Finding a signal inside another signal

Question

I have a 1D array of 32 bit floating point numbers representing the current draw of a radio operating over a 40 minute span. The number of elements in the array is 29 million and change. There is a signal of interest inside this array that I would like to be able to identify in future signals. I've copied the part of the array that I am interested in and used a python script to generate a parameterized mask. I would like to use this mask to look for similar signals in my 40 minute array. The issue is the amount of time it takes to "sweep" this mask over all the data to see if all the data points fit under the mask.

Generally, I would like to be able to identify signals of interest in large data sets and extract their indices.

I am using Python, numpy, and SciPy.

Images 1 Small signal and mask 2 Full 40 minute radio current draw

Given a signal waveform of interest, what is the most effective way to detect whether that signal exists inside another signal?

Here is a link to the raw data. It's a zipped csv format. The values are time and current draw in amps. Sampling rate of the data: One sample every 8.192E-05 seconds. Signal of interest begins around 917.65 seconds and lasts less than one second.

https://www.dropbox.com/s/rw5yt5zv5rpraoh/ExportedData.zip?dl=0

Answer 1

As I wrote in the comments, this can be solved using cross-correlation.

Take your mask of length $L$, invert it and start performing the convolution between this inverted mask and the original signal (or, directly, use scipy.signal.correlate() in Python with the two signals, the original one and the mask). If you get a peak in your correlation at sample $m$, this means that if you take your mask and overlay it over the original signal such that the first sample of the mask is at the $m$-th sample of the original signal (using the function with the parameter mode='same'), then you will have a good match (i.e., for that instant of time, the signal is pretty similar to the mask). Then you can conclude that from $m$ to $m+L−1$ the signal has a shape that looks like the one of the mask.

I checked the signal you've uploaded and, as you pointed out, following the procedure above one gets some non-desired peaks due to the fact that the original signal has some high values that don't correspond to a similarity in shape with the mask, but nevertheless return a high value of cross-correlation because the sum of the elementwise product is high. This could be solved "filtering out" these non-desired values.

The most similar signal to the mask will be... the mask. Thus, calculate the correlation between the mask and itself (the autocorrelation at $0$). This will give you a number that would represent the maximum possible similarity to the mask. Then, you can ignore those values in the original cross-correlation that are greater than the autocorrelation at $0$, as they surely correspond to those high peaks the original signal has. This should do the trick.