3
$\begingroup$

I have a 1D array of 32 bit floating point numbers representing the current draw of a radio operating over a 40 minute span. The number of elements in the array is 29 million and change. There is a signal of interest inside this array that I would like to be able to identify in future signals. I've copied the part of the array that I am interested in and used a python script to generate a parameterized mask. I would like to use this mask to look for similar signals in my 40 minute array. The issue is the amount of time it takes to "sweep" this mask over all the data to see if all the data points fit under the mask.

Generally, I would like to be able to identify signals of interest in large data sets and extract their indices.

I am using Python, numpy, and SciPy.

Images 1 Small signal and mask 2 Full 40 minute radio current draw Small signal and mask

Full 40 minute radio current draw

Given a signal waveform of interest, what is the most effective way to detect whether that signal exists inside another signal?

Correlated signal

Here is a link to the raw data. It's a zipped csv format. The values are time and current draw in amps. Sampling rate of the data: One sample every 8.192E-05 seconds. Signal of interest begins around 917.65 seconds and lasts less than one second.

https://www.dropbox.com/s/rw5yt5zv5rpraoh/ExportedData.zip?dl=0

$\endgroup$
  • $\begingroup$ Welcome to DSP.SE! What about correlation? $\endgroup$ – Tendero Aug 30 '18 at 15:16
  • $\begingroup$ I'm not sure how to interpret the results running the correlation on the mask and the original signal. correlated signal I am looking to extract the index of array where the signal matches. $\endgroup$ – pyth0n Aug 30 '18 at 15:41
  • $\begingroup$ The index in which the correlation achieves its maximum gives you information about the position in which the mask and the original signal have a good match. I can't give you a more thorough answer right now, but I'll try to write one later. Do you have any signal processing background (i.e. already know what correlation stands for and what it means)? $\endgroup$ – Tendero Aug 30 '18 at 15:44
  • $\begingroup$ I have some signals processing background. I have an Electrical Engineering degree and MSEE. I understand (kinda) what the correlation is doing with the two signals, just not sure how to use what it is telling me. $\endgroup$ – pyth0n Aug 30 '18 at 15:50
  • $\begingroup$ Well, basically, you take your mask of length $L$, invert it and start performing the convolution between this inverted mask and the original signal. If you get a peak in your correlation at sample $n_0$, this means that if you take your mask and overlay it over the original signal such that the first sample of the mask is at the $n_0$-th sample of the original signal, then you will have a good match. This means that, for that instant of time, the signal is pretty similar to the mask. Then you can conclude that from $n_0$ to $n_0+L-1$ the signal has a shape that looks like the one of the mask. $\endgroup$ – Tendero Aug 30 '18 at 15:58
3
$\begingroup$

As I wrote in the comments, this can be solved using cross-correlation.

Take your mask of length $L$, invert it and start performing the convolution between this inverted mask and the original signal (or, directly, use scipy.signal.correlate() in Python with the two signals, the original one and the mask). If you get a peak in your correlation at sample $m$, this means that if you take your mask and overlay it over the original signal such that the first sample of the mask is at the $m$-th sample of the original signal (using the function with the parameter mode='same'), then you will have a good match (i.e., for that instant of time, the signal is pretty similar to the mask). Then you can conclude that from $m$ to $m+L−1$ the signal has a shape that looks like the one of the mask.

I checked the signal you've uploaded and, as you pointed out, following the procedure above one gets some non-desired peaks due to the fact that the original signal has some high values that don't correspond to a similarity in shape with the mask, but nevertheless return a high value of cross-correlation because the sum of the elementwise product is high. This could be solved "filtering out" these non-desired values.

The most similar signal to the mask will be... the mask. Thus, calculate the correlation between the mask and itself (the autocorrelation at $0$). This will give you a number that would represent the maximum possible similarity to the mask. Then, you can ignore those values in the original cross-correlation that are greater than the autocorrelation at $0$, as they surely correspond to those high peaks the original signal has. This should do the trick.

$\endgroup$
  • $\begingroup$ Thank you for spending so much time to explain everything. People like you make the world a better place. $\endgroup$ – pyth0n Aug 31 '18 at 18:09
  • $\begingroup$ @pyth0n Glad to help! $\endgroup$ – Tendero Aug 31 '18 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.