See my notebook here

I want to search for irregular time signals in a data set of ~3 500 000 time signals. By eyeballing I have found hundreds of flat and oscillating signals, but just a few that are irregular (see fig. 1). There are 1000-5000 data points/signal, although sometimes data points are missing or there is even a large gap.

My first plan for designing the search for irregular signals was:

  1. Apply Fourier transformation (DFT, FFT o.s.) on all the signals $f(t)$ to get $f(t)=A \cdot \Sigma_k^n e^{ikx} \hat{f}_k$
  2. Afterwards, approximate by setting the smallest contributions $\hat{f}_k$ to 0
  3. Then, calculate PCA on the 3 500 000 sets of $[\hat{f}_1, \hat{f}_2, ... \hat{f}_n]$
  4. there should be only "orthogonal" contributions left, on them I can perform K-means clustering
    • or calculate the neighbours to the irregular signals $f_i(t)$ I have already found by the affinity $|f_i(t)-f(t)|=[ (\hat{f}_{i1} - \hat{f}_{1})^2 + (\hat{f}_{i2} - \hat{f}_{2})^2 + ... + (\hat{f}_{in} - \hat{f}_{n})^2 ]^{1/n}$

Steps 1. and 2. above gave really good results in python by using numpy.fft. Even when I set 98% of the smallest $\hat{f}_k$ to 0 (fig. 2 and 3) the back transformation of the irregular signal into the time domain was very well approximated (fig. 4). But I figured out, that K-means and $|f_i(t)-f(t)|$ might not help me in finding more irregular signals, because they might be not concentrated like the flat or oscillating ones (fig. 5). And I also just learned about DTW (Dynamic Time Warping), that in some cases helps to identify similarity of signals.

My questions are: Is using numpy.fft a legitimate method or will the gaps in the data compromise the results? Is there a better similarity measure than $|f_i(t)-f(t)|$ I presented above?

I am new to FFT, k-means and DWT, so explicit answers are appreciated!







Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.