0
$\begingroup$

See my notebook here

I want to search for irregular time signals in a data set of ~3 500 000 time signals. By eyeballing I have found hundreds of flat and oscillating signals, but just a few that are irregular (see fig. 1). There are 1000-5000 data points/signal, although sometimes data points are missing or there is even a large gap.

My first plan for designing the search for irregular signals was:

  1. Apply Fourier transformation (DFT, FFT o.s.) on all the signals $f(t)$ to get $f(t)=A \cdot \Sigma_k^n e^{ikx} \hat{f}_k$
  2. Afterwards, approximate by setting the smallest contributions $\hat{f}_k$ to 0
  3. Then, calculate PCA on the 3 500 000 sets of $[\hat{f}_1, \hat{f}_2, ... \hat{f}_n]$
  4. there should be only "orthogonal" contributions left, on them I can perform K-means clustering
    • or calculate the neighbours to the irregular signals $f_i(t)$ I have already found by the affinity $|f_i(t)-f(t)|=[ (\hat{f}_{i1} - \hat{f}_{1})^2 + (\hat{f}_{i2} - \hat{f}_{2})^2 + ... + (\hat{f}_{in} - \hat{f}_{n})^2 ]^{1/n}$

Steps 1. and 2. above gave really good results in python by using numpy.fft. Even when I set 98% of the smallest $\hat{f}_k$ to 0 (fig. 2 and 3) the back transformation of the irregular signal into the time domain was very well approximated (fig. 4). But I figured out, that K-means and $|f_i(t)-f(t)|$ might not help me in finding more irregular signals, because they might be not concentrated like the flat or oscillating ones (fig. 5). And I also just learned about DTW (Dynamic Time Warping), that in some cases helps to identify similarity of signals.

My questions are: Is using numpy.fft a legitimate method or will the gaps in the data compromise the results? Is there a better similarity measure than $|f_i(t)-f(t)|$ I presented above?

I am new to FFT, k-means and DWT, so explicit answers are appreciated!

fig1

fig2

fig3

fig4

fig5

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.