0
$\begingroup$

I am starting a project on this Keggle dataset containing EEG registrations (sampled at 128 Hz) of several subjects. What I am really interested in is the final machine learning model, but still, to fit it I need to reproduce the preprocessing of the EEG signal necessary to features engineering reported in the associated published paper.

In the paper the authors write that they have performed a short-time Fourier transform using time window of 15 seconds and 1024 fast discrete Fourier Transform, using a Blackman windowing.

I would like to reproduce this using the scipy.stftfunction in Python.

From my understanding of function documentation, the relevant parameters for me are: x, the raw timeseries, fs, the sampling frequency here equal to 128, window here equal to blackman, nperseg equal to 1920 (i.e. 15 * 128) and nfft, equal to 1024that is the number reported in the paper. So, the call that I am trying to use is

f, t, z = stft(timeseries, fs = 128, window = "blackman", nperseg = 1920, nfft = 1024)

However, if I try this call, I got an error message as follow:

ValueError: nfft must be greater than or equal to nperseg.

If I remove the nfft parameter, the call run without error, but I am not confident with the output. For example the time points in t go from 0 to 600 (this check out, since I have 10 minutes of eeg) but in step of 7.5 seconds and not 15 as I would have expected given the time window chosen. Moreover, the frequencies sampled reported in f go from 0 to 64 Hz (again, this check out), but in step of 0.06 Hzand not in step of 0.125 Hz as reported in the paper.

Can someone point out how I got this wrong and how can I replicate the published preprocessing with the specified parameters ?

$\endgroup$
0
$\begingroup$

Here is how an Short Term Fourier Transform works

  1. You break your time series up in multiple segments. Each segment is nperseg samples long.
  2. The segments overlap by noverlap samples. Your time resolution then is nperseg - noverlap , i.e. that is the time distance between neighboring segments. It's often called the "hop size"
  3. You multiply the segment with window. Window and segment must have the same length
  4. You perform an FFT over the windowed segment. The length of the FFT must be equal or longer than that of the segment (otherwise you would be truncating samples). If the FFT length is longer, the data will be zero-padded. The frequency resolution is the sample rate divided by the FFT length (not the segment length).

Technically, overlap (or hop size) and FFT length allow you to dial in the time and frequency resolution independently from each other and from the segment size. However, there is a limit to how "physically reasonable" that can be: higher resolutions just provide interpolation, not more data.

Hence the "standard" way of doing this is to use an FFT length that's equal to the segment length and an overlap of 50%.

As stated in point 4, the FFT must be equal or longer than your segment. That's your error. The article you cite is behind a pay wall, so we can't tell what they are doing. Maybe you can quote the relevant portion of the text (provided the copyright terms allow that).

using time window of 15 seconds and 1024 fast discrete Fourier Transform

That doesn't make much sense. That would mean they are skipping a lot of data. Maybe the just focus on the actual heart beats and ignore the time in between?

$\endgroup$
3
  • $\begingroup$ Thanks for your comprehensive answer. Indeed this is EEG and not ECG, so it would not made much sense for them to skip a lot of data, as there's no "silence" in the data. You can find here the relevant snapshots of the methods. Indeed I noticed that they first talk about 15 seconds windows, but then they state that they have a FFT per second. It would be great if you can help me clarify this. $\endgroup$
    – fednem
    Oct 5 at 16:33
  • $\begingroup$ Sorry, I confused EEG and ECG. My bad. I took a look at their explanation and it does not make sense to me. Using a 1024 FFT with a window length of 1920 is NOT a good idea and there is also no mentioning of overlap (despite having a Blackman window). Either something is missing in their description or they did a very questionable spectral analysis (to put it mildly) and I wouldn't trust the results, at least not without a detailed analysis of how that questionable spectral analysis effects the outcome. Maybe you can reach out to the authors ? $\endgroup$
    – Hilmar
    Oct 5 at 16:58
  • $\begingroup$ thanks for taking the tame to look at it, I will try to ask directly to the authors. $\endgroup$
    – fednem
    Oct 6 at 6:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.