0
$\begingroup$

I'm trying to extract statistical features from power spectral density values in Python. My data is actigraphy data with sampling rate 1/60 Hz (once per minute). This is a sample from my data, "activity" column are actual measurements.

I calculate periodogram without any problem. However, values that I get are huge, e.g. mean of periodogram values is about 10 milions, while variance is about 10e16. Is this normal? Original measurements are typically from range [1, 1000].

However, with large values I can manage, the real problem is that for kurtosis (and only for it) I get infinities and error:

RuntimeWarning: overflow encountered in square s = s**2

If I change my sampling rate to 1 Hz (which is not true, but it's a default value), then I get regular numbers. What can be the cause of this behavior?

Is there a way to safely calculate statistics from periodogram in such cases?

My code:

x = df["activity"].values
psd = scipy.signal.periodogram(x, fs=(1/60))[1]

features = {
    "minimum": np.min(X),
    "maximum": np.max(X),
    "mean": np.mean(X),
    "median": np.median(X),
    "variance": np.var(X),
    "kurtosis": sp.stats.kurtosis(X),
    "skewness": sp.stats.skew(X),
    "coeff_of_var": sp.stats.variation(X),
    "iqr": sp.stats.iqr(X),
    "trimmed_mean": sp.stats.trim_mean(X, proportiontocut=0.1),
    "entropy": sp.stats.entropy(X, base=2),
}
$\endgroup$
1
$\begingroup$

Is this normal?

Yes.

The way you call periodogram() it simply does a single FFT of the entire vector and squares the result to get the power. The mean of your vector is about 150 and it's 24000 sample long, i.e. the FFT at DC is 24000*150. When you square this you get something in the order of $13\cdot 10^{12}$. periodogram() also scales to spectral density (and not bin power), so it divides by the sampling frequency, so your DC will approach $10^{15}$

That's probably not what you intended (just guessing here), but that's what you do. Consider using pwelch() perhaps with a much shorter FFT length?

The spectrum doesn't look particularly interesting. Since it's all positive values there is a huge spike at DC but other than that it just looks like slightly colored noise.

$\endgroup$
1
  • $\begingroup$ Thanks, now I understand my results! Could you specify what parameters do you suggest for Welch's method for such data? I have window type to choose, length of segment, percentage of overlap (default 50%) and length of FFT itself. I have no experience with this method at all. In particular the length of FFT is of questions, since I think I can use default hyperparameters for other parameters. Or, on the other hand, would using power spectrum instead of power spectral density help? I think I wouldn't divide by frequency then. $\endgroup$
    – qalis
    Aug 26 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.