1
$\begingroup$

I have a $50\text{Hz}$ data set of temperature variance destruction or $\chi$ in the ocean for 365 days and hence is too bulky to handle. I want to convert it to hourly averages.

Simple question: How do I decide the best averaging method given that I do not have any pre-held notion about the data i.e I do not know what to expect? How can I justify its use for the current data? How do I make sure I do not lose the seasonal, diurnal trends etc. when I average?

My further calculations heavily depend on this processing and also this processed signal will give me spectra of different physics in the ocean. So I want to tread carefully. Any literature, suggestion should do to start off.

PS: In few posts and forums, I have read people commenting on moving averages as a "lousy" way to do it.

$\endgroup$
  • $\begingroup$ Please explain temperature variance destruction. Can't know what to preserve if you don't know what it is. Are the 50Hz samples "noisy" or smooth. If smooth, maybe fitting a line to the data over each hour, storing the mean and slope, which is like a value and a derivative. That reduces your data size and preserves hourly trends $\endgroup$ – Stanley Pawlukiewicz Jul 8 '17 at 14:03
1
$\begingroup$

Essentially, you want to take your original signal and downsample it. Since you need to take care of aliasing before downsampling, you'd need to perform low-pass filtering of the signal.

Your original sampling frequency is $F_S=50Hz$, you want to downsample it to $1/h$ (1 per hour), which is $F_S/(60*60*50)$. So, your anti-aliasing filter needs to have cut-off frequency of $0.5/h$. After filtering you can just take every (50*60*60)th sample of the filtered sequence to have hourly averaged data. Note that downsampling to hourly averages requires, that the actual information in the signal does not vary too quick over time.

Code for some rudimentary filter design is below. Since your cutoff-frequency is so low, you need a very long filter to perform a good filtering. This is in tradeoff against calculation time.

Fs = 50
T = 24*60*60  # data of one day

cutoff = 0.5/(60*60)  # cutoff frequency

# Create some simulated measurement data
t = np.arange(0, T, 1/Fs)
x = np.sin(2*np.pi*t*3/(60*60*24))  # frequency: 3 periods/24 hours
y = x + 0.1*np.random.randn(*x.shape)

b = signal.firwin(15*3611, cutoff, nyq=Fs/2)
a=1

plt.subplot(1,2,1)
plt.plot(b)
plt.subplot(122)
w, h = signal.freqz(b, a, 32*4096)
plt.plot(w, 20*np.log10(abs(h)), '-x')
plt.xlim((0, 3/(60*60)))
plt.axvline(cutoff)

enter image description here

r = np.convolve(y, b)
r = r[:len(y)]

plt.plot(t, y)
plt.plot(t, r)
r_down = r[::50*60*60]
t_down = t[::50*60*60]

plt.stem(t_down, r_down)

enter image description here

As you can see, I get accurate downsampled data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.