I recently had to process a very large file to obtain the power spectrum. The file had about 60m measurements (arguably its not that huge, but its getting to that point). Although, in the end (initially I tried with Labview) I managed to find a few tools (Matlab, R) that allowed me to perform the calculation I needed on my PC, I couldn't help but wondering what would be the mathematically correct way to break up that problem, so that I can perform it piecewise.
I guess, my question boils down to:
Can I perform FFT on pieces of a very large signal, and then assemble the results to obtain the FFT of the total signal?
What happens with the periodogram?
The way I interpret the periodogram is that its an fft on a smaller portion of data. So, would I get a much different result if I took a subsets of the signal (overlapping) and performed fft and the created the periodogram?
Update of information on signal No:2 (500kHz):
In advance apologies for changing the sampling rate. I just got some more results with higher sampling rate because I suspect the noise is so high frequency that cannot be captured.
- Signal Description:
The signal comes from measurements in a wind tunnel.
The recorded values are the electrical signal of pitot tube, when the wind tunnel speed is set at a specific value.
The reason, the measurements were taken was because noise was creeping in (probably electrical), the signal and there is a need to investigate the frequencies that contribute so that an appropriate filter is applied (one that does not affect the measurement of wind turbulence).
Data acquisition rate: 500kHz (I just got another set of data with 500kHz)
Data recording rate: 500kHz
Recording duration: approximately 1 min (this is the typical length).
The raw signal is recorded with no preprocessing.
For the FFT the only preprocessing involves removing the mean value from the signal (so as to remove the DC component). (A window is not applied).
- Sample of data
Following are two excerpts (1000 data points from a 500kHz data).
The first is the data with the suspected source of the noise turned on (it is actually the inverter that power the wind tunnel). Although the inverter is turned on there wind speed is set to zero.
The second set, is an excerpt with the inverter turned off (obviously the wind speed is set to zero), at the same scale on the y axis.
The following graph is exactly the same as the second with autoscale on the y axis.
- Longer interval excerpt 0.06[s] (30000 points)
This is to give a better overview of the signal. I switched to points (instead of lines) to give a more clear picture. The following two graphs are again with the electric noise a) on and b) off