Frequency content Signal processing of very large files

Question

I recently had to process a very large file to obtain the power spectrum. The file had about 60m measurements (arguably its not that huge, but its getting to that point). Although, in the end (initially I tried with Labview) I managed to find a few tools (Matlab, R) that allowed me to perform the calculation I needed on my PC, I couldn't help but wondering what would be the mathematically correct way to break up that problem, so that I can perform it piecewise.

I guess, my question boils down to:

Can I perform FFT on pieces of a very large signal, and then assemble the results to obtain the FFT of the total signal?

Bonus question:

What happens with the periodogram?

The way I interpret the periodogram is that its an fft on a smaller portion of data. So, would I get a much different result if I took a subsets of the signal (overlapping) and performed fft and the created the periodogram?

---

Update of information on signal No:2 (500kHz):

In advance apologies for changing the sampling rate. I just got some more results with higher sampling rate because I suspect the noise is so high frequency that cannot be captured.

• Signal Description:

The signal comes from measurements in a wind tunnel.

The recorded values are the electrical signal of pitot tube, when the wind tunnel speed is set at a specific value.

The reason, the measurements were taken was because noise was creeping in (probably electrical), the signal and there is a need to investigate the frequencies that contribute so that an appropriate filter is applied (one that does not affect the measurement of wind turbulence).

• Metadata:

Data acquisition rate: 500kHz (I just got another set of data with 500kHz)

Data recording rate: 500kHz

Recording duration: approximately 1 min (this is the typical length).

• Preprocessing:

The raw signal is recorded with no preprocessing.

For the FFT the only preprocessing involves removing the mean value from the signal (so as to remove the DC component). (A window is not applied).

• Sample of data

Following are two excerpts (1000 data points from a 500kHz data).

The first is the data with the suspected source of the noise turned on (it is actually the inverter that power the wind tunnel). Although the inverter is turned on there wind speed is set to zero.

The second set, is an excerpt with the inverter turned off (obviously the wind speed is set to zero), at the same scale on the y axis.

The following graph is exactly the same as the second with autoscale on the y axis.

• Longer interval excerpt 0.06[s] (30000 points)

This is to give a better overview of the signal. I switched to points (instead of lines) to give a more clear picture. The following two graphs are again with the electric noise a) on and b) off

Answer 1

Can I perform FFT on pieces of a very large signal, and then assemble the results to obtain the FFT of the total signal?

This is the basis for the FFT algorithm, in that a large DFT can be more efficiently done as 2 DFT's each half the size (and so on and so on until the only thing left is 2 point DFT's).

(The efficiency comes about since a DFT requires on the order of $4N^2$ multiplications and $4N^2$ additions, while an $N$ pt DFT derived from two $N/2$ pt DFT requires only $4(N/2)^2 + 4N$ multiplications and $4(N/2)^2 + 4N$ additions.)

Below shows the combining approach for splitting the DFT into two separate DFT's, and the same approach can be continued further back dividing the blocks in half each time to whatever minimum size is desired.

The graphic above is given by the mathematical equation below showing the divide and conquer reduction as the basis for the FFT algorithm.

$$\sum_{n=0}^{N-1}x[n]W_N^{nk} = \sum_{r=0}^{N/2-1}x[2r]W_{N/2}^{rk} + W_N^k\sum_{r=0}^{N/2-1}x[2r+1]W_{N/2}^{rk}$$

Where $W_N^k = e^{-j2\pi k/N}$

Alternatively, individual FFT blocks could be averaged, but in this case it would be the magnitudes that would be averaged to result in a spectral smoothing but that would not improve the spectral resolution the way increasing the total time duration of the DFT as the process above provides. Averaging the blocks would be equivalent to adjusting the video bandwidth on a spectrum analyzer (post detection filtering), in that the noise floor would be smoothed, but not reduced, while increasing the total FFT time duration would be equivalent to adjusting the resolution bandwidth on a spectrum analyzer.

Answer 2

I post it as the answer, because the text does not fit into a comment format.

First, I wonder what is a purpose of computing PSD for the signal acquired with the zero wind speed. You are correct in describing the data acquired when the inverter is turned on: the regular peaks separated by 0.25ms intervals betray the default switching frequency (4kHz) of the wind tunnel fan's VFD leaking into a DAQ circuitry.

When VFD is off, only the noise from the Pitot tube's strain gauge is detected: the graph you show is a representative waveform of colored noise. For your (and possible other reader's) convenience I also cite from Investigations on Noise Level in AC- and DC-Bridge Circuits for Sensor Measurement Systems:

... strain gauges show thermal Nyquist noise. When a strain gauge is deformed, the piezoresistive effect causes a small change in resistivity, superimposed on a large base resistance. Using a Wheatstone bridge, the large base resistance is removed in the read out, but it is still active concerning Nyquist noise. ... In most applications, Wheatstone bridges are excited with DC voltage and read out at low frequency. As a result, there will be 1/f noise on top of Nyquist noise.

Alternatively, the excitation can be done with AC 1, avoiding 1/f noise.

The noise waveform shape in your graph recorded with the inverter turned off makes me think that the conditioning electronics of your setup uses DC excitation.

I believe your wind tunnel instrumentation is installed and maintained the best way possible, so what is the purpose of measuring background noises? If it is more for you and about learning something better, as you write in your comment, here are more refs on noise in sensor bridge circuits: Analog's app note and MCC's Data Acquisition article, all these are about electronics, not signal processing.

The things may become more elaborate when you are interested in turbulence spectral analysis or unsteady velocity fluctuations or what else is measured in wind tunnels with the Pitot/Prandtl tubes. This is where the periodogram analysis of recorded (or live) time series may become applicable (only guess, not sure). And certainly you should examine the range of turbulence frequencies, how much it overlaps with flicker (1/f) component frequencies of a measurement circuitry noise and if it can cause the detection and analysis problems.