# Frequency content Signal processing of very large files

I recently had to process a very large file to obtain the power spectrum. The file had about 60m measurements (arguably its not that huge, but its getting to that point). Although, in the end (initially I tried with Labview) I managed to find a few tools (Matlab, R) that allowed me to perform the calculation I needed on my PC, I couldn't help but wondering what would be the mathematically correct way to break up that problem, so that I can perform it piecewise.

I guess, my question boils down to:

Can I perform FFT on pieces of a very large signal, and then assemble the results to obtain the FFT of the total signal?

Bonus question:

What happens with the periodogram?

The way I interpret the periodogram is that its an fft on a smaller portion of data. So, would I get a much different result if I took a subsets of the signal (overlapping) and performed fft and the created the periodogram?

Update of information on signal No:2 (500kHz):

In advance apologies for changing the sampling rate. I just got some more results with higher sampling rate because I suspect the noise is so high frequency that cannot be captured.

• Signal Description:

The signal comes from measurements in a wind tunnel.

The recorded values are the electrical signal of pitot tube, when the wind tunnel speed is set at a specific value.

The reason, the measurements were taken was because noise was creeping in (probably electrical), the signal and there is a need to investigate the frequencies that contribute so that an appropriate filter is applied (one that does not affect the measurement of wind turbulence).

Data acquisition rate: 500kHz (I just got another set of data with 500kHz)

Data recording rate: 500kHz

Recording duration: approximately 1 min (this is the typical length).

• Preprocessing:

The raw signal is recorded with no preprocessing.

For the FFT the only preprocessing involves removing the mean value from the signal (so as to remove the DC component). (A window is not applied).

• Sample of data

Following are two excerpts (1000 data points from a 500kHz data).

The first is the data with the suspected source of the noise turned on (it is actually the inverter that power the wind tunnel). Although the inverter is turned on there wind speed is set to zero.

The second set, is an excerpt with the inverter turned off (obviously the wind speed is set to zero), at the same scale on the y axis.

The following graph is exactly the same as the second with autoscale on the y axis.

• Longer interval excerpt 0.06[s] (30000 points)

This is to give a better overview of the signal. I switched to points (instead of lines) to give a more clear picture. The following two graphs are again with the electric noise a) on and b) off

• Start with investigation into the nature of a recorded signal: does it have features of regularity, is it steady-state, stochastic, maybe it is a Gaussian process, or what else. Use all the data you have: the recorded signal, metadata, the signal origin, pre-processing etc. When you know, or partly know the details of your signal nature, and you still have questions about what techniques are preferable to obtain power spectra of multimillion-sample signals, communicate this knowledge here to help us make the answers more focused Apr 13 '21 at 8:14
• @V.V.T Hi, I made an update on the information I got on the signal. I am not certain, the exact difference of meaning of "steady-state", "stochastic", or "Gaussian process", so I preferred not to attempt to qualify the signal with those terms. If you (or anybody else) could provide more guidance on what else could be helpful, I would be very happy to provide either graphs of other calculations. Apr 13 '21 at 10:29
• Related. Key question is: what's the lowest frequency, or longest period you seek to capture? This sets the shortest direct FFT you can afford to take. Apr 13 '21 at 11:16
• @OverLordGoldDragon What you are asking (the longest period) is not entirely clear to me (I am helping out a fellow with wind experience but without the computing knowledge to perform analysis). I suspect it should be in the order of the 0.1 of a Hz with respect to the wind (hence the 1 min recording time), but with respect to the electrical noise I am completely at loss. May be a plot will help? Apr 13 '21 at 11:31
• Noise of such low freq should be minimal to nonexistent (relative to higher). If you seek 0.1Hz, that's 1/0.1 = 10 secs, so shortest FFT is fs*T = 100kHz * 10s = 1e6 samples. An excerpt of raw data would be most useful to others here. Apr 13 '21 at 11:54

Can I perform FFT on pieces of a very large signal, and then assemble the results to obtain the FFT of the total signal?

This is the basis for the FFT algorithm, in that a large DFT can be more efficiently done as 2 DFT's each half the size (and so on and so on until the only thing left is 2 point DFT's).

(The efficiency comes about since a DFT requires on the order of $$4N^2$$ multiplications and $$4N^2$$ additions, while an $$N$$ pt DFT derived from two $$N/2$$ pt DFT requires only $$4(N/2)^2 + 4N$$ multiplications and $$4(N/2)^2 + 4N$$ additions.)

Below shows the combining approach for splitting the DFT into two separate DFT's, and the same approach can be continued further back dividing the blocks in half each time to whatever minimum size is desired.

The graphic above is given by the mathematical equation below showing the divide and conquer reduction as the basis for the FFT algorithm.

$$\sum_{n=0}^{N-1}x[n]W_N^{nk} = \sum_{r=0}^{N/2-1}x[2r]W_{N/2}^{rk} + W_N^k\sum_{r=0}^{N/2-1}x[2r+1]W_{N/2}^{rk}$$

Where $$W_N^k = e^{-j2\pi k/N}$$

Alternatively, individual FFT blocks could be averaged, but in this case it would be the magnitudes that would be averaged to result in a spectral smoothing but that would not improve the spectral resolution the way increasing the total time duration of the DFT as the process above provides. Averaging the blocks would be equivalent to adjusting the video bandwidth on a spectrum analyzer (post detection filtering), in that the noise floor would be smoothed, but not reduced, while increasing the total FFT time duration would be equivalent to adjusting the resolution bandwidth on a spectrum analyzer.

• If I understood your post correctly, basically what I could do (in the limiting case) is read in 2 values at the time (at different locations of the signal) and then from the results I could build the whole thing. This is an insight I never managed to grasp when I tried understanding/implementing an FFT for myself about 20 years ago. Apr 14 '21 at 9:28
• @VVT oh I did that in error —- there are log2(N) stages but each stage requires all samples Apr 14 '21 at 16:54
• @NMech I made a previous comment now deleted suggesting that each stage in the DFT pipeline can be less than the total number of samples; although you can process in smaller DFT blocks all the way 2 to DFT’s with regards the the pipeline stages you will ultimately need to build the full length before iterating to the next pipeline.... I suggest not implementing the FFT from scratch but to use the property described to combine two half size DFTs (that are computed using the FFT) or quarter size etc Apr 14 '21 at 16:57

I post it as the answer, because the text does not fit into a comment format.

First, I wonder what is a purpose of computing PSD for the signal acquired with the zero wind speed. You are correct in describing the data acquired when the inverter is turned on: the regular peaks separated by 0.25ms intervals betray the default switching frequency (4kHz) of the wind tunnel fan's VFD leaking into a DAQ circuitry.

When VFD is off, only the noise from the Pitot tube's strain gauge is detected: the graph you show is a representative waveform of colored noise. For your (and possible other reader's) convenience I also cite from Investigations on Noise Level in AC- and DC-Bridge Circuits for Sensor Measurement Systems:

... strain gauges show thermal Nyquist noise. When a strain gauge is deformed, the piezoresistive effect causes a small change in resistivity, superimposed on a large base resistance. Using a Wheatstone bridge, the large base resistance is removed in the read out, but it is still active concerning Nyquist noise. ... In most applications, Wheatstone bridges are excited with DC voltage and read out at low frequency. As a result, there will be 1/f noise on top of Nyquist noise.

Alternatively, the excitation can be done with AC 1, avoiding 1/f noise.

The noise waveform shape in your graph recorded with the inverter turned off makes me think that the conditioning electronics of your setup uses DC excitation.

I believe your wind tunnel instrumentation is installed and maintained the best way possible, so what is the purpose of measuring background noises? If it is more for you and about learning something better, as you write in your comment, here are more refs on noise in sensor bridge circuits: Analog's app note and MCC's Data Acquisition article, all these are about electronics, not signal processing.

The things may become more elaborate when you are interested in turbulence spectral analysis or unsteady velocity fluctuations or what else is measured in wind tunnels with the Pitot/Prandtl tubes. This is where the periodogram analysis of recorded (or live) time series may become applicable (only guess, not sure). And certainly you should examine the range of turbulence frequencies, how much it overlaps with flicker (1/f) component frequencies of a measurement circuitry noise and if it can cause the detection and analysis problems.

• I am amazed at how much you were able to infer about the setup just from looking at the results. I will need to look into the references you provided. Although your answer was not directly related to my question, it was much more helpful and insightful regarding the problem of my friend. Apr 14 '21 at 9:31
• Because, it will take me a while to digest what you've just provided, the bottom line is (if I understood correctly ) that I should propose that we change to AC excitation of the voltage. Apr 14 '21 at 9:34
• also to answer to your question, there are other measurements at different velocities. The reason I presented this the zero velocity, was because with the inverter off, I could not provide similar/repeatable condition measurements. Apr 14 '21 at 9:36