3
$\begingroup$

My question is regarding how one could/should smooth a rather noisy power spectrum obtained from measurements. There are various methods one can use (a moving average, binning, etc) and I am not sure what the correct approach would be, given the situation I will describe.

Although at first sight it might sound like I am asking about what kind of periodogram to use (see for example this question with excellent answers), I think my situation might be a bit different. What I have is a bunch of complex valued autocorrelation functions (ACF): $n$ datasets of length $m = 8192$, centered at $t = 0$, taken with a sampling rate of 1 nanosecond (so 1 GHz sampling rate).

While the specifics are not too relevant to the question, they can also not be changed; the data is obtained using an FPGA that receives a complex value every nanosecond, and it internally calculates the autocorrelation function using a bunch of steps involving DFT's implemented in the binary hardware.

So the data I have I am essentially stuck with. Given this set of ACFs, I'd now like to get their power spectra, which is of course just the Fourier transform. Here is an example: enter image description here

There are a few things to see. First, there are two peaks between -100 and 0 MHz. These are the main objects of interest; they change their width and height in each dataset, and I'd like to integrate them to see how their area changes. As they are now they are rather noisy, and I think they would benefit from smoothing.

The second thing is the major peak between 100 and 200 MHz; this peak is always there (always at the exact same frequency with this resolution), and it should be kept. That means that it shouldn't be smoothed, nor should its neighbours take it into account when smoothing. I suppose one solution is just to take it out, smooth, and put it back in.

Finally there is some noise background that varies from about 0 to 0.002. This I am not interested in; it doesn't have to go away completely, but it should still be flattened out so that I can see it as a sort of static background.

So, taking this all into account, I am looking for some advice on what to do. A simple approach would be binning; splitting the data up into $k$ segments of length $l$ and taking the central moment of each segment as my new dataset. Of course it reduces the number of points. Another alternative is using a moving average, which replaces each datapoint by its r-range average. And then there are a million more smoothing methods; I just don't know what is more suitable.

And I suppose this is the main question. Given that I have a dataset that looks like shown above, with the features I described, how should I go about narrowing down what type of smoothing method is most suited? The main deal, as I mentioned before, is finding the area of the two peaks between -100 and 0. Also making the data look a bit more pleasant is not unimportant. The moving average does all this, but it feels so.. primitive.

For reference, the dataset (as comma seperated values) can be found here. Note that this is already the PSD; the Fourier transform of my original datasets.

Improve this question
$\endgroup$
6
  • $\begingroup$ Did you have a complex input signal or is the frequency axis of the plot wrong, because normally it should be symmetric around 0 Hz. And to have you tried Welch's method? $\endgroup$
    – fibonatic
    Jul 26, 2016 at 14:32
  • $\begingroup$ It is a complex input signal yes. I thought I wrote this, but it seems like I didn't. As for Welch's method, isn't that for when you have a time trace, not an ACF? $\endgroup$
    – user129412
    Jul 26, 2016 at 15:12
  • $\begingroup$ You are correct I skipped over that part. How much does the width and height of the two peaks change between the different ACF's? Is it possible that it is caused by noise or small variations of the measurement setup? And how many ACF's do you have? Because I might try to calculate the average at each frequency. $\endgroup$
    – fibonatic
    Jul 26, 2016 at 16:00
  • $\begingroup$ So, as for the two peaks. It is actually rather intricate; they are a feature of some physical process we are measuring. Without going into the specifics, atoms are absorbing light and then re-emitting it at different frequencies (incoherent emission). We are changing some parameters and seeing how the incoherent emission changes, and we want to quantify that. They change substantially; one dies out and the other becomes about twice as big and wide. As for the ACF, we have one for each parameter setting, but they are already averaged 64 million times. $\endgroup$
    – user129412
    Jul 26, 2016 at 16:18
  • $\begingroup$ Didn't fit into the last comment, but there is definitely a noise background; this is this 0.002 that you see. We're already taking a lot of measures to reduce this as far as possible (the 64M averages, and also subtracting the background without any signal applied) $\endgroup$
    – user129412
    Jul 26, 2016 at 16:19

1 Answer 1

Reset to default
1
$\begingroup$

Two things: 1) Unless there's some kind of synchronicity between your spectrum analyzer oscillator and the signal source, the spectrum should be fairly symmetrical around 0Hz (except for the AWGN). So I strongly suspect something wrong there, because it isn't symmetrical. 2) Since (I assume) you are getting more than one spectral sweep, for each frequency bin take something like the average of the bin over multiple sweeps. For example, a given frequency bin could be the average of the last several sweeps for that bin, or its an IIR of FIR of the bin history. Do NOT smooth across frequency unless you only have one sweep of data and nothing else.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thanks for the reply. The spectrum is not symmetric as we are dealing with complex data; I forgot to make this clear in the question. As for averaging, the hardware that collects the data takes the FFT and averages this 64 million times already. It then IFFTs and gives me the ACFs. So in essence what you describe is already being done at this stage I am afraid, and we end up with the single sweep of data. $\endgroup$
    – user129412
    Jul 26, 2016 at 16:22
  • $\begingroup$ Do you really mean that this is the result of 64 million pass's on each point? Really! $\endgroup$
    – rrogers
    Jul 27, 2016 at 0:46
  • $\begingroup$ So apparent 'signal' here is on the order of ~70dB below the noise floor?! More reason to suspect it is just correlated systemic noise. Also, I still say any asymmetry in a correlated signal magnitude is a sign of systemic (synchronous) correlated noise--unless your real/imaginary data is ~true~ IQ (for example, its based on a local clock that we can presume IS synchronized with the signal, or you're measuring true current vs. voltage in a system that allows any phase of those things, etc.) $\endgroup$
    – Digiproc
    Jul 27, 2016 at 9:17
  • $\begingroup$ Yes, it is 64M. It is from a physics experiment done with single photons, and things are just rather sensitive. We send in a signal at -65dBm which gets attenuated down by another 60 dB before it reaches the object of interest. After that it goes through a rather extensive amplification chain. As for the noise floor, that I can't say. The y-axis in this figure is arbitrary at this point. But it is certainly not systemic noise; the outcome is as expected from theory and occurs only at specific parameter settings. I understand the scepticism though! Without the context it is all rather suspect. $\endgroup$
    – user129412
    Jul 27, 2016 at 14:51
  • $\begingroup$ I could of course include this in the main post, and looking back it might have made sense. I just didn't want to make it too much about the specifics of the experiment. As for the data, it is true IQ. Everything is on the same trigger, and on the same rubidium clock. $\endgroup$
    – user129412
    Jul 27, 2016 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.