Correct baseline for slow sinusoidal noise signal

Question

I acquired some noise signal (5GHz samplerate, 520 samples) data from a device and recognized that it is not gaussian distributed. There seems to be some slow sinusoidal signal underlying. I tried to visualize it by applying a moving average filter to my waveforms. Here are some examples:

It also shows when the noise data is integrated over an arbitrary region. Instead of getting a gaussian shaped bell curve, two distinct peaks (and a small one) appear, which is what would be expected when having a sinusoidal interference:

Of course I'd like to remove this interference somehow. But I have not come up with a good method yet. I tried to apply a linear regression and then subtracting it from my waveform sample points before integrating a small region of the waveform (as an approximation to a sinewave) but that did not lead to satisfying results. My knowledge about signal processing is very limited. I would appreciate some ideas on how to approach this problem. Or buzzwords to guide me to certain approaches.

Edit 27.03.2023: In depth analysis of the problem

First I want to give some detail about the data acquiring system. It is this digitizer: https://caen.it/products/v1742/

It is based on the so called DSR4 Chip and it enables you to select sample rates of 5, 2.5, 1 or 0.75 GHz. It is used for physics experiments, where fast pulses need to be measured.

Now let me show you some example noise waveforms I recorded at the different sampling frequencies and a record length of 1000 samplepoints (which is the maximum).

I fitted an underlying sine wave to each waveform, representing the assumed sinusoidal distortion. I think it is clearly visible. I now did this for about 10000 waveforms at each sampling frequency and filled histograms with the sine distortion frequency and its phase obtained by the fit.

As one can see the structure of the histograms looks pretty similar for all sampling frequencies (some fits may have failed, resulting in a sine distortion frequency of 0 maybe).

Also, when doing a 2D histogram plotting the frequencies against the phases at the respective sampling frequency the structure is the same for all sampling frequencies (not shown here).

Now doing a 2D histogram plotting all distortion frequencies against the sampling rate you obtain something interesting:

There is a clear linear relationship between the sine distortion frequencies and the sample rate. It gets clearer when one takes the mean of the distortion frequencies:

Normalizing the distortion frequencies by dividing by the corresponding sample frequency you get something almost constant:

So the distortion is somehow generated by the device itself, but what the reason may be, I have no idea.

My actual signals I am measuring look like this:

And the power spectrum:

My knowledge about signal processing is very limited so I'd appreciate any help on how to remove this distortion effectively.

EDIT 29.03.2023

Here are the corresponding noise spectra (linear and log x-scale) of the waveforms from above (also shown again here). They were acquired at the longest data capture possible. I try to find signs of flicker noise. Maybe with some imagination you could argue that there is a declining slope at low frequencies characteristic for 1/f noise seen when using a logscale on the x-axis, but I am not sure if that is the right interpretation. Other than that there really seems to be a very fast distortion as I can clearly see a peak in the middle of each spectrum (not the last one though) - so scaling with sampling freq? But this might be a whole other story.

Answer 1

The consistency in the pattern over multiple measurements suggest a sinusoidal distortion, and the frequency as observed in the course plots appears to be on the order of 7 to 10 MHz. However subsequent updates by the OP showing the spectrum with a much longer capture show no indication of sinusoidal interference. The resulting measurements suggest a non-stationary drift such as that seen with $1/f$ noise and phase noise, commonly in electronics traced to “flicker noise”. Here are some additional links from Wikipedia and Science Direct with further details on flicker noise that can provide further insight for this phenomenon.

A high pass filter with a cutoff of 500 KHz (based on where the spectrum hits the floor in the plot provided by the OP) would eliminate all the non-stationary effects of the increasing noise at lower offset frequencies, but that is only useful if there is no signal of interest below this frequency. We can see from this final spectrum provided by the OP that the noise power will progressively increase at a higher and higher rate as the cutoff of the high pass filter is reduced; in shorter data captures this would give the appearance of a sinusoidal interference- however as confirmed with a longer data capture, such fluctuation is random.

A possible reason for the appearance of a sinusoidal pattern if non-stationary noise (such as phase noise, 1/f noise etc) is present, is that a data capture of finite duration is itself a high pass filter (it takes an infinite amount of time to truly observe DC!). The cutoff of such a highpass filter is approximately $1/T$ Hz where $T$ is the capture duration. Thus if we have a capture duration that results in a cutoff below where the noise floor starts to increase, the combination of the high pass of the observation with the increasing noise results in a peaking around the observation cut-off frequency, which is consistent with seeing approximately 1 cycle of a sinusoidal oscillation in all of the OP's captures, and why such observed frequency curiously scaled by the sampling rate: it wasn't due to the higher sampling rate, but because the OP captured for a shorter duration as the sampling rate increased. This is diagrammed in the spectrum plot below showing how the random noise with a component that increases at decreasing frequency offset could appear to have a sinusoidal component near the inverse of the observation time:

If it was due to clock feed through (if other clocks are used in the system or are operating nearby) this would have been indicated by a tight spurious result (narrow tone) in the spectrum and if identified may be easy to resolve in the hardware directly (judicious shielding and bypass caps etc) or a notch filter in the signal processing such as demonstrated here.

Answer 2

This looks mostly like an DC drift (which is fairly normal for any physical data acquisition system that isn't AC coupled).

I think the easiest thing to try here would be a "DC blocking filter"/"Highpass filter". The exact design of the filter would depend on the requirements of your application (what is your frequency range of interest) and the spectral content of your signal. I suggest doing a spectral analysis first.