Removing a non linear noise trend from data

Question

I have a dataset that looks as follows Now, this is from an actual measurement (of the reflection coefficient of a microchip with four LC resonators connected in a periodic fashion) so there is obviously noise involved. For reference, this is what it should look like in theory (with different numerical parameters)

So there's a few things about the data. First of all there is gaussian(?) noise around the 'mean'. This is not really an issue; I'm only really interested in the positions/sharpness/depths of the resonances, and the noise doesn't complicate it too much. I suppose that if it would be very problematic, one could use Gaussian filter?

But then there's a more problematic issue, and that is also why I put accents around mean. The data is not in a straight line; there is some nonlinear trend over the range of the data. This is what my question is about; getting rid of this trend so that the data is on a straight line. Once that is done, getting it to have its mean at 1 is of course easy, and I'll have a dataset that can be fitted with the model.

My issue is that I don't really know where to start. I did some searching, but either my search terms are terrible or this hasn't really been discussed; it is quite specific after all. I'd like your advice on how to proceed, as personally I don't have any clue. Maybe I should fit the data with some general function? It does seem to be somewhat oscillatory, but not very nicely.

I apologize for not providing some code to generate similar looking data. I would, if I had an idea of how to go about that, but I don't as I'm not sure what kind of function would generate this type of trend. Instead I have uploaded the x-coordinates (Frequency in GHz) and the y-coordinates (complex reflection coefficient) here. As commented, this is a rather strange format for the data if you are not using Mathematica. Here is a different version which should work anywhere, and/but already has only the absolute value of the y-coordinates (as mathematica's way of storing complex numbers is a little nonstandard)

(Dropbox, should be safe and registration free to download it). I should definitely add that what is plotted above is the absolute value of the y-coordinates.

Note that it doesn't require registration. It'll pop up a registration dialog when you open the site, but you can close this, and then you can click the download button (although it seems greyed out, but it isn't). They write this here too dropbox.com/help/20

As for the source of the noise. This is one dataset of many, taken with a Vector Network Analyser. It's connected to the device with SMP cables, which are calibrated before doing the measurement. The calibration is supposed to take care of these trends, but in this case it didn't. So, does theory tell us something.. I'm not sure. I think the noise comes from the cables, and that it might have some oscillatory component to it, but beyond that I can't say to be honest, I have almost no experience with the system. The reason I don't just remeasure is that it is done at liquid helium temperatures, so I'd rather post process

Answer 1

The trend can be perceived as some low frequency signal, so a simple highpass filter should remove it. It will naturally slightly decrease the peaks, but by very little, because such narrow peaks are mainly composed of high frequencies.

Here is some concept code in Python (using the NumPy, SciPy and Matplotlib libraries)

import numpy as np
import scipy.signal as scisig
import matplotlib.pyplot as plt
# assumes it has just two columns with f, r values
f, data = np.loadtxt('CQDataAbs.dat', unpack = True) 
# order4 IIR Butterworth highpass filter coefficients,
# the critical frequency is relative to the Nyquist frequency,
# determining the right value might require some experimentation
# note that using a higher order with such low cuttof will likely
# be numerically unstable, if you need higher orders,
# split it into several lower-order stages applied sequentially
b, a = scisig.butter(4, 0.0003, btype='highpass')  
# forward-backwards filter application with edge padding (mitigates edge defects),
# results in zero phase lag and an effective filter order 8 as it is applied twice
filtered = scisig.filtfilt(b, a, data)  
plt.plot(f, filtered, 'b.')
plt.show()

And here is the result

The shoot-up at the end may indicate that there are some higher frequencies in that area. Perhaps you can just disregard that region during fitting if possible.

If you were interested in the trend itself, you would obtain it by using a lowpass filter instead of a highpass filter. With zero phase lag application the result of highpass filtering should be almost identical (within numerical accuracy) to subtracting this lowpass trend from the original data.

Note that you may find that a different filter type might give you better results. A Butterworth IIR filter (with forward-backwards application to remove phase lag) is usually a good choice when the shape of the signal is important, because the frequency response within the pass-band is maximally flat. You could of course use some large FIR filter for better performance if you have enough processing power. With FIR filters you also don't have to worry about numerical instability that can occur in IIR filters of high orders.

Just in case you think about using a moving average (which is a FIR lowpass filter): That will likely perform terribly in this case, because in each window containing a peak the average will shoot up and subtracting that from the original data will greatly decrease the peaks. From a frequency perspective, the moving average filter does not mitigate higher frequencies very well, which are very significant within the region of the peaks.