# Removing a non linear noise trend from data

I have a dataset that looks as follows Now, this is from an actual measurement (of the reflection coefficient of a microchip with four LC resonators connected in a periodic fashion) so there is obviously noise involved. For reference, this is what it should look like in theory (with different numerical parameters)

So there's a few things about the data. First of all there is gaussian(?) noise around the 'mean'. This is not really an issue; I'm only really interested in the positions/sharpness/depths of the resonances, and the noise doesn't complicate it too much. I suppose that if it would be very problematic, one could use Gaussian filter?

But then there's a more problematic issue, and that is also why I put accents around mean. The data is not in a straight line; there is some nonlinear trend over the range of the data. This is what my question is about; getting rid of this trend so that the data is on a straight line. Once that is done, getting it to have its mean at 1 is of course easy, and I'll have a dataset that can be fitted with the model.

My issue is that I don't really know where to start. I did some searching, but either my search terms are terrible or this hasn't really been discussed; it is quite specific after all. I'd like your advice on how to proceed, as personally I don't have any clue. Maybe I should fit the data with some general function? It does seem to be somewhat oscillatory, but not very nicely.

I apologize for not providing some code to generate similar looking data. I would, if I had an idea of how to go about that, but I don't as I'm not sure what kind of function would generate this type of trend. Instead I have uploaded the x-coordinates (Frequency in GHz) and the y-coordinates (complex reflection coefficient) here. As commented, this is a rather strange format for the data if you are not using Mathematica. Here is a different version which should work anywhere, and/but already has only the absolute value of the y-coordinates (as mathematica's way of storing complex numbers is a little nonstandard)

(Dropbox, should be safe and registration free to download it). I should definitely add that what is plotted above is the absolute value of the y-coordinates.

Note that it doesn't require registration. It'll pop up a registration dialog when you open the site, but you can close this, and then you can click the download button (although it seems greyed out, but it isn't). They write this here too dropbox.com/help/20

As for the source of the noise. This is one dataset of many, taken with a Vector Network Analyser. It's connected to the device with SMP cables, which are calibrated before doing the measurement. The calibration is supposed to take care of these trends, but in this case it didn't. So, does theory tell us something.. I'm not sure. I think the noise comes from the cables, and that it might have some oscillatory component to it, but beyond that I can't say to be honest, I have almost no experience with the system. The reason I don't just remeasure is that it is done at liquid helium temperatures, so I'd rather post process

• Could you re-upload the data with some sane and consistent format? It's a mix of fractions 630057/100000, decimal numbers and some strange, float with exponent-like format 1.0151542652853962+0.31807123944301263*I – Ondřej Grover Jun 13 '15 at 18:18
• This is what Mathematica makes out of it, didn't realise that this would of course not work cross-software package. I'll have a look! – user129412 Jun 13 '15 at 18:22
• Should be better now. – user129412 Jun 13 '15 at 18:26

The trend can be perceived as some low frequency signal, so a simple highpass filter should remove it. It will naturally slightly decrease the peaks, but by very little, because such narrow peaks are mainly composed of high frequencies.

Here is some concept code in Python (using the NumPy, SciPy and Matplotlib libraries)

import numpy as np
import scipy.signal as scisig
import matplotlib.pyplot as plt
# assumes it has just two columns with f, r values
f, data = np.loadtxt('CQDataAbs.dat', unpack = True)
# order4 IIR Butterworth highpass filter coefficients,
# the critical frequency is relative to the Nyquist frequency,
# determining the right value might require some experimentation
# note that using a higher order with such low cuttof will likely
# be numerically unstable, if you need higher orders,
# split it into several lower-order stages applied sequentially
b, a = scisig.butter(4, 0.0003, btype='highpass')
# forward-backwards filter application with edge padding (mitigates edge defects),
# results in zero phase lag and an effective filter order 8 as it is applied twice
filtered = scisig.filtfilt(b, a, data)
plt.plot(f, filtered, 'b.')
plt.show()

And here is the result

The shoot-up at the end may indicate that there are some higher frequencies in that area. Perhaps you can just disregard that region during fitting if possible.

If you were interested in the trend itself, you would obtain it by using a lowpass filter instead of a highpass filter. With zero phase lag application the result of highpass filtering should be almost identical (within numerical accuracy) to subtracting this lowpass trend from the original data.

Note that you may find that a different filter type might give you better results. A Butterworth IIR filter (with forward-backwards application to remove phase lag) is usually a good choice when the shape of the signal is important, because the frequency response within the pass-band is maximally flat. You could of course use some large FIR filter for better performance if you have enough processing power. With FIR filters you also don't have to worry about numerical instability that can occur in IIR filters of high orders.

Just in case you think about using a moving average (which is a FIR lowpass filter): That will likely perform terribly in this case, because in each window containing a peak the average will shoot up and subtracting that from the original data will greatly decrease the peaks. From a frequency perspective, the moving average filter does not mitigate higher frequencies very well, which are very significant within the region of the peaks.

• Hm, strange. I thought you could just download it from the link without having a dropbox account. I tried to do it on my other computer (without the program) and it worked fine. I'll re-upload it to a different site in any case, and give your suggestion a try, thanks! – user129412 Jun 13 '15 at 18:07
• Hm, actually, the file is too big for pastebin. I'll look for a free alternative. However, are you sure dropbox doesn't work? Sure, it'll pop up a registration dialog when you open the site, but you can close this, and then you can click the download button (although it seems greyed out, but it isn't). They write this here too dropbox.com/help/20 – user129412 Jun 13 '15 at 18:11
• You're right, those scoundrels just beg for registrations in this way and scared me off. Ok, will download it now. I suggest you add this info about the pop-up within the question. – Ondřej Grover Jun 13 '15 at 18:14
• I have to admit that I don't usually work with Python, but Mathematica instead. However the same principles should apply of course. They have a function HighpassFilter (reference.wolfram.com/language/ref/HighpassFilter.html) which should do something similar, but probably with less options. On top of that, I've never really done any signal processing before (my first real introduction to experimental science, hooray!) so I might be a bit slow in understanding, but I'm trying to read up and such. – user129412 Jun 13 '15 at 18:19
• Looking at the documentation of that function is seem to be implemented as a FIR filter. That means it may require a larger $n$ parameter for the kernel length in case you get suboptimal performance with the default. As long as you are willing to learn and study, everybody will gladly help you ;) – Ondřej Grover Jun 13 '15 at 18:27