# How to measure the agreement between to curves?

I have values (plotted below) of expected RSSI values over time that I would like to compare with my measured RSSI values. What I was looking for was a way to quantify it so I can change parameters and be able to compare/contrast different approaches.

It is a hard problem in my mind because I don't know how to compare the signals and yet take into account the large-scale (overall shape) and small-scale (individual fluctuations) of signal.

For instance, here is a plot of one set of signals:

In the image I can see that the red measure signal roughly follows the model, but it also does an OK job of simulating some of the sinusoidal qualities of the model (in some places). Any thoughts?

<> In response to pichenettes' comments (which seem reasonable), I took a diff of the two values and plotted the abs(fft(diff)) and got this:

I am not sure what to make of that though. Since we don't have any actual freqs, I am not sure how to scale the axis, and then if I did, what metric would you use?

• What about computing something like the square error in different frequency ranges (or breaking down the different into different frequency bands)? In the lower frequency range it'll measure the overall tracking abilities - irrespective of fast bumps. In the higher frequency, it'll measure the ability to track abrupt changes irrespectively of larger DC errors. Commented Apr 10, 2013 at 14:22
• OK, I added a new plot to the original post (as an edit) to show the fft(real(diff)), but I am not quite sure what to make of it. Commented Apr 10, 2013 at 17:48
• I would smooth both of them first; then you get a very good agreement (assuming that's the result you want). P.S. I always recommend sharing the data you used to make your plots so we can help more easily.
– Emre
Commented Apr 10, 2013 at 18:37
• How much do you care about matching the phase at higher frequencies? The sense I get is that you might want to compare the time-domain signal directly (after a low-pass filter), then compare frequency-domain for higher frequencies, possibly looking only at magnitude and ignoring the phase. Commented May 16, 2013 at 22:48
• @toozie21 do you already know the time locations at which the signal properties change? e.g. 8 ms, 17ms .. so on. Commented Aug 12, 2013 at 2:20

## 3 Answers

If the signals are not aligned, yet you have a clue that they more or less "mean" the same thing, or refer to similar data, you could use dynamic time warping (DTW) algorithm to obtain a better correspondence (then simply taking the value at the same location). You could measure RMS, MSE or whatever you like, using those correspondences. For DTW, you might want to check: http://en.wikipedia.org/wiki/Dynamic_time_warping

A good way to elaborate this idea is utilized in Earth Mover Distance (EMD), which computes the minimal effort to bring the signals to alignment, as a distance measure. EMD is presented here: http://ai.stanford.edu/~rubner/emd/default.htm

EMD gives you a direct distance, which you might use for further analysis.

I'm using RMS of the error vector as a measure. Since I'm dealing with complex modulation schemes I'm also using EVM as a measure.

I would probably combine a few approaches. First I would smooth both waveforms or do a spline interpolation to remove the large scale variations. You might want to combine another step after that of a cross-correlation to line them up, supposing time bias doesn't matter to you. Once you've got the cross-correlation peak you could even interpolate that peak on a parabola shape and then re-sample one waveform to match the other. I'd compute the RMSE between the two waveforms at that point and produce one metric indicating the slowly varying delta.

After that I'd subtract the interpolated value from the original so the deviations on small time scales are normalized. From there you could try to RMSE them against each other or even just compute a variance of each, to get a notion of how much "noise" you have about the slowly varying waveform, depending on what you need and what you're actually trying to measure.