Model-based Kalman filtering a noisy signal

Question

In a healthcare application, I need to calculate urine flow by differentiating the mass of urine emitted by a person over time. The measuring instrument consists of a load-cell under a fluid container, with its signal being sampled at 2kHz.

My original dataset is here.

The signals are noisy due to the load cell, and drippings over the fluid container, with an FFT showing an exponential-like decay in the magnitude of higher frequencies. Due to this noise, when I try to differentiate the volume signal to obtain the flow signal, I get a very rough signal.

I have already tried smoothing the volume signal in many ways (savatski-golay, gaussian convolution, butterworth low-pass), but by differentiating the smoothed signal I get either a still very wavy signal, or one that is so smooth that it no longer contains the expected characteristics of the flow profile.

I studied a bit about differentiation of noisy signals so as to estimate, say, velocity and acceleration directly from the position signal, and the main suggestion is to use a Kalman filter, which I never got to the point of making work, or even understanding conceptually.

There are some factors influencing on the signal characteristics, some of them due to the physiological model being represented, others not:

1. Physiological component: the flow is expected to be determined by muscle contractility (mostly involuntary bladder muscles) and urethral shape (affected by static structures like prosthate, and active structures like pelvic floor muscles);
2. Principle of measurement component: the fluid mechanics of the urine falling into the container, especially "turbulent" elements like drippings;
3. Instrumentation component: The load-cell/strain-gauge noise, with a much higher frequency than the two previous factors.

So my goal would be to implement a Kalman filter algorithm (I can use Python or C#) that gives me estimates for both Volume and Flow, based on the measured Volume, and that models the physiological component while "removing" the turbulence of the impact on the container (e.g. drippings) and the instrumentation noise.

I don't expect to get a ready answer, but since this is a bit beyond my skill level, I'm quite lost regarding where to start.

Answer 1

I ended up with a quite satisfactory solution, not by using a Kalman filter, but by using the Savitzky-Golay differentiating filter.

The algorithm is described more or less like this:

1. In a for loop, apply a running window to get a segment of the unfiltered volume signal around a range of given time instants, as measured by the load cell;
2. For each segment, perform a polynomial interpolation. I am using quadratic;
3. Differentiate the found polynomial;
4. Evaluate the derivative at the given time instant;

This gives me a very natural-looking flow curve, based on the premise that the flow cannot physically contain high frequencies. So the "model" ended up being a well-chosen low-pass filter, with the bonus that I can differentiate a continuous function (the smoothing polynomial) instead of the discrete differentiation that tends to amplify the noise.