Detect valleys of a signal

I have raw values that are coming directly from a sensor (on the fly). These values are coming from a wearable sensor installed on a cows neck. When the cow ruminates (meaning that it chews food that brought back from the 2nd stomach of it), generates some kind of acceleration values that have some kind of specific amplitude and frequency (the chew duration is mostly between 800 msec to 1200 msec). A rumination period is between 30 to 120 seconds every time and the cow ruminates around 10 hours per day.

I do a quick calculation on these values and I end up with a graph like the bellow image.

I would like to be able to detect every time when the signal falls (like the highlighted peaks) as it is a representation of a rumination period. That fall in the graph represents the bolus event of the cow (meaning that the cow just shallow-ed the chewed food and brought back a new one [from the stomach] to chew again).

I tried to do it with a threshold approach (if the signal falls bellow a set threshold), but it fails to operate always correctly. This is because:

1) The signals intensities change slightly during the day and between different sensors eg. On the image, the negative peaks do not fall bellow the black horizontal line which is my threshold. As such I have false detection (not detecting them at all). In case I increase the threshold, I do wrong detections since a lot of other points in the signal might pass it.

2) When the signal is slightly noisy I do wrong detections or no detections at all. As an example, the 9th yellow point of the signal should be somehow detected as valley but it is well above the threshold. Of course it could be mistakenly detected as it has the same amplitude as some nearby points.

Does anybody know how I can detect such points (dips) in time?

Note: These data are coming real-time and they are not stored somewhere. As such I must somehow be able to detect dynamically the fall of the signal without having any knowledge of the future values of it (well I can have some small history of course - lets say last 200 values or something).

Extra info: These data values are actually coming from a 3-axis accelerator sensor. I calculate the magnitudes of them (sqrt(x^2 + y^2 + z^2)) which they deviate around 1g and then I get the Moving Standard Deviation of the last second (62.5 values). That is what is displayed on the graph above. The yellow points are at around 0.01g deviation.

Further: After being asked by some users, I also attach an image of the original signal (only a tiny part of it). With yellow are marked the areas of interest (on the magnitude). The rest 3 graphs are the X,Y,Z (legend exists on the top-right of the graph).

Also, I attach a hyperlink where you can download sample signals in CSV format. The format is as following: 1st, 2nd, 3rd columns are the X, Y and the Z axis accelerometer values. The 4th column is the magnitude of them (sqrt(x^2 + y^2 + z^2)) which you can also manualy calculate if you want so.

https://1drv.ms/f/s!Au5-DmkSvYQzgZlLo2KJ_UBAVeYDWw

There are files with 4 quite good signals. The Good3s.csv contains some small junk at the start of it (some 10000 first values can be discarded). The Good4e.csv has the same at the end. There are also 2 more signal files that are quite noisy and you can parse them if you like hardcore situations.

• You say your threshold is the black line, but these drops aren't passing it. So what's the problem? Please edit your question to clarify what the fail condition is, exactly. Sure, those dips are lower than the rest, but what qualifies them as significant as opposed to other dips? For example, between the eighth and ninth drops (the first ones after 4000), there are quite a few other drops which are only a tiny bit higher than the ninth one, but you haven't highlighted them. You need to be able to quantify when it's a "problem", we can't help you with that.
– Wasabi
Aug 14, 2018 at 18:47
• It would help if you digitized your signal first, if you go through an A/D converter first your signal detection problem will be much easier. Aug 14, 2018 at 20:21
• @WilliamHird What exactly you mean by saying digitized signal? This signal is already in a digital form. In order for me to be able to do calculations (magnitude and standard deviation), that means that I already have the signal in digital format. Aug 14, 2018 at 20:29
• @ekalyvio: As a preface, let me say that I don't want to hurt your feelings, but the waveform shown in your question above aint no digital waveform, I think you would be wise to take some basic electronics courses before posting questions in an open forum like S.E. Just my opinion. Aug 15, 2018 at 2:12
• @WilliamHird As I explained in my question, what you see in the graph is the Standard Deviation of the magnitudes of the signal (not the signal it self) and I am searching for a solution in order to be able to detect these dips that are occurring. I really can not find any reason of what you can not understand of my such simple question. I think that my question is quite complete. Aug 15, 2018 at 8:54

First, you'll probably have better luck posting this on dsp.stackexchange. That's a more specialized group that does stuff like this all the time.

In terms of your problem, here's a couple of options.

One is a machine learning approach. e.g. create a training set by taking a bunch of data and hand marking the points that are good versus bad (like you've done above). Then take an Artificial Neural Network, feed it the training data and teach it to recognize the pattern you want.

Another approach might be to take a Statistical process control approach. Lots of tools there for detecting when something gets out of line.

Another way might a floating threshhold. Compute the mean of the last X samples, and if the signal drops more than Y below that mean, call it bad. could also compute the standard deviation of the last X samples, and if the signal drops more than Z std devs then call it back.

or, potentially just high-pass filter the signal to remove the slowly drifting mean and then you can use a fixed threshhold.

• Thank you very much for your wise suggestions. That is what I am actually searching for. Suggestions to open my mind. The floating filter idea is that I am having for quite some days now. I tried to calculate the moving means of the St.Devs and tried to detect sudden drops of the St.Dev from its mean but in reality, due to the 'lets call it' noise in the signal, the moving avg. fluctuates quite fast for small number of samples (it operates like filtering some of the noise and nothing else). Aug 15, 2018 at 9:11
• If your moving average is fluctuating too fast, you might just need to take more samples. Try to double or quadruple the number of samples used. There are also other filtering options besides moving average (for example butterworth filter). Aug 15, 2018 at 22:53
• I had exactly the same suggestion a couple of weeks ago. I ended asking info for such filter on engineering.stackexchange.com/questions/23021/… As it turns out I might have to do some more extra reading... Aug 16, 2018 at 9:16

I did not read carefully through the whole question as I don't have time now, but have you tried some form of robust peak detection? See e.g., https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.find_peaks.html

Then you can set parameters such as the minimum (and maximum) distance, the prominence, the minimal height. If you look for minima just flip the sign of your signal.

So your basic task is to detect "quiet" parts where the amplitude of some vibrations drops below a threshold.

Can you make assumptions on the length of the "silent" phase until you want to detect it? After analyzing your data I see that most of the silent phases have a duration of ca. 200 samples and repeat every 3000 samples. Can you make assumptions on repetitions? (For example: after a "detection" there can't be another one for 2000 samples")

Below I've added my current code for GNU Octave (which IMHO is very great for prototyping algorithms, design filters, analyze data and so on)

# https://dsp.stackexchange.com/questions/51269/detect-valleys-of-a-signal
# https://en.wikipedia.org/wiki/Standard_deviation#Rapid_calculation_methods

graphics_toolkit fltk

pkg load image  ## for "nlfilter"
pkg load signal ## for "butter"

FS = 62.5;  # in the question "of the last second (62.5 values)"

fns = glob ("*.csv");

k = 1;

## calculate magnitude;
mag = sqrt (sumsq (d(:, 1:3), 2)) - 1;

## remove the slow varying trend
[b, a] = butter (8, 0.01);
mag -= filter (b, a, mag);

## find parts where magitude is below +/- thres
thres = 0.04;
q = abs (mag) < thres;

## ensure, that the first and last sample never is "quiet"
q(1) = q(end) = false;

## starts of quiet part
qs = find(diff(q) == 1);

## ends of quiet part
qe = find(diff(q) == -1);

## length of the quiet parts
len = qe - qs;

## only consider parts longer than 2s
idx = find (len > 2 * FS);

plot (mag)

hold on
# draw line above detection
line ([qs(idx) qe(idx)].', repmat (thres, 2, numel (idx)), "color", "red", "linewidth", 5)
hold off


The red lines are the "detections"

PS: I guess you know https://en.wikipedia.org/wiki/Standard_deviation#Rapid_calculation_methods but just in case not

• Yes. There are for sure 2 assupmptions you can make. 1) The quiet parts can repeat between at least 20 up to 120 seconds (20x62.5 - 120x62.5 samples). 2) The quiet part is from 1 to 5 seconds max (1x62.5 - 5x62.5 samples). I will have a try of the code in Octave today and I will let you know. Aug 19, 2018 at 1:19
• Good solutions but it again suffers the same problem as mine. If the 'thres' variable is too low (eg 0.02) it fails to detect some cases. If the 'thres' variable stays at 0.04 it detects a lot of other garbage. Example with file Good4e.csv. Aug 19, 2018 at 9:28
• Sure, I haven't said this is the perfect, final solution. As an engeneer I think you should tell us what you are really trying to do. The "noise" what you see isn't really white noise, it has multiples of 1.18Hz (1.18, 2.36, 3.54Hz...) in it. It we would know more about the overall system we could perhaps point you more effectively into the right direction. I can just guess that you have some vibrating system (which has it's resonances) and you wan't to detect some event. Perhaps in this case it would make sense to detect if a frequency is missing from that spectrum?
– Andy
Aug 19, 2018 at 9:48
• Well said... It is actually a wearable neck sensor for cows. The graph that you see is the chewing activity of the cow (rumination). The quiet parts are the bolus event of it (when they eat the chewed food and bring back from the stomach an other one to chew). I will update the question on top with more info. Aug 19, 2018 at 10:57