0
$\begingroup$

I am working on some signals one of which I have attached below:

enter image description here

What can I do (i.e. functions that I can apply to this signal) to get something like the signal below:

enter image description here

Note how one can draw a line at 0.25 to differentiate between rownumber <13000(or close), and rownumber>13000

-currently, I have tried checking how far each value of the signal is away from the mean, but that didn't seem to help either (i.e., the signal still did not look anywhere near my desired). By the way, I am "not" a signal processing guru, which means I probably would appreciate explanation intended for someone with little experience in signal processing.

Edit: Based on Dan's answer, I decided to clarify something on the question. I am using this signal for a machine learning related research, and the assumption is that I only have the sparse area, but this figure helps give insight into the nature of the dense part of the signal. Hence, in the above paragraph when I said I tried the deviation from the average, I mean as I received new signals (which looks like the dense part shown in the fig), I tried to find out how far it is from the mean of the sparse part of the signal. So I guess my new revised question will be- if you have the sparse part of the signal, and you want to use its nature, shape or amplitude to differentiate between the dense (which at the initial time is unknown) and the sparse signals, how would you do this?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You can apply a simple moving average filter and based on the result of the filter make a decision if you are in the sparse area or dense area, as your average will certainly be larger in your dense area. There is a resolution vs false alarm trade space that you will need to make; your result will be more accurate the longer you can make your moving average filter but your precision in determining where the transition occurred wil be reduced.

So in this sense, to compare to what you tried, instead of determining how far each signal is from the mean, take the mean over shorter intervals (as you move through the sequence), and based on the result of the mean compared to a threshold, decide if you are in the sparse or dense area, and if your decision is sparse, you can clip the result as you have done.

An alternate and simple algorithm would be a window counter, where you allow a certain number of counts (window) between samples above a certain threshold: if the sample comes in above threshold before the count is up you declare a positive, and if not you declare a negative and the count in reset. Further, a certain number of consecutive positives or negatives is required to change state between dense and sparse (this adds hysteresis and avoids a chatter). I don't know the detailed characteristic of your dense to suggest actual count intervals, but this is something you can consider playing with.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks for your reply @DanBoschen Note my updated question. I felt the question needed more clarification. $\endgroup$
    – cidi30bg
    Mar 20, 2017 at 4:52
  • $\begingroup$ I saw your update- if I understand correctly I would still give the same answer: even if you do not have access to the dense part yet, you want to use the nature of the sparse part to determine when a dense part would be present: So I would monitor the mean over the most current samples, and the deviation of this current mean from past means to determine if I am still in sparse or now in a dense area. Also look further into Kalman filtering as a possible more formal approach; I am not certain yet how to frame it in that context but maybe others have good suggestions down that path. $\endgroup$
    – Dan Boschen
    Mar 20, 2017 at 12:18
  • $\begingroup$ Thanks @DanBoschen, yeah the answer still holds for the edit, however, I tried it, and the moving average doesn't seem to solve my problem. It still closely resembles the original signal with only a reduction in amplitude. But thanks for your feedback $\endgroup$
    – cidi30bg
    Mar 20, 2017 at 16:04
  • $\begingroup$ That just means your moving average window is not long enough. The span of the moving average window needs to go over several (up to 10, likely more) of the up down bursts to be meaningful. Also you could also implement an exponential averager which may give you a more desired result; if widening the moving average window improves your result then I can give you more info on the optional exponential averager if interested. $\endgroup$
    – Dan Boschen
    Mar 20, 2017 at 16:08
  • $\begingroup$ Another comment that is handy with the MAF: if your signal content has a particular frequency that repeats, and multiples of that frequency, you can completely null those by choosing your MAF length T in seconds from f= 1/T where f is the frequency to null in Hz. (This will null f, 2f, 3f, etc). Doing an FFT on your signal can reveal if this is the case, or if there is dominant energy to reject beyond a certain frequency which can help you decide how long to make the filter (in order to estimate the average value which is DC you want to filter out the higher frequencies). $\endgroup$
    – Dan Boschen
    Mar 20, 2017 at 16:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.