# Finding Signals when the baseline varies greatly by signal set

I have milliwatt consumption data from home appliances. I'm collecting this data in hundreds of households. Based on the machine model, the baseline value for a given household can vary from 0 to 500 milliwatts. The baseline non-zero values vary by 5 to 15 milliwatts and the data is reported every 13 seconds. The vast majority of data is baseline, and there is only signal when people are running the appliance. Perhaps 10 hours a week is real data. The signals are very pronounced, going from baseline to values exceeding 100,000. I have 1/3 of a billion records (growing by several million records a day) so signal detection must be automated.

I really just need, start and end time of the event. We wrote python code that looked for the jump in signal value and return to baseline. The trouble is that each machine has a different baseline value.

Are there simple tools/techniques that can determine from recent context the end of a pronounced signal? My preference is python but I'm happy to use anything that works. Thanks in advance for any feedback.

• could you post a reasonable length snippet of your raw data. One really can't guess much about the statistics from a word description of your problem – user28715 Nov 20 '18 at 4:34

This sounds like a job for a simple high-pass filter, followed by a simple threshold, to me.

1/3 billion records isn't much data if you're used to radio signal sampling rates, so computationally, this should be most definitely doable.

So, with scipy.signal there's a reasonable signal processing library for Python, but I'm from the GNU Radio world, where you can use Python to plug together a signal processing flow graph.

In your case, it'd look like this:

Vector Source -+> high pass filter --> threshold --> [custom block]


which you can click together graphically in GNU Radio companion and let it generate Python, or you can directly write in Python.

The [custom block] is from the top of my head something you'd have to write yourself. You can do that in Python or C++, but Python will be slower to run. I'd still do it in Python for a first prototype:

When the first input is 0, it simply consumes the the input (effectively dropping that data). When it's 1, save the number of the first 1-sample in a row. Then wait until the 1s stop and safe locally something

self._list_of_signals.append( (first_item_number, last_item_number ) )


I think you get the idea.

You feed in the data into the vector source in your python framework, e.g. with the data from your database, and then let the flow graph run, i.e. process all the data it has.

When it's done running, you ask your custom block to give you the list of timestamps. With their first sample number times the sampling rate, you get a time stamp, and then you just put (start, stop) into your database of "trimmed" data.

I would suggest tracking the baseline value using a slow low-pass filter and the instantaneous value using a relatively fast low-pass filter. You can declare an event started when the output of the fast filter exceeds the output of a slow filter by a threshold (additive or multiplicative) and similarly declare the event ended when the fast filter has receded to the value of the slow filter at the start of the event.

You will have to do some empirical tuning of the filter coefficients and thresholds based on the nature of your data. The method is almost trivial to implement in Python, but should give good results.

Another suggestion would be to look into fast-attack/slow-release and slow-attack-fast release filters (used in the audio signal processing community).