0
$\begingroup$

tl;dr I'm looking for something like Scipy's decimate function, but one that takes in a generator, rather than a Numpy array.

I am turning on and off a light using a pseudo-random sequence of 0s and 1s, something like:

instructions = [0, 1, 1, 0, 1, 0, 1, 1, 1…]

Each 0 or 1 corresponds to ideally 50 milliseconds. The instruction reaching the light is subject to jitter (as much as +- 5ms). However, I have a sensor running at 20 kHz which records when the light receives the next instruction. I assume that the light turns on pretty instantaneously when it receives the instruction.

What I have recorded and saved to disk is a sequence like:

instructions_arrive = [0, 998, 1950, 2999, 3080, 4000, 4900, ...]

Which is the light change event as sampled by the 20 kHz sensor.

I now want to downsample this 20 kHz signal to about 1 kHz. I am aware that I can use something like Scipy's decimate; however, this would require me to decompress the signal to a Numpy array where each element is one of the samples at 20 kHz. Instead, I would like to pass something like a generator, where I calculate the sample values by looking at the instructions_arrive array. This will save me a lot of memory.

My backup option is to use Numpy's memmap feature.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Are you sure you're barking up the right memory tree? Even a full day of 32 bit values (a precision which you probably don't have?) would be like 6.6 GB of memory only, "not very much" on a modern machine you'd run python on. But anyways, yeah, decimation is a process that you can perfectly well stream data through; it only needs the current samples (as many as the filter is long) as state. $\endgroup$
    – Marcus Müller
    Jul 19, 2022 at 19:52
  • $\begingroup$ You are right about the data. I just felt that in principle streaming should be possible, but I couldn't find any feature in SciPy. $\endgroup$
    – Kevin
    Jul 19, 2022 at 19:57
  • 1
    $\begingroup$ sometimes scipy is not the tool of choice, but as you say, with memmap it would work beautifully. If you want more flexibility/higher throughput, try GNU Radio and its Rational Resampler block. $\endgroup$
    – Marcus Müller
    Jul 19, 2022 at 20:00
  • 2
    $\begingroup$ if you want to stay within scipy's realm, just low-pass filter appropriately and throw away samples as needed – the filtering routines should allow you to apply previous filter state to the next signal segment. $\endgroup$
    – Marcus Müller
    Jul 19, 2022 at 20:02
  • $\begingroup$ Would a moving average of the higher sampled result be sufficient? If so this would be super easy to implement as your own generator. I recommend using collections.deque as an efficient 20 sample buffer and yield the sum every 20 samples, OR simply accumulate every 20 samples, yield the result and reset the accumulator (integrate and dump). If you want to push the data rather than pull it, then implement as a simple coroutine generator and use .send() to push in each new sample from the sensor. $\endgroup$
    – Dan Boschen
    Jul 20, 2022 at 0:28

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.