I need to plot large oscilloscope data sets. They are too large (100 million Values) to plot all of them in a for loop. If I try to do that it results in a MemoryError. So with the help of Stackoverflow I came up with a solution that isn't pretty but it works.

Original Dataset: enter image description here

To reduce the amount of data to be plotted I can remove all rows of the dataset, that have a y-value between 1 and -1. This can be done comfortably with pandas:

df is the dataset in form of a pandas.DataFrame().

df[1] is the column which is to be examined.

df.drop(df[(df[1] < 1) & (df[1] > -1)].index, inplace = True)

This results in:

enter image description here

This reduces the length of the dataset from 100 million to about 200 thousand. My Question is: Is there a way to reduce data length and keep its original look (prevent the ugly cut-off)?

This stack seems to work differently than Stackoverflow. Pardon me for all wrongdoings.

  • $\begingroup$ Are you simply interested in plotting the waveform? $\endgroup$
    – A_A
    Apr 23 '19 at 9:57
  • $\begingroup$ Yes, but like I wrote, it seems to be too large to handle for matplotlib. It can plot one dataset but as soon as I start looping over several datasets, it throws a MemoryError $\endgroup$ Apr 23 '19 at 10:38
  • $\begingroup$ You didn't say what software you're using to plot. Matlab? Python? $\endgroup$
    – not2qubit
    Apr 23 '19 at 21:39
  • $\begingroup$ @not2qubit like the Topic name says [python3] $\endgroup$ Apr 24 '19 at 6:15

If you are simply interested in plotting the data then any data reduction technique would do,even if it appears to be crude.

Effectively, the plotting function itself will not plot all the data, because the space assigned to the plot has a finite number of $N_x \times N_y$ pixels assigned to it. For example, if your plotting area was $1024 \times 768$, then any waveform with more than 1024 samples would need to somehow be re-scaled or re-sampled. Plotting functions do that internally anyway but always retain a link to the original data to do the whole process again, usually onUpdate.

What you could try to do is to resample your waveform and specifically downsample it.

Downsampling is composed of two steps: The step of anti-alias filtering and the step of decimation.

If you are purely interested in plotting a waveform, you can try to simply decimate it with a low factor (say, retaining 1 sample out of every 5) and then send it for plotting. It will still have more data than required by the plot but it will be of more manageable size.

The reason for opting for pure decimation here is because the filtering step (although necessary) can be costly (in terms of computation) and take a bit longer. If you can afford the filtering time however, I would suggest that you do a proper downsampling.

If you are using Python, you can try this function (or this one) to handle the downsampling step.

It goes without saying here that if you wish to process the signal further, you must downsample it properly.

Finally, another reason why you are running out of memory might be the fact that you are retaining all data in memory even when you are done using it. Therefore, once you have sent a waveform for plotting, try del()eting it, possibly followed by gc.collect() to ensure that that part of memory will become immediately available.

Hope this helps.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.