# Decimating Long Data in Chunks

I have a very long data up to 8 millions of samples and want to decimate it to order of 1000s. I'm using following Matlab code to do this:

f = numberOfSamples/N; %N is number of points after decimation
p = factor(f); %
temp = rawData; %with length of numberOfSamples
for i = 1:length(p)
temp= decimate(temp,p(i),10); %decimate with 10th order iir filter
end


Number of samples before and after decimation is not known, but the ratio, i.e. decimation number, is an integer for sure(that's given.) In other words, those are input to my program. How can I decimate(including the decimation filter) the data in chunks and have the same result? One chunk computes some part of the decimated signal and the other chunk computes another part, etc. I hope question is clear, any help is appreciated.

• Are you interested in the same results exactly, or you just want do decimate the signal? The difference is with the filters (i.e. if you want to use the exact same filter used by decimate). In any case, since your decimation factor huge, I would recommend multistage decimation. – ThP Apr 9 '15 at 7:30
• I need them the same as much as possible, because other guy is going to use the output in his identification algorithm, which was trained by the data above program produced. The code I posted does the multistage decimation(right?), but I'm not sure how to do in chunks. – Aenid Apr 9 '15 at 11:49

The decimate function in matlab does 2 things. First, the signal is filtered. From your code, and the function doc page, I see that it uses Chebyshev Type I IIR filter of order 10 with normalized cutoff frequency 0.8/p(i) and passband ripple 0.05 dB. Next, the signal is downsampled.
The filter can be created using

[a_lp,b_lp] = cheby1(10,0.05,0.8/p(i));


You should use the initial condition argument in the filter function in order to work in blocks.

Now, the downsampling is a little bit tricky because you have to make sure that the decimation distance is kept between blocks. IMO, the easiest way to do so is to use block size that is multiple of the decimation factor.

N = K*p(i); % K integer
li = 1;
lid = 1;
ri = N;
rid = K;
z = [];
while ri<=numberOfSamples
[filteredData,z] = filter(a_lp,b_lp,rawData(li:ri,z);
decimatedData(lid:rid) = filteredData(1:p(i):end);
li = li + N;
ri = ri + N;
lid = lid + K;
rid = rid + K;
end


(Not sure about the indexes in decimatedData).