Decimating Long Data in Chunks

Question

I have a very long data up to 8 millions of samples and want to decimate it to order of 1000s. I'm using following Matlab code to do this:

f = numberOfSamples/N; %N is number of points after decimation
p = factor(f); % 
temp = rawData; %with length of numberOfSamples
for i = 1:length(p)
    temp= decimate(temp,p(i),10); %decimate with 10th order iir filter
end

Number of samples before and after decimation is not known, but the ratio, i.e. decimation number, is an integer for sure(that's given.) In other words, those are input to my program. How can I decimate(including the decimation filter) the data in chunks and have the same result? One chunk computes some part of the decimated signal and the other chunk computes another part, etc. I hope question is clear, any help is appreciated.

Answer 1

The decimate function in matlab does 2 things. First, the signal is filtered. From your code, and the function doc page, I see that it uses Chebyshev Type I IIR filter of order 10 with normalized cutoff frequency 0.8/p(i) and passband ripple 0.05 dB. Next, the signal is downsampled.
The filter can be created using

[a_lp,b_lp] = cheby1(10,0.05,0.8/p(i));  

You should use the initial condition argument in the filter function in order to work in blocks.

Now, the downsampling is a little bit tricky because you have to make sure that the decimation distance is kept between blocks. IMO, the easiest way to do so is to use block size that is multiple of the decimation factor.

Here is something to start with:

N = K*p(i); % K integer
li = 1;
lid = 1;
ri = N;
rid = K;
z = [];
while ri<=numberOfSamples
  [filteredData,z] = filter(a_lp,b_lp,rawData(li:ri,z);
  decimatedData(lid:rid) = filteredData(1:p(i):end);
  li = li + N;
  ri = ri + N;
  lid = lid + K;
  rid = rid + K;
end

(Not sure about the indexes in decimatedData).