I have a huge dataset, way too much to be loaded into memory. I need to downsample the data to a new rate. Normally, this would consist of upsampling by N, applying FIR LPF, then decimating by M to achieve the new sampling rate N/M. An important point here is that with this method, your first sample of the input will correspond to the same time as the first sample of the output. Because my dataset is so large, it cannot fit into memory all at once. I am loading in sections of data at a time and resampling each section individually, then stitching them together. The problem is that this creates a discontinuity at the edges of different sections. For example, say that a segment is only five samples and my input signal for the first two segments have times $$ T_{in,1}=\begin{bmatrix}0 & 1 & 2 & 3 & 4 \end{bmatrix}\\ T_{in,2}=\begin{bmatrix}5 & 6 & 7 & 8 & 9\end{bmatrix} $$ My input frequency is 1Hz. Let's say that I upsample by 8, then LPF, then decimate by 9 to achieve an output frequency of .88Hz (period of 1.125 seconds). My output samples at each segment will correspond to times $$ T_{out,1}=\begin{bmatrix}0 & 1.125 & 2.25 & 3.375 & 4.5\end{bmatrix}\\ T_{out,2}=\begin{bmatrix}5 & 6.125 & 7.25 & 8.375 & 9.5\end{bmatrix} $$ Now when I merge them I get $$ T_{out}=\begin{bmatrix}0 & 1.125 & 2.25 & 3.375 & 4.5 & 5 & 6.125 & 7.25 & 8.375 & 9.5\end{bmatrix} $$ Now I have two samples spaced only 0.5 seconds apart in the middle, but I need to have equal spacing between all of my samples. My book has a section on interpolating where they give the following equation. $$ \DeclareMathOperator{\sinc}{sinc} x_m=\sum_{n=-\infty}^{\infty}x_nh(t_m-t_n) $$ Given that I have an array of times that I need interpolated values for, it seems like calculating points based on this equation may be an alternative to the upsampling/decimation method, but I would need to know the impulse response as a function of time. I know that FIR filter impulse responses are typically described as h[n]. Is there a good way to use this equation with a digital filter? How does h[n] relate to h(t) for a digital filter? Is there a better way to solve my problem?
1
-
$\begingroup$ What are your actual input and desired output rates? Also, leaving this here, might give you ideas on how to resample in batch. Pay particular attention to the part on polyphase filters $\endgroup$– JdipCommented Feb 14, 2023 at 6:56
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Normally, this would consist of upsampling by N, applying FIR LPF, then decimating by M
No. The standard method for FIR based rational sample rate conversion is a polyphase filter which is much more efficient in terms of memory and processing than brute force up-sampling + filtering.
I am loading in sections of data at a time and resampling each section individually, then stitching them together.
That can certainly work, but you need to carry over all state variables properly from frame to frame. In most cases this includes overlapping adjacent frames by half or the full filter length but that depends a bit on the details of the implementation.
