How to align timeseries by decimating while preventing aliasing?

Question

I have two pandas DataFrames, dfb and dfv, where dfb has a higher sampling rate than dfv. I want to downsample dfb to align it with dfv. However, I am aware that I need to apply a low-pass filter to avoid aliasing. Can you suggest any improvements to the following function and what is the best way to apply a low-pass filter on a DataFrame before downsampling a time series?

import pandas as pd
from scipy.signal import decimate

# Minimal reproducible example
# Generate example dataframes

lenb =5000
lenv =200

dfb = pd.DataFrame({'a': np.arange(0, lenb,1)}, index=pd.date_range('2022-01-01', periods=lenb, freq='2s'))
dfv = pd.DataFrame({'c': np.arange(0, lenv,1)}, index=pd.date_range('2022-01-01', periods=lenv, freq='10s'))

from scipy.signal import decimate

def newindex(df, ix_new, interp_method='linear'):
    """
    Reindex a DataFrame according to the new index *ix_new* supplied.

    Args:
        df: [pandas DataFrame] The dataframe to be reindexed
        ix_new: [np.array] The new index
        interp_method: [str] Interpolation method to be used; forwarded to `pandas.DataFrame.reindex.interpolate`

    Returns:
        df3: [pandas DataFrame] DataFrame interpolated and reindexed to *ixnew*

    """

    # create combined index from old and new index arrays
    ix_com = np.unique(np.append(df.index, ix_new))

    # sort the combined index (ascending order)
    ix_com.sort()

    # re-index and interpolate over the non-matching points
    df2 = df.reindex(ix_com).interpolate(method=interp_method)

    # drop all the old index points by re-indexing to new index
    df3 = df2.reindex(ix_new)
    #print(len(df3)), print(len(ix_new))
    return df3


def downsample_dataframe(dfb, dfv, filter_order=3):
    freq_dfb  = pd.infer_freq(dfb.index)
    freq_dfv  = pd.infer_freq(dfv.index)
    q         = int(pd.to_timedelta(freq_dfv).total_seconds()/pd.to_timedelta(freq_dfb).total_seconds())

    dfb_downsampled = pd.DataFrame()
    for column_name in dfb.columns:
        signal = dfb[column_name]
        signal = decimate(signal, q, zero_phase=True, axis=0, n=filter_order)
        dfb_downsampled[column_name] = signal

    # Create new index starting from dfb.index[0] with a cadence of freq_dfv
    new_index = pd.date_range(start=dfb.index[0], freq=freq_dfv, periods=len(signal))
    dfb_downsampled.index = new_index

    # Now reindex dfb to index of dfv
    dfb_downsampled = func.newindex(dfb_downsampled, dfv.index)
    return dfb_downsampled



dfb_downsampled = downsample_dataframe(df_B, df_V, filter_order=3)

Are there any suggestions on how to improve this function? Is there a suggested method for aligning two timeseries?

Answer 1

• scipy.signal.decimate applies an anti-alias filter before downsampling. See the documentation.

• Alternatively, you can use the resample_poly function if you need a non-integer decimation factor.

---

Another thought: decimating can destroy information. If the higher SR signal (in your case, dfb) has frequency content above one-half of the lower SR, that content will be filtered-out by the anti-aliasing filter prior to down-sampling.
If that's the case, you can instead up-sample (interpolate) dfv to match dfb's length. You can use scipy.signal.resample_poly for this as well. Interpolating, contrary to decimating, keeps the original information alive.

Answer 2

The issue may be a fractional sample delay offset, if the filter used for decimation or interpolation has an even number of coefficients or in the case of decimation specifically, not an integer multiple plus 1 of the decimation rate. This is because the delay in samples for a linear phase filter is $\frac{N-1}{2}$ where $N$ is the number of coefficients. Further when decimating by $D$, the output delay with be the delay prior to downsampling divided by $D$. So for example, if we used a decimation filter of length 33 but down-sampled by 10, the delay after the filter will be 16 samples, which will be a delay of 10/16 or 5/8 of a sample at the output rate!

The solution for the case of decimating is to ensure the decimation filter has an odd number of total coefficient $N$ (I recommend making your own decimation or interpolation filter, using least-squares with the firls command and NOT Parks-McClellan with the 'remez' command, as a constant stop band is also bad for higher order decimation rates), and that $(N-1)/(2D)$ is an integer (which is the output sample delay). Then knowing the output delay just subtract that many samples from the output.

It is easier to land on an integer number of output samples when interpolating; in this case we only need to ensure the interpolation filter used has an odd number of coefficients for the same reasons given above.

Answer 3

Some comments:

• What's "aligned" depends on purpose: length can be changed by changing sampling rate (resampling) or duration (padding). If the sequences are to remain pointwise comparable or preserve physical dimensionality, the sampling rates must match. I go in depth in this post.

• Jdip's answer makes a good point on information. However it also applies in reverse: if the lower SR signal is strongly aliased, upsampling it is a distortion. I'm unsure what gives the cleanest results in that case, but if it's samplings of something with similar SR requirements, then the higher SR signal is an information superset (loosely speaking), which gives leeway - we could downsample it in a way to make its distortions match the lower SR's, which helps with comparisons. But if the goal's to keep the most info, upsampling the lower SR's the way.