# How to align timeseries by decimating while preventing aliasing?

I have two pandas DataFrames, dfb and dfv, where dfb has a higher sampling rate than dfv. I want to downsample dfb to align it with dfv. However, I am aware that I need to apply a low-pass filter to avoid aliasing. Can you suggest any improvements to the following function and what is the best way to apply a low-pass filter on a DataFrame before downsampling a time series?

import pandas as pd
from scipy.signal import decimate

# Minimal reproducible example
# Generate example dataframes

lenb =5000
lenv =200

dfb = pd.DataFrame({'a': np.arange(0, lenb,1)}, index=pd.date_range('2022-01-01', periods=lenb, freq='2s'))
dfv = pd.DataFrame({'c': np.arange(0, lenv,1)}, index=pd.date_range('2022-01-01', periods=lenv, freq='10s'))

from scipy.signal import decimate

def newindex(df, ix_new, interp_method='linear'):
"""
Reindex a DataFrame according to the new index *ix_new* supplied.

Args:
df: [pandas DataFrame] The dataframe to be reindexed
ix_new: [np.array] The new index
interp_method: [str] Interpolation method to be used; forwarded to pandas.DataFrame.reindex.interpolate

Returns:
df3: [pandas DataFrame] DataFrame interpolated and reindexed to *ixnew*

"""

# create combined index from old and new index arrays
ix_com = np.unique(np.append(df.index, ix_new))

# sort the combined index (ascending order)
ix_com.sort()

# re-index and interpolate over the non-matching points
df2 = df.reindex(ix_com).interpolate(method=interp_method)

# drop all the old index points by re-indexing to new index
df3 = df2.reindex(ix_new)
#print(len(df3)), print(len(ix_new))
return df3

def downsample_dataframe(dfb, dfv, filter_order=3):
freq_dfb  = pd.infer_freq(dfb.index)
freq_dfv  = pd.infer_freq(dfv.index)
q         = int(pd.to_timedelta(freq_dfv).total_seconds()/pd.to_timedelta(freq_dfb).total_seconds())

dfb_downsampled = pd.DataFrame()
for column_name in dfb.columns:
signal = dfb[column_name]
signal = decimate(signal, q, zero_phase=True, axis=0, n=filter_order)
dfb_downsampled[column_name] = signal

# Create new index starting from dfb.index[0] with a cadence of freq_dfv
new_index = pd.date_range(start=dfb.index[0], freq=freq_dfv, periods=len(signal))
dfb_downsampled.index = new_index

# Now reindex dfb to index of dfv
dfb_downsampled = func.newindex(dfb_downsampled, dfv.index)
return dfb_downsampled

dfb_downsampled = downsample_dataframe(df_B, df_V, filter_order=3)


Are there any suggestions on how to improve this function? Is there a suggested method for aligning two timeseries?

• is there a reason you do not want to use scipy.signal.decimate?
– Jdip
Jan 26 at 20:43
• Thank you! No actually I dont have an issue with using scipy. signal.decimate. I am not sure if it is the optimal method though, considering that the final goal is to downsample the high resol timeseries to align it with a lower resolution timeseries but also prevent aliasing. Also I am not sure if the method I suggest is correct and was not able to find some examples online. Jan 26 at 20:59

• scipy.signal.decimate applies an anti-alias filter before downsampling. See the documentation.

• Alternatively, you can use the resample_poly function if you need a non-integer decimation factor.

Another thought: decimating can destroy information. If the higher SR signal (in your case, dfb) has frequency content above one-half of the lower SR, that content will be filtered-out by the anti-aliasing filter prior to down-sampling.
If that's the case, you can instead up-sample (interpolate) dfv to match dfb's length. You can use scipy.signal.resample_poly for this as well. Interpolating, contrary to decimating, keeps the original information alive.

• Thank you! Does scipy.signal.resample apply a low-pass filter? Do you think upsampling would make sense? I thought that it wouldnt make much sense to upsample dfv. In that case would just pd.resample work as well? Jan 26 at 21:28
• Thnak you so much! Final question; pd.resample does NOT apply a downsampling filter right? Jan 26 at 21:46
• I'm not sure that scipy.signal.resample applies the needed filtering. scipy.signal.resample_poly does, though. I'll edit my answer accordingly ;) I do think upsampling makes sense, but decimating could also make sense, depending on the frequency content of your signals. I don't know if pd.resample includes filtering. Just use scipy.signal.resample_poly !
– Jdip
Jan 26 at 21:48

The issue may be a fractional sample delay offset, if the filter used for decimation or interpolation has an even number of coefficients or in the case of decimation specifically, not an integer multiple plus 1 of the decimation rate. This is because the delay in samples for a linear phase filter is $$\frac{N-1}{2}$$ where $$N$$ is the number of coefficients. Further when decimating by $$D$$, the output delay with be the delay prior to downsampling divided by $$D$$. So for example, if we used a decimation filter of length 33 but down-sampled by 10, the delay after the filter will be 16 samples, which will be a delay of 10/16 or 5/8 of a sample at the output rate!

The solution for the case of decimating is to ensure the decimation filter has an odd number of total coefficient $$N$$ (I recommend making your own decimation or interpolation filter, using least-squares with the firls command and NOT Parks-McClellan with the 'remez' command, as a constant stop band is also bad for higher order decimation rates), and that $$(N-1)/(2D)$$ is an integer (which is the output sample delay). Then knowing the output delay just subtract that many samples from the output.

It is easier to land on an integer number of output samples when interpolating; in this case we only need to ensure the interpolation filter used has an odd number of coefficients for the same reasons given above.

• Thank you so much! Thats quite a few new words to me. Would you mind also providing some suggestions or modifications to my python code? Jan 28 at 23:30
• @Jokerp Unfortunately it's not a matter of simply replacing your code but would get into design details. This question may help you: dsp.stackexchange.com/questions/66410/… Jan 29 at 1:20