2
$\begingroup$

I have two pandas DataFrames, dfb and dfv, where dfb has a higher sampling rate than dfv. I want to downsample dfb to align it with dfv. However, I am aware that I need to apply a low-pass filter to avoid aliasing. Can you suggest any improvements to the following function and what is the best way to apply a low-pass filter on a DataFrame before downsampling a time series?

import pandas as pd
from scipy.signal import decimate

# Minimal reproducible example
# Generate example dataframes

lenb =5000
lenv =200

dfb = pd.DataFrame({'a': np.arange(0, lenb,1)}, index=pd.date_range('2022-01-01', periods=lenb, freq='2s'))
dfv = pd.DataFrame({'c': np.arange(0, lenv,1)}, index=pd.date_range('2022-01-01', periods=lenv, freq='10s'))

from scipy.signal import decimate

def newindex(df, ix_new, interp_method='linear'):
    """
    Reindex a DataFrame according to the new index *ix_new* supplied.

    Args:
        df: [pandas DataFrame] The dataframe to be reindexed
        ix_new: [np.array] The new index
        interp_method: [str] Interpolation method to be used; forwarded to `pandas.DataFrame.reindex.interpolate`

    Returns:
        df3: [pandas DataFrame] DataFrame interpolated and reindexed to *ixnew*

    """

    # create combined index from old and new index arrays
    ix_com = np.unique(np.append(df.index, ix_new))

    # sort the combined index (ascending order)
    ix_com.sort()

    # re-index and interpolate over the non-matching points
    df2 = df.reindex(ix_com).interpolate(method=interp_method)

    # drop all the old index points by re-indexing to new index
    df3 = df2.reindex(ix_new)
    #print(len(df3)), print(len(ix_new))
    return df3


def downsample_dataframe(dfb, dfv, filter_order=3):
    freq_dfb  = pd.infer_freq(dfb.index)
    freq_dfv  = pd.infer_freq(dfv.index)
    q         = int(pd.to_timedelta(freq_dfv).total_seconds()/pd.to_timedelta(freq_dfb).total_seconds())

    dfb_downsampled = pd.DataFrame()
    for column_name in dfb.columns:
        signal = dfb[column_name]
        signal = decimate(signal, q, zero_phase=True, axis=0, n=filter_order)
        dfb_downsampled[column_name] = signal

    # Create new index starting from dfb.index[0] with a cadence of freq_dfv
    new_index = pd.date_range(start=dfb.index[0], freq=freq_dfv, periods=len(signal))
    dfb_downsampled.index = new_index

    # Now reindex dfb to index of dfv
    dfb_downsampled = func.newindex(dfb_downsampled, dfv.index)
    return dfb_downsampled



dfb_downsampled = downsample_dataframe(df_B, df_V, filter_order=3)

Are there any suggestions on how to improve this function? Is there a suggested method for aligning two timeseries?

$\endgroup$
2
  • $\begingroup$ is there a reason you do not want to use scipy.signal.decimate? $\endgroup$
    – Jdip
    Jan 26, 2023 at 20:43
  • $\begingroup$ Thank you! No actually I dont have an issue with using scipy. signal.decimate. I am not sure if it is the optimal method though, considering that the final goal is to downsample the high resol timeseries to align it with a lower resolution timeseries but also prevent aliasing. Also I am not sure if the method I suggest is correct and was not able to find some examples online. $\endgroup$
    – Jokerp
    Jan 26, 2023 at 20:59

3 Answers 3

4
$\begingroup$
  • scipy.signal.decimate applies an anti-alias filter before downsampling. See the documentation.

  • Alternatively, you can use the resample_poly function if you need a non-integer decimation factor.


Another thought: decimating can destroy information. If the higher SR signal (in your case, dfb) has frequency content above one-half of the lower SR, that content will be filtered-out by the anti-aliasing filter prior to down-sampling.
If that's the case, you can instead up-sample (interpolate) dfv to match dfb's length. You can use scipy.signal.resample_poly for this as well. Interpolating, contrary to decimating, keeps the original information alive.

$\endgroup$
3
  • $\begingroup$ Thank you! Does scipy.signal.resample apply a low-pass filter? Do you think upsampling would make sense? I thought that it wouldnt make much sense to upsample dfv. In that case would just pd.resample work as well? $\endgroup$
    – Jokerp
    Jan 26, 2023 at 21:28
  • $\begingroup$ Thnak you so much! Final question; pd.resample does NOT apply a downsampling filter right? $\endgroup$
    – Jokerp
    Jan 26, 2023 at 21:46
  • $\begingroup$ I'm not sure that scipy.signal.resample applies the needed filtering. scipy.signal.resample_poly does, though. I'll edit my answer accordingly ;) I do think upsampling makes sense, but decimating could also make sense, depending on the frequency content of your signals. I don't know if pd.resample includes filtering. Just use scipy.signal.resample_poly ! $\endgroup$
    – Jdip
    Jan 26, 2023 at 21:48
2
$\begingroup$

The issue may be a fractional sample delay offset, if the filter used for decimation or interpolation has an even number of coefficients or in the case of decimation specifically, not an integer multiple plus 1 of the decimation rate. This is because the delay in samples for a linear phase filter is $\frac{N-1}{2}$ where $N$ is the number of coefficients. Further when decimating by $D$, the output delay with be the delay prior to downsampling divided by $D$. So for example, if we used a decimation filter of length 33 but down-sampled by 10, the delay after the filter will be 16 samples, which will be a delay of 10/16 or 5/8 of a sample at the output rate!

The solution for the case of decimating is to ensure the decimation filter has an odd number of total coefficient $N$ (I recommend making your own decimation or interpolation filter, using least-squares with the firls command and NOT Parks-McClellan with the 'remez' command, as a constant stop band is also bad for higher order decimation rates), and that $(N-1)/(2D)$ is an integer (which is the output sample delay). Then knowing the output delay just subtract that many samples from the output.

It is easier to land on an integer number of output samples when interpolating; in this case we only need to ensure the interpolation filter used has an odd number of coefficients for the same reasons given above.

$\endgroup$
2
  • $\begingroup$ Thank you so much! Thats quite a few new words to me. Would you mind also providing some suggestions or modifications to my python code? $\endgroup$
    – Jokerp
    Jan 28, 2023 at 23:30
  • $\begingroup$ @Jokerp Unfortunately it's not a matter of simply replacing your code but would get into design details. This question may help you: dsp.stackexchange.com/questions/66410/… $\endgroup$ Jan 29, 2023 at 1:20
2
$\begingroup$

Some comments:

  • What's "aligned" depends on purpose: length can be changed by changing sampling rate (resampling) or duration (padding). If the sequences are to remain pointwise comparable or preserve physical dimensionality, the sampling rates must match. I go in depth in this post.

  • Jdip's answer makes a good point on information. However it also applies in reverse: if the lower SR signal is strongly aliased, upsampling it is a distortion. I'm unsure what gives the cleanest results in that case, but if it's samplings of something with similar SR requirements, then the higher SR signal is an information superset (loosely speaking), which gives leeway - we could downsample it in a way to make its distortions match the lower SR's, which helps with comparisons. But if the goal's to keep the most info, upsampling the lower SR's the way.

$\endgroup$
4
  • $\begingroup$ Thank you for your response. I have modified my function a bit. It seems to me that even with zero_phase=True, there is some phase shift. Do you have any suggestions? Would it make sense to resample this way? Do you have a better suggestion? $\endgroup$
    – Jokerp
    Jan 28, 2023 at 16:58
  • 2
    $\begingroup$ I think it's too simplistic / misleading to say "downsampling the higher SR signal isn't [a distortion]" It most certainly is if the higher SR signal has any energy in the spectral regions that will alias due to down-sampling. If we know that it has been filtered properly and sufficiently then we can simply down-sample. In many applications this isn't assured, including losing precision due to aliasing of the noise floor (including quantization noise), so is an important consideration. $\endgroup$ Jan 28, 2023 at 20:02
  • 1
    $\begingroup$ Apples to applies both require a similar filter (interpolation to remove the images and decimation to remove the aliasing). But there is a practical reason to favor decimation if the spectral requirements allow: for efficiency, power and cost considerations it's an advantage to process signals at the lowest sampling rate possible. $\endgroup$ Jan 28, 2023 at 20:03
  • 1
    $\begingroup$ @DanBoschen You're right, I didn't think it through. In my immediate application, direct subsampling is optimal, but it's more a special case. There may be some generality depending on purpose, unsure. $\endgroup$ Jan 29, 2023 at 10:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.