0
$\begingroup$

I am having some issues making thin IIR filters that work at the sample rate of my data and was wondering if there is a way to improve my method? I am also curious about the deficiencies in using the method of filtering of applying and FFT, bin-zeroing and the IFFTing back to get the filtered data.

For example I use scipy.signal.iirdesign with scipy.signal.filtfilt to filter a 300KHz signal with a 5MHz sample frequency with bandwidth of 4KHz and a transition width of 1KHz. If I do this I find I get NaNs in the filtered signal, if I reduce the sample frequency by indexing every $N^{th}$ element of the data (e.g. by Signal[::N]) I get a correctly filtered signal if I use N=4 (i.e. sample frequency of 5/4MHz). However this is not ideal as I am having to reduce the sample rate of my data and can therefore get no higher temporal resolution. I can increase this frequency by widening the bandpass but this is undesirable as other frequencies are often present nearby.

I have also found I can create very thin filters by FFTing, zeroing bins and then IFFTing and this works great for a perfect sine wave signal with some noise. However I am interested in extracting a sine wave with random instantaneous phase jumps, and this appears (in simulated tests I have done where i know the true sine wave a-priori) to result in ripples in the amplitude of the signal with the ripples extending furthur the larger the phase shift between the first half of the signal and the next but this is sometimes a preferable method if I can not construct a filter with scipy that will work with the original sample frequency of the data and requires down-sampling.

The following plots demonstrate what I observe.

The blue is the true signal, the red is the filtered signal.

The first plots are a perfect sine wave with no phase shifts or amplitude variation:

IIR:

IIR filter applied to sine wave

IFFT:

IFFT filter applied to sine wave

The IFFT works much better, but this is not that surprising to me since this is a perfect sine wave and is not particularly realistic to observe.

The next plots are of sine wave with a phase shifts of pi/4 in the center with no amplitude variation:

IIR:

IIR filter applied to a sine wave with a central phase shift

IFFT:

IFFT filter applied to a sine wave with a central phase shift

The IIR filter used for this has a bandwidth of 1KHz and a transition width of 0.5KHz, it also has a sample frequency that is 4 times less than the original data. The IFFT filter had a bandwidth of 500KHz and appears to fit what the data is doing much closer, it also benefits from having the same sample frequency as the original data.

It would appear from this that using the IFFT filtering method is vastly superior but I know that it should introduce a sinc function to the time signal and the reason FIR and IIR filters are used to approximate a perfect rectangular window in frequency space is that doing so is a bad idea due to the introduction of this sinc function. What is wrong with IFFT filtering in this way with zero binning and how might I improve my method of bandpass filtering such that I can retain a high sample rate and a reasonable thin frequency window?

Improve this question
$\endgroup$
8
  • 2
    $\begingroup$ There are more problems than the ones you asked about: Note that "instantaneous" phase jumps usually imply a bandwidth of vastly more than 4 kHz. Note that a non-zero-padded FFT/IFFT filter will do circular filtering (e.g. the beginning of the result will be contaminated with some of the impulse response of filtering the end of the data.) $\endgroup$
    – hotpaw2
    Commented Apr 29, 2017 at 0:47
  • $\begingroup$ I see, might there be a different method to extract a signal at a particular frequency that undergoes phase jumps like this? Alternatively, If I can quantify over how much time these phase jumps occur how does this time relate to the bandwidth of the filter I need to use? $\endgroup$
    – SomeRandomPhysicist
    Commented Apr 29, 2017 at 7:16
  • 1
    $\begingroup$ I concur with @hotpaw2. As a 1st order approximation, with a 4 kHz BW source signal, you'll only be able to resolve events down to 250 microseconds at best. At that point there's no harm in frequency shifting to baseband and FIR filtering and decimating in stages, using much more relaxed filters. $\endgroup$
    – Andy Walls
    Commented Apr 29, 2017 at 11:06
  • $\begingroup$ BTW, why are you filtering a 300 kHz BW PSK signal with a 4 KHz wide filter? That destroys information that I suspect you want. $\endgroup$
    – Andy Walls
    Commented Apr 29, 2017 at 11:18
  • 2
    $\begingroup$ Narrow Band-pass filtering will dull sharp events down to blurs that cover 100's of microseconds. You now get to guess where the event may have been hidden inside that blur. $\endgroup$
    – hotpaw2
    Commented Apr 29, 2017 at 15:05

1 Answer 1

Reset to default
2
$\begingroup$

An IFFT of a single non-zero bin (or conjugate mirrored pair) produces a single pure sinusoid (with no phase or amplitude changes). An IFFT of just a few adjacent non-zero FFT bins produces something like a slowly amplitude modulated sinewave (a beat note) possibly very slowly modulated in frequency. Note that none of those IFFT results resemble a sinewave that includes a sudden instantaneous phase shift. The information about the location of a possible sudden phase change is located in all the other FFT bins that an ultra-narrow FFT filter throws away. e.g. you would have to de-zero a vast number of FFT bins to get that change to reappear in the IFFT result. That is why sharp changes have very wide-band spectrums.

This renders your proposed filter (zeroing all but a few FFT bins) nearly useless for determining the location of any sharp phase shift.

Improve this answer
$\endgroup$
4
  • $\begingroup$ ahhh, ok, I see, that makes sense. I presume a narrowband IIR filter will suffer the same phenomena. Is there a good method to recover a narrow frequency band from a noisy signal with it's phase information intact? I've been looking at using Kalman filters for this. $\endgroup$
    – SomeRandomPhysicist
    Commented Apr 29, 2017 at 15:39
  • $\begingroup$ That may be a different enough question to deserve a new separate question. I might start by heterodyning down to IQ baseband. $\endgroup$
    – hotpaw2
    Commented Apr 29, 2017 at 15:46
  • $\begingroup$ This question seems very analogous to "How do I build an optimal PSK receiver?", for which there is a lot of existing literature. Are the phase shifts you're looking for deliberately generated, or are they coming from natural phenomena? $\endgroup$
    – Andy Walls
    Commented Apr 29, 2017 at 16:48
  • $\begingroup$ Coming from natural phenomena, they are stochastic and unpredictable. $\endgroup$
    – SomeRandomPhysicist
    Commented Apr 29, 2017 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.