# Estimating discrete filter attenuation error

I recently faced a problem that I found quite harder than I thought it would be to solve in the biginning. Here's a little bit of context : I have a library of filters, designed in house. I would like to script some unit test for each of them.

One of the unit test I want, is to validate the attenuation at the cutoff frequency. For the sake of the question, let's talk about a simple 1st order IIR low pass. In the unit test, I generate a stimulus (a sine wave) that I feed to my filter. Then I take the output of the filter, compare the variance of both signals (i.e. the power).

Something like :

fc = 100
stimulus = sin(2*pi*fc*time)
output = IIR1st_filter(stimulus,fc)
Pin = var(stimulus)
Pout = var(output)
attenuation = 20*log10(Pout/Pin)
target_attenuation = 20*log10(1/2)
tolerance = 0.005
if abs(attenuation-target_attenuation) > tolerance
test.failure();
end


Good news is that it works fine! The problem is the tolerance. I selected that value of 0.005dB arbitrarily because it seemed to be big enough to withstand the precision error in my scenario. Still, I am uncomfortable to select such value emprically while I am sure there is some literature out there that could help me select a tolerance more rigorously.

I know that there is a bunch of parameters to be considered.

• Initial state of the filter : I need to give some time to the filter to start attenuating properly. Until now, I remove the first sine cycle to chop off the majority of the error
• Discrete domain : The fact that I am working with a discrete signal surely makes the power ratio a little off the target of -6.0205999132...dB
• Length of acquisition : The more data I have, the more precise I am.
• Float precision : Most likely to be negligible, but still
• Other?

So my questions are:

• How can I take all of these parameters in account to select a the smallest tolerance possible without measuring with my thumb?
• How long of data should I discard before my filter is in steady state. 1 cycle? Equivalent of the biggest group delay in my bandwidth?
• Is it worth it to try to be as rigorous as this, or would an experienced DSP designer just use his thumb? Seems that for complex filters, having some theory to backup the numbers might be a good idea.

I work under Matlab/Simulink, mainly with 32 bits float samples.

Thank you!

• If you're working in floating point, why don' t you just evaluate the filter's frequency response at the cut-off frequency? – Matt L. Jan 19 '18 at 9:04
• Well, the filte rimplementation is done in Simulink. So to evaluate the frequency response, I would need to scan the model and extract the coefficients. Since I do not have the DSP toolbox, I would need to evaluate the frequency response by myself, which is more work than looking at the signal variance. Last thing, the unit tests are black box tests, whch check the end result, not the design itself, If somebody change the implementation of the filter, I may not catch his mistake. – Pier-Yves Lessard Jan 19 '18 at 15:27