Why does my stop-band filter attenuate other frequencies?

Question

Here is my code for a two-toned signal, where I use a stop-band to remove the higher tone and then plotted the before (in blue) and after (in red) in the frequency domain after convolving my signal with the filter coefficients.

If you put this code into Matlab you can clearly see that the higher frequency has been successfully removed by the filter, yet for some reason the amplitude of the lower frequency has been cut in half, and the more I increase the # of filter coefficients, the more it just flattens my whole curve, why does this occur? And how can I prevent it so that the stop-band does not propagate outwards? Here is the image and the code:

fSampling = 8000;
tSampling = 1/fSampling;

t = 0:tSampling:0.005;
F0 = 1000;
F1 = 3000;

xt = sin(2*pi*F0*t) + sin(2*pi*F1*t);
ht = fir1(40,.25,'stop');
yt = conv(xt,ht);

fAxis = -4000:125:4000-125;

xF = fft(xt,64);
MagXF = fftshift(abs(xF));

plot(fAxis,MagXF);
hold on

yF = fft(yt,64);
MagYF = fftshift(abs(yF));

plot(fAxis,MagYF,'r')

Answer 1

Filters always have an inherent "roll-off" in their frequency response, because you can't practically realize a pass-band that's a perfect rectangular function. For a low-pass filter, the point where the magnitude of your frequency response drops to -3dB is called the pass-band and everything beyond is called the stop-band (technically, everything beyond the corner frequency, but we'll take the corner frequency to be the -3dB level). How fast your frequency response attenuates beyond the pass-band is dependent on the filter length.

If you look at the frequency response of your filter, ht, you'll see that it drops to -6dB at 1000 Hz:

So it makes sense that the power drops by 6dB after filtering, which you see in your figure as a halving in amplitude.

If you had looked at the documentation for the fir1 function that you used, you would've realized this too (emphasis mine):

B = fir1(N,Wn) designs an N'th order lowpass FIR digital filter and returns the filter coefficients in length N+1 vector B. The cut-off frequency Wn must be between 0 < Wn < 1.0, with 1.0 corresponding to half the sample rate. The filter B is real and has linear phase. The normalized gain of the filter at Wn is -6 dB.

Now to create sharper filters, with responses that get pretty close to a rectangle, you'll have to use IIR filters, which come with their own host of problems with stability, etc., but definitely an option. You can see my answer here for some ideas on implementing a discrete-form 2, second-order section IIR filter that gives very sharp corners. The example there is for a band-pass filter, but you can read up the docs for the functions used there and implement a low-pass version yourself.

Answer 2

It takes time for a filter to "decide" if a signal is just inside or just outside a filter transition. One solution is to just move the filter transition away from any signal of interest, such as halfway between your two test signals, where, in you test case, there is very little signal to distort by an imperfect decision.

Answer 3

Use

ht = fir1(40,.5,'stop');

instead, and you will have very low attenuation at F0, and very high attenuation at F1.