I used the filter builder (lowpass) , to cut off the frequency at 0.5 MHz, and thus used a stopband of 0.1pi, and a passband of 0.0125pi. (Sampling Frequency= 80MHz). However, the output was still unsatisfactory, as the spikes at +20 and -20 MHz are still visible. Hence, I reduced the stopband to 0.08pi, from 0.1pi, but then the spikes at 20 and -20 became larger, but in the negative direction.
What could be wrong? And what is the reason for the spikes becoming more negative as the stopband frequency is decreased?
Input Spectrum- Figure 2
Output of filter- Figure 3
Output of filter, with reduced stopband- Figure 4