I am working with a STFT spectral processing toolkit I wrote in C, now focusing on a module for spectral filtering. In its first implementation, the module simply multiplied an user drawn spectral envelope (made of magnitudes only, i.e all phases set to zero) by the incoming spectral frames.
Unfortunately I recently realized (being the whole framework intended for music usage) that with certain shapes I get an annoying signal distortion. After investigating (I had not a formal training in digital filters, so be gentle please...) I understood the cause of the problem.
The issue can be seen in two different ways but they are likely two sides of a same money. Simply I create peak distortion. Since in DFT peaks usually share many bins (more so if using a low frame resolution), multiplying the spectrum by a curve can alter/distort the peak shape if too abrupt, causing all possible artifacts.
Another way of seeing the problem is that even by multiplying the spectrum by a real filter curve (i.e with zeroed imaginaries) I generate an unwanted circular convolution.
Focusing on the first way of seeing the problem as I described above, I invented a trick: I identify every peak and copy it entirely to destination by scaling it of a same amount dictated by the filter curve value at its tip. This trick does wonders, but fails miserably in presence of overlapping peaks, i.e two or many peaks merging in a single apparent peak (this happens often when using frame sizes < 2048 or so, as one often has to do to reduce latency, and/or in presence of very low frequency harmonic signals. I posted another question here at this proposal about detecting overlapping peaks btw).
So I had to try another approach aimed at avoiding circular convolution - please read on, this will be my actual question.
I used an overlap-save approach: both the input and the filter spectra are doubled in size and zero padded on the right, to allow room for ringing, i.e to create a proper linear convolution which won't go wrap at the opposite end. As a premise, I discovered experimentally that I had to shift the filter kernel by half frame for proper results and to warrant an impulse response centered at the impulse itself (I never found this fact mentioned anywhere btw... strange). The method would work as expected without causing peak distortions but now I am facing an unexpected comb filter effect. After further investigation, I discovered the culprit: many filter spectral shapes produce kernels which don't fade to zero at their ends but which are either solid (non fading) sinusoidal blocks or have a kind of dc offset (look like truncated, as if they needed a larger framesize) Such "truncated" kernels will naturally produce a comb filtering effect.
What else I am ignoring here ? Is actually spectral filtering a so complex task to accomplish ? Any suggestion or elaboration is welcome. Thanks.
Addition: I have the intuition that I should perhaps apply a window (like a rised cosine) to the kernel to get rid of this annoyance, to have it fading gently to zero at its ends..