1
$\begingroup$

I try to use a denoising algorithm to process an audio in streaming mode. What I've done is spliting the audio into 0.5s long segments and use denoising algorithm respectively.Then use np.cat or np.hstack to merge them together.But in this way,the resulting audio will have a lot of spikes.And the resulting spectrum will have a lot of vertical lines.

How can I improve this?

the original noisy audio is below: enter image description here

If I don't use the cross-fade,the result will be like this: enter image description here

I've tried to use sqrt-based cross fade to do this.like the following:

    sqrt_fade_in = np.sqrt(np.linspace(0, 1, overlap_len))
    sqrt_fade_out = np.sqrt(np.linspace(1, 0, overlap_len))
    for i in range(num):
    noisy_data = noisy_data.squeeze(0)
    if i == 0:
        data = noisy_data[:chunk_size]
    else:
        data = noisy_data[i * (chunk_size)-overlap_len:(i + 1) * chunk_size-overlap_len]  #silde:chunk_size-overlap_len
    data = data.unsqueeze(0)
    start_time = time.time()
    result_data = enhance_one_track(data)
    #--------------overlap with cross-fade--------------#
    fade_out = overlap_data[-overlap_len:]*sqrt_fade_out
    fade_in = result_data[:overlap_len]*sqrt_fade_in
    print(time.time() - start_time)
    if i == 0:
        result[:chunk_size] = result_data
    else:
        result[(i-1) * chunk_size:(i + 1) * chunk_size-overlap_len]=np.concatenate(
            (overlap_data[:chunk_size-overlap_len],fade_in+fade_out,result_data[overlap_len:chunk_size]),axis=None
        )
    overlap_data = result_data

But it didn't work out well.

enter image description here After that, I also tried to zero fill the audio before and then feed it into the algorithm for processing, and finally cut off the filled part when connecting the audio.

def zero_padding(noisy_data,overlap_rate,chunk_size=8000,padding_rate=0.1):
    # # The slices are fed to the algorithm for processing
    overlap_len = int(chunk_size*overlap_rate)
    num = int(noisy_data.shape[1] / chunk_size)
    result = np.zeros((noisy_data.shape[1],))
    # Sets the length of the padding data
    padding_len = int(chunk_size*padding_rate)
    for i in range(num):
        noisy_data = noisy_data.squeeze(0)
        data = noisy_data[i * (chunk_size):(i + 1) * chunk_size] 
        data = pad_audio(data,padding_len)
        data = data.unsqueeze(0)
        start_time = time.time()
        result_data = enhance_one_track(data)
        # result_data = hpss(result_data)
        # plt.plot(result_data)
        # plt.show()
        result_data = result_data[padding_len:-padding_len]
        # result_data = hpss(result_data)
        print(time.time() - start_time)
        # merge
        result[i * (chunk_size):(i + 1) * chunk_size]=result_data
    return result

The end result is as follows: enter image description here I have also tried to combine the above two ways: enter image description here

These are the efforts I've made, but they don't work very well.

$\endgroup$
13
  • 1
    $\begingroup$ I have no idea what your denoising alg does, so my question is, for all of the frequency components left over after denoising, are they phase aligned with what they were before the denoising took place? $\endgroup$ Sep 3 at 2:49
  • 3
    $\begingroup$ Could you add a plot of the orginal noisy signal? Also, to systematically assess the performance, use a clean audio signal to which you add noise yourself. This way you have a clean reference and can calculate all kinds of metrics, e.g. SNR. $\endgroup$
    – Max
    Sep 4 at 8:40
  • 3
    $\begingroup$ The version with no crossfading doesn't look any worse, than the one with it. You could try some nonelinear stuff, like taking 100 samples at each side of the cut and do some interpolating to align them, eliminating the clicks. Overlapping does not seem to help, since the denoising alg is nonlinear itself. $\endgroup$
    – Max
    Sep 4 at 10:11
  • 2
    $\begingroup$ I would support to try something like @Max suggest. You only need to make sure that no clipping occurs due to your interpolating. $\endgroup$ Sep 4 at 11:00
  • 1
    $\begingroup$ @Killuaisaack: Clipping means that your signal values exceed the possible range. For instance in a wave-file only values between -1 to +1 are possible. So if your interpolation function would lead to values outside that range your signal will be "clipped" to +1 or -1. This will lead to simillar vertical lines. $\endgroup$ Sep 5 at 6:42

1 Answer 1

2
$\begingroup$

Your denoising algorithm seems to be nonlinear in what it is leaving behind. The leftover noise is not phase aligned with the original noise or the noise of the next chunk. So, by overlapping and adding you essentially sum two uncorrelated signals, boosting the noise level by 3dB.

Additionally, it may well be that this is the case for the signal part as well. This leads to discontinuities in the overlapping parts, which will sound like clicks or cracks audiowise.

The first problem (noise adding up) can be tackled by not using overlap. This leaves the second problem, the discontinuities. Perceptively, these cracks can be worse than short gaps in the signal depending on their level.

They can be addressed by a smoothing interpolation to even out the discontinuitys. The python package scipy has a rich toolbox for this, scipy.interpolate. Plain linear interpolation will not do the trick, as it does not take into account the first derivative of the samples, leaving discontinuities behind. A good starting point to try out seems to be scipy.interpolate.make_interp_spline. It is quite powerful. Have a look at the documentation as well as this well documented example. With this, you should get the idea of what to do.

$\endgroup$
2
  • $\begingroup$ Thank you very much for your answer. I still have a bit of confusion, I also tried other interpolation functions before you answered, after using these interpolation functions, I will get an interpolation curve. Should I resampling to get discrete points and then insert them into my audio interface to change the result? $\endgroup$ Sep 5 at 7:18
  • 2
    $\begingroup$ In principal, yes. Look at the example. This: xx = np.linspace(x[0],x[-1],100) yy = [cs(x) for x in xx] does what you say. $\endgroup$
    – Max
    Sep 5 at 7:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.