0
$\begingroup$

I'm trying to understand RNNoise (https://arxiv.org/pdf/1709.08243.pdf, https://github.com/xiph/rnnoise). One aspect in particular, the input signal BFCC coefficients and output signal BFCC coefficients are compared to create a "gain per critical band". Then, this gain per critical band is applied to the input signal by applying an interpolated gain term per DFT bin.

The relevant part of the paper is this:

Also, rather than directly estimating spectral magnitudes, we instead estimate ideal critical band gains, which have the significant advantage of being bounded between 0 and 1. We choose to divide the spectrum into the same approximation of the Bark scale [12] as the Opus codec [13] uses. That is, the bands follow the Bark scale at high frequencies, but are always at least 4 bins at low frequencies. Rather than rectangular bands, we use triangular bands, with the peak response being at the boundary between bands. This results in a total of 22 bands. Our network therefore requires only 22 output values in the [0,1] range.

The equations look like this:

enter image description here

I'm trying to understand now, separately from RNNoise, how I could apply the idea of critical band gains on my own.

Say I'm using 24 band Bark-scale BFCCs (for example, from Essentia: https://essentia.upf.edu/reference/std_BFCC.html).

The BFCC coefficient calculator takes an overlapping sequence of audio frames, applies a window + spectrum, and generates coefficients. In one specific loop iteration, I will have (pseudocode) 2 frames (from 2 signals), and I want to modify one of them to transform it into the other with a "BFCC gain":

// use a BFCC coefficient generator using magnitude spectrogram (e.g. dbamp outputs)
// 24 bands between 0 and 15500.0 Hz (the classical Bark bands)

frame_1 = float[1024];
frame_2 = float[1024];

bfcc_1 = BFCC(spectrum(window(frame_1))); // float[24], bfcc coefficient for 24 bands
bfcc_2 = BFCC(spectrum(window(frame_2))); // float[24], bfcc coefficient for 24 bands

bfcc_gain = bfcc_2/bfcc_1; // 24 floats for "gain per critical band"

frame_2_prime = apply_bfcc_gain(frame_1, bfcc_gain);

At this point I'm pretty confused as to what the magic function apply_bfcc_gain could look like.

I've had a few unsuccessful attempts.

Option 1 - apply gain to DFT frequency bins that fall in each band

bark_bands = [
  (0, 100),
  (100, 200),
  (200, 300),
  (300, 400),
  (400, 510),
  ...
]

bark_gains = float[24]; // the "BFCC gains" above

fft_frame = fft(frame);

for i in range(nfft):
    freq_bin = i*sample_rate/nfft;
    if 0 < freq_bin < 100:
        fft_frame[i] *= bark_gains[0]
    else if 100 < freq_bin < 200:
        fft_frame[i] *= bark_gains[1]
    ...

frame_with_gain = ifft(fft_frame);

Some important details are omitted for brevity (namely the overlap-add reconstruction).

This is obviously wrong (and sounds terrible, perceptually). I know it can't be right but it's the only form of solution I can wrap my head around so far.

Option 2 - apply a Bark filterbank

I could maybe create a filterbank of 24 bandpass filters (with the appropriate frequencies for the 24 bark bands), and use the BFCC gain term as the dB gain, and pass in the input signal.

This is not a technique I'm overly familiar with (but have heard of auditory filterbanks).

I also wouldn't know how to define the Q values.

Option 3 - do it the right way, like RNNoise

The code which accomplishes the BFCC gain application is minimal in the RNNoise codebase (but I have trouble understanding it).

It's here:

// g is a float[24] of BFCC gains
// gf is an array of all possible gains for every nfft/frequency bin

interp_band_gain(gf, g);

for (i=0;i<FREQ_SIZE;i++) {
   X[i].r *= gf[i];
   X[i].i *= gf[i];
}

Basically a gain term is applied for every FFT bin. The magic is in the function interp_band_gain(gf, g) (https://github.com/xiph/rnnoise/blob/master/src/denoise.c#L150), which takes in an array of 24 critical band gains, and transforms them into a gain per frequency bin which can be applied on the DFT:

static const opus_int16 eband5ms[] = {
/*0  200 400 600 800  1k 1.2 1.4 1.6  2k 2.4 2.8 3.2  4k 4.8 5.6 6.8  8k 9.6 12k 15.6 20k*/
  0,  1,  2,  3,  4,  5,  6,  7,  8, 10, 12, 14, 16, 20, 24, 28, 34, 40, 48, 60, 78, 100
};

void interp_band_gain(float *g, const float *bandE) {
  int i;
  memset(g, 0, FREQ_SIZE);
  for (i=0;i<NB_BANDS-1;i++)
  {
    int j;
    int band_size;
    band_size = (eband5ms[i+1]-eband5ms[i])<<FRAME_SIZE_SHIFT;
    for (j=0;j<band_size;j++) {
      float frac = (float)j/band_size;
      g[(eband5ms[i]<<FRAME_SIZE_SHIFT) + j] = (1-frac)*bandE[i] + frac*bandE[i+1];
    }
  }
}

I think there's some form of scaled summation of the gains from every possible band at each frequency bin, but I can't understand it.

From the part of the paper I quoted above:

Rather than rectangular bands, we use triangular bands, with the peak response being at the boundary between bands. This results in a total of 22 bands.

So perhaps I need to create some triangle shapes (with, e.g. scipy.signal.triangle), where the peak of each triangle (the 1.0 value) is the boundary between each frequency band. That way, each frequency bin might have an overlap of the right side of the triangle of the current band, and the left side of the triangle in the next band:

amplitude

1        / \  / \
        /   \/   \
       /    /\    \
      /    /  \    \ 
     /    /    \    \ 
0   0    100 x 200      hz

In this (simple) illustrated example, the gain applied at the DFT bin of "x" Hz would be:

amplitude value of the triangle on the left (100Hz bark) * 100Hz bark band BFCC gain
+
amplitude value of the triangle on the right (200Hz bark) * 200Hz bark band BFCC gain

Can anybody help? Thanks.

edit

I wouldn't even mind blindly copying the RNNoise code, except for this magic array which is unexplained:

static const opus_int16 eband5ms[] = {
/*0  200 400 600 800  1k 1.2 1.4 1.6  2k 2.4 2.8 3.2  4k 4.8 5.6 6.8  8k 9.6 12k 15.6 20k*/
  0,  1,  2,  3,  4,  5,  6,  7,  8, 10, 12, 14, 16, 20, 24, 28, 34, 40, 48, 60, 78, 100
};

This is copy-pasted from Opus code (given that RNNoise uses the same Bark bands as Opus). It's some transform from Bark band edges to DFT indices (in 5ms hops, perhaps?).

I don't know how to derive this for my own set of Bark bands.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.