0
$\begingroup$

I am working with Python to isolate voiced segments from music using the Jamendo corpus for singing detection. Training a model, I break my audio into frames, and have a label (0,1) for each frame.

Unfortunately, due to rounding errors, my labels fall always 1 or 2 frames short.When extracting MFCC features from ~Train Music/03 - sample.ogg, my output is shaped as (13,3709).

This seems reasonable because the length of that file is 3797322 samples and with a hop_length of 1024, the number of frames works out slightly above 3708.

More details about the MFCC feature extraction in the following code snippet:

def MFCC(audio):
    s = [] 
    for y in audio:
        mfcc = librosa.feature.mfcc(y= y, sr = 16000, n_mfcc=13, n_fft=2048, hop_length = 1024)
        s.append(mfcc)

    return(s)

To now "mark" each frame with the class label (1,0 for singing, no singing respectively), I am converting from seconds to frame numbers but get a maximum value of 3708 (for the longest duration in the file). This is done as per the following function:

def labelToFrames(textCorpus):
    output = []              
    for block in textCorpus:
        block_start = int(float(block[0]) * 16000 / 1024)   
        block_end = int(float(block[1]) * 16000 / 1024)     
        singing = block[2]
        block_range = np.arange(block_start, block_end, 1)  # Step size is 1 (per frame number)
        for x in block_range:
            ms_start = '{0:.3f}'.format(x * 1024 / 16000)           # Converting back to milliseconds
            ms_end = '{0:.3f}'.format((x * 1024 / 16000) + 0.064)   # Converting back to milliseconds
            add_to_output = [ms_start, ms_end, singing]
            output.append(add_to_output)
    return output

I guess librosa's mfcc function is more complex than my 16000/1024 calculation (?). I have tried using Decimal, math.floor and also math.ceil within my block_start and block_stop variables, but I can't seem to match my audio frame length. Are there any other techniques recommended to avoid this error?

EDIT:

To address some discussion points:

y,sr = librosa.load("~Train Music/03 - sample.ogg", 16000, mono = True) followed by y.shape, it yields (3797322,). d = np.abs(librosa.stft(y, n_fft=2048,hop_length = 1024, win_length = 2048))**2 gives a shape of (1025, 3709). Finally performing MFCC on that gives (13, 3709)

Sticking with 44.1khz, the label data is referenced 10 frames short.

$\endgroup$
  • 1
    $\begingroup$ This has nothing to do with "rounding errors", or machine learning. What exactly do you want to do? Derive which frame contains the data of $k$ ms offset into the file?With the exception of the first frame, all other offsets are covered by two frames because n_fft is 2048 but the hop_length is 1024. $\endgroup$ – A_A Apr 6 '18 at 2:18
  • $\begingroup$ Firstly, this is for a machine learning algorithm so it has everything to do with machine learning. Secondly,I was fairly certain it is a rounding error due to Python's librosa package being more complex on the MFCC calculation than my manual calculations,as I am always between .1 and 1.1 frames away on my labelled data from the actual frame length of my audio! The thought did cross my mind with the first frame starting at 0, yet, that doesn't seem to explain such a tiny discrepancy of .1 between labels and frames. DSP is not my usual area btw as a data scientist! $\endgroup$ – Ciaran De Ceol Apr 6 '18 at 6:48
  • 3
    $\begingroup$ As someone who is trying to answer this question, I am still unclear as to what is the motivating problem behind the question. What are you trying to achieve? $\endgroup$ – A_A Apr 6 '18 at 8:23
  • $\begingroup$ Essentially, I am curious as to why my vector length is not matching my frame length of audio. My code, to me, seems to ensure that my vector length should be 3709 to match exactly the 3709 frame length of my audio - which I need to train my machine learning classifier. Any help would be appreciated. I am thinking to just trim the final or first frame of audio? Although I find sometimes, depending on the audio file, the vector label is 2 frames short. $\endgroup$ – Ciaran De Ceol Apr 6 '18 at 9:14
  • $\begingroup$ OK, what is inside textCorpus? What is the format of that variable? $\endgroup$ – A_A Apr 6 '18 at 9:26
0
$\begingroup$

I am not sure what exactly is amiss here (referring to the code) but here is a list of things that might help:

  1. Librosa's mfcc returns a matrix that is n_mfcc x ceil(len(y)/hop_length), so no surprises there.

  2. Divisions between integers can return reals in Python 3 but not Python 2. In any case, if you have a division of 16000/1024 try to enter it as 16000/1024.0 to make sure that what is returned is a real, even if it is going to be rounded in any way later on. In general, work with reals all the way up to the point where you need the rounding and then round at that point.

    • That's not so important around block_start but it would be around ms_start.
  3. To find the start of a frame containing time offset $to$, you need to $Frn_{start}= \lfloor{\frac{to}{\frac{NFFT-HopLength}{Fs}}}\rfloor$. Where $\lfloor{\cdot}\rfloor$ denotes floor.

    • Which, yes, it is essentially what you are doing here since int will floor.

Assuming that there is no further mismatch between the hardcoded sampling rates and the actual file that is read from the fisk or a mismatch between the audio file and its text file containing the annotations, I cannot see where is the problem with the labeling.

Can you share a specific example where the frame number is off by 1 or 2?

Hope this helps.

EDIT:

Since the annotation file is a simple text file, can you please collect the files that seem to be having issues and confirm that the annotations are indeed "correct"? To do this you can use Audacity. Audacity accepts simple tab delimited text files with annotation information. The Jamendo corpus includes annotations in space-delimited format and files ending in .lab. The conversion is ultra simple using gnumeric, to import the file and export it as tab delimited. Once you do that, open the MP3 in Audacity and import its labels from the newly exported text file.

In the following screenshot, I have imported the emporte-moi.mp3, converted it to mono and downsampled to 16kHz. This did not seem to have affected the annotations, although certain points, I would have annotated differently because it seems to "cut" when the voice still fades out. But that's a detail probably.

Would be useful to see if there is something going on with the annotations of the files that seem to be "out of sync" with the frame count.

enter image description here

$\endgroup$
  • $\begingroup$ Thanks for such a detailed response. On point 2. Unfortunately, the addition of a decimal point followed by 0 did not remedy this problem. 3. Int indeed floors. I also tried using Python's math.floor to no avail. If I take in a sample audio file y,sr = librosa.load("~Train Music/03 - sample.ogg", 16000, mono = True) followed by y.shape, it yields (3797322,). d = np.abs(librosa.stft(y, n_fft=2048,hop_length = 1024, win_length = 2048))**2 gives a shape of (1025, 3709). Finally performing MFCC on that gives (13, 3709) $\endgroup$ – Ciaran De Ceol Apr 6 '18 at 13:55
  • $\begingroup$ I am working with the Jamendo corpus of audio corpus for singing detection which is commonly used. So I doubt there is issues with the corresponding label files either. I am totally stumped with this one and have been for a couple of weeks now. $\endgroup$ – Ciaran De Ceol Apr 6 '18 at 13:57
  • $\begingroup$ I'm wondering would it be easier to adjust the code entirely to trim the longer of the two? $\endgroup$ – Ciaran De Ceol Apr 6 '18 at 14:01
  • $\begingroup$ @CiaranDeCeol what exactly do you perceive as "the problem" with the two ways of deriving the frames in your first comment here? $\endgroup$ – A_A Apr 6 '18 at 14:05
  • 1
    $\begingroup$ @CiaranDeCeol, can I please ask you to add all of this information to the body of the question? The reason for this is that it is not entirely clear from the original "body" of the question what exactly the problem is and from the point of view of the board, it is important to keep accurate questions and answers if they are going to be helpful to others too. And it looks like your "problem" might be coming up for other people as well. This is not by any means the end of the search for a solution but a checkpoint. $\endgroup$ – A_A Apr 6 '18 at 15:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.