In sound anomaly detection, I see a processing step that I am not able to understand. We derive the Mel spectrogram the standard way. The signal shape is (say) 64 Mels by (say) 313 time bins. So far so good. Now the extra step is as follows.

##log_mel_spectrogram - This is our spect. 64 by 313
##frames = 5 (not to be confused with window frames used while deriving spect)
##Now starts the mysterious step
vectorarray_size = len(log_mel_spectrogram[0, :]) - frames + 1
vectorarray = numpy.zeros((vectorarray_size, dims), float)
for t in range(frames):
    vectorarray[:, n_mels * t: n_mels * (t + 1)] =
       log_mel_spectrogram[:, t: t + vectorarray_size].T

The code basically duplicates a lot of data and creates n (=5) frames & concatenates them and transposes it. So we end up with a shape 309 by 320 (64 * 5)

This is then fed to a downstream model. I am unable to grasp the significance of this duplication. Could someone help me understand why this could be useful.

Edit (after 3 days and announcement of bounty): I could find an url here https://dcase.community/challenge2020/task-unsupervised-detection-of-anomalous-sounds which explains the baseline system. It looks like this whole "frame" business is to provide a context window for each time step. Some parts are beginning to make sense, but questions remain. Why not feed the entire signal of all time steps to the auto encoder directly. A good explanation would help future novices as well.


1 Answer 1


A common way of dealing with sound classification is to use spectrograms. We create an image and feed it to any image model. An alternate way to do classification is to use the raw stft data after some processing.

When we feed raw stft data to an anomaly detection model (typically an autoencoder), we need to process the data a bit. Feeding the entire 313 * 64 2D array by flattening it to a linear layer (the first layer of the auto encoder) may not be the best approach. Instead we take a few time slices at a time, club them together and feed it to the model. We will have multiple such clubbing for a single audio file. So each audio file now gets a series of time-series data. The number of steps in each time series is determined by the context window (how many time slices we want to club together at a time...which in above case was 5).

Applying this to the situation above, we basically take 5 time slices at a time from the original stft (which had 313 timeslices) and create multiple such 5-framed outputs for each audio file. Imagine a sliding context window of size 5, which we keep moving from the first time slice of the audio file to the 313th time slice. We get 308 5-framed outputs for each audio file. Each of these 308 outputs has 5 time slices from the original stft and each time slice has n_mels(=64) energy values. So the output we get from the above pre-processing is a 308 by 320 (ie 5 times 64) per file.

While training the model, for each audio file we call the model 308 times feeding the 320 Mel energies each time. During inference too, we would do the same and then average out the (308) reconstruction errors for each audio file to get a single reconstruction error for that entire file based on which the anomaly non-anomaly decision can be taken.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.