Let me preface that I am new to audio processing and audio analysis ;) (I asked the same question on reddit, I wanted to increase it's reach))

I am trying to classify specific events (like a gong or the clank of hitting a pipe) from audio data in an environment where people are. Recording people is a big no no here. (Well the recording was okayed under the premise that no talks can get transcribed and that no people are recognizable)

The first idea was to simply use a band filter and cut out the frequency range of normal speech but some of the signals I am interested might also fall into that range so I would rather avoid that.

Then I looked into spectrograms which looked promising for classification in general. I found the librosa library in python and started doing stft. I had planned to save the amplitude

S = np.abs(librosa.stft(signal, n_fft=<int>)

to maybe work on some other feature extraction or post processing later on but it looks like no matter what I do with the stft can get transformed back with the griffinlim algorithm to have a signal which is clearly identifiable even when using some unusual n_fft frame lengths. Here is an example: I am mostly interested in the signal between seconds 4 and 20 around 2000 Hz, before and after there is someone speaking.


spectrogram with n_fft=2048

spectrogram with n_fft=32768

spectrogram with n_fft=128

The first reverted signal (2048) sounds almost like the original, second (32768) and third (128) are modified but perfectly fine to understand. Even at n_fft=128 you can clearly hear the words when transforming back the amplitude to an audio signal (I am very amazed by this fact)

So my questions would be:

How does one approach this problem? Are there better ways than what I have tried?

Should I completely drop the idea to temporary storage any form of amplitude - frequency - time series data? And just work with aggregates and image data?

If I only use the images of power spectrograms are those images obfuscated enough to make a reconstruction of the audio signal not feasible?

(There is a data pipeline which takes the audio stream from a microphone. The first module would preprocess the audio signal in a way to do clarification later on and save it somewhere while fullfilling the privacy concerns. The real signal will only be briefly in memory)

Thank you for any comments or suggestions where / what to look at!


2 Answers 2


If I only use the images of power spectrograms are those images obfuscated enough to make a reconstruction of the audio signal not feasible?

That depends on how you parameterize your spectrogram. In general, for speech, where phase isn't that important anyways and can with reasonable accuracy also be reconstructed from amplitude alone, I'd be a bit careful with that. There's enough papers that show you can reconstruct audio from spectrograms.

Should I completely drop the idea to temporary storage any form of amplitude - frequency - time series data? And just work with aggregates and image data?

Well, the problem runs a bit deeper – you could totally come up with an image representation that contains 100% the audio samples, leading to full playback capabilities, so "image" is not a safeguard at all. "Aggregate" is not a safeguard either, because, for example, downsampling a 48 kHz-sampled signal to 16 kHz sampling rate would leave basically all speech untouched and would very much be an "aggregate" of the original signal.

The problem you're having is that you want to destroy most information about speakers, whilst keeping as much information as possible about the things you want to detect. For that, you would need to remove most features from the signal that say something about the speakers (or the spoken), but preserve these that say something about the phenomena of interest.

Problem is that you have no clue whatsoever yet about the detection-relevant features, and thus don't know what you need to preserve.

So, you don't know which kind of aggregate is safe to store and still useful for your task.

I think there's logically no good way out here legally and morally, but technically, there's lots of things you could do.

Legally and morally, you cannot record people talking and save the full recording.

Technically, you'll need to have the signals at hand, and quite a few of them, to figure out what features of the recordings need to be saved and what needs to be eliminated.

That's a chicken/egg problem.

So, I'd propose you go the clean route here: You come up with a multi-step plan: First, based on explicitly OK conversations, you find a way of anonymizing the signals, whilst keeping at least the simpler classifications possible, then you co-optimize anonymization and classification as you progress, with more signals, but this time such that you can use the anonymization from the first step as basis, so to not infringe anyone's privacy.

I'd honestly consider starting this all on recordings of people conversing that are not from your environment, but done by someone else. I'm sure you can find a lot of audio libraries already for exactly that purpose – for example, in the training material for modern neural network-based audio codecs.

Superimpose "clean" recordings of your gong, clang, … randomized on randomized snippets from your conversation library. Keep these snippets short.

This has, in addition to solving the privacy chicken/egg problem, the added advantage that you can create years worth of audio computationally, while actually getting labeled data for your real environment will require someone to actually record the events. For example, if you need to detect the door bell, and roughly every 2h someone rings, and to make your detector work reasonably well, you need some 100 door ringings while different conversations are going on – well, that's going to drag out your project a bit, won't it.

Optimize a system that represents these "mixture" snippets with as few coefficients as possible, and makes a detection based on these coefficients (instead of on the snippet itself).

The thing would roughly look like this:

Audio snippet           N "condensed"      M "detection
(duration · f_sample)    coefficients      likelihoods"

|        \
|         \                                 +-------+
|          \                               /        |
|           \                             /         |
|            \                           /          |
|             \                         /           |
|              +-----------------------+            |
|                                                   |
|                                                   |
|                                                   |
|              +-----------------------+            |
|             /                         \           |
|            /                           \          |
|           /                             \         |
|          /                               \        |
|         /                                 +-------+
|        /

encoder          latent representation        decoder

This is the structure of an autoencoder: your encoder is a (a priori unknown) function of your input that converts a high-dimensional vector (here: audio PCM values, e.g. 48 kHz · 0.75 s = 36000 samples) into a much smaller vector, the latent representation. The decoder is another (a priori unknown) function that takes these small vectors and reconstructs a larger output vector from it (for example: a vector containing a detection probability for each event).

That sounds easy enough as a principle, right?

The trick is finding two such functions. As a matter of fact, you usually start with two random functions which have parameters you can modify, and then try them out with the inputs (i.e., the "mixture samples" from above) you have (or can generate on the fly). Because, for these samples, you know whether an event is present, and if so, which event, you can calculate a "goodness" of the output (e.g., if there's no event in there, all elements of the output vector should be close to 0; if there's an event in there, all should be small, but the one output element that corresponds to the event) or a "loss" (how much worse than an all-knowing oracle is our system?).
You use that "goodness" (or "loss") to adjust the parameters accordingly.
We call this "optimization of parameterized functions to fit a task whose success is measured with a loss function" training. As you might have noticed, this is all very classical machine learning; supervised learning. The standard choice for "parameterizable functions" are neural networks here.

Don't start with humongous neural networks as you find them e.g. in image classification with hundreds of classes, or speech models or anything. A handful hidden layers on both the input and the output side might totally suffice.

Let your computer do that optimization according to loss function with a couple thousand synthesized snippets, optimize the two functions until you get reasonable detection performance, and verify whether it still works when you use conversation audio snippets that did not appear in your optimization.

Then, start being a bit mean, and add increasingly much noise to your synthesized input snippets. Have more than one event at a time… Maybe add a term to your loss function that also slightly "pushes" the encoder function to use fewer coefficients. See how small you can get the latent representation without losing performance.

As soon as your "lab" autoencoder works well, you can use the encoder on your real-world audio. You only save the latent representation, which by now you're fairly confident mostly contains information needed by the decoder to detect events, and not information about speakers or speech.

  • $\begingroup$ Fantastic answer $\endgroup$
    – Jdip
    Commented Jul 16, 2023 at 11:12

Joint Time-Frequency Scattering, a CWT-based time-frequency method, performs well at speaker-independent classification, with speaker information attenuated via imposing frequency transposition invariance, since speaker identification is sensitive to pitch. Refer to Anden et al.

The amount of information perserved in spectrogram depends mainly on hop_size and nfft; the bigger the spectrogram, the more information preserved, and ideally the signal can be recovered within a global phase shift, which is a very strong inversion. Since it's audio, whose audible features are phase invariant, that's even better recovery. More info in STFT: why overlap the window?.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.