I have a requirement to detect/reduce sidetalk/background noise in real-time audio. I am stuck in how can I detect this from audio time-frequency domain analysis. I am already getting the time-freq data from stft (I am using java for an easier way to integrate with our project). Can I do this without any machine/deep learning algo as I have not so much idea about these.. and whenever I read any articles those mainly come to this point and hand over the data's to machine/deep learning algo. But when I visualizes the audio data in tfft (spectrum) via wavepad I clearly could identify the voice and noise within same/other freq band. But what could be the algo behind this? How can I detect these from time-freq stft data?
Thanks in advance
$\begingroup$
$\endgroup$
2
-
$\begingroup$ I added a magnitude threshold in each stft frame while reconstructing the audio(inverse stft), by this I can separate most of the noise but the problem is voice is also getting distorted and some unwanted noise is being generated over the whole audio. I think the threshold should be dynamic in voice and noise areas but how. $\endgroup$– Nafiul Alam FujiCommented Feb 2, 2022 at 2:08
-
$\begingroup$ Generally speaking, zeroing out frequency bins is not the best thing to do. This can lead to temporal aliasing when going back from frequency representation to the time domaiin. This may or may not contribute to your problem but I suggest you try a soft mask instead of a binary one. An example of soft masking (in this case for harmonic-percussive separation though) can be found in "Fundamentals of Music Processing - Audio, Analysis, Algorithms, Applications" by Meinard Muller (link.springer.com/book/10.1007/978-3-319-21945-5). Of course soft masks can be found in other sources too. $\endgroup$– ZaellixACommented Feb 6, 2022 at 17:33
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
After some r&d on this area, I found out that there is only one good way to approach the problem. That is Short Time Fourier Transform (STFT) as mere RMS of an audio frame is mixed energy of all frequencies present there. It can give an idea but will fail in many of the cases. but through STFT we are getting frequency bins (range) and those frequencies specific power where I can detect voice compared to a certain threshold and can detect noise/side-talk below that threshold. Now you can go 2 ways with the Infos.
Number 1 is obvious with complex scaling of the frequency power in the frequency domain and then reconstructing the audio in real-time. But after trying this I failed horribly as frequency scaling has a great impact on the time domain. (tried overlapping_samples+windowing+zero-pad+scaling+overlap-save etc and will continue exploring, make me know if you can help me as a newcomer)
And then I thought of another easier & fun way (Number 2) for a minimal/average noisy environment. You can collect as much information in frequency domain like voice/non-voice probability by the threshold, threshold pass/fail count in a single frame, and design an adaptive algorithm to calculate a "scale" value and apply this scale in the time domain. This won't give 100% background noise removal(as it will be present during actual speech) but when you are not talking it can work like magic if you can implement it correctly. So the background noise or side-talk will be scaled to really low and won't hurt others when you are not talking. Here is the result after I designed my adaptive algo and applied.
