I want to isolate video game "action" sounds from the video game soundtrack. Given all ~20 songs, I'd like to filter them out from any live audio stream during gameplay.

What is a fast and robust way to do this?

More info: I am using this to preprocess the audio input for a neural network which is making decisions in real-time, so it needs to be fast (on the order of 100ms or less). It also needs to be robust, since the soundtrack can be interrupted in-game events or pausing the game.

I am thinking of binning the audio with small bins, taking a FFT, identifying which song is playing by feeding the spectogram to a convolutional neural network, and subtracting the song. I fear this approach would be too slow, though.

  • $\begingroup$ "input for a neural network" <-> "needs to be fast" why? Sounds like you want to analyze a recording ("offline analysis"), in which case you've got all the time in the world. $\endgroup$ – Marcus Müller May 1 '17 at 7:04
  • 1
    $\begingroup$ @MarcusMüller He stated that he would like to filter thos songs "from any live audio stream during gameplay", so I think that he needs to be fast because of that: it has to be done on the fly. $\endgroup$ – Tendero May 1 '17 at 14:33
  • 2
    $\begingroup$ Sounds like cross correlations with the "known" audio sequences would give you good inputs for your neural network to train on. What kind of time requirements do you have and what type of processing power do you have at your disposal? $\endgroup$ – hops May 1 '17 at 19:27
  • 1
    $\begingroup$ I would expect you to implement it with the CUDA cores as long as you have a sufficiently powerful GPU, this should be no problem. Of course hundreds of possible sounds is a lot, so I'm not sure this is feasible. I don't know how many cores are used by your Neural Network or how many are available for you to use total though. See pg 26 of this document for more info: docs.nvidia.com/pdf/CUBLAS_Library.pdf $\endgroup$ – hops May 1 '17 at 23:22
  • 1
    $\begingroup$ Would you perhaps be better off using online ICA to detect the game noises instead of filtering out lots of large background sound? $\endgroup$ – wwwslinger May 2 '17 at 17:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.