I'm an animator trying to get a program that will use machine learning to automate the lip syncing process. The program I need made will have two files given to it: a seed, and an audio file. The seed will be a video made in another program, it will be anywhere between a few minutes to a few hours of hand synced audio, done in a different program. The seed will be made uniquely for any new speaker, so as to train the program for every person. A transcript of the video can be added if it would help, but I'm okay removing this if you believe it won't. Then the audio will be added, this is the one needing to be synced. It will then need to go through, listen frame by frame and compare the sounds to the seed video. When it finds a match, it will save whatever image is in the video at that frame into a sequence. The final output will be a png image sequence that can be played at 30 frames per second in a video editor along with the audio. For the sake of quality, I would prefer that the seed also be an image sequence I upload into it with the seed's audio, this way the exported image sequence will have transparency, if that's possible. Also when the audio is being recorded, there is a video being recorded as well of the speaker, if adding some sort of facial tracking could also help, I'd like that as well. Lastly, I'd prefer this to run as quickly as possible. So not running real-time, seeing as the audio being synced can sometimes be up to several hours long. I will down the line like an ability to do lip syncing live (with perhaps a 5 second delay) and it just displays in a window and syncs to a chosen microphone or audio device.

So for example, when the program is listening to the audio that needs to be synced, let's say it hears the "Ah" sound. It will then search the seed audio for the best match to this sound, let's say it's the first time the word "and" is said. It hears a matching Ah" sound, and then looks to the video to see what image is currently being displayed. It copies that image, then moves on to the next frame/sound. In the end it should produce an image sequence that could play back at 30 frames per second in an editor alongside the audio.

So, which program or AI \ DSP \ ML techniques will help in this?

  • $\begingroup$ i think you need to get into time-scaling and splicing, regarding the audio. about matching the lip movement of the face of your recorded voice actor to the lip movement of the animation, that is of course about image processing and that is not my strength. $\endgroup$ – robert bristow-johnson Aug 3 '20 at 4:20
  • 2
    $\begingroup$ There is a very large amount of work implied here that points towards speech recognition rather than pattern matching to detect what is being said. Two existing pieces of software can be used towards that aim: videogrep and "deep lip reading". My perception is that the way this question is formed makes it off-line for this board. Possibly this would be more suitable (?). If it is about DSP techniques really, it needs more focus. $\endgroup$ – A_A Aug 3 '20 at 8:12
  • $\begingroup$ @A_A so are there any python libraries that allow to extract audio based on words like videogrep? $\endgroup$ – dsp101 Aug 4 '20 at 11:41
  • $\begingroup$ anybody has any clues? $\endgroup$ – dsp101 Aug 12 '20 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.