Determining the 'amount' of speaking in a video

Question

I'm working on a project to transcribe lecture videos. We are currently just using humans to do the transcriptions as we believe it is easier to transcribe than editing ASR, especially for technical subjects (not the point of my question, though I'd love any input on this). From our experiences we've found that after about 10 minutes of transcribing we get anxious or lose focus. Thus we have been splitting videos into ~5-7 minute chunks based on logical breaks in the lecture content. However, we've found that the start of a lecture (at least for the class we are piloting) often has more talking than later on, which often has time where the students are talking among themselves about a question. I was thinking that we could do signal processing to determine the rough amount of speaking throughout the video. The idea is to break the video into segments based on the amount of typing they require, as opposed to splitting based on time.

I've done a little research into this, but everything seems to be a bit overkill for what I'm trying to do. The videos for this course, though we'd like to generalize, contain basically just the lecturer with some occasional feedback and distant student voices. So can I just simply look at the waveform and roughly use the spots containing audio over some threshold to determine when the lecturer is speaking? Or is an ML approach really necessary to quantify the lecturer's speaking?

Hope that made sense, I can clarify anything if necessary.

Appreciate the help as I have no experience with signal processing.

Answer 1

The most straightforward and easy thing you could do is rectify the signal (i.e. take the absolute value of its samples) and then run a moving average filter to get an approximate estimate of the "envelope" of your signal. Imagine the moving average filter like estimating the mean of the signal values between samples 0-100, then 1-101 then 2-102 and so on.

The result of this will be an inaudible slow varying waveform. You can then do a histogram on its values and use that to estimate a threshold which could then be used to "tag" the interesting parts of your video.

However, I feel that this is a lot of hard work for this kind of problem.

Therefore, I would also like to suggest that you use a video editor that presents the video "signal" as two separate ribbons, one for the actual video and another representing the amplitude of the sound. You can then quickly inspect the audio ribbon and estimate quickly the parts that could be seen as "interesting" and use the video editor to either directly transcribe those areas or cut the video down into smaller segments. Unfortunately, at the moment, I can`t recall a freely available video editor with this feature but if I do I will amend this response.

Answer 2

This question is somewhat similar to this How to identify useful elements of audio recording and ignore lulls?

I offered a similar solution to A_A Using a small moving average would remove the need for hysteresis as I used in my code, but the end result should be the same.