1
$\begingroup$

I have a video clip that has a frame rate of 25 fps (MJPEG compression) and an audio clip with a sample rate of 48kHz. Both of them play for the same duration - 27 seconds.

Is there any standard way to sync the two signals of different sampling rates if I wish to write them to file?

If this is not doable cleanly in MATLAB, I am happy with a some software tool on Linux or MacOS that can do this syncing.

E.g. the VideoFileWriter object can take both an audio sample and a video frame at a single step, however I have about 1917 audio samples corresponding to a single video frame.

Some details:

I have read in the audio signal using [signal, fs] = audioread(filename). signal is a 1336321 x 1 array.

All the 697 frames in the video are saved as a cell array of RGB images.

Improve this question
$\endgroup$
15
  • $\begingroup$ Welcome to SE.DSP. What do you mean by "write them to file"? $\endgroup$
    – Laurent Duval
    Aug 12, 2016 at 16:32
  • 1
    $\begingroup$ Maybe duplicate every frame 1917 times, and feed that to VideoFileWriter? $\endgroup$
    – MBaz
    Aug 12, 2016 at 19:33
  • $\begingroup$ @LaurentDuval: by "write to file" I mean generate a video that has the frames as the video component and the audio file as the sound component and write this resultant file to disk. $\endgroup$
    – AruniRC
    Aug 13, 2016 at 19:50
  • $\begingroup$ I did not try to do something similar, but I suspect there are simpler ways outside Matlab. Did you check solutions like stackoverflow.com/questions/17013363/… $\endgroup$
    – Laurent Duval
    Aug 13, 2016 at 19:57
  • $\begingroup$ @MBaz: and then set the frame rate of the resultant video clip to be the same as the audio clip (i.e. 48 kHz)? $\endgroup$
    – AruniRC
    Aug 14, 2016 at 9:06

2 Answers 2

Reset to default
2
$\begingroup$

Thanks to helpful comments from @MBaz, I managed to come up with a solution:

we can associate multiple audio samples with a single frame using the VideoFileWriter object. This fact and use-case is missing in the documentation.

First, some stats about the audio and video files. The stereo audio samples are in a 2xN array signal. The video frames are in a cell-array of RGB images, frames.

%% Write audio and video to file
%   Write both audio and video samples into a single video file. 
%   Multiple audio samples are matched with one video frame.
%   MATLAB problem: Compression is not possible when audio is included.

% Display A/V stats
fprintf('\nVIDEO STATS:');
fprintf('\nNum. of frames: %d', length(frames));
fprintf('\nFrame rate: %.02f fps', annot.video.frame_rate);
fprintf('\n\nAUDIO STATS:');
fprintf('\nNum. of samples: %d', size(signal,1));
fprintf('\nSampling rate: %.02f Hz\n', infoAudio.SampleRate);

Here's the portion that matches multiple audio samples to a single video frame. Note that there are 697 video frames and 1336321 audio samples. Some basic fiddling to ensure proper matching. The audio subsampling effect is imperceptible in the output AVI files that I tried. 

numAudio = size(signal,1);
numRep = floor(numAudio/length(frames));
numDiff = numAudio - numRep*length(frames); % mismatch

if numDiff
    % if length(frames) does not evenly divide nAudioSamples, then 
    % subsample audio to match numRep*length(frames)
    selector = round(linspace(1, numAudio, numRep*length(frames))); 
    subSignal = signal(selector, :);
end
assert(numRep*length(frames) == size(subSignal,1));

Finally, we use the VideoFileWriter object available in the Computer Vision System Toolbox of MATLAB to write audio and video to file.

videoFWriter = vision.VideoFileWriter(fullfile(shotPath, 'avclip', ...
                                        [num2str(shotNum) '_av_clip.avi']), ...
                                      'AudioInputPort', true, ...
                                      'FrameRate',  annot.video.frame_rate);
for i = 1:length(frames)
   fprintf('Frame: %d/%d\n', i, length(frames));
   step(videoFWriter, frames{i}, subSignal(numRep*(i-1)+1:numRep*i,:)); 
end
release(videoFWriter);
Improve this answer
$\endgroup$
1
  • $\begingroup$ You might want to give your answer the check mark! $\endgroup$
    – Peter K. ♦
    Jun 16, 2017 at 2:01
-1
$\begingroup$

U can use simulink we can associate multiple audio samples with a single frame using the VideoFileWriter object. This fact and use-case is missing in the documentation.

First, some stats about the audio and video files. The stereo audio samples are in a 2xN array signal. The video frames are in a cell-array of RGB images, frames

Improve this answer
$\endgroup$
1
  • $\begingroup$ A part of another answer has been copy/pasted. This is not a valid answer. $\endgroup$
    – AruniRC
    Apr 25, 2018 at 20:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.