Lip sync over IP

Question

What are the most used techniques for lip sync (audio/video sync) for conferencing over IP (and related issues with playout buffering)? I could find a lot of paper about audio intrastream synchronization, but only a few dealing with audio and video all together.

Answer 1

Before going to IP, I would like to draw your attention to lip synchronization techniques used in broadcast domain. This is also one of the toughest ones because here not only you want to maintain a very accurate lip sync - but require this on an ever running system 24x7x365 for a television networks.

In order to get more insight about this i would recommend reading up about packetized compressed Audio/Video is actually MPEG2 -System layer. See this: http://downloads.bbc.co.uk/rd/pubs/reports/1996-02.pdf. Section 8 explains the Time synchronization aspect.

Essentially, every encoder records timestamps and stamps it on the respective Audio video. Later on, when decoder plays it, it does two things - one, ensures that decoer's own clock is "enslaved" with encoder's clock, and two it ensure that every picture is presented on the screen and audio frame presented to speaker exactly when that respective time occurs. This is only and best way that audio remains in synchronization with video. These timestamps are called PTS/DTS values which are of resolution of 90 Khz clock.

Now this is not the end of the solution. Understand that over time clocks skew but since only the exact time is referenced, decoder playout exactly in same time order.

Now the major concern remains is that decoder's clock needs to remain in control/synchronization of encoder's clock. The first thing done in MPEG is using a higher precision at 27 MHz, (300 times higher). Further, this needs to remain consistent during any transmission path in the middle. (this is called clock recovery process).

A detailed paper that explains the time stamping schemes is here: Timestamping Schemes for MPEG-2 Systems Layer and Their Effect on Receiver Clock Recovery (1998) by Christos Tryfonas , Anujan Varma, IEEE Transactions on Multimedia

Here is another widely cited paper about this as a general theory. R. P. Singh, Sang-Hoon Lee, Chong-Kwoon Kim, “Jitter and Clock Recovery for Periodic Traffic in Broadband Packet Networks”, IEEE Trans. on Communications, Vol. 42, No. 5, May 1994. or this A New Method for Clock Recovery in MPEG Decoders

Now coming to IP networks

here there are other aspects to understand:

1. There is a much higher level of uncertainty of bandwidth (even on a reasonably good networks).

2. Usually, the system is serving one (in unicast) or fewer (multicast) users simultaneously unlike much broader assumption by broadcast.

3. The overall system is only a finite duration as opposed to live 24x7 broadcast.

Almost all basic principles remains same. However, few things are done additionally.

1. Receiver needs to keep a significantly larger buffer. Typically this is in addition to buffers that are described in MPEG2 TS case. The larger the buffer the larger is the delay to start but lesser chance to run out of content.

2. If you have followed above, time at which the clock packets arrive are used for decoder's clock synchronization. Actually, Decoder clock synchronization is quite often omitted in case of IP based systems - because even if the clocks diverge if you are running only a 10 minute clip the effect is not quite visible. However, if there is a long VoD session, may be such a clock recovery can apply - but still the jitter of IP network will be way too much.

In such cases, one uses RTP protocol in this case, additional time stamps on the per packet basis is applied which indicates the typical theoretical time of arrival it would have had, if there was no jitter. In addition to this - in order to make local clocks of the receiver synchronized to that of sender NTP time stamps based packets are used. RTP flows are synchronised by receivers based on information that is contained in RTCP SR packets generated by senders. For details, read section 6 of RTP rfc gives a complete account on how RTCP packets can be used for synchronization.