How can I synchronize signal from an LED and TTL pulses better?

Question

I would like to synchronize my video and voltage data using Python. (I obtain these data using Bonsai and Open Ephys for an open field exploration experiment.)

I sent $150\text{ ms}$ long TTL pulses to a computer recording the voltage data, at the same time as to an LED in the field of view of my camera to turn it on. I sent the pulses at random times with 20-60 second gaps. I would like to align the LED pulses with the TTLs to synchronize my datasets.

The two recordings did not necessarily start and end at the same time, so it was possible for one of the datasets to have fewer pulses than the other one.

The voltage data was recorded at $30000\text{ Hz}$ sampling rate, and the video at about $30\text{ Hz}$.

I tried to downsample the voltage data, correlate the two arrays and find the maximum correlation to calculate the lag and then shift the video data by the offset.

My first problem is downsampling the voltage data. I calculated the real sampling rates from the average step size, and found that the camera is about $30.08207\text{ Hz}$.

I tried to use resampy's resample:

resampy.resample(voltage_data, voltage_data_sampling_rate, video_sampling_rate)

where voltage_data_sampling rate was $30000\text{ Hz}$ and video_sampling_rate $30.08207\text{ Hz}$. This returns an array that is the same size as the original voltage array.

1. How could I get a signal from this that has the same sampling rate as my video?

To resample my data without using this I generated an array with multiples of the rate of sampling rates ($\frac{30000}{30.082074} = 997.271$)), converted these to integers and took these indices. I understand that this is very basic, but I thought it would allow me to check if I can write the rest of this code.

I calculated the correlation using numpy.correlate:

correlation = np.correlate(video, voltage, "full")

I set a threshold based on standard deviation and the median and replaced all values with $0s$ below threshold to remove noise.

I checked which array was shorter and padded the end with $0s$.

short_array = np.pad(short_array, (0, len(long_array)-len(short_array)), 'constant')

I calculated the lag between the two signals based on the maximum correlation:

lag = (np.argmax(corr) - (corr.size + 1)/2)/avg_sampling_rate_video

This result is about right. It reduces my 2-5 second lags to about $100\text{ ms}$.

plt.plot(downsampled_voltage_time, downsampled_voltage_TTL)
plt.plot(video_time, video_sync_pulses)

I need this to be better than that, I would like the lag to be near $30\text{ ms}$.

2. Is the downsampling causing this problem, or are there other problems with this approach?

Answer 1

What you are probably missing here is that the frame rate of a webcam can be all over the place, especially if you are capturing video by getting a handle to the imaging device and then requesting frames by it using a timer.

In any case, wherever you are capturing the frame, capture its timestamp as well. This will allow you to cross reference the frames with the voltage time series better.

One of the first things that you have to do is synchronise at least the first pulse. So, independently of what you are going to do in subsequent processing steps, the first two pulses (one from the voltage time series and one from the video time series) must be synchronised. To do this, you simply shift one (or the other) time series so that the rising edges of the pulses coincide.

Now, extract the same amount of waveform in terms of seconds from both time series. So, extract for example 5 seconds from each. One is going to be 150 samples long the other 150000 samples long. Always subsample the more detailed one.

The two waveforms now will be as synchronised as your equipment allows them to be. If you plot them side by side you might find some mismatches because your camera's frame rate has some jitter.

The only way to escape this is by "counteracting" the FPS jitter by dropping voltage time series samples so that the voltage time series (which has higher resolution) "catches up" (or actually, lags behind) with the video.

You can do this either by matching each individual pulse's width or by Dynamic Time Warping (DTW).

The first method is straightforward: Extract a pulse period from both waveforms, measure the pulse width (in either waveform it has to be 150ms, 4.5 frames in the video waveform, 4500 samples in the voltage waveform), downsample the voltage waveform by a factor equal to the ratio between the pulse widths, move on to the next pulse period.

The second method basically does the same thing and can even return a signal which is exactly this "timing advance" signal by which you have to transform one of the waveforms so that it catches up with the other. In this case, things are even more straightforward: Downsample the voltage to the video sampling frequency and this will give you an approximate matching. Then, with both waveforms in the same sampling frequency, run them past DTW and apply the transformation.

Hope this helps.

Answer 2

I correlated the two datasets as described in the question and then trimmed them to ensure that the first pulse had a corresponding pulse in the other dataset. I was then able to detect the rising edges in both and shift one of them.

This way I have around 30 ms precision.