# Correlation between 2 signals of uneven dimensions

As a part of my work, I am trying to correlate the audio signal in a video with the pixels of each frame. The steps I follow are:

1. Audio sampling rate and frame rate of the video are known. So, division of the audio sampling rate with the frame rate gives us the number of audio samples associated with each frame of the video.
2. That being done, I want to correlate the audio signal (the collection of audio signal amplitudes with 2 channels) with the 2-D pixel intensity of each frame, to basically decipher whether which part of the frame has maximum correspondence with the audio.
3. But, obviously the 2 signals ( audio signal with 2 channels and the 2-D frame pixel intensity values ) are having different dimensions. In this scenario, how do I compute the correlation between them?

I did encounter the idea of performing canonical correlation, but then again, the unequal dimensions of the 2 signals makes it a problem.

The intuitive way to go about this would be to consider every single pixel in your video (assuming that's only intensity, not e.g. color) a 1D signal over time.

Then you'd get width×height number of crosscorrelation functions between pixel intensities and your audio.

The different sampling rates of audio and video just mean that you'd need to reduce your audio to a signal of a rate that is identical to the video frame rate. (You could also interpolate the video to the audio rate, but then you'd be mostly making statements about the correlation properties of your interpolation function, not of your video).

The question is how you can reduce your audio to something that only takes say 25 values in a second. My wild guess is that this is something where you either use your inherent ideas of what would be a clever measure (e.g. "loudness in 40ms steps") or you just do the same for all 12.5 Hz wide bandwidths of your audio, and see which correlates best.

• Thanks. Regarding the reduction of audio dimension, can you suggest some sources I can look up? – Curiosity Feb 6 '19 at 15:52
• not really – as I tried to explain in my last paragraph, I'd typically expect you to have at least a mental model of what information "in" your audio correlates well with the video, and that'd be your starting point; if you don't, as said, a subband decomposition would be a start. – Marcus Müller Feb 6 '19 at 19:15

Interesting. It is just an idea but this what I would try: Consider the pixel data as a complex signal and compute a cross correlation between this complex signal and the real audio signal. Maybe there will be interesting results.

• Yes, that can be done. But, then again, the two signals need to have the same length for computing cross-correlation. I mean, the pixel data is 2-dimensional, and the audio is 1-D. How to proceed in that case? – Curiosity Feb 6 '19 at 15:51