Take the 2-minute tour ×
Signal Processing Stack Exchange is a question and answer site for practitioners of the art and science of signal, image and video processing. It's 100% free, no registration required.

We always hear about this vector of data VS this other vector of data being independent from each other, or uncorrelated, etc, and while it is easy to come across the math regarding those two concepts, I want to tie them into examples from real-life, and also find ways to measure this relationship.

From this stand point, I am looking for examples of two signals that are of the following combinations: (I will start with some):

  • Two signals that are independent AND (necessarily) uncorrelated:

    • The noise from a car engine (call it $v_1[n]$) and your voice ($v_2[n]$) as you are talking.
    • A recording of humidity every day ($v_1[n]$) and the dow-jones index ($v_2[n]$).

Q1) How would you measure/prove that they are independent with those two vectors in hand? We know that independence means that the product of their pdfs equals their joint pdf, and thats great, but with those two vectors in hand, how does one prove their independence?

  • Two signals that are NOT independent, but still uncorrelated:

Q2) I cant think of any examples here ... what would some examples be? I know we can measure correlatedness by taking the cross-correlation of two such vectors, but how would we prove that they are also NOT independent?

  • Two signals that are correlated:
    • A vector measuring an opera singer's voice in the main hall, $v_1[n]$, while someone records her voice from somewhere inside the building, say in the rehearsal room ($v_2[n]$).
    • If you continuously measured your heart rate in your car, ($v_1[n]$), and also measured the intensity of blue lights impinging on your rear windshield ($v_2[n]$)... I am guessing those would be very correlated... :-)

Q3) Related to q2, but in the case of measuring cross-correlation from this empirical stand point, is it enough to look at the dot product of those vectors (since that is the value at the peak of their cross-correlation)? Why would we care about other values in the cross-corr function?

Thanks again, the more examples given the better for building of the intuition!

share|improve this question
    
@DilipSarwate Thanks Dilip, I will take a look at it. For now some examples would be good though. –  Mohammad Jan 27 '12 at 6:36
    
You can't "prove" that they are independent in the same way that even a well-constructed poll can't "prove" how everyone is going to vote- and for the same reasons. –  Jim Clay Feb 21 '12 at 18:15
    
@JimClay Feel free to relax the criterion 'prove' - what I am trying to get at are ways to measure/quantify independence. We often hear about so and so being independent, well, how do they know that? What measuring tape is being used? –  Mohammad Feb 21 '12 at 18:23
    
i would like to know whether cros corelation can be used for two analog signal one of high resolution & other of low resolution for analysis purpose. –  user3338 Nov 25 '12 at 4:42
    
If we have some random variable X and construct 2 signals a**=$f_1$(x) and **b**=$f_2$(x) with $f_1$ and $f_2$ being orthogonal and **x=a+b. Would this imply that such signals are independent? Does this require some additional conditions? This property would be interesting because it avoids constructing joint pdf of a and b. –  Mladen Nov 5 '13 at 16:55
add comment

2 Answers 2

A few elements... (I know that this is not exhaustive, a more complete answer should probably mention moments)

Q1

To check whether two distributions are independent, you need to measure how similar their joint distribution $p(x,y)$ is to the product of their marginal distribution $p(x) \times p(y)$. To this purpose, you can use any distance between distributions. If you use the Kullback-Leibler divergence to compare those distributions, you will consider the quantity:

$\int_x \int_y p(x, y) \log \frac{p(x, y)}{p(x) p(y)} dx dy$

And you will have recognized... the Mutual Information! The lower it is, the more independent the variables are.

More practically, to compute this quantity from your observations, you can either estimate the densities $p(x)$, $p(y)$, $p(x, y)$ from your data using a Kernel density estimator and do a numerical integration on a fine grid ; or just quantify your data into $N$ bins and use the expression of the Mutual Information for discrete distributions.

Q2

From the Wikipedia page on statistical independence and correlation:

Distribution plots

At the exception of the last example, these 2D distributions $p(x, y)$ have uncorrelated (diagonal covariance matrix), but not independent, marginal distributions $p(x)$ and $p(y)$.

Q3

There are indeed situations in which you might look at all the values of the cross-correlation functions. They arise, for example, in audio signal processing. Consider two microphones capturing the same source, but distant from a few meters. The cross-correlation of the two signals will have a strong-peak at the lag corresponding to the distance between microphones divided by the speed of sound. If you just look at the cross-correlation at lag 0, you won't see that one signal is a time-shifted version of the other one!

share|improve this answer
    
Thank you pichenettes: 1) Can you please elaborate on your first point - I am really having a hard time understanding just how, from two data vectors, x[n] and y[n], I can possibly come up with their JOINT PDF, $p(x,y)$. I can understand how taking a histogram of x[n] will give me pdf of X, ($p(x}$), and the same with Y, but how on earth does one come up with a joint given two vectors?? I am asking concretely - exact concrete mapping of a PDF from observed samples. This is what is confusing me the most. (contd) –  Mohammad Feb 21 '12 at 19:50
    
(contd) 2) So to summarize: If the covariance matrix of x, and y is diagonal, then they are uncorrelated, but NOT necessarily independent correct? To test for independence was the issue with follow up question (1). However, if we show they are indep, then of course their covariance matrix HAS to be diagonal. Have I understood right? What is an example of 2 physical signals I can measure in real life that would be dependent, but not correlated? Thanks again. –  Mohammad Feb 21 '12 at 19:52
1  
Let's say you have two signals $x_n$ and $y_n$ represented as vectors of $N$ elements. You can get an estimate of $p(x, y)$ using, for example, a Kernel density estimator: $p^*(x, y) = \sum_i \frac{1}{N}K(x - x_i, y - y_i)$ where $K$ is a Kernel function. Or you can use the same technique as for building an histogram, but in 2D. Build a rectangular grid, count how many pairs $(x_n, y_n)$ fall in each cell of the grid, and use $p^*(x, y) = \frac{C}{N}$ where N is the size of your signals and $C$ is the number of elements in the cell associated with point $(x, y)$. –  pichenettes Feb 21 '12 at 20:08
1  
"2 physical signals that would be dependent, but not correlated": Let's say we hack the GPS of a NY cab to record a (latitude, longitude) history of its position. There's a good chance the lat., long. data will be uncorrelated - there's no privileged "orientation" of the point cloud. But it'll hardly be independent, since, if you were asked to guess the latitude of the cab, you would provide a much better a guess if you knew the longitude (you could then look at a map and rule out the [lat, long] pairs occupied by buildings). –  pichenettes Feb 21 '12 at 21:12
    
Another example : two sines wave at an integer multiple of the same frequency. Null correlation (Fourier basis is orthonormal) ; but if you know the value of one there is only a finite set of values that the other one can take (think of a Lissajous plot). –  pichenettes Feb 21 '12 at 21:18
show 9 more comments

Inferring whether two signals are independent is very hard to do (given finite observations) without any prior knowledge/assumptions.

Two random variable $X$ and $Y$ are independent if the value of $X$ doesn't give any information about the value of $Y$ (i.e. doesn't affect our prior probability distribution for $Y$). This is equivalent to any nonlinear transformation of $X$ and $Y$ being uncorrelated i.e. $$\text{cov}(f_1(X),f_2(Y)) = E(f_1(X),f_2(Y)) = 0$$ for any non-linear $f_1$ and $f_2$ assuming wlog both variables have zero mean. The difference between independence and uncorrelatedness is that $X$ and $Y$ are uncorrelated if the above holds, only for $f_1(x) = f_2(x) = x$, the identity function.

If we assume joint Gaussianity, then all joint moments greater than order 2 are equal to zero and in this case uncorrelated implies independent. If we have no prior assumptions then estimation of the joint moments $E(X^iY^j)$ will give us information on 'how dependent' they are upon one another.

We can generalise this to signals $X(t)$ and $Y(t)$ by considering the cross-spectra $$S_{X,Y}(f),S_{X^2,Y}(f),S_{X,Y^2}(f) \dots$$ across all frequencies $f$.

Example:

After reading 'pichenettes' comment I was inspired to use his idea as an example. Consider the signals $$X(t) = \sin(2 \pi ft)$$ $$Y(t) = \sin( 2 \pi ftk)$$ for $k \in \mathbb{Z}$ and $k \neq 1$. Clearly there is no linear transform sending $X(t)$ to $Y(t)$ as they oscillate at different frequencies. However, it is well known that we can write $\sin(kx)$ as a function in $\sin(x)$ and therefore, $$Y(t) = f (X (t))$$ for some polynomial $f$.

Hence despite being uncorrelated signals, $X(t)$ and $Y(t)$ are not independent.

share|improve this answer
    
+1. Very nice one! –  Phonon May 10 '13 at 18:10
    
Can you please elaborate on what exactly the cross-spectra of $X_{x^2, Y}(f)$ is exactly? Thank you. –  Mohammad May 13 '13 at 15:08
    
en.wikipedia.org/wiki/Cross-spectrum Where we are considering the cross-spectra between the signals $X^2(t)$ and $Y(t)$. –  rwolst May 14 '13 at 9:53
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.