0
$\begingroup$

A very simple python code shows that the correlation coefficient and spearmann rank between two datasets of uniform random numbers drops proportionally to the square root of the number of points in the dataset. However, when one compares Markov chains produced by accumulating those numbers, there is no such behaviour, and even for large data sets, correlation can easily be like 0.7. What is the origin of such behaviour? Are two random markov chains more "related" to each other than two random datasets? Can this effect be corrected?

Correlation and Spearmann Rank of two random normal populations as function of dataset size

Correlation and Spearmann Rank of two markov chains generated from random normal populations as function of dataset size

import numpy as np
import matplotlib.pyplot as plt
import scipy.stats

x = np.random.normal(0, 1, 10000)
y = np.random.normal(0, 1, 10000)

# Convert random data into Markov chains
for j in range(1, 10000):
    x[j] += x[j-1]
    y[j] += y[j-1]

print("Correlation: ", np.abs(np.corrcoef(x,y)[0, 1]))
print("SpearmannRank: ", np.abs(scipy.stats.spearmanr(x,y)[0]))

plt.figure()
plt.plot(x)
plt.plot(y)
plt.show()

EDIT: Unsurprisingly, the original problem is a bit more tricky. I have (at least) two experimentally obtained data sets I know relatively little about. Formally speaking, there is nothing stopping me from calculating their correlation in order to investigate their relationship. Since I was getting garbage, I tried to investigate correlations of several semi-random datasets, until I arrived at a surprising (for me) result above. Let me refine the question: is it possible to guess from the shape of the data, whether or not correlation is an informative metric?

| improve this question | |
$\endgroup$
  • $\begingroup$ just a pedantic ranting from my side: "Markov Chain" requires your state set to be discrete, but for all you treat your floating point numbers here, your state space is the real numbers, which aren't discrete. I'd call what you've built a Wiener Process $\endgroup$ – Marcus Müller Jul 12 '18 at 8:18
  • $\begingroup$ but then again, I'm confused by the fact that you can attach rank to a set of non-discrete observations. No idea what scipy is doing there, or my understanding of Spearmann Rank is insufficient, as that requires at least some values to happen twice. For a truly continuous random variable, that should (surely) not happen. $\endgroup$ – Marcus Müller Jul 12 '18 at 8:28
  • $\begingroup$ @MarcusMüller, sorry, I am indeed sloppy. I have only ever learned discrete Markov chains, so I just call everything a Markov Chain. You are probably right. I don't know much about spearmann rank either, I was just taught that the estimator works well on continuous data too. I suspect it bins the data, there is even link to the book, but honestly I don't have time to understand how exactly it works at the moment $\endgroup$ – Aleksejs Fomins Jul 12 '18 at 11:37
0
$\begingroup$

Hi: you can't cumulate two sets of random numbers and then calculate a correlation between them. The two samples are non-stationary ( since you are cumulating them ) and what you get is referred to as "spurious correlation" or "non-sensical correlation" in econometrics. ( some would say statistics but the econometricians really figured out what was happening ). I wouldn't do it justice trying to explain it here but, essentially, what you are doing is calculating two series whose means are changing over time so the correlation calculation is illegal statistically. Granger and Newbold figured out the statistics behind why this is in 1976 so if you want all the details, just google for "spurious regression". The econometrics literature on this is enormous and, even though what you are doing has maybe zero to do with econometrics, you are basically generating two series that are each random walks and then correlating them which is a garbage in garbage type of scenario. I hope this helps.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Hi Alexsejs: I read the edit but I'm not clear on what the follow up question is ? So, all I can say right now ( which might help or might not ) is, If you want to do statistics to compare two data sets, definitely DO NOT cumulate the observations. if you have two series that are generated from a uniform distribution over say [a,b] and you have a decent rng ( and don't reset the seed ), then calculating a correlation between the non cumulated observations fine and you should get a low correlation between the two series. But, if you clarify your latest question, that could help. good luck. $\endgroup$ – mark leeds Jul 12 '18 at 8:32
  • $\begingroup$ Thanks for reply. I think I figured out the answer to my own question. Basically, if I have a random dataset, I can calculate standard deviations progressively by adding one extra point at a time. If I observe the estimate drop as ~1/sqrt(N), then it is likely that I have a random source and can apply the correlation. If this is not the case, then either my dataset is too short, or I have memory in the system, suggesting that the underlying model may change in time, and thus I can't study such data using correlations $\endgroup$ – Aleksejs Fomins Jul 12 '18 at 11:22
  • $\begingroup$ okay. it sounds like your possibly dealing with markov regime switching and that's not my thing ( james hamilton wrote the important paper on that if you want to take a look at it ) so I'll stay quiet and wish you luck with your project. $\endgroup$ – mark leeds Jul 12 '18 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.