A very simple python code shows that the correlation coefficient and spearmann rank between two datasets of uniform random numbers drops proportionally to the square root of the number of points in the dataset. However, when one compares Markov chains produced by accumulating those numbers, there is no such behaviour, and even for large data sets, correlation can easily be like 0.7. What is the origin of such behaviour? Are two random markov chains more "related" to each other than two random datasets? Can this effect be corrected?
import numpy as np import matplotlib.pyplot as plt import scipy.stats x = np.random.normal(0, 1, 10000) y = np.random.normal(0, 1, 10000) # Convert random data into Markov chains for j in range(1, 10000): x[j] += x[j-1] y[j] += y[j-1] print("Correlation: ", np.abs(np.corrcoef(x,y)[0, 1])) print("SpearmannRank: ", np.abs(scipy.stats.spearmanr(x,y))) plt.figure() plt.plot(x) plt.plot(y) plt.show()
EDIT: Unsurprisingly, the original problem is a bit more tricky. I have (at least) two experimentally obtained data sets I know relatively little about. Formally speaking, there is nothing stopping me from calculating their correlation in order to investigate their relationship. Since I was getting garbage, I tried to investigate correlations of several semi-random datasets, until I arrived at a surprising (for me) result above. Let me refine the question: is it possible to guess from the shape of the data, whether or not correlation is an informative metric?