# Why is the correlation of two random markov chains so large?

A very simple python code shows that the correlation coefficient and spearmann rank between two datasets of uniform random numbers drops proportionally to the square root of the number of points in the dataset. However, when one compares Markov chains produced by accumulating those numbers, there is no such behaviour, and even for large data sets, correlation can easily be like 0.7. What is the origin of such behaviour? Are two random markov chains more "related" to each other than two random datasets? Can this effect be corrected?

import numpy as np
import matplotlib.pyplot as plt
import scipy.stats

x = np.random.normal(0, 1, 10000)
y = np.random.normal(0, 1, 10000)

# Convert random data into Markov chains
for j in range(1, 10000):
x[j] += x[j-1]
y[j] += y[j-1]

print("Correlation: ", np.abs(np.corrcoef(x,y)[0, 1]))
print("SpearmannRank: ", np.abs(scipy.stats.spearmanr(x,y)[0]))

plt.figure()
plt.plot(x)
plt.plot(y)
plt.show()


EDIT: Unsurprisingly, the original problem is a bit more tricky. I have (at least) two experimentally obtained data sets I know relatively little about. Formally speaking, there is nothing stopping me from calculating their correlation in order to investigate their relationship. Since I was getting garbage, I tried to investigate correlations of several semi-random datasets, until I arrived at a surprising (for me) result above. Let me refine the question: is it possible to guess from the shape of the data, whether or not correlation is an informative metric?

• just a pedantic ranting from my side: "Markov Chain" requires your state set to be discrete, but for all you treat your floating point numbers here, your state space is the real numbers, which aren't discrete. I'd call what you've built a Wiener Process Jul 12 '18 at 8:18
• but then again, I'm confused by the fact that you can attach rank to a set of non-discrete observations. No idea what scipy is doing there, or my understanding of Spearmann Rank is insufficient, as that requires at least some values to happen twice. For a truly continuous random variable, that should (surely) not happen. Jul 12 '18 at 8:28
• @MarcusMüller, sorry, I am indeed sloppy. I have only ever learned discrete Markov chains, so I just call everything a Markov Chain. You are probably right. I don't know much about spearmann rank either, I was just taught that the estimator works well on continuous data too. I suspect it bins the data, there is even link to the book, but honestly I don't have time to understand how exactly it works at the moment Jul 12 '18 at 11:37