If I pick $N$ samples from $P_X$ and $P_Y$, they are two independent discrete distributions.

$X_1,X_2,\ldots,X_N$ are drawn i.i.d from $P_X$, and $Y_1,\ldots,Y_N$ are drawn i.i.d from $P_Y$.

I got $P_X'$ is the type of $X_1,X_2,\ldots,X_N$ and $P_Y'$ is the type of $Y_1,Y_2,\ldots,Y_N$. That is, $P_X'$ converges to $P_X$ almost surely as $N$ goes to infinity, and $P_Y'$ converges to $P_Y$ almost surely as $N$ goes to infinity.

My claim is $D(P_X'\|P_Y')$ will converge to $D(P_X\|P_Y)$ as $N$ goes to infinity too, in the above intuition.

But as the KL-divergence $D(\cdot\|\cdot)$ is not continuous, how to start a rigorous proof that the claim is true or false?

Because there are something in log in KL-divergence, It's hard to write down a form like strong law of large number in wikipedia.

  • $\begingroup$ 1. "wiki" is not the same as "wikipedia"; and wikipedia is large, you really want to link to the specific page. 2. I'm really not sure this is signal processing, and not really math or statistics... $\endgroup$ – Marcus Müller Mar 15 '20 at 14:53
  • $\begingroup$ also, you've already answered your question yourself. If your function $D$ isn't continuous in an environment of $P_X,P_Y$, then you simply can't make a convergence statement. That's pretty much the definition of continuity of a function: plug in a convergent sequence (in this case, $(P'_{\cdot})_N$ for increasing $N$), and the output converges, iff the function is continuous around the convergence point of the sequence. $\endgroup$ – Marcus Müller Mar 15 '20 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.