# About convergence of KL divergence: if the two probability distributions are type, does the law of large number work?

If I pick $$N$$ samples from $$P_X$$ and $$P_Y$$, they are two independent discrete distributions.

$$X_1,X_2,\ldots,X_N$$ are drawn i.i.d from $$P_X$$, and $$Y_1,\ldots,Y_N$$ are drawn i.i.d from $$P_Y$$.

I got $$P_X'$$ is the type of $$X_1,X_2,\ldots,X_N$$ and $$P_Y'$$ is the type of $$Y_1,Y_2,\ldots,Y_N$$. That is, $$P_X'$$ converges to $$P_X$$ almost surely as $$N$$ goes to infinity, and $$P_Y'$$ converges to $$P_Y$$ almost surely as $$N$$ goes to infinity.

My claim is $$D(P_X'\|P_Y')$$ will converge to $$D(P_X\|P_Y)$$ as $$N$$ goes to infinity too, in the above intuition.

But as the KL-divergence $$D(\cdot\|\cdot)$$ is not continuous, how to start a rigorous proof that the claim is true or false?

Because there are something in log in KL-divergence, It's hard to write down a form like strong law of large number in wikipedia.

• 1. "wiki" is not the same as "wikipedia"; and wikipedia is large, you really want to link to the specific page. 2. I'm really not sure this is signal processing, and not really math or statistics... – Marcus Müller Mar 15 '20 at 14:53
• also, you've already answered your question yourself. If your function $D$ isn't continuous in an environment of $P_X,P_Y$, then you simply can't make a convergence statement. That's pretty much the definition of continuity of a function: plug in a convergent sequence (in this case, $(P'_{\cdot})_N$ for increasing $N$), and the output converges, iff the function is continuous around the convergence point of the sequence. – Marcus Müller Mar 15 '20 at 14:57