If I pick $N$ samples from $P_X$ and $P_Y$, they are two independent discrete distributions.
$X_1,X_2,\ldots,X_N$ are drawn i.i.d from $P_X$, and $Y_1,\ldots,Y_N$ are drawn i.i.d from $P_Y$.
I got $P_X'$ is the type of $X_1,X_2,\ldots,X_N$ and $P_Y'$ is the type of $Y_1,Y_2,\ldots,Y_N$. That is, $P_X'$ converges to $P_X$ almost surely as $N$ goes to infinity, and $P_Y'$ converges to $P_Y$ almost surely as $N$ goes to infinity.
My claim is $D(P_X'\|P_Y')$ will converge to $D(P_X\|P_Y)$ as $N$ goes to infinity too, in the above intuition.
But as the KL-divergence $D(\cdot\|\cdot)$ is not continuous, how to start a rigorous proof that the claim is true or false?
Because there are something in log in KL-divergence, It's hard to write down a form like strong law of large number in wikipedia.