My issue is with the fact that the point of the adaptive filter and LMS algorithim is to drive the error e(t) to zero correct? Then surely, Speaker B would hear nothing as the algorithim converges.
A usual arrangement of echo cancelers in a full-duplex system is shown in the following figure (from Digital Communication by Lee and Messerschmitt):
The echo cancelers try to remove any residual transmitted signal in the receiver of the same end. Since the input to the echo canceler is the transmitted signal, it can only remove signal components in the received signal that are correlated with the transmitted signal. It cannot remove any other signal components, such as the signal that is transmitted from the far end (which is the desired received signal). Consequently, the error signal can never become zero, and the desired signal (transmitted from the other end) will not be influenced by the adaptive filter.