Faster than Nyquist signaling is used to improve the spectral efficiency by reducing the time spacing (relaxing the orthogonality constraint) to pack more data in the same channel while tolerating a certain intersymbol interference (ISI).
What is exactly faster than Nyquist? Is it by increasing the sampling rate higher than the Nyquist rate? But this doesn't create ISI. What is the relationship between the Nyquist rate and the spacing in the time domain?
Harry Nyquist made so many contributions that it's easy to get confused.
Related to sampling, Nyquist proved that a signal $s(t)$ bandlimited to $B$ Hz can be reconstructed from samples taken at a rate larger than $2B$. This is unrelated to Faster than Nyquist (FTN) signaling, though.
Related to communications, Nyquist also showed that the maximum ISI-free pulse rate in a channel of bandwidth $B$ is $2B$ pulses per second, using a technique known as "orthogonal signaling".
Note that Nyquist didn't say that communication at a rate faster than $2B$ was impossible; he just said that you will have to handle the ISI somehow. For a long time, people didn't know how to do that, so $2B$ became engrained as a sort of actual bound.
One exception is partial response signaling, which introduces ISI in a controlled manner. However, in this case the main purpose is spectral shaping more than an increase in data rate.
In the 1970s, Mazo published a paper showing that, in some cases, the ISI can not only be handled, but it actually doesn't detract from a system's error rate performance. Start from an orthogonal signaling system operating at pulse rate $R=1/T$; that is, find an orthonormal set of pulses $p(t-kT)$ (for integers $k,l,m$) such that $$ p(t-lT)p(t-mT) = \begin{cases} 1,\text{when $l=m$} \\ 0,\text{otherwise.} \end{cases} $$ Now let's say that you want to transmit the data sequence $\lbrace a_0, a_1, a_2, \ldots, a_M \rbrace$. You can do that by transmitting $$ s(t) = \sum_{k=0}^M a_k p(t-kT) $$ and the receiver (ignoring noise for simplicity) can recover any $a_k$ by calculating $$ a_k = \int_{-\infty}^{\infty} s(t) p(t-kT)\, dt. $$
Mazo's proposal was to convert this orthogonal-signaling system to an FTN system by introducing a time-compression factor $0 < \tau < 1$, so that the transmitted signal becomes $$ s(t) = \sum_{k=0}^M a_k p(t-k \tau T) $$ Note that the bandwidth of $s(t)$ is still $B$, but now we transmit pulses at rate $1/(\tau T) > R$. Mazo's main result was to prove that, for certain pulses $p(t)$ and certain time-compression factors $\tau$, the bit-error rate performance of the accelerated system is the same as the orthogonal system.
At the time, the problem was that handling the ISI was impractical. The ISI acquires a certain structure that can be represented as a trellis, but in any non-trivial case the trellises were huge and well beyond the computational capabilities available.
It is only recently that some new techniques, along with faster computers, have made FTN somewhat practical. I recommend this tutorial paper:
J. B. Anderson, F. Rusek, and V. wall, “Faster-Than- Nyquist Signaling,” Proceedings of the IEEE, vol. 101, no. 8, pp. 1817–1830, Aug. 2013.
Mazo's original paper is:
J. E. Mazo, “Faster-Than-Nyquist Signaling,” Bell System Technical Journal, vol. 54, no. 8, 1975.
