This depends a bit how rigorous you define "allpass" filter.
You can show that any pole can be turned into an allpass filter if, and ONLY if, it you pair it with a zero at the inverse location. A zero at the inverse location is the only way to achieve $|H(\omega)|^2 = 1$ for all $\omega$.
Poles can be complex or real. Complex poles result in second order allpass filters, real poles in first order allpass filters.
Pure delays are simply a special case of a single real pole at z=0 and the matching zero at z=infinity. The transfer function of a one sample delay is almost identical to that of an allpass filter with a pole at .00001 or -.00001. Multi sample delays are simply cascades of single sample delays, so an N-sample delay has N poles at z=0 and N zeros at z=infinity
All of these taken together implies that, indeed, you can represent every allpass filter as a cascade of first and second order allpass filters.
There is a downside to that as well: For first order section the phase is zero at DC and $\pi$ at Nyquist. For a second order it starts at 0 and ends at $2\cdot \pi$ and the phase decreases monotonically between those values. These properties are maintained in a cascade so we can conclude that any N-th order allpass has a phase of 0 at DC and a phase of $N \cdot \pi$ at Nyquist with a monotonically decreasing phase in between. That's too restrictive for many applications. (I'm ignoring a potential multiply with -1 flip here, which would just result on an phase offset of $\pi$).
There is a different class of filters that are "almost" allpass filters, i.e. you can make the magnitude as close to unity as needed over the frequency range of interest. These can not be represented as cascades of 1st and 2nd order allpass filters.
An interesting example of this is a Hilbert transformer: it has unity amplitude gain but a phase shift that's only 90 degrees at Nqyuist. You could interpret this as having an infinite amount of poles and zeroes but that's not very useful. You can't implement an ideal Hilbert transformer, but you most certainly can do something that's "good enough" for your specific application.