The nonlinear trick is in fact often used to recover the symbol clock of $\frac \pi {2N}$ phase-shifted signals, e.g. BPSK, QPSK: Squaring samples $x[n]$ (or even taking $x^4$) leads to a removal of phase information, and you can then optimize timing with something that is pseudo-derivate based like Müller&Muller.
Now, the problem with QAM is usually that there's no two constellation points with the same phase; so an arbitrary amount of squaring can not reduce the number of phases to 1.
For such signals synchronization is complicated even with exact knowledge of what to expect; it's pretty common to see preamble-correlation based synchronization in receivers. Upside of that is that you can not only get timing, but also a fair amount of CSI that way, which leads to the possibility of equalization.
As a first step, however, some kind of autocorrelation-based detector surely sounds promising: Assuming that with a finite set of constellation points, there's also only finitely many sequences of symbols within the influence of a single pulse shaping filter impulse response, which also means that one might expect the signal to repeat after that amount of time. Hence, a peak on the autocorrelation¹ function of your signal probably signifies the length of $N,\, N\in\mathbb N$ symbols.
With that known, you'd still be on the lookout for $N$; remember that the spectrum of a data-modulated signal is the pulse shape convolved with a diraccomb with an inter-dirac distance of $\frac{1}{f_{symbol}}$. Hoping that symbol timing coincides with the pulse shape filter's time domain roots, you could just look at a sufficiently oversampled signal and count the spectral "peaks"; in reality, this will typically not be as clear.
A few other hints: If the spectral shape is wideband and rectangular, possibly even with a bit of "sincy" sidelobe on either side, it's probably OFDM. That'll make your life a little harder, but: OFDM typically uses cyclic prefixes, which means that you can find the OFDM symbol duration using autocorrelation; since the cyclic prefix needs to be a whole number of subsymbols (IDFT bins), you can narrow down the possible OFDM configurations quite a bit.
¹In fact, any periodicity-based estimator might work; have a look at thegr-specest toolbox. It has some parametric spectrum estimators, of which MTM might be very interesting, here.
