There are a few different ways:
- Use a fractional multiply or data types. Unfortunately that's not a standard "C" or "C++" instruction but most embedded processors do support it and have some sort of non-standard way to call it.
- Use a "long int" 64-bit integer data type, do a double precision multiply.
- Convert everything to floating point (float or double) before multiplying.
- Scale the numbers by $2^{-16}$ (arithmetic right shift by 16-bits) before squaring.
Option 1 is typically the "best" if your processor supports it as a native instruction. Option 4 is often "good enough" if you have half-way decent signal to noise ratio but that may depend on the specifics or you application and data.
Note on "fractional" data types: This is a fixed point type that is scaled so that the data range is assumed to be from -1 to +1, hence a multiply can never overflow. That's actually the most "natural" type for fixed point signal processing. Multiplying two 32-bit numbers yields a 64 bit result. The difference between an integer and fractional multiply is that the integer multiply keeps the lower 32 bit and the fractional keeps the higher 32 bit of the result.
Both integer and fractional format are subsets of the "Q" number format. See https://en.wikipedia.org/wiki/Q_(number_format). Loosely speaking integers are Q0 and fractionals are Q31(signed).