For what you intend to do, a low-pass filter is the way to go. Your statement about filtering frequencies vs filtering amplitudes is incorrect. Your signal contains many components at many frequencies, the amplitude of which varies in time, and the high frequency components are those causing the "jaggedness" and you want to get rid of them. Not sure why you say your signal is "constant frequency" - maybe you are getting confused by the sample rate?
What you have tried (averaging) is indeed a special case of low-pass filtering, but one with a frequency response far from being ideal. You should try a properly designed IIR or FIR filter. In particular, the FIR filter is not very different from what you tried - this is just a weighted combination of the samples neighboring each sample. But the choice of the coefficients is important and ensures that only unwanted components are eliminated. Note that an FFT is not the way to go. This question crops up here quite frequently under different forms, but in short - FFT, messing with coefficients, IFFT - is a bad idea.
By design, the output of a moving average filter (what you implemented) has less energy than the input. It is thus impossible for a moving average filter to cause distortion. If the input signal is in the [-1, 1] range, there is no way for an averaging filter to yield values outside this range. The "distorted sound" you observed was probably due to an implementation error on your side (overflow / clipping of integer values, signed values treated as unsigned value, or maybe incorrect in-place processing)...
EDIT: one thing worth mentioning is that there are situations in which a speech signal actually has high frequency components (appears "jagged", is noisy) - for example during a sss or shhh ; and removing those with a low-pass filter will affect its brilliance. Ideally, you'll want your low-pass filter to be active only when you detect that your speech signal is voiced - and inhibit it when you detect an unvoiced, noisy consonant.