So the point is, oversampling with a factor of $N$ requires your signal to be confined to at most $\frac1N$ of your Nyquist bandwidth. Otherwise, you're not oversampling.
If the signal is six time oversampled, then there's no loss of information/aliasing when reducing the sample rate by six.
If I would oversample by 2 and then downsample it by 2 without AA-filter I would have stuff folding back to by final Nyquist band.
No, you wouldn't; there's, by definition of oversampling, no energy in the "upper half" of your spectrum.
Now, through discussion in the comments it became clear that "oversampled signal" might be ambiguous:
If you had your signal alone, AA filtering would do nothing, because there's no spectral component that could be folded back (aliased) into the decimated Nyquist band. However, if we're observing noisy signals, you'd have noise energy (and be it just the quantization noise) that is relatively wideband. Thus, in that situation, although the signal of interest is oversampled, the noise is not, and to avoid getting noise aliased into our band of interest, proper AA filtering is necessary.
This is really the difference between mathematically perfect signals and noisy signals; for example, if you generated a digital sine with 100 Samples per period, you could, without incurring any problems, decimate strongly – because a digitally generated signal typically has zero noise.