I can tell you what to do but I can't tell you how to do it in Python code. To average four spectra, do the following:
1) Multiply input samples x[0] –thru- x[4095] by a 4096-point Hanning sequence.
2) Compute a 4096-pt FFT of the above product sequence producing an X1[m] spectrum.
3) Multiply input samples x[2048] –thru- x[6143] by a 4096-point Hanning sequence.
4) Compute a 4096-pt FFT of the above product sequence producing an X2[m] spectrum.
5) Multiply input samples x[4096] –thru- x[8191] by a 4096-point Hanning sequence.
6) Compute a 4096-pt FFT of the above product sequence producing an X3[m] spectrum.
7) Multiply input samples x[6144] –thru- x[10239] by a 4096-point Hanning sequence.
8) Compute a 4096-pt FFT of the above product sequence producing an X4[m] spectrum.
9) Finally, average the magnitude-squared samples of the four spectra to produce a final |Xf[m]| magnitude spectrum. That is,
|Xf[0]| = (|X1[0]| + |X2[0]| + |X3[0]| + |X4[0]|)/4.
|Xf[1]| = (|X1[1]| + |X2[1]| + |X3[1]| + |X4[1]|)/4.
|Xf[2]| = (|X1[2]| + |X2[2]| + |X3[2]| + |X4[2]|)/4.
...
...
|Xf[4095]| = (|X1[4095]| + |X2[4095]| + |X3[4095]| + |X4[4095]|)/4.
If you desire a power spectrum, then merely square each of the |Xf[m]| magnitude samples. user25139, you're implementing Welch's Method of spectrum analysis. You can search the web for "Welch's Method" for more information.