Remove noise introduced while converting a Delta-Sigma Modulated (DSM) signal into a pulse-code modulated (PCM) signal?

Question

I'm writing a tool that converts a dsdiff audio file (dsd64) into a flac audio file (DSM -> PCM). The project is both a programming exercise and a signal processing exercise; I'm learning a lot from it.

Currently, I'm doing the following to compute a sample in the PCM stream

1. Select a window of the DSM sequence that corresponds to the size of the window of a low-pass (FIR) filter.
2. Replace all 0s in the DSM window with -1s
3. Convolve (multiply+sum) the elements in the DSM window with the coefficients of the (FIR) filter and rescale the result to fir into the chosen bitrate (24-bit).
4. Move the window by downsampling factor many elements (16 in my case) and apply repeat this process.

I tested the conversion using a few (arbitrarily chosen) dsdiff files (songs by the band deep purple). The conversion reproduces the signal (good) but introduces noise (bad) with the following profile (measured during the beginning of the track where there is a bit of silence in the original):

I've been trying to figure out how this is being introduced, and what I need to do to remove it. From my reading of random sources online I understand that this is introduced by integrating the DSM signal, which amplifies the noise floor and maps it to a spectrum similar to the one I see in my plot.

Did I identify the problem correctly? What can I do to verify this and, ultimately, fix it?

Any help is highly appreciated.

Edit:

Added a figure showing the filter's frequency response

Added the Periodogram of the first second of the DSM stream

Answer 1

The spectrum appears to be the noise-shaping associated with Delta-Sigma Modulation. Specifically we see that the noise increase follows a 20 dB/decade slope, indicative of a first-order delta-sigma modulator.

The high end of the noise rolls off consistent with the high frequency roll-off of the filter applied.

Below are some charts I have explaining the first-order Delta-Sigma modulator and how such noise shaping occurs, here specifically with a Delta-Sigma Digital to Analog Converter (but it could easily be a 1 bit digital output just the same).

And equivalently the Delta Sigma Analog to Digital converter diagram is shown below, which will be used to develop a simple loop model for providing an understanding of how noise shaping occurs.

Below is a qualitative loop model, providing an intuitive understanding of noise shaping if one has an appreciation of filtering in simple control loop models. Specifically it is showing an 1 bit A/D Converter as a gain with quantization noise added (this is a highly non-linear process, therefore the model is qualitative versus a quantitative prediction).

The path through the loop from the quantization noise input ($N_q$) to the output follows a high pass function as shown below. Quantization noise would typically be white noise (uniform over all frequency) for a traditional data converter, while with the implementations using Delta Sigma modulation we get this resulting noise shaping resulting in significantly higher precision at a given sampling rate after the high frequency noise is filtered out.

Typically higher order implementations are used, such as the 2nd order Delta Sigma shown below (and higher orders such as 5th order are commonly implemented). This provides much steeper noise shaping (much less noise in the band of interest after filtering) and also serves to significantly reduce pattern noise which would manifest as spurs in the spectrum.