# Problem with implementing DC offset for streaming data

I am new to the DSP world. I'm trying to run a raw input signal through a LPF, remove any DC offset, amplify the signal, and then decode the signal. The problem with my code seems to be the implementation of my DC offset technique. I believe it's best to remove any DC values before amp'ing a signal, but after I run my code, I get chunks of data that are 'stitched' together poorly.

Here is my original input signal:

I receive this signal in windows/chunks of streaming data. I then run each chunk through a LPF. To keep the LPF output near zero on the x-axis (for amplification purposes) I perform a DC offset by taking the mean value of each chunk of data and subtracting that value from each sample in that particular chunk. Below you can see the result of this. The blue line is the output from the LPF; the orange line is the output from the LPF and after the DC offset has been performed.

During very large amplitude transitions, like the portion of the LPF output signal (blue line) that has the large negative slope to it, you'd expect the DC offset to skew the continuity of the signal greatly (as seen in the large saw-tooth portion of the orange signal). After the input signal amplitude seems to level out, though, the hope would be that the DC offset would not introduce such drastic transitions, even at small values/levels. But when you zoom in on the orange line, you see that abrupt transitions (circled below) occur between chunks of data still. These are perfectly vertical lines connecting the chunks of data. These abrupt changes make it basically impossible to decode any information from the signal once this signal is amplified.

Assuming that the only way to decode this signal is to get rid of these abrupt changes or at least smooth them out, how do I properly implement a DC offset? I've seen a few solutions but I'm not sure if they are correct or proper. Is my solution the best, or should I perform the offset based off of a buffer of old data chunk mean values, or should I run the data over a high-pass filter, or should I find a best fit line over several sample chunks and detrend the signal based off of the best fit line? What is the de facto/professional way of doing this?

Thanks for any help. My (cleaned up) code is below for reference.

import numpy as np

class <generic>:

def handle_new_data(self, floatdata):

# Turn into a numpy array
self.audio_buf_data = np.array(floatdata[0, :][:])

# Create timestamps for data samples
t = np.arange(self.audio_buf_data.size) / SAMPLING_RATE  # time axis

# Convert data to 16-bit
int16_data = self.audio_buf_data * 32768

########## ENVELOPE DETECTION / AM Demod ###########

# Cut signal in half (analog diode equivalent)
abs_signal = np.abs(int16_data[:])

# LPF
amplitude_envelope = self.butter_worth(abs_signal)

# Remove DC offset from LPF signal (Keep signal around x-axis so that amplifying signal gives max effect)
amplitude_envelope = amplitude_envelope - (amplitude_envelope.max() - ((amplitude_envelope.max() - amplitude_envelope.min()) / 2))

# Amplify LPF output without clipping signal
amplified_signal = amplitude_envelope * (32768 * 0.99 / amplitude_envelope.max())

# decode signal to get data
decode_func(amplified_signal)

• the way to remove a DC is with a high-pass filter have a zero directly on $z=1$ and a pole very close to it. it's the same as a LPF (with DC gain of 0 dB) being subtracted from a wire. $$y[n] = y[n-1] - (1-p) y[n-1] + x[n] - x[n-1]$$ where $p$ is the real pole and very close to one. Aug 26 '20 at 22:54
• Thx, but that does not help me with the second part of my question: how do I prevent the abrupt transitions between data chunks, as I circled above? Aug 27 '20 at 17:24
• have you tried the simple high-pass filter illustrated above? Aug 28 '20 at 8:49