What is causing these spikes in the output of the low pass filter?

Question

I am low pass filtering the PCM of an arbitrary Bell 103 (300 bps) signal. However, the output of the low pass filter shows random spikes in amplitude.

I have tried to increase the bandwidth, but no luck. Can someone explain why these spikes arise? My goal is to do some sort of envelope detection on this signal afterwards, to decode the 1's and 0's. For example by using a rectifier first.

The filter coefficients:

coeffs = [0.265842424474100 0.243276902464240 0.271096721997119 0.243276902464240 0.265842424474100]

The Matlab script used:

close all;
clear all;

% Generate a random bit sequence as input
bitsIn = randi([0 1], 16, 1)';

% Generate the amplitude of the FSK signal
amplitude = randi([7000 10000], 1, 1);

% Generate a randomized length of randomized startup noise
% x = randi([-20 20], 1, randi([0 3000], 1, 1));
x = []; % TODO

% FSK module the input bit stream
for i = 1:length(bitsIn)
    if (bitsIn(i) == 1)
        freq = 1270;
    else
        freq = 1070;
    end

    % Append the new PCM data for the new bit, with random phase
    phase = (2 * rand(1)) * pi;
    x = [x (amplitude * sin((2 * pi * freq * (0:26) / 8000) + phase))];
end

% Lowpass filter the resulting FSK samples
d = fdesign.lowpass('N,Fp,Fst,Ast',4,1070,1270,10,8000);
Hd = design(d,'equiripple');
y = filter(Hd,x);

figure();
plot(x);
title('Input data');
figure();
plot(y);
title('Low pass output');
```

Answer 1

I don't have MatLab with the DSP toolbox, but you're calling filter wrong. Use:

y = filter(Hd, [1], x);

to set the feedback taps properly to $[1]$.

You should also inspect your filter design by using

freqz(Hd, [1], 512, 8000);

Although my hunch is that amplitude spikes in the output are directly caused by the filtering of the phase discontinuities you intentionally added when performing FM modulation.

Really, no modern equipment uses frequency switching to perform FSK. FSK emitted from modern equipment is almost always continuous phase. Are you really dealing with an ancient piece of Bell 103 modem hardware?

An example Octave script that generates the complex I/Q samples of continuous phase FSK is here:

https://dsp.stackexchange.com/a/53834/28112

if you need an example of how to generate CPFSK.