0
$\begingroup$

I am getting back to DSP applications after a while and I've written a real time audio player in Python.

Basically I'm reading chunks of data from a .wav file (I am doing this to simulate a real time scenario, where I receive real time audio input from a source) and I am playing each chunk in real time applying a simple filter with a dynamic highcut frequency: it starts from 300Hz and then gradually moves until 20000Hz (basically no high-cut).

I am using PyAudio for the real time playing and scipy for the simple filter. The code is the following, it is self-contained and reproducible, you have just to change the filename variable with the path of your wave file.

import pyaudio
import wave
import time
import numpy as np
import scipy.io.wavfile as sw
import librosa
import scipy
import sys
from scipy.io.wavfile import write


############ Global variables ###################
filename = '../wav/The_Weeknd.wav' #Test file
chunk = 512 #frame size
#Conversion from np to pyAudio types
np_to_pa_format = {
    np.dtype('float32') : pyaudio.paFloat32,
    np.dtype('int32') : pyaudio.paInt32,
    np.dtype('int16') : pyaudio.paInt16,
    np.dtype('int8') : pyaudio.paInt8,
    np.dtype('uint8') : pyaudio.paUInt8
}
np_type_to_sample_width = {
    np.dtype('float32') : 4,
    np.dtype('int32') : 4,
    np.dtype('int16') : 3,
    np.dtype('int8') : 1,
    np.dtype('uint8') : 1
}
STEREO = 2 #channels
#################################################

# Simple class which reads an input test wav file and reproduce it in a real time fashion. Used to test real time functioning.
class Player:
    # Loading the input test file. Crop to 30 seconds length
    def __init__(self):
        self.input_array, self.sample_rate = librosa.load(filename, sr=44100, dtype=np.float32, duration=60)

        #print(self.sample_rate)
        #print(self.input_array.shape)
        self.cycle_count = 0
        self.highcut = 300

    def bandPassFilter(self,signal, highcut):
        fs = 44100
        lowcut = 20
        highcut = highcut

        nyq= 0.5 * fs
        low = lowcut / nyq
        high = highcut / nyq

        order = 2

        b, a = scipy.signal.butter(order, [low,high], 'bandpass', analog=False)
        y = scipy.signal.filtfilt(b,a,signal, axis=0)
        return(y)

    def pyaudio_callback(self,in_data, frame_count, time_info, status):
        audio_size = np.shape(self.input_array)[0]
        #print(audio_size)
        print('frame count: ', frame_count)

        if frame_count*self.cycle_count > audio_size:
            # Processing is complete.
            print('processing complete')
            return (None, pyaudio.paComplete)
        elif frame_count*(self.cycle_count+1) > audio_size:
            # Last frame to process.
            print('1 left frame')
            frames_left = audio_size - frame_count*self.cycle_count
        else:
            # Every other frame.
            print('everyotherframe')
            frames_left = frame_count

        data = self.input_array[frame_count*self.cycle_count:frame_count*self.cycle_count+frames_left]
        data = self.bandPassFilter(data, self.highcut)
        if(self.highcut<20000):
            self.highcut += 10

        print('len of data', data.shape)

        #write('test.wav', 44100, data) #Saves correctly the file!
        out_data = data.astype(np.float32).tobytes()
        print('printing length: ',len(out_data))
        #print(out_data)
        self.cycle_count+=1
        print(self.cycle_count)
        print('pyaudio continue value: ',pyaudio.paContinue)
        return (out_data, pyaudio.paContinue)





    def start_non_blocking_processing(self, save_output=True, frame_count=2**10, listen_output=True):
        '''
        Non blocking mode works on a different thread, therefore, the main thread must be kept active with, for example:
            while processing():
                time.sleep(1)
        '''
        self.save_output = save_output
        self.frame_count = frame_count

        # Initiate PyAudio
        self.pa = pyaudio.PyAudio()
        # Open stream using callback
        self.stream = self.pa.open(format=np_to_pa_format[self.input_array.dtype],
                        channels=1,
                        rate=self.sample_rate,
                        output=listen_output,
                        input=not listen_output,
                        stream_callback=self.pyaudio_callback,
                        frames_per_buffer=frame_count)

        # Start the stream
        self.stream.start_stream()


    def processing(self):
        '''
        Returns true if the PyAudio stream is still active in non blocking mode.
        MUST be called AFTER self.start_non_blocking_processing.
        '''
        return self.stream.is_active()

    def terminate_processing(self):
        '''
        Terminates stream opened by self.start_non_blocking_processing.
        MUST be called AFTER self.processing returns False.
        '''
        # Stop stream.
        self.stream.stop_stream()
        self.stream.close()

        # Close PyAudio.
        self.pa.terminate()

        # Resets count.
        self.cycle_count = 0
        # Resets output.
        self.output_array = np.array([[], []], dtype=self.input_array.dtype).T



if __name__ == "__main__":
    print('RUNNING MAIN')
    player = Player()
    player.start_non_blocking_processing()
    while(player.processing()):
        time.sleep(0.1)
    player.terminate_processing()

What I am doing is simply read chunks of audio from the wav file, process them applying a filter to the single chunk of data and then send that chunk to audio playing.

The code works fine, the audio is correctly reproduced and the filter does its job.

The problem is that I get audio crackles in audio reproduction and I can't figure out why. I studied DSP in the past and I know techniques such as overlap and add ecc... but it's been a while and i don't know if they could solve my problem (or the "error" lies somewhere else)

$\endgroup$
3
$\begingroup$

Couple of things:

  • You really can't print to the console in a high-speed context, especially not in audio callback functions. That printing has side effects, and it's slower than you think, I promise. This alone breaks "real-timedness": You can never guarantee how fast your printing is (or isn't).
  • Ahhhh! You're re-designing the filter for every single piece of audio! That's terrible! It's always the same filter. Calculate it once, and apply it forever.
  • filtfilt is not the function you need. I saw that a hundred times: People use filtfilt because it has a name that reads a bit like it's the filtering operation they want, but it's not. You just need plain convolution.
  • After fixing these showstoppers, you still need to be aware that filters have state, so you need to save the filter state after each bit of filtered audio for the next part.
$\endgroup$
8
  • $\begingroup$ Yes, I'm totally aware of the first 2 points! It's a small experimental code, so those things (prints ecc...) are going to be removed for the final application. Anyway I'm declaring the filter everytime because I want to apply a dynamic high cut (so not a fixed one). From the theoretical point of view, I was worried about the cracking sound I get, which one is the cause of this problem? $\endgroup$ – Mattia Surricchio Dec 28 '20 at 11:33
  • 3
    $\begingroup$ yes, but doing this in the real-time context introduces non-deterministic varying and most importantly large latency, which is one of the reasons you get the non-continuous effects. You can't do this later, this needs to be fixed first. These are not "non-pretty", these are bugs. All four of the reasons I've mentioned are critical and can/will/do lead to cracks. $\endgroup$ – Marcus Müller Dec 28 '20 at 13:11
  • 1
    $\begingroup$ to make your filter adjustable, you really don't have to recalculate it every time – you should do that only when it actually changes, and outside of the callback function, and with a safe way of switching or transitioning between the old and the new filter, and you still need to keep your filter state. $\endgroup$ – Marcus Müller Dec 28 '20 at 13:14
  • 3
    $\begingroup$ overlap and save solves exactly the fourth point. $\endgroup$ – Marcus Müller Dec 28 '20 at 14:07
  • 2
    $\begingroup$ If the crack is periodic on the order of your processing batch (every N samples) then there is almost certainly an issue with your filter states as @MarcusMüller is saying. You want to carry over those tap values between batches. $\endgroup$ – Keegs Dec 28 '20 at 18:14
1
$\begingroup$

Thanks to @Marcus Muller answer and comments in my question, I managed to solve th problem. As he pointed out:

filtfilt is not the function you need. I saw that a hundred times: People use filtfilt because it has a name that reads a bit like it's the filtering operation they want, but it's not. You just need plain convolution.

After fixing these showstoppers, you still need to be aware that filters have state, so you need to save the filter state after each bit of filtered audio for the next part.

These 2 points were the major problem in my code. filtfilt is not meant to work in real time scenarios, while lfilter is better in these cases, you can find a really useful comparison here: lfilter vs filtfilt

The full working code is the following, it basically plays a .wav audio in real time and adds a dynamic filter to it. The final result will be a sliding cut-off frequency while the audio is playing.

import pyaudio
import wave
import time
import numpy as np
import scipy.io.wavfile as sw
import librosa
import scipy.signal
import scipy
import sys
from scipy.io.wavfile import write


############ Global variables ###################
filename = '../wav/The_Weeknd.wav' #Test file
chunk = 512 #frame size
#Conversion from np to pyAudio types
np_to_pa_format = {
    np.dtype('float32') : pyaudio.paFloat32,
    np.dtype('int32') : pyaudio.paInt32,
    np.dtype('int16') : pyaudio.paInt16,
    np.dtype('int8') : pyaudio.paInt8,
    np.dtype('uint8') : pyaudio.paUInt8
}
np_type_to_sample_width = {
    np.dtype('float32') : 4,
    np.dtype('int32') : 4,
    np.dtype('int16') : 3,
    np.dtype('int8') : 1,
    np.dtype('uint8') : 1
}
STEREO = 2 #channels
#################################################

# Simple class which reads an input test wav file and reproduce it in a real time fashion. Used to test real time functioning.
class Player:
    # Loading the input test file. Crop to 30 seconds length
    def __init__(self):
        self.input_array, self.sample_rate = librosa.load(filename, sr=44100, dtype=np.float32, duration=60)

        #print(self.sample_rate)
        #print(self.input_array.shape)
        self.cycle_count = 0
        self.highcut = 300
        self.filter_state = np.zeros(4)

    def bandPassFilter(self,signal, highcut):
        fs = 44100
        lowcut = 20
        highcut = highcut

        nyq= 0.5 * fs
        low = lowcut / nyq
        high = highcut / nyq

        order = 2

        b, a = scipy.signal.butter(order, [low,high], 'bandpass', analog=False)

        y, self.filter_state = scipy.signal.lfilter(b,a,signal, axis=0, zi=self.filter_state) # NB: filtfilt needs forward and backward information to filter. So it can't be used in realtime filtering where i have no info about future samples! lfilter is better for real time applications!
        return(y)

    def pyaudio_callback(self,in_data, frame_count, time_info, status):
        audio_size = np.shape(self.input_array)[0]
        #print(audio_size)
        #print('SIGNORAAAAAA')
        #print('frame count: ', frame_count)

        if frame_count*self.cycle_count > audio_size:
            # Processing is complete.
            #print('processing complete')
            return (None, pyaudio.paComplete)
        elif frame_count*(self.cycle_count+1) > audio_size:
            # Last frame to process.
            #print('1 left frame')
            frames_left = audio_size - frame_count*self.cycle_count
        else:
            # Every other frame.
            #print('everyotherframe')
            frames_left = frame_count

        data = self.input_array[frame_count*self.cycle_count:frame_count*self.cycle_count+frames_left]
        data = self.bandPassFilter(data, self.highcut)
        if(self.highcut<20000):
            self.highcut += 1

        #print('len of data', data.shape)

        #write('test.wav', 44100, data) #Saves correctly the file!
        out_data = data.astype(np.float32).tobytes()
        #print('printing length: ',len(out_data))
        #print(out_data)
        self.cycle_count+=1
        #print(self.cycle_count)
        #print('pyaudio continue value: ',pyaudio.paContinue)
        return (out_data, pyaudio.paContinue)





    def start_non_blocking_processing(self, save_output=True, frame_count=2**10, listen_output=True):
        '''
        Non blocking mode works on a different thread, therefore, the main thread must be kept active with, for example:
            while processing():
                time.sleep(1)
        '''
        self.save_output = save_output
        self.frame_count = frame_count

        # Initiate PyAudio
        self.pa = pyaudio.PyAudio()
        # Open stream using callback
        self.stream = self.pa.open(format=np_to_pa_format[self.input_array.dtype],
                        channels=1,
                        rate=self.sample_rate,
                        output=listen_output,
                        input=not listen_output,
                        stream_callback=self.pyaudio_callback,
                        frames_per_buffer=frame_count)

        # Start the stream
        self.stream.start_stream()


    def processing(self):
        '''
        Returns true if the PyAudio stream is still active in non blocking mode.
        MUST be called AFTER self.start_non_blocking_processing.
        '''
        return self.stream.is_active()

    def terminate_processing(self):
        '''
        Terminates stream opened by self.start_non_blocking_processing.
        MUST be called AFTER self.processing returns False.
        '''
        # Stop stream.
        self.stream.stop_stream()
        self.stream.close()

        # Close PyAudio.
        self.pa.terminate()

        # Resets count.
        self.cycle_count = 0
        # Resets output.
        self.output_array = np.array([[], []], dtype=self.input_array.dtype).T



if __name__ == "__main__":
    print('RUNNING MAIN')
    player = Player()
    player.start_non_blocking_processing()
    while(player.processing()):
        time.sleep(0.1)
    player.terminate_processing()

NB: the code is still a prototype and still has some problems, in particular the following one as mentioned by Marcus:

Ahhhh! You're re-designing the filter for every single piece of audio! That's terrible! It's always the same filter. Calculate it once, and apply it forever.

I have been trying to find a nice way to create a "dynamic" filter (changing the cut-off frequency without re-create the filter each time) but I haven't found a nicer solution.

$\endgroup$
3
  • 2
    $\begingroup$ why don't you ask "how can I implement a dynamic filter" (describing how dynamic "dynamic" is, and the application of it, and constraints) as a new question? Certainly will be read by more people than your plea for comments here! $\endgroup$ – Marcus Müller Dec 29 '20 at 12:23
  • $\begingroup$ You are definitely right! $\endgroup$ – Mattia Surricchio Dec 29 '20 at 12:26
  • 1
    $\begingroup$ A quick and dirty way of dynamically changing your filter is to calculate a, b = scipy.signal.butter in a separate thread and send it to your audio thread. Too-large jumps in filter characteristics will cause their own clicks & pops. $\endgroup$ – TimWescott Dec 29 '20 at 22:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.