I was going to post this as a comment, but then it became so long that I thought, this constitutes really as an answer in fact.
Your question reminds me of the lab-work in my MSc course on Adaptive Filters. We used "Wiener filter" to remove some unwanted background noise (wind and tire noise) from the recorded input to make the speech clearer. The Wiener filter is quite simple to begin with.
You can look at the wiener filter from the scipy package of python - scipy.signal.wiener.
Another advantage of Wiener filter could be that you don't have to do any additional computational step of taking the Fourier transform of the input. That is, Wiener filter can be applied to the time domain signal directly.
A short example of how to use Wiener would be:
from scipy.signal import wiener
import matplotlib.pylab as plt
plt.plot(t,signal,'k')
plt.plot(t,wiener(signal,mysize=55),'r',linewidth=3)
plt.show()
Going by the assumption that the "breathing sound" in your audio inputs would be quite uncorrelated to the actually speech signal, just like the wind and tire noise that we removed to make the speech clear in our audio data, I expect the filter to provided a reasonable output.
(Note: There were then additional processing steps added sometimes to improve the audio output like making the parameters of the filter adaptive. You have to play around with the parameters a bit, as in one case, I remember the Wiener filter produced an output like it was coming from a hollow pipe, if you chose/modified the parameters in a particularly wrong way. But as a start, just applying the Wiener filter to your signal in Python would be a good first step. And you can improve from their onward, step by step.)
And if you are curious as to how this filter works then you catch up more on the math here.
