Yesterday I asked a question and got the answer to detrend my time series which I think is really better than applying a highpass filter. So I read the description of scipy.signal.detrend and there I learned that there is a parameter bp:

bp : array_like of ints, optional
A sequence of break points. If given, an individual linear fit is performed for each part of data between two break points. Break points are specified as indices into data. ...

I tested this feature with this program:

import math
import numpy as np
from scipy.signal import detrend
from matplotlib import pyplot as plt

low  = np.zeros((1500), dtype = float)
high = np.zeros((1500), dtype = float)
for i in range(500):
    p1 = i * math.tau / 1000
    p2 = i * math.tau /  500
    p3 = i * math.tau /   50
    c1 = -math.cos(p1)
    c2 =  math.cos(p2) / 2 + 1/2
    c3 =  math.cos(p1)
    c4 =  math.cos(p3) / 2
    low [i +    0] = c1
    low [i +  500] = c2
    low [i + 1000] = c3
    high[i +    0] = c4
    high[i +  500] = c4
    high[i + 1000] = c4

raw = low + high
detrended = detrend(raw, bp=[500,1000])
trend = raw - detrended

fig, (ax1, ax2, ax3, ax4, ax5) = plt.subplots(nrows = 5, ncols = 1, sharex = True)

ax1.plot(low, linestyle="solid")
ax1.set_title('low frequency data')

ax2.plot(high, linestyle="solid")
ax2.set_title('high frequency data')

ax3.plot(raw, linestyle="solid")
ax3.set_title('to be detrended = low + high')

ax4.plot(detrended, linestyle="solid")
ax4.set_title('detrended data')

ax5.plot(trend, linestyle="solid")


And I got this diagrams as output:

Five diagrams showing the low frequency(1) and high frequency(2) parts that, when added, give the signal to be detrended(3), the detrended signal(4) and the trend(5)

I put the break points at exactly that two points where the low frequency curves has its maxima.

As you can see, there are jumps in the trend line at the break points and therefore also the detrended signal has jumps that didn't exist in the raw data. This is exactly what I feared and expected, but not what I hoped to get.

I hoped to get something like this:

similar to the image before, differences are described in the text

The trendline still has sharp kinks (which is fine), but it no longer has jumps. Therefore, there are no jumps in the detrended data either.

Is there a simple way to detrend a signal with breakpoint that do not cause jumps?

  • $\begingroup$ It's an optional argument. Have you tried just leaving it on its default value? what happens then? $\endgroup$
    – Jdip
    Sep 13, 2022 at 14:04
  • $\begingroup$ @Jdip: I know that. Then it uses one linear trend line or a constant value, depending on the parameter 'type'. But I have to analyze time-series that are many hours long, recorded with a sampling rate of 256 Hz, so they consist of more than 5 million samples each. And I do not think that there is just one 6 hours long linear trend in my data. My idea was to divide my data in sections that are 10 seconds long (resulting in more than 2000 sections) and to detrend each section separately. But when I use scipy.signal.detrend as described, I will get more than 2000 jumps in my data. $\endgroup$ Sep 13, 2022 at 14:25
  • $\begingroup$ I see. Yeah detrending is not a trivial problem. Have you by any chance looked at the Github I linked in my original answer? $\endgroup$
    – Jdip
    Sep 13, 2022 at 14:29
  • $\begingroup$ There are other methods. Look for wavelet-based detrending methods (pick an appropriate wavelet, decompose to an appropriate level, remove the approximation coefficients and reconstruct), spline interpolation, exponential fit detrending, etc... You can also just see what you get with a simple high pass at different cut-off frequencies. You'll have to experiment quite a bit to get to something you like, there's no one-size-fits-all or magic method here... $\endgroup$
    – Jdip
    Sep 13, 2022 at 14:51
  • 1
    $\begingroup$ @Jdip: You are right, detrending is way more complicated than I thought before. I think I've found a solution that fits well for me. I posted it as an answer to my own question. I also browsed through that article on Github. I think I can use a lot of it for my project. Thank you! $\endgroup$ Sep 14, 2022 at 16:47

1 Answer 1


I'm answering my own question

I found a solution for my detrending problem that fits well to me, and maybe other researchers or developers who stumbled across this question might find this solution helpful for their own purposes. That's why I post it as an answer to my own question. But be aware, that other solutions exist and might fit better in other situations.

My solution:

Detrending means to calculate a certain simple trend function (usually a constant or a linear function) from the input data and then to subtract this trend from the original function. The difference is the detrended version of the original data.

But what if a simple linear function seems to be much too simple like in my example? (I have a time series with a few million samples recorded over many hours. Having only one linear function as a trend over the entire record does not seem very likely.) My idea is that the trend is a 'blurred' version of the input data, like a sliding average, calculated from a short section of the input data (a "sliding window") with a fixed length, but moving across the whole input data.

But the hard edges of such a window might cause some unwanted side effects, so I used a spline function with soft edges instead. There are many possible forms that such a spline might have. I have chosen this spline:

import math
import numpy as np
from matplotlib import pyplot as plt

half_splinewidth = 100
spline =  np.zeros((half_splinewidth * 2 + 1), dtype = float)
for n in range(half_splinewidth + 1):
    x = (n / half_splinewidth) ** 2 * math.pi
    w = (math.cos(x) + 1) / 2
    spline[half_splinewidth + n] = w
    spline[half_splinewidth - n] = w

# normalize such that the area under the curve == 1
integral = np.trapz(spline)
spline /= integral


Spline, as described in the text

I applied that spline to the raw data with an algorithm named convolution (wikipedia). Scipy can do this trick (scipy.signal.convolve):

from scipy.signal import convolve

trend = convolve(raw, spline, mode='same')
detrended = raw - trend

The parameter mode influences the behavior at the beginning and at the end of the input data and the length of the output. same means that the output has exactly the same length as the input and that the input is prefixed and appended with so many zeros that the convolution can be calculated.

Having the same length is what I wanted, but assuming leading and trailing zeros is not really optimal. As a result I got this:

enter image description here

The trend differs from the low frequency data at the beginning and the end of the output, but is very close to it in the main section of the data.

The behavior at the edges can be improved my manually adding a short section of a mirrored copy of the input data to the beginning and the end:

# Note that len(spline) == half_splinewidth * 2 + 1
padded = np.concatenate((
trend = convolve(padded, spline, mode='valid')

Note, that here I used mode='valid'.

And here is the result: better version of detrended data and trend

  • 2
    $\begingroup$ That's great @Hubert! Basically, you're applying a high pass filter. The "spline" - not sure about the terminology here, but let's not get into that for now :) - is a FIR low pass filter kernel with order 200 (that you use to filter your raw data using the convolve function). It has a cut-off at approximately 0.5Hz. It's an FIR so has linear-phase. You then subtract the output from the raw data, effectively removing the low frequencies (high-pass!). You designed a high-pass that works for your purposes, congrats! $\endgroup$
    – Jdip
    Sep 14, 2022 at 17:23
  • $\begingroup$ @Jdip: Why "order 200"? Is this the width of the kernel (the thing I called "spline")? $\endgroup$ Sep 14, 2022 at 19:41
  • $\begingroup$ Yep. Your filter has length 201, so order 200 (semantics, semantics !). You could also have used the function “filter” $\endgroup$
    – Jdip
    Sep 14, 2022 at 21:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.