0
$\begingroup$

I am completing Fourier analysis on many different time series of sediment particle flux exiting an experimental flume. Data is collected at a resolution of 1 Hz for durations ranging from ~5,000 to 50,000 seconds. Flux ranges from 0 to over 1,000 particles. The figure below shows an example time series that I am analyzing - the x-axis has units of seconds and the y-axis has units of particles/second: -

I am familiar with time series and Fourier analyses, but I am not an expert. A colleague told me that I should detrend all of my time series before performing the Fourier analysis. I have completed the analysis without, and with detrending the time series and I see no obvious difference in the resultant spectra; this does not surprise me because I see no secular trend, for example, in the data shown in the example figure above. This leaves me with several questions. First, should I detrend the example time series shown in the figure above? If I should, why (or why not)? Second, there are at least several ways I can think of to detrend the example time series (e.g. subtracting a polynomial fit, subtracting the series mean, etc.), do people have any suggestions for a time series like the example one shown above? Last, are there clear criteria I can work with in making a decision of whether detrending is necessary prior to completing a Fourier analysis? Thanks.

|improve this question
$\endgroup$
  • $\begingroup$ Time series have two broad qualities: Trend and Periodicity. But beyond that, there are no hard definitions about them. To an extend they depend on the problem (conceptually) and the time scales involved. And as you say, you can remove a simple drift or the trend in the moving-average sense, or fit an ARMA model and then remove its output and so on. I cannot see why would the flow of particles of a certain size down a canal would vary periodically though. Is the canal shaped in such a way as to form "tuned cavities" for particles of certain size? Can you talk a bit more about the problem? $\endgroup$ – A_A Jan 25 at 7:07
  • $\begingroup$ Here are some more details about the problem @A_A. I ran experiments that produced different riverbed shapes. Different shapes affect how sediment moves along the river bed. Some shapes produce sediment motions as well defined waves, with periodic behavior. Other shapes produce sediment motions that are random in time and space. I am using the spectral analysis to understand of my data sets reflect how riverbed shape is thought to affect sediment motions. I have an unusually rich data set so I am in a good position to explore these ideas. $\endgroup$ – SurfProc Jan 25 at 16:37
  • $\begingroup$ A clarification on the second to last sentence of my comment above: "I am using spectral analysis to understand the coupling between riverbed shape and sediment motions" $\endgroup$ – SurfProc Jan 25 at 16:53
  • $\begingroup$ Very interesting. But by the sound of it and assuming that you start with a flat river bed, any detrending would be interfering with the observations from the phenomenon itself. Besides that, you might also want to have a look at some non-linear techniques for discovering more complex periodicities $\endgroup$ – A_A Jan 27 at 23:09
  • $\begingroup$ The sediment flux measurements are separate from bed elevation measurements, and the analysis discussed here does not interfere with the phenomenon itself. The analysis described here is trying to understand if and what kind of relationship exists between a particular bed elevation state and a finite time series of sediment flux. I will take a look at the suggested links, which look promising by the way. I am quite interested in nonlinear dynamics. $\endgroup$ – SurfProc Jan 28 at 0:28
1
$\begingroup$

How you pre-process data is related to the purpose of the analysis and since you haven't mentioned why you are doing spectral analysis, answering your question can't go beyond making some general remarks.

Detrending is a preprocessing operation. You want to compensate or remove some bias from your data. As an example, many kinds of data are collected in the presence of long term phenomena that is extraneous to what your are interested in finding. Many kinds of data have seasonal, diurnal, weekly, monthly, tidal, .... ect cycles associated with them. Unemployment has a seasonal dependence to it as an example.

I once was tasked with looking for jumps in syndrome data that is reported to health departments and there was strong increases on Mondays and after holidays in reports but that had to do with many doctor's offices being closed on weekends. This was an artifact of how the data is collected and not how epidemics evolve.

Matlab's dtrend function removes a linear fit from data. If the collection interval is small compared to some much longer trend in the data, it reduces the bias introduced by taking a section that is less than the period of the longer term trend, particularly for data like your which is strictly positive. One may conceptualize dtrending as reducing low frequency spectral leakage that would otherwise mask features in your spectra.

Again, without knowing what you are looking for in your data, no specific answer can be recomended for your situation.

|improve this answer
$\endgroup$
  • $\begingroup$ I just added some detail about why I am using spectral analysis in my comment above to A_A. Short of that though, your answer taught me one thing I had not fully appreciated, that the how of pre-processing is driven by the questions being addressed. The main question I am addressing is whether spectral analysis can distinguish periods of random from non-random sediment motions, which are measured at one fixed point in space. My example data reflects steady supply conditions, so long term phenomenon with periods >> than my collection interval should be damped. $\endgroup$ – SurfProc Jan 25 at 16:59

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.