0
$\begingroup$

Goal

Substitute outliers in a time series by most recent valid data

Problem

The time series (end-of-day stock prices) has several 'uncomfortable' properties:

  • It is non-stationary and can have components of low and high frequency (trends and sudden price moves)
  • There can be missing data for some days, typically single occurrences, but sometimes a whole consecutive segment is missing
  • Some values are erroneous (implausibly high or low i.e. due to misplaced decimal sign). These are the real outliers and don't belong to the series.

There is an additional caveat: The substitution algorithm must not look into the future of the time series since it will be used in a machine learning scenario.

Strategy

What I thought of so far:

  1. Detrending the series. Maybe rolling linear detrending by the last n values. Will still be a problem with very short term moves.
  2. Replacing the missing values by extrapolating the trend identified in (1)
  3. Calculating the z-score and replacing values with z > 6 by extrapolating the trend identified in (1)

As the tags suggest, I will realise this in python / scipy. However I don't have much experience with time series analysis. I have no idea if this is an appropriate approach or if I'm missing something important. Any help is appreciated. Thanks.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Problem

The time series (end-of-day stock prices) has several 'uncomfortable' properties:

  • It is non-stationary and can have components of low and high frequency (trends and sudden price moves)

These are not outliers though. These are valid signal behaviour.

  • There can be missing data for some days, typically single occurrences, but sometimes a whole consecutive segment is missing
  • Some values are erroneous (implausibly high or low i.e. due to misplaced decimal sign). These are the real outliers and don't belong to the series.

To reduce the effect of implausibly high or low values you can use a rolling median filter at a suitable time scale (e.g. depending on your sampling frequency, you might want to get the median of a day's transactions).

Missing data is a more challenging problem. A few missing values might be "patched" with some interpolation scheme but if you have almost a day's worth transactions missing, it is probably better to break the analysis into segments too. Otherwise, you would be assuming a lot for a part of the signal that you don't really know what happened in reality.

Hope this helps.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for your answer. The problem with the median filter is that it will require future data of a given point in time. That mustn't happen. I previously didn't make that point clear in my question and just edited it. $\endgroup$
    – ascripter
    Feb 16, 2018 at 11:02
  • $\begingroup$ @ascripter How does the median filter requires future data? Do you mean that you require an online algorithm? $\endgroup$
    – A_A
    Feb 16, 2018 at 11:16
  • $\begingroup$ Not exactly an online algorithm, but if a train a learner on a median filtered time series, each timestep t will implicitly contain data in the interval [t-x,t+x], x being half the filter width. However the interval (t,t+x] won't be availabe later in a prediction scenario. $\endgroup$
    – ascripter
    Feb 16, 2018 at 11:22
  • $\begingroup$ @ascripter that's beside the point and probably also good to appear in your question. Your ML algorithm would learn relationships between samples. The suitability of the median MIGHT be questionable, depending on its effect on the sequence but NOT for the reason you mention. You might want to have a look at this for a similar example where the "future" doesn't matter. $\endgroup$
    – A_A
    Feb 16, 2018 at 11:26
  • $\begingroup$ The effect on the signal might be a problem and distort the model's prediction accuracy, that's what I tried to convey. But ARMA or ARIMA could indeed serve for outlier detection. I'll look deeper into that. $\endgroup$
    – ascripter
    Feb 16, 2018 at 13:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.