0
$\begingroup$

I have a series of time, they look like this:

[122.76258495, 125.2357452 , 127.70768702, 132.65322914,
   135.12374982, 137.58495967, 140.06083022, 142.5266016 ,
   145.00126936, 147.47713604, 149.94647284, 152.41441211,
   154.893105  , 157.35769229, 159.82660719, 162.28600819,
   172.17809076, 174.65187416, 177.11919058, 179.58977094,
   184.53904675, 187.00739281, 189.47979348, 191.9547493 ,
   194.41813837, 196.88205253, 199.36073519, 201.82316893,
   204.29872663, 206.77035981, 209.23571587, 211.70450343,
   214.19166579, 216.64951491, 219.12309888, 221.59243694,
   224.06177911, 226.53636961, 229.01307628, 233.95261524,
   236.42512815, 238.89132881, 241.35466117, 243.83515673,
   246.30092881, 248.77364934, 251.23992941, 253.70513364,
   261.12110344, 263.59184693, 266.06187123, 268.53406706,
   271.00650483, 273.47121761, 275.94262475, 278.41602937,
   283.34598689, 285.83241639, 288.29302028, 293.22749874,
   295.71148137, 298.17624614]

Apparently, they have a fixed interval aroud 2.47, if we calculate its differences:

[2.47316025, 2.47194182, 4.94554212, 2.47052068, 2.46120985,
   2.47587055, 2.46577138, 2.47466776, 2.47586668, 2.4693368 ,
   2.46793927, 2.47869289, 2.4645873 , 2.4689149 , 2.45940101,
   9.89208256, 2.4737834 , 2.46731642, 2.47058035, 4.94927581,
   2.46834606, 2.47240067, 2.47495582, 2.46338908, 2.46391416,
   2.47868266, 2.46243374, 2.4755577 , 2.47163318, 2.46535606,
   2.46878756, 2.48716236, 2.45784911, 2.47358397, 2.46933806,
   2.46934217, 2.4745905 , 2.47670668, 4.93953895, 2.47251292,
   2.46620066, 2.46333236, 2.48049556, 2.46577208, 2.47272053,
   2.46628007, 2.46520423, 7.4159698 , 2.47074349, 2.4700243 ,
   2.47219584, 2.47243776, 2.46471279, 2.47140714, 2.47340462,
   4.92995752, 2.4864295 , 2.46060389, 4.93447846, 2.48398263,
   2.46476477]

But it may have some gaps and noises, how to determine its interval period in a proper way?

$\endgroup$

3 Answers 3

1
$\begingroup$

I think you looking for linear regression.

Your data looks like a linear function in the form of $$ y = kx + b $$

where $x$ is the time index, $k$ is the slope (or regression coefficient) which is exactly what you want and $b$ is the $y$-intercept.

Now you have a set of linear equations

$$ y_1=kx_1+b\\ y_2=kx_2+b\\ \vdots\\ y_N=kx_N+b $$

These can be written in a matrix form as $$ \left[ \begin{matrix} x_1 & 1\\ x_2 & 1\\ \vdots & \vdots\\ x_N & 1 \end{matrix} \right] \left[ \begin{matrix} k\\ b \end{matrix} \right] = \left[ \begin{matrix} y_1\\ y_2\\ \vdots\\ y_N \end{matrix} \right] $$

This can be solved by the least squares method.

Here are some MATLAB code:

y = [122.76258495, 125.2357452, 127.70768702, 132.65322914, 135.12374982, 137.58495967, 140.06083022, 142.5266016 , 145.00126936, 147.47713604, 149.94647284, 152.41441211, 154.893105  , 157.35769229, 159.82660719, 162.28600819, 172.17809076, 174.65187416, 177.11919058, 179.58977094, 184.53904675, 187.00739281, 189.47979348, 191.9547493 , 194.41813837, 196.88205253, 199.36073519, 201.82316893, 204.29872663, 206.77035981, 209.23571587, 211.70450343, 214.19166579, 216.64951491, 219.12309888, 221.59243694, 224.06177911, 226.53636961, 229.01307628, 233.95261524, 236.42512815, 238.89132881, 241.35466117, 243.83515673, 246.30092881, 248.77364934, 251.23992941, 253.70513364, 261.12110344, 263.59184693, 266.06187123, 268.53406706, 271.00650483, 273.47121761, 275.94262475, 278.41602937, 283.34598689, 285.83241639, 288.29302028, 293.22749874, 295.71148137, 298.17624614].';
x = (1:length(y)).';
A = [x, ones(length(y), 1)];
kb = A\y;
yy = A*kb;
figure; plot(x, y, x, yy)

The result shows that $k=2.842576571327591$ and you can see it fits well in the following picture.

enter image description here

Edit:

To fix the outliers, first we calculate the differences and find the ones which are greater than $1.7k$ or some other proper value, where $k$ is the slope we calculated without considering the outliers. Remove that difference and add $k$ back and then do the linear regression again.

d = diff(y);
idx = find(d>1.7*kb(1));
for ii = 1:length(idx)
    y(idx(ii)+1:end) = y(idx(ii)+1:end) - d(idx(ii)) + kb(1);
end
kb = A\y;
yy = A*kb;
figure; plot(x, y, x, yy)

The two curves almost coincide. In this case $k=2.506241278260715$. There are still some outliers whose differences are around 2.84. If you want to remove their influence just lower the threshold.

enter image description here

$\endgroup$
6
  • $\begingroup$ Interesting idea, but it would take the higher order of periods like 2*2.47 and 3*2.47 as the outliers, I think that's why the results are slightly higher than the expected value. $\endgroup$
    – A.Bbom
    Commented Jun 17, 2021 at 7:46
  • $\begingroup$ I get what you mean. Then you should analysis and fix the outlier first, and then apply linear regression. A proper bound may be $1.6k$ or $1.7k$ $\endgroup$
    – ZR Han
    Commented Jun 17, 2021 at 8:18
  • $\begingroup$ I've edited the answer. $\endgroup$
    – ZR Han
    Commented Jun 17, 2021 at 8:30
  • $\begingroup$ Thank you, but the removement of the outliers is not a good idea because these high order of periods can be useful data points with a proper method. For this time series, the number of points are high enough to fit the liner function; but for other time series with only dozen points including some gaps, the fitting results would be misled. $\endgroup$
    – A.Bbom
    Commented Jun 17, 2021 at 8:41
  • $\begingroup$ Yes. But I'm wondering why do these outliers affect the following data. What kind of data did you collect and do you know why the abnormal differences are exactly around two or three times the expected value? $\endgroup$
    – ZR Han
    Commented Jun 17, 2021 at 8:46
1
$\begingroup$

Let's have a quick look at the difference between sample times.

enter image description here

It's quite obvious that these differences come in very discrete steps that are integer multiples of each other. That indicates a regular sample grid but with time gaps of an integer number of samples. That means samples are being "dropped".

To find the base period, you can simply discard all difference that are larger then, say, 1.2 times the minimum and take the mean. That comes out to be 2.470497.

Let's now normalize your time data to the base period and plot it again.

enter image description here

Now we clearly see the integer nature of the grid. Most time difference are very close to one but you also have a few at 2 and even one at 3 and 4 each. That means that you are missing samples at these points.

In order to use the data, you probably need to restore the dropped samples, i.e. you need to identify the drop locations and insert the appropriate number of samples to restore a uniformly sampled signal . This will require some sort of interpolation between the "good" samples.

%%
x = [122.76258495, 125.2357452 , 127.70768702, 132.65322914,...
   135.12374982, 137.58495967, 140.06083022, 142.5266016 ,...
   145.00126936, 147.47713604, 149.94647284, 152.41441211,...
   154.893105  , 157.35769229, 159.82660719, 162.28600819,...
   172.17809076, 174.65187416, 177.11919058, 179.58977094,...
   184.53904675, 187.00739281, 189.47979348, 191.9547493 ,...
   194.41813837, 196.88205253, 199.36073519, 201.82316893,...
   204.29872663, 206.77035981, 209.23571587, 211.70450343,...
   214.19166579, 216.64951491, 219.12309888, 221.59243694,...
   224.06177911, 226.53636961, 229.01307628, 233.95261524,...
   236.42512815, 238.89132881, 241.35466117, 243.83515673,...
   246.30092881, 248.77364934, 251.23992941, 253.70513364,...
   261.12110344, 263.59184693, 266.06187123, 268.53406706,...
   271.00650483, 273.47121761, 275.94262475, 278.41602937,...
   283.34598689, 285.83241639, 288.29302028, 293.22749874,...
   295.71148137, 298.17624614];

     
     %%  find the base period

 dx = diff(x);
 % do a "rough" normalization. Find everythinh that's within 20% of the
 % minimum
 dxMin = min(dx);
 dxBase = mean(dx(dx < 1.2*dxMin));
 
 fprintf('Time Period = %10.6f\n',dxBase);
 plot(dx./dxBase,'x','MarkerSize',10);
 ylabel('diff in samples');
 xlabel('Time index');
 title('Normalized time index difference');
 
$\endgroup$
2
  • $\begingroup$ Nice try but the determination of base period is not always simple like this example. This target has such high frequency signals but others would not. Only a few time points would seriously mislead the initial period determination. $\endgroup$
    – A.Bbom
    Commented Jun 18, 2021 at 1:57
  • 1
    $\begingroup$ I answered based on the data you posted . If your data isn't representative of your problem, than please edit the question and the relevant details. What are your requirements and what are the specific properties of your data and how how they vary $\endgroup$
    – Hilmar
    Commented Jun 18, 2021 at 23:23
0
$\begingroup$

Below, I just weight each time difference by itself divided by the median time difference, then take the mean of that.

Not sure how it works for additional, non-periodic data being inserted, but it seems OK for the data given.

Mean difference: 2.4716331799999978
Standard deviation: 2.60564323024001e-16

Python code below

import numpy as np
import matplotlib.pyplot as plt

time_differences = np.diff(data)
median_difference = np.median(time_differences)
weighting = np.divide(1,time_differences/median_difference)
corrected_differences = np.multiply(weighting, time_differences)
plt.plot(corrected_differences)
print('Mean difference: ' + str(np.mean(corrected_differences)))
print(' Standard deviation: ' + str(np.std(corrected_differences)))
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.