0
$\begingroup$

I have data that is sampled every 5 minutes(or every 300 seconds)from an automated instrument that measures temperature outside. The data is available on a monthly basis i.e every 1 month . That is 8924 sample points. But due to a power failure during an external event only 8867 points are available. I am wanting to check my data for periodicity, waves and frequencies if possible. An expert recommended that I look at Lomb Scargle periodograms for doing this and not discrete FFT.

I am doing this in python's scipy [Lomb Scargle][1] and I am not clear on two things. What exactly are my x values ? I have data from a CSV file that basically is a bunch of dates or timestamps if you wish to call them that.

such as

  12/1/2021 12:04 AM 
  12/1/2021 12:09 AM
  12/1/2021 12:14 AM
  12/1/2021 12:19 AM 

and so on. But there is no guarantee that the dates are available in a uniform manner i.e sampled every 5 minutes. Occasionally you may have a break and the data would not be available for 5 hours or so.

So the sampling rate is every 300 seconds but what exactly is my x array that I need to feed to the Lomb Scargle periodogram ? I am clear on what y is(my temperature data) and I have an idea on what angular frequencies are (the third argument in that API).

Basically I am assuming the angular frequencies are

  2*pi/300.  

Whereas the Nyquist frequency is

  8924/2 

and so the third argument in that scipy API could be

   freq = np.linspace(0,2*pi/300,8924/2)

As a newbie to signal processing if all of this thinking is correct how do I go figuring out what my x values are ? [1]: https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.lombscargle.html

$\endgroup$

1 Answer 1

0
$\begingroup$

The function expects a numerical array of times as inputs. Since your data has minute resolution, you can extract this by converting the timestamps to int64 (which gives a timestamp with nanosecond resolution) and then dividing by the number of nanoseconds in a minute:

import pandas as pd
import numpy as np
import io

data = pd.read_csv(io.StringIO("""\
  12/1/2021 12:04 AM 
  12/1/2021 12:09 AM
  12/1/2021 12:14 AM
  12/1/2021 12:19 AM
"""), header=None)

times = pd.to_datetime(data[0])
print(times)
# 0   2021-12-01 00:04:00
# 1   2021-12-01 00:09:00
# 2   2021-12-01 00:14:00
# 3   2021-12-01 00:19:00

minutes = times.view(np.int64) / 6E10  # 60,000,000,000ns per minute
print(minutes)
# 0    27305284.0
# 1    27305289.0
# 2    27305294.0
# 3    27305299.0
```
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.