0
$\begingroup$

Using the data in the upper graph, I get the weird autocorrelogram of the lower graph:-

corr

I've never seen a autocorrelogram of this shape. They're usually flat along the x axis. Testing the code with simple sine waves produces the traditional flat bottomed graph. What causes the star shape, and is it valid or have I just done the processing badly? It seems specific to my data.

Python /numpy /scipy code: 

correlation = signal.convolve(signalData, signalData[::-1], mode='full')
|improve this question
$\endgroup$
  • $\begingroup$ It's the constant offset of the data causing the problem. If you do an autocorrelation of a constant value, then you get the triangle. The peak is from the varying content of the rest of the signal. I don't have the math to explain it correctly, just a lot of time analysing audio (and other signals) with autocorrelation and other tools. Someone with the maths will have to explain why it works that way $\endgroup$ – JRE Nov 23 '18 at 12:13
  • $\begingroup$ @JRE My maths is like my Cantonese; non existent, so thanks anyway. Can I get it looking more normal? Is there some process? $\endgroup$ – Paul Uszak Nov 23 '18 at 17:24
2
$\begingroup$

I'll go ahead and post a practical answer for you, and we can both see if someone can post a theoretical explanation.

The shape you see comes from the offset in your data. If you were measuring voltage, I'd say you have an AC signal on a DC offset.

I don't know what you are measuring, but it spends most of its time at around 100. It has short intervals where it changes drastically, and makes small changes almost constantly.

You have two ways to get rid of the triangle.

  1. Take the average of all data points and subtract that average from all the data points.

  2. Use a high pass filter to remove the offset.

If this were an audio signal, you could pick a lower cutoff frequency and be happy - there's some point where the audio frequencies are too low to be interesting.

Since you don't mention the source of the data (could be the sum of loans and borrows of elephants in an elephant lending library for all I know,) I'd go with the first option.

If it is some physical quantity, then you may want to consider the high pass filter, after all.

If the process generating the signal has some natural low frequency, then you could use that as your cutoff.

Or, you set the cutoff by the longest time period you want to consider.

If your data set represents 1 second, then autocorrelation for more than 1second is pointless - you could set the cutoff for one hertz. At any rate, you would use 1/(time period) as the cutoff frequency.

If you have all the data in blocks at hand, you can use filtfilt to do your filtering. That preserves the phase and the absolute timing of your signal.

|improve this answer
$\endgroup$
  • $\begingroup$ I'd love to be studying elephants again, but unfortunately it's noise on a Zener diode inside a random number generator. I'm checking for sample correlation in order to measure the entropy rate. So it has to be option 1 as you suggest. Ta - I appreciate the practical answer. There's too much theory in this world and not enough practise. $\endgroup$ – Paul Uszak Nov 23 '18 at 22:56
  • $\begingroup$ The waveform looks odd because it gets mangled as I'm only looking at the lower 8 bits of 10 bit samples. $\endgroup$ – Paul Uszak Nov 23 '18 at 22:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.