4
$\begingroup$

I have sets of data of different deep sky objects. My job is to check for any periodicity.

I use IDL to run an FFT and wavelet methods to check for a signal. To test my code I ran the IDL built in FFT operation on a sine wave, my code works.

I then add noise to the sine wave and increase the noise until I cant distinguish the frequency peak in the FFT output.

My question is what mathematical method can I use to say 'yes there is a signal and this is its frequency' or 'no there is no signal' ? Can or should I get a percent of certainty ?

This post is very close to what I'm asking, but I only have one data set for an object so how could I make a ROC curve?: What statistic is used to determine presence of a signal in noise?

Also I think I dont want a ROC curve. I will respond to any question you may have about my question. Thank you very much for your help. I am currently do research at UC Santa Cruz as an undergraduate.

Improve this question
$\endgroup$
3
$\begingroup$

As a fellow astrophysicist, I suggest you use the standard 3 sigma above noise for detection, and 5 sigma above noise if you want to do analysis on what you find.

If your noise does not have a flat background, you need to subtract a "Flat", an estimation of that the background might be. If you know what the noise should look like, such as exponential, gaussian, etc., try subtracting such a curve. If not, try a first order polynomial. If not good enough, then go with a second order.

Beyond that, it becomes interpretation... What if that tendency you notice in the noise is actually a signal, spread over many frequencies?

Try computing the FFT of a pure white noise image, to get an idea of what your noise is supposed to look like. (Off course, remember that FFTs assumes your signal is periodic beyond its edges. Don't forget to zero-pad your data, and use a correct apodization, or limit your analysis to something smaller than half the size of your image.)

Improve this answer
$\endgroup$
5
  • $\begingroup$ When you say you need to 'subtract an estimation of the background noise' - can you please elucidate - exactly - how you mean this to be done? What series of steps are you referring to? $\endgroup$
    – TheGrapeBeyond
    Nov 27 '13 at 14:22
  • $\begingroup$ It will all depend on your particular situation. If you know that some region of your signal is supposed to be empty, you can use this region and extrapolate that "everywhere without signal should have the same statistics". If the noise can change in a non-linear way, your best guess would be to do a least square fit on different patches of noise, using polynomials of varying order (ie first power if you have linear noise). Off course, if you're dealing with a time spectrum, you will have to do this for every frequencies! $\endgroup$
    – PhilMacKay
    Dec 2 '13 at 17:34
  • $\begingroup$ Let us say that you measured your noise statistics very well, and that the environment is stationary. Where, exactly, would you now "subtract noise" from? What is being subtracted from what? Thanks. $\endgroup$
    – TheGrapeBeyond
    Dec 3 '13 at 17:00
  • $\begingroup$ Unless your noise is correlated with your signal, on average the data you collect will be the sum of your signal and your noise. The signal being what you want to measure, and the data being what your instrument tells you it measured. So you would subtrack the average value of your noise from your data. The result won't exactly be your signal, but it should at least have the same average value. (Since the variance is computed with respect to the average, you can now compute the variance of your data with respect to the average of the signal.) $\endgroup$
    – PhilMacKay
    Dec 4 '13 at 20:39
  • $\begingroup$ Thank you for pointing out where I need to add details to my answer! I'll try to update it when I'll find some time. $\endgroup$
    – PhilMacKay
    Dec 5 '13 at 21:27
1
$\begingroup$

Try to get a spectrogram with the overlap of pure white noise signal and your every column of your image, compare them in frequency domain. Remove the DC signal in your image before such operation.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.