1
$\begingroup$

My problem should probably be built up from the beginning, so lets start there. I performed a certain experiment 25 times. Every time, the experiment consists of 5000 measurements, and each measurement returns either 1 ('yes') or 0 ('no'). I know that each time I measure 1, the probability that this measurement is correct is $F_1$, while if I measure 0 I know that it's correct with probability $F_0$.

Now, subsequently I want to compute the autocorrelation function of these measurements I use the following formula enter image description here

with a slightly different normalization, but in principle that should not matter, because what I want to do is fit this autocorrelation to an exponential decay and find the decay time. To do so, I thought I had two options: Either I fit the 25 datasets separately, and then find the mean of the decay time $t_1$ and maybe say something about its variance, or I can find the mean autocorrelation of the 25 datasets, and fit that one. I've decided that that is probably the best option, as the individual datasets can be quite noisy with strange fluctuations, while the mean looks much cleaner.

But now I'm stuck wondering what I should be doing to find the uncertainty in the fit. Should I have introduced some sort of uncertainty due to the imperfect measurements ($F_0$ and $F_1$), or should I have introduced some sort of standard error of the mean for the 'mean autocorrelation' data? That last one seems plausible, as of course there is also a standard deviation in the 25 datasets, but then I get a little confused as to how I should do that. The formula for the standard error of the mean seems to be $\frac{\sigma}{\sqrt{n}}$, but this is a bit unfair as in my 5000 measurements, the autocorrelation drops to 0 after around 50 measurements. For the fit I therefore also only use the first ~100 points of the autocorrelation, as there's no real need to fit the next 4000 points that are all pretty much equal to 0. So should I instead only calculate the mean and standard error of the first 100 points? Would that be a fair way of finding the uncertainty? The subsequent fitting method will already give error bars for the $t_1$, but this of course heavily depends on the uncertainty in the data.

A final sort of sidetrail, what kind of quantitative method of finding how good I fit should I use for exponential decay? Reduced Chi square, Adjusted R squared, or something else entirely? I suppose this is not intended for dsp though, more for a statistics forum, so feel free to ignore it.

|improve this question
$\endgroup$
  • $\begingroup$ Regarding your final question, look up "coefficient of determination". It is a way of quantifying goodness of fit. $\endgroup$ – John Apr 17 '14 at 17:17
2
+50
$\begingroup$

As far as I can see the main question is: How to account for the uncertainty in the measurements of $X_t$ (i.e. $F$).

One approach to do this would be to run some sort of Monte Carlo simulation on the data. I.e. for each experiment you can resample a large number of times (say 1000 or 10000) where each point has $1-F$ probability of being flipped. By calculating the auto-correlation of each resample you get a distribution of the correlation for each $k$. Then take the 95% (or another value) confidence interval to get an uncertainty in $R(k)$.

It may also be possible to do this analytically but I'm not sure for binary data (my instinct says not or at least not simply).

This uncertainty in $R$ should be accounted for in your fitting of the data when calculating the decay time. This can definitely be done analytically and may be implemented in whatever you are using to do the fit (I think matlab does). Alternatively a similar Monte Carlo simulation could be done but is probably more effort.

If you think all the data follows the same trend (not unreasonable if it is a repeated experiment) you could calculate a mean for each auto-correlation distance as you suggest. This is effectively taking a manual Monte Carlo with 25 resamples. Then you can calculate the standard error as you suggest (note: $n=25$, not $5000$). I would shy away from this approach as it ignores the uncertainty in you initial measurement which you presumably have a good reason for giving.

|improve this answer
$\endgroup$
  • $\begingroup$ Hm, thank you for your answer, it's very useful. The question indeed boils down to how the uncertainty in the measurements propagates all the way down to the decay time of the correlation functions. I'm not sure I completely understand the Monte Carlo part, it sounds a little like a bootstrap method I guess. Do you perhaps have a reference that describes how one would go about this? What you describe later, with the 25 samples, also makes a lot of sense. The 25 data sets are indeed repeated experiments, so I think this might be a valid approach. $\endgroup$ – user129412 Apr 22 '14 at 15:39
  • $\begingroup$ Yes, bootstrap and monte carlo are very similar. As far as I understand the difference is that bootstrap resamples the empirical data (e.g. the X values at a point across the 25 tests) whereas Monte Carlo resamples based on the probability distribution. $\endgroup$ – nivag Apr 23 '14 at 9:21
  • $\begingroup$ For references for anything uncertainty related I tend to look at the GUM. It is primarily about continuous data but some things are applicable to binary info. In particular Supplement 1 is all about Monte Carlo methods. Although it's not the most practical guide. Here is a more practical guide. $\endgroup$ – nivag Apr 23 '14 at 9:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.