First, I just want to start off by saying that I am a programmer, and am no expert in complex algorithms, and will probably not be able to apply complex pseudo-code or other descriptions of a solution. I have already asked a similar question on one of the programming SE forums in hope of finding a code sample, but no one there had a solution.

I have a stream of data that I get at around 10 snapshots per second. I wrote a (C#) controller that takes this data and adjusts some data-gathering parameters based on the distance away from the expected result. Currently, I just do a linear scale operation on my data so that the farther away I am from the expected result the more it corrects. The problem is that the incoming data stream is delayed by somewhere between 0.5 and 2 seconds (I can calculate that at runtime). Because of this delay, it is correcting for results from a while ago and is constantly overcorrecting and even sometimes correcting in the wrong direction. I am looking for an algorithm to do the following:

  • Implement a correction algorithm (I'd prefer PID) that will attempt
    to hone-in on the optimal value
  • Predict a certain amount of time (or number of datasets) in advance based on the history of corrections and results; the values won't be exactly linear, but they will probably be close

What options do I have in terms of algorithms that can accomplish this? As I said earlier, I am a programmer and therefore code samples would be appreciated; I am not great at converting complex pseudo-code in to a working implementation.

EDIT: To address some questions in the comments...

  • I did make a small abstraction in my question. What's actually happening is I get a stream of data, calculate 3 (separate) numbers based on that data, and try to optimize those separately; each of them having their own output parameters. What I need from this algorithm is to be able to operate on a single number and optimize that. I would give it the history and a result and it would modify it's own output parameter. I can then just use the algorithm 3 times for each parameter.
  • The rate does change slightly (usually at a slow pace), though I should be able to calculate the change if needed. But I don't need exact enough numbers to require variable prediction times; I can just pick an arbitrary number of snapshots to predict in advance and it should get me close enough to my result.
  • It is a physical process and is noisy and generally not very accurate. I am taking my data stream and doing a bunch of estimations and tracking and things like that, so there are often outliers. There are many factors that make a basic model relatively easy, but anything more accurate would be completely off. Basically, I can't really model it to the extent of being useful. Just a note, the output values that this algorithm has to control are the 'drift', or rate of change. Currently I just scale the values so that the farther away the input is from the ideal the higher the output is (pretty simple).
  • Are you asking for me to provide a sample of my data? I should be able to at some point, but it may take a few days for me to get the chance. To test it, I was (and am still) planning to just put the algorithm in a test program with a slider and a noise generator and see what happens.
  • $\begingroup$ Interesting question! I have a few to fill in: 1) Is the data a single measurement? Or something else (vector, classification, etc.)? 2) Does the delay change per sample? Or does it change slowly? At all? 3) Is a physical process generating the data? Can you model it? 4) Is it possible to get a plot of what your data (and your predictions) look like? I believe something like a Kalman filter (or predictor) might be appropriate... but I'll need more information to go on. $\endgroup$
    – Peter K.
    Nov 28 '13 at 1:50
  • $\begingroup$ @PeterK., I just added those clarifications; sorry for the length! $\endgroup$
    – Wasabi Fan
    Nov 28 '13 at 2:32
  • $\begingroup$ Thanks; I'll take a stab at it over the next few days. Might need some more clarifications, but I'll start an answer to go into more detail. $\endgroup$
    – Peter K.
    Nov 28 '13 at 4:08
  • $\begingroup$ @PeterK. Any luck? I've taken another stab at it myself, but haven't come up with much. $\endgroup$
    – Wasabi Fan
    Dec 1 '13 at 4:02

This is just a start, because I am not sure I fully understand the problem yet.

Here is a plot of what I think is happening. The blue curve is the "truth" of your quantity and the red curve is a delayed, noisy, measured version of it.

Does this look like your problem? If not, how should it be changed to make it look more like it?

enter image description here

I suspect that a modified Kalman filter (a Kalman predictor) will be able to give you something, but I want to understand what your data looks like first.

Another question: How do you know that there's a delay between the "truth" and your measurement? I'm trying to get a handle on what numbers we have to play with to correct for the time delay.

Scilab code to generate the figure.

// Q 12924
N = 1000;
process_noise = rand(1,N,'norm');
mid = filter(ones(1,20)/20,1,process_noise);
alpha = 0.99;
truth = filter(1-alpha,[1 -alpha], mid);


max_delay = 50;
true_delay = floor(rand(1,1,'uniform')*max_delay);

sigma = 0.01;
measurement_noise = rand(1,N,'norm')*sigma;

measurements = truth(max(1,[1:N]-true_delay)) + measurement_noise;

  • $\begingroup$ Yup! That's pretty much what my data looks like. What I need is something that will attempt to predict, remove noise and try to optimize the result. Just imagine that there are two points on a scale, and I know the difference between them. It has to control the relative speed at which one point is moving to attempt to get as close as possible to the other. $\endgroup$
    – Wasabi Fan
    Dec 6 '13 at 2:53
  • $\begingroup$ I know that they are delayed because I can interrupt the data gathering (manually) and see the time difference in my app. $\endgroup$
    – Wasabi Fan
    Dec 6 '13 at 2:54
  • $\begingroup$ OK. Will make an attempt at it later today. Thanks for the feedback! $\endgroup$
    – Peter K.
    Dec 6 '13 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.