Noise Variance and Sampling in the Linear Regression Context

Question

I am new to signal processing, so please bear with me. My question applies to any general problem of estimation from noisy measurements but I will like to understand this through a problem given here.

Suppose a function $f(t)$ needs to be estimated, say $f(t)$ is a polynomial in time and the coefficients, $\boldsymbol{\theta}$, of this polynomial need to be estimated. There is $N$ seconds of measurements, $\mathbf{y}$, available but it suffers from white Gaussian noise with a variance of $\sigma^2$. A typical approach is to use a polynomial approximation of order $L$ and pose a least-squares problem, i.e. $$\arg \min_{\boldsymbol{\theta}} \: \vert \vert \mathbf{y} - \mathbf{V} \: \boldsymbol{\theta} \vert \vert^2$$ where $\mathbf{V}$ is a Vandermonde matrix and $\boldsymbol{\theta} = \begin{bmatrix} \theta_0 & \theta1 & \ldots \theta_{L-1} \end{bmatrix}^T$ with $\theta_i = \dfrac{\partial^{i} f(t)}{\partial t^{i}} \Big\vert_{t = 0}$ i.e. the $i^{th}$ derivative w.r.t. time.

Question: Let there be $K$ samples spanning these $N$ seconds of measurement. Generally $K$ determines how accurate the estimate $\widehat{\boldsymbol{\theta}}$ is. Generally, the more $K$ you have the better your estimate should be. But if you increase $K$ while keeping the data within the given $N$ seconds, despite having more samples to fit, you also have measurement $y_i$ with the same noise variance $\sigma^2$. This will eventually lead to a 'noisier' data and consequently, a worse fit. So, in my toy example, if I take more samples within $N$ seconds, i.e. more $K$, the estimate is worse than if I would have taken fewer samples because the least-squares is trying to fit the noise rather than the signal. So the questions are:

• How should I scale my noise as I increase $K$ i.e. number of samples within $N$ seconds to derive any benefit from the increased sampling?
• How does it work in an actual sensor? Say I am measuring distance to a point moving in space. My sensor should give me a lower variance on the measurements if I sample more (so the sampling points are closer in time) for me to have a better estimate of the path.

Please comment if anything is unclear! Feel free to be as thorough as you would like.

Answer 1

One note before going into this.
If we talk on discrete data, then having more samples within the same time interval means higher frequency sampling. Now, if you remember how to remember how is discrete noise is derived as samples from continuous white noise (See my answer to How to Simulate AWGN (Additive White Gaussian Noise) in Communication Systems for Specific Bandwidth and my answer to How to Generate Band Limited Gaussian White Noise in MATLAB) you'd see that if you keep the discrete noise variance constant it means you changed for different sampling frequencies it means you changed the variance of the continuous noise. Usually we have a fixed variance of the continuous noise and then we sample the discrete noise for different sampling frequencies.

Now, regarding your specific question, the best way to gain intuition is using a linear function:

$$ {y}_{i} = a {x}_{i} + b + {n}_{i}, i = 0, 1, \ldots, N - 1 $$

Let's say we have only 2 samples: $ N = 2 $.
Think, what would be better, to have them close to each other or as far as possible?
Well, since the answer is obvious you can derive from the answers to your questions.

Being more formal, the Covariance of a Least Squares estimator of the model:

$$ \boldsymbol{y} = X \boldsymbol{\beta} + \boldsymbol{n} $$

Is given by $ {\sigma}_{n} {\left( {X}^{T} X \right)}^{-1} $.

So for a real system from a sensor with fixed noise (From a continuous noise model) the value of $ {\sigma}_{n} $ will be lower the farther away the samples are.