I am new to signal processing, so please bear with me. My question applies to any general problem of estimation from noisy measurements but I will like to understand this through a problem given here.
Suppose a function $f(t)$ needs to be estimated, say $f(t)$ is a polynomial in time and the coefficients, $\boldsymbol{\theta}$, of this polynomial need to be estimated. There is $N$ seconds of measurements, $\mathbf{y}$, available but it suffers from white Gaussian noise with a variance of $\sigma^2$. A typical approach is to use a polynomial approximation of order $L$ and pose a least-squares problem, i.e. $$\arg \min_{\boldsymbol{\theta}} \: \vert \vert \mathbf{y} - \mathbf{V} \: \boldsymbol{\theta} \vert \vert^2$$ where $\mathbf{V}$ is a Vandermonde matrix and $\boldsymbol{\theta} = \begin{bmatrix} \theta_0 & \theta1 & \ldots \theta_{L-1} \end{bmatrix}^T$ with $\theta_i = \dfrac{\partial^{i} f(t)}{\partial t^{i}} \Big\vert_{t = 0}$ i.e. the $i^{th}$ derivative w.r.t. time.
Question: Let there be $K$ samples spanning these $N$ seconds of measurement. Generally $K$ determines how accurate the estimate $\widehat{\boldsymbol{\theta}}$ is. Generally, the more $K$ you have the better your estimate should be. But if you increase $K$ while keeping the data within the given $N$ seconds, despite having more samples to fit, you also have measurement $y_i$ with the same noise variance $\sigma^2$. This will eventually lead to a 'noisier' data and consequently, a worse fit. So, in my toy example, if I take more samples within $N$ seconds, i.e. more $K$, the estimate is worse than if I would have taken fewer samples because the least-squares is trying to fit the noise rather than the signal. So the questions are:
- How should I scale my noise as I increase $K$ i.e. number of samples within $N$ seconds to derive any benefit from the increased sampling?
- How does it work in an actual sensor? Say I am measuring distance to a point moving in space. My sensor should give me a lower variance on the measurements if I sample more (so the sampling points are closer in time) for me to have a better estimate of the path.
Please comment if anything is unclear! Feel free to be as thorough as you would like.