I have a continuous-time system that I want to fit via least squares. I just send $N$ digital samples $x[n]$ through the system and receive (via analog signal chain, ADC etc) $N$ digital samples $y[n]$. My system can be described as
$$ \mathbf{y} = \mathbf{X} \mathbf{c} $$
where $\mathbf{X}$ is a matrix composed of $x[n]$. Hence I solve for the coefficients via:
$$ \hat{\bf c} = (\mathbf{X}^{H} \mathbf{X})^{-1} \mathbf{X}^{H} \mathbf{y} $$
which works wonderful and it's straight forward and easy. Note - $(\cdot)^H$ denotes the Hermitian transpose
Now $\mathbf{y}$ are heavily contaminated by noise so I need many samples. But on each receive call I can only receive a certain maximum number of samples. Even if I would receive an infinite amount - calculating the pseudo inverse becomes very expensive.
What is the easiest way and direct generalization of the above least squares set up so that I can iteratively receive blocks and refine the estimate after each block? The method should match a hypothetical, full least squares solution as closely as possible.
There are a couple of concepts which may be related but I am not sure how they relate to each other:
- I could first receive, say 1000 batches of repetitive $\mathbf{y}$ and average them and solve one least squares. Is this equivalent to solving a larger least squares system with all samples? Disadvantage: I need to send repetitive data.
- I am not looking for LMS or similar sampled-based methods. Ideally I would just use Least Squares ... if the number of measurements would not be as excessive
- What about RLS oder IRLS? To my understanding, they are also sample based (not block based) methods.
- I have heard about gradient descent + Newton type:
$$ \mathbf{c}_{n+1} = \mathbf{c}_{n} = \mu \left(\frac{\partial^2 J}{\partial \mathbf{c}^{*} \partial \mathbf{c}}\right)^{-1} \frac{\partial J}{\partial \mathbf{c}^{*}} $$
which, in my linear model would expand to:
$$ \mathbf{c}_{n+1} = \mathbf{c}_{n} = \mu (\mathbf{X}^{H}\mathbf{X})^{-1} \mathbf{X} \mathbf{e} $$
where $\mathbf{e}$ is the error vector between actual measurements and model with the current estimate $\mathbf{c}_n$. It requires that the unknowns are smaller than the number of samples in each iteration. This looks very close to what I am looking at ... but I am not sure if it is the right thing and how it relates to the other methods.