# Model validation after estimation — system identification — help with an example in Matlab

QUESTION: I want to determine how well the estimated model fits to the future new data. How do I validate the estimated model...what is the procedure? After system identification, how to do model validation with the channel estimates? Please help.

In this example, I am using Least Mean Squares for system identification. The system has coefficients / impulse response given by $\mathbf{h}$ which are estimated. The input signal to the system is $d$. The noisy output is $x$ where, the noise $e$ is Additive White Gaussian. I am trying to estimate the coefficients -- system identification using Least Mean Square and estimating the input $d$ as well. I am using an equalizer for this.

Thank you.

$x_n = r_n + e_n = \mathbf{h}^T \mathbf{d} + e_n$ where $\mathbf{h}$ represents the coefficients, $d$ is the input and $e_n$ is Additive White Gaussian Noise.

%% Channel and noise level
h = [0.9 0.3 -0.1]; % Channel
SNRr = 10;              % Noise Level

%% Input/Output data
N = 1000;               % Number of samples
Bits = 2;               % Number of bits for modulation (2-bit for Binary modulation)
data = randi([0 1],1,N);        % Random signal
d = real(pskmod(data,Bits));    % BPSK Modulated signal (desired/output)
r = filter(h,1,d);              % Signal after passing through channel
x = awgn(r, SNRr);              % Noisy Signal after channel (given/input)

%% LMS parameters
epoch = 10;        % Number of epochs (training repetation)
eta = 1e-3;         % Learning rate / step size
order=10;           % Order of the equalizer

U = zeros(1,order); % Input frame
W = zeros(1,order); % Initial Weigths

%% Algorithm
To get this we have to find error between input and output. As I understood x is an input for our model and y is an output.

%% Algorithm
for k = 1 : epoch
for n = 1 : N
U(1,2:end) = U(1,1:end-1);  % Sliding window
U(1,1) = x(n);              % Present Input

y(n) = (W)*U';             % Calculating output of LMS
e(n) = x(n) - y(n);           % Instantaneous error
W = W +  eta * e(n) * U ;  % Weight update rule of LMS
J(k,n) = e(n) * (e(n))';        % Instantaneous square error
end
end


UPDATE : Please find below the picture that shows the step at which I am stuck. In the picture, the plus sign is the addition of Additive White Gaussian noise. I think the outcome of system identification of the unknown system is that the system should respond by producing the same output when given the same input. Or to predict. I don't want the MSE. • plot the mean square error alongwith the evolution of input data ? – AlexTP Oct 21 '17 at 6:45
• @AlexTP: Please see the update. I have put up the picture based on my understanding of system identification. Please correct me if this is wrong. I don't want the MSE plot. In many articles I have seen that using testing data, the estimated coefficients are applied and the graph of the plot of predicted output using this testing data and the graph of the plot of known output is plotted. This is known as validating model -- Comparing simulated or predicted model output to measured output. – Ria George Oct 21 '17 at 19:12
• But I am confused how to do this as I have noisy measured output---comparing noisy measured output with simulated data on testing set does not make sense to me. – Ria George Oct 21 '17 at 19:14
• What is input to the estimated model? – learner Oct 24 '17 at 12:53
• @learner: The input to the estimated model is the noisy signal $x$ given in the lines U(1,1) = x(n); ;y(n) = (W)*U';  – Ria George Oct 25 '17 at 0:31

I agree with AlexTP's idea of simply plotting the instantaneous error between the model's output and the system's output over time. The exact implementation is a bit tricky though, because you are not only estimating the impulse response (using LMS) but also doing an "inverse" filter operation to estimate the unknown inputs. One way is to use a standard machine learning approach outlined in this answer.

1. Split your known input data $\mathbf d$ into two halves - training and testing (or validation$^*$).
2. Using the training half, run LMS and estimate the channel coefficients $\hat{\mathbf{h}}$.
3. Now use your test set and apply them to the noisy system to obtain outputs $x_n = \mathbf h^T \mathbf d + e_n$ and also apply them to the filter learned in step 2 to get $\hat x_n = \mathbf{\hat h}^T\mathbf d$. Compute the error$^\dagger$ between $\hat x_n$ and $x_n$ and make sure it's acceptable/small.$^\&$
4. If you are happy with the performance in step 3, your filter $\hat{\mathbf h}$ is ready for real world where you use the inverse of the filter $\hat{\mathbf h}$ to estimate the unknown inputs.$^{\ddagger}$

$^*$ Since you aren't doing model selection, you don't need a validation set, or in other words, your test and validation set are the same. If you do care about model selection (eg. choosing the best filter order) you can split your data into 3 pieces - training, testing and validation.

$^\dagger$ I don't think there's a single best way of computing this error. You could simply plot $\hat x_n - x_n$ vs $n$ and make sure it stays close to zero. Or perhaps you could aggregate them as a mean absolute error or mean squared error by summing over all $n$.

$^\&$ Alternatively, you can use the inverse filter of $\hat{\mathbf h}$ and estimate the inputs and compare them with the known inputs. Again, there's no single best way of showing error and would depend on your application.

$^{\ddagger}$ Since this phase has unknown inputs you can't really compute a performance metric for your model. Or as this answer puts it "you can only speculate about the quality of your model output using the results of your validation phase".

• Thank you so so much for a clear cut answer. I do have few concerns in this approach, could you please clarify? (1) As per Alex's idea is to plot the error between the model's output and the system's output over time -- the model's output is clean and the system's output is noisy. So, would the plot be between noisy x and $\hat{x}$? (2) For each run of the program, I will get a new set of estimates and a new $\hat{x}, \hat{h}$. In machine learning, the accuracy and standard deviation are often reported. – Ria George Oct 29 '17 at 5:35
• Say for 10 independent runs for a particular SNR, I will get 10 different estimates for which I will calculate mean square error between $\hat{x}_n$ and noisy x, $x_n$. All results are different, what do I do with these different results?Is there a way to use variance or standard deviation of the error or such difference in results is a common thing and not applied in any statistics?(3) Is your solution applicable to blind system identification and blind channel equalization? – Ria George Oct 29 '17 at 5:37
• (1) Yes, because you don't have access to $\mathbf h^T \mathbf d$, the best you can do is plot $\hat x_n - x_n$ v/s $n$, or just compute a single number by summing over all $n$. (2) If you have independent runs with the same inputs you can still plot $\hat x_n - x_n$ vs $n$ and additionally show a confidence band around it. (3) I don't think this is blind because I used known inputs to estimate the channel. – Atul Ingle Oct 29 '17 at 14:26
• Thank you once again. Sorry the bounty period had expired before I could accept the answer. If you want I can restart the bounty and then award it to your answer. Please let me know. Just a concluding remark, in point (1) why should the error between the plot of noisy observation and estimated observation be less? Isn't one of the purpose of estimation is to reduce noise as well? Shouldn't like things be plotted/ compared?$\hat{x}$ does not have any $e$ component added to it whereas available $x$ has. I don't get this point clearly. Can you please explain a bit more? – Ria George Oct 29 '17 at 17:45
• Re:Bounty. Your choice. I'm ambivalent. Re:(1). Ideally you want to plot the noiseless $\mathbf h^T\mathbf d$ v/s your estimates. But in this case you don't have access to these noiseless values. So the best you can do is plot $x_n - \hat x_n$ vs $n$. This plot is still a useful performance metric. For instance if your noise $e_n$ is iid zero mean you would want this plot to wiggle around zero. Alternatively you can do what I said in the 3rd footnote$^\&$ and plot the error between $\mathbf d$ and $\hat{\mathbf d}$. – Atul Ingle Oct 29 '17 at 20:15