Model Validation After Estimation for System Identification Task (Assistance with MATLAB Code)

Question

QUESTION: I want to determine how well the estimated model fits to the future new data. How do I validate the estimated model...what is the procedure? After system identification, how to do model validation with the channel estimates? Please help.

In this example, I am using Least Mean Squares for system identification. The system has coefficients / impulse response given by $\mathbf{h}$ which are estimated. The input signal to the system is $d$. The noisy output is $x$ where, the noise $e$ is Additive White Gaussian. I am trying to estimate the coefficients -- system identification using Least Mean Square and estimating the input $d$ as well. I am using an equalizer for this.

Thank you.

$x_n = r_n + e_n = \mathbf{h}^T \mathbf{d} + e_n$ where $\mathbf{h}$ represents the coefficients, $d$ is the input and $e_n$ is Additive White Gaussian Noise.

%% Channel and noise level
h = [0.9 0.3 -0.1]; % Channel
SNRr = 10;              % Noise Level

%% Input/Output data
N = 1000;               % Number of samples
Bits = 2;               % Number of bits for modulation (2-bit for Binary modulation)
data = randi([0 1],1,N);        % Random signal
d = real(pskmod(data,Bits));    % BPSK Modulated signal (desired/output)
r = filter(h,1,d);              % Signal after passing through channel
x = awgn(r, SNRr);              % Noisy Signal after channel (given/input)

%% LMS parameters
epoch = 10;        % Number of epochs (training repetation)
eta = 1e-3;         % Learning rate / step size
order=10;           % Order of the equalizer

U = zeros(1,order); % Input frame
W = zeros(1,order); % Initial Weigths



%% Algorithm
To get this we have to find error between input and output. As I understood `x` is an input for our model and `y` is an output. 

    %% Algorithm
    for k = 1 : epoch
        for n = 1 : N
            U(1,2:end) = U(1,1:end-1);  % Sliding window
            U(1,1) = x(n);              % Present Input

            y(n) = (W)*U';             % Calculating output of LMS
            e(n) = x(n) - y(n);           % Instantaneous error 
            W = W +  eta * e(n) * U ;  % Weight update rule of LMS
            J(k,n) = e(n) * (e(n))';        % Instantaneous square error
        end
    end

UPDATE : Please find below the picture that shows the step at which I am stuck. In the picture, the plus sign is the addition of Additive White Gaussian noise. I think the outcome of system identification of the unknown system is that the system should respond by producing the same output when given the same input. Or to predict. I don't want the MSE.

Answer 1

I agree with AlexTP's idea of simply plotting the instantaneous error between the model's output and the system's output over time. The exact implementation is a bit tricky though, because you are not only estimating the impulse response (using LMS) but also doing an "inverse" filter operation to estimate the unknown inputs. One way is to use a standard machine learning approach outlined in this answer.

1. Split your known input data $\mathbf d$ into two halves - training and testing (or validation$^*$).
2. Using the training half, run LMS and estimate the channel coefficients $\hat{\mathbf{h}}$.
3. Now use your test set and apply them to the noisy system to obtain outputs $x_n = \mathbf h^T \mathbf d + e_n$ and also apply them to the filter learned in step 2 to get $\hat x_n = \mathbf{\hat h}^T\mathbf d$. Compute the error$^\dagger$ between $\hat x_n$ and $x_n$ and make sure it's acceptable/small.$^\&$
4. If you are happy with the performance in step 3, your filter $\hat{\mathbf h}$ is ready for real world where you use the inverse of the filter $\hat{\mathbf h}$ to estimate the unknown inputs.$^{\ddagger}$

_{$^*$ Since you aren't doing model selection, you don't need a validation set, or in other words, your test and validation set are the same. If you do care about model selection (eg. choosing the best filter order) you can split your data into 3 pieces - training, testing and validation.}

_{$^\dagger$ I don't think there's a single best way of computing this error. You could simply plot $\hat x_n - x_n$ vs $n$ and make sure it stays close to zero. Or perhaps you could aggregate them as a mean absolute error or mean squared error by summing over all $n$.}

_{$^\&$ Alternatively, you can use the inverse filter of $\hat{\mathbf h}$ and estimate the inputs and compare them with the known inputs. Again, there's no single best way of showing error and would depend on your application.}

_{$^{\ddagger}$ Since this phase has unknown inputs you can't really compute a performance metric for your model. Or as this answer puts it "you can only speculate about the quality of your model output using the results of your validation phase".}