Wiener filter yielding negative MSE (and MMSE)

I implement a Wiener filter using the following code

function [h, yy, mse] = wiener_h(y,x,M)
yy=xcorr(y(1:M));
yy=toeplitz(yy((length(yy)+1)/2:((length(yy)+1)/2)+M-1));
debug = sprintf('Inverting Autocorrelation matrix..')
yy=inv(yy);
%%
debug = sprintf('Crosscorrelation vector')
yx = xcorr(y,x);
yx = yx((length(yx)+1)/2:((length(yx)+1)/2)+M-1); %consider only non-negative lags
debug = sprintf('Optimal Weiner impulse response..')
h =yy*yx;
mmse = var(x)-h'*yy*h;
mse = var(x)-2*yx'*h+h'*yy*h;


where y is the input signal, x the desired output, M number of coefficients, h the optimal (wiener values) and mse the mean square error. The input to the filter is are magnitude values of a pulse (NR, SV, SA) The desired output is TX The output of

[h, yy, mse] = wiener_h(y,x,15000)
plot(t,filter(h,1,y))


where the number of coefficients M is equal to the number of samples N, is mse = -1.14e+03 Whereas the output of

 [h, yy, mse] = wiener_h(y,x,1500)
plot(t,filter(h,1,y))


Where the number of coefficients is 10% of N, is mse = -7.68e+04 The plotted results of the two above (left M=N, right M=0.1N)

Can someone help me understand what's going on please?

• there is no jm in the code, as you mentioned its name? – Fat32 Mar 21 '17 at 13:32
• sorry, edited to correct couple of distractions – Marco Datola Mar 21 '17 at 14:04