LMS algorithm for modeling step-size ambiguity!

Behrouz Farhang-Boroujeny in his adoptive filters, 2nd ed., p. 155 ,told:

It is sufficient for stability: 1/(3*tr(R)).

But his book have attached mfile for modeling:

%   Modeling
% Last updated on April 28, 1998
%
itn=input('\n No. of iterations?      ');
sigman2=input('\n Variance of the plant noise?      ');
sigman=sqrt(sigman2);
wo=input('\n Plant impulse response (vector, w_o)?      ');
a=size(wo);
if a(1)<a(2)
wo=wo';
end

N=input('\n Length of the model (N)?      ');

h=input('\n Coloring filter impulse response (vector, h)?     ');
a=size(h);
if a(1)<a(2)
h=h';
end

Misad=input('\n Misadjustment (e.g., 0.1 for 10%) ?     ');

You can see:

And it is not matching to his formula in his book ... I can't understand where the hell it is coming from?

Scheme of problem:(Note the source of image is irrelevant to my question, it's only image) Core of algorithm: (Note, it is average over runs for smooth learning curve)

for k=1:runs
x=filter(h,1,randn(itn,1));% the randn creates input sequence that colorize by filtering through filter with h impulse response
d=filter(wo,1,x)+sigman*randn(itn,1); % Create desired by sum of noise and y(main plant output)
w=zeros(N,1);

for n=N:itn
xtdl=x(n:-1:n-N+1);
e=d(n)-w'*xtdl;
w=w+2*mu*e*xtdl;
xi(n)=xi(n)+e^2; % Creation of learning curve, gradient(e^2)=-2e(n)x(n)
end
end
xi=xi/runs;

Note: 1/(3*tr(R)) make algorithm unstable ...!!! But without colorizing filter this will converge.

• LMS adaptive filters do have a number of different variants each having a slightly different definition of the step-size parameter as well as tap weight update adjustment. So you they are probably from two different LMS filter definitions. Does it say which LMS type the filter is, if h is input to the filter, then the Matlab code normalizes the step size by dividing the requested misadjustment to the input power hence it could be a variant of NLMS type. – Fat32 Jun 29 '17 at 23:17
• @Fat32 This is not NLMS, this is general implementation of LMS by this schem: researchgate.net/profile/Guan_Gui/publication/264457158/figure/… also additional information added to question ... – mohammadsdtmnd Jun 30 '17 at 3:56
• i guess i would need to see the time-domain sample processing equations. like how the FIR coefficients are updated or adapted. i don't wanna decode your code, but the bottom looks like vanilla-flavored non-normalized LMS with 2*mu as the adaptation gain. i wonder what the xi(n) array is for? – robert bristow-johnson Jun 30 '17 at 5:53
• @robertbristow-johnson xi(n) is Learning curve, isn't that obvious? – mohammadsdtmnd Jun 30 '17 at 9:07
• Look! the link you provided shows a block diagram indicating LMS/F on the adaptive algorithm being considered. And [ieeexplore.ieee.org/document/585044/] clearly defines what LMS/F is and its stability proprties. So it's probably not standard LMS but a varaint called LMS/F. So you have internet, go read that paper. – Fat32 Jun 30 '17 at 10:10

I have the first edition of Behrouz Farhang-Boroujeny's Adaptive Filters book. I found it useful and it was definitely more practical in terms of implementing adaptive filters than other textbooks like those from Haykin and Sayed, primarily because of the included Matlab code. However, like any topic in the area of adaptive filtering, I would use it with a grain of salt.

There are numerous variations of the LMS algorithm, primarily because it is readily implemented and not computationally expensive compared to other adaptive algorithms such as the Recursive Least Squares (RLS) algorithm.

I think a lot of the LMS variations you can find (particularly in Farhang-Boroujeny's book) come about because people tweak one of the more traditional variations, ending up with something that performs well for their particular problem. These often make it into journals because they are different and are indeed novel implementations.

I would suggest you read about the standard LMS algorithm and the normalized version of this algorithm, the NLMS. Once you feel comfortable with your understanding of these algorithms, then implement them and apply them to your problem. Tuning the step-size can then be done by trial-and-error. If you are keen to understand the mathematical details of how the step-size is determined, I suggest the Haykin and Widrow textbook. It has in depth analysis of the convergence behavior of LMS-based algorithms.

Here is an example (my code) of the LMS algorithm in Matlab.

function [ prediction_error, weights ] = LMS_Algorithm( regressive_sequence, ...
step_size, number_of_taps )

% This script-file implements the Least Mean-squares (LMS) adaptive
% algorithm.
%
%
% LMS_Algorithm, NLMS_Algorithm, RLS_Algorithm, APA_Algorithm

% Author:  Michael R. Wirtzfeld
% Modification Date:  Monday, February 27, 2012
% Creation Date:  Wednesday, February 15, 2012

weights = zeros( number_of_taps, size(regressive_sequence, 2) );
prediction_error = zeros( size(regressive_sequence) );

for index = (number_of_taps + 1):1:numel( regressive_sequence )

tap_inputs = [ regressive_sequence( (index - 1):-1:(index - ...
number_of_taps) ) ]';

prediction = weights(:, index)' * tap_inputs;

prediction_error(index) = regressive_sequence(index) - prediction;

weights( :, (index+1) ) = weights( :, index ) + ...
step_size * tap_inputs * prediction_error(index);

end;  % End:  for index = 1:1:numel( regressive_sequence )

weights = weights( :, 1:1:numel( regressive_sequence ) );

%% References

% Section 9.5, Summary of the LMS Algorithm, Adaptive Filter Theory, Third
% Edition, S. Haykin, Prentice Hall, 1996

Here is an example of the NLMS algorithm in Matlab.

function [ prediction_error, weights ] = NLMS_Algorithm( regressive_sequence, ...
step_size, number_of_taps )

% This script-file implements the Normalized Least Mean-squares (NLMS)
%
%
% LMS_Algorithm, NLMS_Algorithm, RLS_Algorithm, APA_Algorithm

% Author:  Michael R. Wirtzfeld
% Modification Date:  Monday, February 27, 2012
% Creation Date:  Wednesday, February 15, 2012

a = 0.0001;

weights = zeros( number_of_taps, size(regressive_sequence, 2) );
prediction_error = zeros( size(regressive_sequence) );

for index = (number_of_taps + 1):1:numel( regressive_sequence )

tap_inputs = [ regressive_sequence( (index - 1):-1:(index - ...
number_of_taps) ) ]';

prediction = weights( :, index )' * tap_inputs;

prediction_error(index) = regressive_sequence (index) - prediction;

normalization_term = a + norm(tap_inputs)^2;

weights( :, (index+1) ) = weights( :, index ) + ...
(step_size / normalization_term) * tap_inputs * prediction_error(index);

end;  % End:  for index = 1:1:numel( regressive_sequence )

weights = weights( :, 1:1:numel( regressive_sequence ) );

%% References

% Table 9.2, Summary of the NLMS Algorithm, Adaptive Filter Theory, Third
% Edition, S. Haykin, Prentice Hall, 1996

These functions were used as part of an investigation to see how a standard tap-delay line implementation of the LMS and NLMS algorithms could model a second-order auto-regressive sequence.

As Fat32 said, most of the textbooks on the topic of adaptive filters use different notations and suggest different recommendations regarding how to set parameters, so it's no wonder that you feel uncertain about what to do.

I hope this helps.

• Thanks for sharing, but I didn't found anything about that Mu calculation even in Haykin, I know (h'*h) is tr[R], this is not trouble but N is really big trouble. :( – mohammadsdtmnd Jul 2 '17 at 15:11
• Furthermore I had saw this formula in this paper: "Transform Domain LMS Algorithm",6th equation ,but no reference or proofing. – mohammadsdtmnd Jul 2 '17 at 15:26
• So, what exactly are you trying to accomplish? It is not clear to me at this point. – Michael_RW Jul 3 '17 at 16:03
• Why N is appeared on that formula? – mohammadsdtmnd Jul 3 '17 at 17:16
• Your LMS algorithm is for equalizer configuration, not modeling ... – mohammadsdtmnd Jul 3 '17 at 19:27