Why use parametric based estimation methods - confusion regarding terms

Question

Using the probability density function (pdf) we can estimate an unknown parameter using methods such as Maximum Likelihood estimation. If the pdf is not available, then Least Squares can be used. Other methods for pdf unavailable situations are Least Mean Squares.

Then there are other methods which are called as heuristic based approaches such as particle swarm optimization, genetic algorithm, ant colony optimization. These are also applied to find an unknown parameter without using the pdf. This is quite a long post, so a big thank you and I really appreciate the patience in taking the time out to read and understand my concerns. My questions are:

1) What is the difference between parameter estimation and optimization?

2) Is there a rule of thumb when to use estimation theory and optimization?

Many research articles and text books use these two terms interchangeably based on my understanding. For example, the abstract in the paper titled, "Parameter estimation with bio-inspired meta-heuristic optimization: modeling the dynamics of endocytosis" which is given below:

We address the task of parameter estimation in models of the dynamics of biological systems based on ordinary differential equations (ODEs) from measured data, where the models are typically non-linear and have many parameters, the measurements are imperfect due to noise, and the studied system can often be only partially observed. A representative task is to estimate the parameters in a model of the dynamics of endocytosis, i.e., endosome maturation, reflected in a cut-out switch transition between the Rab5 and Rab7 domain protein concentrations, from experimental measurements of these concentrations. The general parameter estimation task and the specific instance considered here are challenging optimization problems, calling for the use of advanced meta-heuristic optimization methods, such as evolutionary or swarm-based methods.

However, in signal processing, optimization is rarely combined or taught in conjunction with parameter estimation. For example, in the book by Steven Kay titled, "Fundamentals of Statistical Signal Processing, Volume I: Estimation Theory" never mentions optimization as another method to do estimation. However, Gradient Descent is an optimization algorithm which uses the derivative of the function and equates to zero. If this is the usual way of doing optimization, then Maximum Likelihood estimation also find the derivatives with respect to the unknown parameter using the log-likelihood function, which are equated to zero to obtain the estimates. This brings to my last 2 questions which are:

3) Is MLE an optimization or estimation technique?

4) Heuristic approaches such as particle swarm etc do not find derivatives to obtain the minima or maxima of the objective function. Then why are they called optimization methods if they are not optimizing the cost function in the usual way which is taking the derivatives and equating them to zero?

Answer 1

Hi: I'll try to answer as briefly as possible and only with respect to statistics. not dsp.

In statistics, if you have a nice pdf such as the normal distribution, then maximizing the likelihood is equivalent to minimizing the sum of squares of the residuals ( often called errors ).

In other cases, where you either have a complicated distribution ( maybe the model is non-linear and results in a more complex likelihood), then maximizing the likelihood is not equivalent to minimizing the residual sums of squares. So, you may still choose to maximize the likelihood but because the likelihood is complex, one often needs to turn to a numerical optimization algorithms such as gradient descent, BFGS.

So, I think your confusion is due to the fact that it ( maybe unfortunately ) happens to be the case that maximizing the likelihood of the normal is equivalent to minimizing the residual sums of squares.

Then there are other cases maximizing the likelihood is too difficult or the likelihood is not even derivable in closed form. In this case, people define other objective functions ( heuristics ) that still measure how "well" the model fits the data. Doing that, allows one to use the genetic, particle swarm type optimization schemes that you refer to. These schemes are often called non-parameteric because these objective functions are not tied to a parametric likelihood ( such as the normal,t, gamma etc ). In fact, the likelihood is often not even known.

Basically, what type of optimization one uses depends on whether you have a pdf and how complicated it is.

A reallly rough categorization is as follows:

1) normal case: : straightforward: maximize it or minimize residuals squared.

2) non-normal case: likelihood is complex, maybe non-linear but known : bfgs or gradient descent or some other numerical method that can handle the closed form version of the likelihood.

3) likelihood too complex or unknown or doesn't exist: define one's own objective function that measures the quality of the model and use algorithms such as particle swarm etc.

Note that in 2) above, you have to worry about whether the optimum you hit is local, global etc and this can be extremely tricky. This may be the case in 3) also. I'm not as familar with genetic algs as the numerical methods.

I hope this helps some. I'm not sure of a good reference for what you asked about. Kenneth Lange has a book "numerical methods in statistics' that may be relevant but I don't have it. Good luck.

Answer 2

Maximum Likelihood was developed by the celebrated Statistician, Ronald Fisher, and yes there is intrinsically an optimization involved, you want to minimize your error after all and since your data is random, to minimize the expected error. In the Gaussian (Normal) case, you can choose more than one error criteria and end up with the same estimator. In a book, like Kay’s you typically can do the optimization analytically. In Van Trees, Detection, Estimation, and Modulation Theory, Volume One, First edition, Maximum Likelihood and MAP are derived from different choices of error criteria. Optimization is almost always analytic, so the basic estimation principles don’t require line searches and Hessians to understand. In fact, the stochastic nature of a numerical optimization requires fairly involved mathematics to show convergence well above deterministic optimization, and the Estimation texts like Kay and Van Trees are already a few thousand pages long. Some DSP estimation books will cover some optimization topics specific to complex variables like the complex gradient.

The EM algorithm (which is actually a family of algorithms) is an indirect optimization technique unlike BFGS (or DFP).

It should be mentioned that maximum likelihood also includes mixed combinatorial and continuous parameters. A relatively recent idea called random set theory is gaining here.

There are some differences between Statistics and Signal Processing, but there are differences within Statistics as well. In Maximum Likelihood one assumes the parameter of interest to be an unknown deterministic quantity. In Signal Processing there is a mix of deterministic and random parameter assumptions. A DSPer usually has no problem thinking that a deterministic parameter is a special case of a random parameter with a delta function probability density and a “diffuse prior”.

The general problem of global verses local optimization is a real one, particularly in mixed combinatorial and continuous parameter estimation problems. Most DSP applications tend to be real time with latency requirements so in many cases, a local optimization is good enough or depends a lot on heuristics.