Using the probability density function (pdf) we can estimate an unknown parameter using methods such as Maximum Likelihood estimation. If the pdf is not available, then Least Squares can be used. Other methods for pdf unavailable situations are Least Mean Squares.
Then there are other methods which are called as heuristic based approaches such as particle swarm optimization, genetic algorithm, ant colony optimization. These are also applied to find an unknown parameter without using the pdf. This is quite a long post, so a big thank you and I really appreciate the patience in taking the time out to read and understand my concerns. My questions are:
1) What is the difference between parameter estimation and optimization?
2) Is there a rule of thumb when to use estimation theory and optimization?
Many research articles and text books use these two terms interchangeably based on my understanding. For example, the abstract in the paper titled, "Parameter estimation with bio-inspired meta-heuristic optimization: modeling the dynamics of endocytosis" which is given below:
We address the task of parameter estimation in models of the dynamics of biological systems based on ordinary differential equations (ODEs) from measured data, where the models are typically non-linear and have many parameters, the measurements are imperfect due to noise, and the studied system can often be only partially observed. A representative task is to estimate the parameters in a model of the dynamics of endocytosis, i.e., endosome maturation, reflected in a cut-out switch transition between the Rab5 and Rab7 domain protein concentrations, from experimental measurements of these concentrations. The general parameter estimation task and the specific instance considered here are challenging optimization problems, calling for the use of advanced meta-heuristic optimization methods, such as evolutionary or swarm-based methods.
However, in signal processing, optimization is rarely combined or taught in conjunction with parameter estimation. For example, in the book by Steven Kay titled, "Fundamentals of Statistical Signal Processing, Volume I: Estimation Theory" never mentions optimization as another method to do estimation. However, Gradient Descent is an optimization algorithm which uses the derivative of the function and equates to zero. If this is the usual way of doing optimization, then Maximum Likelihood estimation also find the derivatives with respect to the unknown parameter using the log-likelihood function, which are equated to zero to obtain the estimates. This brings to my last 2 questions which are:
3) Is MLE an optimization or estimation technique?
4) Heuristic approaches such as particle swarm etc do not find derivatives to obtain the minima or maxima of the objective function. Then why are they called optimization methods if they are not optimizing the cost function in the usual way which is taking the derivatives and equating them to zero?