I am having little difficulty to understand that why most of the adaptive algorithms use error power or addition of error power as cost function/minimization criterion. I have read that minimization of error power criterion gives us best results from estimation mean point of view. I am not being able to grasp that either. So can someone please tell me in simple language what are the actual reasons to use above criterion? Why cant we simply use error as the minimization criterion?
In a very simple form: Gaussian-like noises are frequent, and derivatives of squared functions are easy to minimize.
In very simple words: error, directly, cannot be used as a criterion. You can minimize a signed function to $-\infty$, which is not really useful here. As least, you need a cost function with a lower bound.
Power (if by that you mean the squared error) is an increasing function of the error, and positive. So if you minimize power, you can expect to minimize, somehow, a measure of the error (more precisely its absolute value).
The first problem is that the absolute value is not differentiable around zero, and can be difficult to optimize (one may look for a zero in the derivative). And when can you very easily find a zero in a derivative? When it is linear, because since Laplace, Gauss, Seidel such systems can be solved efficiently (matrix inversion). What are the functions that yield linear derivatives? Squares, parabolas...
Basically, for a while, squared functions were the only ones one could solve easily. The standard mean is just the minimum of the sum of differences squared. Interestingly, in linear cases, unrelated disturbances can add with a Gaussian distributions. And Gaussian distributions are well-related to squared errors.
But you are interested, recent advances in optimization allow to optimize more easily other types of functions, including absolute value loss criteria or penalties.