# How to Smooth Gradient Estimates for Steepest Descent Optimization

In steepest descent methods of minimizing a function $f(x), x \in \mathbb{R}^d$, it's common to approximate the gradient by finite differences: $\qquad\qquad \nabla f(x) \approx gradest( x; h ) \equiv { {f( x + h I ) \ - \ f(x)} \over h }$
then minimize along those directions:
$\qquad\qquad x_{k+1} \ \text{from min} \ f( x_k + \lambda \ gradest( x_k ) )$

Can a signal-processing point of view suggest ways of smoothing $gradest()$ in this context ?
For example, one could view the "zig-zags" in the picture above as high frequencies in $gradest()$, to be smoothed out over the last few ${x_k}$. But that may be naive: can one model, then smooth, zig-zags through non-uniformaly spaced points in 5d or 10d ?

• Add a fraction of the gradient approximation from the previous iteration, and look up Nesterov smoothing. – Emre Nov 7 '14 at 8:31
• @Emre, do you know of any connection between Nesterov (smoothing | momentum) and filter theory ? – denis Nov 10 '14 at 16:45
• No, why? As it stands, the question concerns optimization theory. – Emre Nov 10 '14 at 18:13
• @Emre, afaik (correct me), Nesterov is used to accelerate big gradient descent / SGD problems, and is very sensitive to learning rates. Has anyone used it on e.g. Rosenbrock ? (I've tried -- terrible.) – denis Dec 2 '14 at 17:01