A simple counting argument suffices to show that any general-purpose function-minimization algorithm in
dimensions
must involve at least
function evaluations (see, for example, Press et al. [125
, Section 10.6]): Suppose the function to
be minimized is
, and suppose
has a local minimum near some point
. Taylor-expanding
in a
neighborhood of
gives
, where
,
is
symmetric, and
denotes the transpose of the column vector
.
Neglecting the higher order terms (i.e. approximating
as a quadratic form in
in a neighborhood of
), and
ignoring
(which does not affect the position of the minimum), there are a total of
coefficients
in this expression. Changing any of these coefficients may change the position of the minimum, and at each
function evaluation the algorithm “learns” only a single number (the value of
at the selected evaluation
point), so the algorithm must make at least
function evaluations to (implicitly) determine all the
coefficients.
Actual functions are not exact quadratic forms, so in practice there are additional
multiplicative factors in
the number of function evaluations. Minimization algorithms may also make additional performance and/or
space-versus-time trade-offs to improve numerical robustness or to avoid explicitly manipulating
Jacobian
matrices.