An iterative optimization method that updates parameters by moving them opposite to the gradient of an objective function, scaled by a step size, to reduce the function value.

Principle

At each iteration, take a step proportional to the negative derivative (or gradient) to locally decrease the objective; convergence depends on step size, curvature, and smoothness.

Demonstration

For a quadratic cost f(x)=ax^2+bx+c with a>0, iterating x_{n+1}=x_n - η f'(x_n) converges to the minimizer for sufficiently small learning rate η, with linear convergence rate determined by a and η.

Misapplication

Choosing a fixed step size too large for the local curvature causes oscillation or divergence; neglecting gradient noise in stochastic settings can prevent convergence.

Consequence

When the objective is convex and step sizes are chosen appropriately, gradient descent converges to a global minimum; in nonconvex problems it typically finds a local minimum or saddle.

Reversal

Gradient ascent: update parameters in the direction of the gradient to maximize an objective rather than minimize it.

Boundary

Requires availability of gradients or subgradients; not directly applicable to nondifferentiable objectives without modification (e.g., proximal steps) or to constrained domains without projection or Lagrange methods.

Semantic Tension

Often compared to second-order methods: gradient descent uses only first derivatives and has cheaper iterations but slower local convergence compared with Newton-like methods that use curvature.

Synthesis

Gradient descent is the first-order iterative algorithm that reduces an objective by stepping opposite the gradient; its performance balances step-size control, curvature, and noise to produce local or global minimizers under suitable conditions.