An iterative optimization algorithm that updates parameters by moving them opposite to the gradient of an objective function (or an estimate thereof) to reduce the function value and seek a local minimum.

Principle

Use the local first-order Taylor approximation: the negative gradient is the direction of steepest local decrease; step length (learning rate) and curvature determine convergence behavior and rate.

Demonstration

For minimizing a convex quadratic f(x)=x^T A x with A positive definite, gradient descent with an appropriately chosen step size converges linearly to the unique minimizer, with rate governed by the condition number of A.

Misapplication

Using a fixed large step size on a poorly conditioned or nonconvex objective leading to divergence, or assuming global optimality in multimodal landscapes without further guarantees.

Consequence

Provides a simple, scalable basis for many numerical and machine-learning training procedures; with appropriate variants (momentum, adaptive rates, stochastic sampling) it handles large-scale and noisy problems effectively.

Reversal

Gradient ascent follows the gradient to increase the objective and finds local maxima; second-order methods use curvature (Hessian) information to adjust direction and step for faster convergence.

Boundary

Requires differentiability (or subgradient information) of the objective; performance depends on smoothness, convexity, step-size policy, and noise—does not guarantee global optimum in nonconvex problems.

Semantic Tension

Often conflated with stochastic gradient descent or quasi-Newton methods; distinction lies in using exact versus noisy gradients and first-order versus higher-order curvature exploitation.

Synthesis

Gradient descent iteratively moves parameters opposite the local gradient using step-size control to decrease the objective, forming a foundational first-order optimization method whose behavior depends on smoothness and curvature.