## [Gradient Descent](/gradient-descent-0) Definition An iterative optimization method that updates parameters by moving them opposite to the gradient of an objective function, scaled by a step size, to reduce the function value. ## Principle Principle At each iteration, take a step proportional to the negative derivative (or gradient) to locally decrease the objective; convergence depends on step size, curvature, and smoothness. ## Demonstration Demonstration For a quadratic cost f(x)=ax^2+bx+c with a>0, iterating x_{n+1}=x_n - η f'(x_n) converges to the minimizer for sufficiently small learning rate η, with linear convergence rate determined by a and η. ## Misapplication Misapplication Choosing a fixed step size too large for the local curvature causes oscillation or divergence; neglecting gradient noise in stochastic settings can prevent convergence. ## Consequence Consequence When the objective is convex and step sizes are chosen appropriately, gradient descent converges to a global minimum; in nonconvex problems it typically finds a local minimum or saddle. ## Reversal Reversal Gradient ascent: update parameters in the direction of the gradient to maximize an objective rather than minimize it. ## Boundary Boundary Requires availability of gradients or subgradients; not directly applicable to nondifferentiable objectives without modification (e.g., proximal steps) or to constrained domains without projection or Lagrange methods. ## Semantic Tension Semantic Tension Often compared to second-order methods: gradient descent uses only first derivatives and has cheaper iterations but slower local convergence compared with Newton-like methods that use curvature. ## Synthesis Synthesis Gradient descent is the first-order iterative algorithm that reduces an objective by stepping opposite the gradient; its performance balances step-size control, curvature, and noise to produce local or global minimizers under suitable conditions.