A gradient‑based computational procedure that applies the chain rule to a feedforward computational graph to compute derivatives of a loss function with respect to model parameters by propagating error signals from outputs back toward inputs.

Principle

Represent the model as a composition of differentiable operations; compute local derivatives at each node and propagate gradients backward using the chain rule (reverse‑mode automatic differentiation) to obtain parameter gradients efficiently for use by an optimizer.

Demonstration

In a three‑layer neural network, a forward pass computes outputs and a scalar loss; backpropagation computes gradients of the loss with respect to hidden layer weights by multiplying local Jacobians backwards from the loss through the activation and linear layers, enabling gradient descent updates.

Misapplication

Applying backpropagation to operations that are non‑differentiable or have discontinuous gradients (without relaxation), or ignoring numerical issues (vanishing/exploding gradients, poor initialization), leading to incorrect or ineffective training.

Consequence

Provides an efficient means to compute exact gradients for differentiable parameterized models with complexity proportional to a small constant factor times the forward pass, enabling gradient‑based optimization and large‑scale training of neural networks.

Reversal

Forward‑mode differentiation or derivative‑free methods: computing directional derivatives forward through the graph or using black‑box optimization avoids reverse accumulation but is typically less efficient when the number of inputs (parameters) is large.

Boundary

Applies to models expressible as differentiable computational graphs and to scalar or vector losses; does not by itself solve issues of local minima, poor generalization, or discrete decision structures without additional techniques (relaxations, surrogate losses). Numerical stability and memory trade‑offs limit applicability in very deep or recurrent structures without modifications.

Semantic Tension

Tension between backpropagation as an algorithmic instantiation of reverse‑mode automatic differentiation and its interpretation as a biological learning model: mathematically precise gradient accumulation contrasts with heuristic or biologically plausible learning rules.

Synthesis

Backpropagation is reverse‑mode automatic differentiation applied to parameterized feedforward computations: it propagates loss gradients backward across composed differentiable operations to produce parameter gradients used by optimizers to reduce loss.