The family or curve of model solutions parameterized by a regularization hyperparameter (e.g., penalty strength); it traces how estimated parameters change continuously as the penalty is varied from strong to weak regularization.

Principle

By treating the regularization weight as a continuous homotopy parameter, one obtains a trajectory in parameter space that reveals trade-offs between fit and penalty, often enabling efficient warm-start algorithms and model selection by following solutions along the path.

Demonstration

In Lasso regression, the regularization path shows coefficient values as the ℓ1 penalty λ decreases from large (all coefficients zero) to small (least-squares solution); piecewise-linear solver algorithms exploit this structure to compute the full path efficiently.

Misapplication

Assuming path continuity, uniqueness, or piecewise-smoothness for nonconvex penalties or ill-conditioned problems where multiple local minima produce branching or discontinuous jumps in the solution path.

Consequence

Access to the regularization path allows principled selection of penalty strength (cross-validation along the path), interpretable evolution of features entering the model, and the use of homotopy methods to compute solutions efficiently over a range of penalties.

Reversal

Fixing a single regularization parameter yields one final model estimate without information about the trajectory of solutions or trade-offs at other penalty levels.

Boundary

Defined when the estimator is the solution of a parameterized optimization problem whose minimizers vary with a penalty parameter; for nonparametric methods, stochastic regularization, or strongly nonconvex objectives the notion of a single well-defined path may fail.

Semantic Tension

Tension with the concept of a solution manifold: the regularization path is a one-dimensional trace in parameter space for a single penalty schedule, while a solution manifold may be higher-dimensional and include all minimizers across multiple hyperparameters or constraints.

Synthesis

A regularization path is the curve of optimizer values obtained by continuously varying a penalty parameter; it exposes the trade-off between fidelity and regularity, aids selection, and underlies homotopy algorithms when path regularity holds.