A nonparametric Bayesian regression method that models an unknown function as a Gaussian process: a collection of random variables indexed by input points with joint Gaussian distributions specified by a mean function and covariance (kernel) function.

Principle

Given a Gaussian process prior and (typically) Gaussian observation noise, compute the posterior Gaussian process conditioned on observed data; closed-form expressions yield predictive mean and covariance at new inputs.

Demonstration

Use a squared-exponential kernel prior, observe noisy samples of a one-dimensional function, then compute the posterior mean curve and pointwise predictive standard deviation providing interpolation and uncertainty bands.

Misapplication

Employing a stationary kernel without checking nonstationary structure in data, or optimizing hyperparameters without guarding against overfitting small datasets, leading to misleading confidence bands.

Consequence

Produces a predictive distribution (mean and credible intervals) that reflects data and prior assumptions, enables principled uncertainty quantification, and integrates kernel learning with Bayesian inference.

Reversal

Parametric regression imposes a finite-dimensional parameterization and yields point estimates; in contrast, GP regression maintains an infinite-dimensional prior and yields full predictive distributions.

Boundary

Exact Gaussian process regression assumes Gaussian noise and scales cubically with data size; extensions and approximations (sparse GPs, non-Gaussian likelihoods with approximate inference) are outside the exact closed-form regime.

Semantic Tension

Closely related to kernel ridge regression and spline smoothing (same predictive mean under certain priors), yet differs in providing a full Bayesian posterior and principled uncertainty estimates.

Synthesis

Gaussian process regression treats functions as random draws from a covariance-specified Gaussian prior and, via Bayes' rule with data and noise assumptions, yields closed-form posterior predictions and uncertainties.