A linear dimensionality-reduction method that finds orthogonal directions (principal components) in data space which successively maximize variance; computed via eigen-decomposition of the empirical covariance or singular value decomposition of the data matrix.

Principle

Project data onto an orthonormal basis that diagonalizes the empirical second-moment structure; retaining the top-k components yields the best k-dimensional linear approximation in mean-squared error among orthogonal projections.

Demonstration

Given centered samples in R^d, compute the covariance matrix, find its top eigenvectors, and project the data onto the span of those eigenvectors to compress and de-noise while preserving maximal linear variance.

Misapplication

Interpreting principal components as causal factors, applying PCA to strongly non-linear manifolds without preprocessing, or using PCA on uncentered data which shifts principal directions toward the mean offset.

Consequence

Produces decorrelated coordinates ordered by explained variance, enabling compression, visualization, and noise reduction when dominant directions capture signal structure.

Reversal

Projecting onto low-variance components yields directions that minimize variance capture and often emphasize noise; probabilistic factor models invert PCA's deterministic variance-maximization by introducing latent-variable noise models.

Boundary

Assumes linear relations and second-moment sufficiency; not optimal for non-linear structure-preserving reductions (manifold learning) or for heavy-tailed data without robust covariance estimation.

Semantic Tension

Versus factor analysis or independent component analysis: PCA is an orthogonal variance-maximizer without an explicit generative noise model, whereas competing methods impose latent-variable or independence assumptions.

Synthesis

An orthogonal projection method that diagonalizes covariance to extract ordered linear modes of maximal variance, yielding an optimal least-squares low-dimensional linear summary when data are mean-centered.