A directed measure of discrepancy between two probability distributions defined as the expectation under one distribution of the log ratio of their densities; quantifies expected log‑loss when using one distribution to approximate another.

Principle

Nonnegativity with equality only when the two distributions coincide almost everywhere; it is asymmetric and additive under independent product measures, capturing directional information loss.

Demonstration

For two normal distributions with equal variance σ^2 and means μ1 and μ2, the divergence of the first from the second equals (μ1−μ2)^2/(2σ^2).

Misapplication

Interpreting the divergence as a distance metric is incorrect because it is asymmetric and does not satisfy the triangle inequality; using it blindly where symmetry is required leads to misleading comparisons.

Consequence

Used as an objective to fit approximate models to targets, it yields parameter estimates that minimize expected log‑loss and provides a principled criterion for information‑optimal approximation under one‑sided weighting.

Reversal

Symmetrized alternatives (e.g., the Jensen–Shannon divergence) produce bounded, symmetric measures better suited when reciprocity between distributions is required.

Boundary

Defined only when the support of the reference distribution is contained in the support of the approximating distribution (absolute continuity); it can be infinite when supports do not align.

Semantic Tension

Contrasts with metric notions like total variation: Kullback–Leibler emphasizes average log‑likelihood discrepancy relevant to coding and inference, whereas metrics capture worst‑case probability differences.

Synthesis

A directional information measure that quantifies the expected log‑loss incurred by approximating one probability distribution with another, foundational in inference and information theory.