##  [Kullback–Leibler Divergence](/kullback-leibler-divergence-0) 

 Definition

A directed measure of discrepancy between two probability distributions defined as the expectation under one distribution of the log ratio of their densities; quantifies expected log‑loss when using one distribution to approximate another.

 

 

 

 

 

 





## Principle

Principle

Nonnegativity with equality only when the two distributions coincide almost everywhere; it is asymmetric and additive under independent product measures, capturing directional information loss.

 

 

 

 

 





## Demonstration

Demonstration

For two normal distributions with equal variance σ^2 and means μ1 and μ2, the divergence of the first from the second equals (μ1−μ2)^2/(2σ^2).

 

 

 

 

## Misapplication

Misapplication

Interpreting the divergence as a distance metric is incorrect because it is asymmetric and does not satisfy the triangle inequality; using it blindly where symmetry is required leads to misleading comparisons.

 

 

 

 

 





## Consequence

Consequence

Used as an objective to fit approximate models to targets, it yields parameter estimates that minimize expected log‑loss and provides a principled criterion for information‑optimal approximation under one‑sided weighting.

 

 

 

 

## Reversal

Reversal

Symmetrized alternatives (e.g., the Jensen–Shannon divergence) produce bounded, symmetric measures better suited when reciprocity between distributions is required.

 

 

 

 

 





## Boundary

Boundary

Defined only when the support of the reference distribution is contained in the support of the approximating distribution (absolute continuity); it can be infinite when supports do not align.

 

 

 

 

 





## Semantic Tension

Semantic Tension

Contrasts with metric notions like total variation: Kullback–Leibler emphasizes average log‑likelihood discrepancy relevant to coding and inference, whereas metrics capture worst‑case probability differences.

 

 

 

 

 





## Synthesis

Synthesis

A directional information measure that quantifies the expected log‑loss incurred by approximating one probability distribution with another, foundational in inference and information theory.