##  [Mutual Information](/mutual-information-0) 

 Definition

A nonnegative scalar that quantifies the amount of statistical dependence between two random variables by measuring how much knowing one reduces uncertainty about the other.

 

 

 

 

 

 





## Principle

Principle

Express shared information as the reduction in uncertainty about one variable given knowledge of the other without presupposing linear relationships.

 

 

 

 

 





## Demonstration

Demonstration

Two binary variables that are identical have maximal mutual information equal to the variable's information content; if they are independent, mutual information is zero.

 

 

 

 

## Misapplication

Misapplication

Interpreting a large mutual information value as evidence of causal influence confuses dependence with directionality and can be misleading without temporal or intervention data.

 

 

 

 

 





## Consequence

Consequence

Serves as a model-agnostic criterion for feature selection, variable clustering, and detection of nonlinear associations between variables.

 

 

 

 

## Reversal

Reversal

Independence is the inverse situation: zero mutual information implies no statistical dependence under the chosen probability model.

 

 

 

 

 





## Boundary

Boundary

Requires a joint distribution; for continuous variables practical estimation demands density or discretization methods and suffers from sample-size limitations.

 

 

 

 

 





## Semantic Tension

Semantic Tension

Versus correlation coefficients: mutual information captures arbitrary dependence including nonlinear relationships, while correlation measures only linear association strength.

 

 

 

 

 





## Synthesis

Synthesis

A scalar measure of dependence that quantifies how much one variable reduces uncertainty about another, applicable to discrete or continuous random variables with appropriate estimation.