A quantitative measure of the uncertainty or average information content of a discrete probability distribution, defined as H = -Σ p(x) log p(x).

Principle

Information content of an event is its surprisal (−log p); Shannon entropy is the expected surprisal and sets fundamental limits on lossless coding and average description length.

Demonstration

A fair coin distribution {0.5, 0.5} yields H = 1 bit; a biased coin with p(heads)=0.9 yields H ≈ 0.47 bits, reflecting lower average uncertainty.

Misapplication

Equating Shannon entropy directly with thermodynamic entropy without accounting for differing ensembles and physical interpretations, or applying discrete formula indiscriminately to continuous variables without using differential entropy.

Consequence

Shannon entropy bounds the minimum average number of bits needed for lossless encoding and underlies channel capacity formulas and source coding theorems.

Reversal

Maximum order or complete predictability (zero entropy), where one outcome has probability one and no information is gained by observation.

Boundary

Applies to probability distributions over discrete random variables under a specified logarithm base; extensions exist (conditional, joint, differential entropy) but thermodynamic and algorithmic notions of entropy lie in related but distinct frameworks.

Semantic Tension

Often confused with physical entropy; Shannon entropy is an information-theoretic expectation over probabilities, whereas thermodynamic entropy refers to physical microstate counts and energy considerations.

Synthesis

Shannon entropy quantifies the expected information (surprisal) in a probability distribution and provides operational limits for data compression and communication under probabilistic models.