The long-run average Shannon information produced per symbol by a stochastic process, typically defined as the limit H = lim_{n→∞} (1/n) H(X1,...,Xn) or equivalently H = lim_{n→∞} H(Xn | X1,...,Xn−1) when the limit exists.

Principle

Entropy rate quantifies the average unpredictability or information production of a source per time step and forms the fundamental lower bound for lossless compression of stationary processes.

Demonstration

For a stationary Markov chain with stationary distribution π and transition probabilities P(i→j), the entropy rate is H = −∑_{i} π(i) ∑_{j} P(i→j) log P(i→j), the expected conditional entropy of the next state given the current.

Misapplication

Using the marginal entropy H(Xn) of single variables as the entropy rate for dependent sequences ignores temporal dependence and can substantially overestimate compressibility bounds.

Consequence

When well-defined for a stationary ergodic source, the entropy rate equals the optimal per-symbol compression rate in the limit, determines typical set sizes, and governs universal coding limits.

Reversal

A zero entropy rate indicates deterministic or asymptotically predictable behavior; conversely, maximal entropy rate (given constraints) corresponds to memoryless or independent identically distributed processes under those constraints.

Boundary

Defined for stochastic processes (discrete-time) where limits exist; formal statements require stationarity or ergodicity for many coding and typicality results; alternative notions exist for Rényi or metric entropies but differ from Shannon entropy rate.

Semantic Tension

Distinct from one‑letter or marginal entropy: marginal entropy measures uncertainty of a single draw, while entropy rate measures per-symbol uncertainty accounting for temporal correlations in the process.

Synthesis

Entropy rate is the asymptotic per-symbol Shannon information of a stochastic process, capturing its average unpredictability and setting the fundamental compression and typicality limits for stationary sources.