##  [Entropy Rate](/entropy-rate-0) 

 Definition

The long-run average Shannon information produced per symbol by a stochastic process, typically defined as the limit H = lim_{n→∞} (1/n) H(X1,...,Xn) or equivalently H = lim_{n→∞} H(Xn | X1,...,Xn−1) when the limit exists.

 

 

 

 

 

 





## Principle

Principle

Entropy rate quantifies the average unpredictability or information production of a source per time step and forms the fundamental lower bound for lossless compression of stationary processes.

 

 

 

 

 





## Demonstration

Demonstration

For a stationary Markov chain with stationary distribution π and transition probabilities P(i→j), the entropy rate is H = −∑_{i} π(i) ∑_{j} P(i→j) log P(i→j), the expected conditional entropy of the next state given the current.

 

 

 

 

## Misapplication

Misapplication

Using the marginal entropy H(Xn) of single variables as the entropy rate for dependent sequences ignores temporal dependence and can substantially overestimate compressibility bounds.

 

 

 

 

 





## Consequence

Consequence

When well-defined for a stationary ergodic source, the entropy rate equals the optimal per-symbol compression rate in the limit, determines typical set sizes, and governs universal coding limits.

 

 

 

 

## Reversal

Reversal

A zero entropy rate indicates deterministic or asymptotically predictable behavior; conversely, maximal entropy rate (given constraints) corresponds to memoryless or independent identically distributed processes under those constraints.

 

 

 

 

 





## Boundary

Boundary

Defined for stochastic processes (discrete-time) where limits exist; formal statements require stationarity or ergodicity for many coding and typicality results; alternative notions exist for Rényi or metric entropies but differ from Shannon entropy rate.

 

 

 

 

 





## Semantic Tension

Semantic Tension

Distinct from one‑letter or marginal entropy: marginal entropy measures uncertainty of a single draw, while entropy rate measures per-symbol uncertainty accounting for temporal correlations in the process.

 

 

 

 

 





## Synthesis

Synthesis

Entropy rate is the asymptotic per-symbol Shannon information of a stochastic process, capturing its average unpredictability and setting the fundamental compression and typicality limits for stationary sources.