A principled framework for deriving compressed representations of an input variable that preserve as much relevant information as possible about a specified target variable, trading off compactness against predictive fidelity.

Principle

Representations are chosen to minimize retained information about the input while constraining or maximizing retained information about the target; the optimal trade-off is governed by a Lagrangian balance between compression cost and relevance measure.

Demonstration

When compressing high-dimensional sensory descriptors to predict categorical labels, one seeks a stochastic encoder that maps inputs to a limited set of codes which still retain the majority of the mutual dependence with labels, enabling efficient downstream classifiers.

Misapplication

Applying the bottleneck objective without accounting for the true joint distribution of input and target or using inappropriate parametric encoder families, which yields codes that either discard task-relevant structure or overfit to sample noise.

Consequence

Proper application yields representations that are minimal sufficient for the target under the chosen compression level, often improving generalization and interpretability by discarding nuisance variability irrelevant to the task.

Reversal

An objective that preserves maximal information about the input regardless of relevance to the target, producing high-fidelity encodings with little or no compression and potential retention of irrelevant detail.

Boundary

Formulation assumes a well-defined joint probability law for input and target and quantifiable information measures; it excludes deterministic contexts where mutual dependence is ill-posed without a noise model and settings where compression metrics are not information-theoretic.

Semantic Tension

Tension exists between minimal sufficient representations (task-centric compression) and reconstruction-oriented compression that aims to preserve all input detail; the bottleneck favors relevance while reconstruction favors fidelity to the original signal.

Synthesis

The information bottleneck principle formalizes the trade-off of compressing inputs while preserving task-relevant dependence on a target, producing compact representations that prioritize predictive information over irrelevant detail.