A model compression technique in which a smaller 'student' model is trained to reproduce the behavior or softened outputs of a larger 'teacher' model so that the student inherits useful predictive structure while being more compact and efficient.

Principle

Transfer predictive structure by training the student on teacher-generated targets (e.g., probabilities, logits, intermediate representations) often using temperature scaling and loss terms that combine teacher alignment with direct supervision on labels.

Demonstration

Train a compact neural network to match the softened class probabilities produced by a larger pretrained classifier on the same inputs; the student reaches near-teacher accuracy while using fewer parameters and less inference time.

Misapplication

Using distillation to blindly copy a teacher's outputs propagates teacher errors and biases; distilling on unrepresentative data or without balancing teacher alignment and ground-truth loss can degrade generalization.

Consequence

A correctly distilled student provides a smaller, faster model that approximates the teacher's behavior, enabling deployment under resource constraints but usually at some loss of fidelity to rare or fine-grained teacher decisions.

Reversal

Instead of distilling from teacher to student, train the smaller model directly on the original labeled data (or construct the teacher from ensembles of small models) — this prioritizes ground-truth fidelity over teacher mimicry.

Boundary

Applies where a high-capacity model exists and a lighter model is desired; not equivalent to pruning, quantization, or architecture search, which alter model internals rather than transferring predictive patterns; does not by itself guarantee removal of undesirable teacher properties.

Semantic Tension

Competes with transfer learning and fine-tuning: transfer learning reuses learned representations via weight initialization or feature extraction, while distillation aims to transfer predictive behavior explicitly via teacher outputs.

Synthesis

Knowledge distillation frames compression as behavioral transfer: by training a smaller model to match a larger model's softened outputs or internal signals, one produces a deployable approximation that preserves much of the teacher's predictive structure while trading some fidelity for efficiency.