A Monte Carlo variance-reduction technique that estimates expectations under a target probability distribution by sampling from a different proposal distribution and reweighting samples by the likelihood ratio (target/proposal).

Principle

Replace direct sampling from a target measure (which may be difficult or inefficient) by sampling from a proposal that over-samples important regions and correct the bias by multiplying observables by the Radon–Nikodym derivative (density ratio) to obtain unbiased estimates.

Demonstration

To estimate E[f(X)] where X has density p, draw samples X_i from a proposal q with q(x)>0 whenever p(x)f(x)≠0, compute weights w_i=p(X_i)/q(X_i), and form the estimator (1/N) Σ_{i} w_i f(X_i); choosing q proportional to |f|p minimizes variance in the idealized case.

Misapplication

Selecting a proposal q that underweights regions where p·f is large, leading to extreme importance weights, high variance or infinite variance estimators—e.g., using a narrow Gaussian where the target has heavy tails.

Consequence

When applied with a well-chosen proposal, importance sampling dramatically reduces estimator variance and computational cost for rare-event probabilities and tail expectations; it yields unbiased estimators with known weight-correction.

Reversal

Instead of importance sampling from an alternative distribution with weights, use stratified sampling or control variates; these also reduce variance but act by partitioning or correlating estimators rather than reweighting a proposal density ratio.

Boundary

Requires absolute continuity of target with respect to proposal on regions contributing to the expectation (support condition) and computable density ratios; fails if q(x)=0 on sets where p(x)f(x)≠0 or if weights have infinite variance.

Semantic Tension

Often conflated with importance resampling in particle filters: both use weights, but importance sampling refers to weighted Monte Carlo estimation from a proposal, whereas resampling is a downstream step to produce unweighted samples or to avoid weight degeneracy.

Synthesis

Importance sampling = sample from an accessible proposal that emphasizes important regions and reweight by the density ratio to produce unbiased, lower-variance Monte Carlo estimates when support and weight-variance conditions hold.