Matching and propensity score methods

The assumption everything hangs on

Matching answers the counterfactual question by construction: for each participant, find one or more non-participants who are alike on everything that matters, and use their outcomes as the estimate of what would have happened without the programme [3, 4]. The phrase doing all the work is everything that matters. Formally, the design assumes selection on observables (also called conditional independence or unconfoundedness): once the measured covariates are held equal, who participated and who did not is unrelated to what their outcomes would have been [1, 2].

Say this bluntly in any evaluation that uses matching, because it is a strong claim and it is untestable. Participation in real programmes runs on things administrative data rarely record — motivation, information, social connections, unrecorded shocks. If any of those drive both participation and outcomes, the matched comparison is biased, and no amount of statistical machinery on the observed covariates repairs it [1, 2, 4]. This is why matching sits below randomisation and below designs with a known assignment rule in this cluster’s ordering: a randomised trial makes the groups comparable on unobservables by construction, and a regression discontinuity’s assignment rule is verifiable, while matching’s assumption is a judgement call about what the data captured. The defence of a matching study is therefore substantive, not statistical: an argument, grounded in knowledge of how the programme recruited, that the covariate set plausibly spans the drivers of selection [2, 3].

The propensity score

Matching on many covariates at once collapses quickly — with even a dozen characteristics, exact matches stop existing. Rosenbaum and Rubin’s result rescues the enterprise: the propensity score, the probability of participating given the covariates, is a balancing score. Units with the same propensity score have the same distribution of the measured covariates regardless of whether they participated, so matching on this single number achieves the covariate balance that matching on everything separately was meant to deliver [1].

Two cautions travel with the theorem. The score must be estimated (typically by logistic regression of participation on covariates), and a badly specified score model balances nothing — which is why the diagnostics below, not the score model’s fit statistics, are the test of success [2]. And the theorem covers only the measured covariates: the propensity score inherits, and cannot relax, the selection-on-observables assumption [1, 2].

The estimator menu

With scores (or covariates) in hand, several estimators produce the comparison [2, 3]:

• Nearest-neighbour matching pairs each participant with the closest non-participant(s), with or without replacement.
• Caliper matching adds a tolerance: no match is accepted beyond a maximum score distance, trading sample size for match quality.
• Kernel and local-linear matching weight all comparison units by their distance, using more of the data at the cost of including poorer matches with small weights.
• Exact and coarsened matching require identity (or identity within bins) on designated covariates — natural where a few characteristics, such as county or sex, must not differ.
• Stratification divides the score’s range into blocks and compares within them; inverse-probability weighting skips pairing altogether and reweights the comparison sample by the odds of participation.

The honest summary of the methods literature: the choice among estimators matters far less than the quality of the covariate set and the balance achieved [2]. Pre-register the estimator, then let the diagnostics arbitrate.

Common support and balance: the diagnostics that decide

Common support. Matching only works where participants and non-participants overlap. Participants whose propensity scores exceed every non-participant’s have no counterfactual in the data; they must be excluded, and the report must say who was dropped — because dropping them changes the population the estimate describes [2, 3].

Balance. After matching, the measured covariates should be distributed near-identically across the two groups. The standard exhibit is the standardised mean difference (SMD) for each covariate — the difference in means scaled by the pooled standard deviation — shown before and after matching. A widely used rule of thumb treats absolute SMDs under 0.1 as acceptable balance; whatever threshold is used, judge balance on magnitudes, not on significance tests, whose verdicts move with sample size rather than comparability [2].

Sensitivity analysis and honest reporting

Because the core assumption is untestable, a credible matching study asks the next-best question: how wrong could it be before the conclusion changes? Sensitivity analyses quantify how strongly an unmeasured confounder would have to be associated with both participation and the outcome to explain away the estimated effect [2]. An effect that survives only a whisper of hidden bias should be reported as fragile; one that would require an implausibly powerful unmeasured driver earns more confidence — though never the standing of a randomised estimate.

Reporting language should match the design’s epistemics. “Participants improved by X relative to matched non-participants who were similar on the characteristics we could measure” is accurate; “the programme caused X” without the conditioning clause overstates what matching delivers [2, 4]. Report the covariate set and why it plausibly spans selection, the common-support exclusions, before/after balance, the sensitivity analysis, and the estimator specification — in that order of importance.

Matching also combines well with other designs. Matching on baseline characteristics and then applying difference-in-differences to the matched panel removes both observable composition differences and fixed unobservable ones, a pairing common in development practice — see the difference-in-differences page for the assumptions that combination inherits [3, 4].

Practical workflow

1. Reconstruct how selection actually happened — interviews with programme staff are evaluation evidence here, not background. List the drivers; check which the data measure. Covariates must be pre-programme: matching on variables the programme could have affected builds the impact into the match [2, 3].
2. Estimate the propensity score; inspect common support.
3. Match under the pre-specified estimator; iterate the score specification until balance is achieved — balance-checking is legitimate model selection because it never touches outcomes [2].
4. Estimate the effect on the matched or weighted sample; run sensitivity analysis.
5. Report as above. Standard implementations exist in R (MatchIt) and Stata’s treatment-effects commands; the tooling is not the hard part [2].

Data quality determines everything upstream: covariates measured inconsistently across participant and comparison records — different instruments, different recall periods — will produce beautiful balance on numbers that mean different things. The data-quality dimensions page in the data cluster covers the failure modes.

Checklist before you commit to matching

• No stronger design is available: randomisation, an assignment cutoff, and a defensible before/after comparison structure have all been ruled out.
• The selection process is understood, and the measured covariates plausibly span its drivers — argued in writing, not assumed.
• All covariates predate the programme and are measured comparably for both groups.
• Common support holds for most participants, and exclusions are reported.
• Post-matching balance is demonstrated with standardised mean differences.
• A sensitivity analysis for unmeasured confounding accompanies the estimate, and the report’s causal language carries the conditioning clause.