monival.org

01 · Evaluation designs and causal inference

Difference-in-differences

Difference-in-differences (DiD) estimates a programme's impact by comparing the change in outcomes over time between a group that received the intervention and one that did not. It removes fixed differences between the groups and shared trends over time — but it stands or falls on the assumption that, absent the programme, both groups would have moved in parallel.

Last updated · Reviewed against 4 cited sources

When to use it

Difference-in-differences is the workhorse design when randomisation is off the table but a comparison group exists that was exposed to the same environment as the treated group — neighbouring districts, facilities that adopted later, firms below a policy threshold. It suits policies and programmes that switch on at a known time for an identifiable group, and it requires outcome measurements for both groups before and after that switch [1].

Prefer a randomised design when you can assign treatment; prefer regression discontinuity when a sharp eligibility cutoff exists. Reach for DiD when neither holds but history has produced a credible comparison.

What the estimator does

Two differences, subtracted:

  1. First difference — the change in the outcome within each group between baseline and endline. Differencing within a group removes everything about that group that does not change over time (geography, composition, baseline capability).
  2. Second difference — the treated group’s change minus the comparison group’s change. This removes shocks and trends common to both groups (seasonality, macro shocks, a national policy affecting everyone).

What remains is the impact estimate — if the only differential change was the programme itself.

Difference-in-differences identification, shown as two outcome trends

A chart with time on the horizontal axis (baseline to endline) and the outcome on the vertical axis. The comparison group’s outcome rises along one line. The treated group starts at a different level and rises more steeply. A dashed line parallel to the comparison group’s trend, starting from the treated group’s baseline, shows the assumed counterfactual. The vertical gap at endline between the treated group’s observed outcome and the dashed counterfactual is labelled as the impact estimate.

OutcomeBaselineEndlineComparison groupCounterfactual (assumed parallel)Treated groupImpact
Figure 1. The DiD logic: the comparison group's trend, applied to the treated group's baseline, constructs the counterfactual (dashed). Impact is the gap between the observed and counterfactual outcomes at endline.Presentation follows the exposition in Gertler et al. (2016), ch. 7.

The arithmetic is a two-by-two table. With stylised numbers:

Stylised two-by-two difference-in-differences arithmetic
GroupBaselineEndlineChange
Treated30.038.0+8.0
Comparison35.040.0+5.0
Difference-in-differences+3.0
Table 1. Stylised numbers. Each cell is the group mean of the outcome; the DiD estimate is the difference of the two changes.

Everything rests on one claim: without the programme, the treated group’s outcome would have changed by the same amount as the comparison group’s [1]. This is a statement about a world that did not happen, so it cannot be verified directly. It can, however, be made credible or incredible:

  • Pre-trends. With several pre-intervention measurements, show the two groups moving in parallel before treatment. Parallel history is not proof of parallel counterfactuals, but diverging history is close to disproof.
  • Placebo tests. Estimate the “effect” in a period before the programme existed, or on an outcome the programme could not plausibly affect. A significant placebo effect signals a broken design.
  • No differential shocks. Nothing else that moves the outcome may switch on for one group at the same time as the programme — a concurrent grant, drought, or policy aimed at the same population is a competing explanation the subtraction cannot remove.
Measurement timeline for a defensible difference-in-differences designA timeline with five measurement points: two pre-baseline rounds used to test parallel trends, a baseline, the intervention start, and an endline. A band marks the exposure period between intervention and endline.Pre-round 1trend checkPre-round 2trend checkBaselineboth groupsInterventiontreated group onlyEndlineboth groupsexposure period
Figure 2. More than one pre-period measurement turns the parallel-trends assumption from an act of faith into an inspectable claim.

Getting the uncertainty right

The most common technical failure in applied DiD is not the point estimate but the standard error. Outcomes measured repeatedly on the same units are serially correlated, and treating each period as independent information makes confidence intervals far too narrow — Bertrand, Duflo and Mullainathan showed that conventional DiD inference finds “significant” effects of placebo laws at alarming rates [2]. The standing remedy: cluster standard errors at the level where treatment is assigned (the state, the district, the facility), and be honest when the number of clusters is small.

The canonical example

Card and Krueger’s minimum-wage study is the design’s most famous application: New Jersey raised its minimum wage in April 1992 while neighbouring Pennsylvania did not, and fast-food restaurants on both sides of the border were surveyed before and after the change [3]. The comparison of employment changes across the two states — rather than employment levels in either — is exactly the two-by-two logic above, and the ensuing decades of scrutiny of that study are a working seminar in how parallel-trends arguments are attacked and defended.

Staggered rollout: the modern caveat

Real programmes rarely switch on everywhere at once. When units adopt at different times, the traditional shortcut — a two-way fixed-effects regression with a treatment dummy — implicitly compares late adopters against already-treated early adopters, and can even produce a wrong-signed estimate when effects change over time. The post-2018 econometrics literature provides estimators that handle staggered adoption correctly; Roth, Sant’Anna, Bilinski and Poe synthesise them and give concrete practitioner recommendations [4]. If your rollout is staggered, do not run the old regression by default.

Checklist before you commit to DiD

  • Baseline and endline data exist (or can be collected) for treated and comparison groups, with the same instrument and timing.
  • At least one — ideally several — pre-intervention rounds support parallel trends.
  • No other intervention differentially hits either group during the study window.
  • Standard errors are clustered where treatment varies, and the cluster count is reported.
  • If adoption is staggered, a heterogeneity-robust estimator is specified in advance.

Sources

  1. Impact Evaluation in Practice, Second Edition — World Bank / Inter-American Development Bank, 2016.Gertler, Martinez, Premand, Rawlings & Vermeersch. Chapter 7 is the standard practitioner treatment of DiD.
  2. How Much Should We Trust Differences-in-Differences Estimates? — The Quarterly Journal of Economics, 119(1), 249–275, 2004.Bertrand, Duflo & Mullainathan on why serial correlation wrecks naive DiD standard errors.
  3. Minimum Wages and Employment: A Case Study of the Fast Food Industry in New Jersey and Pennsylvania — NBER Working Paper 4509 (published in American Economic Review 84(4), 772–793), 1994.Card & Krueger — the canonical two-group, two-period DiD application.
  4. What's Trending in Difference-in-Differences? A Synthesis of the Recent Econometrics Literature — arXiv (published in Journal of Econometrics 235(2), 2218–2244), 2023.Roth, Sant'Anna, Bilinski & Poe — the survey of staggered-adoption and robust-inference advances.