Statistics Glossary → Difference-in-Differences (DiD)
Difference-in-Differences (DiD)analysis
A method that estimates the causal effect of an intervention by comparing the before-after change in outcome in a group exposed to the intervention with the change in a group that was not.
Concept
Measuring the effect of a policy with a simple before-after comparison is misleading, because the outcome may also have changed over the same period for other reasons (economic conditions, season, general trends). Comparing only the exposed and unexposed groups is misleading too, because the groups may differ from the start. Difference-in-differences (DiD) combines the two comparisons: it subtracts the change in the control group from the change in the treated group.
In the simplest 2×2 design the effect is computed as DiD = (Ȳ treated, after − Ȳ treated, before) − (Ȳ control, after − Ȳ control, before). The change in the control group serves as an estimate of what would have happened in the treated group without the intervention. The same result comes from a regression with group, period and a group × period interaction; the interaction coefficient is the DiD estimate. With many periods a model with unit and time fixed effects (TWFE) is used, while an event study estimates the effect separately for each period before and after the intervention.
The key assumption is parallel trends: without the intervention, the two groups' outcomes would have moved in parallel over time. It cannot be tested directly, but it can be supported by checking whether the groups followed similar trends before the intervention (a pre-trend test). Because a unit's measurements over time are correlated, standard errors should be clustered by unit; with few clusters these standard errors also lose reliability. If units adopt the intervention at different times (staggered adoption), the classical TWFE estimate can be biased.
When to use it
- When a policy, programme or event affected some units but not others, and both groups were measured before and after the intervention (a minimum wage rise, a new law, an environmental regulation applied in one region).
- If there is no pre-intervention measurement but the groups can be balanced on observed characteristics, use propensity score methods; if there is a valid instrument affecting who receives the intervention, instrumental variables regression can be used.
- If the treated group is a single unit and no single suitable control exists, use the synthetic control method; if units can be randomly assigned, a randomised experiment is preferred.
An everyday example
A district council lowers the speed limit on main roads from 50 to 30 km/h in January, while neighbouring districts keep theirs. In the district with the new limit, average monthly accidents fall from 40 to 30; in the neighbouring districts they fall from 38 to 36 because of winter conditions. A simple before-after comparison would suggest an effect of 10 accidents, yet accidents also fell by 2 in the neighbours. Difference-in-differences gives the effect as (30 − 40) − (36 − 38) = −8 accidents. This rests on the assumption that, without the new limit, accidents in both districts would have changed similarly; whether the trends were parallel in earlier years should be checked separately.
Parameters in MerQur
Covariates
Concept: Adds further variables that affect the outcome and may differ between groups or over time. It helps when the parallel-trends assumption is plausible only after accounting for them; variables that are themselves affected by the intervention should not be added as covariates.
Everyday example: If traffic volume grows at different rates in the two districts, monthly vehicle counts are added as a covariate. If a variable that the speed limit itself changes, such as average vehicle speed, were added, part of the effect would be wrongly removed.
What it does: Adds covariates to the model.
Effect on the result: The DiD estimate and SE can change.
On the validation data: kielmc log price DiD -0.0627 -> -0.1315 with covariates; base_did 2x2 4.993 -> 4.850 with x1.
Unit ID
Concept: Links the observations of the same unit across periods. Clustered standard errors and unit fixed effects rely on it; without a unit, only the simple 2×2 comparison can be made.
Everyday example: If the data consist of each district's monthly accident counts, the unit ID is the district. The model then accounts for consecutive months in the same district resembling one another.
What it does: Links the observations of the same unit across periods: the cluster of the clustered SE and the unit fixed effects in TWFE and the event study.
Effect on the result: With "None" only the 2x2 model can be fitted and the SE is HC1; TWFE without a unit gives an error ("unit ID required").
On the validation data: Card-Krueger 2x2 (DiD 2.7536): SE clustered by store 1.3066 (t(409)); without a unit HC1 SE 1.7955 (t(790)).
Post indicator and Intervention start period
Concept: Defines which observations belong to the post-intervention period; this sets the before and after periods and the relative periods in the event study.
Everyday example: If the speed limit changed in January 2024, January 2024 onwards counts as 'after' and earlier months as 'before'. If the wrong start month is entered, part of the effect is assigned to the wrong period and the estimate is distorted.
What it does: Marks the post-intervention periods. Without a post column it is derived from the start period (time >= start); with "automatic" start it is the first period where post is 1. For two-period data the option "time column: second period = post" can be used.
Effect on the result: Changes the relative periods, the pre-trend test and the average post effect. A warning is given if the start period and the post indicator conflict.
On the validation data: kielmc: post = y81 and start = 1981 give the same result (-11863.90). base_did: choosing start 8 (post starts at 6) changes the average post effect 3.907 -> 1.451 and the pre-trend F 0.94 -> 7.55.
Model (Classic 2×2 (group × period) / Two-way fixed effects (TWFE) / Event study (TWFE))
Concept: Determines which model estimates the effect. The classic 2×2 works with two groups and two periods; TWFE extends to many periods using fixed effects for units and periods; the event study estimates the effect for each relative period and tests the parallel-trends assumption with the pre-intervention coefficients.
Everyday example: The event study shows whether the two districts differed in the months before the speed limit changed, and how the gap opened month by month afterwards. If there were clear differences before the intervention too, the parallel-trends assumption is called into question.
What it does: 2x2 (y ~ G + P + G·P); TWFE (unit and time fixed effects, D = G·P); event study (relative period x G dummies, baseline relative period omitted; joint pre-trend F test and the equally weighted mean of the post coefficients).
Effect on the result: The estimated effect, SE and pre-trend test change. With staggered adoption TWFE can be biased; MerQur gives a "staggered adoption" warning.
On the validation data: organ donations: 2x2 -0.02246 (SE 0.00607); TWFE -0.02246 (0.00613); event mean -0.02134 (0.00565); pre-trend F(2, 26) = 4.117, p = 0.028. hospdd TWFE 0.84799.
SE clustered by unit
Concept: A standard error that allows a unit's errors to be correlated across periods. If this correlation is ignored in DiD studies the standard errors are too small; but with few clusters, clustered standard errors also lose reliability.
Everyday example: A month with more accidents than expected in one district may be linked to a high count the following month. Standard errors clustered by district account for this; with only 5 districts, the result should be interpreted with caution.
What it does: SE clustered by unit (G/(G - 1)·(n - 1)/(n - K), K does not count unit effects nested in clusters; t(G - 1)); HC1 (t(n - K)) when off.
Effect on the result: Only SE, test and CI change. A "few clusters" warning is written if G < 30.
On the validation data: Card-Krueger 2x2 SE 1.3066 (clustered) vs 1.7955 (HC1). With 27 states the warning "few clusters (G = 27)".
Reference relative period
Concept: The relative period whose coefficient is fixed at zero in the event study and against which other periods are compared; usually the period just before the intervention is chosen.
Everyday example: If December 2023 (one month before the intervention) is the reference period, each month's coefficient answers 'how much has the gap between the districts changed relative to December?'.
What it does: The period whose coefficient is fixed at 0 in the event study; the pre-trend test includes the periods before it.
Effect on the result: Event coefficients, the average post effect and the pre-trend test change; a period not in the data gives an error.
On the validation data: organ donations: ref -1: mean -0.02134, pre-trend F(2, 26) = 4.12; ref -2: -0.02764, F(1, 26) = 2.86; ref -3: -0.01840, no pre-trend test.
Confidence Level
Concept: Sets the confidence level of the interval for the intervention effect; the higher the level, the wider the interval.
Everyday example: The effect of the speed limit might lie between a reduction of 3 and 13 accidents a month with 95% confidence. If the interval included zero, the existence of an effect could not be asserted with confidence.
What it does: The interval level; critical value t(G - 1).
Effect on the result: Only the interval limits change.
On the validation data: organ donations TWFE: 90% [-0.03292, -0.01200]; 95% [-0.03506, -0.00986]; 99% [-0.03950, -0.00542].
Example on real data
Data: Kessler & Roth 2014, R causaldata
Variables: outcome Rate (organ donor registration rate), treatment group ca (California), time Quarter_Num (quarter), unit State, post indicator after
- Question: Did the organ donor registration policy that California started in the third quarter of 2011 change the registration rate (Rate) relative to other states?
- Assumption: The result rests on the parallel trends assumption; pre-trends should be tested with an event-study model. The number of clusters is small (G = 27); the reliability of clustered SEs is limited and the treated group contains a single state.
- Result: In California the mean rate fell from 0.2713 to 0.2628 (-0.0085) and in the control states it rose from 0.4449 to 0.4589 (+0.0139). The classic 2x2 DiD estimate is b = -0.0225 (SE clustered by state = 0.0061), t(26) = -3.70, p = .001, 95% CI [-0.0349, -0.0100]; the policy reduced the registration rate by about 2.2 points.
- Effect size: R² = .054; because the baseline difference between states (treated b = -0.174) is very large, the model explains little variance, but the DiD effect is precisely estimated.
- APA: Classic 2x2 difference-in-differences (N = 162, 27 states); DiD = -0.022, SE = 0.006, 95% CI [-0.035, -0.010], t(26) = -3.70, p = .001.
Related concepts
Event study designTwo-way fixed effects (TWFE)Parallel trends assumptionCluster-robust (clustered) standard errorsFixed and random effectsCovariateRobust (sandwich) standard errors (HC0-HC3)
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.