MerQuR+Statistics Glossary

Statistics Glossary → Propensity Score Analysis

Propensity Score Analysisanalysis

A method that estimates the average effect of a treatment in observational data without random assignment, by making treated and untreated individuals similar in their observed characteristics (matching, weighting, subclassification).

Concept

In experiments participants are randomly assigned to groups, so the groups are similar on average in every respect and a difference in outcome can be attributed to the treatment. In observational data, people or circumstances decide who is treated: stronger pupils may enrol in a course, more motivated people in a programme. The groups then differ from the start, and the raw difference in outcome mixes the effect of treatment with the initial differences between the groups.

The propensity score is the probability of being in the treatment group given a person's observed characteristics, and it is usually estimated by logistic regression. Treated and control individuals with the same propensity score are, on average, similar in their observed characteristics. Comparable groups are built from this score in three ways: finding for each treated individual the control with the closest score (matching), weighting each individual by the inverse of their probability of receiving the treatment they received (IPW), or dividing the sample into score strata and comparing within each (subclassification). Balance is then checked with standardised mean differences (SMD); |SMD| should usually be below 0.1.

The key assumption is that all variables that jointly affect treatment choice and outcome have been measured and included (no unobserved confounding); this cannot be tested. In addition, both treated and control individuals must exist at every level of the score (common support). The estimand should be stated clearly: the effect among the treated (ATT), in the whole sample (ATE) or among the untreated (ATC) answer different questions and can differ numerically.

When to use it

  • When the effect of a treatment, programme or intervention is to be studied with data lacking random assignment, and the variables that make the groups differ have been measured (comparing treatments from hospital records, voluntary training programmes).
  • If the groups were measured before and after the intervention, use difference-in-differences; if there is an instrument that affects treatment choice but not the outcome directly, instrumental variables regression can offer better protection against unobserved confounders.
  • If assignment was random, propensity scores are not needed; the groups are compared directly with a t-test, ANCOVA or regression.

An everyday example

At a secondary school some pupils attend an optional weekend course and others do not. At year end, course attendees score 12 points higher on average; but it is known that pupils who already had good marks and supportive families were more likely to attend. Propensity score analysis estimates each pupil's probability of attending from previous-year marks, family income and absences, and matches each attendee with a non-attendee whose probability is closest. If the difference between the matched groups falls to 4 points, the effect of the course on those who attended (ATT) is thought to be about a third of the raw difference; unmeasured differences in motivation could still affect the result.

Parameters in MerQur

Outcome variable (None / variable)

Concept: The outcome variable on which the treatment effect is measured. In propensity score methods the groups are first balanced independently of the outcome; the outcome is compared only once balance has been achieved.

Everyday example: In the course example the outcome is the year-end exam score. Matching and balance tables can also be examined without choosing an outcome, so that the matching can be improved without looking at the scores.

What it does: The outcome variable for effect estimation; with "None" only matching or weighting and balance are reported.

Effect on the result: With "None" there is no effect section or effect sentence; the balance and sample tables are the same.

Method (Nearest-neighbour matching / Inverse probability weighting (IPW) / Subclassification)

Concept: Determines how comparable groups are built. Matching finds a similar control for each treated individual; IPW weights each individual by the inverse of their probability of receiving the treatment they received, creating a pseudo-population; subclassification divides the sample into score strata and compares within each.

Everyday example: In matching, each attendee is compared with their most similar 'twin' pupil. In IPW, a pupil who seemed very likely to attend but did not receives a large weight, because few control pupils represent that kind of pupil.

What it does: The way the comparable group is built: nearest-neighbour matching, IPW and propensity-score subclassification (stratum weights; Hong 2010).

Effect on the result: Weights, the balance table, effective sample size, effect and SE type change (matching: HC1 clustered by matched set; IPW: M-estimation or HC0; subclassification and matching with replacement: HC3).

On the validation data: ATT: nearest neighbour 894.37 (SE 705.23; control ESS 185); IPW 1214.07 (798.15; ESS 99.82); subclassification (S = 5) 663.89 (845.28; ESS 103.35).

→ Propensity score matching (nearest neighbour, caliper)

Estimand (ATT / ATE / ATC)

Concept: Determines which average effect is estimated: ATT is the average effect among the treated, ATC among the untreated and ATE in the whole sample. If the treatment effect varies between individuals, the three values differ.

Everyday example: ATT answers 'what did attendees gain from the course?', ATC 'what would non-attendees have gained had they attended?', and ATE 'what would the average effect be if the course were offered to all pupils?'. If the course helps weaker pupils more, ATC may exceed ATT.

What it does: The average effect in the treated group (ATT), the whole sample (ATE) and the control group (ATC).

Effect on the result: The focal group in matching, the weight formula in IPW and subclassification, and the balance SMD denominator (ATT treated, ATC control, ATE pooled SD) change. ATE cannot be selected for nearest neighbour.

On the validation data: IPW: ATT 1214.07, ATE 224.68, ATC -186.92; subclassification: 663.89, -593.57, -650.06; nearest neighbour ATC -2620.74.

→ Average treatment effect (ATE, ATT, ATC)

Matching ratio (1:k)

Concept: Sets how many controls are matched to each treated individual. Using more matches reduces variance but can increase bias, because the matches become less similar.

Everyday example: With 1:1 matching each attendee gets one 'twin'. With 1:3 matching three matches are sought; if non-attendees are limited, the third matches are no longer very similar and the comparison can suffer.

What it does: k controls for each focal unit; first a 1st match for all focal units, then a 2nd.

Effect on the result: The number of matched controls and the effect change; when the control pool is exhausted increasing the ratio has no effect and a "not enough controls" warning is written.

On the validation data: 1:1 894.37 (185 matched controls); 1:2 220.29 (370); 1:3 and 1:10 -559.15 (all 429 controls).

→ Propensity score matching (nearest neighbour, caliper)

Matching with replacement

Concept: Allows a control individual to be matched to more than one treated individual. Matches become more similar and bias falls, but variance rises because the same control is used several times.

Everyday example: If only a few high-achieving pupils did not attend, matching with replacement lets them match with several high-achieving attendees. The matches are then more similar, but the result depends heavily on those few pupils.

What it does: A control can be matched to several treated units (less bias, more variance). Among controls with equal propensity scores the first in data order is chosen.

Effect on the result: The matches, the number of unique controls, the effect and the SE (HC3) change.

On the validation data: ATT 1991.6152735 (SE 942.8197), 82 unique controls.

→ Propensity score matching (nearest neighbour, caliper)

Caliper (SD units)

Concept: The largest allowed difference between the propensity scores of two matched individuals; pairs beyond this limit are not matched. A caliper prevents poor matches, but unmatched treated individuals are left out and the effect applies only to those matched.

Everyday example: With a narrow caliper only truly similar pupils are matched, but some attendees find no match. The estimated effect then applies not to all attendees but to those for whom a similar 'twin' exists.

What it does: Excludes pairs with |PS difference| > c x SD(PS) (c = 0: no caliper); the SD is that of the whole sample.

Effect on the result: The number of matched pairs and the effect change; a warning lists unmatched treated units and the effect applies only to the matched.

On the validation data: c = 0.05: 109 pairs, ATT 2277.59; c = 0.2: 113 pairs, 1571.74; c = 1.0: 139 pairs, 955.44.

→ Propensity score matching (nearest neighbour, caliper)

Number of subclasses

Concept: Sets how many strata the sample is divided into by propensity score in subclassification. More strata make the groups within each stratum more similar, but smaller strata make the estimates more variable.

Everyday example: The pupils are divided into 5 strata by their probability of attending, and attendees and non-attendees are compared within each. With 20 strata balance improves, but some strata may be left with very few pupils.

What it does: S strata by propensity-score quantiles in subclassification (quantiles of the treated group for ATT).

Effect on the result: Balance, ESS and the effect change.

On the validation data: ATT: S = 2: 341.17 (PS SMD 0.70); S = 5: 663.89 (0.131); S = 10: 813.15 (0.034); S = 20: 1176.62 (0.005); control ESS 147.7 -> 61.5.

→ Propensity score subclassification (stratification)

Stabilized weights

Concept: Multiplies the IPW weights by the overall proportion treated, scaling down extreme weights; the mean weight moves towards 1 and the weights become easier to interpret.

Everyday example: A pupil with a 0.02 probability of attending who nonetheless attended gets a weight of 50. Stabilised weights scale this down, so the saved weights are more in keeping with the sample size.

What it does: Multiplies ATE weights by the marginal treatment proportion (reduces the scale of extreme weights).

Effect on the result: In the saturated effect model the estimate, SE, balance and ESS do not change; only the saved weight column changes. Not applied for ATT and ATC.

On the validation data: ATE 224.68 (SE 876.19) in both cases; NHEFS 3.4405 (0.5255).

→ Inverse probability weighting (IPW)

IPW standard error (M-estimation / Robust (HC0))

Concept: Determines how the standard error of the IPW effect is computed. M-estimation accounts for the propensity score itself being estimated from the data with uncertainty; robust HC0 treats the weights as known constants.

Everyday example: Because the probabilities of attending are estimated from a logistic model, they are not exact themselves. M-estimation carries this uncertainty into the standard error of the course effect.

What it does: M-estimation is a stacked sandwich that includes the estimation uncertainty of the propensity-score model (z); HC0 treats the weights as fixed (t(n - k)).

Effect on the result: Only SE, test and CI change.

On the validation data: ATT SE 798.15 (M-estimation) vs 824.05 (HC0); ATE 876.19 vs 909.48; NHEFS ATE 0.4871 vs 0.5255.

→ Inverse probability weighting (IPW)

Confidence Level

Concept: Sets the confidence level of the intervals for the treatment effect and the propensity score model; the higher the level, the wider the interval.

Everyday example: The effect of the course might lie between 1 and 7 points with 95% confidence. If the interval included 0, it could not be confidently concluded that the course works.

What it does: Sets the level of the effect and PS-model confidence intervals.

Effect on the result: Only the CI limits change (including those of the PS model).

On the validation data: Nearest-neighbour ATT: 90% [-268.6, 2057.3]; 95% [-492.4, 2281.2]; 99% [-931.7, 2720.4].

→ Confidence level and confidence interval

Covariate adjustment in the outcome model (Additional options)

Concept: Adds the covariates to the effect model as well, after matching or weighting. In this doubly robust approach the effect estimate stays consistent if either the propensity score model or the outcome model is correctly specified, and remaining small imbalances are corrected.

Everyday example: If, after matching, a small difference in previous-year marks remains between attendees and their matches, marks are added to the effect model and this residual difference is removed from the course effect.

What it does: Adds the covariates to the weighted effect model (y ~ treatment + X; doubly robust approach).

Effect on the result: The effect estimate and SE change; the balance table does not.

On the validation data: Nearest-neighbour ATT 894.37 -> 1344.94; IPW ATE 224.68 -> 712.74; subclassification 663.89 -> 953.88.

→ Doubly robust estimation

Save score/weight columns (Additional options)

Concept: Adds each observation's propensity score, weight, match ID and stratum to the dataset as new columns; the results do not change, but the columns can be used in later analyses or for checking.

Everyday example: If each pupil's probability of attending and the pupil they were matched with are added to the table, teachers can check one by one whether the matches look sensible.

What it does: Adds ps_score, ps_weight (IPW: ps_ipw_weight), ps_match_id and ps_subclass columns to the dataset.

Effect on the result: Does not change the results.

Balance threshold (|SMD|)

Concept: The standardised mean difference (SMD) is the difference between the two groups' means on a variable divided by a standard deviation. Variables exceeding this threshold are flagged as imbalanced; a common criterion is |SMD| < 0.1.

Everyday example: If, after matching, attendees and their matches differ in previous-year marks by SMD = 0.05, the groups are similar on that variable. If family income still has SMD = 0.30, income remains imbalanced and the propensity model or the matching should be improved.

What it does: Variables whose |SMD| exceeds this threshold are flagged and counted as imbalanced.

Effect on the result: Only the text changes; the numeric results are the same.

On the validation data: With threshold 0.05 age, nodegree and re74 are also flagged; with 0.5 the educ and race levels are not flagged.

→ Standardized mean difference (SMD) and covariate balance

Example on real data

Data: LaLonde 1986, R MatchIt

Variables: treatment treat (job training programme), outcome re78 (1978 earnings), covariates age, education, race, marital status, degree and 1974-1975 earnings

  • Question: Does participation in a job training programme (treat) change 1978 earnings (re78) once observed covariates are balanced (effect in the treated, ATT)?
  • Assumption: The probability of participation is significantly related to race (hispan b = -2.08, white b = -3.07), education (b = 0.16), marital status (b = -0.83), degree (b = 0.71) and 1974 earnings. With 1:1 nearest-neighbour matching without replacement, 185 treated units were matched to 185 controls; the largest |SMD| fell from 0.72 to 0.37, but 4 of 9 terms remain imbalanced with |SMD| >= 0.1. The causal interpretation rests on the assumption of no unobserved confounding.
  • Result: ATT = 894.37 dollars (SE = 705.23, clustered by matched set), t(368) = 1.27, p = .206, 95% CI [-492.42, 2281.15]; the programme effect is not statistically significant.
  • Effect size: Because of the remaining imbalance, adding interaction or squared terms to the propensity-score model, using a caliper or trying IPW is recommended.
  • APA: Propensity score matching (1:1, without replacement; treated n = 185, control n = 185); ATT = 894.37, SE = 705.23, 95% CI [-492.42, 2281.15], t(368) = 1.27, p = .206.

Related concepts

Propensity score matching (nearest neighbour, caliper)Inverse probability weighting (IPW)Propensity score subclassification (stratification)Average treatment effect (ATE, ATT, ATC)Standardized mean difference (SMD) and covariate balanceDoubly robust estimationFolded F test (variance ratio)Robust (sandwich) standard errors (HC0-HC3)

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.