MerQuR+Statistics Glossary

Statistics Glossary → Differential item functioning (DIF)

Differential item functioning (DIF)concept

The situation in which members of different groups (e.g. women and men) who stand at the same level of the trait being measured have different probabilities of answering or endorsing an item correctly.

An item being answered differently on average by two groups is not DIF in itself; the groups may genuinely differ on the trait (this is called impact). In DIF, the comparison is made between people matched on the trait being measured. In the Mantel-Haenszel method, people are stratified by total score, a group × correct/incorrect 2×2 table is formed in each stratum, and a common odds ratio across strata (α_MH) is computed. On the ETS scale, Δ_MH = -2.35·ln(α_MH) is used and items are classified as having negligible (A), moderate (B) or large (C) DIF.

In uniform DIF one group is advantaged at every ability level; in non-uniform DIF the advantage changes with ability level, which Mantel-Haenszel does not capture well; logistic regression or item response theory methods are better suited to that case. DIF does not automatically prove that an item is biased; content experts should judge whether the difference stems from a cause unrelated to the construct being measured (e.g. cultural knowledge, language). If the total score itself contains DIF items, the matching can be distorted, so purification steps may be used.

An everyday example

One problem in a maths exam is built around baseball statistics. If, even among pupils of the same maths ability, those raised in a country where baseball is familiar answer it correctly more often, the question is measuring cultural knowledge as well as maths ability. The item shows DIF and should be considered for removal or rewriting.

In MerQur

In Item Response Theory (IRT / Rasch) the 'DIF group' parameter gives, for a two-level grouping variable, a Mantel-Haenszel DIF test stratified by total score together with the ETS A/B/C classification.

Analyses where this concept appears

Item Response Theory (IRT / Rasch)

Related concepts

Item difficulty, discrimination and guessing (1PL, 2PL, 3PL)Mantel-Haenszel common odds ratio