MerQuR+Statistics Glossary

Statistics Glossary → Item Response Theory (IRT / Rasch)

Item Response Theory (IRT / Rasch)analysis

A family of models that ties the probability of each response to the relation between person and item, estimating each item's difficulty and discrimination and each person's level on the measured trait on one common scale.

Concept

In classical test theory a person's score is the number of correct answers, and item difficulty depends on the group taking the test: the same item looks easy in a strong class and hard in a weak one. Item response theory (IRT) solves this by modelling each response as a probabilistic relation between the person's latent ability (θ) and the item's characteristics. Item difficulties and person levels then sit on the same scale, it becomes visible at which ability levels each item is most informative, and people measured with different sets of items can be compared.

In the simplest model, the Rasch model, the probability of a correct answer depends only on the difference between ability and item difficulty (b): P(correct) = exp(θ − b) / (1 + exp(θ − b)). A person whose ability equals the item's difficulty has a 50% chance of answering correctly. The two-parameter model (2PL) adds a discrimination (a) for each item; the steepness of the curve shows how sharply the item separates low- and high-ability people. The three-parameter model (3PL) also allows for answering correctly by guessing (c). For polytomous items such as Likert scales, extensions such as the partial credit, rating scale and graded response models are used.

The key assumptions are unidimensionality (the items measure one common trait) and local independence (holding ability fixed, responses to different items are independent). Item and model fit statistics show where the model conflicts with the data; differential item functioning (DIF) analysis tests whether two groups of equal ability answer an item correctly with different probabilities, i.e. whether the item is biased against one group. Reliable parameter estimation usually requires a few hundred participants, and more complex models need larger samples.

When to use it

  • When developing an achievement test or attitude scale and you want item difficulties, discriminations and the ability range in which items measure well; in computerised adaptive testing and item-bank work.
  • If only the internal consistency of the total score matters, classical item analysis and Cronbach's alpha are a simpler start.
  • If the items measure more than one distinct dimension, consider multidimensional IRT or confirmatory factor analysis with categorical indicators instead of unidimensional IRT.

An everyday example

A district gives a 20-question maths test to 600 eighth-graders. IRT places each question's difficulty on the ability scale: a question on fractions might have a difficulty of −1 (most pupils solve it) and a probability question +2 (only the strongest solve it). If two pupils have the same total score but one got the hard questions right and the other the easy ones, the model takes that into account too. DIF analysis tests, for instance, whether a long word problem is harder for girls than for boys of the same mathematical ability.

Parameters in MerQur

Model (Rasch, 2PL, 3PL, Partial Credit Model (PCM), Rating Scale Model (RSM), Graded Response Model (GRM), Generalized Partial Credit (GPCM))

Concept: IRT models differ in how many item characteristics they use to explain response probabilities: Rasch uses only difficulty, 2PL adds discrimination, and 3PL adds a guessing probability. For polytomous items the partial credit and rating scale models belong to the Rasch family, while the graded response and generalised partial credit models also estimate item-specific discrimination.

Everyday example: If the maths test is multiple choice, even a pupil who knows nothing can tick one of five options at random and be right about 20% of the time; only the 3PL model accounts for this. A likelihood ratio test and BIC decide which model is adequate.

What it does: Selects Rasch/1PL, 2PL, 3PL for dichotomous items and PCM, RSM, GRM, GPCM for polytomous items. With dichotomous data PCM/RSM become Rasch and GRM/GPCM become 2PL; 3PL gives an error for polytomous items. The likelihood ratio test and BIC are used for comparison (de Ayala, 2009).

Effect on the result: Item parameters, log-likelihood, person scores and fit.

On the validation data: LSAT7: Rasch log-likelihood -2664.901 (6 parameters), 2PL -2658.805 (10), 3PL -2658.054 (15). Science: RSM -1636.908, PCM -1619.274, GPCM -1612.682, GRM -1608.869.

→ Item difficulty, discrimination and guessing (1PL, 2PL, 3PL)

Estimation (MML, CML)

Concept: Because people's abilities are unknown, item parameters can be estimated in two ways. Marginal maximum likelihood (MML) assumes abilities follow a given distribution in the population (usually normal) and averages over it; conditional maximum likelihood (CML) exploits the fact that in the Rasch model the total score carries the information about ability, so it makes no assumption about the distribution of persons.

Everyday example: If the test was given only in high-achieving schools, the pupils' ability distribution may not be normal. CML estimates item difficulties independently of that distribution, so it is a sturdier option in such a case.

What it does: MML is marginal maximum likelihood (EM, 61 nodes on [-6, 6] and a Newton polish); CML is conditional maximum likelihood (dichotomous Rasch only, sum b = 0) and gives item estimates independent of the sample distribution (Andersen, 1972). If CML is chosen for polytomous items, MML is used with a warning.

Effect on the result: The selector is enabled for Rasch/PCM; CML is applied only to dichotomous data (with dichotomous data a PCM choice reverts to Rasch); with polytomous data MML is used and a warning is written. Item parameters and the conditional log-likelihood change.

On the validation data: LSAT7 CML: b1 = -0.5415 (SE 0.0792), conditional log-likelihood -1182.700; theta for raw scores 1.4 = -1.486, -0.443, 0.436, 1.487. Theta is undefined for extreme scores.

→ Marginal and conditional maximum likelihood (MML, CML)

Person scores (EAP, MAP, ML, WLE)

Concept: The person score (θ) is a person's position on the ability scale and can be estimated from the response pattern in several ways. Maximum likelihood (ML) relies only on the responses and goes to infinity for someone with all answers right or all wrong; EAP and MAP also use information about the population distribution and therefore always give finite values; WLE is a correction that reduces ML's bias.

Everyday example: A pupil who gets all 20 questions right has an 'infinite' ability under ML, because the test contains no question hard enough to challenge them. EAP gives this pupil a high but finite score and shows its uncertainty through the standard error.

What it does: Sets the theta estimation method and its standard error. EAP is finite and reliable (default); WLE is bias-corrected (Warm, 1989); ML is undefined for all-0 and all-1 patterns (±infinity).

Effect on the result: Person scores, SE and empirical reliability.

On the validation data: LSAT7 Rasch, pattern 00011: EAP -0.934 (SE 0.707), MAP -0.952 (0.691), ML -1.770 (0.948), WLE -1.707 (0.871); empirical reliability 0.432, 0.422, 0.420, 0.519 respectively. GRM (Science) ML empirical reliability 0.7162.

→ Person ability (θ) estimation (EAP, MAP, ML, WLE)

Standard errors (Observed information (Oakes), Cross-product (crossprod), Sandwich (robust))

Concept: The standard error of an item parameter shows how precise the estimate is, and it can be computed in different ways. If the model is correctly specified, errors based on the observed information matrix are appropriate; if you suspect the model does not quite fit, sandwich (robust) standard errors are more trustworthy.

Everyday example: If the probability question's difficulty is 2.0 ± 0.15, it has been estimated quite precisely. If some pupils are suspected of copying, the model may not fit perfectly; sandwich errors then avoid overstating the precision.

What it does: Sets the source of the item parameter standard errors; the SE of IRT parameters are given by the delta method. Observed information is used if the model is correctly specified, sandwich under suspected misspecification (White, 1982).

Effect on the result: Only SE and confidence intervals.

On the validation data: LSAT7 2PL, Item.3 slope: Oakes 0.3211, cross-product 0.3215, sandwich 0.3297. In 3PL, when the guessing parameter sits on the boundary (g = 0), the SE of g is left empty with a warning (LSAT7 Item.5 b SE 0.419).

→ Robust (sandwich) standard errors (HC0-HC3)

Outputs: Item fit (infit/outfit, S-X2)

Concept: Item fit shows how closely the actual responses to each item match the probabilities the model predicts. Infit and outfit mean squares near 1 indicate good fit; values well above 1 flag unexpected responses, values below 1 overly predictable ones.

Everyday example: If several very strong maths pupils got an easy fraction question wrong, perhaps because it was ambiguously worded, its outfit value rises. That is a sign the question needs a second look.

What it does: Gives the Wright and Masters infit/outfit MNSQ and ZSTD (with EAP theta) and the Orlando-Thissen S-X2 statistic.

Effect on the result: Additional table only.

On the validation data: LSAT7 Rasch, Item.1: outfit 0.744, infit 0.939, S-X2(2) = 5.593 (p = 0.061). For polytomous items S-X2 collapses cells with small expected frequencies: Science GRM Comfort 14.27 (df 9).

→ Item fit (infit, outfit, S-X²)

Outputs: Model fit (M2 / C2, RMSEA, SRMSR, CFI/TLI)

Concept: Model fit summarises how well the model reproduces response patterns across the whole test. Limited-information tests (M2), based on how often pairs of items are answered together, and indices such as RMSEA, SRMSR and CFI show the overall level of misfit.

Everyday example: Even if every question looks acceptable on its own, overall fit weakens if, say, the geometry questions are more strongly related to each other than the model expects. That hints that the test may not really be unidimensional.

What it does: Gives the limited-information fit test (M2 for dichotomous, C2 for polytomous items), RMSEA (+ 90% confidence interval), SRMSR and TLI/CFI against the null model; complete data are required.

Effect on the result: Additional table only.

On the validation data: LSAT7 2PL: M2(5) = 11.938, p = 0.036, RMSEA 0.037. Polytomous C2: GPCM 19.039, PCM 31.694, RSM 65.393.

→ Model fit indices (χ², CFI, TLI, RMSEA, SRMR)

Outputs: Model comparison (AIC/BIC, nested likelihood ratio test)

Concept: Fits several IRT models to the same data and compares which is more suitable. Information criteria (AIC, BIC) balance fit against the number of parameters; for nested models, the likelihood ratio test checks whether the extra parameters improve fit significantly.

Everyday example: If both Rasch and 2PL are fitted to the maths test and 2PL has the higher BIC, giving each question its own discrimination does not improve fit enough to justify the added complexity, and the simpler Rasch model is preferred.

What it does: Also fits the family models and gives AIC/BIC and the likelihood ratio test for nested pairs.

Effect on the result: Additional table only.

On the validation data: LSAT7: chi-square(4) = 12.19 between Rasch and 2PL, chi-square(5) = 1.50 between 2PL and 3PL.

→ Likelihood ratio test (LRT)

Outputs: Unidimensionality (eigenvalue ratio)

Concept: Unidimensionality is the assumption that the items measure a single common trait. If the first eigenvalue of the inter-item correlation matrix is much larger than the second, one dominant dimension is assumed; a ratio of about 3 or more is a strong sign.

Everyday example: If the 20 questions really measure one thing, such as 'mathematical ability', the first eigenvalue dominates. If half of them depend on arithmetic and half on verbal reading skill, the second eigenvalue grows too and the ratio falls.

What it does: Gives the eigenvalues of the tetrachoric/polychoric correlation matrix and the first/second eigenvalue ratio (>= 3 indicates strong unidimensionality).

Effect on the result: Additional table only.

On the validation data: LSAT7 2.044 / 0.921 (ratio 2.22). IrtUni 3.1184 / 0.9936.

→ Unidimensionality

Outputs: Save person scores (theta, SE) to the dataset

Concept: Adds each person's estimated ability score and its standard error to the dataset as new columns. These scores can be used in later analyses (e.g. group comparisons, regression) instead of the raw total score.

Everyday example: Once each pupil's θ score is in the dataset, schools can be compared on scores that account for question difficulty rather than on the raw number correct.

What it does: Adds theta and SE from the selected method to the dataset as the IRT_THETA and IRT_SE columns.

Effect on the result: Only adds columns to the dataset; the text does not change.

On the validation data: 2 columns in LSAT7 (1000 values).

→ Person ability (θ) estimation (EAP, MAP, ML, WLE)

DIF group

Concept: Differential item functioning (DIF) occurs when two groups at the same ability level answer an item correctly with different probabilities. The Mantel-Haenszel method stratifies people by total score and compares the two groups' success rates within each stratum; the ETS classes show whether the difference is negligible (A), moderate (B) or large (C).

Everyday example: If a ratio question built around a football score is answered correctly more often by boys than by girls with the same total score, the question may be measuring something besides maths. DIF analysis catches such items so they can be reviewed.

What it does: Performs the Mantel-Haenszel DIF test stratified by total score for a two-level group variable (continuity-corrected chi-square, alpha_MH, Delta_MH = -2.35 ln alpha, ETS A/B/C classes). The first level is the base (comparison) group, the second the focal group; works for dichotomous items only.

Effect on the result: DIF table only.

On the validation data: verbal, Gender (focal 1): S1wantCurse chi-square = 1.708, alpha = 1.700, Delta = -1.248; 24 items are tested, and the common odds ratio can also be read as 1/alpha. 9 items in ETS class C.

→ Differential item functioning (DIF)

Item for category curves (no.)

Concept: Category response curves show, for a polytomous item, the probability of choosing each response category at each trait level. Ordered, well-separated curves indicate that the categories work well.

Everyday example: For the maths-anxiety item 'I get tense before exams', the chart shows that pupils with low anxiety mostly choose 'never' and those with high anxiety 'always', and whether the 'sometimes' category has a range of its own in between.

What it does: Selects the item shown in the category response curves chart.

Effect on the result: Chart only; the numeric output does not change.

On the validation data: The text does not change; only the chart differs.

Example on real data

Data: verbal aggression data, difR package

Variables: items: 24 dichotomous verbal aggression items (S1wantCurse ... S4DoShout)

  • Question: Do the 24 dichotomous items of the verbal aggression scale fit a unidimensional Rasch model, and how are the item difficulties ordered?
  • Model: Rasch (1PL), marginal maximum likelihood (EM); N = 316, 25 parameters; logLik = -4036.905, AIC = 8123.81, BIC = 8217.70; latent variance σ² = 1.919 (SE = 0.196).
  • Item difficulties: The b values range from -1.75 (S2WantCurse, most easily endorsed) to 2.98 (S3DoShout, hardest); "want" items are endorsed more easily than "do" items, and "curse" more easily than "shout".
  • Model fit: M2(275) = 1423.34, p < .001, RMSEA = .115, SRMSR = .106, CFI = .783; the limited-information fit is weak.
  • Item fit: Only S3DoShout lies outside the 0.5-1.5 infit/outfit range (outfit = 1.840). In the model comparison the lowest BIC is for the Rasch model (8217.70; 2PL 8309.13).
  • Reliability: Marginal reliability = .871, empirical (EAP) reliability = .874; the scale discriminates persons well.
  • APA: A Rasch model was fitted to the 24 items (N = 316; MML-EM); M2(275) = 1423.34, p < .001, RMSEA = .115, SRMSR = .106, CFI = .783; marginal reliability = .871.

Related concepts

Likelihood ratio test (LRT)Person ability (θ) estimation (EAP, MAP, ML, WLE)Item difficulty, discrimination and guessing (1PL, 2PL, 3PL)UnidimensionalityDifferential item functioning (DIF)Marginal and conditional maximum likelihood (MML, CML)Item fit (infit, outfit, S-X²)Model fit indices (χ², CFI, TLI, RMSEA, SRMR)

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.