MerQuR+Statistics Glossary

Statistics Glossary → MR × MR Crosstab

MR × MR Crosstabanalysis

An analysis that examines, cell by cell, the association between the options of two separate multiple-response questions by looking at how often each pair of options is chosen together.

Concept

In some studies both questions are multiple-response: for example 'which drinks do you buy?' and 'which foods do you buy?'. The question is whether people who tick a particular option in one question also tick a particular option in the other more often. Since each person may give several answers to both questions, an ordinary crosstab with a single chi-square test cannot be used.

The analysis pairs every option of the first question with every option of the second and builds a 2x2 table of 'A selected? × B selected?' for each pair. For every table it computes a chi-square test, phi and the odds ratio for the strength of association, the Jaccard coefficient for similarity of co-selection, and lift, which shows how many times more often than chance the pair occurs: lift = P(A and B) / (P(A) · P(B)). Lift = 1 means the two options are chosen independently; values above 1 mean they tend to be chosen together.

Because the number of option pairs grows quickly (e.g. 3 × 4 = 12 cells), a multiple-comparison adjustment is needed; the simultaneous pairwise marginal independence (SPMI) test checks all cells together and gives an overall decision. The cell tests come from the same people and are not independent of each other. A significant association shows co-occurrence; it does not show that one option causes the other.

When to use it

  • To find out which option pairs of two multiple-response survey questions tend to be ticked together (e.g. activities done and equipment used, symptoms and treatments sought).
  • If only one question is multiple-response and the other is a single-response grouping variable, use the MR × categorical crosstab.
  • To explore co-occurrence patterns among many binary variables, association rule analysis or multiple correspondence analysis gives a broader view.

An everyday example

The café asks its 200 customers both 'which drinks do you buy?' and 'which snacks do you buy?'. 60% ticked coffee and 40% cake; if the two choices were independent, about 24% of customers (48 people) would be expected to choose both, but 64 people ticked both. Lift = 64 / 48 ≈ 1.33: coffee buyers are about a third more likely than chance to buy cake. On this basis the café might consider a 'coffee + cake' deal.

Parameters in MerQur

MR Set 1 (row) and MR Set 2 (column)

Concept: The two multiple-response question sets whose association is examined; the options of the first form the rows and those of the second the columns. Each row-column intersection is a separate 2x2 table, and the number of cells is the product of the two sets' numbers of options.

Everyday example: In the café example the first set is the drinks (tea, coffee, ayran) and the second the snacks (simit, cake, sandwich, biscuit). The 3 × 4 = 12 drink-snack pairs are then examined one by one.

What it does: The two multiple-response sets whose intersection is examined; the same set cannot be selected twice.

Effect on the result: I x J cells are formed; in each cell the (W_i, Y_j) 2x2 table is built.

On the validation data: The cell counts are as in Table 1 (w1 x y1 = 27 ... w3 x y4 = 0). Swapping the row and column sets gives the same χ², p and (transposed) OR. Sets built from delimited text give the same result as binary sets.

→ Multiple-response set (MR set)

p adjustment (No adjustment, Bonferroni, Holm, Benjamini-Hochberg (FDR))

Concept: Because a separate test is run for each cell, the chance that some cells come out significant by luck grows even when there is no real association. Bonferroni and Holm limit the probability of at least one false alarm; Benjamini-Hochberg controls the share of false alarms among the cells declared significant.

Everyday example: With 12 drink-snack pairs at the 5% level, on average half a cell would come out significant by chance even with no association at all. The adjustment guards against the coffee-cake link being that kind of fluke.

What it does: Applies a multiple-comparison adjustment to the χ² p-values of the 12 cells.

Effect on the result: The adjusted p and the list of significant cells change.

On the validation data: With Bonferroni the significant cells are w1 x y3 (0.00187), w2 x y3 (0.00757), w3 x y1 (0.00222). Holm gives the same three cells, Benjamini-Hochberg five (w2 x y1 0.0251 and w3 x y3 0.0233 are added).

→ Multiple comparisons and the familywise error rate (FWER)

Confidence Level (%90, %95, %99, Other (custom))

Concept: Sets the threshold for the 'significant' decision (α = 1 − confidence level) and the width of the confidence intervals for the cell odds ratios. The chi-square values and p values do not change.

Everyday example: If the 95% confidence interval for the coffee-cake odds ratio stays above 1, the association is reliable; at 99% confidence the interval widens and may approach 1.

What it does: The decision threshold alpha = 1 - confidence level and the level of the OR Woolf interval.

Effect on the result: Interval limits, significance marks and the on-screen threshold note (e.g. "p < 0.1") change; χ² and p do not change.

On the validation data: w1 x y1 OR 2.2023: 95% [1.0842, 4.4731], 90% [1.215, 3.991], 99% [0.8678, 5.5887], 99.9% [0.6702, 7.2365]. At 90%, w1 x y2 (p = 0.087) counts as significant.

→ Confidence level and confidence interval

Additional tests: Cell association measures

Concept: Measures describing the strength of the association in each cell from different angles: phi is the correlation between the two options (from −1 to 1), the odds ratio is how many times higher the odds of choosing the second option are among those choosing the first, Jaccard is the similarity of co-selection (those choosing both / those choosing at least one), and lift is how many times more often than chance the pair is chosen together.

Everyday example: For coffee and cake, if 64 people chose both and 136 chose at least one, Jaccard = 64/136 ≈ 0.47; a lift of about 1.33 means the pair is chosen together a third more often than chance would predict. Lift is an intuitive measure also used in market-basket analysis.

What it does: For each cell gives the signed phi, OR and Woolf interval (Haldane-Anscombe + 0.5 for a zero cell), Jaccard a/(a + b + c) and lift (a·n/(r1·c1)).

Effect on the result: Additional columns only.

On the validation data: w1 x y1 phi 0.1330, Jaccard 0.1731, lift 1.317; w3 x y1 OR 10.266 [2.343, 44.979]; w3 x y4 (a = 0) Haldane OR 0.4229 [0.0243, 7.3617]. For a zero cell the OR is computed by adding 0.5 to the cells (Haldane correction).

→ Odds ratio (OR)

Additional tests: Fisher's exact test

Concept: Gives Fisher's test, based on exact probabilities, for each cell's 2x2 table instead of the chi-square approximation. It is more reliable for cells involving rarely chosen options, where expected counts are small.

Everyday example: If only 15 customers ticked ayran and only 10 ticked biscuits, the expected count in the ayran-biscuit cell falls below 1. Fisher's p is more trustworthy than the chi-square p for that cell.

What it does: Gives the two-sided Fisher p-value for the 2x2 table of each cell.

Effect on the result: An additional column only.

On the validation data: 0.02758 for w1 x y1, 5.75e-05 for w1 x y3.

→ Fisher's exact test

Additional tests: Overall SPMI test

Concept: Takes all cells together and tests in one go whether there is any association between the two multiple-response questions. It gives an overall answer before moving on to the cell-by-cell examination.

Everyday example: The café first asks 'are drink preferences and snack preferences associated overall?'. If the SPMI test is significant, the cell tests show which pairs carry the association.

What it does: The Agresti and Liu (1999) simultaneous pairwise marginal independence test: X²_S = sum of the cell χ² values; Bonferroni p = min(1, IJ x smallest p).

Effect on the result: An additional overall test row. The Rao-Scott (X²_S.adj = 36.17) and bootstrap versions of MRCV are not in MerQur.

On the validation data: X²_S = 64.825, Bonferroni p = 0.001876; no constant is added to tables with a zero cell (w3 x y4).

→ Multiple marginal independence tests (MMI, SPMI)

Example on real data

Data: Bilder and Loughin (2007), R MRCV farmer2

Variables: row set test (contaminants tested for in water), column set depo (manure storage methods)

  • Question: Is there an association between the contaminants farmers test their water for (salt, phosphorous, nitrogen) and the manure storage methods they use (holding tank, natural drainage, pit, lagoon)? Both variables are multiple response.
  • Method: For each (contaminant, storage) pair a 2x2 table is built from all 279 cases and an uncorrected Pearson χ²(1) is computed (3 x 4 = 12 cells).
  • Result: 6 of the 12 pairs have p < .05: nitrogen x natural drainage (χ² = 14.30, p = .0002), salt x lagoon (13.98, p = .0002), phosphorous x natural drainage (11.68, p = .0006), salt x natural drainage (7.08, p = .008), phosphorous x lagoon (6.56, p = .010), nitrogen x lagoon (4.93, p = .026).
  • Direction: Lagoon users test for contaminants more often than expected (e.g. salt: 19 observed, 10.8 expected), natural drainage users less often (e.g. nitrogen: 2 observed, 12.2 expected).
  • Limitation: 25% of the cells have an expected value below 5 and 12 tests are run at once. In the Bonferroni-adjusted overall SPMI test the overall association is still significant (p = .002); at cell level nitrogen x natural drainage, salt x lagoon and phosphorous x natural drainage remain significant.
  • APA: A significant association was found between nitrogen testing and natural drainage use, χ²(1, N = 279) = 14.30, p < .001.

Related concepts

Post hoc tests (pairwise comparisons)Multiple marginal independence tests (MMI, SPMI)Fisher's exact testMultiple-response set (MR set)Odds ratio (OR)Multiple comparisons and the familywise error rate (FWER)Phi (φ) coefficientExact test and exact p-value

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.