MerQuR+Statistics Glossary

Statistics Glossary → Fisher's Exact Test

Fisher's Exact Testanalysis

A test of whether two categorical variables are associated that uses exact probabilities instead of the chi-square approximation, especially for 2x2 tables with small samples.

Concept

The chi-square test relies on an approximation that works well when the table counts are large enough; when expected counts are small (e.g. below 5) the approximation breaks down and the p value can mislead. Fisher's exact test answers the same question, whether two variables are independent, without any approximation.

The test treats the row and column totals as fixed and, assuming independence, computes from the hypergeometric distribution the probability of every table that could arise with those totals. The p value is the sum of the probabilities of all tables as 'unusual' as the observed one or more so. The direction and size of the association are summarised by the odds ratio (OR): OR = 1 means no association, and values above or below 1 show its direction.

Because counts are whole numbers, the exact test tends to be conservative: the actual type I error rate usually stays below the chosen α. The mid-p and unconditional exact tests such as Barnard's or Boschloo's reduce this conservatism. In small samples the confidence interval for the OR is wide; even a significant p value says little precisely about how large the effect is.

When to use it

  • In a 2x2 table when the sample is small or one or more cells have an expected count below 5.
  • With a large sample the chi-square test of independence gives practically the same answer; when two measurements on the same people are compared, use the McNemar test.
  • When both group sizes were fixed by design (e.g. 10 patients per arm), Barnard's or Boschloo's test is a more powerful alternative; for tables with more than two levels the Fisher-Freeman-Halton test is used.

An everyday example

Suppose a small clinic gives a new wound dressing to 10 of 20 surgical patients and the old dressing to the other 10. One patient on the new dressing and 6 on the old develop a wound infection. With only 7 infections in total, some cells have expected counts below 5 and the chi-square approximation cannot be trusted. Fisher's exact test computes how likely it would be for 7 infections to split this unevenly if the two dressings worked equally well; the two-sided p comes out at about 0.057. The difference is striking for such a small sample, but more patients are needed for a firm conclusion.

Parameters in MerQur

Column 1 (row) and Column 2 (column)

Concept: The two categorical variables forming the rows and columns of the table. The two-sided p is unaffected by swapping rows and columns, but which group the odds ratio and risks compare against which depends on this choice.

Everyday example: In the clinic example, dressing type (new / old) goes in the rows and infection (yes / no) in the columns. The relative risk then answers 'the infection risk with the new dressing is how many times that with the old?'

What it does: Selects the row and column variables of the 2x2 (or RxC) table; levels are sorted numeric ascending, text alphabetical. With more than two levels the Fisher-Freeman-Halton Monte Carlo test is used. Level labels such as "None" and "NA" are kept.

Effect on the result: The two-sided p and phi do not change; the OR and one-sided p direction, RR, and RD depend on the row-column roles.

On the validation data: In Convictions, swapping row and column leaves p and OR the same. hsb2 schtyp x female p = .849. Rabbits Penicillin 1 (with Delay = "None" rows) p = .181818.

Direction (Two-sided, Right-tailed (>), Left-tailed (<))

Concept: States whether the research question looks for any association (two-sided) or for one in a pre-specified direction (OR > 1 or OR < 1). A one-sided test is more powerful in that direction, but the direction must be chosen without looking at the data.

Everyday example: If the clinic will adopt the new dressing only if it reduces infection, the question is one-sided. The one-sided p is then about 0.029, while the two-sided p is about 0.057.

What it does: Two-sided (tables with probability not greater than the observed one), > (OR > 1), or < (OR < 1).

Effect on the result: The main p, the decision, the conditional OR CI, and the direction of mid-p / Barnard / Boschloo change. In RxC the direction is ignored and a note is written.

On the validation data: TeaTasting two-sided .4857, > .2429 (CI [0.3136, Inf]), < .9857; FatComp > .0367, < .9967.

→ One-sided and two-sided tests (hypothesis direction)

Confidence Level (%90, %95, %99, Other/custom 80.0-99.9)

Concept: Sets the decision threshold (α = 1 − confidence level) and the width of the confidence intervals for the odds ratio, relative risk and risk difference. The p value does not change.

Everyday example: With only 20 patients the exact 95% confidence interval for the odds ratio is very wide (roughly 0.001 to 1.05) and includes 1, in line with the borderline p value. At 99% confidence it widens further.

What it does: Used for the Woolf CI, the exact conditional OR CI, the RR / RD CI, the Haldane CI, the CI of the MC p in RxC, and the decision.

Effect on the result: The CI widths and the decision change; p does not change.

On the validation data: Convictions exact conditional OR CI: 90% [0.00499, 0.28496], 95% [0.00332, 0.36319], 99% [0.00137, 0.57885]. FatComp at 99% decision "not rejected" (p = .0393). RxC MC CI 95% [.5944, .6005].

→ Confidence level and confidence interval

RR / risk difference (checkbox)

Concept: The relative risk (RR) gives how many times one group's risk of the event is the other's, and the risk difference gives the gap between the two risks in percentage points. They are more intuitive than the odds ratio; when the event is common, the OR and RR can differ considerably.

Everyday example: The infection risk is 10% with the new dressing and 60% with the old. RR = 0.10/0.60 ≈ 0.17 and the risk difference is −50 percentage points: the new dressing seems to prevent about 50 infections per 100 patients, though with 20 patients the interval around this estimate is wide.

What it does: RR for the first-column risk (row 1 / row 2, log-Wald CI) and the risk difference (Wald CI; clipped to [-1, 1]).

Effect on the result: Extra rows only. In RxC a "2x2 only" note is given; when c = 0, RR = ∞, so the RR row is not written (RD is written).

On the validation data: Convictions RR = 0.1529 [0.0402, 0.5814]; RD = -0.6516 [-0.9271, -0.3761]. At 99.9% the RD lower limit is -1.0.

→ Relative risk (RR) and risk difference (RD)

mid-p (checkbox)

Concept: A correction that subtracts half the probability of the observed table from the exact p value. It reduces the excess conservatism that the exact test has because of whole-number counts, keeping the actual type I error rate closer to α.

Everyday example: In the clinic example, Fisher's two-sided p is 0.057 while the mid-p can drop below 0.05. When the two p values fall on either side of the threshold, the finding is in an uncertain zone and more data are needed.

What it does: A less conservative p that subtracts half the probability of the observed table from the exact p (central two-sided).

Effect on the result: Extra row only; in RxC a "2x2 only" note is given.

On the validation data: Convictions .000481, TeaTasting .2571, FatComp .03995 (central two-sided mid-p).

→ Mid-p value

Barnard / Boschloo (checkbox)

Concept: Fisher's test treats both row and column totals as fixed; Barnard's and Boschloo's tests model the situation where only the group sizes are fixed and the event rates of two independent groups are compared. This 'unconditional' approach is often more powerful than Fisher's.

Everyday example: The clinic assigned 10 patients to each dressing by design, so the group sizes are fixed but the total number of infections was not known in advance. Barnard's test then gives p ≈ 0.022 on the same data, a clearer result than Fisher's 0.057.

What it does: Unconditional exact tests for two independent binomial samples with fixed row totals.

Effect on the result: Extra rows only.

On the validation data: Barnard FatComp .02972, Convictions .00034797. In MerQur the Boschloo two-sided p is defined as 2 x min(one-sided) (FatComp .0363).

→ Unconditional exact tests (Barnard, Boschloo)

φ Cohen interpretation (checkbox)

Concept: φ is an effect size showing the strength of association in a 2x2 table on a 0-to-1 scale. Cohen's thresholds (about 0.1 small, 0.3 medium, 0.5 large) help describe the value in words.

Everyday example: In the clinic example φ is about 0.52 and is labelled a 'large' association. That the effect looks large while p is borderline shows that 20 patients are not enough to establish even a large effect firmly.

What it does: Size label for |phi| = √(X²/n) with Cohen thresholds (.1 / .3 / .5); the label in the header row uses the same thresholds.

Effect on the result: Label only.

On the validation data: Convictions .6591 "Large", FatComp .4644 "Medium", TeaTasting .5 "Large". MerQur writes the absolute value of phi; the direction of the association is read from the table.

→ Phi (φ) coefficient

Example on real data

Data: Convictions, Fisher 1962, R ?fisher.test

Variables: column 1 Twin (dizygotic/monozygotic), column 2 Conviction (whether the twin was also convicted)

  • Question: Does the conviction of the co-twin of a convicted twin depend on twin type (monozygotic, dizygotic)? Because the sample is small (n = 30), Fisher's exact test is used instead of the chi-square approximation.
  • Descriptive: Among monozygotic twins 10 of 13 co-twins were convicted, among dizygotic twins 2 of 17.
  • Result: Fisher exact p = 0.0005 (two-sided). The null hypothesis is rejected: conviction is associated with twin type.
  • Effect size: φ = 0.66 (large effect). Odds ratio 0.04, 95% CI [0.0056, 0.2840]; the odds of the co-twin being convicted among dizygotic twins are about 1/25 of those among monozygotic twins.
  • Additional finding: Conditional maximum likelihood odds ratio 0.0469, exact 95% CI [0.0033, 0.3632]; the exact CI is based on this conditional estimate.
  • APA: Fisher's exact test, p < .001, OR = 0.040 (95% CI [0.006, 0.284]), φ = .66.

Related concepts

Monte Carlo p-valueUnconditional exact tests (Barnard, Boschloo)Mid-p valueRelative risk (RR) and risk difference (RD)Phi (φ) coefficientOne-sided and two-sided tests (hypothesis direction)Odds ratio (OR)Exact test and exact p-value

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.