MerQuR+Statistics Glossary

Statistics Glossary → Binomial Test

Binomial Testanalysis

A test that checks exactly, using the binomial distribution, whether the success rate observed in a two-outcome variable (yes/no, success/failure) is consistent with a pre-specified proportion.

Concept

Many questions concern a proportion: the germination rate of a batch of seeds, the success rate of a drug, how often a die lands on six. We have a sample of n trials, with k successes; the observed proportion is k/n. The question is whether this proportion could differ this much from a pre-specified value (p₀: a claim, a standard or a theoretical proportion) by chance. The null hypothesis states that the true probability of success is p₀.

If the trials are independent and the probability of success is the same in each, the number of successes follows a binomial distribution. The test computes directly from this distribution the probability of getting a number of successes as extreme as, or more extreme than, the one observed if the null hypothesis were true; that is why it is an 'exact' test, valid in small samples too, with no need for a normal approximation. The confidence interval for the proportion shows the range of values the true proportion might take; the Wilson and Clopper-Pearson intervals are reliable in small samples and for extreme proportions.

The assumptions are simple but important: each trial has only two outcomes, the trials are independent, and the probability of success is the same in all of them. Repeated observations from the same person, or data collected in clusters, break this assumption. Because of the discrete distribution, the exact test can be somewhat conservative; the mid-p adjustment reduces this. The practical importance of the result is judged from the difference between the observed proportion and p₀ and from effect sizes such as Cohen's h.

When to use it

  • When comparing the proportion observed in a two-outcome variable with a known value (a claim, a standard, a theoretical proportion), especially when the sample is small or the proportion is close to 0 or 1.
  • To compare the proportions of two independent groups, use the chi-square test or Fisher's exact test; to compare more than two categories with an expected distribution, use the chi-square goodness-of-fit test.
  • If only the direction of the differences between two measurements on the same people is studied, use the sign test, which is the binomial test applied to paired data; to compare two binary measurements on the same people, the McNemar test is appropriate.

An everyday example

A seed company claims '90% germination' on its packets. A farming cooperative sows 50 randomly chosen seeds and 40 of them germinate; the observed rate is 80%. The binomial test answers the question: if the germination rate really were 90%, how likely would it be for only 40 or fewer of 50 seeds to germinate? This probability is about 2.5%; the two-sided p ≈ 0.03. The 95% Wilson confidence interval for the true rate runs from about 67% to 89% and does not include 90%; the cooperative can conclude that the company's claim does not hold for this batch.

Parameters in MerQur

Binary column

Concept: The binomial test works on a variable that places each observation into one of two categories. The variable may already be binary (0/1, yes/no), or a numeric variable can be made binary by splitting it at a threshold.

Everyday example: The cooperative's data file has a 'did it germinate?' column for each seed (1 = yes, 0 = no); the test is applied to this column. A numeric column showing how many days each seed took to germinate could also be made binary with a threshold.

What it does: Selects the column whose success proportion is tested. All columns are listed; for numeric columns the cut point is used.

Effect on the result: Determines all results.

On the validation data: In hsb2, schtyp (coded 1/2) counts the value 1 as success when success is empty: k = 168, p̂ = 0.84. When write <= 50 is selected, k = 74/200, p = 0.000289.

Expected proportion H₀ (p₀; 0.001-0.999)

Concept: The expected proportion p₀ is the value the probability of success is assumed to take under the null hypothesis. It comes from the research question, not from the data: it should be a claim, a standard or a theoretical proportion. The observed proportion and its confidence interval do not depend on p₀; only the p value and the decision change with p₀.

Everyday example: Because of the company's claim, p₀ = 0.90. Had the cooperative used p₀ = 0.85, the same 40/50 result might count as consistent with that value; the observed 80% rate and its confidence interval would not change.

What it does: The success probability under the null hypothesis.

Effect on the result: Only the p value, the decision, and the effect size change; the proportion and the CIs do not depend on p₀.

On the validation data: hsb2: p₀ = 0.5 -> p = 0.229247; 0.4 -> 3.6e-05; 0.6 -> 0.112965. Conover: p₀ = 0.75 -> p = 0.382492, 0.7 -> 0.013275.

→ Null hypothesis (H₀) and test value

Confidence Level (%90, %95, %99, Other/custom 80.0-99.9)

Concept: The confidence level sets how 'sure' the confidence interval for the proportion is and the threshold (α) for the significance decision. It does not change the p value itself.

Everyday example: With 95% confidence the true germination rate lies between about 67% and 89%, excluding 90%. Asking for 99% confidence widens the interval, which may then include 90%; p ≈ 0.03 is also not significant at the 99% level.

What it does: Wilson CI level and decision threshold α = 1 - confidence level. The custom value is read only when "Other" is selected.

Effect on the result: The CI width and the "significant" decision change; p does not change.

On the validation data: hsb2 Wilson CI 90% (0.4869, 0.6019), 95% (0.4758, 0.6125), 99% (0.4543, 0.6328), 80% (0.4997, 0.5896), 99.9% (0.4298, 0.6556).

→ Confidence level and confidence interval

Hypothesis direction (Two-sided, P > p₀, P < p₀)

Concept: The direction of the hypothesis states whether the question asks if the proportion differs from p₀ in either direction (two-sided) or only whether it is larger, or only whether it is smaller. A one-sided test detects a departure in that direction more easily; the direction must be chosen before looking at the data.

Everyday example: If the cooperative only asks 'is the germination rate lower than claimed?', the test is left-sided (P < p₀) and p is about 0.025. This direction makes sense, because seeds doing better than claimed would be no cause for complaint.

What it does: Two-sided or one-sided exact binomial test.

Effect on the result: p and the decision change; in one-sided tests the main Wilson CI becomes one-sided ((lower, 1) or (0, upper)). The "All CI methods" block always stays two-sided; in the output the main CI is labelled "[one-sided]" and the block "(two-sided)".

On the validation data: hsb2: two-sided 0.229247, right-tailed 0.114623, left-tailed 0.910518; right-tailed Wilson (0.4869, 1.0000).

→ One-sided and two-sided tests (hypothesis direction)

Success category

Concept: Which outcome counts as a 'success' must be stated explicitly; the proportion and all test results are computed from this definition. The word 'success' need not mean something good; it simply means the outcome being counted.

Everyday example: If the column contains 'yes' and 'no', the success category is given as 'yes' and the germination rate is computed. Had 'no' been chosen, the same data would test the non-germination rate (20%), and p₀ would then have to be 0.10.

What it does: Specifies the value counted as success (text or number); 1 if empty. Required for a text-coded binary column (Yes / No); if left empty, the analysis stops with an error listing the levels.

Effect on the result: k and all derived values change.

On the validation data: hsb2 success = 0: k = 91, p̂ = 0.4550, Clopper-Pearson (0.3846, 0.5267). Pima success = Yes: k = 68; No: k = 132.

Cut point

Concept: The cut-off point turns a numeric variable into a binary one: observations less than or equal to the given value count as successes. The threshold must be chosen from the research question, before looking at the data; otherwise the result rests on an arbitrary choice.

Everyday example: If the company says '90% of seeds germinate within 7 days', the cut-off on the days-to-germination column is set to 7: seeds that germinate in 7 days or fewer count as successes.

What it does: In a numeric column x <= c is counted as success; when filled it overrides the success category; a comma decimal separator is accepted.

Effect on the result: k and all derived values change.

On the validation data: airquality Temp <= 79: k = 80/153, p = 0.627771; <= 85: k = 119. hsb2 female <= 0 (and the entry "0,5") k = 91. For a text column (Pima) the analysis stops with an error.

→ Cut point (dichotomization)

All CI methods (checkbox)

Concept: A confidence interval for a proportion can be computed in several ways. Clopper-Pearson is exact and conservative; Wilson and Agresti-Coull give good coverage even in small samples; Jeffreys comes from a Bayesian derivation but also works well in the classical sense. The widely taught Wald interval (p̂ ± z·SE) is unreliable in small samples and for proportions close to 0 or 1.

Everyday example: For 40/50, the Wilson interval is about 67-89%, Clopper-Pearson 66-90% (with the upper limit just below 90%), and Wald 69-91%. The Wald interval takes in 90% and would wrongly appear to support the claim; this is why the method is avoided for extreme proportions.

What it does: Additionally gives the Clopper-Pearson (exact), Wilson (score), Agresti-Coull, Jeffreys, and Wald CIs (two-sided). Wald is not recommended for small n or extreme proportions (Brown, Cai, and DasGupta, 2001).

Effect on the result: Extra block only.

On the validation data: Color data 90%: Agresti-Coull (0.4181, 0.4773), Clopper-Pearson (0.4174, 0.4779), Wilson (0.4181, 0.4773). Conover 95% Clopper-Pearson (0.7077, 0.7654).

→ Confidence interval for a proportion (Wilson, Clopper-Pearson, Agresti-Coull, Jeffreys)

Effect size (checkbox)

Concept: The effect size shows how far the observed proportion is from the expected one. The raw difference (p̂ − p₀) is the easiest to read; Cohen's h scales the proportions with an angular (arcsine) transformation, making differences between proportions near 0 or 1 comparable too. Roughly, 0.2 counts as a small, 0.5 a medium and 0.8 a large effect.

Everyday example: The difference between the observed 80% and the claimed 90% is −0.10, and Cohen's h is about −0.28: a small-to-medium effect. Had the same 10-point gap been between 50% and 60%, h would come out at about 0.20, because the same difference weighs more for proportions near the extremes.

What it does: p̂ - p₀ and Cohen h = 2·asin(√p̂) - 2·asin(√p₀).

Effect on the result: Extra row only (Cohen, 1988: 0.2 small, 0.5 medium, 0.8 large).

On the validation data: hsb2 h = +0.0901; Conover (p₀ = 0.5) h = +0.4945; Pima h = -0.3257.

→ Cohen's h

mid-p (checkbox)

Concept: Because the exact binomial test rests on a discrete distribution, its actual false-alarm rate often stays below the chosen α, so the test is more cautious than necessary. The mid-p reduces this caution by adding only half the probability of the observed result itself to the p value; it gives a more balanced result, especially in small samples.

Everyday example: For 40/50, the one-sided exact p is about 0.025 and the mid-p about 0.017. The difference comes from dropping half the probability of exactly 40 seeds germinating from the calculation.

What it does: A less conservative p that subtracts half the probability of the observed value from the exact p (2 x min when two-sided); it avoids the excessive conservatism of the exact test (Agresti, 2013).

Effect on the result: Extra row only.

On the validation data: hsb2 two-sided mid-p 0.2041 (exact 0.2292); Conover left-tailed (p₀ = 0.75) 0.1860.

→ Mid-p value

Example on real data

Data: UCLA OARC hsb2 (High School and Beyond, n = 200)

Variables: binary column female (1 = female counted as success)

  • Question: Does the proportion of female students in the sample (female = 1) differ from 50%? The proportion of a two-outcome variable is tested with the exact binomial distribution.
  • Descriptive: 109 of the 200 students are female; the observed proportion is 0.545.
  • Result: Exact two-sided p = 0.229. The null hypothesis cannot be rejected: the female proportion does not differ significantly from 0.5.
  • Confidence interval: Wilson 95% CI (0.4758, 0.6125); because the interval contains 0.5, it is consistent with the test decision.
  • APA: The proportion of female students (.55, 95% CI [.48, .61]) did not differ significantly from .50, exact binomial test p = .229.

Related concepts

Null hypothesis (H₀) and test valueEffect sizeCohen's hMid-p valueConfidence interval for a proportion (Wilson, Clopper-Pearson, Agresti-Coull, Jeffreys)Cut point (dichotomization)One-sided and two-sided tests (hypothesis direction)Exact test and exact p-value

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.