MerQuR+Statistics Glossary

Statistics Glossary → False discovery rate (FDR, Benjamini-Hochberg, Storey q)

False discovery rate (FDR, Benjamini-Hochberg, Storey q)concept

The expected proportion of false positives among the results declared significant; a more lenient control criterion than the familywise error rate when many tests are run.

Familywise error rate (FWER) methods (Bonferroni, Holm) keep the probability of making even one false positive below α; with hundreds of tests this is very strict and real effects are missed. FDR instead controls the question 'what share of my discoveries are false?'. In the Benjamini-Hochberg method the m p values are sorted from smallest to largest, and the largest i satisfying p₍ᵢ₎ ≤ (i/m)·q is found; all hypotheses up to that rank are rejected. With q = 0.05, for example, on average no more than 5% of the rejections are expected to be false positives.

Benjamini-Hochberg is valid when the tests are independent or positively dependent; under arbitrary dependence the more conservative Benjamini-Yekutieli is used. Storey's approach estimates from the data the proportion of null hypotheses that are actually true (π₀) and gives each test a q value: the smallest FDR level at which that test could be called significant. FDR suits exploratory work with many tests (gene expression, many correlations, map cells); for a small number of confirmatory hypotheses, FWER control is more appropriate.

An everyday example

A supermarket chain tests for 200 products whether a new shelf layout changed sales and finds p < 0.05 for 30 of them. Even with no effect at all, about 10 of 200 tests would be expected to come out significant by chance. With Benjamini-Hochberg (q = 0.05), 18 products remain significant, and on average at most 1 of these 18 is expected to be a false alarm.

In MerQur

Offered in Multiple Comparison Correction as Benjamini-Hochberg, Benjamini-Yekutieli, two-stage methods and Storey's q value. It is also among the multiple-testing adjustment options in Kruskal-Wallis, the multiple-response crosstabs, Cochran's Q, correlation analysis, Kaplan-Meier, Moran's I and Getis-Ord Gi*.

Analyses where this concept appears

Multiple Comparison CorrectionKruskal-WallisMR × Categorical CrosstabMR × MR CrosstabCochran's Q TestCorrelation AnalysisKaplan-Meier Survival AnalysisMoran's I (Spatial Autocorrelation)Getis-Ord Gi* Hotspot

Related concepts

Multiple comparisons and the familywise error rate (FWER)Holm (step-down Bonferroni) method