Statistics Glossary → Fail-safe N
Fail-safe Nconcept
The number of additional unpublished studies with zero effect that would have to be added to make a significant pooled meta-analytic result non-significant.
Proposed by Rosenthal (1979), this measure addresses the 'file-drawer problem', the tendency of studies with non-significant results to stay unpublished in a drawer. The studies' z values are combined with Stouffer's method, and the calculation asks how many studies with an average z of zero would have to be added to bring the combined z below the significance threshold (1.645 for one-sided α = 0.05). Rosenthal suggested that a fail-safe N larger than 5k + 10 (k being the number of studies) indicates a result robust to publication bias.
The fail-safe N is popular because it is easy to understand, but it has serious limitations: it focuses on significance rather than effect size, assumes the missing studies have an average effect of exactly zero (in reality they may point the other way) and ignores between-study heterogeneity. Guidelines such as Cochrane's therefore advise against using it on its own. Funnel plots, Egger's regression test and the trim-and-fill method should be used together to assess publication bias.
An everyday example
A review pools 12 published studies showing that a new teaching method works and finds a fail-safe N of 180. This means 180 more studies that found no effect would have to be sitting in file drawers for the result to lose significance. If that many unpublished studies are unlikely to exist, publication bias can hardly explain the result; yet the number says nothing about how large the effect is.
In MerQur
In Meta-Analysis it is computed with Rosenthal's method as 'Fail-safe N' among the publication-bias options.