MerQuR+Statistics Glossary

Statistics Glossary → Bootstrap CI

Bootstrap CIanalysis

A method that finds the confidence interval of a statistic without distributional assumptions or a ready-made formula, by drawing thousands of new samples with replacement from the data at hand and recomputing the statistic in each.

Concept

To compute a confidence interval you need to know how the statistic varies from sample to sample, its sampling distribution. For the mean this distribution has a known formula, but for statistics such as the median, the trimmed mean, the coefficient of variation or the difference between two medians, there is often no simple, reliable formula; and formulas usually rest on assumptions such as normality. The bootstrap solves this problem by treating the sample at hand as the best available stand-in for the population, even though we cannot draw new samples from the population itself.

The method works as follows: from data with n observations, a 'bootstrap sample' of n observations is drawn with replacement; some observations are chosen several times, others not at all. The statistic is computed in this sample. The process is repeated thousands of times (B times) and the distribution of the resulting values is used in place of the sampling distribution. The standard deviation of this distribution gives the bootstrap standard error, and the difference between its mean and the observed value gives the bias. In the simplest, the percentile interval, the 2.5% and 97.5% points of the distribution are taken for 95% confidence; the BCa interval adds a further correction for bias and skewness.

The bootstrap is not magic: it assumes that the sample represents the population well and that observations are independent. In very small samples (e.g. fewer than 10 observations) the sample may not reflect the shape of the population and intervals can be too narrow. It does not work well for extremes or for statistics that depend on a single point of the data, such as the maximum. Because the result relies on random resampling, B should be large enough and the seed recorded for reproducibility.

When to use it

  • For statistics with no standard confidence-interval formula, or whose formula rests on strong assumptions, such as the median, the trimmed mean, the coefficient of variation, the difference between two medians or a correlation.
  • When the data are skewed and the sample is moderate in size, as an alternative to, or a check on, intervals based on the normal distribution; with near-normal data and a large sample, the classical t interval and the bootstrap interval are almost the same.
  • If the goal is not an interval but a test of the difference between two groups, the permutation test is a distribution-free option; if observations are dependent over time or within clusters, a block or cluster bootstrap that preserves this structure is needed.

An everyday example

A hospital wants to report how many days patients stay after a particular operation. The 40 patients mostly stayed 3-5 days, but a few with complications stayed 15-20 days; the distribution is right-skewed and the median is 4 days. Since there is no simple confidence-interval formula for the median, the hospital uses the bootstrap: new samples of 40 are drawn with replacement from the 40 patients' records and the median is computed in each. If after 10,000 repetitions the middle 95% of these medians falls between 3.5 and 5 days, that is the 95% bootstrap confidence interval for the median length of stay.

Parameters in MerQur

Column (list of numeric or binary columns)

Concept: The bootstrap uses the observed values of the chosen variable as the 'mini population' to resample from. Statistics such as a proportion require the variable to be coded as binary (0/1); missing values do not take part in the resampling.

Everyday example: The hospital chooses length of stay (days) as the numeric column. If it chose a column coding 'was the patient readmitted?' as 0/1, the bootstrap would instead give the confidence interval for the readmission rate.

What it does: Selects the column for the statistic (only numeric and binary columns are listed, missing values are dropped). A proportion needs a 0/1 or numerically coded column.

Effect on the result: Determines the whole output.

On the validation data: hsb2 write mean 52.7750, percentile 95% CI (51.445, 54.070); law GPA mean 3.0947, CI (2.9773, 3.2113).

Statistic (Mean, Median, SD, Proportion, Trimmed mean, Variance, coefficient of variation, percentile, Pearson, Spearman, mean difference, median difference)

Concept: The strength of the bootstrap is that it applies to statistics whose sampling distribution has no known formula. For statistics with a formula, such as the mean and the proportion, it checks the classical intervals; for the median, trimmed mean, coefficient of variation, percentiles and differences between two groups, it is often the most practical route.

Everyday example: Because length of stay is skewed, the hospital wants to report the median rather than the mean. The bootstrap gives the confidence interval for the median as easily as for the mean.

What it does: Selects the statistic whose CI is computed. SD and variance use n - 1, the percentile linear interpolation (type 7); the two-group difference is alphabetically first level - second level.

Effect on the result: The observed value, the bootstrap distribution and all CIs change.

On the validation data: aircondit: mean 108.08, CI (46.58, 191.00); median 88.0 (12.5, 115.0); SD 136.23 (41.16, 201.43). law Pearson LSAT-GPA 0.7764 (0.456, 0.962). The observed value and CI are given for 12 statistics in 4 datasets.

2nd column (Pearson and Spearman only)

Concept: In a bootstrap interval for a correlation, the two variables' values are never separated: each observation's pair of values is resampled as a unit. This preserves the relationship between the two variables in every bootstrap sample.

Everyday example: The hospital wonders about the relationship between patient age and length of stay. Each patient's 'age, length of stay' pair goes into the bag as a single card; as cards are drawn, the correlation is recomputed.

What it does: Selects the second variable of the correlation; row pairs are resampled together.

Effect on the result: If missing or equal to the first column, the analysis stops with a warning. No effect for other statistics.

On the validation data: hsb2 write-math Pearson 0.6174, CI for B = 500 (0.5403, 0.6931); the limits can vary within the Monte Carlo margin depending on resampling.

Group / stratum and Stratified resampling (checkbox)

Concept: When the difference between two groups is studied, resampling is done separately within each group, so every bootstrap sample keeps the groups' original sizes. Stratified resampling applies the same logic to other statistics: the share of an important subgroup stays fixed in every bootstrap sample.

Everyday example: If the hospital wants the difference in median length of stay between patients who had laparoscopic and open surgery, the two groups' records go into separate bags and each bag is drawn from up to its own size. That way no bootstrap sample ever loses one of the groups entirely.

What it does: For two-group difference statistics the group (exactly 2 levels) is selected; the stratified checkbox resamples separately within each group level for the other statistics.

Effect on the result: The group is required for the difference statistics, which already sample within strata; with stratified off, the group has no effect for the other statistics.

On the validation data: hsb2 write mean SE 0.6707, stratified 0.6489; CI (51.445, 54.070), stratified (51.490, 54.025).

→ Bootstrap (resampling)

B (number of resamples, 100-100000)

Concept: B is the number of times the resampling is repeated. As B grows, the bootstrap distribution takes shape more smoothly and the random (Monte Carlo) fluctuation in the interval limits shrinks. A few hundred repetitions are often enough for a standard error, but confidence-interval limits need thousands.

Everyday example: With only 200 repetitions the interval might come out as 3.5-5 days on one run and 3-5.5 days on another. With 10,000 repetitions the limits stay almost the same on every run.

What it does: Sets the number of resamples and controls the MC error (a warning for B < 1000).

Effect on the result: The CI limits change with B.

On the validation data: aircondit mean CI for B = 200 (55.33, 187.59), 2000 (47.41, 192.59), 10000 (46.58, 191.00); the MC SD of the percentile lower limit at B = 10000 is 0.53. B >= 2000 is recommended for percentile and BCa (Efron and Tibshirani, 1993).

→ Bootstrap (resampling)

Seed

Concept: Computer random numbers are really deterministic sequences generated from a starting value (the seed). The same seed produces the same random draws and hence the same result, which allows others to reproduce the analysis exactly.

Everyday example: If the hospital's report gives '3.5-5 days' for the median and states the seed used, an auditor with the same data and seed obtains exactly the same interval. A different seed changes the limits only by small amounts in the last digits.

What it does: Sets the random number generator seed for reproducibility (numpy default_rng); the same seed gives the same result.

Effect on the result: The CI limits change by the MC margin.

On the validation data: For aircondit, seed 0 CI (47.00, 189.00), 12345 (46.08, 190.50), 42 (46.58, 191.00).

→ Random seed

CI method (Percentile, BCa)

Concept: The percentile interval takes the extreme percentiles of the bootstrap distribution directly and is the most intuitive method. The BCa (bias-corrected and accelerated) interval makes two corrections, for the statistic's bias and for its standard error changing with its value; with skewed data and biased statistics it usually gives more accurate coverage. The normal, basic and studentized intervals are extra options for comparison.

Everyday example: Because lengths of stay are right-skewed, the bootstrap distribution of the mean is also slightly right-skewed. Where the percentile interval gives 4.8-7.1 days, BCa may correct for this skew and give a slightly right-shifted interval such as 4.9-7.4 days.

What it does: Selects the primary CI: percentile or BCa (Efron, 1987). The percentile, BCa, normal, basic and studentized (mean and mean difference) intervals are always reported in extra rows.

Effect on the result: The primary CI and the citation line change. If BCa is undefined (jackknife variance 0), the label becomes "Percentile (BCa could not be computed)" and a warning is written.

On the validation data: hsb2 write percentile (51.445, 54.070), BCa (51.415, 54.035) (z0 = -0.0168, a = -0.0056). aircondit mean BCa (56.00, 224.08), a = 0.0938.

→ BCa bootstrap confidence interval

Trim / percentile, % (trimmed mean and percentile only)

Concept: The trimmed mean is the mean of the observations left after a given share is removed from each end; it is more resistant to outliers than the ordinary mean. With 0% trimming it equals the ordinary mean, and as trimming approaches 50% it approaches the median. For the percentile statistic, this value instead sets which percentile (e.g. the 90th) is studied.

Everyday example: A 10% trimmed mean leaves out the shortest and longest 10% of stays and limits the influence of the few patients with complications. If, for planning, the hospital asks 'within how many days are 90% of patients discharged?', it wants the interval for the 90th percentile.

What it does: Sets the proportion trimmed from each end in the trimmed mean (⌊np⌋ observations) and p in the percentile statistic.

Effect on the result: Trimming 0% equals the mean, p = 50 equals the median. The valid trimming range is 0% to 50% (50 excluded); outside it a reasoned warning is given.

On the validation data: aircondit trimming 0% = 108.08, 10% = 80.7, 20% = 71.5; p = 50 = 88, p = 90 = 220.

→ Trimmed mean

Counted value (proportion only)

Concept: A proportion is the share of observations having a particular characteristic. Which value counts as a 'success', the case being counted, must be stated explicitly; otherwise the proportion may be computed for the wrong characteristic.

Everyday example: If a column codes patients' post-discharge status as 0 'no problem', 1 'readmitted' and 2 'emergency visit', and only the readmission rate is of interest, the value to count is given as 1.

What it does: Sets the "success" value of the proportion; if blank, the proportion x > 0 is used.

Effect on the result: The proportion and CI change.

On the validation data: For hsb2 female, blank and "1" are the same (0.545, CI 0.475-0.615); "0" 0.455; ses "2" 0.475.

Confidence Level (%90, %95, %99, Other/custom)

Concept: The confidence level sets which percentiles of the bootstrap distribution are taken as the interval limits: 2.5% and 97.5% for 95%. As the confidence level rises, the limits move further into the tails and the interval widens.

Everyday example: With 95% confidence the median length of stay might be 3.5-5 days, and with 99% confidence 3-5.5 days. Because the extreme percentiles rest on fewer bootstrap values, it is wise to choose a larger B at high confidence levels.

What it does: α = 1 - level; the CI percentiles are α/2 and 1 - α/2. The custom value is active only when "Other" is selected.

Effect on the result: The CI limits and the heading change.

On the validation data: aircondit mean percentile CI 80% (62.00, 158.43), 90% (52.83, 175.09), 95% (46.58, 191.00), 99% (35.17, 223.00).

→ Confidence level and confidence interval

Example on real data

Data: UCLA OARC hsb2 (High School and Beyond, n = 200)

Variables: column write (writing score), statistic mean

  • Question: What is a 95% confidence interval for the mean writing score that needs no distributional assumption? (n = 200, B = 10000 resamples, seed 42)
  • Result: Observed mean 52.775; bootstrap percentile 95% CI (51.445, 54.070), width 2.625.
  • Bootstrap distribution: Bootstrap SE = 0.6707 (very close to the classic SE 0.670), bias 0.0045; the bias is negligible for the mean.
  • Other intervals: BCa (51.415, 54.035), normal (51.456, 54.085), basic (51.480, 54.105), bootstrap-t (51.430, 54.070); all are very close to each other and to the classic t interval [51.45, 54.10].
  • Interpretation: If the same method were applied repeatedly, the interval would cover the true mean about 95% of the time; it is not the probability that this one interval contains the parameter.
  • Limitation: The results depend on random resampling; a different seed or program gives differences of the Monte Carlo margin in the last decimals.
  • APA: M = 52.78, 95% bootstrap CI [51.45, 54.07] (percentile, B = 10000).

Related concepts

Trimmed meanBootstrap (resampling)Coefficient of variation (CV)Percentiles (quantiles) and percentile definitionsRandom seedJackknifeSpearman rank correlationBCa bootstrap confidence interval

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.