Statistics Glossary → Weighted Frequency (Survey Freq)
Weighted Frequency (Survey Freq)analysis
An analysis that estimates the population proportions and counts of a categorical variable, with design-correct confidence intervals, from survey data collected under a complex sampling design, and tests the association of two categorical variables while accounting for the design.
Concept
Questions such as 'What percentage of pupils come to school without breakfast?' are among those surveys answer most often. If the sample was drawn with a complex design, the raw sample percentage can misrepresent the population, because some groups were deliberately over- or under-represented. The weighted frequency counts each respondent by the number of people they represent and estimates both the population proportion and the population count.
The proportion is the share of the total weight falling in the category: p̂ = Σ wᵢ·[in category] / Σ wᵢ. The standard error is computed from the stratum and cluster structure; pupils in the same school sharing similar habits can make the proportion less precise than under simple random sampling. When testing the independence of two categorical variables, the ordinary Pearson chi-square test assumes independent observations and so declares 'significance' too often under a complex design; the Rao-Scott correction fixes this by adjusting the chi-square for the design effect.
When proportions are close to 0 or 1, a symmetric (Wald) confidence interval can spill outside the 0-1 range; logit-transformed or Korn-Graubard intervals are more suitable then. Subgroup proportions should be computed by keeping the full design, not by filtering the data. Results should report the confidence interval and design effect alongside the proportion.
When to use it
- When examining the population proportions of a categorical variable, or the association between two categorical variables, in survey data with weights, strata and clusters.
- For unweighted data from a simple random sample, ordinary frequency tables and the Pearson chi-square test are enough.
- For the mean of a continuous variable use the weighted mean; to relate the probability of an event to several predictors use weighted logistic regression.
An everyday example
The province-wide middle-school survey asks 2000 pupils in schools sampled from urban and rural strata 'Did you have breakfast this morning?'. In the raw data 22% might say 'no'; but once weights are applied the proportion for all middle-school pupils in the province might be 25%, because the crowded urban schools, where skipping breakfast is more common, make up a larger share of the population. The analysis gives this proportion with a design-correct confidence interval and uses the Rao-Scott test to check whether skipping breakfast differs between girls and boys.
Parameters in MerQur
Weight (required)
Concept: The sampling weight shows how many people in the population a respondent stands for and is the inverse of the selection probability. When the proportion is computed, each respondent counts not as one person but as as many people as their weight.
Everyday example: If an urban pupil who says they skipped breakfast has a weight of 200, that answer counts as 200 similar pupils in the province skipping breakfast. Doubling every weight leaves the proportion unchanged; only the estimated numbers of pupils double.
What it does: The sampling weight (inverse of the selection probability; the final weight with nonresponse or calibration adjustment if any). Proportions are weighted, the SE is computed by Taylor linearization.
Effect on the result: Proportions and SE; the weight scale does not change the proportion but changes the weighted frequency.
On the validation data: IceCream Group proportion: unweighted (SRS) 0.575, with stratified weights 0.544555.
Stratum
Concept: Stratification divides the population into non-overlapping groups and samples each separately. Because differences between strata do not add to sampling uncertainty, the proportion's standard error shrinks when the strata differ in how common the category is.
Everyday example: If skipping breakfast is clearly more common in towns than in rural areas, treating town and countryside as separate strata ensures both are represented correctly and estimates the provincial proportion more precisely.
What it does: Computes the variance from the deviations of PSU totals within strata; design degrees of freedom = PSU - strata.
Effect on the result: SE, confidence interval and degrees of freedom; the point estimate does not change.
On the validation data: apistrat awards = Yes proportion: unstratified SE 0.0360031, stratified by school type SE 0.0348975.
Cluster / PSU
Concept: The cluster (PSU) is the unit within which people are selected as a group. Because people in the same cluster tend to answer alike, the effective sample size falls and the standard error of a proportion usually grows.
Everyday example: If half the pupils at one school skip breakfast because classes start very early, the answers of its 40 sampled pupils are not independent of one another. Giving the cluster variable lets the standard error account for this similarity.
What it does: Sets the primary sampling unit; the variance is computed from PSU totals.
Effect on the result: The SE usually grows because of within-cluster similarity; degrees of freedom = PSU - strata.
On the validation data: apistrat stype = E proportion (no clusters) SE 0.029125; apiclus1 (clustered by dnum) SE 0.0468026.
FPC (population)
Concept: The finite population correction shrinks the variance by multiplying it by (1 − n/N) when a substantial part of the population is sampled. With a small sampling fraction its effect is negligible.
Everyday example: If 15 of the 30 rural schools were sampled, half of that stratum's schools have already been observed, and the stratum's contribution to the variance of the proportion is reduced accordingly.
What it does: Finite population correction; the variance is multiplied by (1 - n_h/N_h).
Effect on the result: The SE and confidence interval shrink.
On the validation data: apistrat awards: without FPC SE 0.0348975, with FPC 0.0344059; apiclus1 cluster: without FPC 0.0468026, with FPC 0.0463366.
Single-PSU stratum (No contribution (certainty, default), Centered at the average (adjust))
Concept: If a stratum has only one cluster, the variation between clusters within it cannot be computed and its contribution to the variance is undetermined. The option decides whether this contribution counts as zero or whether a more cautious contribution is computed relative to the average of all clusters.
Everyday example: If an island district forms a separate stratum and only one school was sampled there, how much breakfast habits vary between that district's schools cannot be known. The cautious option reflects this unknown in the standard error.
What it does: Sets the variance contribution of a single-PSU stratum; no contribution (certainty, default) or centred at the average (adjust).
Effect on the result: SE and confidence interval only when a single-PSU stratum exists.
On the validation data: apiclus1 (stratum dnum, contains a single-PSU stratum): no contribution SE 0.0298242; centred at the average SE 0.0298469.
Target column
Concept: The target column is the categorical variable whose proportions are to be estimated. A separate proportion, standard error, confidence interval and design effect is computed for each category; the proportions add up to 1.
Everyday example: If the question is not 'Did you have breakfast this morning?' but 'How many days a week do you have breakfast? (never / 1-3 days / 4-6 days / every day)', you get a separate provincial proportion for each of the four categories.
What it does: Selects the categorical variable whose proportions are computed (including numerically coded categorical variables with more than two values).
Effect on the result: Proportion, SE, confidence interval and DEFF per category.
On the validation data: apistrat stype = E proportion 0.713755; awards = Yes proportion 0.638936.
Subgroup (by)
Concept: Domain (subgroup) estimation computes the category's proportion within a subpopulation while keeping all the design information. Subgroup analysis done by filtering the data gives wrong standard errors.
Everyday example: If the breakfast-skipping rate is wanted separately for grades 5, 6, 7 and 8, grade is given as the subgroup, and each grade's proportion comes with a design-correct confidence interval.
What it does: Computes proportions within subgroups (domain estimation).
Effect on the result: Rows per domain.
On the validation data: apiclus1, sch_wide = Yes proportion: domain E 0.916667, domain H 0.785714.
Cross-table column
Concept: The cross-table shows the joint distribution of two categorical variables and tests whether they are associated. Under a complex design the ordinary Pearson chi-square is over-optimistic because it assumes independent observations; the Rao-Scott and Wald tests account for the design effect and give correct p values.
Everyday example: If 28% of girls and 22% of boys skip breakfast, the question is whether this difference also exists in the population. The Rao-Scott test answers it while accounting for the similarity of pupils in the same school.
What it does: Gives the cross-table with a second categorical variable and the design-based tests of independence (Rao-Scott first and second order, Wald, adjusted Wald) (Rao and Scott, 1984).
Effect on the result: Cell proportions, row percentages, Pearson chi-square, RS1, RS2 F, Wald F and adjusted Wald F.
On the validation data: SIS_Survey Department x Response: Wald F 2.86135, adjusted Wald F 2.83776. SchoolType x Response: RS2 F 2.25073, RS1 chi-square 9.00292.
DEFF denominator (Automatic (default; based on the sum of weights), Without replacement (SPSS / R default), With replacement (WR; R "replace" / SAS))
Concept: The design effect (DEFF) is the ratio of the proportion's variance under the design to its variance under a simple random sample of the same size. The simple-random-sampling variance in the denominator can assume sampling with or without replacement; the choice affects only DEFF.
Everyday example: If the DEFF for the breakfast proportion is 1.5, the 2000 pupils selected by the clustered design carry about as much information as 1330 pupils chosen by simple random sampling.
What it does: Selects the simple random sampling variance in the denominator of the design effect (the same rule as in the means analysis).
Effect on the result: Only DEFF and DEFT.
On the validation data: apistrat without replacement 1.05519, with replacement 1.02112. SIS_Survey without replacement 2.26185, with replacement 2.15428.
Confidence Level (%90, %95, %99, Other (custom))
Concept: Sets how much assurance the confidence intervals give of covering the true population proportion. A higher level gives a wider interval.
Everyday example: The proportion of pupils in the province who skip breakfast might lie between 22% and 28% with 95% confidence; asking for 99% widens the range.
What it does: Sets the level of the confidence intervals.
Effect on the result: Intervals only.
On the validation data: apistrat awards proportion (logit interval) lower bound: 95% 0.568721, 90% 0.580367, 99% 0.545582.
Additional output: Weighted frequencies (population counts) + SE, unweighted cell n
Concept: The weighted frequency shows how many people in the population are estimated to fall in a category, while the unweighted cell count shows how many actual respondents the estimate rests on. Estimates for cells with very small unweighted counts are unreliable.
Everyday example: An estimated 21,000 pupils in the province might skip breakfast. If that estimate rests on 480 sampled pupils it is solid; a subgroup estimate resting on only 12 pupils should be read with caution.
What it does: Gives the weighted frequency (population count) per category with its Taylor SE and confidence interval, and the unweighted cell count.
Effect on the result: Additional table only.
On the validation data: apiclus1 stype = E: 4873.97, SE 1333.32.
Additional output: Unweighted n and sum of weights (N̂)
Concept: The unweighted n is the number of actual respondents in the analysis; the sum of weights is the estimated size of the population they represent.
Everyday example: If 600 pupils responded in the rural subgroup and their weights sum to 18,000, those 600 pupils represent about 18,000 rural middle-school pupils in the province.
What it does: Gives the valid unweighted number of observations and the sum of weights per domain.
Effect on the result: Additional table only.
On the validation data: apiclus1 domain H n = 14, sum of weights 473.858.
Additional output: Differences between domains (contrasts)
Concept: Gives the difference in each category's proportion between subgroups, with a design-correct standard error, so that whether two subgroups' proportions really differ is seen in a single test.
Everyday example: If the breakfast-skipping rate is 7 percentage points higher in grade 8 than in grade 5, the contrast gives a confidence interval for that difference; if it excludes zero, the difference is taken to exist in the population.
What it does: Gives the proportion difference of each category between pairs of domains (only with a subgroup selected).
Effect on the result: Additional table only.
On the validation data: apiclus1 sch_wide = Yes, E - H difference 0.130952, p = 0.218375.
Proportion CI method (Logit-transformed (default; SPSS CSTABULATE), Wald (symmetric), Korn-Graubard (beta))
Concept: A confidence interval for a proportion can be built in different ways. The Wald interval is symmetric and can spill outside the bounds when the proportion is near 0 or 1; the logit-transformed interval stays within 0-1 because it is computed on the logit scale and transformed back; the Korn-Graubard interval rests on the beta distribution and is more cautious for very small or very large proportions.
Everyday example: If only 2% of pupils never have breakfast, the Wald interval may give a meaningless lower limit, such as '−0.5% to 4.5%'. A logit or Korn-Graubard interval gives an asymmetric range that stays above zero, such as '1.1% to 3.6%'.
What it does: Selects how the proportion confidence interval is built: logit (default, the interval does not leave 0-1), Wald (symmetric) or Korn-Graubard (beta; Korn and Graubard, 1998). Logit or Korn-Graubard is recommended for extreme proportions.
Effect on the result: Proportion intervals only.
On the validation data: apiclus1 cluster + FPC, lower bound of the difference: logit 0.0354088; Wald 95% 0.0190005, Korn-Graubard 0.0296381; Wald 90% 0.0292816; logit 90% 0.0407295.
→ Confidence interval for a proportion (Wilson, Clopper-Pearson, Agresti-Coull, Jeffreys)
Odds ratio / risk ratio for 2x2 tables
Concept: In a 2x2 table, the odds ratio is the odds of the event (occurring vs not occurring) in one group divided by the same odds in the other; the risk ratio is the direct ratio of the two groups' event probabilities. For both, a value of 1 means no difference between the groups.
Everyday example: If 28% of girls and 22% of boys skip breakfast, the risk ratio is about 1.27: girls are about 1.27 times as likely to skip breakfast as boys. The odds ratio is (0.28/0.72) / (0.22/0.78) ≈ 1.38; the more common the event, the further the two measures drift apart.
What it does: Computes the odds ratio (OR) and the two risk ratios (RR1, RR2) for 2x2 tables; the confidence interval is built on the log scale by the delta method as exp(log +- t(ν) SE). Off by default.
Effect on the result: Additional table only.
On the validation data: With Taylor variance the OR, RR1, RR2 and bounds are given for the apistrat, apiclus1 and nhanes tables; apistrat sch_wide x comp_imp OR = 31.4192421306. With an empty cell MerQur leaves OR and RR2 undefined and computes RR1 = 5.9259.
Variance method (Taylor (default), Jackknife (JK1 / JKn), BRR (balanced half-samples), Fay BRR, Bootstrap (Rao-Wu), Supplied replicate weights) and related fields (Fay coefficient (rho), Bootstrap replicates, Random seed, Replicate weight columns, Replicate type (JK1, JKn, BRR, Fay, bootstrap, other), Scale, Replicate scales (rscales), Deviations from)
Concept: The standard error can be computed by formula-based Taylor linearisation or by replication methods (jackknife, balanced repeated replication, bootstrap). Replication methods form subsamples in line with the design, recompute the proportion in each, and derive the uncertainty from how much those proportions vary.
Everyday example: In the jackknife, schools are left out one at a time and the breakfast-skipping rate is recomputed. If removing one school shifts the rate noticeably, the result is sensitive to that school, and this shows up as a larger standard error.
What it does: Computes the SE by resampling instead of Taylor: jackknife (JK1, JKn; with FPC), BRR and Fay BRR (2 PSUs per stratum), Rao-Wu bootstrap and supplied replicate weights. Cross-table tests take the cell-proportion covariance from the replicates; df = rank(W) - 1.
Effect on the result: SE, confidence intervals, test statistics and degrees of freedom; the point estimate does not change.
On the validation data: apistrat awards = Yes proportion, JKn (200 replicates): p = 0.6389, SE 0.0349, 95% CI [0.5677, 0.7046]; design df = rank(W) - 1.
Example on real data
Data: California API apistrat, R survey package
Variables: target awards (award eligibility), weight pw, stratum stype (school type)
- Question: In the stratified school sample (weight pw, strata stype), what percentage of schools qualified for the award programme (awards)?
- Design: n = 200 schools, 3 strata, no clusters; design df = 197; Taylor SE, logit-transformed confidence interval (t, df = 197).
- Result: The proportion receiving the award (Yes) is p̂ = .639 (SE = .035), 95% CI [.568, .705]; the proportion not receiving it (No) is p̂ = .361 (SE = .035), 95% CI [.295, .432].
- Design effect: DEFF = 1.09 (DEFT = 1.04); the stratified weighted design increases the variance of the proportion by about 9% relative to simple random sampling.
- Interpretation: About two thirds of the schools in the population qualified for the award; for proportions near the 0-1 boundary the logit interval is more suitable than the symmetric Wald interval.
- APA: Yes: p = .639, SE = .035, 95% CI [0.568, 0.705], DEFF = 1.09 (N = 200).
Related concepts
Primary sampling unit (PSU) and cluster samplingDomain (subpopulation) estimationDesign effect (DEFF)Taylor linearization and replication variance methodsFinite population correction (FPC)Odds ratio (OR)Sampling weight (design weight)Stratified sampling (design stratum)
The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.