MerQuR+Statistics Glossary

Statistics Glossary → Effect Size Calculator

Effect Size Calculatoranalysis

A tool that computes not just whether a finding is statistically significant but how large it is, using standard, comparable measures (such as Cohen's d, η², Cramér's V, r and the odds ratio).

Concept

A p value says whether an effect can be explained by chance, not how large it is: in a very large sample even a trivial difference can be significant, while in a small sample an important difference can fail to reach significance. An effect size expresses the practical importance of a finding on a scale independent of the measurement unit; it makes it possible to compare studies, combine them in a meta-analysis and calculate sample sizes for new studies.

The type of effect size is chosen according to the design. For two group means, Cohen's d gives the difference in standard-deviation units: d = (mean₁ − mean₂) / pooled SD; Hedges' g is its small-sample-corrected version. With three or more groups, η², ω² or ε² give the share of total variation explained by the groups; for categorical variables, φ and Cramér's V give the strength of association; for two continuous variables, r gives the relationship; and in 2x2 tables, the odds ratio, relative risk, risk difference and number needed to treat (NNT) describe the change in risk. Effect sizes can also be derived from reported t, F or χ² values, and d, r and the odds ratio can be converted into one another.

Effect sizes are estimated from samples too, so they should be reported with confidence intervals; in small samples the interval can be very wide. Cohen's 'small, medium, large' benchmarks (e.g. 0.2, 0.5, 0.8 for d) are only a rough guide; the importance of an effect should be judged against the field and context. Different measures are not interchangeable, and which one is reported should be stated clearly.

When to use it

  • When reporting a test result together with the size of the finding, comparing studies, or preparing effect sizes for a meta-analysis.
  • When only published summary statistics (means, SDs, n, t, F, χ²) are available, to derive effect sizes from them without the raw data.
  • To plan the sample size of a new study, the expected effect size is set with this tool and passed to a power analysis; to combine the effects of several studies, meta-analysis is used.

An everyday example

A gym compares two diet programmes: the 25 people on programme A lose 5 kg on average in three months and the 25 on programme B lose 3 kg, with a standard deviation of about 2.5 kg in both groups. The difference is statistically significant, but the manager asks 'is it a big difference?'. Cohen's d = (5 − 3) / 2.5 = 0.8: the gap between the group means is about 80% of the typical spread within the groups (the standard deviation), a large effect by Cohen's benchmarks. The confidence interval shows, however, that with 25 people per group the estimate could plausibly lie anywhere from about 0.2 to 1.4.

Parameters in MerQur

Analysis type (Cohen's d (two group means), η²/ω² (ANOVA, 3+ groups), Cramér's V (categorical × categorical), r/r² (correlation), Paired d (d_z, d_av, d_rm), Rank-based (rank-biserial, Cliff's δ, ε²_R), From summary / test statistics + conversions)

Concept: Chooses which family of effect sizes is computed. The choice follows the study design: the standardised mean difference (d) for two independent groups, the share of variance explained (η², ω²) for three or more groups, Cramér's V for categorical variables, r for two continuous variables, paired d for two measurements on the same people, and rank-based measures for ordinal or non-normal data.

Everyday example: The gym compares two programmes, so it chooses Cohen's d. With a third programme, η² or ω² would fit; to compare the same people's weight before and after the diet, paired d would.

What it does: Selects the effect-size family; the measure is determined by the study design (Lakens, 2013, Tables 1-2).

Effect on the result: Determines the computed measures and the visibility of the other form fields. With three-group data, Cohen d gives an explicit error ("exactly 2 groups required"). Hedges g uses the exact correction factor.

On the validation data: Lakens: independent d = 1.1259; Hedges g = 1.0782; η² = 0.2604; ω² = 0.2107; paired d_z = 1.5000; r = 0.7255. mtcars mpg ~ am d = -1.4779, rank-biserial -0.6599; mpg ~ cyl ε²_R = 0.8305.

→ Effect size

Column 1 and Column 2

Concept: The columns used in the calculation, depending on the analysis type: the outcome and grouping variable for a mean difference, or two categorical or two continuous variables for measures of association. The order of the groups sets the sign of d and which group counts as the control.

Everyday example: In the gym example the first column is weight lost and the second the programme name (A or B). With A as the first group d = +0.8; with B first d = −0.8. The size is the same; only the direction of the statement changes.

What it does: Depending on the analysis type, sets the dependent variable and group (or two categorical / two continuous) columns.

Effect on the result: Group levels are sorted alphabetically as strings; the sign of d, the control group of Glass Δ and the exposed/event level in 2x2 risk come from this order.

On the validation data: RCT_table Group x Diagnosis in alphabetical order: exposed = Control, event = Recovered -> RR = 1.8615. With the level-order fields ("Treatment,Control", "Sick,Recovered") RR = 2.5427 [1.7960, 3.5998].

Confidence Level (%90, %95, %99, Other (custom))

Concept: Sets how 'sure' the confidence interval for the effect size is. Since an effect size is itself an estimate, the interval shows the range within which the true effect may lie; the point estimate does not change.

Everyday example: With 25 per group, the 95% confidence interval for d = 0.8 runs from about 0.2 to 1.4: the true effect could be small or very large. At 99% confidence the interval widens and approaches zero.

What it does: Sets the level of all confidence intervals: d and d_z (noncentral t), g (d interval x J), η²/ω²/ε² (noncentral F), φ/V (noncentral χ²), r and rank-biserial (Fisher z), OR/RR/RD (Wald). 90% is recommended for η² accompanying an F test (Lakens; Steiger).

Effect on the result: Only the intervals change.

On the validation data: Lakens d interval: 90% [0.3148, 1.9086], 95% [0.1631, 2.0621], 99% [-0.1331, 2.3627]; RCT OR 90% [2.9951, 7.4805].

→ Confidence level and confidence interval

Standardizer (extra) (Pooled SD only, Average SD and both Glass Δ)

Concept: Sets which standard deviation the mean difference is divided by. The pooled SD assumes the two groups have equal variances; if they differ, the average of the two groups' SDs, or the control group's SD alone (Glass's Δ), may be a more meaningful scale.

Everyday example: Had weight loss on programme A been very spread out (SD 4 kg) and on B tightly clustered (SD 1 kg), the pooled SD would represent neither group well. If B is the control programme, Glass's Δ expresses the difference as 'how many times the usual variation in group B'.

What it does: Only for Cohen d: d_av = difference/√((SD₁² + SD₂²)/2) and two Glass Δ taking each group as the control. For unequal group variances (Cumming, 2012 d*).

Effect on the result: An additional section. The values are computed from means and SDs rounded to 4 decimals.

On the validation data: mtcars d_av = -1.4110 (with the average of the two group SDs); Glass Δ control = group 0 -1.8896, control = group 1 -1.1749 (without small-sample correction).

→ Glass's Δ

2×2 risk measures

Concept: In 2x2 tables, describes the change in the risk of an event from different angles: the odds ratio and relative risk show the proportional difference, the risk difference the absolute difference. The reciprocal of the risk difference (1/|RD|) gives the number of people who must be treated to prevent one event (NNT), or among how many people one harm appears (NNH).

Everyday example: Suppose the gym looks at 'did the person reach their target weight?': 60% on programme A and 40% on B did. The risk difference is 20 percentage points and NNT = 1/0.20 = 5: for every 5 people put on A instead of B, about 1 more reaches the target.

What it does: Only in the Cramér V type and in a 2x2 table: OR (Woolf), RR (log Wald), RD (Wald) and 1/|RD| (NNT/NNH).

Effect on the result: An additional section. In a table that is not 2x2, the risk rows are skipped with a note and V is shown. A neutral label explaining the direction is written for 1/|RD|.

On the validation data: RCT OR = 4.7333 [2.7436, 8.1659], RD = 0.3560 [0.2422, 0.4698], 1/|RD| = 2.81 [2.13, 4.13]; FatComp OR = 8.25 [1.1535, 59.0029].

→ Relative risk (RR) and risk difference (RD)

Column 1 / Column 2 level order

Concept: Sets the order of groups and categories: in a 2x2 table the first row level is the exposed (or treated) group and the first column level the event of interest. This order does not change the size of the effect, only its sign and the direction in which it is read.

Everyday example: If the gym gives the order as 'A, B' and 'reached, did not reach', the relative risk is 0.60/0.40 = 1.5: people on A are 1.5 times as likely to reach the target. With the order 'B, A', the same finding would read as 0.67, in the opposite direction.

What it does: In 2x2 risk measures, gives the row (Column 1) and column (Column 2) level order, comma-separated; the first level = the exposed group (rows) and the event of interest (columns). It also sets the group order for Cohen d and the rank type.

Effect on the result: The sign and direction of RR, RD and d. If left empty, the default (alphabetical) order is used.

On the validation data: RCT RR = 2.5427; FatComp with "1,0"/"1,0" RR = 2.9333 [0.8502, 10.1204].

Summary statistics

Concept: Computes effect sizes without raw data from reported summary statistics (means, SDs and n, or t, F, χ² values) and converts between measures such as d, r and the odds ratio. It is useful for bringing published studies onto a common scale, especially for meta-analysis.

Everyday example: The gym manager read a journal study reporting 't(48) = 2.83 between programmes A and B' but has no access to the raw data. From this t value and the group sizes, d ≈ 0.8 can be computed, so the gym's own result can be compared directly with the published one.

What it does: Only in the summary type: produces effect sizes and d -> r -> OR conversions from the inputs m/sd/n, t/n1/n2, t/df, t/n, F/df1/df2, chi2/n/k/df, d, r, or; derives effect sizes for meta-analysis from reported study results.

Effect on the result: Raw data are not used. If degrees of freedom or table size are not given in a χ² input, no confidence interval is computed and an explicit note is written; an unrecognized input gives an explicit error.

On the validation data: "t = 2.52, n1 = 10, n2 = 10" d = 1.1270 [0.1640, 2.0634]; "F = 6.34, df1 = 1, df2 = 18" at 90%: η²p = 0.2605 [0.0255, 0.5030]; "r = 0.5" d = 1.1547, OR = 8.1205.

→ Effect size

Example on real data

Data: Lakens (2013) movie ratings example

Variables: analysis type Cohen's d; dependent variable rating, group movie (Movie 1, Movie 2)

  • Question: How large is the difference between the ratings given to two movies (n = 10 per movie)?
  • Descriptive: Movie 1: M = 8.70, SD = 0.82; Movie 2: M = 7.70, SD = 0.95; pooled SD = 0.888.
  • Result: Cohen d = 1.13, 95% CI [0.16, 2.06] (noncentral t method); the interval does not contain zero.
  • Effect size: A large effect by Cohen's (1988) criterion (d > 0.8). Hedges g, which corrects the small-sample bias, = 1.08, 95% CI [0.16, 1.97]; Glass Δ, which uses only the SD of Movie 2, = 1.05.
  • Limitation: With a total n = 20 the confidence interval is very wide (0.16 to 2.06): the true effect may be small or very large.
  • APA: d = 1.13, 95% CI [0.16, 2.06].

Related concepts

Effect sizeCohen's d (effect size)Glass's ΔRelative risk (RR) and risk difference (RD)Omega squared (ω²)Epsilon squared (ε²)Cramér's V and Cohen's wNumber needed to treat (NNT)

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.