Interpreting and Communicating Results

Statistical hypothesis testing produces a set of numerical outputs — a test statistic, a p-value, degrees of freedom, and often a confidence interval — that must be read together and communicated with precision. Each component answers a slightly different question, and conflating them is one of the most persistent sources of error in applied research. This topic develops a thorough understanding of what each output means, how the outputs relate to one another, and how to present findings in a way that is both technically accurate and genuinely informative to a broad audience.

Understanding the Test Statistic

Every hypothesis test reduces the information in a dataset to a single number called the test statistic. This number summarises how far the observed data deviates from what would be expected if the null hypothesis were exactly true. The formula used to compute the test statistic depends on the type of test being performed and on the assumptions that are met:

• A z-test (used when the population standard deviation is known, or when samples are large enough for the Central Limit Theorem to apply fully) produces a z-statistic that follows a standard normal distribution under the null hypothesis.
• A t-test (used when the population standard deviation is unknown and the sample is relatively small) produces a t-statistic that follows a t-distribution whose shape depends on the degrees of freedom.
• A chi-square test (used for categorical data, tests of independence, or goodness-of-fit assessments) produces a χ² statistic that follows a chi-square distribution under the null hypothesis.
• An F-test (used in ANOVA and regression) produces an F-statistic that reflects the ratio of explained to unexplained variance.

The magnitude of the test statistic is key: a larger absolute value means the observed data lies further from what the null hypothesis predicts, which in turn corresponds to stronger evidence against the null. For example, a t-statistic of 4.2 represents far greater departure from the null than a t-statistic of 0.8, all else being equal.

The sign (direction) of the test statistic also carries information for tests involving mean differences or regression coefficients. A positive t-statistic for a two-sample test means the first group's mean exceeded the second group's mean in the sample; a negative value indicates the opposite. This directional information is especially relevant when interpreting the results of two-tailed tests, because the same p-value could correspond to effects in either direction.

For complete and reproducible reporting, always state: (1) the name of the test, (2) the numeric value of the test statistic rounded to an appropriate number of decimal places, (3) the degrees of freedom where applicable, and (4) how the statistic was derived. For instance: "An independent-samples t-test was conducted, yielding t(48) = 3.14." This format alone tells a knowledgeable reader exactly which sampling distribution was used and how much information the test was based on.

Interpreting P-Values Correctly

The p-value is defined as the probability of obtaining a test statistic at least as extreme as the one observed, assuming the null hypothesis is true. It is a conditional probability, and that conditionality is everything. Formally:

p-value = P(observing data this extreme or more extreme | H₀ is true)

A small p-value — typically below the pre-specified significance level α — means that the observed data would be quite rare if the null hypothesis were correct. This rarity is interpreted as evidence against the null hypothesis, leading researchers to reject it. A commonly used threshold is α = 0.05, meaning we are willing to incorrectly reject a true null hypothesis no more than 5% of the time in the long run.

Several critical misconceptions about p-values must be guarded against:

• The p-value is not the probability that the null hypothesis is true. It is not P(H₀ is true | observed data). That would require a prior probability for H₀, which frequentist statistics does not provide. Statements like "there is only a 3% chance that the null hypothesis is true" are incorrect.
• The p-value does not measure effect size or practical importance. A p-value of 0.001 does not mean the effect is large or meaningful — it simply means the data are very inconsistent with the null. With a large enough sample, even a difference of 0.0001 units can yield p = 0.001.
• A large p-value does not prove the null hypothesis. Failing to reject H₀ means only that the data do not provide sufficient evidence against it. The null could be false, but the study may have lacked the statistical power to detect the departure. This is the principle that absence of evidence is not evidence of absence.
• The significance level must be set before data collection. Choosing or adjusting α after seeing the p-value is a form of data manipulation. If α is set at 0.05 and p = 0.07, the result is not significant — it would be wrong to then decide that α = 0.10 would have been more appropriate all along.

Significance Levels and Decision Thresholds

The significance level α is the researcher's chosen maximum acceptable probability of committing a Type I error — rejecting the null hypothesis when it is actually true (a false positive). The most widely used values are:

The choice of α involves a direct trade-off with the Type II error rate (β) — the probability of failing to reject a false null hypothesis (a false negative). Lowering α reduces false positives but increases false negatives, and vice versa. The statistical power of a test is defined as 1 − β, and adequate power (conventionally ≥ 0.80) should be ensured through appropriate sample size planning before data collection begins.

It is also essential to apply any necessary corrections when conducting multiple hypothesis tests simultaneously. If 20 independent tests are performed at α = 0.05 each, on average one will produce a false positive even if all null hypotheses are true. Corrections such as the Bonferroni correction (dividing α by the number of tests) or the Benjamini–Hochberg procedure (controlling the false discovery rate) help manage this inflation.

Once α is set and the test is run, the decision rule is binary and deterministic: if p ≤ α, reject H₀ and declare statistical significance; if p > α, fail to reject H₀. There is no statistical basis for treating a p-value of 0.049 as meaningfully different from one of 0.051 — both sit near the arbitrary boundary and deserve cautious interpretation.

Using Confidence Intervals to Complement Hypothesis Tests

A confidence interval (CI) provides a range of plausible values for the population parameter being estimated. For a parameter θ and a 95% confidence interval [L, U], the correct interpretation is: if the same procedure were repeated on many independent samples, approximately 95% of the resulting intervals would contain the true population parameter. It is not correct to say "there is a 95% probability that the parameter lies in this specific interval" — once computed, the interval either does or does not contain the true value.

Confidence intervals add substantial value over p-values alone for several reasons:

• Direction and magnitude. A confidence interval shows not only whether an effect is statistically distinguishable from zero but also how large that effect plausibly is. A mean difference with a 95% CI of [0.1, 0.3] conveys far more than simply knowing p = 0.002.
• Precision of estimation. A narrow CI indicates that the estimate is precise (often because of a large or well-designed study), while a wide CI signals substantial uncertainty. For example, a hazard ratio of 2.5 with a 95% CI of [1.1, 5.7] is statistically significant but highly imprecise — the true effect could be anywhere from a modest elevation in risk to a very large one.
• Alignment with hypothesis testing. The connection between confidence intervals and hypothesis tests is exact: a 95% CI that excludes the null value (e.g., 0 for a difference, 1 for an odds ratio) corresponds precisely to rejecting the null at α = 0.05 using a two-tailed test. This duality means CIs can substitute for formal tests while providing richer information.

Consider a concrete example. Suppose a two-sample t-test comparing the mean exam scores of two teaching methods yields t(98) = 2.45, p = 0.016, with a mean difference of 4.8 points and a 95% CI of [0.9, 8.7]. The p-value alone tells us the result is significant at α = 0.05. The CI tells us additionally that the true advantage of Method A over Method B is plausibly between 0.9 and 8.7 points — a range that allows a subject-matter expert to judge whether the improvement is educationally meaningful.

Distinguishing Statistical Significance from Practical Significance

One of the most important lessons in applied statistics is that statistical significance is not the same as practical importance. A result is statistically significant when the p-value falls below α; it is practically significant when the magnitude of the effect is large enough to matter in the real world. These two forms of significance are logically independent.

The reason they can diverge is that the p-value is influenced by three things simultaneously: (1) the size of the true effect, (2) the sample size, and (3) the variability in the data. With a very large sample, even a negligibly small effect will produce a p-value below any conventional α. Conversely, with a small sample, a large and meaningful effect may fail to reach significance simply because there is not enough data to detect it reliably.

Effect size measures address this by quantifying the magnitude of an effect on a scale that does not depend on sample size. Commonly used measures include:

These benchmarks are useful starting points but should never replace domain knowledge. In clinical trials, a Cohen's d of 0.2 might represent a trivial improvement in a quality-of-life score but a crucial reduction in a rare serious adverse event. Practical significance must ultimately be judged by experts in the relevant field, not derived mechanically from any statistical formula.

Reporting Hypothesis Test Results Clearly

Clear reporting serves two goals: it allows readers to evaluate the analysis critically, and it allows other researchers to reproduce or build upon the work. A well-written results statement should include all of the following elements:

• Type of test — specify the test by name (e.g., "independent-samples t-test," "Pearson chi-square test of independence").
• Test statistic and degrees of freedom — report the numeric value and, where applicable, the degrees of freedom in the standard notation, e.g., t(df) or χ²(df).
• Exact p-value — report p to two or three decimal places (e.g., p = 0.032) rather than using vague inequalities like p < 0.05 wherever possible. For very small values, p < 0.001 is acceptable.
• Confidence interval — state the confidence level and both bounds (e.g., "95% CI [1.2, 4.7]").
• Effect size — report the appropriate measure with its value and, if useful, its interpretation in context.
• Plain-language conclusion — state whether the null hypothesis was rejected and what this means for the research question, written clearly enough for a non-statistician to understand.
• Limitations — acknowledge any factors that might compromise the validity of the conclusions, such as assumption violations (e.g., non-normality, unequal variances), non-random sampling, or small sample sizes that limit power.

A complete example of well-reported results might read: "An independent-samples t-test was used to compare mean systolic blood pressure between patients receiving the new medication (n = 52, M = 118.4, SD = 11.2) and those receiving the placebo (n = 50, M = 124.9, SD = 12.0). The test yielded t(100) = −2.83, p = 0.006, with a mean difference of −6.5 mmHg (95% CI [−11.0, −2.0]) and a Cohen's d of 0.56 (medium effect). The null hypothesis of no mean difference was rejected at α = 0.05. The observed reduction is statistically significant and clinically plausible, though confirmation in a larger, more diverse sample is warranted."

Avoid certain patterns that undermine clarity and accuracy. Writing "the result was marginally significant" or "approached significance" for a p-value of, say, 0.07 is technically misleading — at the pre-specified α = 0.05 the result was not significant, and implying otherwise blurs the decision rule. Similarly, never report only that "p < 0.05" without providing the actual value; exact p-values allow readers to make their own judgements and are required for any meta-analysis.

Common Misinterpretations to Avoid

Even experienced analysts commit errors in interpreting and communicating statistical results. The following are the most consequential mistakes and how to avoid them:

• Treating non-significance as proof of no effect. A non-significant result means the test did not find convincing evidence against H₀ given the data available. It does not establish that H₀ is true. A study with very low power might consistently fail to detect a real and important effect. The appropriate language is "we found no statistically significant evidence of an effect," not "we proved there is no effect."
• Equating a smaller p-value with a larger or more important effect. Because p-values depend on sample size, a study with n = 10,000 might produce p = 0.0001 for a difference of 0.5 points on a 100-point scale, while a study with n = 30 might produce p = 0.08 for a difference of 15 points. The second study's result, though not significant, may be pointing to a far more important effect. Always examine the effect size alongside the p-value.
• P-hacking and selective reporting. Analysing data in multiple ways and reporting only the analysis that produced p < 0.05 is known as p-hacking or data dredging. This practice inflates the actual Type I error rate far above the nominal α, because each additional analysis provides another chance to find a spuriously significant result. Pre-registering hypotheses and analysis plans before data collection is the most effective safeguard. Reporting all tests conducted, including non-significant ones, is also essential for transparency.
• Over-generalising from a single significant result. Statistical significance in one study is not sufficient grounds for strong scientific conclusions. A p-value below 0.05 will occur by chance 5% of the time under H₀. Scientific confidence is appropriately built through replication — multiple independent studies that find consistent effect sizes and directions. Consistency of effect magnitudes across replications is as important as, if not more important than, consistency in achieving statistical significance.
• Ignoring the assumptions underlying the test. Every hypothesis test rests on assumptions (e.g., independence of observations, normality of residuals, homogeneity of variance). Violating these assumptions can distort the p-value and make the stated Type I error rate inaccurate. Always check assumptions, report when they are met or violated, and consider using alternative tests or robust methods when major violations are detected.

In summary, interpreting and communicating statistical results well requires holding several ideas in mind simultaneously: the test statistic and its distribution, the exact p-value and what it genuinely measures, the confidence interval and what it reveals about plausible effect magnitudes, the appropriate effect size and what it implies practically, and an honest accounting of the study's limitations. Each piece of information complements the others, and omitting any of them risks misleading both the immediate reader and the broader scientific community.