Interpreting and Communicating Results
Reaching the end of a statistical analysis is not the same as finishing the work. The most technically flawless hypothesis test loses its value if its results are misread, overstated, or communicated in ways that mislead rather than inform. Interpreting and communicating results well requires a disciplined understanding of what the numbers actually say, a clear sense of their limitations, and the language skills to translate probabilistic evidence into honest, accessible conclusions. This topic explores each dimension of that challenge in depth.
Drawing Statistically Sound Conclusions
Every conclusion in hypothesis testing must be anchored to a specific, answerable question: does the evidence in the data justify rejecting the null hypothesis at the chosen significance level? Nothing more, nothing less. Before any data are collected, the analyst formulates a null hypothesis (H₀) representing the default assumption — typically that there is no effect, no difference, or no relationship — and an alternative hypothesis (H₁) representing the claim being investigated. After computing a test statistic and its corresponding p-value, the conclusion follows a simple but strict rule: if the p-value is less than or equal to the pre-specified significance level α, the null hypothesis is rejected in favour of the alternative; otherwise, the null hypothesis is not rejected.
It is essential to use the phrase not rejected rather than accepted. Failing to reject H₀ does not establish that H₀ is true; it only means the current data do not provide sufficient evidence to overturn it. Think of a criminal trial: a verdict of "not guilty" does not declare the defendant innocent — it declares that the evidence was insufficient for conviction. Statistical conclusions carry the same logical structure.
A statistically significant result means exactly one thing: the observed data would be unlikely to arise by chance alone if the null hypothesis were true. It does not mean the finding is large, important, or practically meaningful. Overstating certainty is one of the most damaging habits in applied statistics. Hypothesis testing is fundamentally probabilistic; there is always a non-zero probability that the conclusion is wrong, and communicating honestly means acknowledging that reality.
Consider a concrete example. A pharmaceutical company tests whether a new drug lowers blood pressure more effectively than a placebo. After conducting a two-sample t-test, they obtain a test statistic of t = 2.45 with a corresponding p-value of 0.016. With α = 0.05, a sound conclusion would read: "At the 5% significance level, we reject the null hypothesis that the drug has no effect on blood pressure (t = 2.45, p = 0.016). The data provide evidence to suggest the drug is associated with a reduction in blood pressure compared to the placebo." Notice that this conclusion references the test statistic, the p-value, the significance level, and the direction of the finding — all without claiming absolute proof.
Interpreting P-Values in Context
The p-value is arguably the most widely cited and most widely misunderstood number in applied statistics. Formally, the p-value is the probability of observing a test statistic as extreme as, or more extreme than, the one actually obtained, assuming the null hypothesis is true. That conditional clause — "assuming H₀ is true" — is crucial and is routinely dropped in informal interpretation, leading to serious errors.
A small p-value (conventionally below 0.05, though this threshold is a convention rather than a law of nature) signals that the observed data are inconsistent with the null hypothesis. The smaller the p-value, the stronger the evidence against H₀. A p-value of 0.002, for instance, says that if H₀ were true, there would be only a 0.2% chance of seeing data as extreme as what was observed — very strong grounds for skepticism about H₀. A p-value of 0.38, conversely, says the data are quite compatible with H₀; there is no particular reason to abandon it.
The most pervasive misinterpretation is treating the p-value as the probability that the null hypothesis is true. This is categorically incorrect. The p-value is computed under the assumption that H₀ is true; it cannot simultaneously measure the probability of that assumption. Saying "p = 0.03, so there is a 3% chance the null hypothesis is correct" is a logical error with the same structure as saying "if I were in Paris, I would be in France; I am in France, therefore I am in Paris." The reasoning runs in the wrong direction.
The p-value must also always be interpreted relative to the pre-specified α level, not in isolation. Whether p = 0.049 is "significant" or not depends entirely on whether α was set at 0.05 before the analysis began. Choosing α after seeing the data — sometimes called "p-hacking" — invalidates the entire inferential framework because it exploits the natural variability of random samples to manufacture the appearance of significance. The threshold must be committed to in advance.
A useful way to appreciate the contextual nature of p-values is through a comparison table of common outcomes:
| P-value Range | Interpretation (relative to α = 0.05) | Decision |
|---|---|---|
| p ≤ 0.01 | Strong evidence against H₀ | Reject H₀ |
| 0.01 < p ≤ 0.05 | Moderate evidence against H₀ | Reject H₀ |
| 0.05 < p ≤ 0.10 | Weak evidence against H₀; below threshold | Fail to reject H₀ |
| p > 0.10 | Little to no evidence against H₀ | Fail to reject H₀ |
These categories are guidelines for interpretation, not rigid rules. A p-value of 0.051 and one of 0.049 are substantively almost identical; treating them as categorically opposite conclusions because one sits above and the other below 0.05 would be scientifically misleading.
Communicating Findings Clearly and Accurately
Statistical literacy varies widely among audiences. A report submitted to a team of biostatisticians can use technical notation freely, but the same findings presented to a hospital executive board or a general readership require plain-language translation. In either case, good communication demands three non-negotiable elements: stating what hypothesis was tested, stating the chosen significance level, and stating the conclusion explicitly.
Word choice matters enormously. The verb "proves" should never appear in a statistical conclusion. Hypothesis testing is an inductive process; it accumulates evidence, it does not deliver certainty. Similarly, "confirms," "establishes," and "demonstrates conclusively" all overstate what the method can deliver. More honest and accurate alternatives include:
- "The data provide evidence to suggest..."
- "The results are consistent with the hypothesis that..."
- "We fail to find sufficient evidence against the assumption that..."
- "At the 5% significance level, the data support the conclusion that..."
A complete written conclusion for a hypothesis test should typically contain the following components in plain language: the research question, the null and alternative hypotheses in words, the significance level, the computed test statistic and p-value, and the real-world interpretation. For example:
"We tested whether the average commute time for employees at Site A differs from the company-wide average of 35 minutes. Using a one-sample t-test at the 5% significance level, we obtained t = −2.87 (p = 0.005). Because p < 0.05, we reject the null hypothesis and conclude that the data provide evidence that the average commute time at Site A is significantly shorter than the company-wide average. The observed sample mean was 29.4 minutes."
Alongside statistical results, reporting the effect size substantially enriches the communication. Effect size measures how large or meaningful the observed difference or relationship is, independent of sample size. Reporting that a new teaching method produced a statistically significant improvement in test scores is far more informative when accompanied by the note that the average improvement was 1.2 points out of 100 — a detail that immediately signals whether the finding is practically interesting.
Distinguishing Statistical Significance from Practical Significance
Statistical significance and practical significance are related but entirely separate concepts, and conflating them is a mistake with real-world consequences. Statistical significance is a function of the p-value, which in turn depends on both the size of the true effect and the sample size. Practical significance asks a different question: is the effect large enough to matter in the real world?
Large sample sizes expose the critical gap between these two notions. With a sample of 500,000 observations, even an effect that is negligibly small in any practical sense will produce a p-value astronomically close to zero and generate a "statistically significant" result. To illustrate: an online retailer might test two versions of a checkout button and find that the green button produces a 0.02% higher conversion rate — a result that is highly statistically significant across millions of sessions, but generates perhaps two additional sales per 10,000 visits. Whether that matters depends on business context, but calling it "significant" without clarification would mislead stakeholders.
The antidote is to report effect sizes alongside p-values. Two of the most widely used effect size measures are:
- Cohen's d: Used for comparing two means. It expresses the difference between means in units of the pooled standard deviation. By convention, d ≈ 0.2 is small, d ≈ 0.5 is medium, and d ≈ 0.8 is large.
- Pearson's r: Used for correlations or as an effect size for t-tests. r ≈ 0.1 is small, r ≈ 0.3 is medium, and r ≈ 0.5 is large.
The following table illustrates how statistical and practical significance can diverge:
| Scenario | Sample Size | Mean Difference | P-Value | Cohen's d | Verdict |
|---|---|---|---|---|---|
| Drug A vs Placebo | 30 | 12 mmHg | 0.031 | 0.78 | Statistically and practically significant |
| Drug B vs Placebo | 50,000 | 0.3 mmHg | 0.002 | 0.02 | Statistically significant; practically negligible |
| Drug C vs Placebo | 15 | 8 mmHg | 0.21 | 0.60 | Not statistically significant; potentially practically meaningful |
Drug C in the table illustrates an equally important scenario: a potentially meaningful effect that fails to reach statistical significance, most likely due to an underpowered study. Dismissing this finding without noting the moderate effect size and the small sample could cause a genuinely beneficial treatment to be abandoned prematurely. Good communication captures all of this nuance.
Acknowledging and Communicating Potential Errors
No hypothesis test is infallible. Two types of error are always possible, and honest reporting acknowledges both.
A Type I error occurs when the null hypothesis is true but is nonetheless rejected — a false positive. The probability of committing a Type I error is exactly α, the significance level. Choosing α = 0.05 means accepting a 5% chance of incorrectly rejecting a true H₀. This is a deliberate, pre-set tolerance for error. When reporting results, explicitly stating the chosen α communicates to the audience what risk of a false positive was deemed acceptable before the analysis began.
A Type II error occurs when the null hypothesis is false but is not rejected — a false negative. The probability of a Type II error is denoted β, and statistical power is defined as 1 − β: the probability of correctly rejecting a false H₀. Power depends on the true effect size, the sample size, and the chosen α. A study with low power may genuinely miss a real effect, and results from such a study should be interpreted cautiously. When the stakes of a false negative are high — for instance, failing to detect a dangerous drug side effect — a low-powered study is not just statistically weak; it is potentially harmful.
| H₀ is Actually True | H₀ is Actually False | |
|---|---|---|
| Reject H₀ | Type I Error (probability = α) | Correct Decision (power = 1 − β) |
| Fail to Reject H₀ | Correct Decision (probability = 1 − α) | Type II Error (probability = β) |
When communicating results, noting the chosen α level transparently allows readers to assess whether the risk of a Type I error was set at an appropriate level for the context. In medical diagnostics or aviation safety, α = 0.01 or even 0.001 may be warranted. In exploratory social science research, α = 0.05 or 0.10 might be defensible. The rationale for the chosen threshold should be part of the methods narrative.
When consequences of error are high — a new surgical procedure, a financial regulation, a public health intervention — recommending replication or further study is not a sign of weak analysis. It is responsible science. A single study, however well-designed, should rarely be treated as definitive. Flagging this limitation in the conclusion section strengthens, rather than undermines, the credibility of the report.
Presenting Results with Appropriate Confidence
A confidence interval is one of the most powerful and informative tools available for communicating statistical results, yet it is often overlooked in favour of reporting only the p-value. A 95% confidence interval, for instance, is a range of values constructed so that, if the study were repeated many times, 95% of the resulting intervals would contain the true population parameter. For a single study, the interval provides a plausible range for the true effect based on the observed data.
Confidence intervals offer several advantages over a standalone p-value. First, they convey the direction of an effect: an interval entirely above zero (e.g., [2.1, 8.4]) indicates a positive effect, while one straddling zero (e.g., [−1.3, 5.7]) is consistent with either a positive or null effect. Second, they convey precision: a narrow interval such as [3.1, 3.9] indicates a highly precise estimate, while a wide interval such as [−5.0, 25.0] signals substantial uncertainty and should temper confidence in any conclusion. Third, they naturally communicate practical significance: an interval that lies entirely within a practically meaningless range delivers the message directly, even if the p-value happens to be small.
Pairing a confidence interval with a p-value gives the audience a richer picture. Consider two studies both reporting p = 0.04:
- Study A: p = 0.04, 95% CI for mean difference = [0.1, 0.3] kg lost per week
- Study B: p = 0.04, 95% CI for mean difference = [0.1, 12.7] kg lost per week
Both studies lead to the same binary decision (reject H₀), but Study B's extremely wide interval signals that the estimate is very imprecise, almost certainly due to a small or highly variable sample. That caveat would be invisible if only the p-value were reported.
Wider confidence intervals are not failures to be hidden — they are honest representations of uncertainty that the audience needs to make informed decisions. A 95% CI of [−0.5, 18.3] for an estimated cost saving in a business intervention tells decision-makers that while the average estimate is positive, the data are compatible with outcomes ranging from a slight loss to a large gain. That range should directly inform how much investment the decision-maker is willing to commit before seeking more data.
Using confidence intervals as the primary communication vehicle also subtly shifts the framing of results away from the binary "significant or not" dichotomy toward a more informative estimation framework. Instead of asking only "is there an effect?", the communication now answers "how large might the effect be, and how certain are we?" This is almost always a more useful question for real-world decision-making, and it reflects the genuine state of statistical knowledge more faithfully than a p-value alone ever can.