Applying Common Hypothesis Tests

1 Applying Common Hypothesis Tests

Hypothesis testing is one of the most powerful tools in a data analyst's or researcher's toolkit. Rather than simply describing data, hypothesis tests allow us to make principled, probabilistic decisions about whether observed patterns reflect genuine phenomena or are plausibly explained by random chance. This topic walks through the most commonly applied hypothesis tests — the one-sample t-test, the two-sample t-test, the paired t-test, the chi-square test, and ANOVA — and then addresses the practical questions of how to choose the right test and how to communicate results clearly and honestly. Throughout, the emphasis is on applying these methods to real data situations rather than merely rehearsing formulas.

Each test rests on a shared logical framework: you state a null hypothesis (the "nothing interesting is happening" claim), choose a significance level that defines your tolerance for false positives, collect data, compute a test statistic that summarizes how far the data depart from what the null hypothesis predicts, and then compare that statistic to a reference distribution to obtain a p-value. If the p-value is below your pre-chosen threshold, you reject the null hypothesis and conclude the data are inconsistent with it. What differs across tests is the type of data involved, the structure of the comparison, and the reference distribution used.

One-Sample t-Test in Practice

The one-sample t-test answers a focused question: does the mean of a single population equal some specified value? This hypothesized value is not arbitrary — it comes from theory, a regulatory standard, a historical benchmark, or a business target. For example, a coffee shop claims its large cups contain 16 ounces on average. A quality-control analyst fills 25 cups and measures each one. The question is whether the sample mean is far enough from 16 ounces to constitute evidence of a systematic underfill or overfill.

The test statistic is computed as follows:

t = (x̄ - μ₀) / (s / √n)

where x̄ is the sample mean, μ₀ is the hypothesized population mean, s is the sample standard deviation, and n is the sample size. The denominator s / √n is the standard error of the mean — it captures how much random variability we expect in a sample mean of this size. The resulting t-statistic follows a t-distribution with n − 1 degrees of freedom under the null hypothesis.

Suppose the analyst finds: x̄ = 15.7 oz, s = 0.6 oz, n = 25. Then:

t = (15.7 - 16.0) / (0.6 / √25)
  = -0.3 / (0.6 / 5)
  = -0.3 / 0.12
  = -2.5

With 24 degrees of freedom and a two-tailed significance level of 0.05, the critical value is approximately ±2.064. Because |−2.5| > 2.064, the result falls in the rejection region. The corresponding p-value is roughly 0.020, which is below 0.05. The analyst rejects the null hypothesis and concludes there is statistically significant evidence that the average fill differs from 16 ounces.

Before trusting this conclusion, the analyst must check the key assumption: that the data are approximately normally distributed. For small samples (roughly n < 30), this matters considerably. A histogram or a normal probability plot (Q-Q plot) of the 25 measurements can reveal gross departures from normality. If the distribution is heavily skewed or contains extreme outliers, a non-parametric alternative such as the Wilcoxon signed-rank test is more appropriate. For large samples, the Central Limit Theorem ensures that the sampling distribution of the mean is approximately normal even if the raw data are not, making the t-test robust to moderate non-normality.

Two-Sample t-Test for Comparing Groups

When the research question involves comparing the means of two independent groups, the two-sample t-test is the natural tool. "Independent" means the observations in one group carry no systematic relationship to observations in the other — for instance, comparing test scores of students taught by two different methods, or comparing blood pressure between a treatment group and a control group in a randomized experiment.

The null hypothesis is H₀: μ₁ = μ₂ (the two population means are equal). The alternative can be two-tailed (H₁: μ₁ ≠ μ₂), or one-tailed (H₁: μ₁ > μ₂ or μ₁ < μ₂) depending on whether theory specifies a direction.

A critical decision before computing the statistic is whether to assume equal variances in the two populations. The standard approach is to use Levene's test or examine the ratio of sample variances. If the variances appear roughly equal, the pooled two-sample t-test is used:

t = (x̄₁ - x̄₂) / (sp × √(1/n₁ + 1/n₂))

where sp = √[((n₁-1)s₁² + (n₂-1)s₂²) / (n₁ + n₂ - 2)]

The pooled standard deviation sp combines variance estimates from both groups, weighted by their sample sizes. Degrees of freedom equal n₁ + n₂ − 2.

If variances differ substantially (a common situation in practice), Welch's t-test is preferred. It does not assume equal variances and uses a modified degrees-of-freedom formula (the Welch–Satterthwaite equation) that can produce a non-integer value. Many statisticians recommend using Welch's test by default because it is valid whether or not variances are equal, whereas the pooled test is invalid when they differ.

Consider an example: a company tests a new training program. Group A (n₁ = 30) receives standard training; Group B (n₂ = 28) receives the new program. Post-training assessment scores yield x̄₁ = 72, s₁ = 10, x̄₂ = 78, s₂ = 9. Using the pooled test:

sp = √[((29)(100) + (27)(81)) / (30 + 28 - 2)]
   = √[(2900 + 2187) / 56]
   = √[5087 / 56]
   = √90.84
   ≈ 9.53

t = (72 - 78) / (9.53 × √(1/30 + 1/28))
  = -6 / (9.53 × √(0.0333 + 0.0357))
  = -6 / (9.53 × √0.0690)
  = -6 / (9.53 × 0.2627)
  = -6 / 2.504
  ≈ -2.40

With 56 degrees of freedom at α = 0.05 (two-tailed), the critical value is approximately ±2.003. Since |−2.40| > 2.003, the null is rejected. The new program is associated with significantly higher scores.

However, statistical significance alone does not tell us whether the 6-point difference matters in practice. This is where effect size becomes essential. Cohen's d standardizes the mean difference:

d = (x̄₁ - x̄₂) / sp = 6 / 9.53 ≈ 0.63

Cohen's conventional benchmarks classify d ≈ 0.2 as small, d ≈ 0.5 as medium, and d ≈ 0.8 as large. A d of 0.63 suggests a moderate-to-large effect — the difference is not only statistically significant but also practically meaningful. Reporting effect sizes alongside p-values gives decision-makers a much richer picture.

Paired t-Test for Before-and-After Data

The paired t-test applies when observations come in natural pairs — the same individual measured twice (before and after an intervention), matched subjects in two conditions, or twin studies. The pairing is not incidental; it is designed to control for individual-level variability that would otherwise inflate the error term and reduce statistical power.

The key insight is that the paired test converts a two-sample problem into a one-sample problem. For each pair i, compute the difference:

dᵢ = x₂ᵢ - x₁ᵢ

Then test whether the mean of these differences, d̄, is significantly different from zero:

t = d̄ / (sd / √n)

where sd is the standard deviation of the difference scores and n is the number of pairs. Degrees of freedom equal n − 1.

Consider a wellness study measuring systolic blood pressure (mmHg) in 10 patients before and after an 8-week diet intervention:

The differences sum to −66, so d̄ = −6.6 mmHg. The standard deviation of the differences is approximately 3.27 mmHg. The test statistic is:

t = -6.6 / (3.27 / √10)
  = -6.6 / 1.034
  ≈ -6.38

With 9 degrees of freedom, the critical value at α = 0.05 (two-tailed) is ±2.262. Since |−6.38| ≫ 2.262, the result is highly significant (p < 0.001). The diet intervention produced a statistically significant reduction in blood pressure.

Why is the paired test more powerful here? If we had treated these as two independent groups, the high variability between patients (some naturally have higher blood pressure) would have inflated the denominator of the t-statistic. By working with differences, we eliminate that between-subject noise and isolate the within-subject change due to the intervention.

Before applying the paired t-test, verify that the differences are approximately normally distributed — not necessarily the raw measurements. For n = 10 this matters; a Shapiro-Wilk test or a Q-Q plot of the differences is appropriate. Additionally, when reporting results, pair the significance test with a confidence interval for the mean difference. Here, the 95% CI for d̄ is approximately (−6.6 ± 2.262 × 1.034), or roughly (−8.94, −4.26) mmHg. This tells stakeholders that the intervention likely reduces blood pressure by somewhere between about 4 and 9 mmHg — a clinically meaningful range.

Chi-Square Test for Categorical Data

The t-tests and ANOVA described above apply to continuous (numeric) outcomes. When the variable of interest is categorical — political affiliation, product preference, disease status, survey response — the chi-square (χ²) test is the workhorse. It comes in two main flavors: the goodness-of-fit test and the test of independence.

Goodness-of-Fit Test: This variant asks whether an observed frequency distribution matches a theoretically expected distribution. For instance, a genetics researcher expects offspring to follow a 3:1 ratio of dominant to recessive phenotypes (Mendelian inheritance). After observing 90 dominant and 30 recessive plants (n = 120), the expected counts under the 3:1 ratio are 90 and 30 respectively — which happen to match perfectly here. More interesting is when they do not match. Suppose instead we observe 100 dominant and 20 recessive:

Expected: dominant = 90, recessive = 30

χ² = Σ [(O - E)² / E]
   = [(100 - 90)² / 90] + [(20 - 30)² / 30]
   = [100/90] + [100/30]
   = 1.11 + 3.33
   = 4.44

With 1 degree of freedom (number of categories minus 1), the critical value at α = 0.05 is 3.841. Since 4.44 > 3.841, the null hypothesis of a perfect 3:1 ratio is rejected. The observed distribution departs significantly from Mendelian expectation.

Test of Independence: This variant determines whether two categorical variables are associated. The data are arranged in a contingency table. For example, a marketer surveys 200 customers on whether they saw an advertisement and whether they purchased a product:

Under the null hypothesis of independence, the expected count for each cell is:

E = (Row Total × Column Total) / Grand Total

Computing expected values:

E(Saw Ad, Purchased)         = (120 × 70) / 200 = 42
E(Saw Ad, Not Purchased)     = (120 × 130) / 200 = 78
E(Did Not See, Purchased)    = (80 × 70) / 200 = 28
E(Did Not See, Not Purchased)= (80 × 130) / 200 = 52

χ² = [(50-42)²/42] + [(70-78)²/78] + [(20-28)²/28] + [(60-52)²/52]
   = [64/42] + [64/78] + [64/28] + [64/52]
   = 1.52 + 0.82 + 2.29 + 1.23
   = 5.86

Degrees of freedom = (rows − 1) × (columns − 1) = 1 × 1 = 1. Critical value at α = 0.05 is 3.841. Since 5.86 > 3.841, we reject independence and conclude that exposure to the advertisement is significantly associated with purchase behavior.

A critical assumption for the chi-square test is that expected cell counts should be at least 5 in every cell. When this is violated — common with small samples or rare categories — the chi-square approximation becomes inaccurate. Alternatives include collapsing categories (if conceptually defensible) or using Fisher's exact test, which computes exact probabilities without relying on the chi-square approximation and is especially appropriate for 2×2 tables with small counts.

ANOVA for Comparing Multiple Group Means

When comparing means across three or more groups, performing multiple t-tests is tempting but statistically dangerous. Running three t-tests to compare groups A, B, and C inflates the family-wise error rate: if each test has a 5% chance of a false positive, the probability of at least one false positive across three tests is about 14%. ANOVA (Analysis of Variance) solves this by testing all groups simultaneously in a single test while maintaining the chosen significance level.

The null hypothesis is H₀: μ₁ = μ₂ = μ₃ = … = μₖ (all group population means are equal). The alternative is simply that at least one mean differs. ANOVA works by partitioning total variability in the data into two sources:

• Between-group variance (MSB): How much do the group means vary around the overall mean? If groups differ, this will be large.
• Within-group variance (MSW): How much do individual observations vary within each group around their group mean? This represents random noise.

The F-statistic is the ratio of these two:

F = MSB / MSW

Under the null hypothesis, both numerator and denominator estimate the same population variance, so F should be close to 1. A large F suggests the between-group spread exceeds what random noise alone would produce, providing evidence against H₀.

Consider a study testing three fertilizers (A, B, C) on plant growth (cm over 4 weeks), with 5 plants per group:

Group means: x̄_A = 22, x̄_B = 28, x̄_C = 19. Grand mean = (22 + 28 + 19)/3 = 23. The ANOVA results in an F-statistic that, when compared to the F-distribution with (k−1) = 2 numerator degrees of freedom and (N−k) = 12 denominator degrees of freedom, yields a p-value well below 0.05. We reject H₀ and conclude that the fertilizers do not all produce equal growth.

But ANOVA only tells us that at least one group differs — not which groups differ from which others. For that, we need post-hoc tests. Tukey's Honest Significant Difference (HSD) test is the most widely used: it tests all possible pairwise comparisons while controlling the family-wise error rate. In this example, Tukey's test would likely reveal that Fertilizer B produces significantly more growth than both A and C, while A and C may or may not differ significantly.

ANOVA rests on three assumptions:

• Normality: Observations within each group should be approximately normally distributed.
• Homogeneity of variances (homoscedasticity): The population variance should be approximately equal across groups. Levene's test formally checks this. If violated, Welch's ANOVA (an extension of Welch's t-test to multiple groups) is the appropriate alternative.
• Independence: Observations must be independent of each other, both within and across groups.

When normality is severely violated, the Kruskal-Wallis test — a non-parametric alternative — can be used. It tests whether the groups have the same distribution using ranks rather than raw values.

Selecting the Right Test for Real-World Data

One of the most practically important skills is choosing the correct test before analyzing data. The wrong test can lead to invalid conclusions, wasted effort, and — in high-stakes settings like medicine or policy — genuine harm. A systematic decision process helps:

Step 1 — Identify the scale of measurement of the outcome variable. Continuous or interval-scale data (height, income, reaction time, blood pressure) calls for tests based on means: t-tests or ANOVA. Categorical data (yes/no, color preference, disease category) calls for the chi-square test or related methods. Ordinal data (satisfaction ratings, ranks) may call for non-parametric tests.

Step 2 — Count the number of groups and their relationship. The following table summarizes the standard mapping:

Step 3 — Check assumptions. Every test has conditions for validity. Skipping this step is one of the most common errors in applied statistics. Run normality checks (Shapiro-Wilk test, Q-Q plots) for t-tests and ANOVA. Check expected cell counts for chi-square. Test for homogeneity of variances before choosing pooled vs. Welch's t-test. If key assumptions are violated, consider:

• Transforming the data (e.g., log transformation to reduce skewness) and then applying the parametric test to the transformed values.
• Non-parametric alternatives: Wilcoxon signed-rank test (instead of one-sample or paired t-test), Mann-Whitney U test (instead of two-sample t-test), Kruskal-Wallis test (instead of ANOVA).

Step 4 — Document your reasoning. Transparent, reproducible research requires explaining why a particular test was chosen, what assumption checks were performed, and how violations were handled. This documentation is as important as the analysis itself, especially in peer-reviewed science, regulatory filings, and audit-ready business analytics.

Interpreting and Communicating Test Results

Generating a p-value is only the beginning of the analytical work. Communicating what the result actually means — to fellow analysts, to managers, to the public — demands careful translation from statistical language to actionable insight.

State the conclusion in plain language. Rather than writing "t(24) = −2.5, p = 0.020, therefore we reject H₀ at α = 0.05," a more useful statement for a business report is: "On average, our large coffee cups are being filled at 15.7 ounces, which is statistically significantly less than the advertised 16 ounces (p = 0.020). This underfill is unlikely to be due to chance alone and warrants adjustment of the filling equipment."

Report a complete statistical summary. Best practice, as codified in guidelines from the American Statistical Association and the APA Publication Manual, includes reporting:

• The test statistic and its direction (e.g., t = −2.5)
• Degrees of freedom (e.g., df = 24)
• The exact p-value (e.g., p = 0.020), not just "p < 0.05"
• A confidence interval for the parameter of interest (e.g., 95% CI for the mean: [15.45, 15.95])
• An effect size measure (e.g., Cohen's d, η², Cramér's V for chi-square)

Distinguish statistical significance from practical significance. This is perhaps the most important interpretive lesson in modern statistics. A study with n = 10,000 may detect a 0.1-point improvement on a 100-point test as statistically significant (p < 0.001), but a 0.1-point improvement almost certainly has no meaningful real-world impact. Conversely, a small pilot study with n = 15 may fail to detect a genuinely important effect simply because it is underpowered. The p-value answers one narrow question — "Is there evidence against the null?" — but does not answer "Should we care?" or "Is this worth acting on?" Effect sizes and confidence intervals speak directly to those practical questions.

Acknowledge limitations honestly. Every study has constraints that bound the conclusions. Common limitations worth addressing include:

• Sample size: A small sample provides limited power and wide confidence intervals. A very large sample can detect trivially small effects.
• Sampling method: Results from a convenience sample may not generalize to the broader population of interest.
• Potential confounders: In observational (non-experimental) studies, a statistically significant association between two variables may reflect a third variable that influences both. Hypothesis tests do not establish causation.
• Assumption violations: If normality or equal variance assumptions were questionable and non-parametric alternatives were not used, the p-value may be unreliable.
• Multiple testing: If many tests were run on the same dataset without correction, some significant results may be false positives.

Together, these principles — choosing the right test, verifying assumptions, computing a complete set of statistics, and communicating with clarity and humility — form the foundation of rigorous, trustworthy statistical practice. The goal is not to "get a significant result" but to extract honest, defensible insight from data.