Week 10/Module 9 - Hypothesis Testing II — Module Topics

Two-Sample Hypothesis Tests

• Logic of Two-Sample Hypothesis Testing — Two-sample hypothesis tests are used to determine whether there is a statistically significant difference between the means (or proportions) of two independent groups.
• Independence Assumption and Sampling — A fundamental requirement of two-sample tests is that the two groups must be independent of each other, meaning observations in one group do not influence or relate to observations in the other.
• Two-Sample Z-Test — The two-sample z-test compares the means of two independent groups when population standard deviations are known and/or sample sizes are large.
• Two-Sample T-Test — The two-sample t-test is used when population standard deviations are unknown and sample sizes are small, relying on estimated standard errors and the t-distribution.
• Assumptions Underlying Two-Sample Tests — Both two-sample z-tests and t-tests rely on a set of statistical assumptions that must be evaluated before the results can be considered valid.
• Interpreting Results and Making Decisions — After computing the test statistic, the result is compared to a critical value or evaluated using a p-value to determine whether to reject the null hypothesis.

Paired Sample Comparisons

• What Are Paired Sample Designs? — Paired sample designs involve collecting two related measurements from the same subject or from matched subjects, rather than from two independent groups.
• Why Pairing Reduces Variability — The primary statistical advantage of pairing is that it removes between-subject variability from the error term, making the test more sensitive to true differences.
• Computing the Paired Difference Score — The foundation of all paired-sample inference is the difference score D, calculated for each pair as the value in condition one minus the value in condition two.
• The Paired-Sample t-Test — The paired-sample t-test evaluates whether the mean of the difference scores is significantly different from zero, using a t-distribution with n − 1 degrees of freedom.
• Assumptions of the Paired-Sample t-Test — Like all parametric tests, the paired-sample t-test rests on several assumptions that must be reasonably satisfied for results to be valid.
• Interpreting and Reporting Results — Proper interpretation of a paired-sample t-test includes reporting the test statistic, degrees of freedom, p-value, and a measure of effect size to convey practical significance.
• When to Choose a Paired vs. Independent Design — Selecting the correct design and corresponding test depends on the research question, how data were collected, and the nature of the relationship between observations.

Selecting the Appropriate Statistical Test

• The Decision-Making Framework Overview — Selecting the correct statistical test requires a structured decision process rather than guesswork. A systematic framework helps researchers avoid errors that lead to invalid conclusions.
• Identifying Data Type and Measurement Level — The scale of measurement of your outcome variable is the first and most critical factor in test selection. Tests designed for continuous data cannot be validly applied to categorical data, and vice versa.
• Assessing Sample Size and Distributional Assumptions — Many parametric tests assume the sampling distribution of the statistic is approximately normal, an assumption that depends heavily on sample size and the underlying population distribution. Evaluating these conditions guides the choice between parametric and non-parametric methods.
• Determining Independence vs. Dependence of Samples — Whether your samples are independent or related (paired/matched) is a fundamental branching point in test selection. Using an independent-samples test on paired data wastes statistical power and may produce incorrect results.
• Considering the Number of Groups or Samples — The number of groups being compared directly determines the class of test to apply. Two-group comparisons and multi-group comparisons require fundamentally different procedures.
• Aligning Test Choice with the Research Question — Beyond data characteristics, the specific inferential goal—testing differences, associations, or relationships—shapes which test is appropriate. A test that answers the wrong question produces irrelevant results even if technically applied correctly.
• Practical Checklist for Final Test Selection — A concise checklist consolidates all decision criteria into a quick-reference tool that can be applied before any analysis begins. Running through each checkpoint reduces the risk of selecting an inappropriate test.

Assumptions and Conditions for Validity

• Why Assumptions Matter in Inferential Testing — Every inferential technique rests on a set of underlying assumptions; violating these can lead to invalid p-values, incorrect confidence intervals, and misleading conclusions.
• Independence of Observations — Most parametric and many non-parametric tests require that observations be independent of one another, meaning the value of one data point does not influence another.
• Normality Assumption and How to Verify It — Parametric tests such as the t-test assume that the population distribution (or the sampling distribution of the statistic) is approximately normal.
• Homogeneity of Variance (Equal Variances) — Independent two-sample t-tests in their classical form assume that the two populations have equal variances, a condition known as homoscedasticity.
• Sample Size and the Conditions for Each Test — Adequate sample size is a practical condition that affects whether the theoretical assumptions of a test are reasonably satisfied and whether the test has sufficient power.
• Conditions Specific to Paired Comparisons — The paired t-test requires that the differences between paired observations follow an approximately normal distribution, rather than requiring normality of the original measurements themselves.
• Using Non-Parametric Methods When Assumptions Fail — Non-parametric tests make fewer distributional assumptions and serve as valid alternatives when parametric conditions cannot be met.

Introduction to Non-Parametric Methods

• What Are Non-Parametric Methods? — Non-parametric methods are statistical tests that do not rely on assumptions about the underlying population distribution, such as normality.
• When to Use Non-Parametric Tests — Non-parametric tests are used when the assumptions required by parametric tests — such as normality or homogeneity of variance — cannot be reasonably satisfied.
• Advantages of Non-Parametric Methods — Non-parametric methods offer flexibility and robustness, particularly in real-world datasets that do not conform to idealized statistical assumptions.
• Limitations and Trade-offs — While non-parametric tests are versatile, they come with trade-offs, most notably reduced statistical power compared to their parametric counterparts when parametric assumptions are actually met.
• Common Non-Parametric Alternatives to Parametric Tests — For most standard parametric tests, there exists a non-parametric equivalent that can be applied when the necessary assumptions are not met.
• Selecting the Right Test: Parametric vs. Non-Parametric — Choosing between a parametric and non-parametric test requires careful consideration of the data type, sample size, and the degree to which distributional assumptions are satisfied.

Interpreting and Communicating Results

• Understanding the Test Statistic — A test statistic summarizes how far the observed sample data deviates from what would be expected under the null hypothesis, expressed in standardized units.
• Interpreting P-Values Correctly — The p-value represents the probability of obtaining a test statistic at least as extreme as the one observed, assuming the null hypothesis is true.
• Significance Levels and Decision Thresholds — The significance level (α) is the pre-determined threshold used to decide whether a p-value is small enough to reject the null hypothesis.
• Using Confidence Intervals to Complement Hypothesis Tests — Confidence intervals provide a range of plausible values for a population parameter and offer additional context beyond a simple reject-or-fail-to-reject decision.
• Distinguishing Statistical Significance from Practical Significance — A statistically significant result does not necessarily mean the finding is large enough to matter in practice; practical significance depends on the size and real-world relevance of the effect.
• Reporting Hypothesis Test Results Clearly — Communicating hypothesis test results requires reporting all relevant statistical information in a structured and transparent manner so that readers can evaluate the evidence independently.
• Common Misinterpretations to Avoid — Several widespread misconceptions about hypothesis testing can lead to flawed conclusions and poor scientific communication.