↑ Week 11/Module 10 - 2 Sample Hypothesis Testing

Module 10: Hypothesis Tests for Two Samples

Supporting Lectures:
EGN3443 Module 10 - Hypothesis Testing for 2 Samples

1. Independent Samples t-Test

Definition

An independent samples t-test compares the means of two separate, unrelated groups to determine if there is a statistically significant difference between them.

Key Characteristics

Calculation Steps

  1. State null and alternative hypotheses

  2. Calculate t-statistic: t = (x̄₁ - x̄₂) / √[(s₁²/n₁) + (s₂²/n₂)] Where:

Example Calculation

Group 1: x̄₁ = 10, s₁ = 2, n₁ = 30

Group 2: x̄₂ = 8, s₂ = 3, n₂ = 25

t = (10 - 8) / √[(2²/30) + (3²/25)]

Web References

Scenarios

An independent samples t-test is appropriate in the following scenarios:

  1. Comparing two treatment groups in a randomized controlled trial (e.g., comparing the mean blood pressure reduction between patients receiving a new medication versus a placebo)
  2. Comparing performance between two different groups that are naturally distinct (e.g., comparing mean test scores between students from two different schools)
  3. Comparing a product/process change with two independent samples (e.g., comparing the tensile strength of metal parts produced by two different manufacturing methods)
  4. Testing the effect of a binary categorical variable on a continuous outcome (e.g., comparing mean salary between male and female employees in a company)
  5. Comparing before-and-after measurements on completely different subjects (e.g., comparing customer satisfaction scores before and after a service change, using different customers in each period)
  6. Testing for differences between experimental conditions with different participants assigned to each condition (e.g., comparing reaction times between two interface designs with different users testing each design)
  7. Comparing naturally occurring groups on some measured variable (e.g., comparing the mean height of plants grown in two different soil types)
  8. Comparing two different methods or tools using independent samples (e.g., comparing the accuracy of two different diagnostic tests on separate groups of patients)
  9. Market research comparisons between distinct consumer segments (e.g., comparing average spending between urban and rural customers)
  10. Ecological studies comparing different populations or locations (e.g., comparing mean pollution levels between two different cities)
  11. Comparing intervention effects between separate groups (e.g., comparing the mean weight loss between two different diet programs with different participants in each program)
  12. Educational research comparing teaching methods with different classes (e.g., comparing test scores between students taught with traditional methods versus those taught with a new approach)

For all these scenarios, the key requirements are:

2. Paired Samples t-Test

Definition

A paired samples t-test compares the means of two related groups, typically measurements taken from the same subjects before and after an intervention.

Key Characteristics

Calculation Steps

State null and alternative hypotheses

H₀: μd = 0 (no difference in population means)
H₁: μd ≠ 0 (significant difference exists)

  1. Calculate differences between pairs

  2. Calculate mean difference (d̄)

  3. Calculate standard deviation of differences (sd)

  4. Compute t-statistic: t = d̄ / (sd / √n) Where:

Example Calculation

Before Treatment: [10, 12, 15, 18, 20]

After Treatment:  [9, 11, 14, 17, 19]

Differences:      [1, 1, 1, 1, 1]

Mean Difference: 1

Web References

Scenarios 

A paired samples t-test (also called dependent samples t-test or paired t-test) is appropriate in the following scenarios:

  1. Before-and-after measurements on the same subjects (e.g., comparing patients' blood pressure before and after receiving a treatment)
  2. Repeated measurements on the same subject under different conditions (e.g., measuring a person's reaction time when tired versus when well-rested)
  3. Matched pairs designs where subjects are deliberately matched on important characteristics (e.g., comparing treatment outcomes between pairs of twins, or patients matched by age, gender, and disease severity)
  4. Crossover studies where each subject receives both treatments in sequence (e.g., comparing the effectiveness of two medications where each participant tries both drugs with a washout period between)
  5. Testing the same samples under two different conditions (e.g., testing the same material samples with two different testing methods)
  6. Performance comparisons using the same individuals (e.g., comparing students' scores on two different types of assessments)
  7. Product comparisons where the same evaluators rate both products (e.g., having the same group of testers rate two different user interfaces)
  8. Longitudinal studies with two time points for the same participants (e.g., measuring cognitive ability at the beginning and end of an educational program)
  9. Split-plot agricultural experiments where measurements are taken from adjacent plots of land (e.g., comparing crop yields from two different fertilizers applied to adjacent sections of the same fields)
  10. Comparing measurements taken from symmetrical body parts (e.g., comparing strength between dominant and non-dominant hands in the same individuals)
  11. Software or algorithm comparison using the same datasets (e.g., comparing the accuracy of two different machine learning algorithms tested on the same datasets)
  12. Taste tests where the same participants evaluate multiple products (e.g., comparing consumer ratings of two different food products tasted by the same panel)

For all these scenarios, the key requirements are:

The paired t-test is particularly valuable when there's high variability between subjects but relatively consistent differences within pairs, as the pairing helps to eliminate between-subject variability and increase statistical power.

3. F-Test for Comparing Variances

Definition

An F-test determines whether two population variances are significantly different by comparing their sample variances.

Key Characteristics

Calculation Steps

  1. State null and alternative hypotheses

  2. Calculate F-statistic: F = larger sample variance / smaller sample variance

Example Calculation

Group 1 Variance: s₁² = 25

Group 2 Variance: s₂² = 16

F = 25 / 16 = 1.5625

Web References

Scenarios

An F-test for comparing variances is appropriate in the following scenarios:

  1. Testing homogeneity of variance before performing a t-test or ANOVA (e.g., checking if the variability in test scores is similar across different teaching methods)
  2. Comparing precision between two measurement methods (e.g., determining if one laboratory technique produces more consistent results than another)
  3. Quality control comparisons between manufacturing processes (e.g., comparing the consistency of parts produced by two different machines)
  4. Assessing variability in biological systems (e.g., comparing the genetic diversity between two populations)
  5. Comparing variability in performance between groups (e.g., determining if experienced workers have more consistent output than novices)
  6. Instrument validation studies comparing the precision of two different instruments (e.g., comparing the variance in repeated measurements from two different sensors)
  7. Comparing treatment consistency in pharmaceutical research (e.g., determining if one drug formulation produces more variable effects than another)
  8. Comparing risk or volatility in financial investments (e.g., comparing the variance in returns between two different investment strategies)
  9. Testing if experimental conditions affect variability rather than just the mean (e.g., determining if a treatment not only changes the average response but also makes responses more consistent)
  10. Comparing environmental variability between sites (e.g., comparing temperature fluctuations between two different ecosystems)
  11. Assessing consistency in human performance under different conditions (e.g., comparing the variability in reaction times between two different interface designs)
  12. Testing if one group is more heterogeneous than another (e.g., comparing if student performance is more variable in one classroom compared to another)

For all these scenarios, the key requirements are:

It's important to note that F-tests for comparing variances are relatively sensitive to departures from normality. If the normality assumption is violated, alternative tests like Levene's test or Brown-Forsythe test may be more appropriate for comparing variances.

4. Tests for Comparing Proportions

Definition

Tests for comparing proportions determine if there's a significant difference between the proportions of two populations or groups.

Key Techniques

Calculation Steps

  1. State null and alternative hypotheses

  2. Calculate pooled proportion

  3. Compute z-statistic: z = (p₁ - p₂) / √[p(1-p)(1/n₁ + 1/n₂)]

Example Calculation

Group 1: 30 successes out of 100

Group 2: 45 successes out of 150

Pooled proportion calculation

Web References

Two-Proportion Z-Test Appropriate Scenarios:

  1. Comparing vaccination rates between two different demographic groups (e.g., comparing COVID-19 vaccination rates between urban and rural populations)
  2. Comparing success rates of two different treatments or interventions (e.g., comparing the recovery rates of patients receiving two different surgical procedures)
  3. A/B testing in marketing (e.g., comparing conversion rates between two different webpage designs)
  4. Comparing defect rates between two manufacturing processes (e.g., determining if one assembly line produces fewer defective products than another)
  5. Comparing response rates between two survey methods (e.g., determining if online surveys yield higher response rates than mail surveys)
  6. Comparing pass/fail rates between two different educational approaches (e.g., comparing the proportion of students passing an exam under traditional vs. flipped classroom teaching)
  7. Comparing adoption rates of two different products or technologies (e.g., comparing the percentage of businesses adopting cloud storage vs. on-premise solutions)
  8. Comparing click-through rates in digital marketing (e.g., comparing the proportion of users who click on two different ad designs)
  9. Comparing infection rates between two populations (e.g., comparing the proportion of infections between vaccinated and unvaccinated groups)
  10. Comparing germination rates in agricultural experiments (e.g., comparing the percentage of seeds that germinate under two different conditions)

Chi-Squared Test of Independence Appropriate Scenarios:

  1. Testing for associations between categorical variables (e.g., determining if educational level is associated with voting preference)
  2. Comparing distributions across multiple categories (e.g., determining if the distribution of product preferences differs between age groups)
  3. Testing for differences in frequency distributions (e.g., comparing if the distribution of blood types differs between two populations)
  4. Market segmentation analysis (e.g., determining if product preferences are independent of demographic factors)
  5. Comparing categorical outcomes across groups (e.g., comparing diagnostic outcomes between different hospitals)
  6. Analyzing survey responses with multiple categorical options (e.g., determining if response patterns to a multiple-choice question differ between groups)
  7. Testing for changes in categorical distributions over time (e.g., comparing if the distribution of transportation methods changed before and after a policy change)
  8. Analyzing contingency tables with multiple rows and columns (e.g., examining the relationship between educational background and career choice)
  9. Comparing product preferences across demographic groups (e.g., determining if beverage preferences differ by age group)
  10. Analyzing patterns in categorical data (e.g., determining if housing preferences are independent of income level)

Both Tests Are Appropriate When:

  1. Comparing binary outcomes between two groups (e.g., comparing the proportion of "yes" responses between two survey groups)
  2. Testing if the proportion of a characteristic differs between two populations (e.g., comparing the percentage of left-handed people in two different countries)
  3. Comparing event rates between two groups (e.g., comparing accident rates between two different manufacturing facilities)
  4. Testing for differences in binary classifications (e.g., comparing the proportion of positive test results between two laboratories)
  5. Analyzing 2×2 contingency tables (e.g., comparing smokers vs. non-smokers in two different regions)

For all these scenarios, the key requirements are:

Note that for 2×2 tables, the two-proportion z-test and chi-squared test of independence are mathematically equivalent (the chi-squared statistic is the square of the z-statistic).

5. Non-Parametric Alternatives

Definition

Non-parametric tests are used when data does not meet parametric test assumptions (normality, equal variance).

Key Non-Parametric Tests

Characteristics

Web References

Mann-Whitney U Test / Wilcoxon Rank-Sum Test (Alternative to Independent t-test)

  1. Comparing ordinal data between two groups (e.g., comparing customer satisfaction ratings between two service providers)
  2. Analyzing data with outliers that would skew parametric tests (e.g., comparing income distributions between two regions when some extreme values exist)
  3. Small sample sizes where normality cannot be verified (e.g., comparing performance metrics between two small teams)
  4. Comparing data from clearly non-normal distributions (e.g., comparing response times that follow a skewed distribution)
  5. Comparing ranked data between independent groups (e.g., comparing athletic performance rankings between two training programs)
  6. When transformations fail to normalize the data (e.g., comparing biological measurements that remain non-normal after attempted transformations)
  7. Comparing scores on assessment tools with ceiling or floor effects (e.g., comparing scores on a questionnaire where many respondents hit the maximum score)
  8. When comparing medians is more appropriate than comparing means (e.g., comparing home prices between two neighborhoods)

Wilcoxon Signed-Rank Test (Alternative to Paired t-test)

  1. Before-and-after measurements when differences are not normally distributed (e.g., comparing pain scores before and after treatment)
  2. Paired comparisons with outliers (e.g., comparing reaction times to two stimuli when some responses are extremely slow)
  3. Small sample paired data (e.g., comparing biomarker levels in a small group of patients before and after intervention)
  4. Ordinal paired measurements (e.g., comparing subjects' preference ratings for two products)
  5. Ranking performance under two different conditions with the same subjects (e.g., comparing athletes' performance rankings under two different training regimens)
  6. When the magnitude of differences varies widely among pairs (e.g., comparing blood pressure changes where some patients show dramatic changes while others show minimal changes)
  7. Cross-over trials with non-normal outcome distributions (e.g., comparing the effects of two treatments in patients who receive both treatments in sequence)

Sign Test (Simplest Alternative to Paired t-test)

  1. When only the direction of difference matters, not the magnitude (e.g., determining if one product is consistently preferred over another)
  2. Extremely skewed distributions where even ranks might be inappropriate (e.g., comparing datasets with extreme outliers)
  3. Paired comparisons where only "greater than," "less than," or "equal to" information is available (e.g., comparing whether performance improved, deteriorated, or stayed the same after training)
  4. When the exact values cannot be measured but the relative ordering can (e.g., comparing taste preferences between two foods)
  5. Very simple before-after comparisons (e.g., comparing if a treatment leads to improvement or deterioration)

Fisher's Exact Test (Alternative to Chi-Square Test for Small Samples)

  1. Small sample comparisons of categorical data where expected cell counts are low (e.g., comparing rare disease occurrences between two small groups)
  2. 2×2 contingency tables with fewer than 5 expected observations in any cell (e.g., comparing adverse event occurrences between two small treatment groups)
  3. Precise probability calculations for contingency tables (e.g., determining exact probability of observed distribution in genetic studies with small samples)
  4. Testing association in small samples (e.g., comparing genetic mutations in small cohorts)

McNemar's Test (Alternative for Paired Nominal Data)

  1. Before-and-after comparisons with binary outcomes (e.g., comparing presence/absence of symptoms before and after treatment)
  2. Matched-pairs designs with binary outcomes (e.g., comparing diagnostic test results in pairs of siblings)
  3. Crossover studies with binary outcomes (e.g., comparing presence/absence of side effects under two different treatments)
  4. Comparing binary responses from the same subjects under different conditions (e.g., comparing agreement/disagreement to statements before and after an informational intervention)

Key Considerations for All Non-parametric Tests:

In general, non-parametric tests are appropriate whenever the assumptions of the corresponding parametric test cannot be met, particularly the assumption of normally distributed data or when dealing with ordinal data.

Conclusion

Understanding these hypothesis tests provides powerful tools for comparing groups and making statistical inferences across various research and practical scenarios.