Module 7: Point Estimation and Confidence Intervals

Supporting Lectures:
EGN3443 Module 7 - Statistical Estimation Theory and Practice

What is Estimation?

In statistics, an estimator is a rule for calculating an estimate of a given quantity based on observed data: thus the rule (the estimator), the quantity of interest (the estimand) and its result (the estimate) are distinguished.[1] For example, the sample mean is a commonly used estimator of the population mean.

There are point and interval estimators. The point estimators yield single-valued results. This is in contrast to an interval estimator, where the result would be a range of plausible values. "Single value" does not necessarily mean "single number", but includes vector valued or function valued estimators.

Estimation theory is concerned with the properties of estimators; that is, with defining properties that can be used to compare different estimators (different rules for creating estimates) for the same quantity, based on the same data. Such properties can be used to determine the best rules to use under given circumstances. However, in robust statistics, statistical theory goes on to consider the balance between having good properties, if tightly defined assumptions hold, and having worse properties that hold under wider conditions.

1. Properties of Good Estimators

Definition

An estimator is a statistical function that provides an estimate of a population parameter based on sample data. The quality of an estimator is determined by three key properties:

1.1 Unbiasedness

• Definition: An estimator is unbiased if its expected value equals the true population parameter.
• Mathematical Representation: E(θ̂) = θ
• Example: Sample mean (x̄) is an unbiased estimator of population mean (μ)
• Calculation:

  E(x̄) = μBias = E(θ̂) - θ = 0
  

1.2 Efficiency

• Definition: Among unbiased estimators, the most efficient has the smallest variance.
• Measure: Cramér-Rao Lower Bound
• Example: Maximum likelihood estimators are often the most efficient
• Calculation: Variance efficiency = Var(θ̂) / Minimum possible variance

1.3 Consistency

• Definition: An estimator converges to the true parameter value as sample size increases
• Mathematical Representation: P(|θ̂ - θ| → 0) = 1 as n → ∞
• Example: Sample variance becomes more accurate with larger samples

1.4 Web References

Wikipedia: https://en.wikipedia.org/wiki/Estimator 

2. Maximum Likelihood Estimation (MLE)

Definition

Maximum Likelihood Estimation is a method of estimating the parameters of a statistical model by maximizing the likelihood function.

Key Steps:

1. Construct the likelihood function
2. Take the natural log (log-likelihood)
3. Differentiate with respect to parameters
4. Set derivatives to zero
5. Solve for parameter estimates

Example: Bernoulli Distribution

• Likelihood Function: L(p) = p^x * (1-p)^(1-x)
• Log-Likelihood: ln(L) = x * ln(p) + (1-x) * ln(1-p)
• MLE Estimate: p̂ = x̄ (sample mean)

Web References:

• Wikipedia - https://en.wikipedia.org/wiki/Maximum_likelihood_estimation 

Sample Problems

• Module 7 Sample Problem 1 
• Module 7 Sample Problem 2 
• Note: In both sample problems, we went through the formal steps (above in 2). Because both problems follow a Bernoulli distribution the maximum likelihood estimator for the parameter p is the simply the mean of the sample. This is not true of all distributions, distributions with greater complexity are beyond the scope of this class. However, the full methodology for the solution is still shown. 
• MLE for Bernoulli: In both examples, we demonstrated that the maximum likelihood estimator for the parameter $p$p of a Bernoulli distribution is simply the sample mean (proportion of successes).
• Profile Likelihood CI: We used profile likelihood to construct confidence intervals, which can be more accurate than Wald intervals, especially when the sample size is small or $p$p is close to 0 or 1.
• Fisher Information: In the second example, we calculated the standard error using Fisher Information, which for a Bernoulli distribution is $\sqrt{\frac{p(1-p)}{n}}$np(1−p)​​.
• Optimization Methods: Both analytical solutions and numerical optimization approaches to find the MLE are shown.

3. Confidence Intervals for Means

Definition

A confidence interval provides a range of plausible values for a population parameter with a specified level of confidence.

Formula for Population Mean (Known σ):

CI = x̄ ± z(α/2) * (σ / √n)

• x̄: Sample mean
• z(α/2): Critical value from standard normal distribution
• σ: Population standard deviation
• n: Sample size

Example Calculation:

• Sample mean (x̄) = 50
• Population σ = 10
• Sample size (n) = 36
• Confidence Level = 95%

z(0.025) = 1.96
CI = 50 ± 1.96 * (10 / √36)
CI = 50 ± 3.27
CI = (46.73, 53.27)

Web References:

• NIST - Confidence Intervals
• Wikipedia: https://en.wikipedia.org/wiki/Confidence_interval 
• Khan Academy - Confidence Intervals

Sample Problems

• Module 7 Sample Problem 3 
• Module 7 Sample Problem 4 

• From both examples, we can observe the following important principles about confidence intervals:

  1. Sample Size Impact: As seen in Example 2 with a larger sample size (n=40), we get a narrower confidence interval compared to Example 1 (n=25), even though Example 2 uses a higher confidence level (99% vs 95%).
  2. Confidence Level Effect: Higher confidence levels result in wider intervals, providing more certainty but less precision.
  3. Distribution Shape: The t-distribution approaches the normal distribution as sample size increases, which is why the margin of error in Example 2 is relatively smaller despite the higher confidence level.
  4. Practical Interpretation: In the engineering context, confidence intervals help quantify uncertainty in measurements of critical parameters like material strength or component lifetime, allowing for safer design decisions.

Bonus 7- 1 Interactive Estimation Problem 

4. Confidence Intervals for Proportions

Definition

Confidence interval for a population proportion estimates the range of true population proportion.

Population proportion is the fraction of individuals in a population that possess a certain characteristic, calculated as the number of individuals with that characteristic divided by the total population size.  For example If a health survey indicates that 150 out of 500 surveyed individuals are smokers, the proportion of smokers in the sample is 
p = 150/500 = 0.3 or 30%.

Formula:

CI = p̂ ± z(α/2) * √[(p̂ * (1-p̂)) / n]

• p̂: Sample proportion
• z(α/2): Critical value
• n: Sample size

Example Calculation:

• Sample proportion (p̂) = 0.4
• Sample size (n) = 100
• Confidence Level = 90%

z(0.05) = 1.645
CI = 0.4 ± 1.645 * √[(0.4 * 0.6) / 100]
CI = 0.4 ± 0.082
CI = (0.318, 0.482)

Web References:

• Open Library Introduction to Statistics 7.4 https://ecampusontario.pressbooks.pub/introstats/chapter/7-4-confidence-intervals-for-a-population-proportion/ 

Sample Problems

• Open Library Introduction to Statistics 7.4 https://ecampusontario.pressbooks.pub/introstats/chapter/7-4-confidence-intervals-for-a-population-proportion/ 
• Module 7 Sample Problem 5 

5. Tolerance Intervals in Engineering Applications

Definition

Tolerance intervals provide a range expected to contain a specified proportion of a population with a given confidence level.

Types:

1. Normal Distribution Tolerance Interval
2. Non-parametric Tolerance Interval
3. Prediction Intervals

Key Formulas:

• k-factor method
• Based on sample mean and standard deviation
• Accounts for sample size and desired coverage

Example: Manufacturing Tolerance

• Part diameter specifications
• Determining acceptable manufacturing variation
• Ensuring 99% of parts are within ±0.1mm of nominal size

Web References:

• NIST Engineering Statistics Handbook - Tolerance intervals for a normal distribution
• Wikipedia - Engineering tolerance 
• Six Sigma - Statistical Tolerancing 

Recommended Study Resources:

1. Montgomery's "Design of Experiments"
2. Casella & Berger's "Statistical Inference"
3. Online Coursera/edX Statistics Courses