Hypothesis testing is how a team decides whether a difference in its data is real or just random variation. You state a default claim of no difference (the null hypothesis), gather data, and ask how surprising the result would be if that claim were true. If it would be very surprising, you have evidence of a real effect.
Used well, it keeps improvement work honest. It stops a team from celebrating a lucky week, and it stops it from dismissing a real improvement because the sample was too small. This guide covers the vocabulary, the decision logic, how to pick a test, and a worked example from a production line.
Why Hypothesis Testing Matters
Separates Signal From Noise
Any two samples differ a little by chance. A test tells you whether the difference you see is bigger than chance alone would usually produce.
Makes Improvement Claims Defensible
"The new fixture is faster" becomes a statement with evidence and a stated risk of being wrong, which holds up in a design review or customer audit.
Prevents Costly Overreaction
Chasing every shift-to-shift wiggle wastes engineering time. A test stops teams from changing a process because of random variation.
Core of the Analyze Phase
DMAIC's Analyze phase uses tests to confirm which suspected causes actually affect the output before any improvement is designed.
Core Terms
| Term | Meaning |
|---|---|
| Null hypothesis (H₀) | The default claim of no difference or no effect, for example the two machines have the same mean cycle time. |
| Alternative hypothesis (H₁) | What you are looking for evidence of: the means differ (two-sided), or one is greater (one-sided). |
| Significance level (α) | The risk of a false alarm you are willing to accept. 0.05 is the common default. |
| p-value | The probability of seeing a result at least this extreme if the null hypothesis were true. It is not the probability that the null hypothesis is true. |
| Type I error | Rejecting a true null hypothesis (a false alarm). Its probability is α. |
| Type II error | Failing to reject a false null hypothesis (a missed real effect). Its probability is β. |
| Power (1 − β) | The probability of detecting a real effect of a given size. 80% or more is a typical target. |
The Four Possible Outcomes
| H₀ is actually true (no real difference) | H₀ is actually false (real difference) | |
|---|---|---|
| Test rejects H₀ | Type I error (false alarm), probability α | Correct: real effect detected, probability = power |
| Test does not reject H₀ | Correct: no effect, none claimed | Type II error (missed effect), probability β |
Lowering α makes false alarms rarer but, with the same sample size, makes missed effects more likely. The reliable way to reduce both is more data.
The Seven Steps
- State the practical question in plain words, for example "Does Line B run faster than Line A?"
- Write H₀ and H₁ in terms of a parameter (a mean, a proportion, a standard deviation).
- Choose α and the smallest difference that matters to the business, then use the Sample Size Calculator to plan how much data you need.
- Collect the data with random, representative sampling, after confirming the measurement system is trustworthy.
- Check assumptions such as independence, approximate normality and whether variances look similar. Plot the data first.
- Run the test and read the p-value together with a confidence interval for the size of the difference.
- Decide and act. Reject H₀ only if the p-value is below α, then ask whether the difference is large enough to matter in practice.
Choosing the Right Test
| Your question | Data type | Common test | If assumptions fail |
|---|---|---|---|
| Is the mean different from a target? | Continuous | 1-sample t-test | 1-sample Wilcoxon |
| Do two independent groups have different means? | Continuous | 2-sample t-test (Welch) | Mann-Whitney |
| Did the same units change before vs after? | Continuous, paired | Paired t-test | Paired Wilcoxon |
| Do three or more means differ? | Continuous | One-way ANOVA | Kruskal-Wallis |
| Do two variances or standard deviations differ? | Continuous | F-test, Levene's test | Levene's test (robust) |
| Is a defect rate different from a target or another rate? | Pass/fail counts | 1- or 2-proportion test | Fisher's exact test |
| Are two categorical variables related? | Counts in categories | Chi-square test of association | Fisher's exact test (small counts) |
The Hypothesis Testing Quick Tester runs several of these from your own numbers and shows the result in plain language.
Worked Example: Do Two Lines Have Different Cycle Times?
A plant wants to know whether Line B, which uses a new fixture, has a shorter cycle time than Line A. The numbers are illustrative. The team times 30 consecutive units on each line.
| n | Mean cycle time (s) | Std. deviation (s) | |
|---|---|---|---|
| Line A (current fixture) | 30 | 12.4 | 1.1 |
| Line B (new fixture) | 30 | 11.8 | 1.3 |
- H₀: μA = μB. H₁: μA ≠ μB. Use α = 0.05.
- Choose a two-sample t-test without assuming equal variances (Welch's test), since the standard deviations differ slightly.
- Difference in means = 12.4 − 11.8 = 0.6 s.
- Standard error = √(1.1²/30 + 1.3²/30) = 0.311 s.
- t = 0.6 / 0.311 = 1.93, with about 56 degrees of freedom.
- Two-sided p-value = 0.059. The 95% confidence interval for the difference is about −0.02 to +1.22 s.
How to read this. Because p (0.059) is above 0.05, the team cannot claim a difference at the 5% level. That is not the same as proving the lines are equal. The confidence interval runs from essentially zero to over a second, so a real improvement of a half second or more is entirely plausible. The sample was simply too small to be sure.
With a standard deviation of about 1.2 s, detecting a true 0.6 s difference with 80% power at α = 0.05 needs roughly 63 units per line (n ≈ 2 × (1.96 + 0.84)² × 1.2² / 0.6²). Collecting more data, not re-running the test until it "works," is the honest next step.
Statistical Versus Practical Significance
With enough data, even a trivial difference becomes statistically significant. A 0.05-second saving on a 12-second cycle may produce a tiny p-value on a large sample and still be worthless. Always compare the size of the effect, shown by the confidence interval, with the smallest difference the business cares about. The reverse also holds: a large, valuable effect can fail to reach significance when the sample is small, as in the example above.
Self-Assessment Questions
- Did we state H₀, H₁ and α before looking at the data?
- Did we decide how big a difference matters and plan the sample size to detect it?
- Is the measurement system good enough to see a difference of that size?
- Did we plot the data and check the assumptions of the test we chose?
- Are we reporting the confidence interval and effect size, not only the p-value?
- If the result was not significant, did we avoid concluding that there is no difference?
Common Mistakes
Reading p as the Chance H₀ Is True
A p-value of 0.03 does not mean a 3% chance the null hypothesis is true. It describes how surprising the data would be if it were.
Treating "Not Significant" as "No Difference"
Failing to reject H₀ can simply mean too little data or too much noise. Check the confidence interval and power.
Testing Until Something Is Significant
Running many tests on the same data inflates the false-alarm rate. Decide what to test before collecting data.
Ignoring the Measurement System
If gauge variation is large compared with the effect, the test cannot tell you much. Run a Gage R&R first.
Quick Reference
Before Collecting Data
- Write the question, H₀ and H₁.
- Set α and the smallest difference that matters.
- Calculate the sample size for the power you want.
- Confirm the measurement system is capable.
Before Reporting
- Check assumptions with plots, not only test results.
- Report the p-value and a confidence interval.
- Judge practical importance, not just significance.
- Say what you could not conclude, not only what you could.
Hypothesis Testing for Process Improvement: Frequently Asked Questions
What is a p-value in simple terms?
A p-value is the probability of getting a result at least as extreme as the one observed if the null hypothesis, usually 'no difference,' were true. A small p-value, typically below 0.05, means the data would be unlikely under the null hypothesis, which is taken as evidence against it. It is not the probability that the null hypothesis is true.
What is the difference between Type I and Type II errors?
A Type I error is rejecting a true null hypothesis, a false alarm, and its probability is the significance level alpha. A Type II error is failing to reject a false null hypothesis, a missed real effect, and its probability is beta. Power, which is 1 minus beta, is the chance of detecting a real effect of a given size.
Does a non-significant result mean there is no difference?
No. A non-significant result means the data did not provide strong enough evidence of a difference, which may be because the sample was small or the variation large. The confidence interval for the difference shows which sizes of effect are still plausible, and a power calculation shows whether the test could have detected the effect you care about.
Sources and Further Reading
- Douglas C. Montgomery and George C. Runger, Applied Statistics and Probability for Engineers.
- NIST/SEMATECH e-Handbook of Statistical Methods, sections on hypothesis tests and t-tests.
- Forrest W. Breyfogle III, Implementing Six Sigma.
- ASQ Certified Six Sigma Black Belt Body of Knowledge.