DOE Toolbox

What the Frame Step Is For

A designed experiment is a question put to the process. The design, the runs, and the analysis can only answer the question that was asked, with the response that was measured, over the ranges that were tried. Frame is the step that makes sure all three are right before time, materials, and line time are spent.

The Frame Step, Step by Step

1. 1

   Confirm a designed experiment is the right tool

   Compare your situation with the cases in the table below. If the cause is known, fix it. If only one factor is in question, test it directly. If you cannot set the inputs on purpose, use observed data.

   • Check that you can control the factors you want to test
   • Check that the response can be measured
   • Check that the process is stable enough to learn from

   Output: A decision: run an experiment, or use another method

   Watch for: Reaching for a DOE because it looks rigorous, not because the question needs it

2. 2

   State the objective as a question and a decision

   Write what you want to learn and what you will do with the answer. Choose the type of experiment: screen many factors, characterize a few, optimize, make the process robust, or confirm a result.

   • One sentence for the question
   • One sentence for the decision it will drive
   • Name the sponsor who will make that decision

   Output: An objective and an experiment type

   Watch for: An objective like “understand the process”, which no result can fulfil

3. 3

   Choose the response and check how well you can measure it

   Pick one primary response, preferably continuous, with a unit, a direction (larger, smaller, on target), and an exact method of measurement. Run a measurement system study.

   • Define how, when, and by whom the response is measured
   • Check resolution against the process variation
   • Run a Gage R&R, or the nested version for a destructive test

   Output: A defined response and a measurement noise estimate

   Watch for: Using a pass/fail judgment, or a gauge that no one has ever checked

4. 4

   Establish the baseline

   Collect or find data on the response under current settings. Plot it in time order and as a histogram. Estimate the mean, the variation from run to run, and whether the process is stable.

   • Use at least 30 recent observations
   • Look for trends, shifts, and mixtures
   • Record how the response relates to the specification

   Output: Baseline mean and standard deviation, and a stability check

   Watch for: Experimenting on an unstable process, where drift will look like an effect

5. 5

   List the candidate factors

   Brainstorm every input that could plausibly move the response. Use a fishbone diagram, a P-diagram, the process map, past data, and the people who run the process.

   • Include operators, maintenance, and engineers
   • Look at what changed when the problem began
   • Record how each factor could be set, and how precisely

   Output: A long list of candidate factors

   Watch for: A list drawn only from the engineer’s favorite theories

6. 6

   Classify the factors and choose which to test

   Sort factors into controllable factors you can set, noise factors you cannot control, and factors to hold constant. Choose the controllable factors to put in the experiment, and decide how to handle the rest.

   • Hold constant what you can
   • Block or record what you cannot hold
   • Prefer more factors at wide ranges over fewer at narrow ranges

   Output: The factors to vary, the factors to hold, and a plan for noise

   Watch for: Letting a noise factor, such as the material lot, change halfway through

7. 7

   Set the levels

   Choose a low and a high level for each factor. Make them wide enough to move the response, and safe enough that every combination can be run. Keep the current setting between them if possible so center points can be added.

   • Ask the process experts for the widest safe range
   • Check the extreme combinations for safety and feasibility
   • Record exactly how each level will be set and verified

   Output: A table of factors, units, and levels

   Watch for: Levels so close together that a real effect disappears into the noise

8. 8

   Set the constraints and the budget

   List how many runs you can afford in material, line time, test time, and money, and what the problem is costing. Check safety, quality, and regulatory approvals.

   • Count the runs and the units per run
   • Compare the cost with the value of the answer
   • Get approval for the line time and the scrap

   Output: A run and cost budget, and approvals

   Watch for: Finding out halfway that you cannot afford the runs you planned

9. 9

   Write the experiment frame and get it signed off

   Put everything on one page. Review it with the sponsor, the process owner, and the measurement owner. Agree what a successful result would look like and what will be done with it.

   • One page, readable by someone who was not there
   • Sign-off before the Design step starts
   • Keep it with the project record

   Output: A one-page experiment frame, approved

   Watch for: Starting the design while the sponsor still has a different question in mind

Is a Designed Experiment the Right Tool?

A designed experiment is the strongest way to find cause and effect, but it costs runs, time, and disruption. Check the situation first.

Writing the Objective

The objective has two parts: what you want to learn, and the decision it will drive. Without the second part, the experiment produces numbers but not action.

Large problems are usually solved with a sequence of small experiments: screen, then characterize, then optimize, then confirm. Plan to spend no more than about a quarter of the available runs on the first experiment, so something is left to follow up. The Design tab describes how to choose.

Choosing the Response

The response is the output you will measure in every run. It is the most important choice in the frame, and the one that most often goes wrong.

Why it matters, with numbers. The weld leak rate is 4.2%. To see the leak rate fall to 2% with 80% power would need about 975 housings per group. The burst pressure behind the leaks varies with a standard deviation of 22.4 kPa, so seeing a 20 kPa change takes only about 20 housings per group. The measurement is about 49 times cheaper to experiment on.

• Choose a response that is close to the cause, and measured directly, such as burst pressure, not the later symptom, such as field returns.
• Use one primary response. Add a few secondary responses you will watch (cost, cycle time, appearance), and decide how to trade them off.
• Write the operational definition: the method, the instrument, the units, the number of readings, and who measures.
• Give it a direction: larger is better, smaller is better, or on target between limits.
• Measure the response in the same way for every run, and ideally with the same person and instrument.

Checking the Measurement System

Every run produces a number, and that number is the true value plus the measurement error. If the error is large compared with the effects you want to find, the experiment fails however well it is run. Check before you start.

What to carry forward. The Design tab needs the size of the measurement error to decide how many runs are needed and whether to average repeated readings. Record it now. If the measurement noise is a large share of the run-to-run variation, plan to average several readings per run.

Establishing the Baseline

The baseline tells you where the response is today, how much it varies when nothing is changed on purpose, and whether the process is stable enough to experiment on. It also supplies the noise estimate that sets how many runs you need.

• Gather data from the current process over a representative period: at least 30 observations, from different shifts and material lots if possible.
• Plot in time order (run chart or individuals chart) and look for shifts and drift. A stable process has no special causes.
• Plot a histogram and compare with the specification. Look for two humps, which suggest a mixture of sources.
• Estimate the standard deviation. This is the run-to-run noise that effects must stand out from.
• Decide the smallest effect that is worth finding. Effects smaller than this do not change the business result, and they are costly to chase.

See Descriptive Statistics, Control Chart Theory, and Graphical Analysis for the methods.

Listing the Candidate Factors

Cast the net wide at first. The factor you did not think of is often the one that matters. Ask the people who run the process, look at what changed when the problem started, and use more than one method.

Classifying and Choosing the Factors

Not every factor belongs in the experiment. Sort the long list into three groups, then choose.

• Hold constant what you can, and write it down: lot, operator, equipment, and method.
• Block what you cannot hold, for example two resin lots, with each lot run as a block (see the Design tab).
• Record what you cannot control, such as humidity, so you can check afterwards whether it mattered.
• Include a factor if you have a reason to suspect it, even if you doubt it. A surprise factor is the best result an experiment can give.
• Do not drop a factor only because it is expensive to set. Raise the cost with the sponsor.

Setting the Levels

The low and high levels define the region the experiment explores. They determine whether an effect shows above the noise, and whether any run can be made at all.

Constraints and Budget

Every experiment lives within limits. Write them down before the design, because they decide how many runs you can have. Then compare the cost with what the answer is worth.

The value of the answer. 130 leak failures a month at $38 each cost about $59,280 a year. Cutting the rate from 4.2% to 0.5% would save about $52,212 a year. The experiment would cost about $5,432, which the saving would repay in roughly 5 weeks.

Worked Example: The Weld Experiment Frame

A plant makes a plastic sensor housing that is joined by ultrasonic welding. Each month about 3,100 housings are tested for leaks at the end of the line, and 130 fail (4.2%), costing $38 each. The team decided to run an experiment on the weld settings. All figures are illustrative, and the same example runs through all five tabs.

What the frame tells Design. Six controllable factors, a noise level of 22 kPa per housing, a smallest effect of 20 kPa, and a budget of 64 housings. One housing per run would leave the effect hard to see, so averaging several housings per run will be considered in Design.

Frame Check: Are We Ready to Design?

Before starting the Design tab, confirm the frame is complete. Use this checklist; progress saves in this browser only, and nothing is sent anywhere.

Questions a sponsor or a coach can ask:

• What do you want to learn, and what will you do with the answer?
• How do you know the response can be measured well?
• What does the process do today, and is it stable?
• Which factors did you consider, and why did you keep these?
• How wide are the levels, and who agreed they are safe?
• What will it cost, and what is the answer worth?

Adapting Frame to the Situation

Scale the framing to the risk and cost. A cheap, fast, reversible experiment needs a half-page frame. An experiment that stops a line, uses scarce material, or affects customers needs the full review.

Common Mistakes and Red Flags

Frame Resources on This Site

Frame Step Frequently Asked Questions

Sources and Further Reading

What the Design Step Is For

The frame says what to learn. The design says how to learn it with the fewest runs that still give a trustworthy answer. A good design makes every run count toward every effect, protects the result from drift and from things you did not control, and says in advance which effects it can and cannot separate.

The Design Step, Step by Step

1. 1

   Choose the design family

   Match the design to the objective and the number of factors: screening, characterization, optimization, robustness, or a mixture.

   • Use the design finder below
   • Plan a sequence if needed: screen, then characterize, then optimize
   • Check that every combination of the chosen levels is safe to run

   Output: A design family

   Watch for: Using a response-surface design to screen, or a screening design to optimize

2. 2

   Choose the number of runs and the resolution

   List the full factorial and the fractions that fit the budget. Choose the resolution that keeps the effects you care about clear of each other.

   • Prefer resolution V or higher if interactions matter
   • Accept resolution IV for pure screening, with a plan for the follow-up
   • Avoid resolution III unless you must

   Output: A design with a known alias structure

   Watch for: A fraction so small that main effects are mixed up with the interactions you suspect

3. 3

   Decide the setups and the units per setup

   Separate two numbers: how many setups (runs) and how many units to make at each. Use the noise components to see what each choice buys.

   • Setup-to-setup variation is not reduced by making more units at one setup
   • More setups usually beat more units per setup
   • Count the setup time

   Output: Setups, units per setup, and total units

   Watch for: Treating several units from one setup as replicates

4. 4

   Check the power

   Use the run-level standard deviation and the number of runs to find the probability of detecting the smallest effect that matters. Compare the options.

   • Use the smallest effect worth finding from the frame
   • Aim for at least 80% power
   • If power is low, change runs, units, or the goal, not the arithmetic

   Output: A power figure for each option, and a choice

   Watch for: Running the experiment hoping the effect is large

5. 5

   Add blocks, center points, and a random order

   Block on anything you know will change during the experiment, such as the day. Add center points if curvature is a concern and the budget allows. Randomize the order within blocks.

   • Confound blocks with high-order interactions
   • Record the center points as a separate type of run
   • Plan for any hard-to-change factor

   Output: Blocks, center points, and a randomization plan

   Watch for: Running one block, then the other, in standard order

6. 6

   Summarize and get approval

   Write the design on one page: factors and levels, runs, units, blocks, aliasing, power, cost, and time. Review it with the sponsor and process owner and note what the experiment cannot tell them.

   • State the aliasing in plain language
   • State the smallest effect it can find
   • Agree what the next experiment will be if it finds something

   Output: A one-page design summary, approved

   Watch for: Hiding a limitation that will surface when the results arrive

Finding the Design Family

The rest of this tab concentrates on the two-level factorial family, which does most of the work in screening and characterization. See Response Surface Methodology, Taguchi Methods, and Robust Design for the others.

Two-Level Factorial Designs

A two-level factorial sets each factor at a low and a high level, written −1 and +1, and runs every combination. It estimates every main effect and every interaction. Each run contributes to every estimate, which is why it needs far fewer runs than changing one factor at a time.

The number of runs doubles with every factor, so beyond four or five factors a full factorial is rarely worth it: most of the runs only estimate three-way and higher interactions, which are usually negligible.

• Orthogonal and balanced. Each factor is at each level in half the runs, and all effects can be estimated independently.
• Coded levels (−1 and +1) put every factor on the same scale, and make the calculation of effects simple. See Analyzing Designed Experiments.
• Interactions come free. A significant interaction means the effect of one factor depends on the level of another.

Fractional Factorials, Aliasing, and Resolution

A fractional factorial runs a selected part of the full factorial: half, a quarter, an eighth. The price is aliasing: some effects can no longer be told apart, because the same pattern of −1 and +1 estimates both. The figure shows the idea with three factors.

The design is described by its generators (here C = AB) and its defining relation (I = ABC). Every effect is aliased with the effect obtained by multiplying it by the defining relation. The resolution is the length of the shortest word in the defining relation, and it tells you what is mixed up with what:

Runs, Replicates, and Units per Run

Two different numbers decide how much information an experiment buys: the number of setups (runs, each with the factors freshly set), and the number of units made or measured at each setup. They are not interchangeable.

In the weld example, a short run of 12 consecutive housings at one setting gave a standard deviation of 18 kPa. The overall weld-to-weld standard deviation is 22.4 kPa, which means setup-to-setup variation accounts for the rest: √(22.4² − 18²) = 13.4 kPa. The standard deviation of a run that averages m housings is √(13.4² + 18²/m).

How Many Runs: Checking the Power

Power is the probability that the experiment finds an effect of a given size when it is really there. It depends on the effect size, the noise of a run, the number of runs, and the error degrees of freedom. For a two-level design:

1. Smallest effect worth finding (from the frame): 20 kPa.
2. Run-level standard deviation: 18.5 kPa when two housings are averaged at each setup.
3. Number of runs N: 32 setups.
4. Standard error = 2 × 18.5 / √32 = 6.52 kPa.
5. Error degrees of freedom: 9 (from the three-factor interactions, which are assumed negligible).
6. Power for a 20 kPa effect = 78%. The smallest effect found with 80% power is about 20.5 kPa.

In Minitab use Stat > Power and Sample Size > 2-Level Factorial Design, which gives the same answer for the number of corner points, replicates, and center points. See Sample Size and Power.

• Low power is not a small problem. A real, useful effect will often be missed, and the team will wrongly conclude that the factor does not matter.
• If power is low, add runs (or replicates), reduce noise, accept a larger smallest-effect, or drop a factor. Do not just run it and see.
• The error estimate must be real. It comes from replicates, from center points, or from higher-order interactions assumed to be zero. A design with none of these cannot test its own effects, and relies on a normal plot of effects.

Comparing the Options

With a budget of 64 housings and 960 line minutes (2 days), the team compared five ways to run six two-level factors. A setup change takes about 10 minutes and each housing about 3 minutes to weld and test.

Blocks, Center Points, and Randomization

Three more decisions protect the experiment from things the factors cannot explain.

In the example, the 2 line days are the two blocks. Dividing the 32 runs by the sign of the A × B × C column puts 16 runs in each day, with the A × B × C interaction (which is aliased with D × E × F) confounded with the day-to-day difference. The team accepts that: three-factor interactions are expected to be small, and the day effect is then removed from every main effect and two-factor interaction.

Other Designs You Will Meet

Worked Example: The Weld Design

The weld team took the approved frame to the design step. Their inputs: six factors at two levels, a noise level of 22.4 kPa per housing (of which 18 kPa is within a setup), a smallest effect of 20 kPa, a budget of 64 housings, and 960 line minutes. They compared the five options above and chose a half fraction. All figures are illustrative.

The first eight runs of the design in standard order (before randomization), showing how factor F is generated from A to E, and which day each run belongs to:

Some of the aliasing, to show what the half fraction does and does not cost:

Design Check: Are We Ready to Run?

Before starting the Run tab, confirm the design is complete. Use this checklist; progress saves in this browser only, and nothing is sent anywhere.

Questions a sponsor or a coach can ask:

• Why this design and not a smaller or a larger one?
• How many setups, and how many units at each?
• What is the smallest effect it would find, and how sure are we?
• Which effects are mixed up with which?
• What happens if a run goes wrong?
• What will we do if it finds something, and if it finds nothing?

Adapting Design to the Situation

Use software for the arithmetic, not for the thinking. The DOE Quick Planner, Minitab, and other packages generate designs and aliasing tables in seconds. The choices described in this tab, such as the budget, the noise, and the smallest effect, are yours.

Common Mistakes and Red Flags

Design Resources on This Site

Design Step Frequently Asked Questions

Sources and Further Reading

What the Run Step Is For

An experiment is a controlled change, and the control is easy to lose. A setting that is a little off, a run done out of order, a lot of material that changed halfway, or a result copied wrongly can each turn a clean design into an ambiguous data set. The Run step protects the plan from the plant.

The Run Step, Step by Step

1. 1

   Review readiness

   Walk through the checklist: material set aside, equipment checked, gauges calibrated, people briefed, approvals in place, time booked.

   • Set aside enough of one lot for every run and the pilot
   • Confirm the instrument calibration and the check standard
   • Book the line, the operators, and the lab

   Output: A readiness sign-off

   Watch for: Starting without enough material of the one lot

2. 2

   Generate and print the run sheet

   Build the randomized sheet with the planned design, blocks, and units per setup, and print enough copies. Keep the seed with the records.

   • Use the DOE Run Sheet Generator
   • Check the first rows by hand
   • Give each setup a run number that goes on every unit

   Output: A randomized run sheet, with the seed recorded

   Watch for: Re-sorting the sheet by setting to “save time”

3. 3

   Brief the team

   Explain why the order matters, who does what, how to record, and what to do when something goes wrong. Make clear that reporting a problem is the right thing.

   • Name the setter, the welder, the tester, and the recorder
   • Agree who can stop the experiment
   • Give them the deviation rules

   Output: A team that knows the rules and the roles

   Watch for: A team that tries to be helpful by fixing things quietly

4. 4

   Run a pilot

   Before the first real run, put a few units through the whole process, including the extreme corners. Check that every setting can be reached, that no combination damages the equipment, and that the measurement chain works.

   • Run the all-low and all-high corners
   • Test the measurement and the recording form
   • Time a setup change

   Output: A go decision, and a corrected plan if needed

   Watch for: Discovering on the first real run that a corner cannot be run

5. 5

   Execute setup by setup

   For each run in order: reset every factor, have a second person read the settings back, make the units, test them with only the run number visible, and record the actual settings and results immediately.

   • Do not skip the reset, even when the next setting is the same
   • Keep everything else the same
   • Record before moving on

   Output: Completed runs, one at a time

   Watch for: Batching the recording until the end of the day

6. 6

   Record and handle deviations

   Write down anything that did not go to plan, when it happened, and what was decided. Use the deviation rules, and call the experiment owner when the rules do not cover the case.

   • One log, in time order
   • Decisions made at the time, with initials
   • Never overwrite a record

   Output: A deviation log

   Watch for: A fix that is not written down

7. 7

   Check the data and close out

   Enter the data from the sheet, check it against the paper, plot the results in the actual run order, and look for drift and obvious errors. Lock the data file and keep the originals.

   • Double-check the entries
   • Plot by run order and by block
   • Keep the paper and the file together

   Output: A checked, locked data set

   Watch for: Analyzing a file that has not been checked against the sheet

Readiness: What Must Be True Before Run 1

The Run Sheet and the Random Order

The run sheet turns the design into a list of setups in a random order, with space to record what actually happened. Use the DOE Run Sheet Generator: choose the design, enter the factors and levels, set the blocks, replicates, center points, and units per setup, and it produces a printable, randomized sheet. It also reports the resolution and the aliasing, so you can check that it is the design you approved.

• Randomize more than the run order. Also randomize which material goes to which setup, and the order in which units are measured, where practical.
• Keep the seed. The same seed gives the same order, so the sheet can be rebuilt if it is lost.
• Do not sort the sheet by a factor to make changeovers easier. If a factor is very hard to change, tell the design team: a split-plot design may be needed.
• Spread center points through the order, so they can show drift.

Roles and Communication

One person may play more than one role in a small team, but the verifier should not be the setter, and the tester should not see the settings. Agree before starting who may stop the experiment, and make it clear that anyone may call a stop.

Executing a Run: The Rules

What to Record

Record more than you think you need. You can ignore extra information later, but you cannot recover what you did not write down.

Handling Deviations

Things will go wrong. What matters is that each one is noticed, written down, and handled by a rule agreed in advance, not by whoever happens to be on the line. The table gives defaults; the experiment owner makes the final call.

Keeping the Measurement Honest

• Same instrument, same method, same person for the whole experiment, where possible.
• Check standard. Measure a reference at the start, in the middle, and at the end of each day. A drift in the check standard means the gauge changed, not the process.
• Blind the tester. Only the run number on the unit; the setting stays with the setter.
• Measure promptly, and in the same conditions: parts can change with time, temperature, or humidity.
• Record raw readings, not rounded or converted ones, and note the units.
• For destructive tests, keep a record of which unit was tested and in what order. The unit is gone, so the record is all there is.

Checking the Data Before Analysis

1. Enter the data from the sheet into the file, with a second person checking a sample or the whole file against the paper.
2. Check plausibility: sort each column and look at the smallest and largest values for typing errors and unit mix-ups.
3. Plot the results in the order they were run. Look for drift, steps, and the effect of blocks. A trend with run order is a warning that something changed.
4. Compare the actual settings with the planned settings, and mark any differences.
5. Reconcile the units: units made, units tested, and units recorded must agree, and each missing one must be explained in the log.
6. Lock the file and keep a copy of the paper. Every analysis starts from this version.

See Graphical Analysis for the run chart and Descriptive Statistics for summaries.

Worked Example: Running the Weld Experiment

The weld team ran the design from the Design step: 32 setups, two housings at each, over two line days. All figures are illustrative.

The first eight setups in the order they were run, with the settings read back and the results:

The events log, in time order:

Run Check: Is the Data Ready for Analysis?

Before starting the Analyze tab, confirm that the experiment is complete and the data are trustworthy. Use this checklist; progress saves in this browser only, and nothing is sent anywhere.

Questions an experiment owner or coach can ask:

• Did every run follow the sheet? If not, what changed and who decided?
• Who verified the settings, and where is the record?
• What was held constant, and did it stay constant?
• Where are the original sheets, and who checked the data against them?
• Does the run chart show any drift or any block effect?
• Is there anything we did not write down?

Adapting Run to the Situation

Scale the discipline to the risk. A cheap, quick, internal experiment needs a simple sheet and a log. A costly or regulated one needs signatures, formal deviation reports, and retained samples.

Common Mistakes and Red Flags

Run Resources on This Site

Run Step Frequently Asked Questions

Sources and Further Reading

What the Analyze Step Is For

The Run step produced numbers. The Analyze step decides what they mean. In a factorial experiment each effect is a simple difference between averages, so the arithmetic is easy; the hard part is judging which differences are bigger than the noise, whether the model describes the process, and whether the answer is useful.

The Analyze Step, Step by Step

1. 1

   Look at the data first

   Read the log, plot the results in run order and by block, and note every deviation. Decide the unit of analysis: one value per setup.

   • Check for drift, steps, and block differences
   • Treat events from the log as part of the data
   • Average the units at each setup

   Output: One response per setup, and a list of known issues

   Watch for: Analyzing every housing as if it were independent

2. 2

   Estimate all the effects

   Fit the full model: every main effect and every two-factor interaction the design can separate. The effect of a term is the average response at its high level minus the average at its low level.

   • Use coded levels so effects are comparable
   • Estimate the block effect separately
   • Keep the aliasing in mind when you read the list

   Output: A table of effects for all terms

   Watch for: Fitting a model with terms the design cannot separate

3. 3

   Judge which effects are real

   Compare each effect with an error estimate. Use the error from higher-order interactions, Lenth's pseudo standard error, or replicates, and look at a normal plot of the effects.

   • Use two methods and see whether they agree
   • Look at the plot as well as the p-values
   • Treat a borderline effect as a question for the next experiment

   Output: A short list of effects that stand out of the noise

   Watch for: Using p-values from a model with no real error estimate

4. 4

   Plot the findings

   Draw the main effects and the interaction plots for the important terms. Plots show the direction and the size, and they show the interaction in a way a table cannot.

   • Lines for main effects; two lines for each interaction
   • Same vertical scale on every plot
   • Mark the specification or the target

   Output: Plots that tell the story

   Watch for: Reading a main effect that is part of a large interaction

5. 5

   Reduce the model

   Keep the real terms, plus their parents. Fit the reduced model, look at the ANOVA, the coefficients, and R-squared, adjusted and predicted.

   • Keep hierarchy
   • Use the reduced model's error to test and to predict
   • Keep the block in the model if the design had blocks

   Output: A reduced model with coefficients and an ANOVA table

   Watch for: Removing terms until only the significant ones are left, ignoring hierarchy

6. 6

   Check the model

   Plot the residuals against the fitted values, in run order, and on a normal probability plot. Look at outliers and influence.

   • Random scatter, no funnel
   • No trend in run order
   • Points near the line in the normal plot

   Output: A checked model, or a reason to change it

   Watch for: Trusting the p-values of a model that fails its checks

7. 7

   State the practical meaning and the limits

   Compare each effect, with its interval, with the smallest effect worth finding. Say what the experiment could not tell you: curvature, factors held fixed, the range tested.

   • Intervals, not only p-values
   • Say which settings do not matter and so can be chosen for cost or convenience
   • List what the next experiment should check

   Output: A one-page conclusion

   Watch for: Reporting effects with no sense of their size or uncertainty

Start With the Data, Not the Model

• Read the log. Every deviation affects how you analyze: a lost unit, a setup that moved in the order, a block change.
• Plot the results in the order they were run, and by block. A step between blocks, or a trend with run order, is a warning that something changed.
• Choose the unit of analysis. The experimental unit is the setup, not the housing. Use one value per setup, normally the average of its valid units. A setup with one unit, such as the one with a cracked housing, is simply noisier, and a weighted analysis can check that it matters.
• Know what you cannot test. With no replicates and no center points, the design cannot give a pure-error estimate or test curvature, so those assumptions need care.

See Graphical Analysis for the run chart and the box plot by block.

Estimating the Effects

With coded levels, the effect of a factor is the average response at its high level minus the average at its low level. For weld force (B), the 16 setups at 600 N average 320.1 kPa and the 16 at 400 N average 289.7 kPa, so the effect is +30.4 kPa. Interactions are estimated in the same way, from the product of the two coded columns. The sum of squares for a term is N × effect² / 4: for force, 32 × 30.4² / 4 = 7,412.

The design is a resolution VI half fraction, so every main effect and every two-factor interaction has its own column: six main effects plus 15 two-factor interactions, which is 21 effects, with 9 degrees of freedom left from the three-factor interactions (A × B × C is confounded with the day). The +2.2 kPa difference between days is small.

How the standard error was found. The 9 distinct three-factor interactions (the tenth, A × B × C, is the block) are assumed to be zero, so their variation is error: mean square 231, standard error of an effect = √(4 × 231 / 32) = 5.38 kPa, with 9 degrees of freedom. An effect is significant at 5% when it exceeds 2.26 × 5.38 = 12.2 kPa.

Which Effects Are Real?

With no replicates there is no direct measure of pure error, so the experiment estimates it indirectly. Use more than one method; where they agree you can be more confident.

Plotting the Findings

• Force and drying time raise burst pressure from low to high by 30 and 21 kPa. Longer drying probably helps by removing moisture from the resin, a mechanism the team had not tested before.
• The force by weld time interaction means the factors trade off. At low force, a longer weld time raises burst pressure by 37 kPa; at high force it adds only 5. Enough force makes weld time nearly irrelevant, so reading the weld time main effect alone would mislead.
• Amplitude, hold time, and clamp pressure had no effect that stands out of the noise, so they can be set for cost, cycle time, or convenience, within the tested range.

The Reduced Model

Keep the real effects, B, C, E and B × C (whose parents B and C are already there), and the day block. Fitted to the 32 setup averages, the reduced model has 26 degrees of freedom for error, so its tests and its intervals use all the information in the experiment.

Noise came in lower than planned. The Design step assumed a run-level standard deviation of about 18.5 kPa. The residual standard deviation here is 11.3. The planning figure was cautious, which is the safe way to plan: the experiment ended up with better power than the 78% promised, and the effects are more certain than the design promised.

Sensitivity check. One setup has a single valid housing, and so is noisier. Refitting with each setup weighted by the inverse of its variance changes no effect by more than 0.3 kPa, so the simple analysis stands.

Checking the Model

See Residual Analysis and Model Checking for how to read these plots and what to do when they show a problem.

Statistical Versus Practical Meaning

The frame set the smallest effect worth finding at 20 kPa. Compare every effect, with its interval, against it.

Factors with no real effect do not prove that the factor has no influence. The interval tells you how large an effect could still be hiding:

• Weld force is the main lever: 30 kPa, about 1.5 times the smallest effect that matters.
• Drying time and weld time each give about 21 kPa, right at the size that matters, with the lower end of the interval below it. Both look worth using, and both are worth confirming.
• The interaction has a practical use. At high force, weld time barely matters, so the team can shorten the weld time to save cycle time without losing strength.
• Amplitude, hold time, and clamp pressure can be set to whatever is cheapest, because even their upper limits sit below 20 kPa. Their ranges were safe, and they were held within them.

When the Results Look Odd

Running the Analysis in Excel and Minitab

The statistics are explained, worked by hand, in Analyzing Designed Experiments in Stat Dojo, and the software workflow is in the Minitab Guide. In summary:

Factorial Regression: Burst versus Blocks, Force, WeldTime, Drying

Analysis of Variance

Source                  DF   Adj SS   Adj MS  F-Value  P-Value
Model                    5   16414.0    3282.8    25.73    0.000
  Blocks                  1      40.5      40.5     0.32    0.578
  Linear                  3   14404.7    4801.6    37.64    0.000
    Force                 1    7411.5    7411.5    58.09    0.000
    WeldTime              1    3486.1    3486.1    27.33    0.000
    Drying                1    3507.0    3507.0    27.49    0.000
  2-Way Interactions      1    1968.8    1968.8    15.43    0.001
    Force*WeldTime        1    1968.8    1968.8    15.43    0.001
Error                   26    3317.0     127.6
Total                   31   19731.0

Model Summary

      S    R-sq  R-sq(adj)  R-sq(pred)
11.2950  83.19%     79.96%      74.53%

Coded Coefficients

Term              Effect    Coef  SE Coef  T-Value  P-Value   VIF
Constant                   304.88   2.00   152.69    0.000
Blocks                       1.12   2.00     0.56    0.578
Force               30.44    15.22   2.00     7.62    0.000  1.00
WeldTime            20.88    10.44   2.00     5.23    0.000  1.00
Drying              20.94    10.47   2.00     5.24    0.000  1.00
Force*WeldTime     -15.69    -7.84   2.00    -3.93    0.001  1.00

Typed excerpt of the reduced-model output, simplified from the calculated example. Menu names follow recent versions of Minitab Statistical Software and can differ slightly in older releases.

Worked Example: Analyzing the Weld Experiment

The team took the locked data from the Run step: 32 setup averages, from 63 valid housings, over two days. All figures are illustrative.

Analyze Check: Do We Understand the Results?

Before starting the Optimize tab, confirm that the analysis is complete and the model can be trusted. Use this checklist; progress saves in this browser only, and nothing is sent anywhere.

Questions a sponsor or a coach can ask:

• Which factors matter, and by how much?
• Which do not, and how sure are you?
• Is any effect large enough to use, and is it certain enough?
• Does the interaction change the advice?
• What did the residual plots show?
• What could the experiment not tell us?

Adapting Analyze to the Situation

Decide the analysis before seeing the data. A pre-planned analysis, with the model terms and the error estimate agreed in advance, protects against finding what you hoped to find.

Common Mistakes and Red Flags

Analyze Resources on This Site

Analyze Step Frequently Asked Questions

Sources and Further Reading