Experimental Strategy: A Blueprint. Analysis and Visualization

By | March 20, 2015

Step 7. We ran the experiment… Now what? We have data, we need knowledge.

What we mean by information — the elementary unit of information — is a difference which makes a difference. (Gregory Bateson)

 At the end of step 6, we had an experimental plan that the entire team could understand and that was (reasonably) robust to Mother Nature’s nuisances. Between step 6 and step 7 is your time in the laboratory executing the plan. This is in your domain, and I hope you bring all the skills and experience of your specific area of science to bear on it.

The result of a designed experiment is an array of factors and responses. Now it is essential to convert this data to information and then to knowledge.

This data array was designed so the effects of the factors are (relatively) independent of each other. This means that you can make “clean” statements about each individual factor and the way in which its effects are modified by the other factors. For example…


In a study done by a plastics company, the key effort was to minimize “Cbow” in plastic doors for small refrigerators (Fig). They had to fit in a tight space in a hotel cabinet. Tens of thousands of dollars worth of sales were at stake. An 8-run experiment was done to measure the effect of foam type “geometry” and wall thickness on the bow at the center of the door. The results are given in the table.

Foam Geometry Wall Bow (.01 inch)
Flex Wide Thick 11
Rigid Narrow Thin 10
Flex Wide Thin 11
Flex Narrow Thick 7
Rigid Wide Thin 12
Flex Narrow Thin 6
Rigid Wide Thick 14
Rigid Narrow Thick 9

Refrigerator CubeIn the simplest factorial experiments, it is remarkably easy to visualize the data by drawing cubes and writing the observations into the appropriate corners of each cube. It is then straightforward (with a little practice) to compare the faces against each other. In this example, the numbers in the top face are all larger than the bottom, and those differences are larger than right-to-left or back-to-front. Hence, Geometry has the largest effect. A fourth factor can be included using side-by-side cubes.

In the case of larger factorials and fractional factorials, computer methods such as Analysis of Variance (ANOVA) and regression will be helpful. Regardless, it is essential to remember that “All analysis of factorial designs is simply the comparison of averages vs. averages.” (Charlie Hendrix). In the example, the averages of top and bottom (Geometry) are 12 and 8, a difference of 4; compare this to the front and back (Wall) which are 9.75 and 10.25, a much smaller difference of 0.5.

 If we keep that in mind, the statistics becomes less daunting!

Modern statistics and DOE programs will slide your data into a regression analysis in the twinkling of an eye. BEFORE that happens, think back to Step 2. We asked: what kind of measurement are you making? Is it Binary (on-off); Subjective (quality); Integer (counted defects); or Real Numbers? Regression analysis is only fully valid when the measurement is in Real Numbers! For the other variable types:

Binary data is usually handled with logistic regression (and requires many samples at each factor setting).

  • Subjective data can be analyzed using regression analysis with considerable care and skepticism. More statistically valid results can be obtained with Multinomial Logistic Regression.
  • Integer (count) data can be studied using Poisson or Binomial regression.

All three of these data types can also be advantageously studied using Partition analysis. In all three cases, the response is essentially a classifier. Recursive partitioning creates a decision tree that strives to correctly classify members of the population based on the independent variables.

In the example below, the refrigerator doors also showed adhesion failures which would cause expensive rejects. This was an integer variable: the counts of failures out of 20 doors in each sample. Partition analysis immediately separated the factors by room temperature (low: blue; high: red). Within those groups there was also and effect of glue: higher glue gave better adhesion. It was then possible to design conditions to get essentially zero failures!

Adhesion partition

These blog posts have been intended to give you taste of my views on experimental strategy. I hope you have benefited from them! I conclude with a final reminder with a picture of the typical path of an experimental program. There are clear paths forward… with many tactical retreats. “The course of true love never did run smooth” (Shakespeare).

I would be delighted to work with you on your project. Contact me at +1 413 822 5006 or cawse@cawseandeffect.com!

Leave a Reply

Your email address will not be published. Required fields are marked *