Step 4. Do we have the resources?
“The amateurs discuss tactics: the professionals discuss logistics.” – Napoleon Bonaparte
It is really frustrating to get half way through an experiment and then get pulled off for “higher priority” work. If that happens during a designed experiment, the return for your effort could be ZERO! In this post you’ll learn how to clear up this uncertainty in step 4 of the Experimental Strategy Blueprint. You’ll learn ways to estimate the necessary resources.
In Step 3, we identified the factors (independent variables) that must be studied in the experiment, so we can make the measurements identified in Step 2. Here we must walk the line between excessive optimism, thinking we can get the needed data in a few days, versus excessive pessimism – fearing that the experiment is hopelessly large and we’ll never get it done. A key to this strategy is:
Only in exceptional circumstances do you need or should you attempt to answer all questions with one experiment. (Box, Hunter, and Hunter)
The usual first problem encountered at this stage is simple: there are too many factors! If we were to try to examine all the factors in detail simultaneously, the number of experiments would balloon. A common solution is to propose a number of small experiments, perhaps studying two or three at a time. This sometimes works – but often conditions will change between experiments and you will not have “apples to apples” comparisons.
This occurred in a recent experiment with a natural products company where two years had been spent doing just that. The result was confusion.
To cut through this confusion, a careful review with the team identified sixteen potentially important factors! A single large (48 run) design was able to compare all of them under consistent conditions and showed that 11 were either not important or that they could be set to one condition and kept there.
This was a Screening experiment – whose primary purpose is to compare all the factors and reduce them to the “critical few”.
Screening experiments are traditionally done as fractional factorial designs, in which each factor is constrained to just two levels, and a designed fraction is taken of all the possible combinations. In the above case with 16 factors, all possible two-way combinations total 216 = 65,536 runs! The classical fractional factorial would use 64 runs. Using more modern computer-aided designs I only needed 48.
A screening experiment may require a small amount of additional experimentation to clear up ambiguities, but the next step is usually a Response Surface experiment. This is designed to clearly locate the optimum response region while identifying important interactions and curvature.
In the natural products example, this was another 24-run design in five factors. One was qualitative (two types of raw material). The others were quantitative process factors at three levels each, measured over somewhat narrower ranges. This identified an optimum operating range that gave satisfactory product.
Two subsequent 24-run Optimization experiments cleared up some details and made the process Robust to the plant operating environment. Commercial product shipments started very soon afterwards!
Estimating Resource Needs
From this example, we can get a framework for estimating our needed resources. This is key to a strategic approach to avoiding the kind of waste we saw at the beginning of this article.
Typically, the first stage of a well-designed experimental strategy should not use more than 25-30% of the total runs. If the problem is sufficiently complex that an initial screening experiment is required, then the full process will generally require 3 to 4 times as many runs. In our example, the 48-run screen led to 120 total runs (about 2.5x).
This sort of resource estimation will support a dialog with management that should lead to a clear statement of management commitment – in time, personnel, material, and equipment – to the study.
You now have a DOE strategy that
- has a first-pass estimate of resource needs, and
- has management understanding of and commitment to the effort needed.
In the next section, we’ll get into the details of the actual experimental arrays.
If you want to jump right to the whole strategy, contact me at +1 413 822 5006 or firstname.lastname@example.org!