Sawtooth Software: The Survey Software of Choice

CVA v3 Experimental Designer Better than Before

With the recent release of CVA v3, all three base Sawtooth Software conjoint systems are now integrated within the SMRT platform. (SMRT stands for "Sawtooth Software Market Research Tools".) While we are pleased with the added ease-of-use and polish the new Windows-based CVA offers, we are particularly excited about the improvements in the experimental designer. For starters, rewriting and compiling the program for the 32-bit operating system produced about a 5x speed improvement. But more importantly, increasing the "pool multiplier" capacity and an alteration to the existing algorithm significantly improved the designer.

Back in 1997, Warren Kuhfeld of SAS reviewed our CVA v2 designer at the Sawtooth Software conference and gave suggestions for improvement (see "Efficient Experimental Designs Using Computerized Searches" available at www.sawtoothsoftware.com in the Technical Papers library). One of Warren's suggestions was that we search among larger candidate sets equal to, if possible, the full-factorial design. The original CVA v2 used a maximum pool multiplier of 20 when creating candidate sets. In other words, if the user requested 25 conjoint questions, a maximum of 25 x 20 = 500 conjoint questions were used as candidates for finding the optimal subset of 25. With CVA v3, we let the user set the pool multiplier as high as 999. This allows CVA v3 to create candidate sets that usually reach or approximate a full-factorial.

In version 3, we also changed the algorithm so that we spend less time initially reducing the candidate set to the desired number of questions. The previous version started with the entire candidate pool and discarded one question at a time (the question that when discarded gives the surviving set the highest D-efficiency). With the new version, we only use an initial pool of up to the first 10x tasks during this step, where x is the number of requested tasks. Then, we concentrate the algorithm's effort on the two-way swaps using the full candidate pool.

The combination of the 32-bit operating environment and the methodological refinements yield improved results in a measurably shorter time than can be obtained using the previous version.

Comparison of v2 and v3 Designers

To demonstrate how much better the v3 designer works than v2, we'll use two examples from Warren's 1997 paper.

The first example is a 2^2 3^3 5^2 design in 30 cards (single-concept). (A 2^2 3^3 5^2 design means that there are two 2-level attributes, three 3-level attributes and two 5-level attributes.) The full factorial is 2700 cards. With 30 requested cards, a pool multiplier of 90 would create the full factorial. Using a pool multiplier of 20 (the maximum under v2) and 20 attempts (passes) using different random starting points, CVA v2 found a design with a D-efficiency of 97.72 in about 8 minutes. Using a pool multiplier of 90 and 10 passes, CVA v3 found a design with a D-efficiency of 98.23 in about 1 minute. (We used a somewhat dated 500 MHz processor for all runs reported in this article.)

The second example is even more dramatic. The design is much more demanding: 2 3 4^2 5^2 6 in 24 cards (single-concept) with 6 prohibitions. With CVA v2, we ran the problem with the maximum pool multiplier of 20 for 10 hours (1200 passes). The best of the 1200 passes resulted in a D-efficiency of 84.68. We tried the same plan under CVA v3 using a pool multiplier of 225, using just 10 passes (each pass took roughly 60 seconds). After these ten passes, v3 found a design with a D-efficiency of 86.46. Four of the first ten passes had a D-efficiency superior to the best result from the 10-hour CVA v2 run.

These results demonstrate that using larger candidate sets, focusing more of the effort on the two-way "swapping" step, and the increased speed under the 32-bit implementation make version 3's designer far superior to v2.