Sawtooth Software: The Survey Software of Choice

Getting The Most From CBC--Part 1

When designing CBC, we tried to make many of its features automatic, so users could accomplish their goals by accepting built-in defaults. However, CBC users have had questions about several issues, and these are our recommendations about them.

Using Prohibitions: CBC lets you specify that certain combinations of levels should not appear in the questionnaire, such as a luxury product at an economy price. Prohibitions should be used only rarely, to avoid showing hypothetical products that would be seen as completely absurd. There are several reasons to avoid using prohibitions:

  1. CBC lets you specify unique price ranges for different combinations of levels, such as brand and package size, so prohibitions are not required to ensure reasonable prices.
  2. Respondents are usually able to deal with product concepts more "extreme" than currently available in the market, and permitting such combinations not only increases the efficiency of estimation of their utilities, but also let you estimate "what might be" as well as "what is."
  3. Too many prohibitions, or a particular pattern of them, may make it impossible to create a good design, and may undermine your ability to analyze the data. Although all prohibitions decrease statistical efficiency, CBC provides a way to test a design before you go to the field to ensure that prohibitions will not have catastrophic effects. If you do use prohibitions, test your design!

Numbers of attributes and levels: CBC permits a maximum of 6 attributes, and a maximum of 9 levels per attribute. But, as with other conjoint methods, you get the best results if you keep things simple. Don't include unnecessary attributes or levels just because there's room for them. Most CBC studies use only three or 4 attributes. You may have to use as many as nine levels for "categorical" attributes like Brand or Package Type, but there's seldom any reason to have more than 5 levels for quantitative attributes like Price. It's usually better to have more data at each price point than to have thinner measurements at more price points, particularly if you're interested in interactions. The interaction between two 9-level attributes involves 64 logit parameters, but the interaction between two 5-level attributes requires involves only 16.

Determining sample size: In a paper presented at the 1997 ART forum (which you can download from our home page) we showed that the statistical gain from increasing the number of choice tasks per respondent was similar to the gain from a proportional increase in the number of respondents. Therefore, one way to determine sample size is as follows:

  1. Count the number of "cells" in the largest interaction you want to measure. For example, with the interaction of two 9-level attributes, there are 81 cells. One way to think about CBC analyses is that you want to estimate the proportion of times that concepts defining each cell are chosen.
  2. Determine how many concepts will be shown altogether, which is the number of respondents times the number of choice tasks per respondent times the number of concepts per task.
  3. The approximate number of concept occurrences per cell will be equal to the total number of concepts shown, divided by the number of cells. Call this quotient n.
  4. Ignoring choices of "None," the average probability of a concept being chosen is 1 over the number of concepts per task. Call this p.
  5. The standard error of a proportion is sqrt [p (1-p) / n] .

For example, consider a "typical" CBC study, with 300 respondents, each with 10 choice tasks, and with 5 concepts in each task. The total number of concepts shown will be 15,000. If the largest interaction we want to measure is 5 x 5, the number of concept occurrences per cell will be 15,000 / 25 = 600. The average probability of a concept's being chosen will be 1/5 = .2, so the average standard error will be sqrt [.2 * .8 / 600] = .016, and the 95% confidence interval will be about +/- 1.96 * .016, or +/- .03.

If we were interested in a 9x9 interaction, the number of cells would be 81 rather than 25, and the number of respondents required for equivalent precision would be about three times as large. If we were only interested in main effects, the maximum number of cells might be 5 rather than 25, and a sample size only a fifth as large could yield equivalent precision for those estimates.

The "typical" CBC study is like our example, in which the total number of concepts divided by the number of cells of interest is about 600. If separate estimates are desired for several segments, then adequate samples should be included for each of them. Finally, remember that you can get about the same increase in precision from proportional increases in the number respondents, or the number of tasks per respondent, for questionnaires with up to 20 tasks.

Reporting results: Counts vs. Simulations: If you don't use prohibitions, CBC produces designs in which each attribute varies independently of the others. This means that you can measure the effect of an attribute simply by observing the proportion of times concepts are chosen when they have each level. Similarly, two-way interactions can be evaluated by seeing how often concepts with each pair of levels are chosen. We call this the "counting" approach. You can also do logit analyses to estimate average utilities, which you can then use in simulations. If you have not used prohibitions, these two approaches will produce similar results.

For simple questions, such as obtaining price sensitivity curves for specific brands, the counting approach is often adequate. Since no complicated analysis is required, results are easy to communicate to others who are not market researchers. However, other objectives may require logit analysis and simulations. For example, if you want to simulate a particular product's showing in a competitive market, the logit/simulation approach is more appropriate.

Still to come: In subsequent issues we'll have some suggestions about use of the "None" option and how to calibrate CBC results to external market share data. We'll also discuss the "IIA Problem" (Independence from Irrelevant Alternatives), when to use CBC's Correction for Product Similarity, and we'll have some suggestions for dealing with respondent heterogeneity.