Sawtooth Software: The Survey Software of Choice

Getting the Most From CBC - Part 2

Our previous issue featured suggestions for using CBC, including use of prohibitions, choosing the numbers of levels, determining sample size, and presenting simple analyses. You can download that article from our Web page. Here are some further suggestions:

Including "None"

CBC provides the option of letting the respondent choose "None," or another constant alternative such as "I'd continue buying my usual product." One issue is whether to include "None" in the questionnaire, and a separate issue is whether to include "None" in the analysis.

We think it is usually a good idea to include the "None" option in the questionnaire, for these reasons:

  • It makes the choice tasks more realistic, because that option is usually available when shopping.
  • It makes the experience more pleasant for the respondent, who is not forced to select an unacceptable alternative.
  • It improves the quality of the data, by letting respondents screen themselves out of questions containing only alternatives they would never consider.

Some researchers like to include a "None" category in simulations, as an aid in estimating how category volume would expand or shrink as products become more attractive. We recommend against doing this, for these reasons:

  • CBC's estimate of how many respondents should choose "None" depends on the number of alternatives in the choice tasks. If you do a simulation with a different number of products, your estimates will not be correct.
  • Although choices of "None" are probably indicative of disliked alternatives, there is little reason to believe their frequency will accurately reflect the actual proportion of respondents refusing to purchase products in the real world.

In summary, we usually suggest including the "None" option in choice tasks, but then neglecting it in the analysis.

Calibrating CBC Results to Market Shares

CBC results usually differ from actual market shares. This is not surprising, since market shares are influenced by product distribution, brand awareness, out-of-stock conditions, point-of-sale promotions, imperfect buyer knowledge, and many other factors not captured in conjoint measurement.

Researchers are often motivated to adjust or "calibrate" simulation results to look like market shares. We suggest not doing so, because no matter how carefully choice results are calibrated to the market, the researcher will one day be embarrassed by differences that remain. However, if the pressure is too great to resist, there are two ways in which CBC results can be adjusted to more closely mimic market shares.

CBC utilities are scaled automatically to reflect the amount of random error in respondents' choices. You can over-ride that scaling by specifying a scaling parameter. Larger values than the default of 1.0 will create greater variation among shares of choice, making large simulated shares even larger, and small shares even smaller. Smaller values of the parameter will create less variation among simulated shares of choice, in the limit making them all equal.

Market shares are often "flatter" than choice shares, because they are affected by additional sources of random noise. If that is the case, you may be able to approximate market shares more closely with a scaling parameter of less than 1.0. Beware, however, that such an adjustment will also make your results less sensitive to changes, including pricing changes.

Sometimes, market shares reflect too much variation, for example when the largest product has nearly 100% geographic distribution but smaller products do not. In that case we suggest not adjusting the scaling parameter, which could make your results too sensitive to pricing changes.

In general, if you must make simulated shares look like market shares, we suggest using "external effects" which will adjust share levels to be like market shares without affecting sensitivity to change. External effects may be calculated for each simulated product by dividing the target share by the simulated share of choice.

IIA and the Red Bus/Blue Bus Problem

The CBC simulator, like most logit conjoint simulators, suffers from "IIA," which is shorthand for "Independence from Irrelevant Alternatives." The basic idea of IIA is that the ratio of any two products' shares should be independent of all other products. This sounds like a good thing, and at first, IIA was regarded as a beneficial property.

However, another way to say the same thing is that an improved product gains share from all other products in proportion to their shares; and when a product loses share, it loses to others in proportion to their shares. Stated that way, it is easy to see that IIA implies an unrealistically simple model. In the real world, products compete unequally with one another, and when an existing product is improved, it usually gains most from a subset of products with which it competes most directly.

Imagine a transportation market with two products, cars and red busses, each having a market share of 50%. Suppose we add a second bus, colored blue. An IIA simulator would predict that the blue bus would take share equally from the car and red bus, so that the total bus share would become 67%. But it's clearly more reasonable to expect that the blue bus would take share mostly from the red bus, and that total bus share would remain close to 50%. Indeed, the IIA problem is sometimes referred to as the "red bus, blue bus problem."

The ACA and CVA simulators offer a "first choice" model, which avoids the IIA assumption entirely. If you do a first choice simulation and add a product identical to an existing product, those two products will get the same total share of first choices as either would alone. However, the first choice model is usually not satisfactory for another reason: it tends to overstate results for the most popular products.

If you don't use the first choice model, then you will run into IIA problems. ACA, CVA and CBC offer a "correction for product similarity" which penalizes products in proportion to their similarity to others. That correction works properly under ideal conditions, but it requires assumptions about how similarity is measured. We don't know of any completely satisfactory answer to the IIA problem. The best insurance is to be aware of it, and to avoid reaching improper conclusions.

Many of our users do "sensitivity analyses" by varying a product up and down on each attribute in turn. When testing sensitivity in this way, we suggest not using the correction for product similarity. It can give misleading answers, particularly for continuous attributes such as price. Suppose all products are initially simulated as having mid-level prices. If one product is then given a higher price, it is not only less desirable because of its increased price, but also less similar to the other products because its price is different from theirs. As a result, a product may gain predicted share as its price increases!

Another common situation involves simulating the effect of line extensions. It is a fairly common error for a researcher to include two versions of the product of interest, but only a single version of other products. That gives an artificial edge to the product with two versions. One rough work-around to this problem is to include two versions of every product, but to permit the two versions to be different only for the one with a line extension.

Finally, because of IIA restrictions, conjoint simulators are not very good at measuring cross-elasticities. They can do quite a good job of measuring the effect of a price change on that product's own share, but they are not very good at measuring the effect of a price change on other products' shares. The reason, again, is that IIA restrictions require that a product's interactions with others will be proportional to their shares.

We end with a note of good news. IIA problems are less serious when simulations are done at the individual respondent level, or even with homogeneous subgroups of respondents. IIA is less of a problem for ACA and CVA than for CBC. And even with choice data, IIA problems may be reduced by first separating respondents into homogeneous groups, as is done with the CBC Latent Class Segmentation Module.