Sawtooth Software: The Survey Software of Choice

Including Holdout Choice Tasks in Conjoint Studies

We think it is wise to include holdout choice tasks in conjoint interviews, even though they may not appear to be needed for the main purpose of the study. They almost always turn out to be useful, for these reasons:

  • They provide a proximal indication of validity, measured by the utilities' ability to predict choices not used in their estimation.
  • They provide a check on the scaling of the utilities. If the most popular concepts are over-predicted, then the scale parameter should be reduced. If the predictions are too flat, then the scale parameter should be increased.
  • They permit identification and removal of inconsistent respondents.
  • They can be used for testing specific product configurations under consideration. Much value can be added by direct measurement of these concepts.

It's hard to design good holdout concepts without some prior idea of respondent preferences. There's no point in asking people to choose among concepts where one dominates in the sense that everyone agrees which is best. And, similarly, it's good to avoid presenting concepts that are equally attractive, since equal shares of preference would be predicted by a completely random simulator. If you present triples of concepts, it's probably best if their shares of choices are somewhere in the neighborhood of 50/30/20.

When conducting CBC studies, if you plan to do segmentation with latent class analysis, it's wise to consider the kinds of groups you expect to get, and to design products in holdout choice sets so that one alternative will be much more preferred by each group. This maximizes your ability to confirm the validity of the multi-group Lclass simulator.

It isn't necessary to have many holdout sets to check the validity of your utilities, or their scaling. However, if you want to use those choices to identify and eliminate inconsistent respondents, you need several choice sets.

For ACA studies, holdout concepts can be included in the computer-administered interview using the Ci3 System. For CBC studies, either the Ci3 System, or the fixed design option of CBC's "TSK:" instruction may be used for presenting holdout concepts.

We've shown an example of a holdout choice task below. It is probably not very useful to include a "None" option in holdout choice tasks, particularly when these are paired with traditional conjoint exercises which don't have a "None" option.

Finally, if you do have several choice sets, it's useful to repeat at least one of them so you can obtain a measure of the reliability of the holdout choices. Suppose your conjoint utilities are able to predict only 50% of the respondents' holdout choices. Lacking data about reliability, you might conclude that the conjoint exercise had been a failure. But if you were to learn that repeat holdout tasks had reliability of only 50%, you might conclude that the conjoint utilities were doing about as well as they possibly could, and that the problem lies in the reliability of the holdout judgements themselves.

If you were shopping for a credit card and these were your only options, which would you choose?
No Annual Fee
15% Interest Rate
Frequent Flier Program
$4,000 Credit Line
$30 Annual Fee
12% Interest Rate
No Frequent Flier Program
$6,000 Credit Line
$60 Annual Fee
9% Interest Rate
Frequent Flier Program
$2,000 Credit Line