Sawtooth Software: The Survey Software of Choice

Sawtooth Software's Bryan Orme and Walter Williams report results of a meta analysis of about 50 commercial CBC and MaxDiff data sets. Specifically, they looked into how the priors settings in CBC/HB (prior variance and degrees of freedom) affect the ability of the estimated part-worth utilities to predict holdout choice tasks. Because most CBC data sets don't have enough holdout choice tasks for this purpose, they developed a special extension to the CBC/HB software that uses a jack-knifing routine to systematically hold out just a few of the choice tasks (and repeat) for hit rate validation while estimating the utilities from the remaining choice tasks.

Bryan and Walter found that the default prior setting of variance=2 leads to lower hit rates than lower prior variance settings (the average optimal prior variance setting across the data sets was 0.78). Not surprisingly, they found that the optimal prior variance setting depends on the data set (their optimal found prior variance ranged from 0.1 to 1.65). They also found that the default prior variance setting for MaxDiff data (prior variance = 1) is fairly close to optimal for the MaxDiff data sets they examined, but actually may lead to a little bit of underfitting.

  Download PDF   View in new window