Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Out-of-Sample Prediction with Hierarchical Bayes

Is it possible to conduct an out of sample prediction test using the results of a CBC estimated with HB? How should one do so?

I have the results of an HB estimation of a CBC (I have both the choice data, and the individual estimated partworths) for a sample of about 800 respondents, each of which answered 12 choice tasks.  I also have the choice data of additional 1000 respondents who answered the same conjoint survey (same attributes, 12 choices each, but not necessarily the exact same choice sets as the original sample, as the choice sets they answered were randomized and orthogonalized).  Is it possible to use the partworth results from the first sample to generate some form of prediction for the choices of the additional 1000 respondents?

I would very much appreciate any guidance or references to academic articles.

Thank you,
asked Apr 6, 2017 by CBC-HE

1 Answer

0 votes
Yes, for methodological studies, researchers often do out-of-sample validation for HB models.  Typically the holdout respondents share a single design, or a small number of blocks/versions so that the number of simulations needed to predict their choices would be small.  In this case though you cannot match predictions at the individual respondent level, of course, you can use your HB model to predict shares of the holdout respondents' choices and you can compare actual to predicted shares in terms of mean absolute errors or correlations (the latter is nice because it allows for statistical testing).

If you used a design for your holdout respondents that had a large number of blocks/versions then you could still use your HB sample to make predictions about the choices of all those holdout respondents.  The simulation work would be much more complicated, however.  I suppose you could compute log likelihoods to assess the fit of your predicted choice probabilities to 0/1 actual choices, though I've not done validation work this way before.  Perhaps someone else has who would have a better idea of how best for you to do this and what challenges you might face.
answered Apr 6, 2017 by Keith Chrzan Platinum Sawtooth Software, Inc. (50,675 points)