Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

First Choice vs. Share of Preference


I am interested to understand why we use First Choice Rule for Hold-out-Hit Rates and the Share of Preference Rule for MAE.
I do a Line Extension with an alternative specific Design.

Thank you for your help!

Regards, Isabelle.
asked Oct 6, 2016 by isabelle (180 points)
retagged Oct 6, 2016 by Walter Williams

1 Answer

+1 vote
Share of preference can also be done at the individual level for "hit rates" and in that case we call it the "probability of the hit" or the "hit likelihood".  Rather than simply count it as a 1/0 (hit or miss) at the individual level, you can take the share of preference (logit share) for the concept that the respondent actually chose.  

You can take the regular arithmetic mean of the hit likelihoods across respondents, though some researchers prefer to take the geometric mean across respondents.

I tend to prefer raw hit rates (1/0) for validation because it removes the issue of scale factor having an effect on the predictive validity.  What I don't like about the probability of the hit under the logit rule is that if you multiply all the utilities for all respondents by a constant such as 1.1, you can often lift the hit rate.  What that means if I am comparing two different utility estimations on hit rate is that their scaling (which can differ just due to some estimation settings) has an effect on the hit rate.  

That said, the probability of the hit is more precise than the raw (1/0) hit and could be especially useful with moderate to small sample sizes.
answered Oct 6, 2016 by Bryan Orme Platinum Sawtooth Software, Inc. (173,090 points)