I would like to compute the RLH (and later also the hit rates and percent certainty) of observed responses for the hold-out choice tasks of a CBC.

Based on the forum posts and tutorials indicated below, I have thus calculated:

a) Utility of concept 'a' for the respondent 'i':

U_ai = U_aif1 + .. + U_aif5, where U_aif1 through U_aif5 is the raw utility of respondent 'i' for the features f1 to f5 of the product concept 'a'.

As described in the link below, the utility of each concept is thus simply the sum of the respondent's part worth utilities for each of the respective features.

(See: https://www.sawtoothsoftware.com/help/lighthouse-studio/manual/estimating_utilities_with_logi.html)

b) Likelihood of observing chosing concept 'a' in each choice task 'x' of respondent 'i' as:

L_xi= P_ai=e^U_ai/(e^U_ai+e^U_bi+...+e^U_di), where P_ai is the probability of picking concept 'a' from the set of concepts shown in choice task 'x' to the respondent 'i', U_ai through U_di are the utilities of concepts ai through di for the respondent 'i', as described in point a) above

(see: https://sawtoothsoftware.com/forum/16929/how-is-rlh-for-maxdiff-calculated?show=16929#q16929)

c) Individual RLH of each respondent i as:

RLHi = (L_x1i * ... * L_x6i ) ^(1/6), where L_x1i through L_x6 i are the likelihoods of the observed responses in the hold-out choice-tasks x1 to x6 using the formula under point b) above.

(see "predict the proportion of respondents who would choose each concept" in:

https://www.sawtoothsoftware.com/help/lighthouse-studio/manual/estimating_utilities_with_logi.html

d) RLH over all respondents as:

RLH = (RLH1 * ... * RLHn) ^(1/n), where RLH1 through RLHn are the RLH of the respondents number i=1 to i=n from point c) above.

To make sure, that these formulas are correct, I first calibrated the raw utilities using all choice tasks, and compared the results with the individual RLH (for point c) and the overall RLH (for point d) that are reported by Sawtooth. However, the results were very different.

Can you spot, whether there is an error in the formulas above, or is there any other diagnostic output I could use (e.g. for steps a) and b), or a tutorial describing the steps more in detail?

Thank you in advance for your reply and happy new year ;-)!

So in step d) I should use the arithmetic, instead of geometric mean.

In step c), to calculate the RLH of a respondent in the same way as sawtooth, I need to first calculate the RLH for each draw, and then compute the arithmetic mean of the RLH for the different draws.

However, wouldn't it make more sense to describe the quality of the fit by using the RLH resulting from the average part worths across different draws? (i.e. the individual part worths, which are reported by sawtooth at the end of the HB analysis as an estimator of the true part worths)