Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

HB report understanting

Hello,

I have run several HB models on my MaxDiff data, but Now I am not sure how to report the results and which model should I choose as the best fit?
I have followed the recommendations in chapter 12 of your advanced CBC book and generated the 95% CI & the % confidence of each level I am valuing. Still, I found that in the articles they report Standard errors, how can these two be correlated?
Furthermore, how can I discover the heterogeneity between respondents regarding each level/item and how to check if they are significant?
Finally, if I have only one hold-out, how can I check the hit rate of my model regarding this hold-out task or test the model predictability?

One last point, how can I calculate the degrees of freedom for the Logit in maxdiff case 2 if I have 31 total levels & presenting 6 of them (each time) and each respondent evaluated 10 different scenarios ?

Thank you very much
Regards'
asked Mar 26 by AMYN Bronze (2,980 points)
edited Mar 26 by AMYN

1 Answer

+1 vote
Please see replies below in CAPS:


I have followed the recommendations in chapter 12 of your advanced CBC book and generated the 95% CI & the % confidence of each level I am valuing. Still, I found that in the articles they report Standard errors, how can these two be correlated? I AM NOT SURE WHAT ARTICLES YOU HAVE SEEN, BUT IN MANY ACADEMIC JOURNALS THE REPORTS OF MNL MODELS SHOW THE STANDARD ERRORS OF THE PARAMETERS.  AS FOR ANY OTHER PARAMETRIC STATISTIC, YOU CAN CALCULATE STANDARD DEVIATIONS AND THEN DIVIDE YOUR STANDARD DEVIATIONS BY THE SQUARE ROOT OF YOUR SAMPLE SIZE TO GET STANDARD ERRORS.   
Furthermore, how can I discover the heterogeneity between respondents regarding each level/item and how to check if they are significant?  WITH HB YOU HAVE RESPONDENT LEVEL UTILITIES.  THESE QUANTIFY THE HETEROGENEITY AMONG RESPONDENTS.  I AM NOT SURE WHAT YOU MEAN BY SIGNIFICANCE IN THIS CONTEXT - YOU HAVE RESPONDENT LEVEL SCORES AND THEY ARE NOT ALL IDENTICAL.
Finally, if I have only one hold-out, how can I check the hit rate of my model regarding this hold-out task or test the model predictability? BUILD A SIMULATOR AND SIMULATE RESPONDENTS' CHOICES FOR THAT HOLD OUT USING THE FIRST CHOICE RULE.  FOR HOW MANY RESPONDENTS DO YOU CORRECTLY PREDICT THE BEST CHOICE?  FOR HOW MANY RESPONDENTS DO YOU CORRECTLY PREDICT THE WORST CHOICE.  THE NUMBER CORRECT DIVIDED BY YOUR SAMPLE SIZE IS A HIT RATE.  

One last point, how can I calculate the degrees of freedom for the Logit in maxdiff case 2 if I have 31 total levels & presenting 6 of them (each time) and each respondent evaluated 10 different scenarios ? HOW MANY PARAMETERS DID YOUR MODEL HAVE - E.G. IF YOU USED MAXDIFF CODING THEN YOUR 31 LEVELS HAVE 30 PARAMETERS (DEGREES OF FREEDOM).
answered Mar 26 by Keith Chrzan Platinum Sawtooth Software, Inc. (90,475 points)
Thank you very much, Keith, for the reply, and just I wanted to clear some points:
1- I am using the standalone program to generate the row scores for the HB, and I am asking about reporting the HB raw score results. SO I got the geometric average for the RLH & the arithmetic average for each respondent scores from the _utilities.csv file, and that is my principal coefficients. Then from the _alfa.csv (or _minbeta.csv in case there are added coefficients or constraints) I used the iterations after convergence to generate the 95% CI by getting the 97.5th & 2.5th percentiles in each coefficient. Also, I calculated the % confidence by measuring the percentage of times the value of one coefficient that is larger than the one coefficient that is next to it in on dimension as the coefficients in each dimension are ordinal from the best to the worst. (Case 2 MAxDiff). I have not got the chance to check this against similar studies reporting HB of MAxDiff, so I checked HB for DCE. One of the studies reported coefficients & SE; variances & SE. [The Patient - Patient-Centered Outcomes Research https://doi.org/10.1007/s40271-019-00402-w]
2- In the same article, I believe they used Standard deviations and coefficients of variation (the ratio of the standard deviation to the mean) to quantify the heterogeneity. I am not sure about the importance of this statistic.

3- I believe that the simulator is for the DCE tasks! Whenever I tried it in the lighthouse, they ask me to generate CBC results. I have such a study in my data, but I have not started analyzing it yet, and it is not the same. What I did is that I tried to do this check using my limited programming skills in R, but the results were less than 0.25.

4- Yes, I have used all my 31 parameters & the model has 30 parameters in it. Thus 30 DF. Thanks
...