Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Using first HB and an then LC for Analysis - Does it make sense to do that?

Hi,

first, I  run an HB analysis and saw that the relative importances differ across respondents (sometimes even a lot), indicating that there might be lots of heterogeniety in the sample. Therefore, I conducted a LC analysis to identify potential segments. Indeed, LC shows me that 3 to 4 segments would be valuable to apply in my case according to certain criteria such as BIC and CAIC. Also, the number of respondents in each segment is relatively equally distributed.
However, since the results of HB concerning predictive accuracy and model quality are more precise than for LC, I am asking myself currently whether it would be valuable to first outline the results of HB, doing the tests (MAE, MSE, hold out sample, t-test for attribute levels) and then to conduct the LC analysis rather than to compare it with the logit model.

Do you have any advice for that? What would you recommend to do?

Also, is there any other approach to test for heterogeniety in HB (e.g. comparing one "hypothetical" or random respondent to the mean of the whole sample and conduct a t-test)?

Thanks for your help!
Best regards
asked Jul 18 by briniminii (490 points)

1 Answer

0 votes
Hello.

First off, you have to think about your objectives.  It sounds like your primary goal to predict well in a simulator?  If so, do you have enough (8-10 or more) holdout questions to allow you to do a powerful test comparing HB, LC and aggregate MNL models?   If so I would run all three models and see which one predicts your holdouts best.  If not, I would run LC if I wanted to make segments to include in my simulator but I would probably run my simulations on HB utilities.  For a lot of reasons respondent-level utilities will tend to give you better simulation results than the other models - this isn't a guarantee, and if you're in a position with enough holdout questions to test it you should, but if you are not, then I think you should default to HB.  

I'm not sure what you mean about testing for heterogeneity in HB.  I sounds to me like you've eyeballed it and found that you have what appears to be heterogeneity.  In mixed logit models run with the method of simulated maximum likelihood, some software packages provide the inputs for a significance test you can run to see if the heterogeneity you're seeing is significant.  I see that used more for academic reporting purposes than for decisions about whether we should allow for heterogeneity or not in applied models, however.
answered Jul 19 by Keith Chrzan Gold Sawtooth Software, Inc. (48,200 points)
...