Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Scale of part worths estimates in CBC/HB


I'm doing some simulation work with the classical CBC-HB-Model:

1) I generated hierarchical part worths ( alpha )  and a covariance matrix ( D )
2) I generated individual part worths (Beta_i) from Normal(alpha,D)
3) For a given (balanced overlap) design ( X ) I calculated individual choice probabilities
4) With these choice probabilities I generated individual chocies y_i from the multinomial distribution

So the parameter generated in 1) and 2) are the true values to be recovered in the estimation.
The estimation has been done with X and y_i as input and default settings with 200000 iterations (with 100000 as burn-in).

The Beta and alpha estimates from CBC/HB are very high correlated with the true values but they are all larger by the factor =~1.45, i.e. estimate=~ 1.45* (true value).

My question  is where this scaling factor is coming from? (It's different from the theoretical scaling factor sqrt(pi^2/6)=1.28). Is this value to be expected in all simulation studies with CBC/HB?

Many thanks.
asked Nov 7, 2011 by Vladimir (120 points)
retagged Sep 13, 2012 by Walter Williams

2 Answers

+3 votes
In the HB algorithm, the utilities tend to 'fall where they may'.  HB uses effects coding and thus utilities within an attribute are zero-centered, but it doesn't do any rescaling.  However, there are things that tend to influence the scale factor:

1) The prior degrees of freedom
2) The prior variance (used to create the prior covariance matrix)
3) The number of levels in an attribute
4) The amount of discrimination among respondents

In general I would not be concerned about the utility scale unless one attribute has a drastically different scale than the others.
answered Nov 7, 2011 by Walter Williams Gold Sawtooth Software, Inc. (15,075 points)
Thank your for the quick answer. My intention was to see how many iterations does it take to achieve the convergence to the true values. Primarily I wanted to do it via tracking the MSE (mean square error) of the parameter estimates during the estimation. But if I don't know the right scaling factor I cannot calculate MSE.
Except for correlations what mesaures comparing true and estimated values would you use to show/track the convergence?
How to assess convergence is a good question, and I don't know of a good way to do it mathematically.  Others who understand it more might be able to give you more direction.

In CBC/HB people usually use the visual graph to see if the data have converged.

If there is a good and robust way to determine convergence, that's the kind of feature I might want to add to v6!
+1 vote
The amount of error that synthetic respondents are projected to use when answering the questionnaire has a huge effect on the scale factor for the estimated utilities.  When I create synthetic respondents and add Gumbel error to the total utility of each alternative before projecting the first choices, I can exactly recover the right scale factor through aggregate logit.  But, I have found that when I do the same with CBC/HB, I'm off by a scale factor.  I've heard of others experiencing this same issue, but I cannot recall if there is a solution for choosing the right scaling of the Gumbel error together with the settings in the CBC/HB setup (prior variance, prior D.F.) that leads to exact recovery of the original scale factor (as baked into the original "true" synthetic utilities.  Perhaps one of the HB experts out there has the answer?
answered Nov 7, 2011 by Bryan Orme Platinum Sawtooth Software, Inc. (132,190 points)
Thank you for your insights. Now I've understood where the difference come from. I didn't realise it before, that HB can behave different than aggregated logit in the matter of scaling. I will keep on this issue, and if you want I can share with you if I find something out.