Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Goodness of fit for ACBC


I am currently estimating different HB models (ACBC) by altering the price function (log, linear, especially different piecewise functions).

So far I understood that Pct. Cert. and RLH are better, the closer to one (obvious). However, I am not sure, whether I understand Avg. variance and RMS right. In the CBC/HB technical paper it is said on page 26 that both values predict a better fit if higher? Do I understand this right?

Here is an example of two of my HB models (ACBC). Pct. Cert. and RLH are quit the same. However, which model is better concerning Avg. Variance and RMS? According to the technical paper it should be the first (left) one?

Pct. Cert.:         0,532          0,516
RLH:                    0,665          0,675
Avg. Variance: 3,523          2,453
RMS:                   2,199          1,867


asked Dec 29, 2015 by Mike

1 Answer

0 votes
Both Avg. Variance and RMS are proxy measures that (when larger) can indeed indicate greater fit at the individual level.  But, obtaining better fit at the individual level is often counterproductive to getting a good model!  Choice data are sparse and getting too much fit to individual choices is usually a bad thing.  There typically needs to be Bayesian smoothing to upper-level parameters (due to the sparse nature of the data) to obtain better models.  When you smooth to upper-level parameters, you sacrifice a lot of fit at the individual level, but obtain better fit (density) to the upper-level parameters.

For example, in HB, you could obtain much better Pct. Cert, RLH, Avg Var, and RMS simply by setting the prior variance higher.  It doesn't mean you're getting a better model, though.

We are currently conducting research on HB estimation for CBC and MaxDiff wherein we're finding that most HB models we're fitting right now with the defaults we have in our software lead to overfitting.  Using different priors (such as lower prior variance) can avoid the overfitting, but each data set is unique so one cannot simply give a blanket policy regarding the proper prior variance settings for all CBC and MaxDiff studies.

ACBC often tends to be less sparse than CBC data, so maybe we aren't overfitting so much in general.  But, it's still an issue to watch.
answered Dec 29, 2015 by Bryan Orme Platinum Sawtooth Software, Inc. (148,140 points)