Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Overfitting Note in v9 Release

Can anyone elaborate on the Lighthouse Studio v9 release note about changing the prior variance?  Maybe this is part of learning that will (or has?) been presented soon?

I have pasted in the note that I am referring to below.

"The default Prior Variance setting for HB estimation within CBC has been changed from 2.0 to 1.0 based on information we've learned about overfitting."

Thanks!
asked Feb 10, 2016 by Joel Anderson Bronze (1,310 points)
retagged Feb 10, 2016 by Walter Williams

1 Answer

+1 vote
 
Best answer
We've examined 45 commercial CBC datasets and are finding that the prior variance of 2 (recommended by academics from the 1990s) is never found to be optimal for these.  Lower prior variance is what is justified.  This isn't surprising since many leading researchers and academics have worried over the last decade that HB (at least using industry standard priors settings) has a tendency to overfit.  Kevin Lattery from SKIM (at the 2016 Turbo Choice Event) said his independent investigations on the same matter concur with what we're finding (that prior variance as high as 2.0 is never supported by the CBC datasets he's examined).  We'll be presenting the results of our meta study covering more than 50 datasets at the Rome SKIM/Sawtooth Software conference.  For now, we believe the evidence is in that prior variance of 1.0 is safer and closer to the right answer for our CBC customers than prior variance of 2.0.  We'll share even more guidance for making proper decisions based on the characteristics of your CBC studies in Rome.  See you there!
answered Feb 10, 2016 by Bryan Orme Platinum Sawtooth Software, Inc. (140,065 points)
selected Feb 10, 2016 by Joel Anderson
Thanks for explaining!  Looking forward to it.
...