Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Why is it more proper to conduct Bayesian Testing for comparing groups of respondents?

Dear Sawtooth Team,

when searching the forum and the „Becoming an Expert“ book it gets clear that there are generally two ways to compare groups of respondents regarding their preferences, when HB analysis is used: The frequentist test and Bayesian testing using group membership as covariate.

There exists various indications that Bayesian Testing is „more proper“ and „statistically more true“ than the frequentist approach. The Becoming an Expert book also specifically refers to this point.

But what are the reasons why Bayesian testing is regarded as being more proper?

Thanks for your help!
asked Jun 22, 2018 by anonymous

1 Answer

0 votes
If using HB to estimate individual-level utilities, the standard way Sawtooth Software handles the results is to collapse the many draws (after convergence) per respondent to create a single point estimate of the utilities.  This is a shortcut practitioner's trick that makes it faster and easier to deal with the utility results during analysis, since there is just one case per respondent rather than 100s or 1000s.  If you have the patience and the sophistication to leverage the many draws per respondent in analysis (such as market simulations), it's considered more true to HB and you may get very slightly better results.

Taking the mean of the draws throws away the granular uncertainty along with additional covariance structure in the individual-level draws.  HB also involves some smoothing of each respondent's parameters to the population means.  So, to take those point estimates and treat them as if they were independent estimates for respondents appropriate for frequentist statistical testing (such as t-tests and f-tests) is not statistically proper.  

If using HB analysis to estimate the individual-level utilities, then Bayesian tests (as are described in the references you cite) are more proper to use rather than frequentist tests on the point estimates.
answered Jun 22, 2018 by Bryan Orme Platinum Sawtooth Software, Inc. (163,515 points)