Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Significant differences between sub-samples

Dear all,

we made use of ACBC in order to calculate preference of respondents. We then made use of CCEA software in order to identify groups of respondents with similar preferences (we opted for the 2-group solution).

Now we would like to identify whether there are significant differences in terms of preferences between two sub-groups.

I know that importances can be used for such an analysis of significant differences, but that does not help in our case too much. We really need to look at the differences in preferences between distinctive attribute levels.

I know that some scholars make use of the zero-centered part worth utilities for such an analysis (by making use of t-tests, or Mann-Whitney U tests when no normal distribution exists). I know also that the zero-centered utilities LIMIT the scale issue (but does the use of zero-centered utilities completely eliminate the scale issue?) However, I was once warned by a reviewer that this scale effect exists and he suggested using WTP.

Let´s have a look at two respondents:

1) Respondent 1:

40$: 71.9 utilities
80$: -71.9 utilities

Attribute level 1 (Attribute 2): -38 utilities
Attribute level 3 (Attribute 2): 72 utilities

2) Respondent 2:

40$: 87.2 utilities
80$: -87.2 utilities

Attribute level 1 (Attribute 2): -88 utilities
Attribute level 3 (Attribute 2): 45 utilities

Both respondents would have a similar willingness to pay when applying a traditional trade-off methodology.

So one can see that both respondents have a same WTP, but exhibit very different preference structures (also potentially different scale as the "range" between both attributes (1+2) for respondent 2 is higher than for respondent 1?

Do I make myself clear?

So my question basically is whether it is actually possible to make use of significant tests (t-tests or mann-whitney u tests) on the part-worth utilities to identify significant differences between distinctive attribute LEVELS.

Thanks for any help in advance!
asked Oct 22, 2014 by Stefanie

1 Answer

+1 vote
Practitioners will be quick to accept t-tests between respondent groups for ACBC by converting the utilities to zero-centered diffs (to normalize, equalize the scale)...then compute a regular t-test.  However, this is not very acceptable for formal statistics or formal journal reviewers.

Bayesians would prefer to estimate ACBC utilities under HB estimation with the groups as covariates, then open the alpha.csv file that records the draws of the alpha vector.  Using only the iterations after convergence is assumed, they would count the % of draws for which one group of respondents is higher than another group of respondents on the estimate of alpha for a given utility level.

If we were referring to utilities for a quantitative attribute (like price or speed), it might be advisable to estimate a single linear parameter for the attribute to capture its slope.   Then, the Bayesian test would only need to consider the difference in slope between covariate groups by examining the 1000s of draws of alpha after convergence.

I would think this Bayesian approach would be defensible for publishing in technical journals.

For more info on covariates in HB, see our white paper: http://www.sawtoothsoftware.com/downloadPDF.php?file=HBCovariates.pdf
answered Oct 22, 2014 by Bryan Orme Platinum Sawtooth Software, Inc. (132,190 points)