Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Comparing Subgroup Utilities (like Male vs. Female respondents)

SMRT allows the option to select different subgroups of respondents when calculating utilities. How do I correctly compare the results of scaled utilities from one subgroup to another? (Without using latent class or hierarchical bayes, just the standard software)

For example, I run the same CBC conjoint analysis on 200 people. 100 men, 100 women. SMRT allows me to calculate utilities for a) all 200 respondents b) just selecting men, c) just selecting women.

However, the utilities and scaling of the numbers are different between a),b) and c). How do I properly interpret or re-scale the utilities in order to compare the results?

I would like to objectively know what the differences are between men and women for:
-Importance of attributes
-Utilities of attribute values

How do I correct for scaling issues?

Thank you very much for your answer!
asked Sep 28, 2012 by anonymous
retagged Sep 28, 2012 by Walter Williams

3 Answers

+3 votes
Please be warned that you probably cannot correctly compare attribute importance between groups of people if you are just using the standard aggregate logit utility estimation routine.  That is because aggregate logit will often lead to seriously misleading attribute importance calculations due to not recognizing differences between people's preferences.

For example, imagine that half of the respondents love brand A and half love brand B (and these are the only two levels in your brand attribute).  Imagine this attribute is extremely important.  Aggregate logit simply averages across everybody, leading to utilities that are tied for Brand A and Brand B.  So, the standard importance computation looks at the difference between Brand A and Brand B preference (no difference) and reports that Brand has zero importance for the sample.  This is obviously not correct.

That's just one of the reasons that HB analysis (the standard way to analyze CBC data by our users) is so valuable.  Each respondent's utility scores are estimated, so that the importance calculation is done within each individual, and the importance results are averaged across people.  Under HB, the average utilities for the sample will still be reported to be tied, but the importance calculation will show that the importance for Brand for the sample is very large (since it is computed at the individual-level).

And, the software (SMRT) will allow you to split the HB utility results out by Male and Female, so you will see the zero-centered diffs (normalized utilities) summarized for the Males and the Females.

Using zero-centered diffs helps adjust for the fact that with CBC analysis (utility analysis) the raw utilities can be much larger magnitude for a group of respondents that answers with less error than another group.  These differences can at times be very large...so the zero-centered diffs normalization affords some protection against making wrong conclusions regarding the utility one one level for one group vs. the same level for a different group.

Of course, importance scores (that are normalized to sum to 100) automatically take out differences in scale factor between people or groups.  Importances calculated from zero-centered diffs or raw utilities are exactly the same (except for maybe a tiny bit of rounding error).
answered Sep 28, 2012 by Bryan Orme Platinum Sawtooth Software, Inc. (160,785 points)
edited Sep 28, 2012 by Bryan Orme
0 votes
each effect has it's standard error in smrt output. You can simply use standard t-test for independent means. Compute pooled error for mean differences using standard errors for effects in each group, then proceed as usual
answered Sep 28, 2012 by lkomenda Bronze (2,830 points)
Follow up on comparing groups
0 votes
There is a formal statistical test for scale and utility differences in aggregate MNL models.  The test is a little complicated to run and the vast majority of times you run it you learn that you have differences in utilities (I've heard that running the test on data sets constructed to have equal utilities produces a result of identifying significant differences about 60% of the time, so the test seems too liberal).

I think what people may do more often in practice is an eyeball test:  plot mean utilities for (e.g.) males and females on an X-Y axis and identify any points that fall at all off the line as likely differences.  If the points do land all on the line, conclude that they're telling the same story.
answered Sep 28, 2012 by anonymous