Sawtooth Software: The Survey Software of Choice

Average utilities and simulations are not in agreement

When I look at average utilities, Level X has a higher utility than Level Y, but in simulations more respondents choose Level Y.

The most common reason for an outcome like this, which could also be extended to cover flat utility reports (levels are undifferentiated), is that you have a lot of disagreement in your respondent sample.

When you have categorical variables, such as Brand or Color, they don't inherently have a preference order.  It's possible that your respondents have vastly different opinions on how good the levels are. To use an extreme example, consider a beverage study study where you just happened to survey a group of respondents where 50% prefer Pepsi, 50% prefer Coke, and everyone is kind of OK with Sprite. You might end up with utilities that look like this for a "Prefer Coke"  and a "Prefer Pepsi" respondent:

  Prefer Coke Prefer Pepsi
Coke 90 -100
Pepsi -100 90
Sprite 10 10

Let's say you had 50 of each respondent. If you calculate an average across all respondents, you'll come up with:

Coke -5
Pepsi -5
Sprite 10

On average, there is no difference between Coke and Pepsi, and Sprite is preferred.  This turns out to be completely the opposite of what your data really tells you.

If you look at your standard deviation, though, Coke and Pepsi will both have about 95.5, while Sprite has 0. If you simulate a Coke, Pepsi, and Sprite, no respondents would choose a Sprite (assuming a first-choice rule). Additionally, if we had other attributes, Brand would also show up as being very important, despite the apparent low spread of the utilities. When you see these types of results, it's a good indicator you have a lot of disagreement among your respondents, and averages are probably going to be misleading.

Sawtooth Software

6:30 AM to 5:30 PM Mountain Time
(GMT-6; GMT-7 Autumn/Winter)
Monday through Friday

Phone: +1 801 477 4700
Fax: +1 801 337 7410

SKIM Software

9:00 AM to 5:30 PM Central European
(GMT+2; GMT+1 Autumn/Winter)
Monday through Friday

Phone: +31 10 282 3500
Fax: +31 10 282 3560

Lighthouse Studio

Lighthouse Studio is our flagship software for producing and analyzing online and offline surveys. It contains modules for general interviewing, choice-based conjoint, adaptive choice-based conjoint, adaptive choice analysis, choice-value analysis, and maxdiff exercises.

Try Lighthouse Studio

  Buy Lighthouse Studio