Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

CBC: How to deal with the importance of non-signifcant attributes?

Hey,

I got a fairly simple CBC-design (3 attributes, 1 with 5 levels, 2 with 2 levels).
CBC counts show that one of the 2-level-attributes is not significant.
However, HB estimation returns an importance of 40% for that attribute, even though the utilities of its levels are fairly close to each other compared to the other attributes:

Utilities:

attribute 1:
level 1    58,85
level 2    39,12
level 3    2,03
level 4    -39,00
level 5    -60,99

attribute 2:   
level 1    11,88
level 2    -11,88

attribute 3:   
level 1    6,23
level 2    -6,23

Importance:
attribute 1        48,95
attribute 2    10,84
attribute 3    40,21

Attribute 3 is the non-significant one according to the counts. Simulations based on the HB run also show that it makes nearly no difference whether level 1 or level 2 of attribute 3 is present.
Still, the importances suggest that its relevance for decision behaviour is much higher.

Now I'm wondering how to deal with these results, especially how to interpret the importance. I'd be happy for any advice.
asked Feb 22, 2013 by anonymous
retagged Feb 22, 2013 by Walter Williams

1 Answer

+2 votes
Remember, counts just average across people, so it masks true importance if people disagree about the level preferences on an attribute (such as brand or color).

For example, imagine you interview 100 respondents.  50 of them believe:

Coke:  +50
Pepsi: -50

The other 50 believe:

Coke:  -50
Pepsi: +50

When you average (count) across the 100 respondents, the net utility is:

Coke:  0
Pepsi: 0

And counts will report that this attribute has a 0 Chi-square (no difference from proportional counts across the levels).  But, this doesn't mean that individual respondents believe brand has zero importance!

That's why importance scores from HB are far superior.
answered Feb 22, 2013 by Bryan Orme Platinum Sawtooth Software, Inc. (147,915 points)
Thanks Bryan,

While I was generally aware of this, I didn't consider it as a possible explanation as simulation results seemed to support the non-significance-hypothesis. I even simulated with several banner data and did not yet find one subgroup where attribute 3 affected choice behaviour.

My client will definitely want an answer on this, so I suppose I'll have to segment my data based on utilities for attribute 3 to identiy groups it does play a role for. Do you agree?
If samples are large enough, one could consider using Sawtooth Software's Latent Class to identify segments with different preferences.  Segment membership from Latent Class could be used to cluster HB’s utility scores while simulations (say Randomized First Choice) could predict each segment’s preference for different product options.
...