Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

What is the best way to test modified CBC design?

We are running CBC ASD project and have done some recoding in the design. Recoding introduced into design decreased D-efficiency by 1% (compared to design original with no recoding), the one way level balance was ok.
Looking at preliminary counts we see that it happens that medium price is chosen more often than low price. Digging deeper into design I found out that there is level imbalance in terms of 2-way frequencies – medium price was more often presented with promotion (compared to low price, which turns out to be slightly more often presented with no promotion).
Is there any better test than examining D-efficiency to make sure recoded design will be orthogonal enough?
Do you see any smart way to compensate for some 2-way level imbalance at analytical stage?
asked Oct 4, 2017 by RafalNeska (170 points)

1 Answer

0 votes

D-efficiency is the metric for design quality.  Orthogonality is one component of efficiency, but efficiency is what we want to maximize.  So I'm not aware that there's a better test.  Looking at efficiency and at one- and two-way level balance the way you're doing really is a great way to evaluate designs.  

You noticed that medium price is being chosen more often than low price.  When you run your utilities, do you see that medium price has a higher utility than low price?  As long as your imbalance isn't too bad, the multivariate analysis should account for the fact that medium price and promotion are shown together more often and in apportioning utility it should separate the promotion and medium price utilities.  I would run utilities and see if this is happening.
answered Oct 4, 2017 by Keith Chrzan Platinum Sawtooth Software, Inc. (52,325 points)
Keith many thanks for your explanation. Unfortunately when we run HB using default settings the average medium price utility is higher than the average utility for the lowest price.  
Is there any way I could work around it?
It may be that your recoding created a nasty correlation between price and promotion that the analysis wasn't entirely able to sort out (I don't know your sample sizes or the number of choices sets you gave each respondent, but these could have contributed as well if you number of observations was too small.  A quick fix would be to impose a constraint so that lower prices to have higher utilities than higher prices, IF you think that is the truth.  This functionality is readily available in the software.

Be careful, though, about constraints.  Sometimes they impose a price order where none should be.  In some categories (we've seen this with categories like laser surgery and home door locks) the lowest price point may be an indicator of low quality and respondents might really prefer a medium price to a lower one.   Also, what was the orderings of the price levels before you did your recoding?
At the moment we have N=150 completed interviews.
We did not change the ordering of price levels when recoding.
I will probably impose constrains (I would not expect lower price to be indicator of low quality - it is the medium price that has highest scores), but can any HB setting be useful when dealing with some correlation between attributes?