sensitivity analysis doesn't "match" the utility report

Question

Hi, I'm running a CBC studies with 5 attributes. One of them is called "throughput" and it has 5 levels: 1-2, 3-4, 5-6, 7-8 and >8. The research question is should we build an instrument with 3-4 or 5-6 throughput?

In the utility report, 1-2 level has an average utility of -14, 3-4 is 6, 5-6 is 0, 7-8 is 6 and >8 is 2. So I interpret the result as respondents prefer a 3-4 throughput to a 5-6 throughput.

However in the sensitivity analysis, when I simulate the SOP, holding everything else equal, the SOP associated with 5-6 (21%)  is somehow higher than the SOP with 3-4 (17%).

I'm confused here. Which level is the optimal configuration then? 3-4 or 5-6?

Answer 1

In my experience, when looking at zero-centered diffs (where the average attribute has a range of utilities of 100 points), a difference of just 6 utility points on average between two levels is probably not statistically significant (unless you had a very large sample size).

Another thing to know is that the math involved in reporting average utilities (under the zero-centered diffs rescaling) is not the same as the math involved in computing shares of preference.  In the first case, we take an arithmetic mean.  In contrast, the share of preference (or randomized first choice) calculations involve an exponential transformation.  When the differences in utility are not very large, it often happens that you will see small disagreements between average utilities and share of preference reports.  This especially can occur in cases of relatively small sample size.

If you are confident that every rational respondent should prefer  greater throughput (holding all other attributes constant, including price), then you might consider re-running the utilities and constraining the utilities for throughput to be positive (worst to best preference order).  This can help in situations in which your prior belief is true and when this attribute doesn't seem to gain enough attention (relative to the other attributes) for your sample size to reveal a preference curve that avoids reversals (out of order preference ordering).

Comments

Thanks Bryan! It is very helpful!