Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Difference d-efficiency w/ and w/o BYO

Hi,

I do not quite understand the difference between both d-efficiencies. The problem I face is that my d-efficiency w/ BYO is about 0.95 while the d-efficiency w/o BYO is only 0.1.

This is due to the variation of the BYO which I reduced from 2-4 to 3-4 in order to increase the minimum levels have been seen. With a variation of 2-4 attributes (near-neighbour) I had a d-efficiency of 0.9 (w/ BYO) and 0.65 (w/o BYO).

I have included the BYO in the tournament.

Thanks in advance!!
asked Nov 22, 2015 by Mike

1 Answer

0 votes
ACBC generates a set of "near-neighbor" cards that have a lot of similarities to the original BYO "ideal" product the respondent specifies.  To generate the near-neighbor cards, it does some swapping and level relabeling steps to see if it can improve the D-efficiency (while still holding strong to the degree of oversampling the BYO-selected levels).

When ACBC computes the final utilities, it uses information from not only the choices among the near-neighbor cards, but the information provided in the BYO selections, where those BYO selections are encoded as additional choice tasks (best level out of all levels of each attribute, formulated as separate tasks).

If when ACBC goes to estimate the D-efficiency (to evaluate whether swaps and relabeling are helping), if the researcher has asked the software to generate a generous amount of near-neighbor cards per respondent, then D-efficiency is >0.  But, if the number of cards is too few, the D-efficiency evaluates to 0 and the algorithm would have no compass by which to try to improve the efficiency of the design via swaps and relabelings.  That's why if the D-efficiency evaluates to 0 for too many tries during the process of generating the near-neighbor cards, the algorithm adds the additional information from the BYO tasks to the D-efficiency estimation.  Now, D-efficiency computations will be >0.

So, the fact that the design report shows you that the D-efficiency without BYO tasks included is super low means you may be treading on the sparse side of information content per respondent.  But, it doesn't necessarily mean your study will be bad (maybe you have enough respondents to overcome less precision at the individual level).  Some researchers purposefully field sparse ACBC or CBC designs, depending on their goals.  Use the design report to count how many times each non-BYO level is shown per respondent.  The goal for quite precise individual-level estimation is around 3 times or more occurrences per non-BYO level per each respondent.  

Remember, ACBC's strategy is to oversample BYO-selected levels.  Oversampling levels really hurts your D-efficiency relative to level-balanced designs (like traditional CBC designs).  So, we expect D-efficiences for ACBC designs at the individual level of often below 0.6 (though once you augment the data matrix with the additional info from the BYO tasks, you see the D-efficiency becomes much higher).  

With ACBC, we know we are sacrificing a lot of D-efficiency for the purpose of oversampling the most relevant levels per respondent.  Many split-sample studies now have proven that sacrificing a great deal of D-efficiency for the benefit of the adaptive procedure of oversampling relevant levels actually works better in the end for ACBC in terms of being able to estimate high quality part-worth utilities at the individual level.
answered Nov 23, 2015 by Bryan Orme Platinum Sawtooth Software, Inc. (146,540 points)
...