Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Randomization of CBC questionnaire versions


in a CBC survey we have 10 questionnaire versions with 12 choice tasks per version. There are two versions with a much smaller number of respondents than with the other versions (the surveys happened to be cancelled by respondents more often for the two respective versions). Thus, a few versions are under represented within the data. According to the Lighthouse studio manual, respondents are assigned to the questionnaire versions in a sequential order. How are the results affected, if the respondents are not approximately equally divided among all versions? Wouldn't it be better to randomly distribute the respondents to one of the questionnaire versions?

Best regards,
asked Jan 31 by Andrew Bronze (1,125 points)
edited Jan 31 by Andrew

1 Answer

+1 vote
CBC Analysis (for Sawtooth Software designs) is very robust to imbalance or even the complete absence of certain blocks (versions) in the design.  That is because each version is designed to have excellent properties of level balance and orthogonality that would allow it to stand quite well on its own (assuming enough tasks relative to parameters to estimate).

Therefore, I would not lose sleep at all over the fact that many more respondents had completed one version of our CBC designs than another.

Whether respondents were assigned randomly across versions or sequentially should not make much difference at all either.
answered Jan 31 by Bryan Orme Platinum Sawtooth Software, Inc. (172,790 points)

thank you very much! Would this also apply to other designs, e.g. imported efficient designs. I read about the loss of orthogonality in the data when blocks are underrepresented in an OMEP. However, do you know of any paper that addresses this issue (not sure if it is) in efficient designs?
So much depends on the nature of the non-Sawtooth design that you imported.  If such designs feature strong one- and two-way level balance within each block (version) of the design and if there are enough tasks for each version relative to the number of parameters to be estimated, then the under- or over-representation of blocks within the overall design should have very minimal effect on the overall efficiency of the design.  It would take simulating robotic respondent answers to both perfect representation and non-representative numbers of blocks across the designs, with analysis, to test this.
Thank you very much! This helps a lot.