Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

How do I best specify the number of questionnaire versions in CBC

I estimate that I will get about 80-100 responses. Is it recommendable to generate 100 different questionnaire versions in CBC (to my knowledge 100 in recommended in general) and to take the risk of falling below that number? Or would you prefer to generate 80 different versions so that some respondents might get the same questionnaire?
(BTW: I have a 10x3x5 CBC Design, both numbers do not indicate deficiencies)
asked Jun 18, 2012 by anonymous

2 Answers

0 votes
I wouldn't advise to generate more designs than respondents.
The SSI Web generation tries to balance out over all of the designs. If you have only 80 respondents, you might not have enough observations for some of the levels/combinations since it needed the additional 20 to level out. In case that happens your CBC/HB estimation might still be deficient.
It also depends on how many prohibitions etc. you are using of course.
answered Jun 19, 2012 by anonymous
0 votes
The gain in design efficiency tapers off pretty quickly as you increase the number of versions you have, but you have an added benefit in preventing order bias by generating multiple versions of the design.  In SSI Web, most of the time it's going to take an extra few seconds to generate 300 versions instead of 30, so it becomes more of a "why not?" type of thing.

Honestly in my opinion, having 20 unused versions probably won't cause any harm, but an 80-version design should be quite satisfactory as well.  The other answer here is a bit misleading, because each version of a design is constructed to be balanced and orthogonal.  A lot of the time people are using a single design from a book or library of designs, so you don't "need" multiple versions, it just carries some benefits we think.

It probably is the "safer" choice to have 80 and repeat some, just so your design tests reflect perhaps your worst case scenario (indeed you should always be testing the design with your expected sample size/worst case scenario sample size).
answered Jun 20, 2012 by Brian McEwan Gold Sawtooth Software, Inc. (44,570 points)