Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Is a 20 version CBC full profile balanced overlap design better than 100 version design?

Hi ,
I am facing a dilemma where do i create a CBC with 20 versions or with 100 versions for a sample size of 200? there are no restrictions hence it is a full profile balanced overlap design. I looked at Efficiency and Err margins of levels and they all come good in both the designs. So my set of questions are
- What are the drawbacks of a 20 version design vs 100 version design for 200 respondents?
- What are the gains of a 20 version design vs 100 version design for 200 respondents? Since in a 20 version design each version gets 10 responses and in 100 version design every version gets 2 responses, does that affect the estimation in any way?
asked May 24, 2016 by vaib13 (125 points)
I'm not sure if the Advanced Test will help you at all if you plan to use HB estimated utilities. I learned that the Advanced Test is only helpful when Logit estimated data will be used.

1 Answer

+1 vote
If you've run the advanced test and looked at standard errors and the strengths of the two designs, and if they look good both ways, I think you've already answered your questions:  it sounds like there's very little practical difference.  Some folks might prefer the smaller design so that they get multiple views on each version, but others might prefer having more versions to cover more of the design space.  But if the statistics, the standard errors and the efficiency, say it doesn't matter, then I'd say it doesn't matter and which you choose is up to you - it's not a "choosing the right thing or the wrong thing" sort of decision at all.
answered May 24, 2016 by Keith Chrzan Platinum Sawtooth Software, Inc. (63,000 points)
Thanks. In general Sawtooth website recommends having higher number of versions i.e. 100 at least for online/web based conjoint. Why do you think that is so? Is having more design space more important than number of responses on each version? given that estimations is HB.
Yes, we do.  Greater coverage of the design space enables you to estimate interactions more easily.  In a paper available in our 2013 conference proceedings, my colleague Aaron Hill and I found that version effects explain only a small amount of the heterogeneity we see in respondent-level utilities, so worries about version effects shouldn't incline one to prefer a study with 10 versions over one with 100.  

That said, there are diminishing marginal returns to adding more versions, and  beyond about 30 versions, gains in efficiency and in ability to measure interaction, often get quite small.
...