Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Choosing number of tasks/atts for partial CBC

Hello,

I have been asked to program a partial conjoint exercise with 14 attributes in total (ranging from 2 to 5 levels), and I have been given "complete" freedom in choosing how many products/tasks/attributes to show. FYI: there are no prohibition to worry about.

Given that we are only collecting data on some of the attributes/levels per respondent, it stands to reason that the more products we show (or the more attributes, or the more tasks or any combination of the three) the better the data we collect and the better our results. After reading the technical paper on CBC advanced design module, I was left with the impression that one way to understand which combination would provide better data is to look at the "Strength of design for this model" score after I run an "Advanced test".

After generating several possible designs and running tests for everything, I can see that there are a number of options. For instance I can show 3 products per task, 7 attribute per product and include 15 tasks and have roughly the same design strength as if I showed 4 products per task, 7 attributes in 13 or even 14 tasks.

Another thing to look at is the estimated standard errors; assuming that for all these possibilities the standard error is less than 0.05.

How do I choose which design to use? When do I stop adding attributes/tasks/products in my design?

Thank you!

Orestis
asked Nov 28, 2011 by anonymous
retagged Sep 13, 2012 by Walter Williams
I should probably add that we are looking to recruit 300 respondents.

Orestis

3 Answers

+2 votes
You can always increase "Strength of Design" and lower your standard errors with partial-profile CBC by including more attributes per task.  But, at some point respondents get overwhelmed by the info on the screen and provide more response error.

In 2003, Patterson and Chrzan investigated the practical issues of partial-profile design with human respondents.  They found that showing more than about 5 attributes at a time was perhaps the tipping point where respondent error began to trump additional statistical information.

Read that paper at: https://www.sawtoothsoftware.com/download/techpap/2003Proceedings.pdf

Starting on page 173.
answered Nov 28, 2011 by Bryan Orme Platinum Sawtooth Software, Inc. (138,915 points)
0 votes
I am currently running a partial-profile CBC study consisting of 21 attributes (ranging from 3 to 4 levels). I present 5 attributes per concept and have 5 concepts per choice-task. From speaking to my study participants and reading their feedback comments, 5 attributes per concept is probably the limit for the number of attributes that participants can process at a time. My preliminary study results indicate study participants are capable of processing the 5 attributes/5 concepts choice-talk sets.

Victoria
answered Nov 28, 2011 by anonymous
I should also add that I present a total of 16 choice-task sets throughout the study and I have over 3,000 people in my study. Not everyone was capable of processing the 5 attributes/5 concepts combination, but two of the choice-task sets were included specifically to filter out people who were not reading the choice-task sets carefully.
0 votes
Hi both,

Thank you very much for your input - it was very helpful.

I am going forwards with showing the 5 attributes per product (as this appears to be the most efficient), 4 products (I believe I can squeeze 5 but then the question doesn't look very nice), which just leaves me with the number of tasks. With 15 tasks I have small standard Errors (all less than 0.03).

Orestis
answered Nov 29, 2011 by anonymous
One of the advice I was given when I was putting my study together was to make sure that every level of each attribute was presented at least three to four times throughout the entire survey. This helped me work out how many choice task sets I needed to present to ensure I presented the levels enough times to be able to calculate their part-worth utilities. I also remember reading that participants can process up to 16 to 18 choice task sets in one sitting. If your choice task sets aren't too demanding, you can get away with presenting 18 choice tasks sets and having more data to analyse. However, if your choice task sets are a bit demanding (such as mine with a total of 21 attributes presented), 16 is probably the limit and I have had a lot of participants tell me that they felt 16 was too much or that they wouldn't have been able to answer more.

I guess the number of choice task sets to present comes down to a) making sure you present each level of every attribute a certain number of times over and b) the difficulty of the choice task sets (the more attributes and levels you have, the more difficult it tends to be as you have more range in choices).

Victoria
Thank you Victoria, that was very useful. I went back and checked and I am showing each levels at least 4 times throughout the survey.
Regarding the number of tasks, I remember that you can go up to 20 tasks, but yes, the more complicated the task the less tasks one should show.
...