Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

MaxDiff: Best alternative Sparse or Express

Hi

I need some advice regarding setting up the best design for a MaxDiff.

I'm attempting to assess 50 items. The business has asked specifically for me to segment respondents based on their responses, so I'd like to be able to estimate individual parameters using HB and create segments via Latent Class (Ideally I'd like to be able to simultaneously estimate item scores and segment respondents).

I'm also very conscious of cognitive load so I don't want to overburden the respondents either, so I'm trying to figure out the best option.

I've read the papers on Sawtooth's website which appear to indicate that Sparse MaxDiff performed the best on simulated data with regards to creating stable results and segments.

However, I'm a bit confused on what is the best approach. I was thinking these were my three options:

Express Design (option 1)

Create two design blocks  of 30 questions. The design would be:

Number of items: 30
Number of alternatives: 5
Number of questions: (3 * 30)/5 = 18 questions

Under this design, each person answers 18 questions, and each person answers 5 items per question.  Therefore on average each respondent evaluates 5*18= 90 items. Does this mean every item is 50/90 =  0.55 x? So, if wanted ensure a large enough sample to do HB segmentation and each item appeared 1000x across the sample, I’d need a sample of at least 0.55 * 1000 ~ 550 people?

Express Design (option 2)

Create three design blocs of 30 questions. The design would be:

Number of items: 20
Number of alternatives: 5
Number of questions: (3*20)/5 = 12

Under this design, each person answers 12 questions, and each person answers 5 items per question.  Therefore on average each respondent evaluates 5*12 = 60 items. Does this mean every item is 60/90 = 0.833x? So, if wanted ensure a large enough sample to do HB segmentation and each item appeared 1000x across the sample, I’d need a sample of at least 0.833 * 1000 ~ 850 people?

 And if I wanted a sparse design,
Number of items: 50
Number of alternatives: 5
Number of questions: (1*50)/5 = 10

Are these the correct calculations?

And which option, based on the requirement of segmentation is the best option?

I'm particularly unsure how to design the blocks for Express Max Diff? The examples I've seen are for 100 items where the analyst has generated 20 design blocks creating 30 items each. But I don't understand why they've used these thresholds.

Would it be possible for someone to explain to me the arithmetic of generating an appropriate number of design blocks for 50 items, including required number of respondents to do a meaningful segmentation?

Or is better to persist with a sparse design only?

Thanks
asked Oct 25 by DavidBeri

1 Answer

0 votes
 
Best answer
David,

I would definitely use the Sparse design.  You could go with the sparser design you describe at the end of your email, with 10 questions of 5 items each, but I think you'd do better to be a little less sparse and have 20 questions of 5 items each and two item-views peer respondent.   

We've now conducted a pretty definitive empirical test of Sparse and Express using human respondents, something we'll be sharing at our Turbo Choice Modeling event in March (Turbo Choice happens in the two days prior to our main Sawtooth Software Conference:  http://www.sawtoothsoftware.com/events-hidden/1839-2018-conference).  Whereas our artificial data showed a slight advantage for Sparse over Express in terms of the validity of respondent-level utilities, with human respondents the advantage is much larger and still in favor of  Sparse.  We even looked at the ability of Sparse and Express to reproduce the segments you'd get from a more fully-informed MaxDiff (3 item views per respondent) and Sparse reproduces segments MUCH better than does Express.  

I don't want to give away too much, because my colleague Megan Peitz will be showing these results in March, but those are the high level results which I think should steer your decision.
answered Oct 25 by Keith Chrzan Gold Sawtooth Software, Inc. (48,525 points)
Thank you Keith, very helpful!
...