Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Procedure for Express Max Diff hosted by external survey vendor

I am conducting a max diff with 200 items and will be using an external vendor to program/host the survey (so the survey will not be programmed/hosted via LightHouse Studio; I will need to send the .csv max diff design file output from LightHouse Studio to the programming vendor AND receive max diff data for analysis from this vendor). I will also need to compute respondent-level HB scores for this max diff.

I considered using Sparse Max Diff (showing each item once) but decided against it because it will result in more max diff exercises per respondent than I'm comfortable with given respondent fatigue.

Instead I'm considering using Express Max Diff (where each respondent completes max diff with a randomly selected subset of 30 items from the total 200 items) - BUT am not sure of the steps to design/analyze in LightHouse Studio given that it will be programmed/hosted by an external vendor. What would be the procedure? - for example:
(1) In LightHouse Studio, create a max diff design for 30 items per the usual process with, say, 100 versions. Output the design to .csv.
(2) Send this .csv design to the external programming vendor along with the full list of 200 items AND tell the vendor to randomly select 30 items (from the 200 item list) to use for each respondent's 30 max diff items.
(3) I don't know what to do in terms of:
(a) What max diff data to get from the vendor.
(b) how to read this data into LightHouse Studio via the paper-and-pencil Accumulated Data Template File for HB analysis.
(c) how to run HB in LightHouse Studio where it will impute scores for items not asked in each respondent's max diff - so that I end up with respondent-level max diff scores for all 200 items.
Thank you.
asked Feb 21 by anonymous

2 Answers

0 votes
I would not try to do this with Express MaxDiff.  Rather, I'd continue with the Sparse MaxDiff approach--if respondents cannot see each item 1x, then you can do the Sparse MaxDiff approach with each respondent seeing each item fewer than 1x on average.  

For example, imagine you have 200 items and you plan to show 5 items per MaxDiff set, for 20 total sets.  That means under Sparse MaxDiff each respondent will see half the items.  Make sure to click the button in the Design tab (under Show Advanced Settings that says "Allow Individual Designs Lacking Connectivity").

However, I would prefer not to use HB for analysis when each respondent hasn't seen each item.  I would think you would be limited to pooled analysis, such as aggregate logit for the sample.

And, given the sparse nature of your design, I would do some pre-planning to evaluate how many times each item would be displayed across respondents.  My personal preference is that for large sparse designs like this I want each item to appear at least 1000 times across the sample.  So, since it takes two respondents to see each item once, it means you'll want to have at least n=2000.

If the main purpose of your research is to identify the top winning few items out of the 200, then the Bandit MaxDiff approach as supported by interviewing in our Lighthouse Studio platform is 3x to 5x more efficient than sparse MaxDiff.  In other words, rather than using 2000 sparse MaxDiff respondents, you could obtain equally precise results (in terms of identifying the top few items) using from 400 to 667 respondents.
answered Feb 24 by Bryan Orme Platinum Sawtooth Software, Inc. (174,440 points)
0 votes
I would recommend to follow Bryan's suggestion but if there is no way and you have to do express max diff then I have a solution for you.

We have  done this in the past using the following steps:

1. Create a dummy lighthouse file with express max diff
2. If sample size is X, generate data for X respondents using data generator
3. open csv file and copy paste actual data in max diff variables and list of selected attributes in constructed list variables
4. Import this new csv file and estimate utilities as if max diff is administered in lighhouse.

I hope this helps.

answered Feb 25 by Saurabh Aggarwal Gold (33,400 points)
edited Feb 25 by Saurabh Aggarwal