Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Run on the fly maxdiff utilities after the survey

I want to compare how well on the fly utilities for max diff match what HB generates.  Does one of the calculation options after you have your data reproduce those on the fly calculations?  If not, is there a way to get to them.

Thanks,

Steve
asked Oct 3 by stevetlg.com (290 points)
retagged Oct 3 by Walter Williams

1 Answer

0 votes
If you did not save them as part of the survey, you can recreate them as long as you are running a somewhat recent version of Lighthouse Studio (9.1 or later).

In the Analysis Manager, add an Analysis Run and set it to be Logit.  Click the Gear Icon and then click on Settings on the left.  Show the Advanced Settings and check the box to Recreate on-the-fly scores and rankings.
answered Oct 3 by Brian McEwan Gold Sawtooth Software, Inc. (37,085 points)
Thanks, that worked perfectly.  Based on some comparisons, we've stumbled on 2 additional questions related to this.  We are using maxdiff for a new measurement tool and in this tool, we don't require respondents to answer each page.  We've found that when someone does answer all the exercises, the post hoc run logit run does match the on the fly.  However, (1) if someone doesn't answer even one of the exercises (there are 48 of them with 4 shows per each of the 24 attribute) the utilities do not match.  We also see a lot of ties in the on the fly utilities - not the post hoc logit, those all look distributed and good.  Also, (2) if someone does not answer the first exercise, even if they answer all the others, the don't get any on the fly utilities at all - they are blank.

Do these two things ring true with expectations?  Is there a way to get the on the fly to always match the post hoc logit?

Thanks!
Interesting, I've got no experience to draw on for this at the moment.  MaxDiff questions are required and you are not able to disable that requirement.  How are you letting people skip questions?
Honestly, not sure, our programmers set it up.  We've been running this tool for the past year and were running into some issues.  Keith and Aaron were in Chicago with us a few weeks ago and they did some testing and could get on the fly utilities with even a bunch of missing data.  At the time, we didn't have insight into the results enough to follow up with the 2 questions I just posted.  Where we've narrowed our issues down to those two specific events.  Don't want to break protocol here, should I send them an email directly or is there someone else who might know?  Much appreciated!
Sure, shoot me an e-mail at brian at sawtoothsoftware.com and I can give you a link to upload files to so we can try to reproduce it.  I'd definitely want to confirm how the data was gathered, though, if something custom was done to allow the questions to be skipped.
...