Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

RLH to identify bad respondents for CBC

We have a CBC with 8 choice tasks with 3 choice option each (5 attributes).

We believe that several of the respondents have just clicked through the survey and would like to exclude them from the analysis.

We came across the following article which describes using RLH for excluding "bad respondents" from a Max-Diff study:https://www.sawtoothsoftware.com/help/lighthouse-studio/manual/hid_web_maxdiff_badrespondents.html

We were wondering whether there are any suggestions for such a "Fit Statistic to Identify Random Responders with 95% Correct Classification" for a CBC study.

Obviously respondents below a RLH value of 0.33 should be excluded given that their value is even below the chancel model. However, we believe that respondents with a RLH value close to 0.33 (e.g. 0.4) still have respondent rather randomly.

Any advice from your side would be very much appreciated.

Best wishes,
asked Mar 8, 2018 by Stefanie

1 Answer

0 votes

It's a tricky thing to develop RLH guidelines for CBC.  Respondents who are answering purely randomly should have RLH's not much different from the chance level.  But, really high RLH's could be a sign of extreme respondent simplification: such as always picking None, or always picking the lowest price product no matter the other features.  So, I would caution against using RLH as the only measure to identify and discard respondents.  At minimum, run HB and look for respondents who have low RLH scores while also having very fast completion times.
answered Mar 10, 2018 by Bryan Orme Platinum Sawtooth Software, Inc. (169,915 points)