Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Disqualification or cleaning criterion for ACBC/HB studies


In the standard CBC module we have the luxury of having a lower RLH estimate produced during the estimation that allows us to Identify respondents who are (likely  to be) answering tasks at random. I understand the derivation of this lower RLH stat is extremely difficult in ACBC/HB studies due to the fact that many respondents see different combinations of choice sets and tradeoffs (based on how they answer the various sections of the ACBC). This being the case is there a substitute measure we can use that might highlight respondents answering at random - other than looking at things like speeding / straightlining etc.

Is running a monotone regression and using the Tau statistic of any use? I'm guessing no for the same reason as RLH, but I'm looking for a way to eliminate troublesome respondents which we all know exist on panels.

Any help?
asked Jun 29, 2017 by Jasha Bowe Bronze (1,680 points)

1 Answer

+1 vote
Best answer
I've not done this before but the thought comes to mind to use a combination of our data generator and maybe take the survey yourself a few times.

The data generator answers questions randomly and could perhaps provide a bit of a baseline as to what kind of an RLH you could expect by a poor respondent (simulate many respondents to get a feel for the range).  I would probably also take the survey a few times myself and just answer randomly as well.

I'm not sure if it's a perfect approach since ACBC does not provide a perfectly balanced design to allow people to contradict themselves as much, but this approach plus looking at the time people spent on the page could perhaps be a nice proxy for identifying a "bad" respondent.
answered Jun 29, 2017 by Brian McEwan Gold Sawtooth Software, Inc. (37,410 points)
selected Jul 3, 2017 by Jasha Bowe