Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Why are most of my survey's respondents disqualified or incomplete?

I recently published my survey and most of the respondents are disqualified or incomplete. I do not have any completed responses.

asked Sep 27, 2018 by Robin
Hi Robin,

Did you place a terminate question at the end of your survey?  If you download or view your data, are people getting to the end of the survey?
Yes I added a terminate question at the end of the survey. Should I delete it?
No, ideally you want a terminate page at the end.  Can you check its settings?  Was the page set to mark someone as complete or disqualified?

Can you download your data into a CSV and see if people are making it to the end of your survey?
Just a comment to add to Brian's feedback: Is the terminate question that is submitting surveys as a complete (if there is one) being skipped?

1 Answer

+1 vote
I think the comments to this question cover most of the possibilities. To summarize, if you are getting a large number of incompletes or disqualified respondents, there are several potential causes and solutions:

1) Cause: It could be that your pool of respondents is poorly matched to your needs, so people are indeed dropping out or being disqualified in higher than expected numbers. Solution: Work with your list provider to make sure they are inviting the right types of people to your survey.

2) Cause: You have something wrong with how you are flagging people when they reach a termination page, or you don't have termination pages in the right places. Or, due to skip logic, respondents accidentally find themselves reaching a "disqualified" page when they should have continued on in the survey. Solution: Check your terminate pages and your skip logic. I've often seen cases where people have accidentally marked good completes as "disqualified" or vice versa. I find it useful to put all my terminate pages at the end of the survey ("complete" page first) so that people can only reach the "disqualified" pages if I purposely skip them there.

3) Cause: You have custom script in your survey that is redirecting respondents somewhere without hitting a "terminate" page. Solution: Move the custom script to a terminate page that respondents will hit. Or, use Lighthouse Studio's "redirect" feature on the terminate page.

4) Cause:  On survey launch, users occasionally misunderstand how we count completes. A respondent is not counted as a "complete" (nor are they counted towards quotas) until they hit a "terminate" page. So for the first 20 minutes or more that a survey is live, virtually everyone will be "incomplete" or "terminate-disqualified" because your good respondents haven't yet had time to complete the survey! Solution: Wait a while for respondents to begin completing their surveys. Then, if you still think things look out of order, review the first three options.

5) Final note: It may be helpful to run the "generate data" function on the Test menu. This goes through the survey randomly a number of times. If you have issues with your skips, terminate page setup, etc., you can often identify it ahead of time using this feature. If you've got fairly stringent screeners, you can pre-define values to help the data generator make it through the screener successfully. Then, export the resulting data and analyze it to make sure questions get answered by the right respondents, skips went to the proper places, and the correct terminate page is reached by each respondent.

If you need further help on your specific situation, please feel free to contact our support team directly at support@sawtoothsoftware.com.

Best wishes,
Aaron Hill
answered Oct 12, 2018 by Aaron Hill Gold Sawtooth Software, Inc. (10,995 points)