Sawtooth Software: The Survey Software of Choice

Writing Better CBC Questionnaires: An Empirical Test

How can we motivate respondents to give reliable, thoughtful, and most importantly realistic responses to CBC (Choice-Based Conjoint) questionnaires? This is especially a concern as the research community fields a greater number of projects among respondents who complete multiple surveys per month. Internet surveys have accelerated the pace of research projects, and respondents seem to have accelerated their processing speed as well. There are certainly cost benefits to both trends, but overall quality may suffer.

Study Design

In August, 2006, we fielded a CBC study over the internet using GMI (Global Market Insite) sample. The subject matter was laptop computers, and we screened respondents for interest in the category and basic knowledge about laptop features. The CBC questionnaire consisted of nine choice tasks designed for utility estimation and six holdout choice tasks (identical in appearance to the utility estimation tasks) sprinkled throughout the other nine tasks (three early in the questionnaire, and the same holdouts repeated very late in the questionnaire, with concept position rotated).

After deleting respondents who were clearly speeding through the interview or providing clearly inconsistent responses (test-retest reliability check), we had 379 respondents for analysis.

Every respondent received an instructional screen prior to the first choice task. The core text (that all respondents received) for the instructional screen was:

We want to learn what aspects are important to you when purchasing a laptop (notebook) computer. To do this, we are going to ask you a series of tradeoff questions.

In each question, the computer running this survey will come up with three different laptop PCs to choose from. Sometimes, these notebooks will have very good features at very reasonable prices. Sometimes, they will have poor features at relatively high prices.

If you really don't like any of the options, you can say that you wouldn't purchase any of them. That's just fine.

We tested six questionnaire writing elements (all as binary factors) that we felt might have a positive effect on the quality of the CBC part-worths:

  1. Explain Reason for Repeated CBC Tasks. In the instruction screen just prior to the first CBC task, we either included or excluded the following additional paragraph:

    We need to ask you repeated questions to learn how you make sometimes complex tradeoffs. If we see how you select laptops in many different situations, we'll do a much better job at learning about your preferences. So, what may seem to you like repetition is very useful to our research study.

  2. Appeal to Respondent to Provide Realistic Answers. In the instruction screen just prior to the first CBC task, we either included or excluded the following additional paragraph:

    We're really depending on you to answer realistically and carefully. Please imagine you were actually shopping for a laptop and really were going to pay for a laptop you might choose during our survey today. Thank you for your effort!

  3. Progress Bar. A Progress Bar was shown (or not) in the footer of the questionnaire.
  4. Instructional Screen after First Choice Task. After the first CBC question, we either displayed (or not) a screen that said:

    Now, the computer running this survey is going to rearrange the brands and features of laptops. This next question is going to have the same layout as the previous one, but the combinations of features for the laptops will be different.

    Again, each time you answer a choice scenario, the computer will scramble the features for the next question.

  5. Countdown after Every Third Task. After each third task, we included (or not) a label like the following at the upper left-hand corner of the choice task:

    (3 of 15 choice scenarios)

  6. Rest Screen after 8th Task. After the 8th choice task, we either displayed (or not) a separate screen that said the following:

    Thank you for your work so far. We know that some of these tradeoffs are challenging. We hope you also find it interesting to consider what features are important when considering a laptop.

    The answers you give to these choice scenarios are very important to help us understand your opinions and preferences. Keep up the good work!

Findings:

We estimated individual-level part-worths using our CBC/HB system. Using multiple regression, we tested whether these binary factors had any effect on the following ten dependent variables:

  • Interview Length
  • Test-Retest Reliability for Holdouts
  • Internal Fit (RLH) of Calibration Tasks from HB
  • Number of Utility “Reversals” for Ordered Attribute Levels
  • Respondent’s Qualitative Feedback on the Questionnaire: 5-point Likert scales reporting whether the respondent found the questionnaire...
    • enjoyable
    • confusing
    • easy
    • made them feel like clicking answers just to get done
    • made them feel like they could express their opinions
  • Importance of Price

Only one effect was significant at p<0.01. Given the number of repeated t-tests (10 regression models x 6 independent variables), it raises the possibility that this effect and especially the few others that emerged at p<0.05 may have been due to chance alone.

Here is the one significant effect at p<0.01: The average interview time was 302 seconds (just over 5 minutes). If the “Countdown” was provided on every 3rd task, respondents took an extra 60 seconds to complete the survey (p<0.01). Why they took extra time is an interesting question. Perhaps the Countdown kept respondents more engaged by giving them a clear indication of a finite number of choice tasks, so they could maintain focus and be less likely to become discouraged (not knowing if/when the end was in sight). Perhaps respondents then gave more attention to each choice task and put forth more effort? The effect of Countdown on test-retest reliability and reversals also suggested it improved data quality (but both effects were non-significant).

So, we are left with evidence suggesting that including the Countdown leads to respondents going slower, and that the Countdown potentially leads to better quality data.

Comments:

Why didn’t more of the experimental factors have a significant effect on the dependent variables we examined? For example, the Appeal to Respondent to Provide Realistic Answers might have led to an increase in price sensitivity, yet it didn’t have a significant effect on price importance (though the coefficient had the expected sign).

Perhaps respondents in online panels are so practiced (and quick) at doing survey research that they are predisposed to commit a certain level of cognitive effort and are not easily swayed by instructional text or visual cues meant to influence their behavior. This reminds us of the finding that that price sensitivity increased with later tasks as reported by Sawtooth Software’s Rich Johnson and Bryan Orme after examining about 20 commercial CBC datasets collected in the early- to mid-1990s. Jon Pinnell (MarketVision Research) has looked at this same issue multiple times based on online panel and has not confirmed this result. One hypothesis is that there were stronger learning effects among respondents to (mostly disk-by-mail) CBC interviews in the early 1990s than there are among online panelists post-2000. There is no doubt that we are dealing with a larger pool of respondents in these online panels who bring to each new survey the experience gained from frequent survey-taking experiences.

This research effort was just one modest experiment using a single sample source. Much more should be done in this area before reaching firm conclusions regarding how questionnaire writing elements affect CBC results. Based on our experience (and suggestions from others, such as Jordan Louviere, who has long advocated a task countdown), the six elements we tested above should lead to higher data quality (the exact wording of the elements might be improved upon, though we think our effort approximates best practice). The results from this single test suggest only one of the elements we tested (the “Countdown”) leads to better data quality, but this certainly should not be taken as the final word on this subject.

(Editor's Note, March 2007: A follow-up study analyzed in February, 2007 found that Countdown had no effect on interview time. Therefore, the finding reported here is questionable. Even so, we recommend using a Countdown in CBC surveys.)

Future Research

We at Sawtooth Software have become increasingly concerned about data quality in CBC surveys. We suspect that researchers are not getting as much (and the right kind of) information at the individual level from standard CBC surveys as they had supposed. Taking a complex CBC interview with many attributes can be a monotonous and mind-numbing experience, so perhaps it’s mostly our fault as researchers if data quality suffers. We’re investigating new ways to address these issues and hope to have useful results to report at the upcoming Sawtooth Software conference in October.