Sawtooth Software: The Survey Software of Choice

Report on Conjoint Analysis Usage 2011

Thank you to those who participated in our annual customer feedback survey. This tool serves as a measuring stick for our business, and allows us to keep in touch with both the overriding sentiment and the specific ideas of customers. We learn how we can better serve you, and what we can do to improve our products.

Here are some top-line findings you may find interesting:

  • 96% of respondents rated their interactions with us as either Good or Excellent.
  • MaxDiff (Maximum Difference Scaling) was used by 52% of the respondents’ firms during the previous year.
  • 52% of you said it would be a positive development if Sawtooth Software offered custom Javascript and Perl services for assisting with challenging SSI Web questions. 2% said it would be a negative development for you.

Of course, an important element of the annual survey is to track the usage of conjoint analysis methods. Respondents were asked how many conjoint analysis projects their firm (or department) conducted during the previous year. Then, they specified what percent of those projects utilized different conjoint-related methodologies. The vast majority of the projects employed Sawtooth Software tools, and among those projects, the percent of projects utilizing CBC, ACBC, ACA, and CVA were as follows:

ss33c

Choice-based conjoint methods (CBC and ACBC) continue to take share away from ratings-based methods (ACA and CVA). In its second year of release, the use of ACBC has increased to 11% of projects. We expect that as alternative-specific designs are supported in the future version of ACBC, its use will continue to grow.

The continued dominance of choice-based methods over the last few years might lead some to wonder whether there still is a place for ratings-based conjoint methods (CVA and ACA) in our practice. What this chart doesn't show is that many researchers use multiple conjoint techniques, depending on the research situation. Although CVA (card-sort conjoint) is the oldest conjoint technique, with its roots dating back to the 70s, it can be quite useful in specific circumstances. ACA is frequently used by researchers who tend to use CBC for the majority of projects. There are some situations in which ACA's clever hybrid conjoint approach works nicely.

The application of MaxDiff methodology is also increasing. During the last year, 52% of respondents to our survey indicated that their firm had used MaxDiff during the previous year. MaxDiff was introduced as a software product in 2004, so this represents significant growth and usage.

ss33d

We also asked respondents who stated that their firms had used CBC to indicate what percent of CBC projects employed different design options. The “Balanced Overlap Design” method, which forces a modest amount of level overlap into the designs, has been shown to improve measurement of interaction effects—and in practical applications with human respondents may actually result in better main effect estimates as well. If respondents tend to be applying “must-have” or “must-avoid” rules to answer CBC questionnaires, designs with level overlap have an advantage over minimal overlap designs. We are pleased to see the use of Balanced Overlap increasing, and will probably make this the default design method in the next version of CBC.

ss33e

We are also pleased to see the prevalence of application of alternative-specific designs. Such designs are an excellent way to deal with studies that seem to be requiring many prohibitions. But, rather than including many prohibitions, what is often needed is to specify that certain attributes altogether only apply to certain brands or product alternatives.

Some of you who took the survey may have noticed that in addition to the options shown in Figure 3 we asked you if you had used a technique in your CBC studies called “Carroll's Mome Raths (MR) estimation method.” This was a fictional (decoy) item we added, to adjust for possible overstatement of usage by our respondents. The percent allocation for the decoy item (which was very small) was subtracted from the other items before reporting them in Figure 3. The decoy item makes reference to Lewis Carroll’s poem Jabberwocky as published in the book popularly referred to as Alice in Wonderland. The first stanza of that poem is as follows (underline added):

'Twas brillig, and the slithy toves
Did gyre and gimble in the wabe;
All mimsy were the borogoves,
And the mome raths outgrabe.