Sawtooth Software: The Survey Software of Choice

Advances and Trends in Marketing Science from the Sawtooth Software Perspective

Dr. John Roberts (Joint Professor at Australian National University and London Business School) recently asked us to pen our perspective on developments in marketing science and trends over the last 10 years. Based on his interest to share that response more broadly, we’ve polished our thoughts a bit more to create this short article.

Dear Dr. Roberts,

Thank you for the opportunity to respond regarding trends in the tools used in marketing science over the last 10 years. It was interesting for us to ponder and record a few thoughts. Due to our company’s focus, it won’t surprise you that our comments focus on conjoint/choice analysis, although this certainly is only one slice (though an important one!) of the marketing science practice.

Most Important Developments directly related to Sawtooth Software tools over the last 10 years:

  1. Mainstreaming of hierarchical Bayes (HB) for estimation of utility scores. Although Sawtooth Software released software for HB analysis of CBC software in 1999 (called CBC/HB), the bulk of the impact has been experienced over the last 10 years. During the last decade HB has become the gold standard estimation approach for analyzing discrete choice data. HB isn’t the only estimation technique for modeling continuous heterogeneity for discrete choice data. Though not an exhaustive list, Mixed Logit, EM algorithms, bottom-up (Louviere), ICE (Rich Johnson), and Latent Class with C Factors (Latent Gold) are other viable approaches, though they are not used with nearly the frequency of HB.
  2. Decline of ratings-based conjoint popularity, including Paul Green’s full-profile conjoint (as represented by Sawtooth Software’s CVA, SPSS, and Bretton-Clarke software) and Sawtooth Software’s ACA (Adaptive Conjoint Analysis). In 2012, just 10% of all conjoint studies conducted by Sawtooth Software users employed ratings-based conjoint.
  3. Emergence of MaxDiff (Best-Worst) scaling as a commonly-used general scaling technique (e.g. for measuring the importance or preference of attributes). Louviere invented the technique in the late 1980s. Sawtooth Software introduced software for MaxDiff in 2004. As of 2012, over 50% of Sawtooth Software users reported that their firms used the technique during the previous year. In the last couple of years, researchers have tried different methods for anchoring the scaling of items in MaxDiff relative to an absolute threshold of importance (or a buy/no buy boundary). While this appears to remove the ipsative nature of MaxDiff scaling, it also lets anchor-interpretation bias (akin to scale-use bias) back into the data—something that researchers were trying to eliminate in the first place by using MaxDiff.
  4. Menu-Based Choice (MBC). Multi-check and mixed bundling applications for choice menus were mentioned with greater frequency in the early 2000s. Sawtooth Software released software for analyzing MBC problems in 2012. About 3% of conjoint studies conducted between 2012 to 2013by Sawtooth Software users employed the MBC methodology (often using our software). Although multivariate probit seems the more academically sound approach to analyzing MBC data, MNL is more practical for practicing researchers and scales up to much more complex MBC problems.
  5. Proliferation of the use of optimization algorithms in market simulations (what-if simulators), including hill-climbing and Genetic Algorithms. Sawtooth Software released its optimization search for market simulators in 2003 (ASM-Advanced Simulation Module). Other optimization plug-ins are available for Excel, leading to the ability to develop quite sophisticated Excel-based optimization/simulation programs.
  6. Release of Adaptive CBC (ACBC) by Sawtooth Software in 2008. ACBC leverages the best ideas espoused with ACA and CBC. It uses a 3-stage interviewing process:
    1. Respondents build their ideal product using a BYO (Build-Your Own) format
    2. Respondents build a consideration set of products by evaluating an array of near-neighbor concepts to the BYO-specified ideal. During this phase, respondents can express certain non-compensatory screening rules.
    3. Respondents use a CBC-looking choice tournament to narrow down the products in their consideration set to a final choice.
    Along with this development came an enhanced awareness of how many respondents employ non-compensatory heuristics for decision-making. Choice researchers now accept that probably 50% or more of respondents use non-compensatory heuristics rather than following an additive, compensatory decision rule consistent with the logit rule (RUM). (This train of thought is all quite related to key findings in the behavioral economics field. Over the last ten years, interest in behavioral economics has grown as evidenced by the popular book by Dan Ariely, “Predictably Irrational.”) About 13% of conjoint studies conducted in 2012 by Sawtooth Software users employed ACBC software.
  7. Release of Cluster Ensemble Analysis (CCEA software) by Sawtooth Software in 2008. Cluster ensembles generally improve recovery of market structure compared to K-means or hierarchical clustering routines. The notion of cluster ensembles came out of the machine learning literature. A single consensus solution is developed from a wide variety of segmentation schemes that are provided in the ensemble. The consensus solution typically demonstrates stronger qualities than any of individual segmentation solutions supplied within the ensemble. A related and valuable development for classification trees is Random Forests.
  8. Across the industry (whether using Sawtooth Software designs or not), an increase in the use of large experimental designs for CBC-type problems (large numbers of design blocks), especially those that incorporate significant amounts of Level Overlap (level repeats) within choice sets. Researchers have become increasingly aware that maximizing D-efficiency doesn’t necessarily improve precision of the parameters when real humans are interviewed, because humans often (usually?) don’t generate data according to MNL assumptions. Non-compensatory respondent behavior is very common (if not the norm) and designs featuring level overlap are superior for avoiding superficial responses and for the more complete and precise recovery of true utility parameters for non-compensatory behaving respondents.

Other meaningful developments and trends not directly tied to Sawtooth Software:

  1. Use of open-source statistical tools such as R by which academics and practitioners develop and share code for a variety of multivariate statistical applications for the marketing sciences.
  2. Quantitative analysis of open-ended social media data, including scraping the web for user-generated content and submitting the open-end comments to automated content and sentiment analysis. Social media research has captured the imagination of researchers and investors alike, with millions of dollars of funding being lavished on upcoming firms possessing the technical and computing science capabilities (along with salesmanship) in this field. Whether the benefits of all this excitement are worth the interest and investment remains to be seen.
  3. Development of hardware and software to manage Big Data, including user-generated content, shopping cart data, click-stream (within-website), and across website browsing behavior.
  4. Neuroscience. Brain scans are drawing a lot of interest (seems so futuristic and almost big-brotherly), though a healthy amount of scepticism abounds regarding the proper interpretation and usefulness of brain scan data. Other kinds of physio-measurements such as eye-tracking and sweat response appear more robust and are providing important insights. Eye-tracking is becoming particularly useful in the retail environment in terms of shelf optimization, advertising, and messaging.
  5. Death of the pretence of representative samples. Increased reluctance of people to participate in research surveys, do-not-call lists, and cell phones, mean that researchers give up any pretence that they are obtaining representative samples. The online panel is widely accepted as a legitimate alternative (at least in countries with high internet penetration), although researchers realize that online panels have biases and weaknesses.
  6. DIY (Do-It-Yourself) research due to economic realities imposed by the recent global recession and encouraged by firms such as SurveyMonkey and now Google with Google Consumer Surveys. While these developments seem to offer cost savings and greater opportunities for hands-on work by internal market research departments at end-user firms, the question is whether this is a good thing or not for the practice of marketing science and especially the health of market research and strategy consulting firms. Time will tell and perhaps the pendulum will swing back more in favor of the consultants as end-users realize the value they were missing and the global economy (hopefully) strengthens.