Sawtooth Software: The Survey Software of Choice

Optional 2-Day Workshops

Monday 8AM-5PM and Tuesday 8AM-5PM

CBC Consulting Challenge Workshop
Aaron Hill, Sawtooth Software and Chris Chapman, Google, sponsored by Survey Sampling International

This rapid-pace workshop is intended for those already experienced with conducting and programming CBC studies within the SSI Web platform, including analysis and market simulations. Participants will break into small consulting teams to carry out a conjoint analysis study that provides insight and guidance for a real business problem. These studies will be fielded (using SSI online panel sample) the evening after the first day, with data analyzed the second day. Each team’s presentation will be critiqued by the instructors who will select a winning team to present their findings/recommendations on Wednesday at an after-hours clinic (5:15-6:15). This is an exhilarating workshop, as problem definition, study design, survey programming, fieldwork, analysis, and client presentation are condensed into an interval of just 34 hours. Previous participants raved about the experience! (Laptop computer required.)

CBC Software Workshop
Keith Chrzan, Brian McEwan, and Gary Baker, Sawtooth Software

If you are relatively new to choice-based conjoint (CBC) or just getting started, join us for two days of hands-on practice with the CBC software and market simulator. We'll cover the main aspects of designing, programming, and analyzing CBC studies. You will have an opportunity to program CBC questionnaires individually as well as analyze data from a real CBC study in a team-oriented case study session. We'll provide coverage of counting analysis, logit, latent class, and HB. The instructors will share best practices, pitfalls to avoid, and experiences based on many years of technical support and consulting.

Attendees receive an evaluation copy of the software that they may use for 90 days (for non-commercial studies and evaluation purposes only).

MBC Software Workshop
Bryan Orme and Walter Williams, Sawtooth Software

Menu-Based Choice (MBC) is a relatively new and flexible choice modeling approach for solving a variety of multi-check (combinatorial) menu-selection problems. Examples include: choosing options to put on an automobile, selections from a restaurant menu, banking options, configuring an insurance policy, purchasing bundled vs. a la carte services including mobile phones, internet, and cable. Using the MBC software requires expertise in terms of experimental design for conjoint exercises, developing MBC surveys using advanced HTML/Javascripting, and some data processing to prepare the files in .csv-format for analysis within MBC. MBC has a built-in simulator and can also automatically export Excel simulators.

This course is intended for those with a strong background in discrete choice and econometric modeling. It is not necessary to own any software to participate: a demo license will be given. The learning is enhanced by working with real practice datasets, including a modeling challenge where attendees compete to fit actual holdout data.

Optional Half-Day Tutorials

Tutorial workshops provide opportunities for a more in-depth learning experience. Each tutorial will be led by an outstanding professional with pertinent research and teaching experience. Tutorials are optional and are an additional cost ($250). Please note that you must register separately for the tutorials.

Monday 8AM-Noon

SSI Web Snorkel
Gary Baker and Justin Luster, Sawtooth Software

Are you a new SSI Web user? Have you only used SSI Web for conjoint analysis or MaxDiff? We'll teach you to snorkel in the deep blue waters of SSI Web instead of wade along the shore!

You'll learn how SSI Web works below-the-surface and see what makes it such a flexible interviewing platform. We'll introduce you to the CiW question types (Select, Numeric, Open-End, Grid, Free-Format, Constant-Sum, Ranking, and Semantic Differential). Many additional SSI Web features will be covered, including Quotas, Skips, Constructed Lists, Randomized Blocks, Looping, the Data Generator, and Data Management. We'll share example files with attendees.

Monday 1PM-5PM

SSI Web Deep Dive
Justin Luster and Gary Baker, Sawtooth Software

Even though SSI Web is easy to begin using, there is an amazing degree of power awaiting the adventurous and advanced user. The course will demonstrate a number of power tricks that will open your eyes to new possibilities to accomplish challenging tasks and impress your clients.

You'll see how you can take on new work and problems you previously thought could not very easily be done. Many of these tricks involve Perl, JavaScript, and CSS. We will teach you the basics of these technologies and how you can apply them to create powerful SSI Web surveys. Some of the topics will include: Free Format questions, custom JavaScript validation, advanced constructed lists, advanced formatting options with CSS, how to search and replace text across your whole study, how to build drag and drop questions, etc. Bring your laptop to work through class exercises (we'll provide temporary licenses if needed).

Tuesday 8AM-Noon

New Frontiers in Conjoint Analysis
Keith Chrzan, Sawtooth Software

The variety of methods available to conjoint modelers has exploded in the 40+ years since the invention of conjoint analysis. Today this growth continues at a hectic pace as academics and practitioners continue to develop new ways of creating experimental designs, collecting respondent evaluations and running statistical models, and of combining these pieces into new conjoint technologies. This tutorial covers some promising new directions in this continuing evolution, introducing attendees to methods and issues like Random Regret Minimization, Conjoint Poker, Attribute and Scope Non-Attendance, new ways of blending stated preferences into conjoint models, and of handling rank order data. While Keith presents some of these topics as curiosities to spur our thinking, others come from theories that challenge the economic assumptions of conjoint analysis tradition, and others still pose difficulties that practitioners need to consider in designing their studies. For those looking to present at the 2015 Sawtooth Software Conference, you may want to view this session as a "one stop shop" for topics that need further research.

Introduction to R for Choice Modelers
Chris Chapman and Steven Ellis, Google

R is the statistics package of choice for many quantitative researchers and statisticians, with immense flexibility but a steep learning curve. This hands-on tutorial introduces R with a focus on data manipulation and the core language. The first half develops foundational skill with the R command line and data structures. The second half applies that skill to use R in choice model experiments. Using open source code, this section introduces working with SSI Web CBC data such as HB utilities, creating synthetic CBC data in R, estimating utilities, and doing basic CBC market simulations. We present R as a complement to SSI Web that adds tools for analysts to conduct additional analyses, test models, and develop their own extensions. A laptop with WiFi is required for this live code tutorial. [Note: Portions of this tutorial were offered at recent ART Forum conferences; this offering has new content on R for CBC, while excluding content on general regression models and plotting.]

Problems and Solutions in Conjoint Applications in Health and Healthcare
John F P Bridges, Department of Health Policy & Management, Johns Hopkins Bloomberg School of Public Health, and F. Reed Johnson, Health Preference Assessment Group, RTI Health Solutions

Stated-preference methods are now widely used to evaluate the value and relative importance of health outcomes and healthcare services. While borrowing extensively from stated-preference research methods in marketing, transportation, environment, and other disciplines, the research questions, attribute characteristics, and type of ultimate users for health and healthcare applications differ substantially from applications in other research areas. Health researchers have adapted conjoint methods to the special challenges of quantifying patients’, physicians’, policy makers’, and other stakeholders’ preferences for health and healthcare. Two recent task forces sponsored by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) have helped to standardize conjoint analysis and discrete-choice experiment research practices in health. This workshop will draw on a broad range of published and unpublished studies to illustrate solutions to common research challenges in developing effective survey instruments, constructing efficient experimental designs, analyzing data, and presenting results. Standards developed by the ISPOR taskforces for conjoint analysis and experimental design for discrete-choice experiments also will be reviewed for guidance on good research practices. At the conclusion of the workshop, participants will be able to identify common problems that researchers encounter in applying stated-preference methods in health and healthcare and will be equipped to apply pragmatic solutions that respect the basic principles necessary for obtaining valid measures of health and healthcare preferences.

John Bridges led the ISPOR task force on good research practices in conjoint analysis and Reed Johnson led the ISPOR task force on good research practices in experimental design.

Tuesday 1PM-5PM

Advanced MaxDiff
Keith Chrzan, Sawtooth Software

If you already have experience fielding and analyzing MaxDiff studies, extend your knowledge by attending this advanced session taught by Keith Chrzan. He'll cover experimental design principles, design matrix coding, score estimation, rescaling approaches, comparisons to other methods (ratings, Q-Sort, constant sum, magnitude estimation), market segmentation via MaxDiff results, and anchored scaling techniques (both Lattery and Louviere approaches).

Chrzan will also present some new research into fusing MaxDiff with CBC questionnaires to obtain the benefits of conjoint modeling together with the convenience of placing all the attribute levels on a common scale.

Segmentation Modeling Projects Using Latent Gold® Choice and Excel-Based StatWizards from Beginning to End
Jay Magidson, Statistical Innovations Inc. and George Boomer, StatWizards LLC

The key to getting clear, meaningful segments is to use appropriate latent class models. In this tutorial Jay and George will take you through each phase of a discrete choice and MaxDiff project:

  • experimental design
  • data setup
  • model development
  • characterization of resulting segments, and
  • simulator development

We show the power and flexibility of StatWizards and Latent GOLD Choice and how they work together seamlessly. Jay and George also describe features in the new releases of Latent GOLD Choice and StatWizards including:

  • Version 5.0 of LG Choice
    • syntax to apply adjustments for scale factors
    • major speed improvement/support for multi-processors
    • Profiling the segments using Step3 estimation
    • Markov choice models to investigate how preferences change over time
  • Design Wizard:
    • Real-time calculation of D- and A-efficiencies
    • Use of Solver to minimize overlap
    • Use of Solver to optimize designs having a limited number of runs
Conjoint Data as Evidence: The Role of Patient Benefit-Risk Tradeoff Preferences in Regulatory Decision Making
F. Reed Johnson, Health Preference Assessment Group, RTI Health Solutions, and John F P Bridges, Department of Health Policy & Management, Johns Hopkins Bloomberg School of Public Health

Agencies that regulate medical technologies, such as the European Medicines Agency (EMA) and the Food and Drug Administration (FDA), are required to improve the transparency of decisions involving benefit-risk assessments and to facilitate greater patient involvement in such decisions. While regulatory decision makers are trained to evaluate clinical evidence on the outcomes of medical technologies, many decisions require a societal judgment about whether therapeutic benefits justify the associated risks. Both EMA and FDA recently have undertaken initiatives and issued guidance on greater patient involvement in such benefit-risk evaluations. Conjoint researchers have begun to respond to the need for quantitative information on patients’ willingness to accept risks in return for specified benefits. The objectives of this workshop are to introduce participants to recent regulatory guidance relevant to stated-preference research, assess the capabilities of existing methods to satisfy regulatory standards for validity and reliability, and critically evaluate existing conjoint-analysis and choice-experiment applications to estimating patients’ risk tolerance. At the conclusion of the workshop, participants will be able to assess the current state of the practice in benefit-risk applications of stated-preference research and to understand both the opportunities for and barriers to greater acceptance of preference data in regulatory decision making.

Reed Johnson and John Bridges have led numerous risk-preference studies and consulted with regulatory authorities on the role of patient preferences in licensing and reimbursing new medical products.

Main Conference Sessions

Wednesday, October 16, 2013

Session 1

9 Things Clients Get Wrong about Conjoint Analysis
Christopher Chapman, Google

Conjoint analysis (CA) is widely known among product managers, thanks to its inclusion in business school curricula and the efforts of research firms and platform providers. CA is approachable to statisticians, econometricians, and survey scientists. Yet underneath this surface simplicity lies danger! I describe several problems drawn from experiences consulting on over 100 CA projects.

Quantitative Marketing Research Solutions in a Traditional Manufacturing Firm: Update and Case Study
Robert J. Goodwin, Lifetime Products, Inc.

In this paper, Lifetime Products provides a progress report on its quest for more effective analytic methods and offers an insightful new ACBC case study. This demonstration of a typical adaptive choice study, enhanced by experiments with research design parameters, will be of interest to new practitioners and experienced users alike.


Session 2

Can Conjoint Be Fun?: Improving Respondent Engagement in CBC Experiments
Jane Tang and Andrew Grenville, Vision Critical

Many respondents find CBC experiments boring. Reducing the number of tasks can shorten the time required and retain engagement, but it is still tedious. We test two ideas: an adaptive tournament based approach and an instant feedback mechanism, to see if they can bring some "fun" into conjoint experiments.

Making Conjoint Mobile: Adapting Conjoint to the Mobile Phenomenon
Chris Diener, Rajat Narang, Mohit Shant, Hem Chander, and Mukul Goyal, AbsolutData

The authors test and compare multiple ways of conducting choice based conjoint analysis on the mobile platform. The results obtained have been evaluated for data quality and respondent experience by comparing them to results of a CBC conducted on a PC, supporting the use of the mobile platform.

Choice Experiments in Mobile Web Environments
Joseph White, Maritz Research

Respondents are increasingly completing surveys in a mobile web environment, raising a potential problem for discrete choice experiments due to the added complexity and visual limitations. This paper looks to understand this potential impact by investigating differences in parameters, respondent error, and predictive validity by survey completion form factor for a large commercial study.


Session 3

Using Complex Choice Models to Drive Business Results
Karen Fuller, HomeAway, Inc. and Karen Buros, Radius Global Market Research

In 2011 and 2013, HomeAway, Inc. completed menu-based research programs to test alternative pricing strategies for homeowner listings on its sites. Based on the results of the studies HomeAway successfully adopted tiered pricing strategies worldwide. This paper will detail how the studies were designed and analyzed to illustrate the success of this important new tool.

Augmenting Discrete Choice Data – A Q-sort Case Study
Brent Fuller, Mike Smith, and Matt Madden, The Modellers

One of the shortcomings of discrete choice models is difficulty in handling attributes with many levels. One option to solve this is to include information from other parts of the survey. We show how a Q-sort survey exercise was used to augment discrete choice data and produce better estimates.

MaxDiff Augmentation: Effort vs. Impact
Urszula Jones, TNS and Jing Yeh, Millward Brown

Augmented MaxDiff is an option for testing large sets of attributes, but is the effort worth it? We compare Augmented MaxDiff in four situations (heavy/light and top/bottom augmentation) to Sparse MaxDiff to evaluate its performance. We will also explain the augmentation process – including Q-Sort question format, design, and estimation.


Session 4

When U = βx Is Not Enough: Modeling Diminishing Returns among Correlated Conjoint Attributes
Kevin Lattery, Maritz Research

Correlated alternatives are a well known problem in conjoint. Less documented are problems with correlated attributes, especially when the number of these attributes varies. We borrow the solution from nested logit, treating the correlated subset of attributes as a nest whose utility function is defined with an additional λ parameter.

Respondent Heterogeneity, Version Effects or Scale?
Keith Chrzan and Aaron Hill Sawtooth Software

HB utilities from discrete choice experiments differ across respondents. Preference heterogeneity is that portion of the heterogeneity not attributable to version effects or differences in respondent reliability. This presentation aims to separate the different sources of respondent heterogeneity to identify how much owes to preference-irrelevant factors like reliability and version effects.

Thursday October 17, 2013

Session 5

Bridging Survey Research with Social Media Monitoring Services
Karlan Witt and Deb Ploskonka, Cambia Information Group

Organizations are drowning in data. And the volume increases each year. This paper describes an approach to arm firms with analytics to digest the river of social media and identify when the firm needs to take action on trends that arise, intelligently deploying resources precisely when and where they are needed.

Brand Imagery Measurement: Assessment of Current Practice and a New Approach
Paul Richard McCullough, MACRO Consulting, Inc.

This paper reviews the practice and limitations of traditional brand measurement techniques and suggests a novel application of Dual Response MaxDiff to provide a superior brand imagery measurement methodology.

ACBC Revisited
Christopher Fotenos, Jeroen Hardon, and Marco Hoogerbrugge, SKIM Group

ACBC, released in 2009, has already received a lot of attention, though CBC is still used most often. We will compare ACBC and CBC, mix and match the two methodologies, in order to see whether improvements can be made in either method.


Session 6

Research Space and Realistic Pricing in Shelf Conjoint
Peter Kurz, TNS Infratest, Stefan Binner, bms marketing research + strategy, and Leonhard Kehl, Premium Choice Research & Consulting

Conjoint Analysis using some type of shelf display is frequently applied around the globe. The authors will give an overview of the areas in which Shelf Conjoint requires specific consideration and designs and will provide suggestions for best practice in regard to some critical aspects: objectives, research space and pricing.

Attribute Non-Attendance in Discrete Choice Experiments
Dan Yardley, Maritz Research

Some respondents ignore certain attributes in choice experiments to help them choose between competing alternatives. By asking respondents which attributes they ignored and accounting for this attribute non-attendance we hope to improve preference models. We also test ways of asking stated non-attendance and the impact of non-attendance on partial profile and different sized designs.

Anchored Adaptive MaxDiff - Application in Continuous Concept Test
Rosanna Mau, Jane Tang, LeAnn Helmrich, and Maggie Cournoyer, Vision Critical

Many firms have a continuous concept test program based on monadic or sequential monadic ratings. MaxDiff is superior to ratings, but does not lend itself easily to tracking across the many waves of a continuous program. We look into how an anchored adaptive MaxDiff can be set up in this environment so that all the concepts tested are comparable across the different testing periods.

How Important Are the Obvious Comparisons in CBC? The Impact of Removing Easy Conjoint Tasks
Paul Johnson and Weston Hadlock, SSI

Removing obvious comparisons from CBC exercises has generated theoretical efficiency gains in simulated experiments, but does this ‘easy task’ elimination actually improve the hit-rates? We compare hit-rates on hold-out tasks for standard CBC groups vs. difficult CBC groups by number of tasks to measure efficiency gains with real respondents.


Session 7

Segmenting Choice and Non-Choice Data Simultaneously: A How to...
Thomas C. Eagle, Eagle Analytics of California, Inc.

We demonstrate how to simultaneously segment both choice and non-choice data from a survey. We extend this to a true multi-dimensional multi-objective segmentation where there are multiple correlated, or non-correlated, nominal latent class variables used to segment the data. All examples are fit using LatentGold’s Syntax Module and the code will be shared.

Extending Cluster Ensemble Analysis via Semi-Supervised Learning
Ewa Nowakowska, GfK Polonia and Joseph Retzer, JRA, LLC

We extend Cluster Ensemble methodology to improve the consensus solution by augmenting ensemble partitions with partitions from Random Forest (RF) Analysis. Consensus is achieved using Sawtooth Software’s CCEA. RF partitions incorporate profiling information indicative of target measures. The consensus is high quality, easier to predict, and useful for marketing strategy.


Session 8

The Shapley Value in Marketing Research: 15 Years and Counting
W. Michael Conklin and Stan Lipovetsky, GfK

We review the application of the Shapley Value to marketing research over the past 15 years. We attempt to provide a comprehensive understanding of how it can give insight to customers. We outline assumptions underlying the interpretations so that attendees will be better equipped to answer objections to the application of the Shapley Value as an insight tool.

Demonstrating the Need and Value for a Multiobjective Product Search
Scott Ferguson and Garrett Foster, North Carolina State University, and Joseph Donndelinger, General Motors Research and Development

The product search algorithms currently available in Sawtooth Software’s ASM focus on optimizing product configurations for a single objective. We demonstrate how multiobjective product search formulations can significantly influence and form your product strategy. Advantages include richer solution sets and the ability to explore tradeoffs between competing objectives.

A Simulation Based Evaluation of the Properties of Anchored Max/Diff: Strengths, Limitations, and Recommendations for Practice
Jake Lee, Maritz Research and Jeff Dotson, Brigham Young University

Several approaches have been proposed to overcome some of the limitations of Max/Diff including dual response, direct anchoring, and a status quo alternative. In this paper we use a series of simulation studies to better understand the properties of each approach, with an eye toward setting a standard for best practices.

Friday October 18, 2013

Session 9

Contexts in Which Best-Worst CBC Are Most Valuable: Application to School Choice
Namika Sagara and Joel Huber, Duke University, and Angelyn Fairchild, Research Triangle Institute

Best-Worst CBC can generate efficient individual valuation when some features are strongly desirable and others are strongly undesirable. For school choice, the ‘worst’ judgments expose features that respondents actively avoid while ‘best’ judgments reflect features that are sought after. Additionally, a linear probability model that combines both judgments discriminates between respondents almost as well as the appropriate HB model.

Does the Analysis of MaxDiff Data Require Separate Scaling Factors?
Jack Horne and Bob Rayner, Market Strategies International

Scale of the error terms around MaxDiff utilities sometimes varies between "best" and "worst" responses. Most estimation procedures however assume that scale is fixed, leading to potential bias in the estimated utilities. We investigate to what degree scale actually does vary between response categories, and, whether true utilities may be better recovered by properly specifying scale when estimating utilities.


Session 10

How to Use Conjoint to Determine the Market Value of Product Features
Greg Allenby, The Ohio State University, Jeff Brazell, The Modellers, John Howell, The Ohio State University, and Peter Rossi, UCLA

Carefully designed conjoint studies can be used to estimate the system of demand for the product in question and competing products. However, equilibrium market prices must involve supply information and competitive sets and do not simply reduce to the computation of some sort of aggregate WTP measure.

The Ballad of Best and Worst
Tatiana L. Dyachenko, Rebecca Walker Naylor, and Greg M. Allenby, The Ohio State University

We investigate psychological processes underlying Best-Worst procedure. We find evidence for sequential evaluation in Best-Worst tasks that is accompanied by elicitation and sequence scaling effects. We propose a model that accounts for these effects, and advise against thinking of Best-Worst data as arising from a simple model.

Conference Adjourned