Sawtooth Software: The Survey Software of Choice

Pre-Conference Workshops (Monday, Tuesday)

Monday

(8:00 AM – 12:00 PM) CBC Software Workshop (Part I)

Brian McEwan, Sawtooth Software

Aaron Hill, Sawtooth Software

Gary Baker, Sawtooth Software

If you are relatively new to choice-based conjoint (CBC) or just getting started, join us for two days of hands-on practice with the CBC software and market simulators (desktop simulator, the online simulator, and an Excel-based simulator). We’ll cover the main aspects of designing, programming, and analyzing CBC studies. You will have an opportunity to program CBC questionnaires individually as well as analyze data from a real CBC study in a team-oriented case study session. We’ll provide coverage of counting analysis, logit, latent class, and HB. The instructors will share best practices, pitfalls to avoid, and experiences based on many years of technical support and consulting.

Attendees receive an evaluation copy of the software that they may use for 90 days (for non-commercial studies and evaluation purposes only). Limited to 25 participants.

(8:00 AM – 12:00 PM) ACBC Survey Programming Workshop (Part I)

Bryan Orme, Sawtooth Software

With the most recent release of ACBC, there are even more options and opportunities than before for designing adaptive CBC surveys. While we will spend some time addressing conceptual and statistical issues, the clear emphasis of this workshop will be on practical examples and survey programming. We’ll be exploring ACBC material that we cannot cover in our standard choice workshops. Topics and examples will include: the Test Design functionality, constructed lists (dropping irrelevant attributes and levels), additional price adjustments within Summed Pricing, conditional display, skipping BYO, Screener, and/or Choice Tournament sections, and abbreviating ACBC surveys for Mobile. (Attendees must bring a laptop with the most recent version of ACBC installed).

(8:00 AM – 12:00 PM) Introduction to MaxDiff Workshop

Keith Chrzan, Sawtooth Software

Since its introduction in the early 1990s, Maximum Difference (MaxDiff) scaling has become an extremely popular method for measuring the value of each member of a set of items. These values can be importances, product ideas, brand images, levels of agreement for psychographic scaling, among other things.

This hands-on workshop introduces the rationale behind MaxDiff to explain why it works so well. The design, administration, analysis, interpretation and reporting of MaxDiff experiments all receive in-depth treatment. Case studies will illustrate the most common uses of MaxDiff and its use in a wide variety of other applications. We will briefly discuss the other methods that comprise the MaxDiff measurement family, MaxDiff Scaling cases 2 and 3, for conjoint and discrete choice modeling, respectively.

For those new to MaxDiff, this session provides suitable background for the papers to be discussed at the main conference. In addition to the conceptual background and practical grasp of MaxDiff described above, attendees will get hands-on experience using the MaxDiff software to program MaxDiff surveys. You will leave the tutorial able to add this powerful analysis to your methodological toolbox.

(8:00 AM – 12:00 PM) Advanced MaxDiff Survey Programming Workshop

Aaron Hill, Sawtooth Software

In this hands-on workshop, you’ll learn how to move beyond the basics of standard MaxDiff. Using programming exercises and examples in SSI Web, we’ll:

  • Demonstrate new software features such as Anchored MaxDiff
  • Show how you can reduce the number of items evaluated by a respondent via Express MaxDiff
  • Use SSI Web’s powerful survey features to build an Adaptive MaxDiff exercise
  • Take advantage of Free Format capabilities to customize the display of your MaxDiff exercise
  • Reveal unconventional ways you can employ MaxDiff’s experimental designer for non-MaxDiff purposes

Participants in this class should bring a PC-compatible laptop on which to program exercises. Download instructions and software licenses for classroom use will be emailed to attendees prior to the workshop. While some exposure to MaxDiff and SSI Web is helpful, it is not necessary.

(8:00 AM – 12:00 PM) Advances in Market Research Segmentation Analysis (Part I)

Joseph Retzer, Radius Global Market Research

The focus of this tutorial will be on conceptual underpinnings of unsupervised learning (Cluster Analysis) algorithms and models. It will begin with a short review of some well-known methods while paying particular attention to the Cluster Ensemble approach. Unsupervised methods covered will include:

  • Hierarchical methods (agglomerative and divisive with various cluster similarity measures e.g. Wards distance, complete, single & average linkage)
  • K-means
  • PAM (Partitioning Around Medoids)
  • Finite Mixture Modeling
  • Clustering Objects on Subsets of Attributes (COSA)
  • Cluster Ensembles
  • Aceclus & SPSS Two-Step Clustering (time permitting)

Cluster quality diagnostics (e.g. silhouette plots, quality indices, etc.) will also be reviewed. Comparison of cluster partitions using the Adjusted Rand Index will be covered along with this discussion. Finally, an illustration of effective post hoc cluster profiling, and presentation of cluster analysis results in general, will be demonstrated. Example code will be provided, typically in R, but also in Sawtooth Software, and Latent Gold when appropriate.

(1:00 PM – 5:00 PM) Introduction to R for Market Researchers

Christopher Chapman, Google

Kenneth Fairchild, Sawtooth Software

Introduction to R for Marketing Researchers is a 4-hour, hands-on introduction to the R statistics platform focused on marketing applications. The course presents the basics of the R language, statistical procedures, and graphics. The tutorial begins with the basics of the command-line language and data processing in R. We then examine common statistical analyses such as regression models. Finally we discuss applied topics for marketing research, including a brief overview of R options for clustering and segmentation, choice models, and automated reporting.

(1:00 PM – 5:00 PM) Advanced MaxDiff Concepts

Keith Chrzan, Sawtooth Software

Whether you are relatively new to the use of MaxDiff or already quite experienced, this class provides material that will stimulate your thinking and improve your practice.

This tutorial will take a deep dive into different ways of generating, rescaling, reporting and anchoring utilities. Because there are often multiple ways these can be done, we will compare and contrast options to identify their relative strengths and weaknesses, so that you walk away with a good feel for how and why to make decisions in the design and reporting of your MaxDiff study.

An important practical topic we will cover is how to address the all-too-common situation wherein we need to handle an especially large number of items without fatiguing respondents. Several variations of MaxDiff have been advanced to do this and we will compare them.

Finally, we’ll wrap up with some newer MaxDiff topics and intriguing research from the recent Sawtooth Software conferences as well as the broader literature.

(1:00 PM – 5:00 PM) ACBC Survey Programming Workshop (Part II)

Bryan Orme, Sawtooth Software

In this advanced workshop, we delve into ACBC programming examples and topics that cannot be covered in our standard choice modeling workshops. This section builds upon material from Part I taught during the morning session. Topics and exercises include: asking follow-up questions about Unacceptable levels, alternative-specific designs, investigating significant interaction terms, summed pricing functions (linear, log-linear, piecewise), unverified Perl to customize question wording, customized utility constraints, latent class segmentation vs. HB+CCEA, and dealing with different country currencies in a unified summed pricing study. (Attendees must bring a laptop with the most recent version of ACBC installed).

(1:00 PM – 5:00 PM) SSI Web Swim

Gary Baker, Sawtooth Software

Are you a new SSI Web user? Have you only used SSI Web for conjoint analysis or MaxDiff? We invite you to come explore the general surveying capabilities of the platform! Topics in this introductory class include:

  • Project management
  • Styles, headers & footers, progress bar
  • Standard question types (select, numeric, open-end, constant sum, grid, ranking, semantic differential, terminate/link)
  • Questionnaire flow: skip patterns and randomization
  • Introduction to constructed lists and piping
  • Introduction to fielding the survey on a server or CAPI

Attendees must bring a laptop PC with SSI Web installed (a demonstration version will be given to you in advance for the purposes of classroom instruction). You will follow along within the software and do programming exercises to solidify the concepts. This class is part of the SSI Web Swim, Snorkel, and Deep Dive series.

Tuesday

(8:00 AM – 12:00 PM) CBC Software Workshop (Part II)

Brian McEwan, Sawtooth Software

Aaron Hill, Sawtooth Software

Gary Baker, Sawtooth Software

If you are relatively new to choice-based conjoint (CBC) or just getting started, join us for two days of hands-on practice with the CBC software and market simulators. We’ll cover the main aspects of designing, programming, and analyzing CBC studies. You will have an opportunity to program CBC questionnaires individually as well as analyze data from a real CBC study in a team-oriented case study session. We’ll provide coverage of counting analysis, logit, latent class, and HB. The instructors will share best practices, pitfalls to avoid, and experiences based on many years of technical support and consulting.

Attendees receive an evaluation copy of the software that they may use for 90 days (for non-commercial studies and evaluation purposes only). Limited to 25 participants.

(8:00 AM – 12:00 PM) How People Choose: Understanding Customer Decision Making

Keith Chrzan, Sawtooth Software

This tutorial focuses on the strategies customers use to make decisions and on the statistical models analysts use to understand and quantify those strategies.

The most widely used model in the field of marketing research is the Random Utility Model (RUM), brought to life in the multinomial logit (MNL) choice model. This model supports choice-based conjoint (CBC) analysis, MaxDiff scaling, ACBC and many other techniques marketing researchers use. As convenient and popular as it is, however, the MNL model makes sometimes unrealistic assumptions about human behavior (e.g. the property called Independence from Irrelevant Alternatives or IIA). These assumptions have led to interest in alternative choice models, for example, non-compensatory models like the lexicographic and elimination-by-aspects (EBA) choice models. In addition, augmented versions of the MNL model which enable two-stage models combining a non-compensatory first stage elimination with a compensatory RUM second stage have been developed and tested.

Recent attempts to incorporate additional behavioral effects like loss aversion and extremeness aversion into the MNL model have led to a class of semi-compensatory models that incorporate context effects, like Random Regret Minimization (RRM) and Relative Advantage Maximization (RAM).

The tutorial reviews, illustrates and compares the models listed above and several others. It reviews evidence that survey respondents may not always make choices according to RUM, that they can change their decision strategies from one situation to the next and that they can employ different strategies even within a single choice occasion.

(8:00 AM – 12:00 PM) Introduction to Market Simulations

Bryan Orme, Sawtooth Software

In this four-hour tutorial, we aim to convince you that market simulations are often the best way to use conjoint analysis data to answer strategic questions. We’ll examine more in-depth the virtues and drawbacks of different market simulation approaches: first choice, share of preference (logit rule), randomized first choice (RFC), and purchase likelihood simulations. The red-bus/blue-bus discussion factors prominently into the choice of market simulation method. We’ll illustrate the common applications for simulations, including demand curve generation, sensitivity analysis, and line extension/cannibalization. We’ll discuss the pros and cons of four software tools for simulations: Excel as well as Sawtooth Software’s SMRT, Online Simulator, and the upcoming SSI Web integrated simulator. Finally, we’ll briefly touch upon optimization, introducing different algorithms for optimizing share, revenue, or profit.

(8:00 AM – 12:00 PM) Research for Solid Pricing Decisions

David Lyon, Aurora Market Modeling

Pricing decisions go right to a firm’s bottom line, so pricing research is one of the most important areas of marketing research. A number of very different approaches are used to measure price sensitivity, and sorting out their competing claims of accuracy, realism and simplicity can be very confusing. While none are a panacea, all have their place, and this session will focus on understanding the basics, pros and cons of each to help establish what to use when. We will discuss a number of approaches to the pricing problem, all based on questionnaire/survey data, including simple willingness-topay questions, monadic designs, van Westendorp and its Newton-Miller-Smith variant, and various forms of conjoint, both ratings-based and choice-based (discrete choice).

(8:00 AM – 12:00 PM) SSI Web Snorkel

Gary Baker, Sawtooth Software

If you've used SSI Web but haven't taken time to understand the power that lies just under the surface, then the SSI Web Snorkel class is for you. We'll get below-the-surface and see what makes SSI Web such a powerful interviewing platform. In this intermediate class, we'll cover:

  • Understanding the browser, the server, and SSI Web
  • SSI Script functions
  • Advanced constructed lists
  • The free format question type
  • Question verification with custom JavaScript
  • The data generator
  • Export and analysis Filters

Attendees must bring a laptop PC with SSI Web installed (a demonstration version will be given to you in advance for the purposes of classroom instruction). You will follow along within the software and do programming exercises to solidify the concepts. This class is part of the SSI Web Swim, Snorkel, and Deep Dive series.

(1:00 PM – 5:00 PM) Introduction to R for Market Researchers (repeat session)

Christopher Chapman, Google

Kenneth Fairchild, Sawtooth Software

Introduction to R for Marketing Researchers is a 4-hour, hands-on introduction to the R statistics platform focused on marketing applications. The course presents the basics of the R language, statistical procedures, and graphics. The tutorial begins with the basics of the command-line language and data processing in R. We then examine common statistical analyses such as regression models. Finally we discuss applied topics for marketing research, including a brief overview of R options for clustering and segmentation, choice models, and automated reporting.

(1:00 PM – 5:00 PM) Advances in Market Segmentation Analysis Part II: A Hands-On Research Tutorial

Ewa Nowakowska, GfK

Joseph Retzer, Radius Global Market Research

The focus of this tutorial will be on combining theory with implementation via a “hands on” programming of cluster analyses and cluster partition evaluation.

The tutorial will begin with an in depth discussion of the Semi-Supervised Learning (SSL) algorithm. The implementation of SSL using Random Forest analysis similarity matrix results in conjunction with Sawtooth Software’s Convergent Cluster Ensemble Analysis package will be demonstrated. SSL can be useful in producing clusters of high quality that are also informative to pre-specified marketing strategy.

Cluster basis variable importance evaluation, as measured by relative variation, will also be performed. Importance measurement will be accomplished using Friedman’s R based cluster analysis evaluation tools. Various data sets will be used for illustrative purposes including one in which an interesting application of geographical data clustering is reviewed. Finally, a presentation of Bayesian Co-Clustering will presented with results produced via JAGS (Just Another Gibbs Sampler) and R, time permitting.

Attendees are encouraged to bring laptops with the R statistical programming language installed (http://www.rproject.org). Additional information pertaining to required R packages will be provided at a later date.

(1:00 PM – 5:00 PM) SSI Web Deep Dive

Justin Luster, Sawtooth Software

Gary Baker, Sawtooth Software

SSI Web has been designed to be very flexible. Custom code can be added to greatly enhance the survey. In this class we will learn the basics of: CSS, JavaScript, jQuery, and Perl.

A little bit of training in these areas can turn you into a whiz when it comes to creating powerful SSI Web surveys. Various examples of using these technologies together with SSI Web will be shown.

Attendees must bring a laptop PC with SSI Web installed (a demonstration version will be given to you in advance for the purposes of classroom instruction). You will follow along within the software and do programming exercises to solidify the concepts. This class is part of the SSI Web Swim, Snorkel, and Deep Dive series.

(1:00 PM – 5:00 PM) Adventures in Choice Modeling

David Lyon, Aurora Market Modeling

Aaron Hill, Sawtooth Software

Bryan Orme, Sawtooth Software

Jaroen Hardon, SKIM Group

Four experienced practitioners will present 8 to 10 examples of challenging and interesting extensions and applications of choice modeling based on their own practical experiences. As an advanced session for well-experienced practitioners of choice-based techniques with a strong analytic bent, this is meant to stretch your mind and open your eyes to new possibilities and approaches. Many of the topics will require custom work beyond the standard Sawtooth Software options, so a programming bent is also useful.

Specific topics are still being finalized, but will likely include:

  • analyzing the effect of prohibitions, and strategies minimizing those effects,
  • effects coding for main effects and interactions,
  • searching for significant interactions in choice models – mechanics and testing,
  • should dominated/dominating alternatives (ones better/worse than the others across all attributes) be avoided in choice designs?
  • extending the idea of conditional attributes to crossalternative contingencies that model pricing and features of lines of tiered products (e.g., Basic, Premium, Deluxe versions), and
  • options and considerations in choosing among CBC, ACBC and MBC – is no-intro ACBC a strong contender?

Other possible topics include:

  • soft constraints,
  • volumetric approaches to choice modeling,
  • evoked-set choice modeling,
  • and anything else new from the presenters’ recent experiences!

General Session Presentations (Wednesday—Friday)

Session 1

(7:00 AM - 5:00 PM) Conference Registration

(7:00 AM - 8:25 AM) Breakfast

(8:25 AM) Welcoming Remarks (Bryan Orme, Conference Moderator)

(8:30 AM) Mobile Choice Modeling - a Paradigm Shift

Jeroen Hardon, SKIM Group

Dirk Huisman, SKIM Group

Mobile platforms (smartphones and tablets) present a paradigm shift in consumer choice behavior which transcends to choice modeling research, by causing consumer shopping behavior to shift from offline to online. We show how choice modelers can respond to this development by introducing new and practical choice modeling formats to better replicate online consumer shopping behavior.

(9:00 AM) MaxDiff on Mobile

Jing Yeh, Millward Brown

Louise Hanlon, Millward Brown

As MaxDiff continues to be a household name in the market research industry, and as the world becomes more technologically mobile; it is important for us to understand the implications of device agnostic data collection for MaxDiff surveys. We will provide learnings and recommendations from two case studies that employed MaxDiff on mobile devices.

(9:30 AM) A Forecaster's Guide to the Future: How to Make Better Predictions

David Bakken, Foreseeable Futures Group

Nate Silver subtitled his best-selling book, The Signal and the Noise, “Why Most Predictions Fail, but Some Don’t.” Extending that idea, we identify five reasons that business predictions and forecasts based on market research fail to deliver and show how market researchers can overcome those factors to create better predictions.

(10:00 AM - 10:30 AM) Refreshment Break

Session 2

(10:30 AM) Wallet Economics?: Credit Card Choice-Based Conjoint - Beyond Preference & Application

Demitry Estrin, Vision Critical

Michelle Walkey, Vision Critical

Vidya Subramani, US Bank

Carla Wilson, VISA

Jane Tang, Vision Critical

Rosanna Mau, Vision Critical

CBC research works well for determining which credit card offers will be accepted. CBC does not, however, help in assessing the profitability of credit cards – which is in large part determined by total spend and spend by merchant category. We look at spend behavior to better understand the profitability of potential credit card offerings.

(11:15 AM) Conjoint for Financial Products: the Example of Annuities

Suzanne Shu, UCLA Anderson

Robert Zeithammer, UCLA Anderson

John Payne, Duke

Attributes of financial products jointly imply an expected present financial value, which should normatively drive customer choice. We conduct a CBC on annuities, and find that several attributes drive choice directly beyond their impact through the financial value. The direct impact and design implications depend on how attributes are described.

(12:00 PM - 1:30 PM) Lunch

Session 3

(1:30 PM) Comparing Message Bundle Optimization Methods: Should Interactions Be Addressed Directly?

Dimitri Liakhovitski, GfK

Faina Shmulyian, MetrixLab

Tatiana Koudinova, GfK

Frequently used Message Bundle Optimization methods (MBOMs) ignore the issue of semantic interactions among messages, or address it indirectly. This research compares the performance of several MBOMs in studies with real and simulated respondents. We also suggest and test an approach that tackles the issue of semantic interactions directly.

(2:15 PM) Using TURF Analysis to Optimize Reward Portfolios

Edward "Paul" Johnson, Survey Sampling International

Kyle Griffin, Survey Sampling International

An optimal reward portfolio is essential to an opt-in online panel. We compare four different techniques (some of which incorporate MaxDiff data) in creating an optimal rewards portfolio. Each technique’s predictions are then compared to actual respondent selections at the individual and aggregate levels.

(3:00 PM - 3:30 PM) Refreshment Break

Session 4

(3:30 PM) Bandit Adaptive MaxDiff Designs for Huge Number of Items

Kenneth Fairchild, Sawtooth Software

Bryan Orme, Sawtooth Software

Eric Schwartz, University of Michigan

When the goal is to identify the top few items in a MaxDiff design involving 100+ items (e.g. images, messages, or profile stimuli), standard MaxDiff designs are inefficient. We demonstrate an adaptive design based on the so-called “multi-armed bandit problem” and Thompson Sampling that leverages preferences from previous respondents and can be 2 to 3 times more efficient than standard MaxDiff.

(4:00 PM) What's the Right Sample Size for My MaxDiff Study?

Stan Lipovetsky, GfK

Dimitri Liakhovitski, GfK

Mike Conklin, GfK

Today, practitioners have no validated ways of determining desired sample size for MaxDiff studies. Our simulation assesses MaxDiff performance under a number of experimental conditions. The results enable practitioners to estimate the required sample size based on parameters of a planned MaxDiff study and desired Confidence Interval /Margin of Error.

(4:30 PM) Developing and Testing a Pay-for-Performance Incentive Mechanism for Conjoint Analysis

Philip Sipos, University of Hohenheim

Markus Voeth, University of Hohenheim

Incentive-alignment in conjoint-studies has established itself as state-of-the-art since its recent introduction to marketing research. However, a mechanism for settings in which the research object is not available does not yet exist. To fill this gap a monetary-based pay-for-performance mechanism independent from the availability of the research object is developed and tested.

(5:00 PM) General Session Ends

(5:15 PM - 6:15 PM) Discover: A Streamlined, Web-Delivered Version of CBC Software (Clinic)

(5:15 PM - 6:15 PM) Menu-Based Choice—MBC (Clinic)

(6:00 PM - 7:30 PM) Reception

Session 5

(7:00 AM - 5:00 PM) Conference Registration

(7:00 AM - 8:25 AM) Breakfast

(8:30 AM) Perceptual Choice Experiments: Enhancing CBC to Get from Which to Why

Bryan Orme, Sawtooth Software

Typical CBC market simulators predict choice likelihoods for multiple product concepts within competitive market scenarios; but they provide no insights into perceptions—the why’s behind choice. We show how to augment CBC questionnaires to add predicted agreement on perceptual/diagnostic dimensions to simulators, by product, as an interactive heat-map.

(9:15 AM) Profile CBC: Using Conjoint Analysis for Consumer Profiles

Christopher Chapman, Google

I describe how choice-based conjoint analysis may be used to assist consumer segmentation efforts, especially to help size profiles that result from qualitative research. I discuss both failed projects and successful ones. Then I describe the keys to success and how conference attendees could try this for themselves.

(10:00 AM - 10:30 AM) Refreshment Break

Session 6

(10:30 AM) RUM and RRM - A Great Mix

Kees van der Wagt, SKIM Group

Jeroen Hardon, SKIM Group

Different from a Random Utility Model (RUM), a Random Regret Model (RRM) assumes consumers take the context of a choice into account: choosing an option that minimizes regret on a few pertinent options, rather than maximizing utility. For some consumers, RRM may better represent how choices are made. We show how to combine RUM and RRM to yield the best predictive validity.

(11:15 AM) Capturing Individual Level Behavior in DCM

Peter Kurz, TNS Infratest

Stefan Binner, bms marketing research + strategy

Discrete Choice Models with HB estimation are state-of-the-art techniques. Yet, results often seem counterintuitive. Recent research papers have shown different findings about the right number of choice tasks, suitable covariates or the relevance of priors. The authors will dig a little deeper into these topics and show the underlying assumptions to derive a better understanding of DCM results.

(12:00 PM - 1:30 PM) Lunch

Session 7

(1:30 PM) Occasion Based Conjoint - Augmenting CBC Data to Improve Model Quality

Björn Höfer, Ipsos GmbH

Susanne Müller, Ipsos GmbH

In specific product categories occasions play an important role in moderating consumer choice. In these the integration of occasions can improve the results of conjoint studies: additional insights to support strategic decisions can be generated and the model quality can be improved. We have developed such an Occasion Based Conjoint (OBC) with an efficient way of data collection and augmentation.

(2:15 PM) Precise FMCG Market Modeling Using Advanced CBC

Dmitry Belyakov, Synovate Comcon

In this presentation I propose solutions to some of the most common FMCG CBC issues – long SKU lists and precise price elasticity modeling. I examine various encoding methods to obtain the best estimates depending on case study conditions.

(3:00 PM - 3:30 PM) Refreshment Break

Session 8

(3:30 PM) Optimizing the Employee Value Proposition through Conjoint Analysis

Tim Glowa, Bug Insights

Garry Spinks, Bug Insights

Allyson Kuper, Bug Insights

While most applications of conjoint focus on solving marketing problems, HR also is increasingly using this tool. Our presentation discusses how conjoint can be used to design, deliver, and position employee benefits and rewards in a more effective and efficient manner, and addresses some of the specific challenges of applying this analytical tool in the HR space.

(4:00 PM) Menu-Based Choice: Probit as an Alternative to Logit?

Christian Neuerburg, GfK Marketing & Data Sciences

Probit models seem to offer a number of advantages over logit-based approaches in the context of menu-based choice experiments. Nevertheless, logit is still the major workhorse for menu-based choice modeling. Based on a large-scale simulation study I investigate if and under which conditions probit is an alternative to logit.

(4:30 PM) Combining Latent-Class, Data Mining, and CBC to Identify Actionable Customer Segments for Telecom

George Boomer, StatWizards

Kiley Austin-Young, Comcast

To add robust demographic segmentation to HB estimation, we combine CBC/HB, latent-class choice and data mining techniques with StatWizards' Simulator Wizard to create a simulator that marries HB's excellent fit with Latent GOLD's powerful segmentation capabilities.

(5:00 PM) General Session Ends

(5:15 PM - 6:15 PM) Benefits of Using Sawtooth Software’s SSI Web Hosting Services (Clinic)

(5:15 PM - 6:15 PM) Teaching Conjoint Analysis at the University (Clinic)

(5:15 PM - 6:15 PM) New SSI Web Conjoint Market Simulator (Clinic)

(6:00 PM - 7:30 PM) Reception

Session 9

(7:00 AM - 5:00 PM) Conference Registration

(7:00 AM - 8:25 AM) Breakfast

(8:30 AM) Uncovering Customer Segments Based on What Matters Most to Each

Ewa Nowakowska, GfK

Joseph Retzer, Radius Global Market Research

Most clustering algorithms require segment members be similar across ALL attributes used. This leads to poor quality solutions when not all variables “matter” to cluster members. Co-clustering takes advantage of pairwise relationships between respondents and attributes to yield high quality clusters defined only by variables most meaningful to the cluster.

(9:15 AM) Climbing the Content Ladder: How Product Platforms and Commonality Metrics Lead to Intuitive Product Strategies

Scott Ferguson, North Carolina State University

Without a potentially large number of conditional constraints, optimization-based product searches are unlikely to return solutions with an intuitive distribution of basic or premium content. This paper demonstrates how applying commonality measures from the engineering design community results in more intuitive solutions and increased confidence in the discovered product strategy.

(10:00 AM - 10:30 AM) Refreshment Break

Session 10

(10:30 AM) A Machine Learning Approach to Conjoint Analysis: Boosting and Blending Ensembles

Kevin Lattery, Maritz Research

This presentation asks, “how might statisticians from machine learning approach conjoint analysis?” We describe a very popular machine learning algorithm, AdaBoost, and extend it to generating ensembles for conjoint. We also look at blending ensembles from different algorithms and theories: HB, Latent Class, Regret Minimization, and alternative variable codings.

(11:05 AM) The Unreliability of Stated Preferences: When Needs and Wants Don't Match

Marc Dotson, Ohio State University

Greg Allenby, Ohio State University

We examine the ability of consumers to provide reliable statements of preference for offerings that they may not need. We find that respondents do not express consistent choices for products that are not relevant to them. Our results have implications for conjoint design and screening criteria.

(11:35 AM - 11:40 AM) Best Paper Ballot Collection

(11:55 AM - 12:05 PM) Closing Remarks and Best Paper Award, Bryan Orme, Conference Moderator

(12:05 PM) Conference Adjourned

Break-Out Sessions (Wednesday—Friday)

Wednesday | Room 1: Survey Development

(1:30 PM – 3:00 PM) Intro to SSI Web

Justin Luster, Sawtooth Software

Gary Baker, Sawtooth Software

SSI Web is our survey software platform, and we've built it to expertly handle challenges from simple to the most complex. This introduction and tour will take you through its general survey question types and show how conjoint and MaxDiff exercises may be added to a survey. We'll demonstrate SSI Web's powerful piping, skip patterns, and randomization techniques, and then show off constructed lists. Finally, we'll show how survey programmers that understand JavaScript and web programming can create amazing survey customizations.

(3:30 PM – 5:00 PM) Intro to SSI Web (repeat session)

Justin Luster, Sawtooth Software

Gary Baker, Sawtooth Software

SSI Web is our survey software platform, and we've built it to expertly handle challenges from simple to the most complex. This introduction and tour will take you through its general survey question types and show how conjoint and MaxDiff exercises may be added to a survey. We'll demonstrate SSI Web's powerful piping, skip patterns, and randomization techniques, and then show off constructed lists. Finally, we'll show how survey programmers that understand JavaScript and web programming can create amazing survey customizations.

Wednesday | Room 2: Analytical Focus

(1:30 PM – 2:10 PM) Online Market Simulator & TURF Optimizer

Brian McEwan, Sawtooth Software

In this session we'll show off two of our online tools for data analysis. Our online conjoint simulator is a client-friendly, attractive, cloud-based simulator with visual outputs for utilities, importances, and shares of preference. We'll also show you our MaxDiff Analyzer tool, and how you can use built-in TURF algorithms and MaxDiff data for bundle optimization. You can even use our TURF tool with your own Likert scale ratings data!

(2:15 PM – 3:00 PM) Discover: A Streamlined, Web-Delivered Version of CBC Software

Justin Luster, Sawtooth Software

David Squire, Sawtooth Software

Come see Discover, our new online tool for creating, fielding, and analyzing choice-based conjoint surveys. The Discover tool is perfect for those new to conjoint analysis. You will be delighted by how simple we have made the process.

(3:30 PM – 4:10 PM) Introduction to MaxDiff

Nathan Bryce, Sawtooth Software

This presentation will review the history, methodology, and motivation for best/worst maximum difference item scaling and demonstrate how to build and analyze a MaxDiff survey using Sawtooth Software products and services. MaxDiff scaling is a discrete-choice trade-off method for measuring the importance or preference for multiple items, such as brands, product features, political platforms, advertising claims, etc.

(4:15 PM – 5:00 PM) What Is R? What Are Its Pros & Cons?

Christopher Chapman, Google

Kenneth Fairchild, Sawtooth Software

This 40-minute session is for those who are interested in a high-level introduction to R. We'll address questions such as: what is R? Is it a statistics program or a programming language? How does one learn R? What is it good for? What are some reasons to use it, and not to use it? We'll illustrate these with brief demonstrations of Bayesian regression models and automated reporting in R.

Thursday | Room 1: Survey Development

(8:30 AM – 9:10 AM) CSS and SSI Web

Justin Luster, Sawtooth Software

CSS allows you to modify the look or style of your surveys. Learning a bit of CSS can really help you to become a more powerful SSI Web user. Come learn a bit about CSS and instantly have more control over how your surveys look.

(9:15 AM – 10:00 AM) Perl and SSI Web

Justin Luster, Sawtooth Software

You can become a much more powerful SSI Web user if you understand some Perl programming. Perl allows you to modify and customize your surveys in powerful ways. Come learn a bit about Perl and instantly create more powerful surveys!

(10:30 AM – 11:25 AM) Making Best Use of Sawtooth Software: Enhancing SSI Web

Saurabh Aggarwal, KnowledgExcel Services

Faraz, KnowledgExcel Services

The focus of this tutorial will be on using customized scripts in SSI Web to program questions which are not available in Sawtooth Software and minimize the use of Free Format questions. The highlight of this session is to showcase how to manage multiple languages in a single SSI Web project. In this session, Saurabh also will demonstrate some of the interesting examples & add-ons you can program in CBC and MaxDiff Studies.

(11:30 AM – 12:00 PM) Constructed Lists in SSI Web

Justin Luster, Sawtooth Software

Constructed lists are a very powerful feature in SSI Web. They allow you to build dynamic questions with item lists that change based on previous answers. SSI Web gives you a ton of flexibility to create these dynamic lists. We will start with the basics and then show you some advanced capabilities.

(1:30 PM – 1:55 PM) SSI Web Hosting Service

Gary Baker, Sawtooth Software

Mike Lodder, Sawtooth Software

Need to launch your survey immediately but don’t want to deal with the complexity of servers? Worried about uptime, reliability, and backups? This session will demonstrate how the Sawtooth Software Hosting service can help with all of these areas—leaving you to focus on what you do best.

(2:00 PM – 3:00 PM) A Decade of SSI Web Knowledge from the Land Down Under

Paul Moon, Beaton Research + Consulting

Regardless of our Sawtooth Software proficiency, we often stumble across barriers that challenge us, regularly search for enhancements and even attempt to foresee a better way of doing things. This session includes everything from interesting tips to neat tricks, from oversights to common blunders, and maybe something innovative too.

You do not need to be an expert in SSI Script, Unverified Perl, JavaScript, CSS or HTML. You will be provided with a beneficial edge when you program your next Sawtooth Software survey. The spotlight will focus on the following Sawtooth Software topics: a) the power of SSI Script, b) you can do so much more with constructed lists, c) utilize your SSI Web question limit efficiently, d) program immaculate sub-group headings in a select type question, e) how to automatically relocate your terminating page to another website, f) defining variables in your survey using several methods, g) some worthwhile features you should know about skips.

So if you desire a boost to your Sawtooth Software “bag of tricks,” you are kindly invited to join us for an enlightening session.

(2:15 PM – 3:00 PM) Intro to ACBC

Aaron Hill, Sawtooth Software

If you are new to conjoint analysis, Adaptive Choice Based Conjoint (ACBC) can seem a bit intimidating. ACBC can handle conjoint surveys with lots of attributes, complex pricing, and small samples, and create efficient models without overtaxing respondents. This session will introduce you to the ACBC methodology and explore the features and components that make this tool unique. We will demonstrate the different sections of the ACBC survey, show how to create an ACBC exercise, and discuss some “Best Practices” to make sure your next ACBC project is a success.

(3:30 PM – 4:10 PM) SSI Web Data Generator

Mike Lodder, Sawtooth Software

What originally started as an in-house test tool is now accessible for general use. Available since SSI Web 8.2.0, the Data Generator can help you find errors in skip logic, verify quotas, uncover JavaScript issues, and discover design problems. The tool has also been used to create synthetic datasets for research on research. This session will show how to use the full capabilities of this powerful feature. We’ll explain how it works, show test strategies, and teach you how to predefine and customize survey answers to explore various paths that respondents could take.

(4:15 PM – 5:00 PM) JavaScript and SSI Web

Justin Luster, Sawtooth Software

You can become a much more powerful SSI Web user if you understand some JavaScript. JavaScript allows you to modify and customize your surveys in powerful ways. Come learn a bit of JavaScript and instantly create more powerful surveys!

Thursday | Room 2: Analytical Focus

(8:30 AM – 9:10 AM) Introduction to MaxDiff (repeat session)

Nathan Bryce, Sawtooth Software

This presentation will review the history, methodology, and motivation for best/worst maximum difference item scaling and demonstrate how to build and analyze a MaxDiff survey using Sawtooth Software products and services. MaxDiff scaling is a discrete-choice trade-off method for measuring the importance or preference for multiple items, such as brands, product features, political platforms, advertising claims, etc.

(9:15 AM – 10:00 AM) Intro to Bayesian Estimation

Jeffrey Dotson, Brigham Young University

This breakout session will provide attendees with a brief introduction to the mechanics of Bayesian estimation and an opportunity to peek under the hood of Sawtooth Software's CBC/HB algorithm. We will discuss what can go wrong (and why), with a goal of diagnosing and resolving problems as they occur.

(10:30 AM – 11:10 AM) What Is R? What Are Its Pros & Cons? (repeat session)

Christopher Chapman, Google

Kenneth Fairchild, Sawtooth Software

This 40-minute session is for those who are interested in a high-level introduction to R. We'll address questions such as: what is R? Is it a statistics program or a programming language? How does one learn R? What is it good for? What are some reasons to use it, and not to use it? We'll illustrate these with brief demonstrations of Bayesian regression models and automated reporting in R.

(11:15 AM – 12:00 PM) Models for MaxDiff and Related Data and their Performance with Latent Class and HB Methods

Jay Magidson, Statistical Innovations

Gary Bennett, Logit Research

Jaroen Hardon, SKIM Group

By eliciting the worst (least preferred) as well as best (most preferred) options, MaxDiff data are considered the “gold standard” for estimating preference for a group of stimuli (objects). In this presentation we use simulated and real MaxDiff data to compare various MaxDiff and extended Best-Worst (Multi-attribute CBC and “Dual Response None”) models and examine whether the utility estimates improve over those from the standard best-only design. We also compare Latent Class (LC) and Hierarchical Bayes (HB) methods that allow for individual differences. By focusing on the form and magnitude of bias in model estimates, and when it occurs, our results are somewhat surprising.

(1:30 PM – 2:10 PM) New SSI Web Conjoint Market Simulator

Brian McEwan, Sawtooth Software

Walter Williams, Sawtooth Software

Join us for a walkthrough of the new conjoint simulator slated for release in 2015. We’ll show how to create a new project and many of the new features we’re working on. We’ve revamped the main view to allow for multiple, concurrent simulations with a host of new tricks. Additional improvements include greater control over sensitivity and optimization searches, visual displays of utilities, importances, and simulation results, and better options (than SMRT) for incorporating product availability, awareness, and external effects.

(3:30 PM – 5:00 PM) Flexible Drivers Analysis Using Decision Trees

Joseph Retzer, Radius Global Market Research

Ewa Nowakowska, GfK

Commonly used drivers models such as ordinary least squares, multinomial logit, ordered logit, etc., are capable of relating a response variable (continuous or categorical, depending on the model) to a set of explanatory variables.

An alternative approach broadly referred to as "decision tree analysis" offers a highly flexible, typically non-parametric (no assumptions about the distribution of the data are necessary) approach providing easily interpretable, often highly predictive, graphical insights. These models are referred to as "trees" because the output resembles that of an upside down tree.

This tutorial will focus on the most popular approaches to decision tree analysis including, CHAID, CART, Boosting/Bagging and Random Forests. R code for implementing each model will also be provided and discussed.

Friday | Room 1: Survey Development

(8:30 AM – 9:10 AM) Pro-Tips for Survey Design

Sharon Alberg, Maritz Research

Want to do the best modeling? Deliver the best results? Both start with a well-designed survey. This 40-minute class goes beyond the usual top ten lists of problems to avoid, giving you concrete guidance on making your next project as successful as possible. Just some of the areas we cover:

  • Structure of content—ordering questions, randomization strategies, and brand quotas
  • Screening—best consumer, business and household questions to ask for weighting to census information
  • Survey format—optimal formats for online and smartphones
  • Content of questions—common mistakes when writing questions
  • Scaling—scales that work best for individual and multiple methodologies
  • Data collection—incentives, reminders, and fielding issues to manage
  • Trending issues—what kinds of changes can bias trending on tracking studies

(9:15 AM – 10:00 AM) Mobile Surveys, Plus How SSI Web Handles Mobile

Justin Luster, Sawtooth Software

Sharon Alberg, Maritz Research

How should survey development differ for Mobile? Also, come see how SSI Web surveys adapt for mobile devices. We will share various tips for how to make sure your surveys look great on small mobile devices. We will also review research done on taking conjoint analysis surveys on mobile devices. Mobile is hot. Come make sure your surveys are ready.

(10:30 AM – 11:10 AM) Data Privacy and Holding Back the Legislative/Regulatory Deluge

Howard Fienberg, Marketing Research Association

Howard Fienberg, MRA's lobbyist, will cover some of the worst legislation in the country and the top legal concerns facing researchers. He will also explore the current data privacy regulatory landscape, where things are headed, what policymakers and the Federal Trade Commission want from us, and how researchers can stay ahead of the curve.

(11:15 AM – 11:50 AM) SSI Web on Android – Collect Data without an Internet Connection

Justin Luster, Sawtooth Software

Mike Lodder, Sawtooth Software

Need to collect data for a survey without an Internet connection? Need to sync survey updates to multiple locations? Come see the new SSI Web CAPI on Android for mobile survey data collection. Data collection without an Internet connection just got a whole lot easier.

Friday | Room 2: Analytical Focus

(8:30 AM – 9:10 AM) Introduction to Segmentation and CCEA

Keith Chrzan, Sawtooth Software

Segmentation analysis is a staple of the marketing researcher’s toolbox. We introduce segmentation analysis using an intuitive, conceptual approach rather than a heavily mathematical one. After covering basic concepts and illustrating graphically how different segmentation approaches work, we will demonstrate how to use the CCEA (Convergent Cluster & Ensemble Analysis) program and its powerful segmentation algorithms.

(9:15 AM – 10:00 AM) Rating Scales in Market Research

Keith Chrzan, Sawtooth Software

Perhaps no topic raises more controversy than the seemingly simple one of how to construct rating scales. This session seeks to cut through the clutter and give evidence-based advice, where it exists, on how to make decisions about several properties of rating scales (number of response categories, balance, labeling, polarity, etc.). The session also recommends some scales for common uses in marketing research and covers the results of some specific case studies comparing alternative rating scales that can be used in brand image and customer satisfaction measurement.

(10:30 AM – 11:10 AM) Weighting Principles and Perils

David Lyon, Aurora Market Modeling

Some researchers act as if any weighting scheme amounts to fudging the data, while others seem to think weighting can easily fix every possible problem in their sample or data. The truth, of course, is somewhere in between. This tutorial will review the upsides and downsides of weighting, its relationship to sample design, when it is required and when it is optional. It will stress using the notion of the “effective sample size” (which is completely unrelated to the “weighted total”) as a key indicator of how much downside there may be. The ESS also offers a simple approach to calculating standard errors, confidence intervals and statistical tests that accurately account for the weighting.

(11:15 AM – 11:50 AM) Which Conjoint Method Should I Use?

Aaron Hill, Sawtooth Software

This session will introduce the many conjoint and discrete choice analysis options offered by Sawtooth Software and help you determine when it is appropriate to use each method. Example case studies will illustrate various outcomes achieved with different conjoint approaches.