|
||||||
Technical Papers| Home |
Solutions |
Products |
Services |
Education |
Support |
| | ||||||||
|   SS Winter 2005European Conference and Training, May 25-27, Berlin, GermanyWe are pleased to announce the 2005 Design & Innovations Conference, to be held May 25-27, 2005, in Berlin, Germany. This event is co-sponsored by Sawtooth Software and SKIM, our European representative. It will closely follow the format of the successful US Sawtooth Software conferences.The first two days will be filled with valuable tutorials and workshops, and the last day will follow a conference format, with speakers recruited through a general call for papers. The tutorials and workshops include:
To learn more about this exciting event, or to register, please visit www.skim.nl, and browse to the SKIM Software Division announcements.
General Online Surveys Take Front SeatFor some time, Sawtooth Software had invested most of its development effort toward conjoint analysis and advanced analytics. We'd done this quite successfully, and as a result, many of you have come to think of us as a "conjoint company." It may surprise some of you to know that Sawtooth Software's first interviewing product (Ci2, in 1983) was a general interviewing product. The majority of our revenues over the first decade of business came from general interviewing.About two years ago, we turned our attentions back to general interviewing software, and focused on critical improvements to our CiW interviewing system (a component within our SSI Web platform). Our efforts have paid off--in fact, the new system has exceeded even our high expectations. Our users have been very impressed with the increased power and ease of use for SSI Web. If you aren't using CiW for your general web interviewing, we think it is time that you consider whether CiW should be your principal interviewing platform. Long-Time User Sounds Off John Fiedler (Oreon, Inc.) has been a longtime user of our products, and brings a wealth of experience and perspective to the table. He is currently using SSI Web (CiW) to conduct a complex study with piping, list-building, and challenging skip patterns. Here's what John has to say about SSI Web v5: "Version 5 of CiW has been worth waiting for and more than well worth paying for. It's been almost 20 years since we first used Ci2; then there was Ci3 through the 1990s. The early versions of CiW were a step forward in moving many of us from CAPI interviewing onto the Web. But those same early versions were a step backwards in terms of control and flexibility." "Despite my long commitment to Sawtooth Software, I was forced to use other Web-based interviewing software to field complex studies on the Internet. No longer. Version 5 of CiW seems to offer unlimited flexibility and control. Fifteen hours of work produced a questionnaire with 1,100 data fields and over 60 pages; last year, a similar project took a month to program using competitive software. The upgrade paid for itself in a day." Download a Trial Copy Today! You can download a functional, demo version of SSI Web from www.sawtoothsoftware.com/downloads.shtml The demo includes a step-by-step tutorial, on-line help, and an extensive manual (.PDF format). "Constructed Lists" Added to CiW v5 With the release of v5, our general online interviewing software has taken a huge leap forward in usability and power. Those who have used our general interviewing software through the 1990s (Ci3) will recognize the term "constructed lists." Survey questions often display lists of brands or other response options that we might generically refer to as lists. Some lists of response items are always presented in a fixed order, such as the standard:
In other lists, we might want to randomize the order of presentation, to control for order effects. CiW of course can randomize presentation order for lists.
With "constructed" (dynamic) lists, we can present a new list of response options to respondents, based on answers given to a previous question, such as:
The list for Q2 is constructed "on the fly," based on answers to Q1. This is just one example of how constructed lists may be used. Constructed lists may be developed using a variety of conditions, such as list items chosen or not chosen in a previous question, high or low ratings, forced inclusion, or probabilistic inclusion. New Licensing and Pricing Policies Effective May 1, 2005As of May 1, 2005 Sawtooth Software will begin selling software and version upgrades under a new per-seat licensing and pricing policy.Background The existing Category licensing and pricing policy has been in place for over 20 years. Even though it has been easy to manage and has served us quite well over the years, it is not sustainable into the future. There are three main flaws with the current Category licensing policy:
The Category licenses have also been difficult for users to understand. We suspect that only a minority of customers actually understand the Category policy. As you surely understand, it is difficult to alter procedures that have been in place for over 20 years. We have rarely raised any price for our software over our history. Yet, we recognize that this new pricing policy will mean a price increase for many of our users. We respect and appreciate our colleagues and friends, and therefore approach this change very seriously. New Policy: Effective May 1, 2005
As many of you know who have asked for additional discounts, we do not offer them. Our users can be confident that we have applied and will continue to apply our licensing policies uniformly and equitably. Conference 2004 Summary of FindingsNineteen presentations were delivered at the eleventh Sawtooth Software Conference, held October 6-8, in San Diego, CA. We've summarized some of the high points below. Since we cannot possibly convey the full worth of the papers in a few paragraphs, the authors have submitted complete written papers within the 2004 Sawtooth Software Conference Proceedings available for ordering from Sawtooth Software.
It's Ethical Jim, But Not in the Way We Used to Know It! (Ray Poynter, UK): Ray questioned how good the data are when we pursue some respondents relentlessly, after they have already refused multiple times to participate. He condemned improper uses of marketing research, such as SUGGING (selling under the guise of research) and push-polling. Legislation is increasing which may impact our ability to conduct research among the general population. As a result, the use of permission-based panels will likely increase. However, Ray questioned how ethical it is to report a 60% response rate for a panel survey, when the panel includes fewer than 1% of the population. Ray suggested we spend more resources training our staff in professional standards and to act ethically. He stressed that we consider the consequences of our research, and whether each project we undertake truly benefits respondents/consumers. And, we shouldn't necessarily "do unto others as you would have them do unto you" because their preferences may not be the same!
A Structured Approach to Choosing and Using Web Samples (Theo Downes-Le Guin, Doxus): Theo suggested that we estimate the likely sources of error, and use this information to decide, separately, regarding sample source and data collection mode. We should find the appropriate balance among manageability, cost efficiency, and reduction of error. Practical questions to ask include: "If a frame-based approach is available, does it offer a compelling advantage?" "If an intercept-based approach is the only alternative, have we made the risks/pitfalls known to our clients?" "Can we combine more than one sampling approach, and then compare the results to guide future decisions?"
Optimizing the Online Environment: Examination of a Configurator Analysis Case Study (Donna Wydra, Socratic Technologies, Inc. and Debra Kassarjian, Taco Bell): The authors demonstrated a different technique that they felt improved upon the previous lines of research: virtual product configuration. The survey was computer-based, with an attractive Flash-based exercise wherein respondents built their optimal burrito. Respondents could choose ingredients, and (with a variety of stimulating visual effects) watch the burrito being built before their eyes. After they built their idea burrito, respondents were asked Van Westendorp pricing questions, to investigate a reasonable range of price expectations. Cluster analysis revealed groups of individuals that chose similar ingredients. Even though the virtual configurator (as well as previous research) had limitations in terms of forecasting demand for new product concepts, the authors felt that the results were more insightful than previous research. The Options Pricing Model: A Pricing Application of Best-Worst Measurement * (Keith Chrzan, Maritz Research): Best/Worst (or MaxDiff Scaling) is a relatively new technique forwarded by Jordan Louviere and colleagues back in the early 1990s. Keith showed how this technique could be applied to estimate the relative demand and price sensitivity for automobile options (such as sunroof, anti-lock brakes, etc.). Best/Worst questionnaires generally show a subset of the possible options, and ask respondents to indicate which is the best within the set and which is the worst. Eleven automobile features were tested at four price points each, among 202 consumers. Keith analyzed the data using HB analysis, which yielded individual-level utility parameters (on a common scale) for each of the options at each price point. This permits the researcher to forecast which options (and at which price levels) would be most accepted in the marketplace. Utilities were converted to relative probabilities of choice by taking the antilog (exponentiating). Correlation between model prediction and self-reported actual purchase of options on the most recently purchased vehicle was 0.92. This suggested that the client could use the best/worst data to reasonably project future sales for not-yet-offered options and determine appropriate prices for each. (*Winner of Best Presentation award, based on attendee ballots.)
Conjoint Analysis: How We Got Here and Where We Are: An Update (Joel Huber, Duke University): Practitioners replaced the rankings (early card-sort conjoint) with ratings (and later, choices), and replaced the full factorial designs with highly fractionated ones. Joel suggests that conjoint analysis has worked well because the simplification that respondents do in the conjoint survey often mirrors choice in the marketplace. Conjoint reflects how individuals might choose, given full information and more experience in making choices. Choice has dominated ratings-based conjoint lately due to a number of factors. It is argued to better relate to market behavior, it emphasizes the competitive context, it is better for dealing with price/cost, and people are willing/able to make choices about just about anything. Surprises over the years have been the power of market simulators to account for differential substitution, the success of HB in predicting individual choices, and the difficulties of finding adaptive designs that outperform orthogonal ones.
The "Importance" Question in ACA: Can It Be Omitted? (Chris King, Aaron Hill, and Bryan Orme, Sawtooth Software): The authors conducted a test to see if the importance questions could be skipped. Respondents were randomly assigned to receive a version of ACA that either included or didn't include the Importance section. Respondents who didn't receive the Importance section completed an extra six Pairs questions, though the total interview time was over a minute shorter than the respondents who completed traditional ACA. The authors found that traditional ACA (with the Importance question) achieved slightly better hit rates (for holdout choice tasks), but the share predictions were better when importances were omitted (in favor of six more pairs). Also, the final utility results without the pairs showed more discrimination among attributes ("steeper" importances) and provided different information from the utilities using the self-explicated importance information. The authors concluded that the Importance question could, if using HB analysis, be omitted.
Scale Development with MaxDiffs: A Case Study (Luiz Sa Lucas, IDS-Interactive Data Systems): Luiz found an excellent match between the scales he developed from MaxDiff, and the "power rule" relationships found in the previous studies. Luiz also investigated the use of latent class to develop segments of individuals, based on their MaxDiff judgments of the seriousness of offenses. He also collected attitudinal statements to classify people in different psychological segments, as suggested by previous research. The segmentation developed from MaxDiff seemed consistent with that in the literature. Luiz's case study lends additional credibility to the claim that MaxDiff results are both ratio scaled (after exponentiating the logit-scaled coefficients) and reliable. And, a MaxDiff questionnaire is probably easier for respondents to complete than the questionnaires employed by the previous authors.
Multicollinearity in CSAT Studies (Jane Tang & Jay Weiner, Ipsos-Insight): For both simulated and real CSAT data cases, the authors analyzed bootstrap samples to demonstrate that these methods result in much more stable estimates of drivers of satisfaction than standard OLS or stepwise OLS. They concluded that when the objective is to establish relative importance, rather than forecasting the dependent variable, methods that take into consideration all possible combinations of the explanatory variables are much more robust. Moreover, the margin of victory for these techniques increased as sample size decreases.
Insights into Patient Treatment Preferences Using ACA (Liana Fraenkel, Yale School of Medicine, and Dick Wittink, Yale School of Management): The use of conjoint allows patients to express individual tradeoffs between efficacy, side effects, mode of treatment and costs. This avoids the limitation that physicians lack the time and the training to capture each individual patient's unique perspectives. A field experiment is in process to show whether the use of ACA changes prescription decisions, enhances patient satisfaction, improves health outcomes, and reduces total cost of care.
Modeling Conceptually Complex Services: The Application of Discrete Choice Conjoint, Latent Class, and Hierarchical Bayes to Medical School Curriculum Redesign (Charles E. Cunningham, Ken Deal, Alan Neville, and Heather Miller, McMaster University): Using qualitative research, the authors developed a list of attributes important to the quality of the program. Fourteen attributes each on four levels were used in a web-based, partial-profile CBC interview. The authors employed Latent Class to investigate the preferences of segments of students. Two segments emerged: students whose preferences better aligned with McMaster's small group, problem-based tutorial curriculum, and a smaller group of students who seemed to favor a more traditional medical school program Based on market simulations, the authors made specific recommendations regarding lower-cost (but significantly preferred) options that could improve the existing curriculum, despite an increase in tutorial group size. The results also underscored the need to identify prospective students during the admission process that are a better fit with McMaster's curriculum.
Over-Estimation in Market Simulations: An Explanation and Solution to the Problem with Particular Reference to the Pharmaceutical Industry (Adrian Vickers, Phil Mellor, and Roger Brice, Adelphi International Research, UK): The authors described their typical questionnaires, including how they strive to establish a realistic setting (asking the physician to consider a specific patient when completing choice tasks), the use of partial profiles if more than six attributes, and HB estimation to obtain individual-level estimates. In addition to those standard procedures, they establish a cut-off threshold, applied at the individual level, and external effects, also applied at the individual level, to reflect other market realities for releasing a new drug. The full model takes into account any reluctance the physician may have about prescribing a new drug and also the volume restriction that results from third party payers and/or a physician's own consideration of what is a "fair" allocation among available product options. The amount of reduction depends upon such things as how serious and common the condition is, and the competitiveness of the market. The model still assumes 100% awareness of the new product, and the authors believe it represents an important step closer to predicting a realistic market share.
Estimating Preferences for Product Bundles vs. a la carte Choices (David Bakken and Megan Kaiser Bond, Harris Interactive): Based on a real client need, David and Megan designed a more complex CBC-like choice task that incorporated the notion of mixed bundling. Respondents (web-based survey) were instructed that they could either purchase the components as a bundle from a single manufacturer, or they could purchase the components separately from different manufacturers. Such a complex task required careful questionnaire design, pre-test, and detailed explanations and examples. The key to their solution was to develop two types of models from the data: a set of part worths predicting purchases of the bundles (including a coefficient for non-purchase of bundle), and a set of part worths predicting selections of each a la carte item given rejection of the bundle. The authors built a spreadsheet simulator that included a simple "if" condition to determine whether the shares of preference for each individual would be captured by the bundled alternatives or the a la carte alternatives.
The Importance of Shelf Presentation in Choice-Based Conjoint Studies (Greg Rogers, Procter and Gamble, and Tim Renken, Coulter/Renken): The authors fielded a CBC study using the different layout approaches in CBC, covering multiple product categories. The criteria for success were: 1) ability to capture the same price elasticity estimates as are obtained using an IRI Marketing Mix Model (regression-based model using actual scanner sales data), and 2) ability to predict actual market shares. They found that the shelf layout provided slightly better fit to econometric models of price sensitivity, but that the grid layout provided slightly better fit to share. Importantly, the authors found that the estimates of price sensitivity from CBC were on average unbiased with respect to the scanner data models, though they often missed by a significant margin when any one product or brand was considered. They concluded that both exercises yield similar results, though the shelf display seems to have greater face validity, and is therefore easier to sell to clients.
The Effect of Design Decisions on Business Decision-Making (Curtis Frazier and Urszula Jones, Millward Brown-IntelliQuest): The authors found that partial profile tended to dampen the relative importance of price and increase the importance of brand, relative to full-profile. Including a "None" concept (either within the task, or as a second-stage question following the choice task) tended to increase the relative importance of price. To test how these differences might affect business decisions, the authors created hundreds of potential simulation scenarios, and used the part worths from the various treatments to determine "optimal price" points to maximize revenue for a client's hypothetical offering. As suggested by the findings with regard to "importances," partial profile designs lead to a significantly higher optimal price point. Including a "None" option in the questionnaire yielded the lowest derived optimal price points. Also, asking the "None" as a separate follow-up question produced much higher overall "None" usage, relative to when "None" was included in the choice task. Curtis and Urszula hypothesized that when "None" is included in the choice task, respondents may wish to appear cooperative by avoiding use of the "None." The authors concluded that different design decisions often have modest effect on holdout hit rates and share prediction accuracy, but can have a much bigger impact on business decisions, such as finding the right price points and projecting overall demand by relying on the scaling of the "None."
Application of Latent Class Models to Food Product Development: A Case Study (Richard Popper, Jeff Kroll, Peryam & Kroll Research Corporation, and Jay Magidson, Statistical Innovations): Different models were used to detect consumer segments according to their liking ratings for the crackers. Four main models were tested: Latent Class (LC) Cluster model (nominal factor), LC Factor model (discrete factors), LC Regression model with a random intercept (nominal factor + one continuous factor) and a parsimonious non-LC regression model (two continuous factors). Latent GOLD software was used. The authors concluded that there was clear evidence of segment differences in consumers' liking ratings. Respondents reacted similarly to the variations in flavor and texture, but differed with regard to how they reacted to the products' appearance. Many other details regarding the relative strengths of the different models are covered in the full written paper.
Assessing the Impact of Emotional Connections (Paul Curran, Greg Heist, Wai-Kwan Li, Camille Nicita, Bill Thomas, Gongos and Associates, Inc.): The authors felt that a key aspect to the research was a "negative priming" exercise. Based on previous research in psychology, the idea was to mentally overload and distract respondents in order to hinder overly-conscious thinking. After the negative priming exercise, respondents completed a discrete choice task involving eight emotional drivers on two dimensions for each vehicle: "How do you want to feel about a vehicle?" (Importance) and "Which do you associate with the [brand]?" (Brand Association). The authors examined the data by respondent segments based on automobile ownership. The most important emotional drivers were "peace of mind" and "smart and practical." A perceptual map (using correspondence analysis) displayed the results of the brand associations. VW was strongest on "fun to own" and "happy and carefree," Honda owned "peace of mind," and Saturn was strongly associated with "care for others." The authors concluded that "Independent or self-reliant" represented a positioning opportunity that no one vehicle measured currently fills.
Item Response Theory (IRT) Models: Basics, and Marketing Applications (Lynd Bacon & Jean Durall, LBA Ltd., and Peter Lenk, University of Michigan): The authors suggested that IRT provides a rich framework for test item construction, which may have potential in marketing research. Test items can be characterized by their two parameters: discrimination and difficulty. Both of these parameters are combined in the item information function (IIF), which summarizes how much information an item has in estimating the latent trait for different values of the latent trait. One can easily imagine developing a large bank of marketing research items for different concepts, such as loyalty and satisfaction, where the items are indexed by their IIF. A marketing researcher could then select items from these banks to construct survey instruments for various purposes, such as studying highly loyal customers or dissatisfied customers. IRT enables the design of adaptive, online surveys. After obtaining an initial estimate of a subject's latent trait, an adaptive survey might select items to better estimate the trait with fewer responses. Instead of using a "shot-gun" approach to survey design, marketing researchers could be more strategic and systematic by employing the IRT framework.
Avoiding IIA Meltdown: Choice Modeling with Many Alternatives (Greg Allenby, Ohio State University and Jeff Brazell, The Modellers, LLC): Predictive accuracy of actual market shares for automobiles showed a small improvement for the error-restricted models relative to traditional error specifications, despite the restrictive assumptions made about the error terms. The authors also illustrated their solution using a packaged-goods problem.
A Second Test of Adaptive Choice-Based Conjoint Analysis (The Surprising Robustness of Standard CBC Designs) (Rich Johnson, Sawtooth Software, Joel Huber, Duke University, and Bryan Orme, Sawtooth Software): 1009 respondents completed a web-based study, randomly receiving one of four questionnaires (ACBC vs. CBC crossed by full profile vs. partial-profile). The hit rates and share predictions for the ACBC vs. CBC treatment were similar. However, holdout predictions were better for full profile than partial profile when predicting choices of full-profile holdouts, probably due in part to methods bias. The authors were puzzled why ACBC didn't perform better. Upon further investigation, they discovered that the adaptive designs were indeed about twice as efficient with respect to the information they were given (initial self-explicated part worths). However, the self-explicated information used to generate the designs was not accurate enough to produce efficient designs with respect to the final CBC-derived part worths. Omitting the "importance" question was apparently a mistake. The authors suggested that rather than depending on self-explicated information, a better method in future research might include using part worth information from previous respondents (through HB or Latent Class) as initial estimates, and updating those estimates after each respondent answer. On the Design of MaxDiff ExperimentsWhat Is MaxDiff?MaxDiff (Best/Worst) scaling has received a growing amount of interest lately. Papers on this topic have won "best presentation" at our two most recent Sawtooth Software conferences. MaxDiff provides a trade-off based alternative to standard rating scales for evaluating the desirability or importance of items. The items may be product features, products themselves, political candidates, brand names, etc. The basic idea behind MaxDiff is to present respondents with (typically) from 4 to 6 items at a time, and ask which item (among the set) is "best" and which is worst. Respondents often complete a dozen or more choice sets. MaxDiff questioning leads to more information gained per respondent effort than the classic Method of Paired Comparisons. For example, if among items A, B, C and D the respondent indicates that B is "best" and C is "worst," we can infer that B>A, B>C, B>D, A>C, D>C. From these two choices, MaxDiff can infer five of the six possible paired comparisons involving four items. Indeed, MaxDiff has been shown to provide results superior to Paired Comparisons in a recent methodological test (see "Maximum Difference Scaling: Improved Measures of Importance and Preference for Segmentation" by Steve Cohen, available in our Technical Papers Library at www.sawtoothsoftware.com). Conducting MaxDiff Studies MaxDiff studies may be fielded via paper-and-pencil or computerized questionnaires. Constructing these questionnaires requires determining which items appear in the different choice sets (an experimental design). MaxDiff experimental designs may be generated using our "Best/Worst Designer" software system. Analysis may be done using our Latent Class or CBC/HB software systems. (We hope to implement MaxDiff more integrally within a future version of SSI Web.) Common questions for MaxDiff studies include:
We recently performed a simulation study to investigate these questions with respect to individual-level estimation under hierarchical Bayes (HB). A detailed write-up of this investigation is found in the article entitled "Accuracy of HB Estimation in MaxDiff Experiments" also available in our technical papers library on the Web. Simulation Study Methodology and Results We used computer-generated respondents following known utility rules (plus random error) to answer different MaxDiff questionnaires. The questionnaires varied in terms of the number of items overall, number of items per set, and sets offered per respondent. Then, we estimated parameters for each "respondent" under HB and observed how well the estimated parameters could predict holdout choice questions answered by the same "respondents." Figure 1 summarizes the results across all questionnaires in our experimental design, in terms of relative hit rates.
 Conclusions Based on the results of our simulation study, we conclude:
These results should be interpreted with caution, since they are based on simulated results. We look forward to studies that use actual respondents to co |
|||||||
Technical Papers
|
About Us | Contact Us | Privacy | FAQ's | Forum |
|
© 2008 Sawtooth Software, Inc. All rights reserved. |