|
||||||
Technical Papers| Home |
Solutions |
Products |
Services |
Education |
Support |
| | ||
|   CVA v3 Experimental Designer Better than BeforeWith the recent release of CVA v3, all three base Sawtooth Software conjoint systems are now integrated within the SMRT platform. (SMRT stands for "Sawtooth Software Market Research Tools".) While we are pleased with the added ease-of-use and polish the new Windows-based CVA offers, we are particularly excited about the improvements in the experimental designer. For starters, rewriting and compiling the program for the 32-bit operating system produced about a 5x speed improvement. But more importantly, increasing the "pool multiplier" capacity and an alteration to the existing algorithm significantly improved the designer.Back in 1997, Warren Kuhfeld of SAS reviewed our CVA v2 designer at the Sawtooth Software conference and gave suggestions for improvement (see "Efficient Experimental Designs Using Computerized Searches" available at www.sawtoothsoftware.com in the Technical Papers library). One of Warren's suggestions was that we search among larger candidate sets equal to, if possible, the full-factorial design. The original CVA v2 used a maximum pool multiplier of 20 when creating candidate sets. In other words, if the user requested 25 conjoint questions, a maximum of 25 x 20 = 500 conjoint questions were used as candidates for finding the optimal subset of 25. With CVA v3, we let the user set the pool multiplier as high as 999. This allows CVA v3 to create candidate sets that usually reach or approximate a full-factorial. In version 3, we also changed the algorithm so that we spend less time initially reducing the candidate set to the desired number of questions. The previous version started with the entire candidate pool and discarded one question at a time (the question that when discarded gives the surviving set the highest D-efficiency). With the new version, we only use an initial pool of up to the first 10x tasks during this step, where x is the number of requested tasks. Then, we concentrate the algorithm's effort on the two-way swaps using the full candidate pool. The combination of the 32-bit operating environment and the methodological refinements yield improved results in a measurably shorter time than can be obtained using the previous version. Comparison of v2 and v3 Designers To demonstrate how much better the v3 designer works than v2, we'll use two examples from Warren's 1997 paper. The first example is a 2^2 3^3 5^2 design in 30 cards (single-concept). (A 2^2 3^3 5^2 design means that there are two 2-level attributes, three 3-level attributes and two 5-level attributes.) The full factorial is 2700 cards. With 30 requested cards, a pool multiplier of 90 would create the full factorial. Using a pool multiplier of 20 (the maximum under v2) and 20 attempts (passes) using different random starting points, CVA v2 found a design with a D-efficiency of 97.72 in about 8 minutes. Using a pool multiplier of 90 and 10 passes, CVA v3 found a design with a D-efficiency of 98.23 in about 1 minute. (We used a somewhat dated 500 MHz processor for all runs reported in this article.) The second example is even more dramatic. The design is much more demanding: 2 3 4^2 5^2 6 in 24 cards (single-concept) with 6 prohibitions. With CVA v2, we ran the problem with the maximum pool multiplier of 20 for 10 hours (1200 passes). The best of the 1200 passes resulted in a D-efficiency of 84.68. We tried the same plan under CVA v3 using a pool multiplier of 225, using just 10 passes (each pass took roughly 60 seconds). After these ten passes, v3 found a design with a D-efficiency of 86.46. Four of the first ten passes had a D-efficiency superior to the best result from the 10-hour CVA v2 run. These results demonstrate that using larger candidate sets, focusing more of the effort on the two-way "swapping" step, and the increased speed under the 32-bit implementation make version 3's designer far superior to v2. Call for Papers--Tenth Sawtooth Software ConferenceOn April 15-17, we will hold our tenth Sawtooth Software Conference in San Antonio, Texas. Our research conference brings together some of the best minds in our industry to talk about practical issues in computer/Web interviewing and quantitative market research. It is not a sales-oriented event for our software, but a chance to exchange ideas and receive education from a variety of sources and perspectives. Papers presented at our previous Sawtooth Software Conferences are cited frequently in journal articles.We're looking for exceptionally strong papers. If you'd like to be on the program, please respond promptly (by October 11, email: bryan@sawtoothsoftware.com) with a one-page abstract describing your proposed paper, with special attention to what the audience will "take away" from the presentation. We are interested in papers on a variety of subjects, including Web interviewing, market segmentation, mass customization, customer satisfaction modeling, conjoint/choice analysis, perceptual mapping, hierarchical Bayes methods, forecasting, pricing research, market simulations and case studies. These papers need not involve Sawtooth Software's programs or approach. In an effort to provide more balance to the program, we are equally interested in papers that are NOT about conjoint/choice modeling. For all topics, we are eager to see evidence of managerial relevance, external validity, evidence of profit impact, etc. Presenters receive a complimentary conference registration. To be accepted, a paper must show promise of being sufficiently practical to be of use to the least sophisticated members of the audience, while having enough substance to be of interest to the most sophisticated members. In addition to standard presentation slides, authors are required to submit a journal-quality written paper for publication in the Conference Proceedings. We strive for the highest quality in our conferences. If your abstract is accepted, a member of the steering committee will review early drafts of your presentation and offer suggestions. Authors are expected to consider these suggestions conscientiously and rework their presentations as needed. Sawtooth Software reserves the right to remove any author from the program or proceedings that fails to meet deadlines or produce high quality work. Sawtooth Software Conference 2003 Steering Committee Members are:
 Go back to IndexUpcoming Workshops AnnouncedWe often receive requests from researchers looking for practical materials and training opportunities for conjoint analysis and web interviewing. Our day-and-a-half workshops are excellent opportunities to gain the skills and insights you need to advance your understanding of conjoint analysis and your career as a provider/consumer of conjoint analysis studies.You do not need to own the systems to attend the training. Free demonstration versions are provided for use in all in-class exercises. The ACA and CBC Workshops are led by Dr. Tom Pilon, a leading expert in the use of conjoint design and analysis. Tom is a private consultant with over 20 years of experience in this area. He brings a wealth of experiences and applied knowledge to share with students, providing detailed best practices during the sessions. The workshops provide the foundations and underlying theory about the techniques in general, as well as the knowledge and experience required to successfully use the Sawtooth Software conjoint analysis products. Class size is limited to 15 people per session to ensure ample opportunity to interact with the instructor and the other students. The cost is $700 to attend the 1.5 day workshop held in Dallas, Texas. The ACA and CBC classes focus on the use of our Windows-based conjoint analysis products, but given the overlap in our Web and Windows-based software, much of the information is relevant to both. We also offer a Web Interviewing workshop ($500) that covers general Web i nterviewing along with ACA/Web and CBC/Web. The basis of this workshop is our SSI Web system for Internet interviewing. It is taught by Sawtooth Software personnel. Our next workshops are:
Web Site Targets Academics Teaching Conjoint AnalysisWe are pleased to announce a new area on our web site designed to assist professors in teaching conjoint analysis. Different lecture outlines are suggested for teaching traditional full-profile conjoint analysis, discrete choice, or Adaptive Conjoint Analysis (ACA). Even though the site is aimed at academics, we hope that professional organizations will also use the materials for training purposes.The focus is on providing free materials for download and student exercises that can be done entirely using Microsoft Excel(TM). Professors may take advantage of the site without buying software licenses from us or paying royalties. The site features:
We encourage academics to adapt/modify our materials to their needs. The materials may be used in whole or in part as long as the citation "Copyright Sawtooth Software" or "Portions Copyright Sawtooth Software" is included. Students may also complete an optional primary research project using academic versions of our ACA, CBC or CVA software systems. Academic versions for teaching purposes are available for just $300 for the entire school. Please visit our new academic area at www.sawtoothsoftware.com/academic.shtml. Perspectives on Recent Debate over Conjoint Analysis and Modeling Preferences with ACAThis article is contributed by Bryan Orme, Vice President at Sawtooth Software.A Controversial Article Inspires Debate In a recent article in Marketing Research, "What's Wrong with Conjoint Analysis," Larry Gibson argued that conjoint analysis is not needed given the simplicity and predictive accuracy of self-explicated methods. He wrote that conjoint analysis cannot handle the number of factors that self-explicated methods can and that clients usually need to study. This argument may be valid for full-profile conjoint, but it largely ignores the strengths of the more flexible hybrid methods, like ACA. Regarding hybrid methods, he expressed that self-explicated methods produce valid enough data that subsequent conjoint tasks are not justified. It is important to note that Gibson's firm (Eric Marder & Associates) has developed a unique method of self-explicated scaling called "SUMM," (see a description of the technique in the useful book The Laws of Choice by Eric Marder). In SUMM, respondents record their preference for each level of each attribute using an "unbounded scale." Rather than using a predefined numeric scale, respondents write "L" letters to indicate that they like a level (as many "L" letters as they wish--the more used, the greater the preference). Respondents write "D" symbols next to levels they dislike (as many "D" letters as they wish--the more used the more disliked the feature). A "N" letter is used to signify neutral. "Utilities" are developed by scoring +1 for every "L" and -1 for every "D." The technique, as described, employs paper-and-pencil surveys. Gibson's position swims against the prevailing tide of practice and opinion in our industry--which by itself doesn't mean he is wrong. His article clearly struck a nerve, as a number of researchers published responses in defense of conjoint analysis in the subsequent issue of Marketing Research (Paul Green & Abba Krieger, Rich Johnson, and Dick McCullough). These authors provided evidence regarding the value of full-profile methods for small conjoint problems, and asking conjoint questions in addition to collecting self-explicated ratings data for larger problems. I suspect Gibson knew his article would spark controversy, and tailored his prose for healthy effect. I don't fault him for stirring the pot, but I am concerned about some serious errors of fact and logic in his article. In his defense, his self-explicated technique is different (and perhaps superior among some populations, for certain product categories, and in paper-and-pencil mode) from the one the responding authors used for comparisons. Until more comparative work is done with his SUMM method, the argument cannot be resolved by referring to current research. But future research cannot validate or disprove the use of SUMM for all situations. There are many useful techniques for modeling preferences, and no one technique dominates the others. We should not dismiss his claim that SUMM seems to work better for him than the conjoint techniques he has tried. But it is misleading to imply that a single method, such as SUMM, generally dominates conjoint and choice analysis. I wonder how the SUMM technique might be used to model interaction effects, alternative-specific effects (designs where some attributes only apply to certain brands), conditional pricing, or cross-effects, which are important for many choice studies. I also wonder whether SUMM can work well for computerized interviewing. Despite Gibson's recent condemnation of conjoint analysis, when smaller problems involving, say, brand, package and price are being studied, Gibson's firm uses a technique they call STEP. STEP is really just a single full-profile CBC (discrete choice) task, featuring an allocation-based response. Respondents allocate a fixed number (say 10) of stickers across alternative products within a set. Gibson misleads his readers by seeming to dismiss the use of conjoint analysis when his firm often uses this CBC-like approach. But Gibson and his colleagues are quick to distinguish STEP from CBC. They argue that only one choice task should ever be asked of a respondent (per product category being studied) as subsequent tasks result in biased information. Indeed, if money were no object, one could achieve good results using very large sample sizes, with each respondent receiving one task. However, Rich Johnson and I showed in a 1996 paper entitled "How Many Tasks Should You Ask in CBC Studies" (available at www.sawtoothsoftware.com in the Technical Papers library) that the responses to subsequent tasks provide similar information, with a moderate but predictable bias. Brand becomes less important in later tasks, and price becomes more important. I would argue that the responses to the first CBC task might reflect what happens the first time a price change occurs (buyers may not notice and might stay loyal to their favored brand). But, later tasks may, in some cases, better reflect long-range equilibrium share. But what makes asking respondents additional tasks so valuable is that it enables us, under HB, to estimate individual-level utilities. Individual-level utilities resolve many IIA problems that plague aggregate-level modeling. I haven't used the self-explicated SUMM technique, but it is likely to lead to greater discrimination among most and least important attributes than the basic rating-scale method used in ACA. Green correctly noted in his article "What's Right with Conjoint Analysis?" that respondents do not discriminate enough among the most and least important attributes when using self-explicated importance rating scales. In practice, I often observe importance ratings from the self-explicated section of ACA that show on average a 2:1 ratio between two attributes' importances. But when the conjoint information is analyzed separately, derived importances often suggest a 3:1 ratio or even greater. Personal Perspectives on ACA In the nine years of fairly intensive experience that I have with ACA, I admit to having had a hot and cold relationship with the technique. Upon first seeing ACA, I was enthralled. Later, I became disenchanted when I observed how unimportant attributes seemed to be ascribed too much importance, and the importance of critical attributes seemed to be damped. ACA's pricing information was also disappointing, with price receiving too little importance. ACA simulators would often over-predict preference for expensive, feature-rich products. Another concern revolved around the assumption of linear utility increments for a priori or ranked attributes within the priors. Jon Pinnell helped me overcome my dislike for ACA in pricing problems, by documenting both the bias and a solution (that he and prior researchers had been using) called "dual conjoint" (see "Multistage Methods for Measuring Price Sensitivity"). Dual conjoint requires that respondents complete some full-profile (or near-full profile) conjoint tasks in addition to ACA. The price weight from ACA can then be adjusted to fit the additional data. For conjoint problems involving more than six or so attributes, this solution often works reasonably well. Peter Williams later published a paper at our conference entitled "Calibrating Price in ACA: The ACA Price Effect and How to Manage It" that uses fixed holdout choice tasks rather than a separate full-profile experiment. Williams didn't invent this approach, but he was the first to formally present it at our conference. The holdout approach is easier to implement, and I believe it is more commonly used than the dual conjoint approach. Both of these papers are available at www.sawtoothsoftware.com in the Technical Papers library. Enter Hierarchical Bayes Even after overcoming the pricing hurdle, I remained concerned that ACA "flattened" attribute importances. About three years ago, I pondered whether something could be done so that the importances from the self-explicated section in ACA might provide greater discrimination. Reading about the "unbounded scale" approach in SUMM further stimulated my thinking in this direction. Within the next few months, Rich Johnson completed the ACA/HB product for hierarchical Bayes estimation of ACA. I marveled at how well ACA/HB improved ACA modeling. After this breakthrough, I realized that HB provided an excellent solution and discarded ambitions to search for a modified priors question. Let me discuss how ACA/HB largely dispensed with most of my lingering concerns. The traditional ACA estimation combines the metric information from the priors and pairs section using OLS. If the priors importance weights are too flat, then the resulting final utilities are also biased toward flatness. Moreover, if attributes are ranked or specified as a priori, ACA assumes equidistant utilities within the priors, again biasing the final utilities toward this assumption. In contrast, the default estimation technique for ACA/HB uses the priors information to enforce ordinal constraints only. The metric information, which determines the importance of attributes and scaling of the levels within each attribute, is estimated solely from the conjoint pairs. ACA/HB usually leads to greater discrimination in the importance of attributes than when using OLS estimation. It leads to better prediction of holdout choices for all but two data sets I know of. Even after using ACA/HB, you will usually see that ACA importances still show less discrimination than those derived from full-profile methods--especially CBC. Why does this happen? Respondents use simplification strategies to deal with full-profile tasks that include many attributes. When faced with so much information, they focus on just the top few features. I suspect that ACA's flatter importances may be quite predictive of actual purchases for many high involvement product categories wherein which buyers make careful decisions. I'm surprised that so many ACA users have not yet embraced ACA/HB. Yes, it is a complex statistical procedure--but it is also a very simple product to use. By using the defaults, it is as simple as pressing a few keystrokes and then waiting for the run to finish. With today's fast 1 GHz+ speed computers, it usually takes 30 minutes to an hour per utility run for a typical data set with 500 respondents. Let me discuss a recent project we were involved in to illustrate HB's value. Why I Prefer HB Estimation: An Example Our research wing, Sawtooth Analytics, was contracted by a recent ACA buyer to assist with a pricing project. The study involved 11 attributes, most of them being binary (on/off) features. We suggested ACA, with follow-up full-profile CBC-like holdout questions. Each respondent received three holdout tasks, and each holdout featured a selection from three possible products: a cheap one (with few features), a mid-priced product, and an expensive feature-rich product. We used the regular OLS utilities to predict choices to the holdouts and saw the typical ACA pricing problem: too much share (by a significant margin) for the most expensive products in each set. We coached our ACA client regarding the adjustment for price (multiply the price part worths by a factor greater than unity, and import the new results back into the market simulator). After adjusting price by about a factor of 4, the fit was much improved for the least and most expensive products, but predictions were dismal for the medium-priced product. The errors in prediction weren't due to just price bias. The average MAE (mean absolute error) for prediction of all products across all three holdout sets was over 12. Those familiar with this type of analysis will recognize that this is a poor fit. An MAE of 12 means, for example, that the target share is 30, but we predict 42. Next, we used ACA/HB to reanalyze the ACA data. Many of the features within this product were quite new to respondents, and we suspected that they may not have been able to settle on a reliable scaling of preferences and importance early on in the priors section. To test this assertion, we used ACA/HB to estimate two separate sets of utilities. In the first set, we fit the pairs information after applying the priors information as ordinal constraints. For the second set, we estimated utilities only using the information from the conjoint pairs. Comparing the two sets of utilities, we noted a large discrepancy in the importance of many of the features--particularly price. We used both sets of ACA/HB utilities to predict the holdouts, again adjusting price for best fit. The ACA/HB run that fit only the pairs information provided the best fit (not a typical finding, given our experience with many data sets). These utilities required a price weight of less than 2, and the resulting MAE was around 4 (significantly better than the MAE of 12 when using the default ACA OLS utilities). We realized that we were discarding information by ignoring the priors data, leading to lower within-respondent precision of estimates. But we believed the priors information in this case damaged predictive accuracy. The sample size was a healthy 1000 (which meant that the added error would have minimal effect on the precision of aggregated share predictions). Since the client was most concerned about achieving an accurate market simulator, it seemed clear that using HB with the conjoint pairs information only was the right choice. In short, having ACA/HB makes one a much more confident and competent ACA researcher. The research is much more defensible, because it applies a leading-edge statistical technique with a more correct method for combining priors and pairs information. The researcher has greater options for building a model that is more predictive of real-world behavior. Under OLS, there is one default approach, and the typical user can do little but accept it. In this most recent experience, by using ACA/HB we were able to identify a disconnect between the priors and pairs information. We ended up discarding the priors. Through HB's ability to stabilize estimates for individuals by borrowing information from the population parameters, we estimated more realistic utilities than under traditional ACA estimation and delivered a more predictive market simulator. Suggestions and Conclusion Despite my improved outlook for ACA, I wouldn't use it exclusively. I am suspicious of claims that there is one best technique for all (or even most) preference modeling situations. For studies involving about six or fewer attributes and sample sizes greater than 100, I generally first consider CBC. When sample sizes become quite small, I often favor traditional full-profile conjoint (with HB estimation under HB-Reg). For problems involving more attributes than is reasonable for full-profile methods, I tend to favor ACA--paying special attention to adjust for the price bias, and using ACA/HB to try different ways of combining the priors and pairs data. Even after choosing the approach, there are numerous issues to be resolved regarding the formulation of attribute levels, choice of response scales, allocation versus discrete choice, full- versus partial-profile, number of alternatives per task, number of pairs/tasks. These decisions are often vexing, but it is the challenge that makes the process all the more enjoyable. If there was just a single easy and best approach for all situations, our profession wouldn't be nearly as fulfilling and managerially useful.
|
|
|
About Us | Contact Us | Privacy | FAQ's | Forum |
|
© 2008 Sawtooth Software, Inc. All rights reserved. |