|
||||||
Technical Papers| Home |
Solutions |
Products |
Services |
Education |
Support |
| | |||||||||||||||||
|   SS Summer 2005SSI Web v5.2 Even Better with Quota ControlWe are working hard to make SSI Web your software of choice for both general web interviewing and for projects that include conjoint sections. Many of our conjoint analysis customers have been pleasantly surprised with the capabilities of our general interviewing component, CiW. They’ve been able to save money and time by using CiW.
Back in October, we released SSI Web v5, which was a major improvement in our web interviewing platform. We had originally planned for Quota Control to be part of that release, but were unable to include it then. In May, we released v5.2 (a free update for v5 users) with Quota Control functionality. The desire for Quota Control can be economic: respondents completing the survey are paid a cash incentive, and the researcher does not want to go over-budget. Quota Control is also used to achieve a sample reflecting known characteristics of a population: representative proportions of male, female, low income, high income, etc. Using Quota Control can also help us conserve our relationship with respondents: we limit the amount of time we ask of respondents to a few screener questions if we do not need their completed record. Consider a B-to-B study, wherein the researcher wants to interview 1000 total respondents, with a certain number of completes within different company sizes (large, small) and types (manufacturing, services):
A respondent starts the survey and is first asked what size and type company she works for. These are called screener questions. After screener questions are asked, a determination is made (at the Qualification Point) whether this respondent is needed. If this respondent's Quota Cell is already filled with enough completed records, then she is immediately skipped to the end and disqualified. Otherwise, she is allowed to continue the survey. This process is represented by the diagram below:
 Predictive Quota Control SSI Web has more sophisticated rules for determining if the Quota Cell Limit has been or is expected to be reached. You can specify that SSI Web should assume that, say, 80% of the respondents who have passed the Qualification Point and are actively taking the survey will eventually finish (i.e. a 20% drop-out rate). You can also specify that respondents who haven't completed the survey within a certain amount of time should be marked as Inactive. If these options are in place, when a new respondent reaches the Qualification Point, SSI Web considers the number of respondents in this respondent's quota cell already completed and actively in process (discounted by 20%) to decide if the new respondent is needed. We refer to this ability to more intelligently determine if more respondents are needed within a Quota Cell as Predictive Quota Control. Predictive quota control can certainly limit the amount of overage, but it still cannot ensure that quota cell limits are not exceeded. Download a Free Demo Version Today
Download a functioning demo version at www.sawtoothsoftware.com/ssiweb.shtml.
Call for Papers: Sawtooth Software Conference 2006On March 29-31, we will hold our twelfth Sawtooth Software Conference in Delray Beach, Florida. Our research conference brings together some of the best minds in our industry to talk about practical issues in online interviewing and quantitative market research. It is not a sales-oriented event for our software, but a chance to exchange ideas and receive education from a variety of sources and perspectives. Papers presented at our previous Sawtooth Software Conferences are cited frequently in journal articles.We're looking for exceptionally strong papers. If you'd like to be on the program, please respond promptly (by September 1, email: bryan@sawtoothsoftware.com) with a one-page abstract describing your proposed paper, with special attention to the findings and what the audience will "take away" from the presentation. You must also include a 50-word description of your paper to include in the conference brochure, should your abstract be accepted. We are interested in papers on a variety of subjects, including Web interviewing, market segmentation, scale development, customer satisfaction modeling, conjoint/choice analysis, MaxDiff, perceptual mapping, hierarchical Bayes methods, forecasting, pricing research, market simulations, and case studies. These papers need not involve Sawtooth Software's programs or approach. In an effort to provide more balance to the program, we are encouraging papers that are not about conjoint/choice modeling. For all topics, we are eager to see evidence of managerial relevance, external validity, profit impact, etc. Presenters receive a complimentary conference registration. To be accepted, a paper must show promise of being sufficiently practical to be of use to the least sophisticated members of the audience, while having enough substance to be of interest to the most sophisticated members. In addition to standard presentation slides, authors are required to submit a journal-quality written paper for publication in the Conference Proceedings. We strive for the highest quality in our conferences. If your abstract is accepted, a member of the steering committee will review early drafts of your presentation and offer suggestions. Authors are expected to consider these suggestions conscientiously and rework their presentations as needed. Sawtooth Software reserves the right to remove any author from the program or proceedings that fails to meet deadlines or produce high quality work. Sawtooth Software Conference 2006 Steering Committee Members are:
 Go back to Index2005 Customer Feedback ResultsFor the third year running, we fielded a customer feedback survey. A very sincere thanks to all who participated. The quality and quantity of openend responses were tremendous! We are reviewing the comments, and hope to be able to implement many new features and changes based on the suggestions.“Longer support hours!” was one of the most common themes—especially from our east coast customers in the US. When we moved to Washington State 10 years ago, this put us at a time zone disadvantage with respect to the east coast. Based on a recent hire of another customer support consultant, we will open the office to business and technical support 1.5 hours earlier than before. Starting August 1 our new hours will be 7AM-5PM, Pacific. Here are a few interesting findings from the 2005 Customer Feedback Survey. Overall Impression The overall impression of our company is high, and improving. In 2003, 92% of respondents rated the overall quality of interactions with us as Excellent or Good. In 2004 and 2005, 93% and 94% said the same. We’ve plotted the top box (Excellent—FAR exceeded what I normally expect from a software company) percentage below:
 A key indicator for success is the quality of the help provided. In 2005, 98% of respondents who had an interaction with us either Strongly or Somewhat Agreed that they were confident the support/advice they received was correct. We’ve plotted top box results below:
 For three years running, we’ve tracked the relative use of Sawtooth Software conjoint methods. In 2005, the momentum continues for increased use of CBC. CBC accounted for 54% of all conjoint/tradeoff projects conducted by customers responding to the survey over the last 12 months. At Sawtooth Software, we’ve been interested in the relative use of our three conjoint packages. The relative proportions are shown below, weighted by projects conducted.
 For the first time, we included Best/Worst (MaxDiff) scaling as a response option in the conjoint/tradeoff method usage question. (Although MaxDiff scaling is not in the formal sense a conjoint methodology.) 12% of respondents that conducted preference modeling employed Best/Worst during the year, and it accounted for just under 4% of the total preference modeling projects conducted. We currently offer a Best/Worst Experiment Designer, and CBC/HB or Latent Class software for parameter estimation for MaxDiff. However, we haven’t yet integrated a complete MaxDiff solution within SSI Web yet. We hope to offer that soon! Adaptive CBC: Summary of Three ExperimentsThis article is excerpted from an article by Rich Johnson, Bryan Orme, Joel Huber, and Jon Pinnell entitled “Testing Adaptive Choice-Based Conjoint Designs” presented at the 2005 Design and Innovations Conference, held in Berlin. The full text may be downloaded from our Technical Papers library at www.sawtoothsoftware.com.Choice-Based Conjoint analysis (CBC) has achieved a dramatic increase in use during the past decade. One important reason for the growth of CBC is that choices are more like actual marketplace behavior than are the rankings or ratings used by other conjoint methods. Another reason is that hierarchical Bayes (HB) methods now permit estimation of partworths for individuals, where previously it had seldom been possible to collect enough choices from each individual to support individual-level analysis. However, despite the important contribution of HB, there remains considerable incentive to make choice designs more efficient. Huber & Zwerina (1996) showed that efficient choice designs have four characteristics: orthogonality, level balance, minimal overlap, and utility balance. The first three of these are present in designs provided by Sawtooth Software’s CBC System, but utility balance is not. Various combinations of authors have been involved in three (2003, 2004, and 2005) recent Sawtooth Software experiments to test an algorithm for adaptive CBC (ACBC) which accounts for individual-level utilities when designing choice tasks. An Algorithm for Adaptive CBC: In a good design the estimation error for the parameters is as small as possible. The adaptive algorithm creates a unique design for each respondent, where the questions are chosen to maximize D-efficiency. Because partworths affect D-efficiency (utility balance yields greater statistical efficiency), preliminary estimates of partworths are needed for each respondent. ACBC begins with an estimate of the respondent’s partworths, obtained differently in the three experiments we conducted. The first choice task is always random, subject to requiring only minimum overlap among the attribute levels represented. After the first choice task has been completed, the information matrix for completed choice tasks is calculated. Alternatives for the next choice set are constructed to maximize D-efficiency, based on the preliminary partworth estimates. There were differences in the three experiments in how the initial estimate of partworths was obtained, whether estimated partworths were updated during the questionnaire, and in how the next task was chosen to improve design efficiency. (Those details are contained in the full written paper, but not in this summary.) The First Experiment: This study was done with approximately 1000 allergy sufferers who were members of Knowledge Networks’ web-based panel. The product category was antihistamines, described by 9 attributes having two or three levels, for a total of 23 levels. The respondents were randomly allocated to receive either standard CBC, or an Adaptive CBC questionnaire. Self-explicated questions similar to ACA’s “prior” section were used to estimate partworth utilities for each respondent for use in the adaptive CBC design algorithm. Success was measured by hit rates measuring accuracy of prediction of individual choices in holdout choice tasks, and MAEs (mean absolute errors) of share predictions for the holdout tasks. Hit rates for ACBC and CBC were nearly identical, but share predictions showed a directional improvement for ACBC. However, the experiment didn’t allow for testing significant differences between share prediction accuracy. The Second Experiment: The second study dealt with laptop computers, and was conducted with approximately 1000 members of AOL’s Opinion Place panel. The data were contributed by SPSS. Respondents were selected using a “river” methodology, recruited from a variety of popular Web portals. There were two differences from the first experiment that may have affected the results.
This inconsistency led the authors to search for possible causes. They calculated D-Efficiencies based on preliminary partworths, and found the ACBC designs to have average efficiencies nearly twice those of the CBC group. Thus it appeared that the adaptive algorithm behaved as expected. However, when computed using final partworth estimates, the efficiencies for the ACBC group were much smaller. For the Full Profile treatment the ACBC designs were only 9% better than the CBC designs. For Full Profile respondents, the average correlation between self-explicated estimates and final partworths was only 0.445. The authors concluded that the preliminary partworths had not been sufficiently accurate to be useful in guiding the adaptive algorithm. The authors concluded by observing that the adaptive algorithm appeared to have worked as expected, but that the preliminary partworths available to it were not effective. They suggested further research in which other means might be used to estimate preliminary partworths. The Third Experiment: The third experiment was done with approximately 450 respondents who were members of MarketVision Research’s Web panel. The product category was hotels, described by 9 attributes. An important difference between this experiment and the previous ones was the way preliminary partworths were estimated. Self-explicated questions were asked about desirabilities of levels within attributes so we might investigate the effect of constraining partworths to have desired rank orders, but those answers were not used to construct preliminary partworths. Instead, a hierarchical Bayes procedure was used to estimate partworths and to update them “on-the-fly” after each question. Although in this third study it appeared that hit rates for ACBC had a slight edge over CBC, that improvement turned out to be illusory. A covariance analysis was done to remove any spurious difference in hit rates due to the difference in test-retest reliabilities, and when the groups were equated on reliability the difference in hit rates disappeared. We are left with results for hit rates very similar to those for the first experiment: no significant difference between treatments. Results for share predictions showed slightly lower share prediction predictive accuracy for ACBC relative to CBC. ACBC again failed to demonstrate superiority over regular CBC. We believe that the algorithm used in this third study was superior to previous versions, and are disappointed that the results were not better. In the last section we consider possible reasons for its failure. Discussion: The ACBC algorithm was tested extensively with simulated respondents before the first study with human respondents. It had worked well in that artificial context. But simulated respondents provide fewer challenges. Accurate partworths for our simulated respondents were available at the beginning of the interview, simulated respondents did not change their preferences during the interview, and they made choices faithfully according to the logit model. We can think of several possible reasons for ACBC’s failure to perform as well with human respondents, some of which are listed below. Initial estimates of partworths not good enough. The adaptive algorithm requires preliminary estimates of partworths. The first study, where ACBC seemed to work well, collected self-explicated judgments of attribute importances and desirabilities of attribute levels in a way similar to that of ACA. Perhaps this provided better preliminary partworth estimates than either of the later two studies. The second study did not ask for attribute importances, and its preliminary partworths did not correlate well with final partworths. In the third study the “on-the-fly” partworth estimates became quite good for later tasks, but may not have been good enough during the early tasks. Respondents change their values during the interview. We know from other research (Johnson and Orme, 1996) that brand tends to become less important and price more so as interviews progress. Perhaps there are other changes as well, and partworths obtained from information available early in the interview may not be capable of leading to a design that is efficient for estimating partworths at the end of the interview. Respondents don't use logit models to make choices. We know respondents don’t make choices by summing partworths. There is ample evidence that they use various schemes to simplify the job of answering choice tasks in market research questionnaires. Despite this, the multinomial logit model has been generally successful in predicting respondent choices. For example, in the three studies reported here, hit rates were usually nearly as large as test-retest reliability percentages, indicating that the hit rates were nearly as good as possible. D-Efficiency may not be a good criterion to maximize. Although the logit model has many desirable properties and provides a useful approximation to respondent choice behavior, high D-Efficiencies may not translate into favorable hit rates and MAEs. Utility balance is desirable under the logit model, but for respondents who use response strategies different from the logit model, the other three design characteristics may be more critical. Although D-Efficient designs produce good estimation of logit parameters, those parameters may predict choice behavior less well than parameters developed from designs that maximize orthogonality, level balance, and minimal overlap. Designs produced by Sawtooth Software’s CBC System, to which we have been comparing ACBC designs, do precisely that. Perhaps designs which do not take account of partworths, such as those produced by regular CBC, have an advantage for predicting hit rates and choice shares in holdout concepts. In the meantime, we are impressed that standard CBC designs appear to be surprisingly robust, and regular CBC appears to be hard to beat. References Huber, Joel and Klaus Zwerina (1996), “The Importance of Utility Balance in Efficient Choice Designs,” Journal of Marketing Research, 33 (August) 307-317. Johnson, Richard M. and Bryan Orme (1996), “How Many Questions Should You Ask in Choice-Based Conjoint Studies?”, Available at http://sawtoothsoftware.com/technicaldownloads.shtml#howmany Johnson, Richard M., Joel Huber, and Lynd Bacon (2003), “Adaptive Choice-Based Conjoint,” Sawtooth Software Conference Proceedings. Johnson, Richard M., Joel Huber, and Bryan Orme (2004), “A Second Test of Adaptive Choice-Based Conjoint Analysis (The Surprising Robustness of Standard CBC Designs),” Sawtooth Software Conference Proceedings. CBC/Web Workshop
|
||||||||||||||||
|
About Us | Contact Us | Privacy | FAQ's | Forum |
|
© 2008 Sawtooth Software, Inc. All rights reserved. |