|
||||||
Technical Papers| Home |
Solutions |
Products |
Services |
Education |
Support |
| | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|   SS Summer 1998Computerized Full-Profile Conjoint Analysis: Pairwise vs. Single-Concept PresentationThis article is adapted from an article entitled "Conducting Full-Profile Conjoint Analysis over the Internet" scheduled to be published in the July issue of Quirk's Marketing Research Review.If you've visited our web site lately, chances are you've seen the experimental conjoint study we've been conducting over the Internet. The subject of the study was credit cards, and its purpose was to compare pairwise full-profile (FP) conjoint and single-concept presentation. Both types of questionnaires can be designed and analyzed using our CVA system. The conclusions we've drawn apply to all computerized FP studies, whether over the Internet, DBM or CAPI. In a 1997 survey of conjoint analysis usage in the marketing research industry, ACA (Adaptive Conjoint Analysis) was found to be the most widely used conjoint methodology in both the US and Europe. Traditional FP conjoint was also reported as a popular method. In general, we believe traditional FP conjoint is an excellent approach when the number of attributes is around six or fewer, while ACA is generally preferred for larger problems. FP conjoint analysis studies can be done either as paper-based or as computerized surveys (Internet surveys, disk-by-mail, or CAPI). Because they typically involve fixed designs and, unlike ACA, are not adaptive, computerized FP surveys really offer no real benefit over the paper-based approach in terms of the reliability or validity of the results. In fact, paper-based FP may work better than computerized FP. Even though computerized FP probably offers no significant benefit over paper-based surveys in terms of reliability or validity, real benefits might be realized in survey development, data collection costs, and speed. Pairwise and Single-Concept presentation are two popular approaches for FP conjoint. With Pairwise questions, respondents make comparative judgements regarding the relative acceptability of competing products. The Single-Concept approach probes the acceptability of a product, and de-emphasizes the competitive context. Both methods have proven to work well in practice, but we are unaware of any study other than this one that has directly compared these two approaches. Details of Experiment We designed an Internet survey to compare the Pairwise and Single-Concept approach for computerized FP conjoint analysis. The subject for our study was credit cards, with the following attribute levels:
Respondents completed both Pairwise and Single-Concept conjoint questions (in rotated order). Additionally, holdout choice sets were administered both before and after the traditional conjoint questions. A total of 280 respondents completed the survey. Respondents self-selected themselves for the survey, which was launched from a hyperlink on Sawtooth Software's home page. This sampling strategy is admittedly poor had we been interested in collecting a representative sample. But the purpose of our study was not to achieve outwardly projectable results, but rather to compare the within-respondent reliability of alternative approaches to asking FP computerized conjoint. Measuring the Reliability of Conjoint Methods Reliability and validity are two terms often used to characterize response scales or measurement methods. Reliability refers to getting a consistent result in repeated trials. Validity refers to achieving an accurate or "true" prediction. Our study focuses only on issues of reliability. Holdout conjoint (or choice) tasks are a common way to measure reliability in conjoint studies. We call them "holdout" tasks because we don't use them for estimating utilities. We use holdouts to check how well conjoint utilities can predict answers to observations not used in utility estimation. If we ask some of the holdout tasks twice (at different points in the interview), we also gain a measure test-retest reliability. We included a total of three repeated holdout choice questions in our Internet survey. These displayed three credit cards and asked respondents to choose the one they would most likely sign up for. Respondents on average answered these holdouts the same way 83.0% of the time. This test-retest reliability is in line with those reported for other methodological studies we've seen that were not collected over the Internet. But one can argue that our respondents (marketing and market research professionals) were a well-educated and careful group. We cannot conclude from our study that Internet interviewing is as reliable as other methods of data collection. We use the holdout choice tasks to test the reliability of our conjoint utilities. We would hope that the conjoint utilities can accurately predict answers to the holdout questions. We call the percent of correct predictions the holdout hit rate. Some have referred to hit rates as a validity measurement, but prediction of holdout concepts asked in the same conjoint interview probably say more about reliability than validity. The holdout hit rates for the Pairwise and Single-Concept approach were 79.3% and 79.7%, respectively. This is a virtual tie; the difference is not statistically significant. These findings suggest that both methods perform equally well in predicting holdout choice sets. Qualitative Evidence In addition to completing conjoint tasks, we asked for qualitative evaluations of the Pairwise versus the Single-Concept approach. Respondents perceived that the Pairwise questions took only 13% longer than the Singles. We asked a battery of questions such as whether respondents felt the conjoint questions were enjoyable, easy, frustrating, or whether the questions asked about too many features at once. We found no significant differences between any of the qualitative dimensions for Pairwise vs. Single-Concept presentation. Conjoint Importances and Utilities We calculated attribute importances in the standard way, by percentaging the differences between the best and worst levels for each attribute. Conjoint importances describe how much impact each attribute has on the purchase decision, given the range of levels we specified for the attributes. Importances and utilities for Pairs vs. Single-Concept presentation were as follows:
Conjoint Utilities
The only significant difference for either conjoint importances or utilities between the two full-profile methods occurred in the utility for the middle level of annual fee ($20). In a presentation at our 1997 Sawtooth Software Conference, Joel Huber of Duke University argued that respondents may adopt different response strategies for sets of products versus Single-Concept presentation. He argued that when faced with comparisons, respondents may simplify the task by avoiding products with particularly bad levels of attributes. Annual fee was the most important attribute. The larger gap between the worst and middle level (44-0) for Pairs versus Single-Concept (34-0) is statistically significant at the 99% confidence level (t=3.93) and supports Huber's "undesirable levels avoidance" hypothesis. Conclusion
Our data tell a comforting story, suggesting that both computerized
Pairwise and Single-Concept FP ratings-based conjoint are equally
reliable and result in the same importances and roughly the same
utilities. Computerized FP conjoint seems to have worked well for
a small design such as our credit card study. Given that the researcher
has determined that the Internet is an appropriate vehicle for
interviewing a given population, our findings suggest that FP
conjoint can be successfully implemented via the Internet for a
small study including four attributes.
The Benefits of Accounting for Respondent Heterogeneity in Choice ModelingMarketers have long recognized that people are unique. In market research, heterogeneity most often refers to differences in preference from person to person.If you attended our 1997 Conference, you heard a number of talks addressing the benefits of capturing heterogeneity in CBC analysis, such as more accurate simulation models and better understanding of market structure. You also heard the term IIA (Independence of Irrelevant Alternatives) often referred to as the "Red Bus/Blue Bus Problem." This article will demonstrate that IIA is much more problematic with aggregate logit and much less a problem with methods that recognize respondent heterogeneity, such as Latent Class and ICE (Individual Choice Estimation). The "Red Bus/Blue Bus" Problem In the classic "Red Bus/Blue Bus" example, different modes of transportation are available. Assume the following aggregate level utilities for a given set of individuals: Aggregate Logit Utilities Car 0.61 Bus -0.28 Bicycle -0.33 Red 0.01 Blue -0.01Respondents on average prefer cars to buses or bicycles, and the color of the vehicle is virtually unimportant. Simulated shares of choice for three different modes of transportation are given below: Aggregate Logit Simulations: Scenario 1
Car 55.4
Red Bus 23.0
Bicycle 21.6
-------
100.0
(If you would like to review how shares are calculated using logit utilities,
please refer to pages 4-24 and 4-25 of the CBC manual.) Suppose the bus
company wanted to increase traffic on its routes by repainting half of its
fleet blue. We would expect that this move shouldn't significantly
increase bus ridership, but the aggregate logit simulation suggests
differently:Aggregate Logit Simulations: Scenario 2
Car 45.1
Red Bus 18.8
Blue Bus 18.4 (Net Bus = 37.2%)
Bicycle 17.7
-------
100.0
The net bus ridership has increased from 23.0% to 37.2% (a 62% relative
increase), which is not logical. Under aggregate logit simulations, IIA
typically results in unrealistic share inflation for very similar (or
identical) products.If we analyze the same data set with Lclass analysis, we get much better results. Lclass detects three segments of individuals with quite different preferences for transportation. The segment sizes and utilities for each group are listed below: Lclass Utilities Segment: 1 2 3 Group Size: 16.7% 16.7% 66.6% Car -5.9 -1.5 1.5 Bus 2.3 1.4 -0.8 Bicycle 3.6 0.1 -0.8 Red -0.05 0.07 0.03 Blue 0.05 -0.07 -0.03Notice that the Lclass utilities are larger in magnitude than the aggregate logit utilities. With logit and lclass analysis, the better the data fit the responses to choice tasks, the greater the absolute values of the utilities. The Lclass utilities fit the choices much better within each latent class (segment) than a single average set of utilities fits the entire sample. Let's again simulate results for Scenario 1, this time using the Latent Class utilities. Latent Class Simulations: Scenario 1
Segment 1 2 3 Total
Group Size: 16.7% 16.7% 66.6% 100.0%
Car 0.0 3.9 83.1 56.0
Red Bus 20.6 76.6 8.6 21.9
Bicycle 79.4 19.5 8.3 22.0
------- ------- ------- -------
100.0 100.0 100.0 100.0
With Lclass, not only do we get an overall share of choice for the entire
sample, but we learn about the composition of the market. Segment 1
favors bicycles; Segment 2 favors buses. The largest segment, Segment 3,
favors cars. For this simulation which features dissimilar products, the
average (weighted) shares across all segments closely match the shares
computed under aggregate logit. With respect to Scenario 1, other than
better understanding the composition of the market, Lclass hasn't
significantly changed the overall simulation results.Now let's add blue bus to the simulation. Latent Class Simulations: Scenario 2
Segment 1 2 3 Total
Group Size: 16.7% 16.7% 66.6% 100.0%
Car 0.0 2.4 76.9 51.6
Red Bus 16.8 45.9 7.9 15.8
Blue Bus 18.5 40.0 7.5 14.7 (Net Bus = 30.5%)
Bicycle 64.7 11.7 7.7 17.9
------- ------- ------- -------
100.0 100.0 100.0 100.0
Net share to buses has again increased, but instead of a 62% relative share
increase as with aggregate logit, under Lclass the net share for buses
increases from 21.9% to 30.5%, representing a 39% increase. Accounting
for heterogeneity has reduced the Red Bus/Blue Bus problem. Where's the magic? Most of the preference for buses comes from Segment 2. In Scenario 1, 76.7% of the share from Segment 2 was cast to the Red Bus. When we add a Blue Bus, there is really very little room for share inflation within that segment since buses already capture nearly all the share. Imagine that instead of accounting for heterogeneity by fitting utilities to three (or even ten) separate groups, we fit a set of utilities for each respondent. ICE, our newest add-on module for CBC, accomplishes just that. The result is a model that is even better than Lclass at handling the classic Red Bus/Blue Bus simulation problem (and generally superior in terms of holdout predictability). In a real data set we recently collected, the relative share inflation for duplicated products in simulations was 54%, 28% and 7% under aggregate logit, Lclass and ICE, respectively. Even though careful marketing research analysts would never run simulations as unrealistic and methodologically flawed as the Red Bus/Blue Bus situation, most CBC simulations usually involve at least some degree of differential similarity among products. These situations can benefit from recognizing heterogeneity using Lclass and ICE. Cross-elasticities Another weakness of aggregate-level logit is its inability to account for cross-elasticities with the standard main-effects or main-effects-plus-interactions models available in CBC. (One can model cross-elasticities with aggregate-level logit, but this requires specialized modeling expertise and software other than CBC.) Cross-elasticity is defined as the relative percent change in quantity demanded of brand A resulting from a percent change in price of brand B. With aggregate-level logit, when a brand lowers its price, it steals share from other brands in proportion to the other brands' shares. In other words, the cross-elasticities are held constant. If we account for respondent heterogeneity (with Lclass or ICE), some degree of cross-elasticity can be captured and modeled. Assume two Lclass segments with the following utilities: Lclass Utilities Segment: 1 2 Group Size: 50.0% 50.0% Cola A 0.5 -0.5 Cola B 0.5 -0.5 Diet Cola C -0.5 0.5 Diet Cola D -0.5 0.5 Low Price 1.0 1.0 Medium Price 0.0 0.0 High Price -1.0 -1.0Cola A and Cola B are preferred (and highly substitutable) by Segment 1, while diet colas C and D are preferred (and highly substitutable) by Segment 2. Such situations often occur in the real world when competing brands are similarly positioned and appeal to unique segments. To simplify our example, we've made the segments equally sensitive to price. Scenario 1 shows the simulated shares at the average price: Scenario 1: All Brands at Medium Price
Segment: 1 2 Total
Group Size: 50.0% 50.0% 100.0%
Cola A, Medium Price 36.6 13.4 25.0
Cola B, Medium Price 36.6 13.4 25.0
Diet Cola C, Medium Price 13.4 36.6 25.0
Diet Cola D, Medium Price 13.4 36.6 25.0
------- ------- -------
100.0 100.0 100.0
In Scenario 2, Cola A lowers its price:Scenario 2: Cola A Lowers its Price
Segment: 1 2 Total
Group Size: 50.0% 50.0% 100.0%
Cola A, Low Price 61.0 29.7 45.4
Cola B, Medium Price 22.5 10.9 16.7
Diet Cola C, Medium Price 8.3 29.7 19.0
Diet Cola D, Medium Price 8.3 29.7 19.0
------- ------- -------
100.0 100.0 100.0
When Cola A lowers its price, the diet colas each lose (25% - 19% = 6%)
share points. Cola B, however, loses a larger amount (25% - 16.7% = 8.3%)
share points. Capturing respondent differences in this example has resulted
in detecting differential cross-elasticities. Without accounting for the
differences between these two segments, simulations would not reveal
differential substitutability between the brands.Interactions One of the benefits of aggregate-level logit is the ability to model interactions, such as interactions between brand and price. However, if interactions result from differences in preferences between segments, these can also be captured by recognizing heterogeneity with Lclass or ICE without having to include interaction terms. We won't take the space in this article to give a numeric example. Rather, assume a market with two styles of women's dress-pants: a straight leg and a wide leg. Further assume that the straight leg pant tends to appeal to the more price-sensitive segment, and the wide leg appeals to the less price-sensitive segment. If we recognize market heterogeneity when computing utilities, sensitivity simulations can naturally account for an interaction between price and pant styles using only main effects utilities (no interaction terms). Share for the wide leg pants should be less sensitive to price changes than the straight leg. Summary and Conclusion Aggregate-level logit has been faulted for its IIA properties. Specifically, aggregate logit modeling can fail when products with differing degrees of similarity are included in simulations. Corrections for product similarity (such as Sawtooth Software's Model 3) can help in such situations, but it is best to begin with an underlying model that is less susceptible to the "Red Bus/Blue Bus" problem. Using Lclass can reduce the need to correct for product similarities under Model 3. For a recent data set we've studied, developing individual-level utilities using ICE near fully accounted and corrected for product similarities. Aggregate logit also cannot account for differential cross-elasticities without customized modeling beyond the standard capabilities of Sawtooth Software's CBC System. Lclass or ICE can account for differential substitutability if such relationships are accounted for by differences among underlying segments or individuals.
CBC has been praised for its ability to detect interactions by pooling
data across respondents. To the degree that interactions can be accounted
for by differences among segments or individuals, models that recognize
heterogeneity can reflect interactions with main-effects only models.
If directly modeling interactions and capturing heterogeneity are both
of concern, our Lclass Module can be an excellent approach.
Conjoint Internet Module AvailableOver the last few years, we've received numerous requests for conjoint software that can create interviews for the Internet. Our CVA users recently received the announcement regarding our newest software product: the CVA Internet Module. This is a Windows-based add-on to our popular CVA package that gives you everything you need to create and host your own Internet/Intranet full-profile conjoint interview.The CVA Internet Module leverages the advantages of the Internet and web browsers to create attractive interviews. Plus, you can add up to 50 additional standard survey questions such as numerics, selects (single or multi) and open-ends to create a complete survey instrument. The CVA Internet Module has three main components: Passwords Module, Write Questionnaire Module, and the On-line Data Management Module. 1. Passwords Module. The imposter/repeater problem is one of the most oft-cited criticisms of Internet research. It is critical to restrict access to your survey. The Passwords Module lets you assign up to 9,999 different passwords. (If you want, you can also turn password verification off.) Assigning a unique password for each respondent keeps respondents from taking the interview more than once. It also lets them interrupt a survey and resume it at a later session, without losing all of their data. 2. Write Questionnaire Module. The CVA Internet Module is a Windows-based system that lets you compose survey questions quickly and easily. No programming is involved. You simply click and choose options and type text into fields. The CVA Internet Module automatically reads your conjoint design and attribute labels directly from the base CVA System. You can ask a single question per "page" or group questions together. Intuitive dialogs help you compose and format conjoint and other standard survey questions. You can add up to 50 additional survey questions to your full-profile conjoint questionnaire. The following question types are supported:
3.On-line Data Management Module. When you conduct an Internet/Intranet survey, all of the data reside in the same directory on the server. This makes it easy to perform many study/data management tasks from any computer connected to the Internet. You can view data, download, or run simple marginals in real-time, while your survey is still running. You can also check password quota cells and adjust quotas. Two levels of permission are available with the Data Management Module: read and read/write. You may wish to give your client read capabilities so that he/she can view results at any time, without the ability to do anything that could harm the study.
You can view an on-line demonstration of the CVA Internet Module at:
www.sawtoothsoftware.com/products/cva/. We've also written a new
article that is available for downloading from our technical papers
library on the web entitled: "Conducting Full-Profile Conjoint Analysis
over the Internet."
1999 Research Conference ReminderWe hope by now that you have heard about our next Sawtooth Software Conference to be held February 2-5 in San Diego, California. This will be a top quality event held in the San Diego–La Jolla Marriott Hotel. The program is starting to take focus, and we are already very pleased with the amount of new information, new techniques and practical guidance that is coming forward.Attendees to our conferences have commented on the forum's relevance and applicability. Our conference tends to be more practical and down-to-earth than many market research conferences today. This is a 3 ½ day event. The first day is dedicated to tutorials, and the remaining 2 ½ days are set aside for presentations (roughly 30 minutes each) and follow-up discussion. By the way, if you came last year, you know that the food will be incredible! Please mark February 2-5 on your calendar. We'll be sending out conference registration information soon. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Technical Papers
|
About Us | Contact Us | Privacy | FAQ's | Forum |
|
© 2008 Sawtooth Software, Inc. All rights reserved. |