
	Technical Papers Pricing & Ordering Your Cart

Sawtooth Software » Education » Sawtooth Solutions

Education

Support

SS Spring 1997

1997 Sawtooth Software Conference: Speakers and Registration Information (SS Spring 97)

The sixth Sawtooth Software Conference will be held August 20-22, 1997 in Seattle, Washington. The conference will again focus on Computer Interviewing and Analytical Methods for Marketing Research.
Some of the scheduled speakers and their topics are:
Joel C. Huber, Duke University
What We Have Learned from 20 Years of Conjoint Research: When to Use Choices Vs. Graded-Pairs Vs. Full Profile Ratings
Warren Kuhfeld, SAS Institute
Efficient Experimental Designs Using CVA Design Software
Dick Wittink, Cornell University and Bill McLauchlan, McLauchlan & Associates
Solving the Number of Levels Effect in Conjoint
Keith Chrzan, Boehringer Mannheim and Ritha Fellerman, IntelliQuest
A Comparison of Full- and Partial-Profile Best/Worst Conjoint Analysis
Joop Hox, University of Amsterdam
Overcoming the Problems of Special Interviews on Sensitive Topics: Computer Assisted Self-Interviewing Tailored for Young Children and Adolescents
Bryan Orme, Sawtooth Software and Mark Alpert, The University of Texas at Austin
Conjoint Validity for High-Involvement Purchases
Karlan Witt, IntelliQuest
Best Practices in Conducting Internet Surveys Online
Ray Poynter, Sandpiper, Inc.
An Alternative Approach to Brand Price Trade-Off
Sawtooth Conferences are professional rather than commercial activities. The focus of the conference is not about our software, and you do not need to be a Sawtooth user to attend (although clinics will be offered each day after the regular program for anyone desiring information about our products).
Registration is $600 until June 30, which includes breakfasts and lunches. After June 30, registration will be $650. Attendance will be limited to 200 participants, so we suggest that you register early to avoid disappointment.
To request registration and hotel information or with any other questions, contact Marilyn Stanford at 360/681-2300.
Go back to Index

Getting the Most From CBC - Part 2 (SS Spring 97)

Our previous issue featured suggestions for using CBC, including use of prohibitions, choosing the numbers of levels, determining sample size, and presenting simple analyses. You can download that article from our Web page. Here are some further suggestions:
Including "None": CBC provides the option of letting the respondent choose "None," or another constant alternative such as "I'd continue buying my usual product." One issue is whether to include "None" in the questionnaire, and a separate issue is whether to include "None" in the analysis.
We think it is usually a good idea to include the "None" option in the questionnaire, for these reasons:

It makes the choice tasks more realistic, because that option is usually available when shopping.
It makes the experience more pleasant for the respondent, who is not forced to select an unacceptable alternative.
It improves the quality of the data, by letting respondents screen themselves out of questions containing only alternatives they would never consider.
Some researchers like to include a "None" category in simulations, as an aid in estimating how category volume would expand or shrink as products become more attractive. We recommend against doing this, for these reasons:

CBC's estimate of how many respondents should choose "None" depends on the number of alternatives in the choice tasks. If you do a simulation with a different number of products, your estimates will not be correct.
Although choices of "None" are probably indicative of disliked alternatives, there is little reason to believe their frequency will accurately reflect the actual proportion of respondents refusing to purchase products in the real world.
In summary, we usually suggest including the "None" option in choice tasks, but then neglecting it in the analysis.
Calibrating CBC Results to Market Shares: CBC results usually differ from actual market shares. This is not surprising, since market shares are influenced by product distribution, brand awareness, out-of-stock conditions, point-of-sale promotions, imperfect buyer knowledge, and many other factors not captured in conjoint measurement.
Researchers are often motivated to adjust or "calibrate" simulation results to look like market shares. We suggest not doing so, because no matter how carefully choice results are calibrated to the market, the researcher will one day be embarrassed by differences that remain. However, if the pressure is too great to resist, there are two ways in which CBC results can be adjusted to more closely mimic market shares.
CBC utilities are scaled automatically to reflect the amount of random error in respondents' choices. You can over-ride that scaling by specifying a scaling parameter. Larger values than the default of 1.0 will create greater variation among shares of choice, making large simulated shares even larger, and small shares even smaller. Smaller values of the parameter will create less variation among simulated shares of choice, in the limit making them all equal.
Market shares are often "flatter" than choice shares, because they are affected by additional sources of random noise. If that is the case, you may be able to approximate market shares more closely with a scaling parameter of less than 1.0. Beware, however, that such an adjustment will also make your results less sensitive to changes, including pricing changes.
Sometimes, market shares reflect too much variation, for example when the largest product has nearly 100% geographic distribution but smaller products do not. In that case we suggest not adjusting the scaling parameter, which could make your results too sensitive to pricing changes.
In general, if you must make simulated shares look like market shares, we suggest using "external effects" which will adjust share levels to be like market shares without affecting sensitivity to change. External effects may be calculated for each simulated product by dividing the target share by the simulated share of choice.
IIA and the Red Bus/Blue Bus Problem: The CBC simulator, like most logit conjoint simulators, suffers from "IIA," which is shorthand for "Independence from Irrelevant Alternatives." The basic idea of IIA is that the ratio of any two products' shares should be independent of all other products. This sounds like a good thing, and at first, IIA was regarded as a beneficial property.
However, another way to say the same thing is that an improved product gains share from all other products in proportion to their shares; and when a product loses share, it loses to others in proportion to their shares. Stated that way, it is easy to see that IIA implies an unrealistically simple model. In the real world, products compete unequally with one another, and when an existing product is improved, it usually gains most from a subset of products with which it competes most directly.
Imagine a transportation market with two products, cars and red busses, each having a market share of 50%. Suppose we add a second bus, colored blue. An IIA simulator would predict that the blue bus would take share equally from the car and red bus, so that the total bus share would become 67%. But it's clearly more reasonable to expect that the blue bus would take share mostly from the red bus, and that total bus share would remain close to 50%. Indeed, the IIA problem is sometimes referred to as the "red bus, blue bus problem."
The ACA and CVA simulators offer a "first choice" model, which avoids the IIA assumption entirely. If you do a first choice simulation and add a product identical to an existing product, those two products will get the same total share of first choices as either would alone. However, the first choice model is usually not satisfactory for another reason: it tends to overstate results for the most popular products.
If you don't use the first choice model, then you will run into IIA problems. ACA, CVA and CBC offer a "correction for product similarity" which penalizes products in proportion to their similarity to others. That correction works properly under ideal conditions, but it requires assumptions about how similarity is measured. We don't know of any completely satisfactory answer to the IIA problem. The best insurance is to be aware of it, and to avoid reaching improper conclusions.
Many of our users do "sensitivity analyses" by varying a product up and down on each attribute in turn. When testing sensitivity in this way, we suggest not using the correction for product similarity. It can give misleading answers, particularly for continuous attributes such as price. Suppose all products are initially simulated as having mid-level prices. If one product is then given a higher price, it is not only less desirable because of its increased price, but also less similar to the other products because its price is different from theirs. As a result, a product may gain predicted share as its price increases!
Another common situation involves simulating the effect of line extensions. It is a fairly common error for a researcher to include two versions of the product of interest, but only a single version of other products. That gives an artificial edge to the product with two versions. One rough work-around to this problem is to include two versions of every product, but to permit the two versions to be different only for the one with a line extension.
Finally, because of IIA restrictions, conjoint simulators are not very good at measuring cross-elasticities. They can do quite a good job of measuring the effect of a price change on that product's own share, but they are not very good at measuring the effect of a price change on other products' shares. The reason, again, is that IIA restrictions require that a product's interactions with others will be proportional to their shares.
We end with a note of good news. IIA problems are less serious when simulations are done at the individual respondent level, or even with homogeneous subgroups of respondents. IIA is less of a problem for ACA and CVA than for CBC. And even with choice data, IIA problems may be reduced by first separating respondents into homogeneous groups, as is done with the CBC Latent Class Segmentation Module.
Go back to Index

Including Holdout Choice Tasks in Conjoint Studies (SS Spring 97)

We think it is wise to include holdout choice tasks in conjoint interviews, even though they may not appear to be needed for the main purpose of the study. They almost always turn out to be useful, for these reasons:

They provide a proximal indication of validity, measured by the utilities' ability to predict choices not used in their estimation.
They provide a check on the scaling of the utilities. If the most popular concepts are over-predicted, then the scale parameter should be reduced. If the predictions are too flat, then the scale parameter should be increased.
They permit identification and removal of inconsistent respondents.
They can be used for testing specific product configurations under consideration. Much value can be added by direct measurement of these concepts.
It's hard to design good holdout concepts without some prior idea of respondent preferences. There's no point in asking people to choose among concepts where one dominates in the sense that everyone agrees which is best. And, similarly, it's good to avoid presenting concepts that are equally attractive, since equal shares of preference would be predicted by a completely random simulator. If you present triples of concepts, it's probably best if their shares of choices are somewhere in the neighborhood of 50/30/20.
When conducting CBC studies, if you plan to do segmentation with latent class analysis, it's wise to consider the kinds of groups you expect to get, and to design products in holdout choice sets so that one alternative will be much more preferred by each group. This maximizes your ability to confirm the validity of the multi-group Lclass simulator.
It isn't necessary to have many holdout sets to check the validity of your utilities, or their scaling. However, if you want to use those choices to identify and eliminate inconsistent respondents, you need several choice sets.
For ACA studies, holdout concepts can be included in the computer-administered interview using the Ci3 System. For CBC studies, either the Ci3 System, or the fixed design option of CBC's "TSK:" instruction may be used for presenting holdout concepts.
We've shown an example of a holdout choice task below. It is probably not very useful to include a "None" option in holdout choice tasks, particularly when these are paired with traditional conjoint exercises which don't have a "None" option.
Finally, if you do have several choice sets, it's useful to repeat at least one of them so you can obtain a measure of the reliability of the holdout choices. Suppose your conjoint utilities are able to predict only 50% of the respondents' holdout choices. Lacking data about reliability, you might conclude that the conjoint exercise had been a failure. But if you were to learn that repeat holdout tasks had reliability of only 50%, you might conclude that the conjoint utilities were doing about as well as they possibly could, and that the problem lies in the reliability of the holdout judgements themselves.

If you were shopping for a credit card and these were your only options, which would you choose?

Visa
No Annual Fee
15% Interest Rate
Frequent Flier Program
$4,000 Credit Line Mastercard
$30 Annual Fee
12% Interest Rate
No Frequent Flier Program
$6,000 Credit Line Discover
$60 Annual Fee
9% Interest Rate
Frequent Flier Program
$2,000 Credit Line

Go back to Index

Achieving Better Response Rates for Disk-By-Mail Surveys (SS Spring 97)

The following is an excerpt from an article entitled "Best Practices in Disk-By-Mail Surveys" by Karlan Witt of IntelliQuest, Inc. and Steven Bernstein of Apple Computer, originally presented at our 1992 Sawtooth Software Conference. The complete article is available for downloading from our Technical Paper Library on our home page: www.sawtoothsoftware.com.
Typical Response Rates on DBM Studies
Response rates on IntelliQuest disk-by-mail studies have ranged from 35% for an over-surveyed group conducted during the summer, to 70% when a high profile client was disclosed as the sponsor. With this potential 2X difference in response rate, it is important to heed all factors affecting response rate.
The most influential factor to impact response rates is the disclosure of a highly respected corporate sponsor. The next most influential aspect is the sample itself. Very senior executives, decision makers, and employees with select functions will produce lower response rates. The reported survey length also affects the response rate dramatically.
Potential Bias in DBM
Achieving a high response rate is beneficial in two ways:

it decreases cost per completed interview
it increases representativeness of survey results
For DBM surveys, even more than for other types of data collection methodologies, non-respondents are potentially systematically different from respondents in at least one way: their access to personal computers. Although respondents may be screened for access to PCs, this may introduce a source of non-response bias.
Factors Affecting Response Rate
The following are critical aspects to consider:
1. Saliency of survey topic. The more interesting and relevant the topic is to the target audience, the higher the response rate.
2. Length of survey. Two components of survey length affect response rate. The first is the expected length of time to complete the survey, if reported to the respondent in the cover letter. This eliminates certain respondents who are unwilling to commit that time to the interview. The second component is perceived time elapsed while taking the survey. Some respondents may begin an interview, but they may terminate if they perceive the survey is too long. It is important to note that for disk-based surveys, respondents' perception of elapsed time is less than the actual time lapsed.
The shorter the survey, typically the higher the response rate. This is a critical component to gaining an appropriate response from the over-surveyed populations and the respondents who place a high value on their time.
3. Respect for respondents' time; high professional ethics. Although respondents will accept longer interviews under a DBM methodology, the researcher must respect respondents' time.
4. Composition of research sample. Certain populations, such as purchase decision influencers and senior executives, are asked to participate in many surveys, and others place a very high value on their time. These groups typically demonstrate lower than average response rates.
5. Access to personal computers. The majority of DBM surveys are conducted on IBM-compatible personal computers. Whether a Macintosh survey software diskette is offered as an additional option depends largely on the target audience and the objectives of the research. In either case, respondents must be known to have or must be screened for access to a personal computer.
Depending on the subject matter being measured, respondents without access may or may not be different from respondents with personal computers. Respondents without access to PCs should be asked to respond to primary demographic and firmographic questions, as well as attitudinal questions about the subject being measured to analyze the potential for bias in the non-respondent sample.
6. Convenience of taking the survey. A DBM survey permits completing the survey at a time of the respondents' choosing. This convenience provides an advantage for DBM surveys over telephone or other data collection methodologies, and produces a higher overall response rate. Additionally, providing all materials necessary for the respondent to complete and return the survey (such as the postage-paid return disk mailer) will increase response rate.
7. Sponsorship of survey disclosed. One of the key factors affecting response rate is whether the sponsor of the research is disclosed. While it is clearly not appropriate in most studies, disclosure will increase the response rate, especially if the sponsor is respected by the target audience, such as in product follow-up surveys.
Disclosing the sponsor may also benefit the sponsoring company. In one customer satisfaction study, 35% of respondents stated that their attitudes toward the sponsor improved as a result of receiving the survey from the sponsor.
8. Guarantee of anonymity or confidentiality. Mailed surveys in general offer respondents some degree of anonymity; lack of anonymity is often a source of non-response in other data collection methodologies. This anonymity helps both on item non-response and unit non-response level.
9. Priority or First Class mail. Respondents react to a package as soon as it arrives. The packaging and professional appearance of the package and its contents will be the respondents' first impression. The goal is to have respondents complete the survey immediately.
In one IntelliQuest study, a split sample was used to test the effect of First Class vs. bulk rate postage on response rate. The response rate from the sample using First Class postage was 32%, while the response rate for the bulk rate sample was 27%. In debriefing with respondents from another study, it was found that faster mailing methods (for example, Federal Express or USPS Priority Mail) connote that the survey is of great importance to the sponsor of the research, and the respondents are therefore more likely to respond, and respond soon after receiving the survey.
10. Personalized cover letters and envelopes. This is a specific illustration of the packaging discussion above. The more professional the packaging and presentation from the research sponsor, the higher the response. While personalized cover letters increase response rate, even small typographical errors in the cover letter may have an adverse effect on response rate.
11. Incentive. Incentives are one of the most interesting and most debated response rate enhancers in survey research. Most sources report that incentives of any kind increase response rate. To examine the effect of offering incentives, IntelliQuest performed an experiment where potential respondents were randomly assigned to one of two groups. One of these groups was offered a coffee mug as an incentive for responding. The other was not offered an incentive. The promised incentive increased the response rate from 45% to 54%.
Selection of incentives may also affect response. Incentives should be appealing and motivating to the target respondents. Incentives may be job-related, such as an executive summary of the research results or a chance to win office equipment, or personal, such as a chance to win cash, a trip, or other such prizes. We have found that a choice of prizes is effective, particularly when the choices consist of targeted prizes. For instance, early adopters of technology respond to high-end technology gadgetry.
In targeting incentives, be cautious not to offend the intended respondents. In a debriefing of Fortune 500 senior executives, we have found that some respondents felt the use of a $1 bill was insulting to them, considering the value of their time; conversely many thought $1 communicated that the survey was important to the survey sponsor.
12. Pre-notification/pre-screening. In many studies it is necessary to contact respondents in advance of the mailed survey to:

Identify the individual who should receive the survey
Pre-qualify individuals for the study
Identify to which market segment, or quota group a respondent belongs
Screen for access to a personal computer
Verify address
Even when it is not necessary to conduct a pre-screening call, we have found that pre-notifying respondents, either by mail or phone, increases response rate. Pre-notification legitimizes the survey and communicates its importance to the survey sponsor.
Additionally, pre-qualifying respondents by telephone ensures that all respondents receiving the survey are eligible to participate. If non-qualified respondents receive survey disks and do not respond, they may be counted in the non-response category. It is not non-response bias if an unqualified respondent does not respond.
It is important for respondents to receive the survey package soon after the pre-notification. For a telephone pre-notification, we have found it most effective for respondents to receive the package within two to three days. With written pre-notification (letter or postcard), we have found it most effective for the package to be received approximately five to seven days after the notification.
13. Second mailing or follow-up postcard or phone call. As with pre-notification, a reminder call or postcard increases response rate. This follow-up may be used to thank respondents if they have already responded, and gain share of mind among those who have not yet responded. In one DBM study, the use of reminder phone calls almost doubled the response rate with a difficult-to-survey population.
Effect of Incentives/Reminder Postcards
Incentives and reminder postcards are an effective way for increasing response rates. To study the effects of incentives and reminder postcards on response rate, we conducted a study in 1987 where four groups were selected to receive a combination of a $1 incentive/no incentive and a reminder postcard sent five days later/no reminder postcard.
As shown in the following graph, the group that received both the $1 incentive and the reminder postcard had a 46% response rate. The group that received neither had a 33% response rate. The group that received $1 incentive and no reminder had a 39% response rate, and the group that did not receive an incentive, but did receive a reminder postcard had a 34% response rate. For this study, it seems that a $1 incentive worked well by itself and better in conjunction with the reminder card. The reminder card, when used alone, increased response rate only slightly.
We have found that incentives typically pay for themselves because the increased response rate requires fewer survey packages to be mailed to achieve the same number of completed interviews.

Response Rates For Experimental Study

$1 Incentive/Reminder post card 46%
$1 Incentive/No reminder post card 39%
No Incentive/Reminder post card 34%
No Incentive/No reminder post card 33%