CBC Tutorial and Example

CBC Tutorial and Example

Introduction

This tutorial leads you step-by-step through a simple CBC study, from thinking about a marketing problem, planning a questionnaire and creating a new study in SSI Web, to entering a list of the attributes and levels, generating the questionnaire and analyzing the results.

SSI Web is a powerful program, and this tutorial only touches on the basics. You'll become aware of different or more sophisticated approaches by reading other sections in this manual, by viewing on-line help and by experimenting with SSI Web.

We also suggest the following two articles, available within our Technical Papers Library at www.sawtoothsoftware.com/techpap.shtml: "CBC Technical Paper" and "Getting the Most from CBC."

(Note: this example and the data presented are purely fictional. The suggested steps represent one approach to this type of marketing problem, but may not necessarily reflect the single "best" way to conduct such a research study using the SSI Web system.)

The Marketing Problem

You have been hired as a consultant to a little-known startup company called Performance Plus. The brilliant design engineers at Performance Plus have developed a golf ball that flies farther than average balls. The proposed name for the ball is "Long Shot." Because Performance Plus is not well known among golfers, your client has considered approaching a well-known maker of golf clubs and balls (Golfers, Inc.) to market the ball using their existing brand name for balls: "Eclipse." Sales for Eclipse have been declining, and they welcome the opportunity to benefit from the new technology. A line extension brand called "Eclipse+" has been proposed. However, the royalty Golfers, Inc. is proposing seems high.

Two other manufacturers already market "high performance" balls: High-Flyer Pro (by Smith and Forester), and Magnum Force (by Durango). High performance balls command a price premium of between 20% to 50% over traditional balls, and the market is growing rapidly.

One day between rounds of golf with the president of Performance Plus (you gracefully lost), you jointly draft up the following questions on the back of your score card.

1) How well could Long Shot by Performance Plus hope to compete with existing competitors Smith and Forester and Durango in the high performance ball market?

2) Should Performance Plus form an alliance with Golfers, Inc.? How much incremental value does the Golfers, Inc. brand name "Eclipse" add?

3) How do golfers trade off performance vs. price for high performance balls?

4) How should "Long Shot" be manufactured and positioned in terms of performance and price to maximize profitability?

5) Do novice or experienced golfers differ in their preferences for high performance golf balls? Could this lead to a target market strategy?

Defining a List of Attributes and Levels

One of the first steps in using any Conjoint technique is to define the problem in terms of attributes and levels. Back at the office, you review the points written on the back of the score card, and you write down the following:

Brand

High-Flyer Pro, by Smith and Forester

Magnum Force, by Durango

Eclipse+, by Golfers, Inc.

Long Shot, by Performance Plus

Performance

Drives 5 yards farther than the average ball

Drives 10 yards farther than the average ball

Drives 15 yards farther than the average ball

Price

$4.99 for package of 3 balls

$6.99 for package of 3 balls

$8.99 for package of 3 balls

$10.99 for package of 3 balls

Each of these levels is mutually exclusive; golf balls are described using a single level from each attribute. The range of the attributes is based on technical specifications that Performance Plus has given you. As for prices, your research suggests that the two competitors are currently selling packages of three balls for between $7.99 to $10.99. You select a wider range to cover the price levels you are interested in, but don't consider any prices below $4.99, having determined that such a low price cannot cover the manufacturing and marketing costs and result in a profit.

A Study Plan and Sample Scripted Questionnaire

Now that you have formulated some attributes and levels, you begin to think about the rest of the questionnaire, and plan a design and field strategy.

You decide to invite golfing enthusiasts (by email) to visit a site on the web and take a CBC survey. You've purchased an opt-in list of 2000 email addresses from a widely-read golfing magazine. Each of the participants who complete the survey will be entered into a drawing for a free set of golf clubs from a leading manufacturer. You expect a completion rate of between 15% to 25%, resulting in a final sample size of between 300 to 500.

You have scripted the following introductory text screens and questions in your word processing package:

Start:

Thank you for your willingness to participate in this study.

You are part of a special group of golfers we've selected to ask for feedback regarding golf equipment. By completing this survey, you'll be eligible for a drawing to receive a free set of PrimoDriver clubs.

(Click the Next button to continue)

Frequency:

First, we'd like to ask you about how often you golf. Would you say that you golf...

m 5 times or fewer per year

m from 6 to 25 times per year

m from 26 to 50 times per year

m more than 50 times per year

Skill:

We'd like to get a feel for your skill level. Over the last year, what was your best score on a par-72, 18-hole course?

m 120 strokes or more

m Between 100 and 120 strokes

m Between 85 to 100 strokes

m Above par, but lower than 85

m At or below par (72 or lower)

Venue:

Do you usually golf at...

m Public courses

m Private courses

IntroCBC:

In the next section, we'll be asking you specifically about high performance golf balls. These balls are premium balls specially engineered to travel farther than the typical ball with good accuracy.

We'd like you to imagine that you are considering purchasing golf balls for your next golf outing. We'll show you some different high performance golf balls, and ask which one you would purchase.

Some of the golf balls you are going to see are not currently available on the market, but we'd like you to imagine that they were available today. It is important that you answer in the way you would if you were actually buying golf balls.

If you wouldn't purchase any of the balls we'll show you, you can indicate that by choosing "None". By choosing none, you indicate that you'd buy another brand, or would continue using existing balls in your golf bag.

At this point in the questionnaire, you plan to ask the Choice-Based Conjoint questions (tasks). You want to measure preference for the different levels of the attributes and their impact on choice for high performance golf balls. To accomplish this, you decide to specify 15 CBC tasks (below is an example of one such task). The features will combine freely to form up to 48 (4 x 3 x 4) different product offerings. Even though there are only 48 total possible product concepts in this relatively small design, there are thousands of different ways to display multiple product concepts at a time in choice sets.

Choice Task (one of 15 constructed tasks using a controlled randomization design strategy):

If you were considering buying golf balls for your next outing and these were the only alternatives, which would you choose?

High-Flyer Pro, by Smith and Forester

Drives 15 yards farther than the average ball

$10.99 for package of 3 balls

Magnum Force, by Durango

Drives 15 yards farther than the average ball

$6.99 for package of 3 balls

Eclipse+, by Golfers, Inc.

Drives 5 yards farther than the average ball

$6.99 for package of 3 balls

None: I Wouldn't Purchase Any of These

Even though these CBC tasks are called "random tasks," that term is easily misunderstood. Some people take it to mean that the composition of the questions is randomly (haphazardly) selected. However, the questions are carefully chosen according to design principles of balance and independence (orthogonality). We call the tasks "random" because respondents are randomly selected to receive one of many carefully constructed, unique versions of the CBC questionnaire.

In addition to the random choice tasks, you also plan to ask a few fixed "holdout" tasks that will be constant across all respondents. You plan to do this for three main reasons:

1) By doing some research on the Internet and at local golf stores, you have learned the performance specifications and average prices for the two existing balls on the market (High-Flyer Pro and Magnum Force). Your client has also given you a good feel for the product specifications they think could match up well with these two brands. Therefore, you and your client can imagine a specific potential product scenario you'd like to directly assess.

2) Your client has no experience with conjoint methods and seems a bit uncertain regarding the reliability and accuracy of the market simulator. Demonstrating that the market simulator can accurately predict responses to some holdout fixed scenarios should boost his confidence in the method.

3) You'll may want to test how well different part-worth utility analysis methods work for the study. You'd like to have a couple of "holdout observations" by which to compare the predictive accuracy of alternative solutions you might develop.

These fixed tasks will not be constructed randomly, but are to be displayed exactly with the specific levels you assign. These two tasks will be asked as the 7th and 9th tasks. In total, you'll ask 15 random + 2 fixed = 17 tasks. In the first fixed holdout, you plan to show the relevant competition versus your client's offering with your client's brand name. In the second fixed task, you plan to show that same relevant competition versus the Eclipse+, by Golfers, Inc.

Here are those two fixed holdout concepts:

Fixed Holdout Task #1 (7th overall choice task):

If you were considering buying golf balls for your next outing and these were the only alternatives, which would you choose?

High-Flyer Pro, by Smith and Forester

Drives 15 yards farther than the average ball

$10.99 for package of 3 balls

Long Shot, by Performance Plus

Drives 10 yards farther than the average ball

$6.99 for package of 3 balls

Magnum Force, by Durango

Drives 5 yards farther than the average ball

$8.99 for package of 3 balls

None: I Wouldn't Purchase Any of These

Fixed Holdout Task #2 (9th overall choice task):

If you were considering buying golf balls for your next outing and these were the only alternatives, which would you choose?

Magnum Force, by Durango

Drives 5 yards farther than the average ball

$8.99 for package of 3 balls

Eclipse+, by Golfers, Inc.

Drives 10 yards farther than the average ball

$6.99 for package of 3 balls

High-Flyer Pro, by Smith and Forester

Drives 15 yards farther than the average ball

$10.99 for package of 3 balls

None: I Wouldn't Purchase Any of These

Notice that your client's offering is in the center position for both tasks, but the position of its competitors is rotated. The two fixed holdout tasks are separated by a random choice task so that respondents will generally not recognize that it is a repeated task with just one small difference (the brand name attached to your client's ball).

Lastly, you decide to ask about gender and income. These might prove useful as a respondent filter in analysis, for bench-marking versus future waves of research, or for future target/database marketing efforts.

Gender:

Are you..

m Male

m Female

m Refused

Income:

This is the last question in the survey. What is your total household income for the last calendar year, before taxes?

m Less than $30,000

m Between $30,000 and $60,000

m Between $60,000 and $100,000

m More than $100,000

m Refused

Signoff:

That completes our survey. Thank you very much for your input.

Now that you have developed a list of attributes and levels, and scripted a rough-draft of your questionnaire and the design of the choice tasks, you are ready to create the study and compose the questionnaire within the SSI Web system.

Using SSI Web to Write the Questionnaire

Start by opening the Sawtooth Software SSI Web program. If you have a standard installation, you start it by clicking Start | Programs | Sawtooth Software | Sawtooth Software SSI Web. "SSI" stands for "Sawtooth Software Inc." and "Web" conveys the idea that this software is for developing Web-based surveys (although SSI Web can also be used for standalone computer interviewing with its CAPI interviewing module.) CBC is one component within that software program. The next step is to open a new study, and define a study name.

After starting SSI Web, choose File | New Study…. The New SSI Web Study dialog appears, with your cursor active in the Name field.

The Location field lists the folder (directory) that SSI Web currently is using to store studies. You can use any folder you like for this tutorial project. You can browse to and create new folders by clicking the Browse... button to the right of the Location field, and then (after browsing to the folder in which you want to create a new sub-folder to contain this study) by clicking the "new folder" icon. Each studyname in SSI Web has a maximum number of twenty-four characters (either letters or numbers), and SSI Web requires that the study be located within its own folder. The folder that SSI Web automatically creates for your study also carries the same name as the study. For this tutorial, you might create a name such as golfexample. From the New SSI Web Study dialog, specify golfexample as the studyname.

Click the OK button. You are returned to the main menu, and a new "Study Navigator" window is displayed along the left-hand side of the screen.

As you work with this study, items appear in the Study Navigator window, listing the functional areas you have been using or that are now available to you. This list of items provides a Shortcut link to access those parts of SSI Web. Alternatively, you can access those same areas by clicking icon buttons on the toolbar or by using the pull-down menus.

Now that we have created an SSI Web study called golfexample, we can add our CBC exercise to that study. Click Compose | Write Questionnaire... to access the Write Questionnaire dialog. Place your cursor at the place in the questionnaire when you'd like to add the CBC exercise. Right now, there are no questions in your survey, so you can just click the last page break in the questionnaire list, and click the Add... button. Specify to add a new CBC exercise named CBCgolfexercise to the study (you can have multiple CBC exercises within the same study).

Click OK, and the CBC Exercise Settings dialog is shown:

Default text is supplied within the Header 1 field. Paste the text we'll be using for this task into the Header 1 field:

If you were considering buying golf balls for your next outing and these were the only alternatives, which would you choose?

Entering the List of Attributes and Levels

When composing conjoint analysis studies, one typically begins by specifying the list of attributes and levels in the software.

To enter the list of attributes and levels you developed, go to the Attributes tab on the CBC Exercise Settings dialog.

To add the first attribute, click the Add button (at the bottom left of the Attributes panel).

Type in the first attribute name: Brand. The attribute name is a label that the respondent may see in the interview (if you select the option to show the attribute labels at the left of each choice task). If you want to use a shorter label to display to you as the questionnaire author for program settings and data export, specify that label in the Internal Label field. If you do not specify an internal label, the label in the Attribute Display Text is used. Click OK to accept this information and close the dialog. Now that at least one attribute name is in the list, the Add button under the Levels panel becomes active. Also note that the Brand attribute is highlighted in the Attributes panel. With the Brand attribute highlighted in the Attributes panel, click the Add button under the Levels panel to add levels within the Brand attribute. The Level Text dialog is displayed. Type High-Flyer Pro, by Smith and Forester in the Level Display Text field. To add the next level of Brand, press the ENTER key twice. Type in the next level: Magnum Force, by Durango. Repeat the process for the remaining two levels of brand.

When you are ready to add the next attribute (Performance), click the Add button under the left Attributes panel, type the attribute label, and click OK to place that new attribute on the attribute list. With that new attribute highlighted on the attributes list, click the Add button under the Levels panel to add the three levels of that attribute.

Follow the same pattern for the last attribute, Price. For your convenience, we repeat the full list of attributes below. Note that you can copy-and-paste attribute level text from this document (and other text documents) into the text fields within SSI Web. After highlighting the words to copy with your mouse, use the shortcuts Ctrl-C to copy, and Ctrl-V to paste into the desired field. (Hint: you can select a list of attributes or levels from Word and paste into SSI Web using the Paste list member(s) from the clipboard

icon. This can save a great deal of time.)

Brand

High-Flyer Pro, by Smith and Forester

Magnum Force, by Durango

Eclipse+, by Golfers, Inc.

Long Shot, by Performance Plus

Performance

Drives 5 yards farther than the average ball

Drives 10 yards farther than the average ball

Drives 15 yards farther than the average ball

Price

$4.99 for package of 3 balls

$6.99 for package of 3 balls

$8.99 for package of 3 balls

$10.99 for package of 3 balls

Now that you have finished entering the list of attributes and levels, we'll use the other tabs to continue developing our CBC exercise.

Additional Study Parameters

After you have specified your list of attributes and levels, you can specify other study parameters to govern your CBC exercise. From the CBC Exercise Settings dialog, click the Design tab. The upper left-hand corner of that dialog shows the following:

Here, you can specify how many Random (experimentally designed) choice tasks and how many fixed (user specified) tasks you want. For this golf ball study, we want 15 random and 2 fixed tasks. Change the Number of Random Choice Tasks field to 15.

As planned, the questionnaire shows three products on the screen, plus a None. Change the Number of Concepts per Choice Task to be 3 (excluding the None option). Recent research has suggested that the Balanced Overlap design approach is a robust approach for studies like this, so select Balanced Overlap as the Random Task Generation Method.

It is useful to have many versions of the questionnaire (each showing a different combination of golf balls). This leads to efficient designs for estimating part-worth utilities, and also controls for order bias. The default in the software is to generate 300 unique versions of the "Randomized" CBC tasks. Such a quantity may be overkill, but the software can handle this quite easily, so it seems prudent to go with a large number such as 300.

Recall that we want to include a "None" option in this CBC questionnaire. This gives respondents the option to indicate that they would choose none of the products displayed on the screen. Click the Format tab. Toward the bottom of that dialog, you'll see:

The Traditional None Option is selected default, and you can specify the text to use for your None option by clicking the Settings... button. Specify the following text within the None Option dialog:

None: I Wouldn't Purchase Any of These

Click OK to return to the Format tab. The upper part of that tab shows the following settings:

The Format Options control how wide the choice tasks might appear on the respondent's monitor, how much white space is included between concepts and attribute levels, and how many columns to use to display the product concepts.

Number of Columns describes how many columns will be used to arrange the product concepts on the screen. Our questionnaire calls for three products plus a none, for a total of 4 columns. Change the Number of Columns to 4.

The Alternating Concept Colors are used in alternating concepts in the choice task to help make the choice task more readable, and provide a visual indicator from one choice task to another that the question has changed (sometimes choice tasks can look so similar that respondents might not notice from, say, the first task to the next that anything is different.) It appears that no alternative colors have been selected, but indeed alternating colors have already been selected as part of the Style that is currently being used for your SSI Web study. If you would like to override the default colors used in your style, you can select new colors here.

You can override the default font color and size specified in the Style that is currently being applied to your SSI Web study (Styles are selected by going to Compose | Survey Settings, Style tab) by clicking the Paintbrush

icon.

To preview how the CBC task will look when it runs on the respondent's computer using Windows Explorer, click the Preview button. The question is displayed in "Preview" mode (no data are saved, and no special "error checking" or "validation" is performed for the question).

This may or may not be the look you want for the questionnaire. You can modify the font and sizes of the font using the Paintbrush

icon. Or, you can select a different Style (selecting styles was described in the 45-minute tutorial for SSI Web).

Go ahead and close the browser window. At this point, you might try modifying some of the settings on the Format tab. After each change you make, you can click Preview to see the effect of the changes.

Generating the Experimental Design

In CBC studies, we use a carefully chosen experimental design plan. The "experiment" involves observing how respondents react to different golf ball designs. The "design" reflects the attribute combinations that make up the golf balls, and how these combinations are placed into choice sets. Ideally, each respondent would receive a unique questionnaire version, with different combinations of golf balls arranged within sets in unique ways. There are only a finite number of possible combinations, so there potentially can be some identical choice tasks across respondents, but the idea is to improve measurement of the effects of the attribute levels (including reducing order and learning effects) by ensuring a high degree of variability in the choice tasks across individuals.

CBC lets you generate up to 999 unique versions of the questionnaire to be uploaded to the web server. By pre-specifying the design plans in a file residing on the server, we can simply assign each new respondent to the next design version within the file, which places minimal demands on the web server. Even if you have more than 999 respondents, once a reasonably large number of designs are allocated across respondents, the statistical gains of using even more questionnaire versions are very minimal. The default in the software is to use 300 design versions. Even 300 designs is probably overkill, but the software manages hundreds of designs for typical studies quite easily, and it seems beneficial to reduce order bias and improve design efficiency by using numerous questionnaire versions.

If you aren't already at the CBC Exercise Settings dialog, go there by navigating to the CBC exercise from the Study Navigator panel, or by clicking Compose | Write Questionnaire and editing a question within the exercise in the question list. Click the Design tab, and the following is displayed:

Many of the features of this dialog are beyond the scope of this tutorial. You can read additional details by pressing F1. We'll cover some of the basics here.

Under the Design Settings area, we specify the Random Task Generation Method. This indicates the strategy that CBC uses to generate the "random" choice tasks in our study. We'd recommend using the Balanced Overlap method for this study.

Number of Questionnaire Versions refers to how many unique versions of the CBC questions that SSI Web will save into a file. When respondents enter the questionnaire, they are assigned the next questionnaire version. Once a respondent is assigned the 300th version, the next respondent begins again with version #1, and so forth.

Randomize Attribute Position within Concepts lets you randomize the order in which the attributes appear in the choice task. We specified the attribute list in the order Brand, Performance and Price. If we use the default (which is not to randomize the order), Brand will always appear in the top and Price always at the bottom. This seems like a natural presentation order for this study, so we'll retain the Do not Randomize Attribute Order setting.

Concept Sorting within Tasks controls how the concepts are arranged within the choice task. By default, the order of presentation is randomized. But, if we wanted the first brand always to appear in the first concept position, etc., we could specify to sort concepts based on the Natural Order for Brand. You can investigate details regarding Concept Sorting by pressing F1. For this tutorial, we'll retain the defaults.

Once we are comfortable with our settings, we click Generate Design.

You are given two warnings, notifying you that you have yet to specify the composition of the two fixed tasks in your study. We'll do that later, so you can ignore the warnings.

The following report is displayed:

CBC Design Efficiency Test
Copyright Sawtooth Software
12/2/2011 3:31:51 PM

Task generation method is 'Balanced Overlap' using a seed of 1.
Based on 300 version(s).
Includes 4500 total choice tasks (15 per version).
Each choice task includes 3 concepts and 3 attributes.

A Priori Estimates of Standard Errors for Attribute Levels
-------------------------------------------------------------
Att/Lev   Freq.   Actual    Ideal      Effic.
1    1      3375 (this level has been deleted) High-Flyer Pro, by Smith and Forester
1    2      3376   0.0275   0.0275   0.9970    Magnum Force, by Durango
1    3      3375   0.0275   0.0275   1.0023    Eclipse+, by Golfers, Inc.
1    4      3374   0.0275   0.0275   0.9999    Long Shot, by Performance Plus

2    1      4500 (this level has been deleted) Drives 5 yards farther than the average ball
2    2      4500   0.0231   0.0231   0.9964    Drives 10 yards farther than the average ball
2    3      4500   0.0230   0.0231   1.0052    Drives 15 yards farther than the average ball

3    1      3375 (this level has been deleted) $4.99 for package of 3 balls
3    2      3374   0.0276   0.0275   0.9954    $6.99 for package of 3 balls
3    3      3376   0.0276   0.0275   0.9994    $8.99 for package of 3 balls
3    4      3375   0.0276   0.0275   0.9934    $10.99 for package of 3 balls

Note: The efficiencies reported above for this design assume an equal number of respondents complete each version.

The report indicates that 300 versions of the CBC questionnaire were created. We know that each version of the questionnaire had 15 "random" choice tasks, so a total of 300 x 15 = 4500 choice tasks were generated. Some of the details of the report are beyond the scope of the unit, but you can obtain more information within the Help text. The "Freq." column indicates how many times each level is represented in the design. Within each attribute, you can see that CBC has nearly perfectly balanced the presentation of levels. The "Effic." Column indicates the relative efficiency of the design. Design efficiency runs from the worst (0.0) to the best (1.0), and we can see that this particular design is quite efficient with respect to main effects.

If you include prohibitions in your design, sometimes designs can be quite inefficient. If the Effic. values are relatively low, this is a warning that your estimates may lack precision, and you should reconsider your setup. In some cases, you will receive a warning stating that the design is deficient or will notice asterisks rather than data in the test design report. You should re-evaluate your design should this occur. Failure to correct the problem can result in unusable data.

Click Close to close the test design report, and click OK to close the CBC Exercise Settings dialog.

Let's now examine the questionnaire we've built to this point:

You'll note that SSI Web has added 15 Random (CBCgolfexercise_Random1 through CBCgolfexercise_Random15) and 2 Fixed CBC tasks (CBCgolfexercise_Fixed1, CBCgolfexercise_Fixed2) to the survey. These questions are based on your settings and the default template within SSI Web. The Fixed tasks aren't yet in the proper positions (we wanted them in the 7^th and 9^th task positions in the survey). You also haven't yet specified the attribute combinations used for your user-defined Fixed tasks, so SSI Web has inserted the questions with default levels that we'll modify shortly.

Before we fine tune those CBC questions, let's turn our attention to formatting other aspects of the survey.

Passwords and the Start Screen

Placing a survey on the web makes it convenient for a geographically dispersed population to take surveys. However, the danger is that the survey may become available to people that have not been invited to take the survey. Also, some respondents might try to take the survey multiple times. Assigning respondent passwords is a way to deal with both of these issues. Password assignment is beyond the scope of this tutorial, so to make things simple we'll assume that no passwords are to be used.

The Start screen is placed as the first page in your questionnaire, whether you are using passwords or not. If you are using passwords, it is the page in which respondents type passwords to access the survey. If not using passwords, you'll use this screen to specify any introductory/welcome text.

Let's enter the introductory text into the Start screen using the Write Questionnaire dialog. You can access the Write Questionnaire dialog by selecting Compose | Write Questionnaire… or by clicking the "pencil" icon

on the toolbar.

The introductory text and other survey questions would most likely be initially developed within a word processing document. Assuming you really had such a document, you might use the Ctrl-C to copy, and the Ctrl-V shortcuts to paste the information into SSI Web. We suggest you simply copy-and-paste the text within this document into your SSI Web questions rather than re-type the text (if viewing this document with Acrobat Reader, you can use the "text select" icon

from the Acrobat Reader toolbar.)

The introductory screen is as follows:

Start:

We'll insert this introductory screen in the Start question that is automatically the first question of any SSI Web survey. From the Write Questionnaire dialog open the Start question by highlighting it in the list of questions and clicking Edit.... Alternatively, you can double-click the Start question, and it automatically opens. The Questionnaire Access and Passwords dialog is displayed:

Most question types in SSI Web have "Header 1", "Header 2," "Body" and "Footer" sections. (The Start question has all but the "Body" section.) These are "text" areas in which you can insert any text (including HTML). When the question is viewed with a web browser, the sections are organized roughly as follows:

For this introductory screen (that includes three paragraphs of information), it seems to make sense to place the first paragraph in the "Header 1" area, the second paragraph in the "Header 2" area, and the third paragraph in the "Footer" area.

Type (or cut and paste) the following text for the Start question into the text areas in the Start question. Put the first paragraph in Header 1, the second paragraph in Header 2, and the third paragraph in Footer.

Thank you for your willingness to participate in this study.

You are part of a special group of golfers we've selected to ask for feedback regarding golf equipment. By completing this survey, you'll be eligible for a drawing to receive a free set of PrimoDriver clubs.

(Click the Next button to continue)

To preview how the question will look when it runs on the respondent's computer using Windows Explorer, click the Preview button. The question is displayed in "Preview" mode (no data are saved, and no special "error checking" or "validation" is performed for the question).

SSI Web automatically places paragraph breaks (extra blank lines) between the Header 1, Header 2, and Footer sections. If you put all the text in a single section, you may see that when the web browser interprets the text, it all runs together without any blank lines between the paragraphs (unless you insert some HTML instructions to force blank lines between paragraphs). We'll talk about using HTML within your text to take greater control over the layout, font, and style later in this tutorial.

Click the OK button in the Preview window to close that window and return to the previous SSI Web dialog.

After viewing the text in preview mode, you might decide that it is really too small, or that you want to make the text bold. You can change the size and styles (bold, italic, underline) or the text justification for the three major text sections by clicking the Paintbrush icon

on the Start Question Text tab. After changing any setting, click Preview again.

After you are happy with the layout of the Start screen, click OK to return to the Write Questionnaire dialog. If you need to move a question once it has been added to the List of Questions, simply highlight the question to be moved and click the

buttons to move the questions within the list (you can also highlight a question and click Ctrl-X or Ctrl-C to cut or copy and Ctrl-V to paste questions within the current questionnaire, or even to another SSI Web study.)

Now you are ready to specify the first Select-Type question:

Frequency:

First, make sure that you highlight the page break directly beneath the Start question on the List of Questions (when you add a new question, it is placed directly below the highlighted question/page break on the list). At the Write Questionnaire dialog, click Add..., choose Select as the question type, and type Frequency for the question name. Click OK and the Select Question dialog is shown.

Place the heading text for the question in the Header 1 field. To specify the response options, click the Response Options tab. From the Response Options tab, use the Add... button to add the four response options for this question. When you are finished, the question should look something like:

Now that you have seen how to specify Single Select question types in SSI Web, you have the tools you need to specify the remaining four select-type questions for the golf ball questionnaire (Skill, Venue, Gender and Income—please refer to the questionnaire text for these questions as presented near the front of this tutorial).

Remember that when you add a new question to the List of Questions, it is inserted directly below the question that was highlighted when you clicked the Add... button.

Formatting Text in SSI Web Questions

In addition to the Select questions we added to the questionnaire previously, there are two text-only instructional screens to add. Referring to the questionnaire we outlined earlier, these "questions" are named IntroCBC and Signoff. Even though the respondents aren't asked to provide specific inputs to these, we refer to these as "questions" in the sense that we add them to the questionnaire as if they were standard questions, and they are listed separately by their "question" name in the List of Questions.

After the Venue question follows the text-only instructions:

IntroCBC:

This text layout is a bit more challenging than we dealt with before (when we put three paragraphs of introductory text into the Start question). There are more than three separate paragraphs here, and we'll need to deal with the additional element of bolding selected text.

Add a Text/HTML Filler question directly following the Venue question in the questionnaire. To do so, highlight the Venue question, click Add…, and specify the Question Name as IntroCBC and the question type as Text/HTML Filler.

The Text/HTML Filler dialog is displayed.

Earlier in this tutorial, when we used the Start question to format the opening page of the survey, we placed each paragraph of text in a separate Header 1, Header 2, or Footer section. We saw that SSI Web automatically places blank lines between text in these sections. However, with the text in the IntroCBC question, there are many more paragraph breaks. We'll take the opportunity here to introduce the concept of using a few simple HTML instructions within our survey text.

HTML stands for "HyperText Markup Language" and provides simple ways for you to enhance the look of your surveys, such as by bolding or underlining words, or making paragraph breaks within text. Browsers know how to interpret HTML instructions when displaying the page on the screen. If you know HTML, you can use it whenever you wish within SSI Web surveys to accomplish your aims. If you don't know HTML, it isn't very difficult to learn a few HTML tricks, or you can use the toolbar available within the editor that appears when you click "Pencil" icons

in SSI Web:

toobar

Either type or cut-and-paste the text for the IntroCBC question into the Header 1 text field (Any of the other three fields could be used also, though the Body field is indented). Click the "Pencil" icon to display the larger text editing window. Initially, the text appears something like this:

Even though it appears that there are extra blank lines between the paragraphs, web browsers ignore these hard returns (as well as more than one consecutive space characters), so you'll need to provide HTML instructions to insert these paragraph breaks (hard return plus blank line). HTML instructions are placed within <> brackets, called "tags." For example, the HTML instruction to create a paragraph break begins with an "open" paragraph tag written as  and optionally ends with a "close" paragraph tag written as . The text to be formatted as a separate paragraph is enclosed within these tags. You can either directly type HTML within your document, or you can highlight text to be modified and click the icons on the HTML toolbar in the editor. If you highlight the first paragraph with your mouse:

In the next section, we'll be asking you specifically about high performance golf balls. These balls are premium balls specially engineered to travel farther than the typical ball with good accuracy.

and then (with the text in the first paragraph highlighted) click the "Paragraph" icon

on the toolbar, this inserts a  prior to the text and a  after the text:

In the next section, we'll be asking you specifically about high performance golf balls. These balls are premium balls specially engineered to travel farther than the typical ball with good accuracy.

When the browser interprets this text, it doesn't display the "tags" but instead separates the text enclosed within the tags as a separate paragraph. Repeat the same for each of the paragraphs in the IntroCBC question.

Next, we need to bold certain words in the text. The HTML tags for bolding text are  (with the text to be bolded placed between the open and close bold tags). You can either directly type these tags within the document, or highlight the text to be bolded with the mouse and click the "Bold" icon

on the toolbar. After you finish separating the paragraphs with  tags and bolding the appropriate text with  tags, it should look something like:

Click OK to return to the Text/HTML Filler dialog and then Preview to see how the web browser displays this question. It should look like:

When you add the Signoff at the end of the survey, make sure to add it at the end of the survey as a Terminate/Link question type. On the Settings tab of the Terminate/Link question, click Terminate Respondent to indicate that respondents that reach this question are finished. SSI Web may warn you at this point that a terminating question must stand alone on its own "page" in the survey. This leads us into our next discussion on page layout.

Page Layout and Study Settings

It may be helpful at this point to review how SSI Web breaks the survey into separate pages, and some basic global settings that affect the look and functionality of your SSI Web questionnaire. The Write Questionnaire dialog also displays the page breaks that may be currently set for your questionnaire.

This dialog shows how the various questions we've specified (or that SSI Web has automatically added to the list) are arranged across different pages. Page breaks are indicated by "---------------<Page>". Notice that the preliminary questions (Frequency, Skill and Venue and IntroCBC) we've added to the questionnaire are all currently arranged on the same page. Let's assume we wanted to break these up, one question per page. You add a page break after the Frequency question by highlighting Frequency on the list and clicking the Add Page Break button. Frequency is now placed on its own page. Repeat this operation to place Skill, Venue, IntroCBC and CBCgolfexercise_Random1 on separate pages. Make sure to place the final Terminate/Link screen Signoff on its own (last) page.

While we had been adding page breaks for our questionnaire, you may have noted that the two fixed CBC tasks (CBCgolfexercise_Fixed1 and CBCgolfexercise_Fixed2) are not yet in the 7^th and 9^th CBC task positions. You can move a question on the list by highlighting the question in the List of Questions and clicking the

buttons to move the question to another point in the questionnaire. Move CBCgolfexercise_Fixed1 directly after CBCgolfexercise_Random6 and move CBCgolfexercise_Fixed1 directly after CBCgolfexercise_Random7. You may need to re-arrange the page breaks so that each CBC question remains on its own page.

Specifying Fixed Tasks

A fixed choice task looks just like typical "Random" (experimentally designed) tasks. However, rather than let the design algorithm determine the combination of attribute levels to be shown for each respondent, you specify the codes for the levels to be displayed in each fixed product concept. Recall that the purpose of specifying fixed holdout tasks for this example is to achieve a controlled and direct measure of the existing competitors' offerings versus our client's proposed offerings. You'll also be able to use the results to check the ability of the market simulator you develop using the 15 random choice tasks to predict the responses to the 2 fixed holdout tasks.

Recall that our list of questions already includes two fixed tasks named CBCgolfexercise_Fixed1 and CBCgolfexercise_Fixed2. These tasks will look exactly like the experimentally designed (random) tasks within the survey, except that you'll specify precisely which levels to show in those two tasks. Edit any of your CBC questions (or use the Study Navigator panel to directly go to the CBC exercise). Go to the Design tab on the the CBC Exercise Settings dialog.

To modify the levels displayed in the fixed choice task, click the Fixed Task Designs… button.

From this one dialog, you can specify the attribute combinations to be used in each of your fixed tasks. To change from one fixed task to another, use the Fixed Choice Task drop-down box at the upper-left hand corner of the dialog. We'll first modify CBCgolfexercise_Fixed1.

Recall that the first fixed holdout was to display the following three products:

High-Flyer Pro, by Smith and Forester

Drives 15 yards farther than the average ball

$10.99 for package of 3 balls

Long Shot, by Performance Plus

Drives 10 yards farther than the average ball

$6.99 for package of 3 balls

Magnum Force, by Durango

Drives 5 yards farther than the average ball

$8.99 for package of 3 balls

None: I Wouldn't Purchase Any of These

Use the drop-down controls provided to specify the three product concepts as shown.

Click the OK button when finished. Preview the fixed holdout task to make sure it looks as you expect. Make any necessary changes.

Next, you need to modify the second fixed holdout task. From the CBC Fixed Choice Task Settings, used the Fixed Choice Task drop-down control to select CBCgolfexercise_Fixed2. Modify it to have the appropriate specifications for our second fixed choice task:

Magnum Force, by Durango

Drives 5 yards farther than the average ball

$8.99 for package of 3 balls

Eclipse+, by Golfers, Inc.

Drives 10 yards farther than the average ball

$6.99 for package of 3 balls

High-Flyer Pro, by Smith and Forester

Drives 15 yards farther than the average ball

$10.99 for package of 3 balls

None: I Wouldn't Purchase Any of These

Again, preview this task to make sure it looks as expected.

Adding Polish and Style
You have probably noted that the survey we've created is pretty bland looking. We can add some polish and style by clicking the Survey Settings icon

from the Write Questionnaire dialog (or by clicking Compose | Survey Settings from the main menu).

First, let's select a style from the Styles tab. Select a style you wish to use, and click Use Selected at the bottom of the dialog to implement the style. We'll select Panama as the style, but you might want to experiment with other styles.

On the General Format tab, you can select to use graphical Submit, Previous, and Select/Checkbox buttons. A library of buttons is provided, available in the C:\Program Files\Sawtooth Software\SSI Web\graphics folder.

On the Headers and Footers tab, you can specify a Header and a Footer. We'll specify Golf Ball Questionnaire as the header text.

The Progress Bar tab lets you add a progress bar to your survey. We'll click the check box to add a progress bar to the page footer.

When you go back and test run the survey, your survey should have a bit more polish. Under Panama style, our survey now looks like:

Test Running Your Survey Using "Local Test Server"

Although the Preview Survey function is nice, you generally will want to test your survey in runtime mode, with any skip patterns, response verification, and randomizations in place. You could upload your files and the Perl scripts to the Web Server, but SSI Web provides a more convenient way to test run your survey locally on your PC or laptop.

When SSI Web was installed on your computer, web server software called "Apache" was also installed. Apache lets your computer run the questionnaire just like a remote Web Server would.

Local Test Server

From the Write Questionnaire dialog, Click the Test Survey button

and select to Test Survey Locally.

Your browser opens up the first survey page. This is your survey, as it would appear to a respondent over the Internet (there are slight visual differences from browser to browser).

To close the survey, simply close the browser window (using the X in the upper-right hand corner). To run the survey again, click the Test Survey button

.

After you have edited the survey to the point that you are pleased with its look, content, and functionality, you should examine the test data you've collected using Local Test Server (this process is described in a separate tutorial called "Getting Started with SSI Web: A 45-minute Hands-On Tour" available on the Help menu), then at least examine the CBC data using Counting analysis (described later). The results should generally reflect your preferences (assuming you answered in a rational manner). If they don't, this suggests something may be wrong with your study setup.

After you are convinced that the survey is functioning properly, you should also pretest your survey among your colleagues. They can give you feedback regarding the usability of the survey, and you can examine the resulting Counts data or perhaps even the part-worths (assuming you have enough data) to make sure the data at least have face validity.

Pretesting and Fielding

Let's now return to the marketing problem and story we began at the beginning of this document.

After you have tested the survey using Local Test Server, you post the survey to the Web on your web server, on a server provided by an ISP (Internet Service Provider), or through Sawtooth Software's web hosting service. (Setting up your survey on the web is beyond the scope of this unit, but is described in detail in the SSI Web help documentation. Fielding options also include hosting on your own company's server, or using Sawtooth Software's hosting services).

The next day, you send an email to your client, with a link to take the survey. The president of Performance Plus first takes a survey and suggests some minor wording changes. After you make those changes, you invite six other individuals at Performance Plus take the survey. You download the results and analyze the resulting data. Due to the small sample size, the results are a bit noisy (and there is an obvious bias toward your client's balls), but the data seem to feel right.

Next, you recruit six golf enthusiasts within your city to come to a central site to take the survey. You watch silently as they take the survey. At the end of each session, you ask each respondent follow-up questions to ensure that there weren't any parts that were difficult to understand or just didn't make sense. You ask them specifically about the choice tasks, making sure that there wasn't too much information on the screen at once, and that they didn't feel overwhelmed with the task. After debriefing the test respondents and analyzing their data to ensure that the results looked reasonable, you make a few small adjustments to the questionnaire and attribute descriptions, and proceed to field.

Using a bulk email program, you send an email to the list of 2000 golfing enthusiasts from the opt-in list mentioned at the beginning of this unit. After a week, and a reminder email sent to those who had not yet completed the survey, you achieve 300 total completes.

The completed data reside on the server where you uploaded the SSI Web survey. The details for viewing and downloading the data from the server are provided in other areas of the SSI Web documentation and are beyond the scope of this CBC unit. Even so, it is useful to at least describe the steps you would perform to access and download the data from the Web, estimate the part-worth utilities, and begin analysis using the market simulator.

Data Management, Utility Estimation, and Moving the Data into SMRT

SSI Web includes an Admin Module so that you can monitor or access your project from any computer connected to the Web. The Admin Module is password protected, and your passwords for access are specified on the Field | Web Server Management dialog. SSI Web generates random administrative access passwords whenever you create a new study, but you can modify them to suit your needs.

To download the data from the 300 respondents to the golf ball study, you can browse to the administrative module for your study on the web site (again, we are speaking hypothetically, as for this tutorial study there is no such site set up). Once at the administrative module, you would download your data (by clicking Data Management | Download from the main menu), making sure to save your data (named studyname_data.sqlite, where STUDYNAME is your study name) to the same folder on your hard drive in which you developed the SSI Web project. Alternatively, from the SSI Web interface, you can simply click Field | Download Data... to download your data automatically.

Once you have downloaded the data, you are ready to export the CBC data to a .CHO format file (with accompanying attribute labels in a file with extension .ATT), and estimate part-worths in preparation for running market simulations. Assuming you had downloaded the data as described above, you would click File | Data Management and then click Add Job.... Under Export Settings, you would name your export job something that reminds you what this export job does, and then you'd select the CBC file type. You would click OK to return to the Data Management dialog. You should select the new export job name, and then click Export all Checked Jobs. If you try this with this tutorial study, you will receive an error stating that you have no data within your project folder to export. However, hypothetical data for this project are stored in a tutorial folder within the accompanying SMRT software that you received together with your SSI Web system.

SMRT stands for "Sawtooth Software Market Research Tools" and contains the market simulation tool used to analyze data resulting from any of Sawtooth Software's conjoint analysis systems. SMRT is the platform that (given the proper license with its user identification codes) runs CBC for Windows studies and includes the tutorial data for this golf ball study. We'll discuss how you would move data from your CBC project into SMRT (hypothetically, since the data aren't actually in your CBC project) but then open the SMRT software system and continue using the golf ball tutorial data that were installed there.

CBC researchers employ a variety of techniques for analyzing their data. One common approach for initially understanding the basic, summary preferences for the market is called "Counting" analysis. This produces proportions from 0.0 to 1.0 for each level in your study, reflecting how often this level was chosen, when available in a choice set. The higher the proportion, the higher preference for the level. While this method of analysis has good intuitive appeal, there are more powerful ways to analyze the data based on estimation of part-worth utilities and subsequent market simulations.

There are three part-worth estimation routines that Sawtooth Software provides. The first technique is called multinomial logit, which "pools" respondent data in a single aggregate model. This technique was the first part-worth estimation technique that Sawtooth Software used for analyzing CBC data in the early 1990s. It is a good technique for quickly summarizing the results for the sample, but it is quite susceptible to the IIA (red-bus/blue-bus) problem, which is discussed in many technical papers and in the CBC help documentation. Two other estimation techniques are commonly used for developing the final simulator models delivered to clients: Latent Class and Hierarchical Bayes (HB). These techniques model respondent heterogeneity (recognize differences between respondents in terms of preferences) and in most cases provide more useful part-worths (than aggregate logit) for the purpose of market simulations.

It is beyond the scope of this tutorial to discuss these part-worth estimation methods. These part-worth estimation programs all work seamlessly with the .cho and .att files exported by SSI Web. If you use the logit estimation routine built into the SMRT software system, you are immediately ready to perform additional market simulations and analysis within SMRT. If you use either Latent Class or HB estimation, these routines produce a text-only format files containing case IDs and part-worth estimates that can easily be imported by SMRT.

For simplicity, we'll assume that you plan to use logit estimation provided by SMRT to produce quick topline results. To move the STUDYNAME.cho file into SMRT for analysis by logit, you would (again, this is for discussion purposes only—you should not actually perform these steps for this golf ball tutorial):

Start the SMRT software by clicking Start | Program Files | Sawtooth Software | Sawtooth Software SMRT.

Within SMRT, create a new study for analyzing the results (select File | New and choose a folder and a studyname). You can choose any folder or studyname that you want, as this new study functions independently of your original SSI Web study.

Import the STUDYNAME.CHO file into your SMRT study by clicking (from the SMRT software menu) File | Import…, selecting the file type as Choice Data (*.cho) and browsing to your .CHO file. Once you import the CBC data from the .CHO file, you can analyze the data.

Analyzing the CBC Data Using Counts

SMRT is a companion software system to SSI Web that can be used for analyzing the results of CBC studies. To start SMRT, click Start | Programs | Sawtooth Software | Sawtooth Software SMRT. Open the golf ball study by clicking File | Open and then browsing to find the Tutor2.smt file located in the ...Program Files\Sawtooth Software\SMRT\Tutorial folder.

Hypothetical data are provided to be used for the remainder of this tutorial. To access the data, please select the Tutor2 study (select File | Open, and double-click the Tutor2.smt file located in your Samples directory).

If you used a randomized design (which you did for this project), usually the first step in analyzing the choice results is to conduct a "Counting" analysis. CBC's Counts program reports the percent of times each attribute level was chosen when it was available on the screen. Counts provides an intuitive measure of the impact of each attribute level on overall choice for golf balls. Counts are proportions ranging from 0 to 1. For example, a Count of 0.31 for an attribute level would mean that when a golf ball was displayed including that particular level, respondents chose it 31% of the time.

To access the Counts program, return to the CBC main menu, and select Analysis | Counts.

By default, the Counts program analyzes all one-way and two-way Count proportions. Notice also by default that the Counts program uses only responses to the 15 randomized choice tasks you fielded (under Choice Tasks to Include, All Random is checked). The two fixed holdout tasks are not included. That is because results from Counts assume random designs where each attribute level appears an equal number of times with each level of the other attributes. Randomized designs (with no prohibited level combinations) make it possible to analyze the effect of each attribute level independent of all other levels. That characteristic does not usually hold if fixed holdout tasks are included in the analysis.

Click Compute! and the following report is displayed in the report window:

Brand

Total

Total Respondents 300

High-Flyer Pro, by Smith and Forester 0.367

Magnum Force, by Durango 0.336

Eclipse+, by Golfers, Inc. 0.189

Long Shot, by Performance Plus 0.159

Within Att. Chi-Square 416.495

D.F. 3

Significance p < .01

Performance

Total

Total Respondents 300

Drives 5 yards farther than the average ball 0.168

Drives 10 yards farther than the average ball 0.282

Drives 15 yards farther than the average ball 0.339

Within Att. Chi-Square 259.984

D.F. 2

Significance p < .01

Price

Total

Total Respondents 300

$4.99 for package of 3 balls 0.397

$6.99 for package of 3 balls 0.292

$8.99 for package of 3 balls 0.231

$10.99 for package of 3 balls 0.132

Within Att. Chi-Square 478.331

D.F. 3

Significance p < .01

Brand x Performance

Total

Total Respondents 300

High-Flyer Pro... Drives 5 yards farther... 0.263

High-Flyer Pro... Drives 10 yards farthe... 0.401

High-Flyer Pro... Drives 15 yards farthe... 0.436

Magnum Force, ... Drives 5 yards farther... 0.234

Magnum Force, ... Drives 10 yards farthe... 0.340

Magnum Force, ... Drives 15 yards farthe... 0.431

Eclipse+, by G... Drives 5 yards farther... 0.091

Eclipse+, by G... Drives 10 yards farthe... 0.206

Eclipse+, by G... Drives 15 yards farthe... 0.266

Long Shot, by ... Drives 5 yards farther... 0.083

Long Shot, by ... Drives 10 yards farthe... 0.180

Long Shot, by ... Drives 15 yards farthe... 0.216

Interaction Chi-Square 25.644

D.F. 6

Significance p < .01

Brand x Price

Total

Total Respondents 300

High-Flyer Pro... $4.99 for package of 3... 0.549

High-Flyer Pro... $6.99 for package of 3... 0.394

High-Flyer Pro... $8.99 for package of 3... 0.332

High-Flyer Pro... $10.99 for package of ... 0.204

Magnum Force, ... $4.99 for package of 3... 0.477

Magnum Force, ... $6.99 for package of 3... 0.323

Magnum Force, ... $8.99 for package of 3... 0.323

Magnum Force, ... $10.99 for package of ... 0.216

Eclipse+, by G... $4.99 for package of 3... 0.292

Eclipse+, by G... $6.99 for package of 3... 0.232

Eclipse+, by G... $8.99 for package of 3... 0.144

Eclipse+, by G... $10.99 for package of ... 0.080

Long Shot, by ... $4.99 for package of 3... 0.284

Long Shot, by ... $6.99 for package of 3... 0.202

Long Shot, by ... $8.99 for package of 3... 0.124

Long Shot, by ... $10.99 for package of ... 0.039

Interaction Chi-Square 58.408

D.F. 9

Significance p < .01

Performance x Price

Total

Total Respondents 300

Drives 5 yards... $4.99 for package of 3... 0.281

Drives 5 yards... $6.99 for package of 3... 0.193

Drives 5 yards... $8.99 for package of 3... 0.114

Drives 5 yards... $10.99 for package of ... 0.079

Drives 10 yard... $4.99 for package of 3... 0.400

Drives 10 yard... $6.99 for package of 3... 0.326

Drives 10 yard... $8.99 for package of 3... 0.264

Drives 10 yard... $10.99 for package of ... 0.130

Drives 15 yard... $4.99 for package of 3... 0.520

Drives 15 yard... $6.99 for package of 3... 0.353

Drives 15 yard... $8.99 for package of 3... 0.312

Drives 15 yard... $10.99 for package of ... 0.185

Interaction Chi-Square 20.754

D.F. 6

Significance p < .01

None

Total

Total Respondents 300

None chosen: 0.211

At first, this report may seem overwhelming, so we'll break it up and discuss it in pieces. First, Counts reports that 300 respondents were used. Then, the count proportions for brand name are displayed:

     High-Flyer Pro, by Smith and Forester               0.367
     Magnum Force, by Durango                            0.336
     Eclipse+, by Golfers, Inc.                          0.189
     Long Shot, by Performance Plus                      0.159

High-Flyer Pro was the most preferred ball name and brand on average, being chosen 36.7% of the times that is was presented and available for choice. Your client's ball name and brand, Long Shot, by Performance Plus, was the least preferred at 15.9%. If your client markets their ball under the Eclipse+ name with the Golfers, Inc. brand, choice probability improves from 15.9% to 18.9%. These are ratio quality data, so one might infer from the counts that using the Golfers, Inc. name and brand increases the probability of choice by 19% (18.9/15.9 - 1).

Next comes the counts for Performance:

Drives 5 yards farther than the average ball 0.168

Drives 10 yards farther than the average ball 0.282

Drives 15 yards farther than the average ball 0.339

Respondents on average chose balls that flew 15 yards farther more than twice as often as those that flew 5 yards farther. There appears to be a non-linear effect of performance on choice probability. Likelihood of choice nearly doubles as performance increases from +5 yards farther to +10 yards farther than the average ball. Much less is gained in terms of choice probability by increasing flight from +10 yards to +15 yards. A preliminary conclusion might be that your client should make sure their ball is rated to travel at least 10 yards farther than the average ball. But improving the performance to travel 15 yards farther than the average ball might not be worth the extra manufacturing cost, if that cost is significant.

Last comes price:

$4.99 for package of 3 balls 0.397

$6.99 for package of 3 balls 0.292

$8.99 for package of 3 balls 0.231

$10.99 for package of 3 balls 0.132

As expected, respondents prefer lower prices over higher ones. Probability of choice decreases monotonically for each step increase in price.

To this point, you've only analyzed one-way (main effect) effects of attribute levels on choice. You can further consider the probabilities of choice when a combination of two attribute levels are available for choice. There are three tables of two-way probabilities in our study. Rather than look at all three, let's examine the table (brand x price) that appears (by the Chi-Square statistic) to show the most promise of being interesting (and reflect a potentially significant interaction effect) for our study:

High-Flyer Pro... $4.99 for package of 3... 0.549

High-Flyer Pro... $6.99 for package of 3... 0.394

High-Flyer Pro... $8.99 for package of 3... 0.332

High-Flyer Pro... $10.99 for package of ... 0.204

Magnum Force, ... $4.99 for package of 3... 0.477

Magnum Force, ... $6.99 for package of 3... 0.323

Magnum Force, ... $8.99 for package of 3... 0.323

Magnum Force, ... $10.99 for package of ... 0.216

Eclipse+, by G... $4.99 for package of 3... 0.292

Eclipse+, by G... $6.99 for package of 3... 0.232

Eclipse+, by G... $8.99 for package of 3... 0.144

Eclipse+, by G... $10.99 for package of ... 0.080

Long Shot, by ... $4.99 for package of 3... 0.284

Long Shot, by ... $6.99 for package of 3... 0.202

Long Shot, by ... $8.99 for package of 3... 0.124

Long Shot, by ... $10.99 for package of ... 0.039

This table shows the probability of choice for each brand when it was shown at each price. This table is more involved than the simpler one-way tables you examined. Therefore, it may be helpful to plot the results with a graphics or spreadsheet package.

If you plot price on the x-axis and probability of choice on the y-axis, it would appear like the familiar demand curves one learns about in economics. We should note, however, that counting analysis has some drawbacks and inaccuracies associated with it. There are more accurate ways to generate demand curves with CBC using the market simulator. Also, demand curves from CBC assume perfect information, equal distribution, and other assumptions mentioned in the online Help.

The "pseudo demand curves" seem to suggest that the Eclipse+ name is preferred over all levels of price to the Long Shot name. Also, the gap appears to widen slightly as prices increase—though we cannot tell from this chart whether that is a significant or a chance occurrence.

After you have spent some time looking at the choice results for the randomized tasks, you might consider also looking at the results for the fixed holdout tasks. Recall that we asked these fixed scenarios to gauge preference for what may play out as actual future market scenarios. We also asked these fixed questions so that we could see how well the market simulator predicts the preferences for those two questions.

Recall that the first holdout task was asked as the seventh task, and the second holdout as the ninth. To analyze responses for the first holdout, you need to use the Counts program again, this time isolating only the seventh choice task. To do that, while in Counts, in the Choice Tasks to Include Area, un-check the All Random option, then click the Filter... button. Un-check all of the random choice tasks (by clicking the boxes in front of the task labels), and select just the first fixed holdout choice task, called Fixed_T1. Click OK to close the dialog. You will only be interested in one-way (Main Effect) counts, so in the Level of Analysis area, un-check the 2-Way Interactions selection (leaving only Main Effect checked). Click Compute!. The Counts for brand are displayed:

High-Flyer Pro, by Smith and Forester 0.307

Magnum Force, by Durango 0.213

Eclipse+, by Golfers, Inc. *

Long Shot, by Performance Plus 0.237

Recall that we did not display Eclipse+ in the first holdout task, so it is marked with an asterisk. For review, here is the configuration for the first fixed holdout task, with the choice probabilities listed:

30.7%

High-Flyer Pro, by Smith and Forester

Drives 15 yards farther than the average ball

$10.99 for package of 3 balls

23.7%

Long Shot, by Performance Plus

Drives 10 yards farther than the average ball

$6.99 for package of 3 balls

21.3%

Magnum Force, by Durango

Drives 5 yards farther than the average ball

$8.99 for package of 3 balls

24.3%

None: I Wouldn't Purchase Any of These

If you isolate only the second fixed holdout task (by selecting only Fixed_T2 in the Choice Tasks to Include dialog) and click Compute!, you'll get the following probabilities for the second holdout task:

18.0%

Magnum Force, by Durango

Drives 5 yards farther than the average ball

$8.99 for package of 3 balls

28.7%

Eclipse+, by Golfers, Inc.

Drives 10 yards farther than the average ball

$6.99 for package of 3 balls

29.6%

High-Flyer Pro, by Smith and Forester

Drives 15 yards farther than the average ball

$10.99 for package of 3 balls

23.7%

None: I Wouldn't Purchase Any of These

Each of these choice probabilities are based only on 1 task x 300 respondents = 300 total choice tasks, whereas the previous counting data was based on a total of 15 tasks x 300 respondents = 4,500 total tasks.

In accordance with the previous conclusion we made based on the random choice tasks, the holdout data suggest that there is marginal benefit from using the Eclipse+, by Golfers, Inc. name.

As a final note on counting analysis, one should not put too much emphasis in the None percentage. We shouldn't conclude that since the None percentage is running at about 25% that roughly three-quarters of these respondents would be expected to purchase a premium ball for their next golf outing. It is our experience that respondents tend to exaggerate their likelihood to purchase or choose products in survey research. The actual proportion of buyers that purchase premium balls would probably be significantly less.

Now that you have spent some time learning about the Choice results using the Counts program, you are ready to run logit analysis and construct a market simulation.

Analyzing the Choice Data Using Logit

Aggregate logit is an older and less advanced technique for analyzing choice results. But, it is a quick way to analyze results, and a recommended starting point for learning about the essential characteristics of your data. There are great advantages to Latent Class and Hierarchical Bayes analysis programs, but for simplicity we'll focus on logit for this tutorial.

Logit analysis estimates an effect, or logit "utility" for each level of each attribute. It also can be used to estimate interaction effects. A utility refers to a degree of worth or preference for a product feature. As with any complex analytical tool, we suggest you learn as much about the technique as you can, to ensure that you interpret the results wisely. Much more information about logit analysis and logit utilities can be found in the online help.

To compute logit, select Analysis | Compute Utilities. First click the Settings button. You should provide the following settings:

Respondents to Include (All)

Respondent Weights (Equal)

Choice Tasks to Include (All Random)

Effects Coding (Main)

Output Precision (5 decimal places)

After you have checked these settings and closed the Logit Settings dialog, click Compute!. The following report is displayed:

Main Effects

Choice Tasks Included: All Random

Total number of choices in each response category:

1 1135 25.22%

2 1377 30.60%

3 1039 23.09%

NONE 949 21.09%

Files built for 300 respondents.

There are data for 4500 choice tasks.

Iter 1 log-likelihood = -5517.00384 rlh = 0.29335

Iter 2 log-likelihood = -5498.70622 rlh = 0.29466

Iter 3 log-likelihood = -5498.66372 rlh = 0.29466

Iter 4 log-likelihood = -5498.66372 rlh = 0.29466

Iter 5 log-likelihood = -5498.66372 rlh = 0.29466

Converged.

Log-likelihood for this model = -5498.66372

Log-likelihood for null model = -6238.32463

------------

Difference = 739.66091 Chi Square = 1479.322

Effect Std Err t Ratio Attribute Level

1 0.52865 0.03219 16.42166 1 1 High-Flyer Pro...

2 0.37167 0.03213 11.56654 1 2 Magnum Force, ...

3 -0.37734 0.03728 -10.12212 1 3 Eclipse+, by G...

4 -0.52298 0.03878 -13.48669 1 4 Long Shot, by ...

5 -0.49336 0.03006 -16.41310 2 1 Drives 5 yards...

6 0.12924 0.02652 4.87399 2 2 Drives 10 yard...

7 0.36411 0.02589 14.06139 2 3 Drives 15 yard...

8 0.66815 0.03219 20.75429 3 1 $4.99 for pack...

9 0.16510 0.03350 4.92893 3 2 $6.99 for pack...

10 -0.09740 0.03527 -2.76111 3 3 $8.99 for pack...

11 -0.73585 0.04092 -17.98437 3 4 $10.99 for pac...

12 -0.02819 0.03825 -0.73693 NONE

The column labeled "Effect" contains the utilities for each level of each attribute. The larger the utility, the more preferred the level. The utilities sum to 0 within each attribute (they are zero-centered). These utilities are used within CBC's market simulator to compute interest (share of choice) among products in competitive scenarios.

The next column displays the standard errors for each logit effect. Earlier under Counting analysis, we saw that Eclipse+ was preferred to Long Shot, but we could not tell at that point if the result was statistically significant. Here, we note that the utility for Eclipse+ is 0.14564 utility points higher than Long Shot (-0.37734 - -0.52298). We also note the standard errors for each of these utilities. The pooled standard error for the difference between these two utilities is equal to the square root of the sum of the squared standard errors, or sqrt (0.03728² + 0.03878²) = 0.0537912. To get the t-value, we divide the difference in the two utilities by the pooled standard error 0.14564 / 0.0537912 = 2.708. If we look this up on a standard t-table, we see that this difference is significant at roughly the 99% confidence interval. Therefore, we conclude that the Golfers, Inc. name is more preferred than the Performance Plus name.

The logit run above reflects main effects only (no interactions). The results from Counts suggested that interaction effects might be significant. Adding interaction terms to the logit model (and the resulting simulator) might significantly improve our ability to predict respondent choices. We will not go into detail in this tutorial regarding how to test for significance of interaction terms in logit. To briefly summarize the process, since we believe that changes in price may act differently on different brands, we decide to rerun the logit model specifying the brand x price interaction. We compare the overall fit (log likelihood) of the model before and after the inclusion of the interaction terms. With this data set, we find that the interaction term improves the fit.

To include the interaction between brand and price in your logit run, click Analysis | Compute Utilities. Click the Settings... button. Make sure your settings are the same as previously outlined in this section. However, instead of checking Main Effects, un-check it and then click Effects... and a grid is displayed.

Checks are displayed along the diagonal of the grid, representing main effects only (no interactions). To specify the interaction between brand and price, additionally check the box in the lowest cell of the left-most column. Click OK. You are returned to the Logit Settings dialog. Click OK again to close the dialog. Then, click Compute!. Specify a name by which to reference that run, such as "Main Effects + A1xA3". Click OK. A new solution is computed including main effects and interactions (it may take a few minutes).

Running Market Simulations

Once you have computed logit utilities, you can use those within a market simulator to predict choices for any combination of products defined using the attributes and levels you measured. You can test what-if scenarios, conduct sensitivity analysis, and predict interest in different product concepts in competitive scenarios.

To open the Market Simulator within SMRT, click Analysis | Market Simulator or double-click Market Simulator from the study Navigator window.

The logit runs that you have saved are displayed in the Utility Runs list. If you saved a run (or multiple runs) while working through the last section of the tutorial, these should appear on the list.

The first time you enter the Market Simulator for a study, the list of Simulation Scenarios is empty. Create a simulation scenario by clicking Add....

When you click the Add... button, the Scenario Specification dialog is displayed.

Recall that one of the reasons for including fixed holdout tasks in your survey was to check the accuracy of the market simulator. We use the simulator to predict the outcome of a choice task that wasn't included when estimating the attribute utilities. To check this, you decide to specify the first fixed product scenario.

The first step is to type a scenario name into the Name field, in the upper left-hand corner of the dialog. Type Fixed Holdout Task 1.

Next, you'll specify the three products that were included in that choice task. The area you type the products into looks like the grid of a spreadsheet. The first column is where you type the product label. As you click the various columns associated with the attribute levels, the level codes will appear in the window below to remind you what codes are associated with each attribute's levels.

The attribute level codes for the three products that were shown in the first holdout fixed task were:

Product Name	Brand	Performance	Price
High-Flyer Pro	1	3	4
Long Shot	4	2	2
Magnum Force	2	1	3

Specify these three products in the product entry grid. To add new rows to the grid (for additional products) click the Insert Product button.

After you have specified these three products, there are a few other things we should do before running the simulation. First, we need to specify a simulation method. For the purposes of this tutorial, the method we'll select is called Share of Preference. This method works well when the products that are specified are basically unique, with little or no overlap in attribute levels used to define those products. The three products we entered are unique in terms of brand, performance and price, so our situation seems to fit this well. If some products shared a level specification (e.g. having the same performance), we would probably favor using the default Randomized First Choice method.

Next, we need to tell the simulator to estimate a share of preference associated with the None product. (We generally suggest not estimating share of preference for the None product; but in this case, where we want to predict responses to a holdout task that included a None choice, it makes good sense). To do so, click the Advanced Settings... button. (The settings on this dialog are described in more detail in online Help).

Specify a "None" Weight of 1, and click OK to close the dialog.

Click OK again to close the Scenario Specification dialog and return to the main Market Simulator dialog.

Choose a utility run to use for your simulation. You should have saved a run that computed main effects plus the interaction between brand and price. Select that run by highlighting it in the list of Utility Runs.

To simulate shares for a scenario you have specified, place a check mark in the box next to the scenario name in the Simulation Scenarios list, and then click Compute!.

The following report is displayed (we've inserted commentary within brackets):

 Scenario: Fixed Holdout Task 1
 Utility Run: Main Effects + A1xA3

 Average Utility Values
 Rescaling Method: Diffs

 Total
 High-Flyer Pro, by Smith and Forester 47.62
 Magnum Force, by Durango 36.42
 Eclipse+, by Golfers, Inc. -32.49
 Long Shot, by Performance Plus -51.56

 Drives 5 yards farther than the average ball -41.06
 Drives 10 yards farther than the average ball 10.52
 Drives 15 yards farther than the average ball 30.54

 $4.99 for package of 3 balls 59.03
 $6.99 for package of 3 balls 18.33
 $8.99 for package of 3 balls -7.17
 $10.99 for package of 3 balls -70.19

 High-Flyer Pro... $4.99 for package of 3... -4.60
 High-Flyer Pro... $6.99 for package of 3... -6.50
 High-Flyer Pro... $8.99 for package of 3... 0.75
 High-Flyer Pro... $10.99 for package of ... 10.35
 Magnum Force, ... $4.99 for package of 3... -11.64
 Magnum Force, ... $6.99 for package of 3... -20.31
 Magnum Force, ... $8.99 for package of 3... 1.08
 Magnum Force, ... $10.99 for package of ... 30.87
 Eclipse+, by G... $4.99 for package of 3... -2.01
 Eclipse+, by G... $6.99 for package of 3... 14.19
 Eclipse+, by G... $8.99 for package of 3... -5.78
 Eclipse+, by G... $10.99 for package of ... -6.40
 Long Shot, by ... $4.99 for package of 3... 18.24
 Long Shot, by ... $6.99 for package of 3... 12.61
 Long Shot, by ... $8.99 for package of 3... 3.96
 Long Shot, by ... $10.99 for package of ... -34.82

 None -0.38

<<The above average utilities are rescaled logit utilities using the zero-centered "diffs" method. The diffs method rescales utilities so that the total sum of the utility differences between the worst and best levels of each attribute across attributes (main effects) is equal to the number of attributes times 100. Note: the attribute utilities are influenced by the number of respondents in the simulation and respondent weighting, but are not affected by the product specifications you enter. After you have seen these once for a particular group of respondents, you may choose to omit them in subsequent simulations by un-checking the Display Utilities box in the Scenario Specification dialog.>>

Product Simulation Settings

Simulation Mode: Simulation

Model: Share of Preference

None Weight: 1

Exponent: 1

Product Specifications

Brand Perf... Price

High-Flyer Pro 1 3 4

Long Shot 4 2 2

Magnum Force 2 1 3

<<Above are the product level codes you specified for the three products in this simulation. Below are the simulated shares of preference (choice) for these products.>>

Shares of Preference for Products

Total

High-Flyer Pro 30.99

Long Shot 22.16

Magnum Force 22.00

None 24.85

In the table below, we've summarized the actual and predicted choices shares for the first holdout task. Notice that the simulated shares of preference are very similar to the actual share of choices respondents gave to fixed holdout task number one (which we analyzed earlier using Counts):

Product Name	Actual Choice Shares	Simulated Choice Shares	Absolute Error
High-Flyer Pro	30.67	30.99	0.32
Long Shot	23.67	22.16	1.51
Magnum Force	21.33	22.00	0.67
None	24.33	24.85	0.52

To quantify how closely the simulated shares match the actual choice shares, we've computed the absolute value of the difference between the actual and simulated shares in the last column. If we average the errors in the last column, we find the Mean Average Error (MAE) is 0.76. (We should note that the MAE for this example is much lower than is generally observed when comparing simulation predictions versus actual holdout shares).

We won't take the space here to show the MAE computation for holdout task #2, though you may decide to specify another simulation scenario and compute predicted shares for that task.

You can use MAE (or Mean Squared Error or Chi Square) to compare the results of different simulation models (i.e. Logit versus Latent Class or HB). You can also use it to tune the scaling parameter (exponent) or other advanced settings covered in more depth in the online help.

The Market Simulator is a powerful tool for testing nearly an unlimited number of possible market scenarios. You can use the simulator to answer the strategic questions related to the fictitious golf ball situation posed at the beginning of this unit.

For example, the main question facing Performance Plus was whether they could hope to compete with the two existing performance balls in this market. The simulation results suggest that, if they were able to provide the specified performance at the simulated price and could achieve the level of distribution and awareness of the existing brands (whose specifications and prices didn't change), they could expect a market share roughly equal to Magnum Force, and lower than the leader, High-Flyer Pro.

A secondary question was whether they should market their new golf ball under the "Eclipse+, by Golfers Inc." name. If you specify the second holdout task as a new simulation scenario, the following shares result (displayed next to the previous simulation for fixed holdout task #1):

                   Simulated Shares of Choice

         Fixed Task 1               Fixed Task 2
         High-Flyer Pro  30.99      High-Flyer Pro  29.20
         Long Shot       22.16      Eclipse+        26.65
         Magnum Force    22.00      Magnum Force    20.73
         None            24.85      None            23.42

By marketing their ball under the "Eclipse+, by Golfers, Inc." name instead of "Long Shot, by Performance Plus", share of choice increases from 22.16 to 26.65, representing a 20% relative increase. If you have a good memory, you'll note that this conclusion is similar to what we inferred from Counts data, and by comparing the actual choices of the two fixed holdouts. The simulated shares, however, should be both more reliable and more accurate. They are not subject to some inherent weaknesses of counting analysis, and they leverage much more data (15 randomized tasks) than isolating only the choices given to the two fixed holdout tasks.

It is important to remember that the shares of preference resulting from conjoint predictions are not equivalent to actual market shares, and often look quite different. Many other factors in the real world factor into market shares and cannot be measured and reflected solely by conjoint data. Conjoint assumes perfect information, equal distribution and availability, and that each respondent is in the market and able to purchase. Conjoint results reflect the potential market acceptance, given proper promotion, distribution and time.