MaxDiff Tutorial and Example

MaxDiff Tutorial and Example

Introduction

This tutorial leads you step-by-step through a simple MaxDiff study, from thinking about a research problem and planning a questionnaire to entering a list of MaxDiff items in SSI Web, generating and testing the questionnaire, and analyzing the results.

SSI Web is a powerful program, and this tutorial only touches on the basics for running a MaxDiff experiment. You'll become aware of different or more sophisticated approaches by reading other sections in the on-line help and by experimenting with SSI Web.

(Note: this example is based on a presentation delivered at the 2003 Sawtooth Software Conference by Amanda Kraus, Diana Lien, and Bryan Orme. Though many of the items we use in this example are identical to the ones used in the conference presentation, the data we have developed for this example are purely fictional. The suggested steps represent one approach to this type of research problem, but may not necessarily reflect the single "best" way to conduct such a research study using the SSI Web system.)

The Research Problem

As the economy has improved (2003 perspective), the Navy has faced declining reenlistment rates among its sailors. The Navy is looking for ways to improve the reenlistment rate. Previous research has shown that reenlistment terms, quality of life issues, and the work environment are all significant factors when sailors face the reenlistment decision. Pure economic aspects matter as well, such as the amount of the reenlistment bonus. You have been brought on as a consultant to help the military design and execute a research study to prioritize a list of possible job and quality of life enhancements as well as reenlistment terms. The Navy is keenly interested in how reenlistment bonuses may affect sailors' desire to enlist relative to other non-monetary issues. Those improvements which are most likely to increase the reenlistment rates at the lowest expense to the military should be implemented first. They would like to implement this study over the Web, and expect that between 1,000 and 2,000 sailors might complete the study.

At first, you consider whether a conjoint method such as CBC would be appropriate. You ask your military client, Jane, if the resulting model needs to be able to project the likely reenlistment rate when combinations of job enhancements are taken together. Jane assures you that although that type of analysis would be of interest to military planners, the primary consideration is the ability to prioritize the items from most to least effective in terms of improving reenlistment rates. The planners need to be able to directly compare the effect of each job enhancement or monetary bonus.

After pondering the problem, you recognize that a strength of MaxDiff is its ability to place multiple items on the same scale using a trade-off methodology. With conjoint analysis (such as CBC), the client would not be able to directly compare the preference for levels of one job-related item to another. MaxDiff doesn't formally support adding the preference scores across items and predicting the overall benefit of the job package or likelihood of reenlistment given a specific job package (as would CBC), but that is not a requirement for the study. Therefore, you decide to use MaxDiff.

Developing a List of Items

You meet with Jane to discuss the elements (items) to be studied in the MaxDiff exercise. Jane explains that a series of focus groups had been conducted over the last four months wherein the reasons for reenlistment versus leaving the Navy were discussed. Hundreds of ideas had been presented, but there seemed to be some common themes. Furthermore, Jane had discussed the myriad ideas with military planners. Many of the ideas were clearly off-the-table (such as "king-sized beds on submarines," "more parties," and "supermodels as supervisors"). However, there were a number of areas of overlap: job conditions that the military was in the position to change and those that also were on the minds of the sailors.

The list included the following:

Length of reenlistment obligation

Location or duty assignment guarantees for next assignment

Faster promotion

Living conditions when in port (2-person barracks vs. 4-person barracks)

Increase in sea pay

Percent of on-job time using your skills and training

Time off each week allowed for on-line college coursework

Online career counseling services

Personal counseling services

Reenlistment bonus

This looked like a good start, but it seemed obvious to both you and Jane that some further refinement and quantification of benefits was needed. The items needed to carry specific meaning to the sailors and the results needed to be actionable for the military planners. Most of the items seemed to require multiple levels, whereas some seemed to stand on their own. After thinking about the reasonable items (and some back-and-forth with Jane's internal clients, the military planners) a final list was drawn up:

Factors Affecting Reenlistment Decision:

3-year reenlistment obligation

5-year reenlistment obligation

Location guarantee for next assignment

Duty guarantee for next assignment

Both location and duty guarantees for next assignment

Get promoted 6-months sooner than expected

Live in 2-person barracks when in port

Live in 4-person barracks when in port

Choice between living in 2-person barracks or 4-person barracks when in port

$50 per month pay increase when out at sea

$75 per month pay increase when out at sea

Half of on-job time using your skills and training

Three-quarters of on-job time using your skills and training

Time off each week allowed for on-line college coursework

Receive online career counseling services

Receive in-person personal counseling services

Receive phone-based personal counseling services

Current reenlistment bonus ($2,500)

Increased reenlistment bonus ($5,000)

Increased reenlistment bonus ($10,000)

Next, you worked with Jane to decide which items would need to be prohibited from displaying with each other versus which could freely combine with all other items within the same set. For example, it would not make sense to ask sailors to choose between a $50 pay increase versus a $100 pay increase when out at sea. Every rational sailor would choose higher pay over lower, so this would be a wasted comparison. Other items such as living in a 2-person barracks versus a 4-person barracks didn't seem to have a rational preference order (though having a choice between 2-person barracks or 4-person barracks would seem to be preferred to no choice). Perhaps some sailors would prefer to live with more people, whereas others would prefer just one roommate.

You organized the list into groupings of items (listed as groups A through F), where the items within groups A through E would require at least some prohibitions (restrictions that they could not be displayed together within the same choice set).

A) Prohibit these two items:

3-year reenlistment obligation

5-year reenlistment obligation

B) Prohibit third item with appearing with either item 1 or item 2:

Location guarantee for next assignment

Duty guarantee for next assignment

Both location and duty guarantees for next assignment

C) Prohibit third item with appearing with either item 1 or item 2:

Live in 2-person barracks when in port

Live in 4-person barracks when in port

Choice between living in 2-person barracks or 4-person barracks when in port

D) Prohibit these two items:

$50 per month pay increase when out at sea

$75 per month pay increase when out at sea

E) Prohibit these items from appearing together:

Current reenlistment bonus ($2,500)

Increased reenlistment bonus ($5,000)

Increased reenlistment bonus ($7,500)

F) No prohibitions with other items:

Half of on-job time using your skills and training

Three-quarters of on-job time using your skills and training

Get promoted 6-months sooner than expected

Time off each week allowed for on-line college coursework

Receive online career counseling services

Receive in-person personal counseling services

Receive phone-based personal counseling services

Scripting the Questionnaire

A real research project would involve other questions about each sailor, including gender, current rank or job specification, current location, inclination to reenlist, etc. Standard survey questions (such as select questions with radio buttons or check-boxes) can be developed within the SSI Web platform (of which MaxDiff is a subcomponent). Rather than take time within this tutorial to add such classification questions, we'll focus on the MaxDiff section of the survey. Interested users can look to the CiW Tutorial within this same online help and manual for further help with standard survey questions.

You decide to include the following introductory text leading into the MaxDiff questions:

Start:

Thank you for taking time to help the Navy consider ways to improve your working environment and career development. Our goal is to make the Navy a satisfying career for you and to improve your desire to reenlist when the decision arises.

We would like you to imagine that you are facing the decision whether to reenlist or not--today. Please think about the issues involved, including future career planning, family/relationship considerations, job location, and compensation.

Click the Next button below to continue...

IntroMaxDiff:

If you were making the decision to reenlist or not today, there would certainly be many factors that affect your decision to reenlist or not. Real life decisions are not always easy.

We want to gain some insight into how you make this decision by asking you to evaluate a series of tradeoff scenarios. In each scenario, we'll show you four possible factors affecting your career in the Navy and the reenlistment decision.
We'll ask which factor (among this set of four) would make you most want to reenlist, and which would make you least want to reenlist.

We need to ask you repeated tradeoff scenarios (involving different factors each time) so that we can learn what is truly important to you when deciding whether to reenlist.

Layout for MaxDiff Questions

Next, you need to decide the layout of the MaxDiff questions. Research has suggested that showing more than about five items at a time in MaxDiff questions may not be very beneficial. Some of the items in your study are a bit on the wordy side, so you decide to show just four items at a time.

The MaxDiff documentation suggests showing each item between 3 to 5 times on average per respondent across the sets. With 20 items in the study, the number of questions to display so that each item is shown on average 3 times per respondent is equal to:

   3(K/k)

   where K is the total number items in the exercise, and k is the number of items per set

Therefore, you would need to ask each respondent a minimum of 3(20/4) = 15 sets. The "nice to have" number of sets would be 5(20/4) = 25. After discussing with the client, you decide that respondents should be able to complete 20 sets with little problem (each item displayed on average 4 times to each respondent). You estimate that each set will take about 20 seconds to complete (prior research presented at the Sawtooth Software Conference reports that figure), for a total average interviewing time of between 6 to 7 minutes for the MaxDiff section.

To try to keep respondents engaged and working productively through the MaxDiff exercise, you plan to show respondents a progress bar and also provide a count-down message after every five MaxDiff questions.

Here's an example of the MaxDiff question layout you plan to use:

After every five MaxDiff questions, you plan to show some text that provides encouragement and a countdown, such as:

ProgressCounter:

Good work so far. We know that some of these tradeoffs are challenging.

We need to ask you to evaluate 15 more sets so that we can better determine how you make your reenlistment decision.

And at the end, you want to display a screen that thanks respondents, and then immediately sends them to a Navy site www.navy.mil.

Finish:

Thank you very much for your help. Your data will be used to help Navy planners as they work to make the Navy an even better career choice.

Click the Next button to finish submitting your data and to return to the Navy's official website.

Using SSI Web to Write the Questionnaire

Start by opening the Sawtooth Software SSI Web program. If you have a standard installation, you start it by clicking Start | Programs | Sawtooth Software | Sawtooth Software SSI Web. "SSI" stands for "Sawtooth Software Inc." and "Web" conveys the idea that this software is for developing Web-based surveys (although SSI Web can also be used for standalone computer interviewing with its CAPI interviewing module.) MaxDiff is one component within that software program. (Although not emphasized here, MaxDiff may be used for conducting paper-based MaxDiff projects.)

After starting SSI Web, choose File | New Study…. The New SSI Web Study dialog appears, with your cursor active in the Name field.

The Location field lists the folder (directory) that SSI Web currently is using to store studies. You can use any folder you like for this tutorial project. You can browse to and create new folders by clicking the Browse... button to the right of the Location field, and then (after browsing to the folder in which you want to create a new sub-folder to contain this study) by clicking the "new folder" icon. Each studyname in SSI Web has a maximum number of twenty-four characters (either letters or numbers), and SSI Web requires that the study be located within its own folder. The folder that SSI Web automatically creates for your study also carries the same name as the study. For this tutorial, you might create a name such as navyproject. From the New SSI Web Study dialog, specify navyproject as the studyname.

Click the OK button. You are returned to the main menu, and a new "Study Navigator" window is displayed.

As you work with this study, items appear in the Study Navigator window, listing the functional areas you have been using or that are now available to you. This list of items provides a Shortcut link to access those parts of SSI Web. Alternatively, you can access those same areas by clicking icon buttons on the toolbar or by using the pull-down menus.

Adding the Introduction Screens to the Survey

(If you are already familiar with the Start (passwords) question, Text/HTML Filler and Select questions within SSI Web surveys, you may choose to skip forward to the section entitled "Adding MaxDiff Questions to the Survey").

Let's enter the introductory text using the Write Questionnaire dialog. You can access the Write Questionnaire dialog by selecting Compose | Write Questionnaire… or by clicking the "pencil" icon

on the toolbar.

The introductory text and survey questions would most likely be initially developed within a word processing document. Assuming you really had such a document, you might press Ctrl-C to copy, and Ctrl-V to paste the information into SSI Web. We suggest you simply copy-and-paste the text within this document into your SSI Web questions rather than re-type the text (if viewing this document with Acrobat Reader, you can use the "text select" icon

from the Acrobat Reader toolbar.)

The first question to compose is the introductory screen:

Start:

We'll insert this introductory screen in the Start question that is automatically the first question of any SSI Web survey. The Start question is also the place where you would ask respondents to type their passwords to access the survey. We'll not use passwords for this sample survey.

From the Write Questionnaire dialog open the Start question by highlighting it in the list of questions and clicking Edit.... Alternatively, you can double-click the Start question, and it automatically opens. The Questionnaire Access and Passwords dialog is displayed::

Most question types in SSI Web have "Header 1", "Header 2," "Body" and "Footer" sections. (The Start question has all but the "Body" section.) These are "text" areas in which you can insert any text (including HTML). When the question is viewed with a web browser, the sections are organized roughly as follows:

For this introductory screen (that includes three paragraphs of information), it seems to make sense to place the first paragraph in the "Header 1" area, the second paragraph in the "Header 2" area, and the third paragraph in the "Footer" area.

Type (or cut and paste) the following text for the Start question into the text areas in the Start question. Put the first paragraph in Header 1, the second paragraph in Header 2, and the third paragraph in Footer.

We would like you to imagine that you are facing the decision whether to reenlist or not--today. Please think about the issues involved, including future career planning, family/relationship considerations, job location, and compensation.

Click the Next button below to continue...

The main dialog only shows a portion of the text, but you can click the "pencil" icon

adjacent to each text area to expand the text editing window.

To preview how the question will look when it runs on the respondent's computer using Microsoft Internet Explorer, click the Preview button. The question is displayed in "Preview" mode (no data are saved, and no special "error checking" or "validation" is performed for the question).

SSI Web automatically places paragraph breaks (extra blank lines) between the Header 1, Header 2, and Footer sections. If you put all the text in a single section, you may see that when the web browser interprets the text, it all runs together without any blank lines between the paragraphs (unless you insert some HTML instructions to force blank lines between paragraphs). We'll talk about using HTML within your text to take greater control over the layout, font, and style later in this tutorial.

Click the OK button on the Preview window to close that window and return to the previous SSI Web dialog.

After viewing the text in preview mode, you might decide that it is really too small, or that you want to make the text bold. You can modify the font and sizes of the font using the Paintbrush

icon. Or, you can select a different Style (selecting styles was described in the 45-minute tutorial for SSI Web). After changing any setting, click Preview again.

After you are happy with the layout of the introductory screen, click OK to return to the Write Questionnaire dialog.

Remember that when you add a new question to the List of Questions, it is inserted directly below the question (or page break) that was highlighted when you clicked the Add... button. If you need to move a question once it has been added to the List of Questions, simply highlight the question to be moved and click the

buttons to move the questions within the list (you can also highlight a question and click Ctrl-X or Ctrl-C to cut or copy and Ctrl-V to paste questions within the current questionnaire, or even to another SSI Web study.)

Now, add a the second introductory screen on a separate page. The text for that screen is as follows:

If you were making the decision to reenlist or not today, there would certainly be many factors that affect your decision to reenlist or not. Real life decisions are not always easy.

We want to gain some insight into how you make this decision by asking you to evaluate a series of tradeoff scenarios. In each scenario, we'll show you five possible factors affecting your career in the Navy and the reenlistment decision.

We'll ask which factor (among this set of five) would make you most want to reenlist, and which would make you least want to reenlist.

We need to ask you repeated tradeoff scenarios (involving different factors each time) so that we can learn what is truly important to you when deciding whether to reenlist.

Note that there already is a page break following the Start question. Highlight the page break and click Add.... Select Text/HTML Filler as the question type with IntroMaxDiff as the question name. Click OK to add and then edit the question by double-clicking it from the list of questions.

This introductory screen has four paragraphs and it also includes two words that need to be underlined (most and least). Therefore, we'll show you how to how to format text in HTML. While we're showing you how to add underlining to text, we will show you how to insert paragraph breaks as well.

Copy and paste the full text for the IntroMaxDiff question from this document into the Header 1 field. Next, click the "pencil" icon

adjacent to the Header 1 text area to expand the text editing window. Highlight the first full paragraph with the mouse and click the "paragraph break" icon

which inserts the HTML text at the beginning of the paragraph and at the end of the paragraph. Repeat the same steps for all four paragraphs.

Next, we need to underline the words most and least. Highlight "most" and click the "underline" icon

. Repeat the same for the word "least". When you are finished, the dialog should look like:

Click OK to close the text editor dialog and Preview to see how it is displayed in the browser window.

At this point, we are ready to add the MaxDiff questions to the questionnaire.

Adding MaxDiff Questions to the Survey

In this section, we'll add the MaxDiff questions to the survey, directly following the two introductory screens we previously added. From the Write Questionnaire dialog, insert a page break beneath the IntroMaxDiff question. While the page break beneath IntroMaxDiff is highlighted, click Add... and select MaxDiff as the Exercise type. Specify a name for this exercise, such as Reenlist. (Note: you can have more than one MaxDiff exercise within the same SSI Web questionnaire. In this tutorial, we'll only insert a single exercise.) Click OK and the MaxDiff Exercise dialog is shown.

The MaxDiff Exercise dialog has six tabs:

Question Text (here you type the text to display directly above and below the question)

Label Text (here you specify the text to display directly next to the item text)

Items (here you specify the list of items in the exercise)

Format (you may format your questionnaire, including widths and colors from this tab)

Design (the number of items to display per set and number of sets is specified here)

Skip Logic (this dialog is inactive and only present for consistency with other dialogs in the SSI Web system)

We'll begin by specifying the 20 items used in this exercise within the Items tab. Click the Items tab. A list box is displayed which allows you to specify a new list of items to be used in this SSI Web study. Or, you can select from an existing list of items. Since we have not yet specified a list within this study, we'll add a new one. By default, the list name is provided as ReenlistList. Let's go ahead and use this default name.

(Note that other options such as making items on the list "Other Specify" items or "None of the Above" options are available. These are for consistency with other dialogs in the SSI Web System, but are of no use in MaxDiff exercises.)

From this dialog, we could click the Add... button and type the items individually, but it is easier to copy-and-paste the list of items from an existing document. Copy-and-Paste the list of items by highlighting the items below...

3-year reenlistment obligation
5-year reenlistment obligation
Location guarantee for next assignment
Duty guarantee for next assignment
Both location and duty guarantees for next assignment
Get promoted 6-months sooner than expected
Live in 2-person barracks when in port
Live in 4-person barracks when in port
Choice between living in 2-person barracks or 4-person barracks when in port
$50 per month pay increase when out at sea
$75 per month pay increase when out at sea
Half of on-job time using your skills and training
Three-quarters of on-job time using your skills and training
Time off each week allowed for on-line college coursework
Receive online career counseling services
Receive in-person personal counseling services
Receive phone-based personal counseling services
Current reenlistment bonus ($2,500)
Increased reenlistment bonus ($5,000)
Increased reenlistment bonus ($10,000)

...and pressing Ctrl-C to copy them into the clipboard, and then switching to the Items tab within the SSI Web System and clicking the Paste list member(s) from the clipboard

icon. The 20 items are all automatically inserted within the list.

Next, we'll specify the text to display above the MaxDiff sets into the Question Text tab. Click the Question Text tab. Some default text is available within this question, but we'll replace it with wording more appropriate for our study. Paste the following text into the Header 1 field:

Imagine you were making the decision to reenlist in the Navy today.

Which of the following four factors would make you most and least likely to want to reenlist?

Insert paragraph breaks and underlining as we showed you earlier in this tutorial. Delete the wording in Header 2 and the Footer.

We are almost finished formatting the basics for our MaxDiff exercise. Click the Label Text tab, and type the labels to display over the radio buttons respondents click to indicate best and worst items. The text to use is as follows:

Best label: Makes me Most want to reenlist

Worst label: Makes me Least want to reenlist

Next, we need to tell SSI Web how many MaxDiff questions (sets) to display in the questionnaire. Click the Design tab. Basic design settings for your MaxDiff exercise are displayed:

You may recall that we previously decided that each set would present 4 items at a time and that the respondent would receive 20 questions (so each item was shown four times on average). Specify 4 in the Number of Items per Set (Question) field. Specify 20 in the Number of Sets (Questions) per Respondent field.

At this point, we might preview the MaxDiff question to see how we are doing so far. Click the Preview button, and you may see something like this:

Note that by default the preview displays the first four items in our list therefore some of the items that we wanted to prohibit are being displayed together. We can add the prohibited combinations under this same Design tab.

Inserting prohibitions is a capability provided within the Advanced Settings on this dialog. (After retiring the Preview dialog) click Show Advanced Settings to reveal additional controls, including the Prohibitions... button.

Click the Prohibitions... button. We need to prohibit the following pairs of items for our exercise:

Prohibit this item...	...from appearing with this item.
3-year reenlistment obligation	5-year reenlistment obligation
Location guarantee for next assignment	Both location and duty guarantees for next assignment
Duty guarantee for next assignment	Both location and duty guarantees for next assignment
Live in 2-person barracks when in port	Choice between living in 2-person barracks or 4-person barracks when in port
Live in 4-person barracks when in port	Choice between living in 2-person barracks or 4-person barracks when in port
$50 per month pay increase when out at sea	$75 per month pay increase when out at sea
Current reenlistment bonus ($2,500)	Increased reenlistment bonus ($5,000)
Current reenlistment bonus ($2,500)	Increased reenlistment bonus ($10,000)
Increased reenlistment bonus ($5,000)	Increased reenlistment bonus ($10,000)

Click an item to prohibit from the list on the left, then click the item to prohibit from the list on the right. Continue picking items until all prohibitions have been specified. Click the OK button when finished.

Now, we are ready to "generate the design." The design is all the combinations of items that respondents will see across their 20 choice sets (questions). (Note: it may take a few minutes for the computation to finish.) The sets need to be chosen carefully, so each item appears about an equal number of times, and that each item is shown about an equal number of times with every other item (non-prohibited item, that is). Click Generate Design, and the software tries 1000 separate attempts to create a balanced experiment (using a different random number start each time). The best attempt is selected, and the results are shown to you in a design report.

When the computation finishes, you are shown a report (which you have the opportunity to save to a text file if you wish).

We'll explain the report, one step at a time. First, design specifications are reported:

Number of Items (Attributes): 20
Number of Items per Set: 4
Number of Sets per Respondent: 20
Number of Versions: 300
Number of Iterations: 1000
Random Number Seed: 1
Number of Prohibitions: 9
Iteration 612 was chosen

Above, we see that the software made 1000 total attempts (iterations) to generate a design plan, and the 612th attempt was chosen as "best." In total, the software created 300 versions of the questionnaire (where each version features different item combinations within the sets) to display to respondents. Multiple versions of the questionnaire are generated to improve the stability of the item scores and also to reduce psychological order and context effects that might occur if every respondent received the same combinations of items. Although you plan to interview 1000 or more respondents, it would probably be overkill to generate even more unique questionnaire versions. (Some might argue that 300 versions is already overkill!) Additional versions would provide imperceptibly little measurable benefit, though the software lets you specify up to 999 versions. Each respondent will receive one of these versions of the questionnaire, and many of the questionnaires will therefore be repeated.

Below, the one-way balance is reported. This shows how many times each item is displayed across all 300 versions of the questionnaire plan, where each version had 20 sets displaying 4 items at a time. If the design was perfectly balanced, each item would be displayed 1200 times. As can be seen, the design is nearly perfect with respect to one-way frequencies:

One Way Frequencies:
  Item  Times Used
----------------
    1    1200
    2    1200
    3    1200
    4    1200
    5    1200
    6    1200
    7    1200
    8    1200
    9    1200
   10    1200
   11    1201
   12    1200
   13    1200
   14    1200
   15    1200
   16    1200
   17    1200
   18    1199
   19    1200
   20    1200

Mean = 1200.000000
Std Dev. = 0.316228

The quality of the design is summarized as the standard deviation across the 20 reported frequencies, or 0.316228. This number is used to classify the quality of the design for multiple attempts (iterations) the software makes. Attempts that demonstrate a lower standard deviation across the frequency counts are deemed better.

Next, the report summarizes how many times each item appeared together with every other item within sets. Ideally, each item would appear an equal number of times with every other item. You may recall that we prohibited nine combinations from occurring, so there cannot be perfect balance. These prohibited combinations are shown with 0 frequency in the table below (for example, item 1 shown with item 2), highlighted in red.

maxdifftwoway

Although the balance in the non-prohibited cells is not nearly as good as the one-way counts (the nine prohibitions certainly inhibited the MaxDiff designer from finding a balanced design), the results are still quite acceptable in practice. In the two-way table, each non-prohibited cell is represented a minimum of 162 and a maximum of 217 times. How would you know if you were in deep trouble due to prohibitions? If you have too many prohibitions, you would receive an error message reporting that it is impossible for the designer to ensure connectivity across items in the study. If you have to prohibit a single item from appearing with more than half of the other items in the study, then this may be excessive. Again, we'd like to stress that it is much more important that the one-way frequencies be nearly balanced than to achieve approximate balance in the two-way frequency table.

(Note for advanced users: We were interested in the potential loss in design efficiency resulting from these nine prohibitions, so we performed a simple test. We assumed 250 respondents receiving this study as just designed versus the same study in which there were no prohibitions. The design with no prohibitions had perfect one-way balance, and nearly perfect two-way balance (low of 189 and high of 190 counts per two way cells). Upon estimating MNL parameters and studying the standard errors, we determined that the difference in overall design efficiency between these two designs was only 1%. For those familiar with conjoint analysis, the important point is that prohibitions are much less detrimental to MaxDiff designs than conjoint designs. That is because prohibitions in conjoint are within-concept whereas prohibitions in MaxDiff are between-concept. And, the nine prohibitions in MaxDiff eliminate a much smaller proportion of the full-factorial combinations than would nine prohibitions in a typical conjoint plan where each attribute has three to five levels.)

Finally, the report shows how often each item appears in the four positions within sets (top, bottom, etc.). This type of balance has no bearing on statistical design efficiency. But, balance here helps reduce psychological order effects (e.g. we wouldn't want a given item to always appear in the top position each time). The positional balance for our design is nearly perfect:

Positional Frequencies:
          | Pos.       1       2       3       4
----------+-------------------------------------
Item     1|          300     300     300     300
         2|          300     300     299     301
         3|          299     301     300     300
         4|          300     299     301     300
         5|          300     300     301     299
         6|          300     301     300     299
         7|          300     300     300     300
         8|          300     300     300     300
         9|          300     300     300     300
        10|          300     299     301     300
        11|          300     300     301     300
        12|          300     300     300     300
        13|          300     300     299     301
        14|          300     301     299     300
        15|          300     300     300     300
        16|          300     300     300     300
        17|          300     300     299     301
        18|          301     299     300     299
        19|          300     300     300     300
        20|          300     300     300     300

Mean = 300.000000
Std Dev. = 0.524404

At this point, we have the basics programmed for this MaxDiff study. You can click OK to return to the Write Questionnaire dialog, where you'll see the twenty MaxDiff questions automatically inserted within your project:

Inserting the "Progress" and "Finish" Questions

When respondents have completed five MaxDiff questions, you had wanted to display a screen that says:

ProgressCounter:

Good work so far. We know that some of these tradeoffs are challenging.

We need to ask you to evaluate 15 more sets so that we can better determine how you make your reenlistment decision.

Insert a new Text/HTML Filler question by highlighting the Page Break directly beneath the Reenlist_5 question, and then clicking Add.... Select Text/HTML Filler as the question type and name the question ProgressCounter. Copy-and-paste the text for this question from this document and format the text into paragraphs. Preview the question to make sure it looks as expected. Click OK to return to the Write Questionnaire dialog, and insert another page break directly following this ProgressCounter question.

If you would like, you can insert similar "progress screen" reminders after the 10th and 15th MaxDiff questions.

Last, we need to insert the final screen in which we tell respondents that they have completed the survey and link them to the Navy's official site.

Finish:

To do this, scroll to the bottom of the list of questions on the Write Questionnaire dialog. Highlight the final question in the list (Reenlist_20) and insert a page break. Then, click the Add... button and specify a Text/HTML Filler question named Finish.

Copy-and-paste the text for this question from this document, implement any formatting you desire. Click OK to return to the Write Questionnaire dialog.

Next, insert a page break directly following the Finish question. Then, insert a new Terminate/Link question directly following that entitled FinishLink. No text should be shown on this page. This page simply redirects respondents immediately to the Navy's website. While editing the FinishLink question, click the Settings tab. Check the Terminate Respondent box indicating that respondents seeing this page should be terminated and marked as completed records. Then, check the second Link to a Questionnaire or Website box and type within the provided field so that the respondent is sent to http://www.navy.mil. When you are finished, that dialog should look like:

Page Layout and Study Settings

It may be helpful at this point to review how SSI Web breaks the survey into separate pages, and some basic global settings that affect the look and functionality of your SSI Web questionnaire. We also want to add a progress bar to our survey. The Write Questionnaire dialog also displays the page breaks that are currently set for your questionnaire.

Return to the Write Questionnaire dialog. This dialog shows how the various questions we've specified (or that SSI Web has automatically added to the list) are arranged across different pages. Page breaks are indicated by "---------------<Page>".

From the Write Questionnaire dialog, click the Survey Settings icon

(alternatively, click Compose | Survey Settings...). This dialog controls global aspects of the questionnaire (meaning that the settings here control the general behavior of the questionnaire rather than page- or question-specific aspects.) Examples include the fonts and colors used in the questionnaire, the wording for the "Next" button, and the global header and footer that can appear at the top or bottom of each page of the survey. The details of this dialog are beyond the scope of this unit, but you can learn more about the items on the dialog by pressing the F1 key. However, we do wish to insert a progress bar. This is easily done by clicking the Progress Bar tab, and checking the Automatically show the progress bar box with the position selected as Page Footer.

Save your changes and exit the Write Questionnaire dialog.

Adding Polish and Style
You have probably noted that the survey we've created is pretty bland looking. We can add some polish and style by clicking the Survey Settings icon

from the Write Questionnaire dialog (or by clicking Compose | Survey Settings from the main menu).

First, let's select a style from the Styles tab. Select a style you wish to use, and click Use Selected at the bottom of the dialog to implement the style. We'll select Panama as the style, but you might want to experiment with other styles.

On the General Format tab, you can select to use graphical Submit, Previous, and Select/Checkbox buttons. A library of buttons is provided, available in the C:\Program Files\Sawtooth Software\SSI Web\graphics folder.

On the Headers and Footers tab, you can specify a Header and a Footer. We'll specify Reenlistment Questionnaire as the header text.

The Progress Bar tab lets you add a progress bar to your survey. We'll click the check box to add a progress bar to the page footer.

When you go back and test run the survey, your survey should have a bit more polish. Under Panama style, our survey now looks like:

Test Running Your Survey Using "Local Test Server"

Although the Preview Survey function is nice, you generally will want to test your survey in runtime mode, with any skip patterns, response verification, and randomizations in place. You could upload your files and the Perl scripts to the Web Server, but SSI Web provides a more convenient way to test run your survey locally on your PC or laptop.

When SSI Web was installed on your computer, web server software called "Apache" was also installed. Apache lets your computer run the questionnaire just like a remote Web Server would.

Local Test Server

From the Write Questionnaire dialog, Click the Test Survey button

and select to Test Survey Locally.

Your browser opens up the first survey page. This is your survey, as it would appear to a respondent over the Internet (there are slight visual differences from browser to browser).

To close the survey, simply close the browser window (using the X in the upper-right hand corner). To run the survey again, click the Test Survey button

.

After you are convinced that the survey is functioning properly, you should also pretest your survey among your colleagues. They can give you feedback regarding the usability of the survey, and you can examine the resulting Counts data or perhaps even compute individual-level scores using the Analysis | MaxDiff Scores dialog (assuming you have enough data) to make sure the data at least have face validity.

At this point, you would set up the study on your server (or use the hosting services provided by Sawtooth Software). Server setup is beyond the scope of this tutorial, but is covered in this SSI Web manual. You would invite the respondents to complete the survey (probably using an email message with a clickable link).

Data Management

SSI Web includes an Admin Module so that you can monitor or access your project from any computer connected to the Web. The Admin Module is password protected, and your passwords for access are specified on the Field | Web Server Management dialog. SSI Web generates random administrative access passwords whenever you create a new study, but you can modify them to suit your needs.

To download the data for our study, you would browse to the administrative module for your study on the web site (again, we are speaking hypothetically, as for this tutorial study there is no such site set up). Once at the administrative module, you would download your data (by clicking Download Data from the main menu), making sure to save your data (named studyname_data.sqlite, where STUDYNAME is your study name) to the same folder on your hard drive in which you developed the SSI Web project. Alternatively, from the SSI Web interface, you can simply click Field | Download Data... to download your data without logging into the Admin Module.

Once you have downloaded the data, you are ready to analyze the data by Counting analysis (Analysis | MaxDiff Counts) or through a more sophisticated methods called hierarchical Bayes analysis through clicking Analysis | MaxDiff Scores.

Analysis: Counts and Computed Scores

At this point, we would like you to open the Tutorial study which contains hypothetical MaxDiff data for you to analyze. Click Help | Sample Studies | Sample MaxDiff.

We'll assume that you have collected data over the web and have downloaded the data to the folder on your hard drive containing the study files (as described in the previous section).

A quick summary method called "counts" is often used for summarizing respondents' preferences. Counting analysis simply looks at how many times an item was available within sets, and counts how many times it was chosen either best or worst. The percent of times an item was chosen as best or worst (when it was available) is termed the "count proportion."

Run counting analysis for the tutorial data set by clicking Analysis | MaxDiff Counts.... The MaxDiff Counts dialog is displayed. It has various settings, including the ability to select which MaxDiff exercise to use (we only have one in our study) and to select which sets to include in the analysis (we'll use all sets). Click Compute Counts..., and after a few moments of computation, a counts report is displayed:

MaxDiff Counts Analysis

MaxDiff Exercise Reenlist

Sets Included All

Respondents Included Completes Only

Total Number of Respondents 250

Total Number of Sets 5000

Number Times Best Times Worst

Item of Times Selected Count Selected Count

Number Shown Best Proportion Worst Proportion

3-year reenlistment obligation 1 999 310 0.310 177 0.177

5-year reenlistment obligation 2 1000 72 0.072 561 0.561

Location guarantee for next assi 3 999 191 0.191 286 0.286

Duty guarantee for next assignme 4 1000 202 0.202 283 0.283

Both location and duty guarantees 5 1000 361 0.361 135 0.135

Get promoted 6-months sooner than 6 1000 183 0.183 307 0.307

Live in 2-person barracks when in 7 1000 252 0.252 231 0.231

Live in 4-person barracks when in 8 1000 208 0.208 276 0.276

Choice between living in 2-person 9 1000 294 0.294 186 0.186

$50 per month pay increase when o 10 1000 230 0.230 271 0.271

$75 per month pay increase when o 11 1000 246 0.246 225 0.225

Half of on-job time using your sk 12 1001 185 0.185 311 0.311

Three-quarters of on-job time usi 13 1001 199 0.199 251 0.251

Time off each week allowed for on 14 1000 190 0.190 309 0.309

Receive online career counseling 15 1000 234 0.234 268 0.268

Receive in-person personal counse 16 1001 231 0.231 238 0.238

Receive phone-based personal coun 17 1000 195 0.195 279 0.279

Current reenlistment bonus ($2,50 18 1000 276 0.276 189 0.189

Increased reenlistment bonus ($5, 19 999 386 0.386 110 0.110

Increased reenlistment bonus ($10 20 1000 555 0.555 107 0.107

The report shows results for 250 respondents and 5000 sets (remember, each respondent received 20 choice sets; 250 x 20 = 5000). You may recall that we had originally generated 300 questionnaire versions for respondents to potentially complete. We only interviewed 250 respondents, so not all questionnaire versions were used. However, we see from the "Number of Times Shown" column that each item was shown almost exactly the same number of times (subsets of questionnaire versions are also carefully balanced). In the "Times Selected Best" column, we see how many times each item was selected best (across all sets and respondents). Dividing the "Times Selected Best" by the "Number of Times Shown" results in the "Best Count Proportion." The higher the proportion, the more impact this item has on reenlistment.

Counts proportions are a good way to get a quick (top-line) look at the data. This method of analysis is very intuitive. But, we don't suggest you use counts results in final reporting. Because of prohibitions in your design or slight imbalances, the data are not as accurate as when using the more time-consuming HB estimation routine, automatically provided under Analysis | MaxDiff Scores.... Also, the counts probabilities tend to be artificially "smoothed" relative to the final estimated scores where we see larger differences between items.

The item selected best most often was "Increased reenlistment bonus ($10,000)." Thus, we can see that these sailors (at least our hypothetical ones) pay most attention to monetary rewards. Even the $5,000 reenlistment bonus tends to trump every other issue. If you are interested, you can also examine the proportion of respondents choosing each item as worst. This data set was generated with artificial symmetric properties, so there is no difference between bests and worsts other than random noise. However, this isn't always the case with real data sets, where respondents may not use a consistent scoring approach when judging bests and worsts.

Let's turn our attention to the more powerful and accurate method of analysis offered in the MaxDiff System: hierarchical Bayes (HB). A typical data set may take about 10 to 15 minutes to run using HB (large data sets may take hours!). HB estimates a set of scores for each respondent. These scores may be saved to an Excel-compatible file. How HB actually works is beyond the scope of this tutorial, and we refer you to technical papers on HB on our website, www.sawtoothsoftware.com.

Click Analysis | MaxDiff Scores... to run the HB estimation routine and estimate scores for each respondent. The MaxDiff Scores dialog is shown. As with the MaxDiff Counts dialog, you can select which MaxDiff exercise to use and which sets to include.

Click Prepare for Scores Computation.... First, the software reads the data set and prepares it in a more compact way for more efficient processing. As part of this step, the software reports how many respondents it has found and how many times items were chosen in each position of the question.

** Data Summary **
Total respondents = 250
Total 'Best' choices = 5000
Total 'Worst' choices = 5000
Average sets per respondent = 20.00

'Best' responses by position:
   (1) 24.38%
   (2) 26.34%
   (3) 25.08%
   (4) 24.20%

'Worst' responses by position:
   (1) 24.76%
   (2) 25.28%
   (3) 24.78%
   (4) 25.18%

Click Compute Scores, and the estimation of scores using hierarchical Bayes (HB) begins. You do not need to have any special training in statistics to consistently achieve good results with this analysis. HB uses an iterative method that requires many thousands of iterations to stabilize and complete the run. We have chosen defaults to govern the HB run that should work well for most every situation. Advanced users can modify the settings if they desire.

Partway through the estimation, the MaxDiff Scores dialog may look something like this:

maxdifftut11

This display shows how well the estimated scores are stabilizing over the planned thousands of iterations. The graphic plots the estimated scores at each iteration of the process. The estimates first all start at 0, and then trend toward their final values (which may be either negative or positive). Once the process has "converged" the estimates will tend to randomly "wobble" up and down, but there should be no noticeable trend remaining.

The graphic is divided into two regions: a gray region (iterations prior to assumed convergence) and a white region (iterations after assumed convergence). Once the iterations cross into the white region, you should look for no noticeable trend in the data. The plotted scores should just continue to wobble randomly up and down, with no strong and sustained move in any specific direction. If there is noticeable trend after the plotted values are in the white region of the graphic, then you should repeat the estimation using more iterations (under the Advanced button).

(Note that for sake of brevity we didn't include "constraints" in this estimation to force the scores for some items to be higher than others, such as $10,000 bonus higher than $5,000 bonus. You do not need to enforce such constraints during estimation, though some researchers choose to do so.)

When the estimation finishes, the graphical plot of the estimation remains on the screen. Click the button stating: The estimation is complete. Click here to close.

When you click the "close" button, a report is given which resembles a spreadsheet, with multiple "sheets" that can be accessed by clicking the tabs at the bottom of the dialog labeled "Summary," "Rescaled," and "Raw." The first (Summary) sheet contains a summary of the final scores, 95% confidence intervals,and other information regarding the settings for your run:

The second (Rescaled Scores) and third (Raw Scores) sheets contain the scores for each respondent, including a fit statistic. Most researchers will pay most attention to the Rescaled scores as they are easiest to interpret and present to others. These individual-level item scores are positive values ranging from 0 to 100 that sum to 100. This approach has the valuable property of ratio-scaling. That is to say, an item with a score of 20 is twice as important (or preferred) as an item with a score of 10.

Respondents who answer randomly will generally have low fit statistics. We have provided guidelines (fit statistic thresholds) for identifying "bad" respondents elsewhere in this documentation. You can use the suggested cutoff rules to discard respondents that are most likely answering randomly.

Click the Save As... button to save this report. Two formats are offered:

Microsoft Excel format --an Excel Workbook .XLS file is saved with multiple sheets (one for each of the three sheets displayed in this dialog)

.CSV (Comma Delimited) --only the current displayed sheet is saved to the file. To save all information from the three sheets to .CSV format, select each sheet separately and save each to a unique .CSV file.