|
Designing the Study
|
|
| Number of Items (Attributes)
|
|
|
| In this field, you see the number of items (i.e. levels or features) to include in the exercise. This field is not editable, but is computed by examining the current list you have specified on the Items tab Depending on your license, you can specify up to 500 items on the Items tab.
|
|
|
| Number of Items per Set (Question)
|
|
|
| Generally, we recommend displaying either four or five items at a time (per set or question) in MaxDiff questionnaires. However, we do not recommend displaying more than half as many items as there are items in your study. Therefore, if your study has just eight total items, we would not recommend displaying more than four items per set.
|
|
|
| If you would like to use the Method of Paired Comparisons approach, then specify just two items per set.
|
|
|
| Research using synthetic data suggests that asking respondents to evaluate more than about five items at a time within each set may not be very useful in MaxDiff studies. The gains in precision of the estimates are minimal when using more than five items at a time per set. The small statistical gains from showing even more items may be offset by respondent fatigue or confusion.
|
|
|
| Number of Sets (Questions) per Respondent
|
|
|
| We generally recommend asking as many sets (questions) per respondent such that each item has the opportunity to appear from three to five times per respondent. For example, consider a study with 20 items where we are displaying four items per set. With 20 total sets, we know that each item will be displayed 4 times (assuming the design is perfectly balanced). This leads to the following decision rule and formula:
|
|
|
| For best results (under the default HB estimation used in MaxDiff), the suggested number of sets is at least:
|
|
|
| 3K/k
|
|
|
| where K is the total number of items in the study, and k is the number of items displayed per set.
|
|
|
| The software will warn you if the number of sets you have requested leads to each item being displayed fewer than 2 times per respondent.
|
|
|
| These recommendations assume HB estimation (the default method in MaxDiff). Advanced researchers may elect to export the data (File | Export Data | Prepare MaxDiff Data Files (*.cho, etc.)...) for analysis using Sawtooth Software's Latent Class system, or another analytical program of their choice. When pooling data across respondents using Latent Class or aggregate logit, you may be able to achieve reasonable population-level scores using even fewer sets per respondent than these recommendations. If you use relatively few sets per respondent (relative to the number of parameters to estimate), aggregate modeling may be the better option to stabilize and summarize parameters across subgroups or the population.
|
|
|
| 1. | Create a design that features item frequency balance. A balanced design is one in which the one-way frequencies are nearly equivalent (how many times each level appears across the entire design) and two-way frequencies are also nearly equivalent (how many times each pair of items appears within the same set across the entire design). When one-way and two-way frequencies are balanced, this is termed orthogonality.
|
|
|
| 2. | Ensure that each version of the design (respondent questionnaire) features connectivity (meaning that all items are linked either directly or indirectly), unless the Allow Individual Designs Lacking Connectivity option is checked. Without connectivity it becomes very difficult for the default estimation procedure in MaxDiff (HB) to scale the items properly relative to one another for each respondent.
|
|
|
| 3. | After the designs have been generated based on the guidelines in 1 and 2, swap the order of the items within each set so that each item appears approximately an equal number of times in each position. Positional balance reduces or virtually eliminates order bias. Finally, randomize the order of the tasks within each version.
|
| · | Number of versions used
|
| · | Design seed used
|
| · | One-way and two-way frequencies
|
| · | Positional balance
|
| Number of Versions
|
|
|
| Though it is possible to estimate scores relatively efficiently using just a single questionnaire version for all respondents, there is practical benefit to using multiple versions (sometimes called "blocks") of the questionnaire. With multiple versions of the questionnaire, different respondents see different series of questions. Across respondents, this dramatically increases the variation in the way items are combined within sets, which can reduce potential context biases (which are usually modest).
|
|
|
| The default is 300 questionnaire versions, which ensures a very large overall design and a great deal of combinatorial variation in the sets. Using 300 versions is more than adequate, since beyond a handful of questionnaire versions the further practical benefits of increased variation are barely detectable. Since using SSI Web makes it nearly automatic to produce and use multiple versions of the questionnaire, it makes sense to use a large number. The software allows you to request up to 999 questionnaire versions. The two main take-aways are, 1) it makes sense to use many questionnaire versions, 2) it is not necessary for each respondent to receive a unique questionnaire version.
|
|
|
| If using the separate MaxDiff designer for conducting paper-and-pencil studies, we recommend using at least four different versions of the questionnaire. You need to find a good balance between difficulty of study administration and design quality.
|
|
|
| Number of Iterations
|
|
|
| By default, the MaxDiff designer repeats its algorithm 1000 times and returns the best design found across those passes. The quality of the design is based on the one-way, two-way, and positional balance (in that order of precedence). For many designs, 1000 iterations can be done within a few seconds. We encourage you to increase the number of passes, as it may enable you to find a slightly better design. Up to 999,999 iterations may be specified.
|
|
|
| Design Seed
|
|
|
| Specify a value from 1 to 999,999,999. This integer (seed) is used for determining the starting point for the design algorithm. Different seeds will yield different designs, all having approximately the same overall design efficiency. You can try different starting seeds to see if you obtain slightly better designs.
|
|
|
| Favor Two-Way Balance
|
|
|
| The MaxDiff designer by default puts primary emphasis on balancing the number of times each item is included (one-way balance), with secondary emphasis on two-way balance (how often each paired combination of levels appears together). If you have specified prohibitions, these two goals are at odds with one another. The software puts more emphasis toward balancing the two-way frequencies if you check this box. Recognize, however, that with prohibitions it may be impossible to achieve near-perfect balance in either the one-way or two-way frequencies.
|
|
|
| We strongly suggest checking this box when displaying two items at a time within sets (pairwise presentation). The design algorithm will almost always produce one-way frequencies that differ very little from when one-way balance is favored. But, the two-way frequency balance will usually be far superior.
|
|
|
| Allow Individual Designs Lacking Connectivity
|
|
|
| Connectivity is the property that the items cannot be divided into two sets wherein no comparison is made between any item in one set and another item from the other. Even if creating choice sets randomly, if respondents receive enough sets, it would be difficult for the items to lack connectivity for any given respondent. However, if you select a design with relatively many items and relatively few tasks per respondent, many individual versions of the questionnaire could lack connectivity. The default for the software is to reject any version (questionnaire for one respondent) that lacks connectivity. But in the case of many items and few sets per respondent, it may become impossible to satisfy connectivity. If you would like to permit versions lacking connectivity (which may be the case if you ask few questions of any one respondent, probably leading to aggregate analysis), then you can check this box. Make sure to specify many questionnaire versions. To ensure stable estimation, advanced users may wish to generate dummy (random) response data, export the data for estimation under aggregate logit, and examine the size of the standard errors to ensure reasonable precision.
|
|
|
| Prohibitions
|
|
|
| Occasionally, your study might include multi-level items characterizing degrees or amounts of some feature. For example, in studying features of fast food restaurants, you may wish to include the following three items in your study list:
|
|
|
| Order and receive food within 3 minutes
|
| Order and receive food within 6 minutes
|
| Order and receive food within 10 minutes
|
|
|
| When considering a fast-food restaurant, it would seem rational that all respondents would prefer faster service to slower service. Therefore, it makes no sense to include more than one of these levels within the same set. You should specify prohibitions between these levels by clicking the Prohibitions... button so that they never appear compared with one another.
|
|
|
| For those familiar with conjoint analysis, it may be heartening to learn that prohibitions are not as damaging in MaxDiff experiments as with conjoint analysis. For example, in a conjoint study where there are 4 prohibitions specified between two 4-level attributes, 4 of the possible 16 combinations (or 25%) are prohibited. Depending on the pattern of prohibited combinations, this could be very detrimental. However, in a typical 20-item study, there are 1/2(20*19) = 190 possible combinations of items (taken two at a time). Prohibiting 4 of these represents only about 2% of the possible combinations. This will have very little impact on the overall results. Please see Robustness of MaxDiff Designs to Prohibitions for more details.
|
|
|
| When you compute the Generate Design button, if the software returns warnings that the questionnaire versions lack connectivity, this is clear sign that your prohibitions are excessive (after ruling out the possibility that you are not displaying enough sets per respondent).
|
|
|
| · | 20 items in the exercise
|
| · | 4 items per set
|
| · | 20 sets per respondent
|
| · | 2 versions of the questionnaire
|
| Version,Set,Item1,Item2,Item3,Item4
|
| 1,1,10,1,19,7
|
| 1,2,19,17,15,3
|
| 1,3,17,10,16,6
|
| 1,4,6,20,12,13
|
| 1,5,13,2,9,5
|
| 1,6,2,9,13,11
|
| 1,7,15,18,1,14
|
| 1,8,7,3,20,8
|
| 1,9,5,6,18,9
|
| 1,10,3,12,2,16
|
| 1,11,8,3,10,4
|
| 1,12,15,7,5,2
|
| 1,13,11,16,14,19
|
| 1,14,4,19,11,12
|
| 1,15,14,11,17,8
|
| 1,16,20,15,13,4
|
| 1,17,9,1,17,18
|
| 1,18,18,8,16,6
|
| 1,19,1,5,12,20
|
| 1,20,14,4,7,10
|
| 2,1,4,11,6,15
|
| 2,2,10,13,14,18
|
| 2,3,20,10,9,1
|
| 2,4,11,20,8,5
|
| 2,5,16,13,5,7
|
| 2,6,12,19,7,17
|
| 2,7,12,14,8,17
|
| 2,8,13,18,3,2
|
| 2,9,18,7,4,11
|
| 2,10,17,4,15,16
|
| 2,11,5,14,10,15
|
| 2,12,19,2,3,8
|
| 2,13,2,15,17,20
|
| 2,14,16,12,5,9
|
| 2,15,4,2,19,10
|
| 2,16,6,11,14,3
|
| 2,17,8,7,4,6
|
| 2,18,3,1,11,19
|
| 2,19,1,9,16,20
|
| 2,20,12,9,2,14
|