Developing a List of Items
Top  Previous  Next

Choice studies such as those supported by the MaxDiff software, which include,

·MaxDiff  
·Method of Paired Comparisons (MPC)  
·Choices from sets of three (triples), sets of four (quads), etc.  

usually involve at least eight items (i.e. features, or attributes). With seven items or fewer, a designed choice experiment may risk overkill. When dealing with so few items and with reasonably sophisticated respondents, perhaps an allocation-based task would be sufficient. Even so, we could imagine conducting a paired comparison exercise with as few as four or five items, if the demands of the project required a simple respondent task (appropriate for people of all educational and cultural backgrounds) and avoidance of the response-style bias typically seen with traditional ratings scales.

On the high end, we at Sawtooth Software have been involved in a paired comparison experiment that involved 160 employment-related benefits/conditions. Sample size was in the thousands of respondents, and we pooled information across respondents for estimation. We're certain other researchers have conducted even larger studies than that.


Specifying a List of Items

On the Items tab of the MaxDiff Exercise dialog, you can select an existing list to use in the MaxDiff exercise, type a new list by clicking the Add... button, or copy-and-paste a list of items from another program by clicking the Paste list member(s) from the clipboard button pastequestion. Depending on your license, you can specify up to 500 items. Items can involve either text, graphics, or a combination of text and graphics.



Recommendations for Item Descriptions:

Item text should be clear and succinct. Anything you can do to help respondents process information more quickly and accurately is helpful.  
 
When possible, items should be as specific and actionable as possible. For example, it would probably be more useful to describe a level as "Fuel efficiency increased from 5 liters/hour to 4.5 liters/hour" instead of "Improved fuel efficiency." That said, we realize that such quantitative certainty doesn't make sense for many studies and items. For example, in an image study, we may very well ask respondents to select which picture of an automobile most "Makes me feel successful when I drive it." How one defines "successful" in a concrete or quantitative way is probably not useful to the goals of the study.  
 
Items can be made to be multi-level and mutually exclusive. For example, in a study of fast-food restaurant features, rather than ask about "Fast service" generically, you might create three separate items that probe specific levels of fast service:  
 
Order and receive food within 3 minutes  
Order and receive food within 6 minutes  
Order and receive food within 10 minutes  
 
When considering a fast-food restaurant, it would seem rational that all respondents would prefer faster service to slower service. Therefore, it makes no sense to include more than one of these levels within the same set. You should specify prohibitions between these levels (from the Design tab) so that they never appear compared with one another. It is also possible during estimation to require that faster levels of service receive a higher score than slower levels of service (by specifying monotonicity constraints).  
 
Another example of multi-level, mutually-exclusive levels was shown in the award-winning paper at the 2004 Sawtooth Software Conference by researcher Keith Chrzan. Using a MaxDiff methodology, he studied price sensitivity for options for automobiles. Each option was available at multiple prices, so he could plot the resulting scores as relative demand curves. (Of course, the same option never appeared within the same set at different prices.)  
 
You may mix multi-level items with single-level items within the same study. However, we should note that the increased frequency of multi-level attributes in the design might bias respondents to pay more attention to that dimension. But, we are not aware of any research yet in MaxDiff or MPC that substantiates that concern.  
 
Graphics are possible! Remember, SSI Web lets you use graphics as list items.  
 
It is sometimes helpful to create a "reference" level with monetary meaning. Some researchers have found it helpful to associate one of the levels with a specific monetary equivalent. For example, in studying job-related conditions and benefits, it might be useful to include a level that says: "Receive an immediate $500 bonus." Or, if studying improvements to a product, we might include a level that says: "Receive a $20 off coupon." In both cases, we can associate a specific item score with a specific and immediate monetary gain. The scores for other levels may then be compared to this monetary-based reference level. Also note that it is possible to make the monetary reference point multi-leveled with mutually exclusive levels, such as "$5 off coupon," "$10 off coupon," "$15 off coupon," "$20 off coupon." This provides multiple monetary-grounded reference points for comparing the scores of other items in the study.  
 
In other studies, such as those studying potential modifications to existing products, services, or (to be more specific) employment conditions, it might make sense to include a reference level reflecting no change; for example, "No change to current work environment." That way, item scores that might have a negative affect (relative to "no change") can be identified.  
 
Using Constructed Lists in MaxDiff Experiments

Warning: This is an advanced feature that if not used carefully can result in errors in data collection and incorrect model results. Please test your survey thoroughly prior to fielding the study. After collecting data, analyze the data via Counts to ensure that all items from the parent list have been shown a reasonable number of times to the sample of respondents. If some items were never shown to any respondents, they will be marked as having been shown zero times (with an asterisk) and estimation using HB will be incorrect unless these items are omitted from the analysis.  

SSI Web software can build constructed (dynamic) lists based on answers to previous questions or some other rule you define (such as randomization or always forcing a client's brand onto a list). You may use the constructed list in later questions or even in MaxDiff exercises. When at the items tab on the MaxDiff Exercise dialog, you may select a constructed list rather than the typical pre-defined list used in MaxDiff experiments. This powerful option even allows for the inclusion of respondent-supplied other-specify text as an item in the MaxDiff exercise.

The key limitation for using constructed lists is that all constructed lists for respondents must have the same length. SSI Web requires you to use SETLISTLENGTH as the last instruction in the constructed list logic used in a MaxDiff list. The Number of Items per Set specified on the Design tab must match the number of items specified in the SETLISTLENGTH instruction.

Warning: the SETLISTLENGTH instruction ensures that the constructed list has a fixed size or length. If the SETLISTLENGTH instruction needs to add members from the parent list it will not add "Other Specify" or "None of the above" type members. This can cause a problem if there are no other members to add. Please thoroughly test the constructed list in the MaxDiff exercise prior to fielding the study.  

SSI Web uses the design generated from the Design tab, but draws items from the constructed list according to the design by position on the constructed list. The values stored for analysis (both for the design shown to respondents and for the answer given) correspond to the original pre-defined list.

Warning: the design report displays frequencies for a design having the same number of items as specified in the SETLISTLENGTH instruction and in a fixed order. However, each respondent's actual items may vary, and therefore the true frequency of occurrences for each parent item and co-occurrences of items may differ significantly from that shown in the design report.  

When estimating item scores, MaxDiff estimates (via HB) a full set of item scores for all items in the pre-defined list, even if each respondent received only a customized subset of the items. These scores for items not seen take on values near population averages. Depending on your reason for using a constructed list that is a subset of the pre-defined list, this may or may not be a desired outcome.