Choosing the Next Paired-Comparison Question

The first part of an ACA interview is concerned with screening the attribute levels and learning enough about the respondent's preferences to construct initial part-worth estimates. After that is done we begin the paired-comparison section, in which pairs of concepts are shown and preference questions are asked. Following each response (submitted page of pairs questions) we update our estimates of part-worths and then decide what pair of concepts to present next.

The number of possible concepts is very large, and we need some reasonably efficient procedure to choose a pair of them at each stage that will be most beneficial in some way. There are several principles to keep in mind when thinking about how to choose concepts.

·

Concepts should be chosen by a method that gives the author as much control as possible over the interview, in terms of the complexity of the concepts and the number of questions asked.

·

The design should be as "balanced" as possible. Observations should be spread as evenly as possible over all attribute levels, and the columns of the design matrix should be as nearly orthogonal as possible.

·

We should ask the respondent questions that require careful consideration. There is no point in asking questions for which we already know the answer, such as "High quality at a low price" versus "low quality at a high price." We learn more if we choose concepts nearly equal in attractiveness.

Our procedure addresses these points. The author may specify the number of attributes to appear in each concept. The range is from two to five. It is possible to start with only two attributes per concept and, after the respondent has gained experience, to increase their complexity.

The concepts in a pair always have different levels of the same attributes. Our procedure for choosing those concepts is:

·

Count the number of times each pair of attributes has appeared together in any concept. Pick a set of attributes at random from among those whose members have previously appeared together the fewest times.

·

For each of the chosen attributes, repeat similar logic to find levels that have been paired least frequently.

·

Examine all possible ways of combining these levels into concepts (with just two attributes there are only two possible ways; with 5 attributes there are 16 ways. Find the pair of concepts most nearly equal in attractiveness, using the current estimates of the respondent's part-worth utilities.

·

Randomly determine which concept will appear on each side of the screen.

Accordingly, ACA presents pairs of concepts that are as nearly equal as possible in estimated utility. At the same time, constraints are imposed to ensure that the overall design is nearly orthogonal. Within concepts, each pair of attributes is presented with equal frequency, and within each attribute, each pair of levels is presented with equal frequency. In addition, if the paired-comparison questions show only two attributes at a time, further steps are taken to insure that the overall design is "connected."

Such an approach has these benefits:

1) It gives the respondent the impression that the system is paying attention to his or her answers, and it seems to be asking increasingly insightful questions.

2) It helps keep the respondent operating within the defined range of a response scale rather than at its ends.

3) It provides data on "tight" inequalities if estimation is later to be done by nonmetric methods.

ACA lets the author specify certain pairs of attribute levels that must not appear together in the same concept. The procedure described above is modified slightly to take account of such prohibitions. When concepts are described on only two attributes, ACA chooses the first few questions in a slightly different way. (When the concepts are described on only two attributes, it would be possible to blunder into a design in which the attributes would be divided into subsets in such a way that those in one subset would never be paired with those in another subset. Such designs would provide no information about the relative importance of attributes in different subsets, and ACA automatically corrects the design in such a situation.)

ACA's designs usually have good statistical efficiency, although they are not strictly orthogonal. Statistical efficiency is increased as more attributes are used in each concept, and it is also possible to produce concepts more nearly equal in attractiveness when there are more attributes with which to work. However, using larger numbers of attributes has the unfortunate consequence of making the questions more complicated, and respondents are more easily confused.

Both anecdotal and experimental evidence has shown that it is usually best to start with only two attributes per concept and, after a few pairs, to increase the number of attributes to three. Beyond three attributes, gains in efficiency are usually offset by respondent confusion due to task difficulty.