Customizing the "Importance" Question

The stated (self-explicated) importance questions in ACA have been the target of some criticism over the years (see "The 'Importance' Question in ACA: Can It Be Omitted?" available within our technical papers library at www.sawtoothsoftware.com/techpap.shtml). Some of the main arguments against the current implementation of an importance rating scale in ACA are:

1.

Respondents tend to say that many attributes are very important, which can result in "flatter" final importances (lower discrimination) than reality.

2.

Respondents may have a hard time understanding the framing of the question (assigning a weight to the difference between extreme levels).

3.

There is evidence that respondents may not answer the questions as reliably as the within-attribute ratings (ACARAT questions).

Starting with version 6 of SSI Web, the Importance questions can be customized or omitted altogether, drawing upon aggregate importances from previous respondents as "priors" to inform the pair generation algorithm and subsequent calibration concepts. With this approach, ACA/HB is required for proper part-worth utility estimation (since after dropping importances there is not enough information available at the individual level to estimate part-worths under standard OLS estimation).

Some users, however, may not have the luxury of relatively large sample sizes needed to stabilize part-worth estimates after dropping the Importance questions and furthermore may not be interviewing from a common server (or SSI CAPI installation) such that prior importance scores could be shared and updated across respondents. To help overcome some of the issues noted above, some researchers may wish to substitute customized self-explicated importance questions and have ACA use that information for informing the subsequent pair generation algorithm, calibration concepts, and final part-worth utility estimation under OLS.

Custom Importance Question Ideas

One example that may work well is to first educate respondents about the full list of attributes (including the range of levels for each attribute), and then ask the respondent to identify the one most important attribute. Then, the researcher asks the respondent to assign a "10" to that most important attribute, and to rate the other attributes with respect to that most important attribute. Respondents could be instructed that if an attribute is half as important as the most important one, they should give it a "5."

As another example, assume that ACA is being used to interview just a few key business managers within an organization regarding the desirability of different projects or capital expenditures. Further suppose that these managers are analytically sophisticated and are quite comfortable with the notion of assigning weights to attributes using a constant-sum (chip-allocation) question. Under these conditions, the researcher might believe that these particular respondents will provide better information if using a constant-sum allocation of importance across attributes rather than the standard ACA approach for asking about attribute importance.

Implementing Custom Importance Questions

To implement a customized importance questions, click the Set Prior Importances Based on other Questions/Variables option, and then click the Specify Questions... button.

For each attribute in the table, specify a question name (or variable value). For example, CS_1 returns the value contained in variable CS_1. This might refer to a constant sum question named CS where the first response option (_1) refers to the first attribute.

The customized importance scores must contain only positive numeric values. ACA assumes these positive values have ratio scaling (i.e. that a score of 10 is twice as important as a score of 5). For internal utility calculation purposes, the importance data are rescaled to the range 0 and 7. The rescaled importance values (range of 0 to 7) are stored in the data file in place of the "skipped" variables Exercise_Importance1 through Exercise_Importancen.

Variables containing customized importance information can come from most any question type within SSI Web, asked at any point in the survey, including Free Format and hidden variables. They may even be passed into the questionnaire through the "one-click access to survey."