Sawtooth Software: The Survey Software of Choice

Reducing the Number-of-Attribute-Levels Effect in ACA with Optimal Weighting

This article is taken from a more complete technical paper available for downloading from our Technical Papers library.

Dick Wittink was the first to document the number-of-attribute-levels (NOL) effect in conjoint analysis. He found that the number of levels on which an attribute was defined had a direct impact on the resulting attribute importance. One could increase the apparent importance of an attribute simply by adding more levels!

The NOL effect occurs in varying degrees in all conjoint methods, and even can play a role in self-explicated approaches. Both psychological and algorithmic explanations have been proposed to explain the effect. This paper will demonstrate that the optimal weighting option in ACA Version 4 for combining Priors and Pairs utilities significantly reduces the NOL effect relative to Version 3's equal weighting.

How Utilities Are Calculated in ACA

Before examining the NOL effect and optimal weighting in ACA, it may be helpful to review how ACA utilities are determined.

ACA is an adaptive hybrid conjoint model combining self-explicated evaluations with paired conjoint comparisons. The self-explicated half of the model is referred to as the Priors. The paired comparison conjoint section is referred to as the Pairs.

ACA computes utilities using Ordinary Least Squares (OLS) regression. The Priors contribute as many cases (rows) to the design matrix as levels in the study. The Pairs section contributes as many cases as pairs questions.

Criticism Leads to Innovation

In an article published in the Journal of Marketing Research, Green et al. (1991) criticized ACA Version 3 for combining the information from the Priors and Pairs in a single OLS matrix operation. They argued that the coded dependent variables in the Priors and Pairs were not necessarily congruent.

Johnson (ACA, 1987) had developed these coding procedures based on many years of experience and experiments on combining self-explicated ratings with paired comparison conjoint evaluations. The response scales and the coded dependent variables were determined empirically to work well in practice. Following Green's criticism, Johnson released a new version of ACA (Version 4) in 1993 which provided an option for Optimal Weighting.

Optimal Weighting

Under optimal weighting, utilities are calculated independently (at the individual level) for the Priors and the Pairs (ridge regression is used to stabilize the Pairs utilities). Subsequent calibration concepts are rated on a 100-pt purchase likelihood scale. These additional observations are used to determine the relative weights that should be applied to the Priors and Pairs utilities, using a simple linear model:

       y = a + bX1 + cX2

   where,

       y =  The logit transform of the calibration concept rating
       a =  Intercept
       b =  Weight for the Priors utilities
       X1=  Utility of concept as predicted by Priors utilities
       c =  Weight for the Pairs utilities
       X2=  Utility of concept as predicted by Pairs utilities

The scaling of the dependent variable and coding of the independent variables are not required to be congruent across the two halves of the design, since the utilities are calculated independently for each exercise.

Optimal Weighting and the Number of Levels Effect

A paper (Wittink et al. 1997) given at our 1997 Conference suggested to us that the NOL effect in ACA may be in part due to an incompatibility in the way respondents use the scale in the Pairs and Priors sections. We examined two data sets to test this hypothesis.

The first data set was a commercial study which included 336 respondents and 20 attributes, varying in number of levels from 2 to 5. The data were collected under Version 4 of ACA, but equal weighted utilities (Version 3 method) were also accessible in ACA's audit trail file, and these are labeled as Version 3. Importances were calculated at the individual level for the 20 attributes, for both Version 4 and Version 3 utilities. (Attributes that have two levels are represented on the graph as "2," three levels as "3," etc.)The 45-degree line reflects where the data points should lie if the two methods for calculating utilities resulted in the same attribute importance. If a data point lies above the line, the importance from Version 3 exceeds the importance from Version 4's method.

All of the 5-level attributes lie above the line, and all 2-level attributes fall below the line, strongly suggesting that the Version 4 method reduces the NOL effect. The 2-level attributes are significantly less importantunder Version 3 than Version 4.

Under Version 3, the average importance for 2-level attributes is 12% less than the corresponding Version 4 importances, with an average t-value for the mean difference of 8.1 (p<0.001). Under Version 3, the average importance for a 5-level attribute is 6% higher than the corresponding optimally-weighted importances, with an average t-value for the mean difference of 6.3 (p<0.001).

The Version 3 utilities display a pattern consistent with the NOL effect. Attributes with more levels are biased to receive greater importance relative to the Version 4 result.

The second data set was an experimental study conducted among 80 MBAs in 1997. This design was considerably smaller in scope than Study #1. Only 9 attributes were included, each having either 2 or 3 levels. The differences in importances were not as large for this data set as the previous. Only 2- and 3-level attributes were measured, so there was less potential bias from the NOL effect. Even so, the deviations were all in the expected direction. The 2-level attributes on average were 8% lower and the 3-level attributes were 3% higher with the Version 3 approach versus Version 4.

Conclusion

The optimal weighting option in Version 4 reduces the NOL effect relative to the equal weighting approach of Version 3. It is important to note that the principal reason the optimal weighting method reduces the NOL effect is not due to the customized differential weights for pairs and priors, but due to estimating utilities independently within those separate components prior to combining the information.

One cannot argue that we can completely control the NOL effect with ACA. Other factors contributing to the effect undoubtedly remain. We have seen, however, that even equal-weighted ACA is less susceptible to the number of levels effect than traditional full profile methods (Wittink et al. 1991).

The optimal weighting method appears to have been a nice addition to ACA. It probably deserves more credit than it has been given. We at Sawtooth Software have believed for some time that optimal weighting provided modest improvements to ACA utilities relative to equal weighting. We now recognize that this method of calculating ACA utilities plays a significant role in reducing the NOL effect, and recommend that ACA users use optimal weighting, especially when the number of levels varies across attributes in the design.

References

Green, Paul, Abba M. Krieger, and Monj K. Agarwal (1991), "Adaptive Conjoint Analysis: Some Caveats and Suggestions", Journal of Marketing Research, (May), 215-22.

Johnson, Richard M. (1987), "Adaptive Conjoint Analysis," Sawtooth Software Conference Proceedings, Ketchum, ID: Sawtooth Software, 253-65.

Wittink, Dick R., Joel Huber, John A. Fiedler, and Richard L. Miller (1991), "Attribute Level Effects in Conjoint Revisited: ACA versus Full Profile," Advanced Research Techniques Forum, Proceedings of Second Conference, Rene Mora (ed.) Chicago: AMA, 51-61.

Wittink, Dick R., William G. McLauchlan, and P.B. Seetharaman (1997), "Solving the Number-of -Attribute-Levels Problem in Conjoint Analysis," Sawtooth Software Conference Proceedings, Sequim, WA: Sawtooth Software, 227-40.