Sawtooth Software: The Survey Software of Choice

Interaction Effects and ACBC

ACBC is a very recent addition to the conjoint toolkit (2009) but is becoming more widely used. Among Sawtooth Software users, the ratio of ACBC to CBC projects conducted over the previous 12 months was 1:6, up from 1:7 in the previous year.

One of the most often overlooked capabilities of ACBC is its ability to include interactions between attributes within HB estimation. Indeed, one of ACBC’s strengths (on par with or potentially even better than CBC) is to compute interactions between attributes. On the Attributes tab, within the HB Settings dialog, is a button that allows you to specify interaction effects between attributes.

Probably the biggest reason that ACBC users don’t include interactions within their ACBC models is because ACBC doesn’t make it as easy as the CBC software to investigate the possible value of 2-way attribute interaction effects. With CBC, the Counts routine can run a quick Chi-Square test to report which interaction effects are potentially most useful within subsequent logit, latent class or HB modeling. With ACBC, Counts isn’t appropriate, since the adaptive, customized designs do not reflect two-way level balance for the joint occurrence of levels for attributes considered two at a time. This leaves us with model-based approaches to investigating the value of interaction effects.

The quickest model-based approach for assessing the potential value of interaction effects is to use aggregate logit to analyze ACBC data sets. This may be done by exporting the ACBC .cho file from SSI Web, and then by analyzing that .cho file with our standalone Latent Class software (in 1-group mode, representing aggregate logit). First, a main-effects only model is fit, and you record the model fit (the Log-Likelihood). Next, you re-run the model by including a 2-way interaction effect between two attributes of interest. Again, you record the Log-Likelihood, and examine the increase in Log-Likelihood over the main-effects only model. It turns out that 2x the difference in Log-Likelihood between those models is distributed as Chi-Square, with degrees of freedom equal to the number of additional parameters included in the model. If you included the interaction between a 3-level attribute and a 4-level attribute, the number of additional parameters added to the model is (j-1)(k-1) or (3-1)(4-1) = 6. You can test the likelihood that the additional fit was just due to chance by using the “=Chidist(Critical_Value,Degrees_of_Freedom)” function in Excel. For example, if 2x the difference in Log-Likelihood was 12.65, and the additional degrees of freedom for the interaction model was 6, then the function “=Chidist(12.65,6)” returns a p-value of 0.049, meaning that the likelihood of seeing this large of an increase in fit due to chance occurrence in the data is less than 5%. In other words, we are 95% confident that the interaction effect adds significant fit to the aggregate ACBC model.

The problem with this prescription is that examining all potential 2-way interactions between eight attributes in an ACBC study would lead to (8*7)/2 = 28 separate two-log-likelihood tests. If each test took you 2 minutes to run manually, this would be nearly an hour of tedious work. Hence, very few ACBC researchers take the time to investigate significant interaction effects!

We have developed a stand-alone .EXE that will examine a .cho file (either a CBC-generated or ACBC-generated file) and compute all 2-LL tests for all 2-way combinations of attributes. A summary report is given, sorting the interaction effects from most significant (lowest p-value) to least significant. The cost of this tool is $250. If interested, please write This email address is being protected from spambots. You need JavaScript enabled to view it. to order.

While interactions are most helpful for aggregate analysis (such as aggregate logit), we have seen data sets where the inclusion of a few selected interaction effects can substantially improve the fit for HB models. Therefore, we strongly encourage you to investigate potential significant interaction effects within your ACBC models. It isn’t a guarantee that a significant interaction effect observed with 2LL tests in aggregate logit will improve ACBC/HB models, but particularly strong interaction effects observed in aggregate logit will probably improve your ACBC/HB results. You can look at the increase in Pct. Cert. in your ACBC/HB reports to see if you are obtaining much additional fit by inclusion of interaction terms. (There is a formally correct way within HB analysis for performing statistical tests of interaction effects, but that is beyond the scope of this article.)

And, just because CBC or ACBC allows you to model interaction terms, you shouldn’t go crazy by including too many interaction terms in the model! We have heard of users trying to include all possible two-way interaction terms, without regard to whether they provided significant increase in fit. Such practices lead to overfitting of the data, decreased holdout prediction performance, and dramatically longer run times.