Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

When to Include Interaction Effects in HB

Dear Sawtooth Team,

I have conducted two CBC studies with different respondents. The two CBC studies were the same in terms of attributes and attributes levels and only differed regarding the described starting situation/context. Before comparing these two groups I wanted to check, whether there are any interaction effects I need to incorporate in the HB model(s).

For this reason I used the automated interaction search tool.

For treatment group one, the interaction search tool proposes an interaction between attributes A and B.

For treatment group two, the tool proposes an interaction between attributes A and B as well as A and C.

As the automated interaction search tool makes use of logit estimation, I wanted to check, whether I also need to incorporate these interactions, when using (individual-level) HB.

As described in the Sawtooth Software Manual the best approach for doing this (albeit time consuming) is running separate HB models for each interaction effect.

That´s what I did. so I ran three additional HB estimations and checked respectively, whether they can outperform the basic HB model with only main effects in terms of holdout prediction hit rates.

Is this (simply comparing holdout prediction rates) a reasonable approach to decide, whether or not to incorporate interaction effects in HB models? Or are there any other parameters, which I could use to make a decision? Do I need to run an additional estimation for treatment group two, where both proposed interactions (A and B as well as A and C) are incorporated simultaneously?

I only found one interaction (treatment group one, attributes A and B) to slightly increase holdout prediction rate. In absolute terms this model was able to predict one more respondent´s choice correctly (per holdout task) compared to the basic HB model with only main effects. This caused the Hit-Rate to increase by 0.76%. Given this rather small increase I inferred to not incorporate this interaction in the HB model.

Is this a reasonable conclusion?

Also, in the manual you write that you „recently undertook a substantial investigation of about two-dozen CBC datasets, looking for interaction effects that could significantly lift holdout prediction hit rates [were your found] only one or two CBC datasets that seemed to benefit much from inclusion of interaction effects“. Are you referring to the paper `What Are the Optimal HB Priors Settings for CBC and MaxDiff Studies?´

Thanks in advance!
asked May 27, 2018 by AnjaWe (270 points)
edited May 27, 2018 by AnjaWe

1 Answer

0 votes
Best answer
I don't know how many attributes you have in the model, but with the automatic interaction tool it can be important to correct for multiple comparison error.  See this post on our LinkedIn User Group:  https://www.linkedin.com/groups/1715557/1715557-5895943758326820864

The software that automates interaction for HB analysis is called the Model Explorer and yes, the paper about Optimal HB Priors is the one you want there.  You can download this software for free from the Tools and Scripts portion of our website here:  http://www.sawtoothsoftware.com/support/downloads/tools-scripts
answered May 27, 2018 by Keith Chrzan Platinum Sawtooth Software, Inc. (74,825 points)
selected May 29, 2018 by AnjaWe