Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Analysis of Partial Profile Design?

Hi,

I am running an experiment using CBC partial profile design (i.e. 9 attributes, but respondents only see 4 attributes at a time, make a binary choice). I want to know the technical details of how Sawtooth analyzes this kind of data?

When I download the data, Sawtooth codes the levels of the attributes that a respondent did not see as "0".

I have analyzed this data both in R and using the analysis manager in Sawtooth using a logit model. I am able to get comparable results for coefficients using effects coding.

However, in Sawtooth, the log likelihood, chi-square statistic and AIC differ substantially from the results in R. Is there documentation somewhere that explains exactly how Sawtooth estimates partial profile designs?
asked Oct 13 by Alex

1 Answer

0 votes
There is some information in the following forum post:

https://sawtoothsoftware.com/forum/14403/information-criteria-in-analysis-manager-aic-caic-bic

In addition, the chi-square and AIC are calculated as follows:

ChiSq = 2* (LL - nullLL) [Where nullLL is the null log likelihood for the solution]
AIC = (-2 * LL) + (NP * 2)

The null log likelihood is computed by taking the sum of the logs of the number of concepts in each choice task across all respondents.  The log likelihood is a bit more complex, but just like in that forum post realize that the data are being coded where things like tasks without responses have been removed, etc. and could account for minor differences.

If you would like to know how it is calculated in more depth, please feel free to email me at walt at sawtoothsoftware.com.
answered Oct 13 by Walter Williams Gold Sawtooth Software, Inc. (14,100 points)
...