Logit Analysis for CBC

Logit Analysis for CBC

Overview

The previous section described how to analyze CBC results using the Counting approach. This section introduces a more sophisticated technique for analyzing CBC data called multinomial logit. Multinomial logit is a general model for fitting choice data, but in this section, we'll be referring to a specific pooled (aggregate) logit model. We'll also refer to three relatively new methods which have been demonstrated to work even better in practice than aggregate logit: Latent Class, and HB.

These three methods are not available in the base SMRT system, but are offered in advanced add-on modules.

The Market Simulator uses utility (part-worth) information to estimate respondent interest (preference) for different product concepts. Before you can use the Market Simulator, you must compute utilities using either logit, Latent Class, or HB estimation.

Importing CBC Data

Prior to conducting this analysis, you must import the data from your CBC study. We'll assume that you have already downloaded the data from the Web server, and prepared the .CHO and .ATT files using the SSI Web system (by clicking Analysis | Prepare .CHO and .ATT Files for Analysis). To import the CBC data for use within the CBC Analysis Module and Market Simulator (SMRT):

Open the SMRT software by clicking Start | Programs | Sawtooth Software | Sawtooth Software SMRT.

Create a new study, by clicking File | New and providing a study name in the desired directory.

Import your CBC data by clicking File | Import, specifying Choice Data (*.cho) as the Import Type, and clicking the Import button.

What Are Part Worth Utilities?

A part-worth utility is a measure of relative desirability or worth. When computing utilities using logit, every attribute level in a conjoint project is assigned a utility (also referred to as a part-worth). The higher the utility, the more desirable the attribute level. Levels that have high utilities have a large positive impact on influencing respondents to choose products.

When using logit, Latent Class, or HB, the raw utilities are zero-centered within each attribute. For example:

Level Utility

$300 -0.6

$200 0.1

$100 0.5

This example shows respondents preferring lower price levels to higher ones. For information on interpreting conjoint utilities, please read Appendix B of the SSI Web manual.

Choosing Effects for Logit

By default, SMRT estimates utilities for all main-effects. Main-effects reflect the impact of each attribute on product choice measured independently of the other attributes. Main-effect models are the simplest models, resulting in a single utility value associated with each attribute level in the study.

You can add additional terms to the model to account for two-way attribute interactions. For example, if the combination of "Red" with "Mazda Miata" results in greater preference than the main-effect utilities would suggest, an interaction term can capture and reflect that synergy. Interaction terms can also reflect differences in price sensitivity for different brands.

We suggest only adding interaction terms that result in a significant improvement in the overall fit of the model. Including too many terms in a model can lead to overfitting, which has the potential of modeling a good deal of noise along with true effects.

When using aggregate logit, it is important to investigate whether including interaction terms can significantly improve the fit of the model. With methods that recognize respondent differences (Latent Class and HB), many complex effects (including, but not limited to, interactions) can be reflected in market simulations using only main-effects. It is our experience that including interaction terms in logit models can often significantly improve predictability of the model, but that those same terms added to Latent Class are not as valuable and can even, in some cases, be detrimental. Thus, with little effort, one can achieve excellent results with Latent Class and HB using main-effects.

Running Logit Analysis

Logit analysis is an iterative procedure to find the maximum likelihood solution for fitting a multinomial logit model to the data. When only main-effects are estimated, a value is produced for each attribute level, which can be interpreted as an average utility value for the respondents analyzed. After logit converges on a solution, the output is displayed in the results window, and you are given the option to save the run to the utility file. You can save multiple runs to the utility file, and each run is available to you when running simulations.

The computation starts with estimates of zero for all effects (utilities), and determines a gradient vector indicating how those estimates should be modified for greatest improvement. A step is made in the indicated direction, with a step size of 1.0. The user can modify the step size; a smaller step size will probably produce a slower computation, but perhaps more precise estimates. Further steps are taken until the solution stops improving.

For each iteration the log-likelihood is reported, together with a value of "RLH." RLH is short for "root likelihood" and is an intuitive measure of how well the solution fits the data. The best possible value is 1.0, and the worst possible is the reciprocal of the number of choices available in the average task. For these data, where each task presented four concepts plus a "None" option, the minimum possible value of RLH is .2.

Iterations continue until the maximum number of iterations is reached (default 20) or the log-likelihood increases by too little (less than 1 in the 5th decimal place).

The user also has the option of saving variances and covariances of the estimates. The square roots of the variances are equal to the standard errors, and are always displayed. The default is not to display variances and covariances.

Here is the output for a computation, using the same data as were used in the second example described in the previous section on Counting analysis (about 100 respondents, each of whom answered 8 randomized choice tasks, each including 4 concepts and a "None" option).

CBC System Multinomial Logit Estimation

Main Effects

Specifications for this run:

Max iterations 20

Variances and covariances not saved

Step size 1.00000

Max change in loglike 8e-007

Iter 1 log-likelihood = -1471.20588 rlh = 0.24828

Iter 2 log-likelihood = -1462.75221 rlh = 0.25028

Iter 3 log-likelihood = -1462.73822 rlh = 0.25028

Iter 4 log-likelihood = -1462.73822 rlh = 0.25028

Iter 5 log-likelihood = -1462.73822 rlh = 0.25028

Iter 6 log-likelihood = -1462.73822 rlh = 0.25028

Converged.

Log-likelihood for this model = -1462.73822

Log-likelihood for null model = -1699.56644

Difference = 236.82822 Chi Square = 473.656

Effect Std Err t Ratio Attribute Level

1 0.62150 0.05126 12.12462 1 1 Brand A

2 -0.05740 0.06021 -0.95331 1 2 Brand B

3 -0.26472 0.06411 -4.12943 1 3 Brand C

4 -0.29938 0.06509 -4.59957 1 4 Brand D

5 0.13859 0.04899 2.82895 2 1 Shape 1

6 0.07652 0.04962 1.54217 2 2 Shape 2

7 -0.21510 0.05199 -4.13734 2 3 Shape 3

8 0.15207 0.04895 3.10636 3 1 Large Size

9 0.04925 0.04939 0.99716 3 2 Medium Size

10 -0.20132 0.05201 -3.87086 3 3 Small Size

11 -0.52970 0.07101 -7.45947 4 1 Price 1

12 -0.22737 0.06409 -3.54773 4 2 Price 2

13 0.17347 0.05708 3.03928 4 3 Price 3

14 0.58361 0.05185 11.25616 4 4 Price 4

15 -1.07590 0.12434 -8.65299 NONE

The output starts with a listing of values that govern the computation. The values shown here are default values, and the user will rarely need to modify them. The default value for maximum number of iterations is 20, and this computation required only 6.

Logit analyses are often evaluated by Chi Square statistics. The procedure is to determine the log likelihood that would be obtained, given the sample size and the nature of the data, if the estimated effects were all zero. That log likelihood is compared to the log likelihood for the estimates obtained. Twice the difference between those two log likelihood values is distributed as Chi Square, with degrees of freedom equal to the number of parameters estimated.

The number of parameters estimated here is 11, obtained by adding the total number of levels and subtracting the number of attributes. With 11 degrees of freedom, a Chi Square of about 25.0 would be significant at the .01 level. The obtained value of 473.6 is safely larger than this, so we would conclude that respondent choices are significantly affected by the attribute composition of the concepts.

Within each attribute, the effects sum to zero. That is because we actually omit one level for each attribute in doing the estimation, and then supply a value afterward for the missing level that is equal to the negative of the sum of the others. Our use of "effects coding" makes this possible. In the logit analysis output, "effect" is synonymous with utility.

To the right of each estimate is a standard error, and to the right of that is a t ratio. The t ratio is a measure of the significance of the difference between that level's effect and the average of zero for all levels within the attribute.

When there are no interaction effects, as in this example, the relative attractiveness of a concept can be assessed by adding up the effects for its component attribute levels. For example, consider these two hypothetical concepts:

Concept 1 Concept 2

Effect Effect

Brand A 0.62150 Brand B -0.05740

Shape 3 -0.21510 Shape 1 0.13859

Small -0.20132 Large 0.15207

Price 3 0.17347 Price 1 0.52970

Total 0.37855 -0.29644

These two hypothetical concepts are each "scored" by adding up the effects of their component attribute levels. Concept 1 should be strongly preferred to Concept 2 by these respondents. In fact, we can go one step further and estimate how strong that preference would be. If we exponentiate (take the antilog of) each of the total values we can then express them as percentages to predict the proportion of respondents who would choose each concept if they had to choose one or the other:

Total exp(total) Percent

Concept 1 0.37855 1.460 66.3%

Concept 2 -0.29644 0.743 33.7%

Total 2.203

If forced to choose between Concept 1 and Concept 2, about two thirds of these respondents should choose Concept 1 and one third should choose Concept 2.

To show how the "None" parameter is used, we add a third item with total equal to -1.07590, the estimate for choosing "None."

Total exp(total) Percent

Concept 1 0.37855 1.460 57.4%

Concept 2 -0.29644 0.743 29.2%

"None" -1.07590 0.341 13.4%

Total 2.544

In a three-way contest between Concept 1, Concept 2, and the option of choosing neither, about 57% should choose Concept 1, 29% should choose Concept 2, and 13% should choose "None."

To see how logit output and analysis by counting choices output are similar, we use the same procedure to estimate the distribution of choices for four concepts differing only in brand, plus a "None" option. The resulting numbers in the logit output should be similar to the main-effects for Brand as reported in the output for analysis by counting choices.

Effect exp Prop- From Diff-

(effect) ortion COUNT erence

Brand A 0.62150 1.862 0.400 0.387 0.013

Brand B -0.05740 0.944 0.203 0.207 -0.004

Brand C -0.26472 0.767 0.165 0.173 -0.008

Brand D -0.29938 0.741 0.159 0.165 -0.006

None -1.07590 0.341 0.073 0.068 0.005

Total 4.655 1.000 1.000 0.000

Although the estimates derived from logit analysis are not exactly the same as those observed by counting choices, they are very similar. The differences are due to slight imbalance in the randomized design (and due to other factors mentioned at the end of the previous section). The estimates produced by the logit calculation are slightly more accurate. With a larger sample size we might expect differences between the two kinds of analysis to be even smaller.

After logit has finished running, results are displayed to the results window, including summary information about the computation (the total number of tasks and time for the computation). You are given the option to save the run to the utilities file. You can also print, or cut-and-paste the logit report to a file.

We emphasize that logit analysis is an advanced statistical technique that requires understanding to be used properly. We have made it easy to conduct logit analyses with SMRT. The burden is on you to use the results wisely. We urge you to consult appropriate reference materials. We also urge you to consider the limitations of logit analysis relative to the Latent Class or HB models.

Managing Multiple Logit Runs

In the process of analyzing your data, you will probably run logit a number of times. Some runs may test the inclusion of specific interaction effects; others may include just a subset of the sample or tasks. The Market Simulator includes a Run Manager that helps you manage this process.

After each logit run has completed, the results are displayed in the results window, and you are prompted to "save the run." If you choose to save the run, the utilities and set-up parameters (respondents included, tasks included, effects, etc.) are stored in the studyname.UCS file. If you do not store the run, the utilities are deleted. You can store as many runs as you want in the studyname.UCS file. You can view the settings from previous runs. You can also delete runs you no longer need.

The Market Simulator lets you choose a run to use for each simulation, and then reads the appropriate utility information from the studyname.UCS file.

Notes on Determining Significant Interaction Effects

By default, SMRT's logit program calculates all main-effect utilities. Sometimes main-effects models can be improved upon by adding interaction terms. We'll define an "improved" model as one that:

Significantly improves the model fit in terms of log-likelihood,

Improves the accuracy of the market simulator in terms of aggregate shares vs. fixed holdout choice tasks or market shares.

We urge you to only include interaction terms in the model that significantly improve results. To determine whether to include interaction terms, we suggest you first study the two-way tables provided by Counts analysis. The Chi-square statistic is one indication of possible significant interaction, but we suggest you look further. We also suggest plotting the Counts proportions with a scatter plot display to verify that deviations from proportionality in the table seem plausible given what you know about buyer preferences.

Most importantly, we suggest you conduct a "2 log-likelihood" test. We alluded to this test earlier in this section with respect to determining whether the computed logit effects significantly improve our ability to predict respondent choices relative to the null model (effects of zero). A 2 log-likelihood test can also be used to determine whether the addition of an interaction term to the logit model significantly improves the fit. We will illustrate with a simple example.

Assume a simple CBC design with two attributes, each with four levels. Under main-effects, a total of eight part-worths result. Recall, however, that each four-level attribute is coded as just three parameters in the logit model under effects-coding. Assume we run this main-effects logit model and achieve a log-likelihood of -1500.00.

The interaction between two four-level attributes, each effects coded as three parameters, results in an additional 3 x 3 = 9 parameters added to the model. Suppose we include this interaction term and the log-likelihood improves to -1490.00. Adding the interaction term has improved the log-likelihood by 10.00. Two times the log-likelihood is distributed as Chi-square. We refer to a Chi-square table (available in most statistics textbooks), and look up the p-value for a Chi-square of 20.00 with 9 degrees of freedom (the number of additional parameters added to the model). We find that the p-value is roughly 0.02, suggesting a significant improvement in the model by adding the interaction terms with a confidence level of 98%.

Technical Notes on Logit

SMRT uses multinomial logit analysis (MNL) to estimate effects for attribute levels. Additional information is provided by Hosmer and Lemeshow in Applied Logistic Regression (Wiley, 1989).

MNL is similar in many ways to multiple regression and discriminant analysis. Like those methods, it seeks "weights" for attribute levels (or for combinations of them, if interactions are included in addition to main effects). Those weights are analogous to "part-worth utilities" in conjoint analysis, and are computed so that when the weights corresponding to the attribute levels in each concept are added, the sums for each concept are related to respondents' choices among concepts.

MNL assumes that the relative probabilities of respondents' choosing each concept within any task can be estimated from the weights in the following way:

Sum the weights for the attributes appearing in each concept to get a value analogous to that concept's "total utility."

Convert total utilities to positive values by exponentiating them. The resulting values may be considered analogous to relative probabilities, except that they do not lie within the unit interval.

Normalize the resulting values so that within each task they sum to unity, by dividing the values for concepts within each task by their sum.

Suppose we have such probabilities for the concepts in one choice task. Then we can say that, according to our model, the likelihood of a respondent's choosing each concept is equal to the probability for that concept. If we assume choices for different tasks to be independent, then we can compute the likelihood (under our model) of seeing any particular pattern of choices for any number of respondents and tasks. That likelihood is just the product of a number of computed probabilities. If there are N respondents, each having responded to k choice tasks, then the likelihood of that pattern of responses is just the product of Nk such probabilities.

MNL chooses weights that maximize the likelihood of the observed pattern of respondent choices, using probabilities derived from the weights as just described.

Since likelihoods are obtained by multiplying many probabilities together and each is less than unity, likelihoods are usually very small positive numbers. It is more convenient to think about their logarithms, and users of logit estimation usually speak of "log-likelihoods" rather than likelihoods themselves. This is just a matter of convenience, because the set of weights that maximize the likelihood must also maximize the log likelihood.

In SMRT's logit routine we also report another somewhat more intuitive measure, "root likelihood," which we abbreviate "RLH." This is just the geometric mean of the probabilities corresponding to the choices made by respondents, obtained by taking the Nk'th root of the product of the Nk probabilities. The best possible value of RLH is unity, achieved only if the computed solution correctly accounts for all the choices made in all tasks by all respondents.

The computation method used by SMRT is iterative. It starts with a "null model" consisting of weights all equal to zero, and computes the likelihood of the respondents' choices, given probabilities corresponding to that initial solution. During that computation information is also accumulated about a "gradient vector," indicating how the initial solution should be changed for greatest improvement. Those changes are made, and a second iteration evaluates an improved solution, while also accumulating information about how a third iteration might improve on the second.

The iterative process continues until one or more of the following occurs:

The number of iterations exceeds a limit. The default is a limit of 20.

The change in log-likelihood from one iteration to the next is less than a limit. The default is 1 in the fifth decimal place.

We do not anticipate that users will need to change these defaults, but it can be done by clicking the Advanced Logit Settings button. The dialog includes four settings:

Output option. If checked, more information about variances and covariances of estimates is provided in the output.

Step size (default = 1.0). This number governs the sizes of changes made in the iterative computation. If set to a smaller number, such as .5 or .1, the computation will be slower but may be somewhat more precise.

Iteration limit (default 20).

Log-likelihood convergence criterion (default l in the fifth decimal place).

Effects Coding

Logit analysis, like least squares, suffers from the dependencies among attribute levels when all the levels of an attribute are entered into the computation. For this reason one level of each attribute is deleted automatically, although results for that level may be obtained by inference from the others, and those results are provided in the output.

Logit deletes the last level. Remaining levels are represented using "effects coding." This is a way of coding for levels so that the average effect within each attribute will be zero. After the logit computation, we provide the estimate for each deleted level just by computing the negative of the sum of the included levels. Logit analysis permits estimates not just of main effects, but also of two-way interactions.

Tests of Significance

SMRT offers two kinds of significance tests: a t ratio and an overall Chi Square test. Probably the more useful is the Chi Square for the difference between two "nested" models, where the second includes all effects of the first plus one or more others. Twice the difference between the log-likelihoods of those two models is distributed as Chi Square, with degrees of freedom equal to the number of additional effects in the second model. This test is performed automatically in each logit analysis, comparing the results of that analysis to what would be obtained with no effects at all.

The test can also be computed manually to compare any other nested runs, such as one run with all main effects and another with just a subset, or one with just main effects and another with main effects and one or more interactions. (The Chi Square of the difference is the difference of the Chi Squares.)

The other test is a t ratio provided for each attribute level. This tests the difference between that level and the average of all levels for that attribute (which is zero). These t ratios can provide useful guidance, but for measuring whether an effect is significant the overall Chi Square test is preferable.

The standard errors used to compute t tests are taken from the matrix of covariances among estimates that is obtained by inverting a sum of squares and cross-products matrix. If many parameters are being estimated or there are relatively few observations, this matrix may be ill-conditioned (having determinant less than 1.0E-10). in this case a "ridge adjustment" is made by adding unity to each diagonal position of the sum of squares and cross-products matrix, reinverting, and displaying an error message. We do this so the computation will provide useful information; however, the presence of a "ridge adjustment" message indicates the solution to be unsatisfactory, and you should probably remove some interaction terms and re-estimate.