Sawtooth Software: The Survey Software of Choice

Testing for Significant Differences in Conjoint Analysis

When analyzing conjoint analysis data, we may wish to conduct statistical tests to detect significant differences. Common situations include...

  1. Testing whether the share of preference for one product is significantly different from another, or if shares differ by respondent segments.
  2. Testing whether one attribute is more important than another, or a level has a higher utility than another (within the same attribute).

Performing statistical testing as described in this article requires individual-level part worth utilities.

Tests between Two Products

  1. Generate an Individual Shares file by checking Individual Results to File from the Scenario Specification dialog in the Market Simulator. This generates a file of product shares for each respondent called studyname.shr.
  2. Using the individual-level results (perhaps you open this file using Excel or your statistical package), compute a difference in a new column (variable) between the two products (Share1 – Share2).
  3. For this new variable representing the difference, calculate a mean and a standard error across all respondents. (The standard error is equal to the standard deviation divided by the square root of the sample size.)
  4. Compute a t statistic by dividing the mean difference by the standard error of the differences. A t-value of absolute magnitude greater than 1.96 indicates a significant difference at the 95% confidence level.
Differences between Two Segments for the Same Product

When running market simulations using individual-level utilities, a share and standard error are reported for each product in the simulation scenario. To test for a significant difference between a product’s share for two unique respondent groups (such as males vs. females), we first compute a t-statistic:

Where the subscripts 1 and 2 refer to the respondent groups 1 and 2, and SE refers to the standard error of the shares, as reported in the market simulator.

A t-value of absolute magnitude greater than 1.96 indicates a significant difference at the 95% confidence level.

Testing Differences for Importances/Part Worth Utilities

  1. Export the Importance scores or normalized utilities (by the Points, Diffs, or Zero-Centered Diffs method) to a comma-separated values (.CSV) file using the Run Manager + Export option.
  2. Open the file in Excel (or a statistical software package). Create a new column (variable), where the new variable is equal to the difference between the importance scores or utility values of interest. If comparing part worth utilities, remember that you should only compare levels within the same attribute.
  3. Compute a t-statistic by dividing the mean difference by the standard error of the differences. A t-value of absolute magnitude greater than 1.96 indicates a significant difference at the 95% confidence level.
A Discussion of Standard Error

In conjoint analysis, the standard errors reflect the uncertainty in the preference estimates due to sampling and the uncertainty regarding the estimated part worths. They do not completely characterize the accuracy of the model. There are other sources of inaccuracy such as bias or misspecification that are not captured in the standard error.

Standard errors decrease as the sample sizes increases. Quadrupling the sample size cuts the standard error in half.