Sawtooth Software: The Survey Software of Choice

Randomized First Choice Simulations: Strengths, Weaknesses, and a New Refinement in SMRT v4.14

We’ve now had about eight years of experience working with the Randomized First Choice (RFC) simulation model. It’s the default method for market simulation within the SMRT platform. In general, it has worked well. But as might be expected, we’ve learned a few things: how to improve RFC, but also about some weaknesses.

Reflections

Eight years ago, most Sawtooth Software customers were using aggregate models: logit or latent class. RFC provided a clear benefit in these cases. Lately, most Sawtooth Software customers are using part-worths estimated under HB. According to our March 2006 customer survey, 68% of Sawtooth Software users typically use HB-estimated part-worth utilities for their final market simulation models. Once one obtains individual-level utility estimates, RFC typically provides only modest improvements. Thus, the popularity and effectiveness of HB has in turn reduced the impact that RFC has in our industry.

“IIA Meltdown”

However, there may be situations in which RFC simulations offer significant benefit over standard logit-rule (Share of Preference) simulations. In the 2004 Sawtooth Software Conference, Allenby et al. pointed out that standard HB models can face what they termed “IIA Meltdown” when very many alternatives (such as 84 alternatives in a beverage category or even more alternatives in the automobile category) are in the choice design. Although they proposed a model different from RFC, their finding that standard HB simulators face greater IIA troubles with large numbers of alternatives suggests that RFC may be even more useful in these cases.

Weakness with RFC and Price

We have also noted a weakness with RFC simulations. One of the problems with simple Share of Preference (logit rule) simulations is that often some kind of correction for product similarity would be useful. A strength of the RFC model is that such corrections occur automatically. The simple RFC model assumes that all attributes involve a correction for product similarity. However, it is not clear that this is always useful. For instance, price represents an attribute for which it isn’t clear that corrections for product similarity should be made as they are with attributes like brand and form factor.

Many analysts like to derive demand curves via sensitivity analysis within choice simulators. Under RFC, if all products are first aligned on the average price (and the “test” product systematically varied across all price levels), an unwanted kink can occur in the demand curves around the point that was artificially chosen as the average (reference) price. The kink is often slight and harmless when price sensitivity is strong and few products are in the simulation. But, as the number of products in the simulation increases and/or price has less impact, the kink becomes more noticeable and problematic.

In the example below, sixteen products were included in the simulation. We derived an estimated demand curve for the first product by holding the remaining 15 products constant at price 3 and systematically varying the price for just product 1. We plot the share of preference (under both RFC and “Share of Preference” simulation models) for product 1 below, as the price varies from level 1 to level 5.

When product 1 is changed from the average (price 3) to either the next lower or higher price points, it receives an extra boost in share due to becoming unique with respect to price. The penalty for product similarity results in a dip in share when product 1 is aligned with the competitors at price 3. Sometimes, the dip is deep enough that it can lead to a “reversal,” where increasing the price from level 3 to level 4 results in an apparent share increase.

Setting all products to price 3 for purposes of sensitivity simulations creates an unrealistic degree of similarity with respect to price. In the real world, we wouldn’t expect all products to carry the same price. This issue is especially problematic for RFC when conditional pricing is used, and thus level 3 of price really refers to different price values, depending on the brand (or other conditional attribute). When using conditional pricing, it clearly would make sense to turn off RFC’s correction for similarity with respect to the price attribute, but to retain it for other attributes.

RFC’s “automatic” correction for product similarity is produced by adding random error to respondents’ part-worths for each “draw,” with the same error values applying to alternatives that share the same level of an attribute. To turn off similarity correction for an attribute like price, we simply apply independent error (rather than correlated “attribute type” error) to the price part-worths for all alternatives in the simulation. This leads to a more sophisticated RFC simulator, where some attributes involve correlated error (when product alternatives share the same levels of those attributes) and other attributes involve uncorrelated errors (when product alternatives receive independent random error draws for these attributes, irrespective of shared levels).

We are now offering the ability to treat attributes differently with respect to correction for similarity (correlated error) within SMRT v4.14 (under the Method Settings… button on the Scenario Specification dialog). This is a free upgrade that may be downloaded from our website at www.sawtoothsoftware.com.

Maintaining Precision

We should also note that as the number of alternatives in the simulated choice scenario increases, the number of draws used in RFC should also be increased. Otherwise, the sampling error associated with the simulated shares of preference under RFC may be uncomfortably large relative to the signal associated with some relatively tiny product shares. By default, RFC simulations use 100,000 iterations (draws). As the shares of preference for some products drop to quite small numbers (especially below 5%), researchers should consider increasing the number of iterations significantly to avoid loss of precision.

For Power Users

There is another opportunity for analysts to improve RFC modeling that is a powerful idea, but involves extra data processing (is not automatically supported in the interface). Some conjoint/choice designs involve many alternatives. Beverages and automobiles are good examples. Suppose we had conducted a choice study with 200 automobile makes, including trucks, minivans, sedans, and coupes. Further suppose that we had treated the makes (for part-worth estimation) as independent levels of a 200-level attribute. However, we know that these 200 makes fall into four clear categories that should reflect increased competition within each category. One could assume a new attribute with four levels (truck, minivan, sedan, and coupe), each level with utility of zero, for which we apply attribute-type error under RFC in choice simulations.