Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Measuring Extent of Redundancy in a Simulation Scenario

Hi All,

I am aware that RFC is the best cure for simulations where scenarios with redundancy (of some extent) is possible.

But at the same time we are trying to use other simpler simulation methods whenever possible for other reasons (ease to use of a simulator on the client side on any desktop with Excel as opposed to have them install plugin on dedicated machine only, the need to run extensive optimization search which with RFC becomes a pain etc.).

In that sense  we always review what types of scenarios they are interested in. Sometimes we have a green light for a simple Share of preference or First Choice. Sometimes we would warn them not to run something that would have somewhat redundant options.

But it would be best if any market simulator would have auto-detection of  a scenario if it has some redundancy in it. I mean not a complete redundancy, which is simple, but some soft version of it - almost the same product in a scenario  with only 1 or 2 minor features different.

I am wondering if anyone came up with any idea of how to measure this extent in a scenario? I guess a simple look at product specifications only (by counting how many attributes & levels are the same vs different between products) might not work as importance of attributes varies and is not homogenous

Thanks for any ideas or references
asked Jan 2 by furoley Bronze (825 points)

1 Answer

0 votes
Well, RFC is one potential remedy for problems involved with simulating scenarios involving pairs of products that are highly similar (share many of the same attribute levels).  But, it's not the only solution that has been proposed.

It seems you'd like the benefits of RFC without the requirement of the plug-in.  A straightforward and academically-sound approach is to take about 30 to 200 of the respondent draws from HB (the estimates of beta across different iterations of the HB algorithm, such as every 100th draw starting at the 10000th draw), then to apply either the first choice or "share of preference" decision rule in simulations.

This approach should be essentially as good as RFC.  How many draws you use per respondent will of course affect your simulator's speed.  And, you'll need to make adjustments to more correctly report the sample size and to estimate the standard errors (to avoid making it look like you have 30 to 200 times larger sample size).
answered Jan 2 by Bryan Orme Platinum Sawtooth Software, Inc. (152,255 points)
Thank you Bryan for your suggestion! If I understood it right, only "First Choice" would work well in resolving redundancies with multiple draws per respondent. The "Share of preference" would give me biased share for "None" in a scenario with redundant product regardless if I am using 1 or 200 sets of utilities per respondent, right? Having 30-200 sets would not resolve for this, unless it is used with First Choice decision rule.

I like this suggestion - Thank you very much! We definitely will use this technique.

Thinking in this given direction, I feel it can be modified (is not improved) by exporting std. errors of HB estimates and then using this table of errros with random normally distributed factor in order to reproduce equivalent estimates  from whatever number of draws I need.
I guess the same computational effort but much less data to use.
Would it work?
I think you're right that applying the First Choice rule on the draws is a stronger way to correct for product similarity than applying the standard logit equation (share of preference) rule.  However, the logit rule when applied on the draws will tend to reduce product similarity problems just a tiny bit better than logit rule on the point estimates (one summary set of utilities per respondent).

Simulating draws can also work, either applying empirically estimated standard deviations by parameter, or by doing a more complete job by simulating draws from the upper-level covariance matrix.
Thank you Bryan, your answers are always mind-opening. Really appreciated