Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Sensitivity analysis in market simulator with alternative-specific design

Hello everyone,

I will explain my intention using the example of the Advanced Design Module technical paper (in a simplified form)

CBC-Question: If you lived in mile away from your workplace in downtown New York and these were your choices for getting there which would you choose?

There are two alternatives (primary attributes):
Ride the bus
Drive my car

And there are three conditional attributes for each alternative. For example:

The Bus has a pickup-frequency

Picks up every 20 minutes
Picks up every 15 minutes
Picks up every 10 minutes
Picks up every 5 minutes

and there is a parking fee for the use of the car:

Parking fee $5.00/day
Parking fee $8.00/day
Parking fee $10.00/day

Now, I would like to examine when a respondent decides for bus and when for the car. I would like to use the market simulator for that (or is there a better alternative?)

In the market simulator, I would always compare the two alternatives with each other and adjust (for example) the pickup-frequency for the alternative “Ride the bus” and run the simulator again (all other attributes remain the same)

At the end, I would like to have such a "sensitive-analysis" table, which I can represent graphically:

pickup-frequency: 5        10       15      20
Bus:                              0.8     0.6     0.4     0.3
Car:                               0.2     0.4     0.6     0.7

The values in the table could then be utilities or share of preference. As a result, it can be said that the car becomes more attractive when the bus pick-up time is 15 minutes (all other attributes remain the same).

I would also like to do this analysis for all other attributes to see when the bus wins and when the auto wins.

My question:
Which market simulator method is best for this? (assuming the market simulator is the best solution)

I tend to use the Share of Preference Model because I'm interested in the Information about the relative preference for the remaining products. But I can't use the CBC/HB-utilities for that, right? And then there's the IIA problem. But in my opinion, the products "Ride the bus" and "Drive my car" would not be similar in their attributes.

Thank you for your help!
asked Oct 11, 2019 by mlemtes (130 points)

1 Answer

+1 vote
The Share of Preference model is another name for the logit equation, which is the math aggregate logit, latent class logit, and HB logit uses for predicting respondent choices and measuring the fit of the model.

You say you worry about IIA (Independence from Irrelevant Alternatives) problems.  If using aggregate logit (one set of utilities fitting on average all of the respondents), then IIA problems are at their worst.  IIA problems include constant substitution rates between products and share inflation for similar products.

One of the reasons that HB logit utilities work so well is that IIA problems are significantly reduced for predictions of sample (market) preference due to representation of heterogeneity.  While IIA still holds within the unit of analysis (the individual), when the share of preference votes are accumulated across respondents, the average of those shares is not bound by IIA.  

The HB utilities for CBC experiments are regularly used in  market simulations and sensitivity simulations.  This is the most common approach among Sawtooth Software users for building market simulators, rather than using aggregate logit or latent class logit.  

Another simulation approach that can work well is Randomized First Choice.  One caution is that attributes like price probably shouldn't have the correction for product similarity turned on.  Depending on which software you are using for this, the correction for product similarity is automatically turned off.  (You can open the RFC settings to ensure that has happened).
answered Oct 11, 2019 by Bryan Orme Platinum Sawtooth Software, Inc. (172,790 points)
Thank you for your answer, Bryan

Is the market simulator in lighthouse generally a good way for a sensitivity analysis?

And is this procedure correct in principle? :

In the market simulator I would compare the alternative “Ride the bus” and "Drive my car" and use a range for the pickup-frequency for the alternative “Ride the bus” and run the simulator.

As a result I get a table as described above, right?

I don't think IIA can become a problem because I would only compare two products in the simulator and that would be "ride by bus" and "drive my car" - both have different attributes. Or am I being wrong here?

I would like to use the model "Share of Preference" in the market simulator for that. But I can't use the CBC/HB-utilities for that, right? But it runs in Lighthouse Studio 9.8.0

How would you proceed in this example to conduct a sensitivity analysis as described?
HB utilities are exactly what you want to use in your market simulator.  If your CBC study was composed within Lighthouse Studio, then you'll be using the built-in HB estimation that is included in the Analysis Manager within Lighthouse Studio.  It is the same algorithm as CBC/HB standalone system uses.  And, of course, if you do the design and analysis of the CBC study within Lighthouse Studio, then you can use the built-in simulator that's within Lighthouse Studio for conducting your simulations.

If your CBC study was designed outside of Lighthouse Studio, then you're needing to use the standalone CBC/HB program to estimate the utilities.  Then, those utilities can be read into our standalone Choice Simulator program, which is identical to the market simulator that's built into the Lighthouse Studio system.  If you have a license for Lighthouse Studio, then you also have a license for the standalone choice simulator--you just need to install it from our website.

IIA (share inflation for similar products) is not an issue for a two-product simulation scenario as it is with three products or more.

You can run sensitivity simulations and then take the data and formulate it in a table for presentation to others, or line graphs, etc.

In sensitivity simulations, you typically set the comparison (fixed) product alternatives to their average or middle attribute levels.  Then, the test product is varied one-at-a-time across each of its levels.  The other attributes not involved in the sensitivity test are typically returned to their middle or average level.
Thank you Bryan. That helps me.

I built my design in Lighthouse Studio and conducted the HB analysis with the Analysis Manager.

Is it always necessary to use the middle level for the other attributes?

Some of my attributes have only two levels and some attributes are not metric (for example: WLAN available, WLAN not available, WLAN only for 15 min.)

Which method for the simulator would you recommend? I would like to use the model "Share of Preference" for that. But I can't use the HB-utilities for that, right? But it runs in Lighthouse Studio 9.8.0
If you are working within Lighthouse Studio, then you estimate HB utilities using the Analysis Manager.  And, when you click Simulator within Lighthouse Studio, you can select your HB utility run under "Utility Sets" on the toolbar.  

Yes, HB utilities are used in market simulators.  That's the most common thing that our users do for CBC and ACBC.  

When conducting sensitivity analysis, you can select "default" attribute levels that the simulator will use when that attribute isn't the one being changed.  You do this via the "Range" function when defining the levels to use for the test product.

The syntax is "=Range(start, end, increment, default)".

For example, specifying:

=range(1,4,1,2)

for an attribute of the test product tells the simulator (when in sensitivity mode) to investigate changes from levels 1 to 4 in increments of 1 unit.  And, when this attribute is not the one being varied, to set it back to level 2 as the default.  

Note, if you want the default level to be exactly between two levels, then you can specify an interpolated value for the default, such as 1.5.
Of course, you can just specify the attribute levels one and time and run the simulator one simulation at a time if you want the brute-force, non-automated way to do it.  In that case, you won't be using the "Range" function.  But, the "Range" function and sensitivity mode saves so much time once you learn it that I recommend it.
...