Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

t-test for hb utilities


I just read your information in Sawtooth Software Forum about how to check for uncertainty of HB part worth utilities using a t-test. I am not sure if I did it the right way. This is how I did it:
1)    I opened the alpha.csv output file and deleted the first 5.000 iterations. (Is this a good value to find convergenz?) Then I calculated the standard deviation of every part worth utility using the remaining 15.000 iterations. (s.d. = 0.36)
2)    Then I calculated the S.E.=0.36/(15.58)   -->  n=243; S.E.=0.0231
3)    Afterwards I sorted those 15.000 iteration of the part worth utility in an histogram to find values for the 95% confidence interval (which are the alpha values of 2.5 and 97.5 in the histogram).
4)    Finally I  calculated the t-value:   “Average Utility” value = -11.61/0.0231=-504.53 (-11.61=“Average Utility” value from the report.xls file). When my the two alpha-values  are -1.11 and 0.31, my t-value is outside this range and is significant.

Could you tell me if this is the right way?

Thank you very much!

asked Oct 16, 2016 by isabelle (180 points)
edited Oct 16, 2016 by isabelle

1 Answer

0 votes

I think you may have some unnecessary steps.

But first, I would discard the first X iterations, where X is the number of preliminary or burn-in iterations you specified in your HB run (you already made a guess about when you could safely assume convergence so just use that guess).

I don't think you need to calculate a s.d. for statistical testing because you have the entire distribution to work with - you don't need to create summary statistics from it in order to use it.  Just count the number of iterations where the value exceeds zero and if this is more than 95% of iterations then you can conclude with 95% confidence that the value for that parameter exceeds zero.
answered Oct 17, 2016 by Keith Chrzan Platinum Sawtooth Software, Inc. (50,675 points)
Thank you very much for your answer. I just have one more question to make sure I understood you correct.
If you say I have to count how often the alpha-values exceed zero, you mean the absolute values? So I count if values are negative? Moreover I am not sure if I can count a 0.4 or if the value has to exceed the 1.0 to count it?

Because it seems like a lot of part worth utilities in my study are not significant different from zero. Does this mean that my results from market simulator are not proper if I use those part worth utilities to do market simulations?

Thank you!!!

I mean positive values, not absolute values if we want to test that a parameter is greater than zero.  Negative values (not absolute values) if we want to test that it is less than zero.  

We expect that some utilities will not be significantly different from zero because the effects coding we use zero-centers each of the attributes.
I get somehow confused. My goal is to find out something about the uncertainty of my part worth utilities.  In "https://sawtoothsoftware.com/forum/9259/meaning-of-st-deviations-in-hb-results" it was said that a t-test has to be done and by calculating s.d. from the alpha draws or did I missunderstood something?

If you tell me that some utilities are not signigficantly different from zero because they are zero centered, how can I get reliable information about how good my part worth utilities are?

Thank you very much for your support!
Yes, you COULD conduct a t-test using an estimate of the standard deviation, but that would make for a hybrid between a Bayesian and a frequentist test.  You could also conduct a Bayesian test just by counting whether more than 95% of all draws exceed zero or not.

You can use the standard deviations, standard errors or confidence intervals, whether frequentist or Bayesian, to draw conclusions about the precision of your utility estimates, even when the estimates themselves are close to zero.