Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

R-sq in individual utilities


I have a question related to the "R-Sq" shown in the ACA_ols_rep for each respondent for individual utilities.

There are some respondent with an R-Sq 0 and some of them 1000.
0 or all respondents with an R-Sq lower than 500 haven't answered consistently, aren't they? My second question relates to the R-Sq of 1000 ... I thought a R-Sq couln't be 1 or rather 1000. Or have these respondents extremely good answers (it doesn't look like).
Which kind of R-Sq. are you using for your estimation?

Thanks alot and kind regards,

asked Feb 13, 2013 by anonymous
retagged Feb 13, 2013 by Walter Williams

1 Answer

0 votes
The R-Sq shown in ACA_OLS is a very simple regression based on just one independent variable and one dependent variable.  Let's say the respondent was given just 3 Calibration Concept tasks.  We first compute OLS utilities across all the ACARAT and ACAPAIR questions, creating individual-level utilities (we don't report the r-squared for that regression step).  Next, we compute the total utility (from that first step) of each concept shown to respondents in the calibration concept step.  Put that in one column (with three rows in this case, where I'm referring to just 3 Calibration Concept questions).  Then, do a logit transform (log-type transformation of the respondent's answers...I won't give details here) for the Calibration Concept ratings.  Place those values in the second column as the dependent variable.  Then, regress the dependent variable on the independent variable (also fit the intercept).  It's not uncommon to be able to get an R-squared that rounds to 1.0 for that (and historically we have multiplied by 100 when reporting that statistic).  

Respondents who get negative betas in this last calibration step are marked with a 0 r-squared (even though the r-squared was certainly greater than zero).  We do this to flag them, because their calibration concept responses are in a pattern that is opposite of the pattern that their earlier ACA responses would predict.  The more calibration concepts you have asked (say, 6 or more), the less likely that r-sq will be 1000 for respondents.  But, typically quite a few respondents just don't understand the Calibration Concept step in ACA and give illogical answers.

I'm not a big fan of Calibration Concept questions for ACA.  The main reason you need them is to be able to predict purchase likelihoods in the market simulator (an "iffy" procedure, since respondents are so bad at guesstimating their purchase likelihoods).  My preferred practice in general is to not ask the calibration concepts and spend those extra moments asking a few more Pairs questions.  Then, I use ACA/HB estimation to obtain better utilities than OLS.  Also, the "Exponent" typically needs to be tuned a bit higher than 1.0 within the market simulators to give better fitting probability predictions of choice shares.  (Good to have two or three holdout choice tasks within the same survey, so you can tune the magnitude of the Exponent to best fit choice scenarios).
answered Feb 14, 2013 by Bryan Orme Platinum Sawtooth Software, Inc. (154,405 points)
Thank you so much Bryan!!
That is exactly the answer to my question!!!

I did what you recommend: I downloaded my data again from the web surver and put them into the SSI Web Lab and then did an ACA/HB estimation instead of an OLS.

I am right, that I can handle these individual utilities the same than these estamiting with the OLS, am I?
I put these individual utilities into SPSS for an linear regression. So the output of the ACA/HB is basically the same than with OLS but a little bit more exact - is that right?
Good.  I'm glad you're using ACA/HB.  ACA/HB actually produces an R-squared based on the consistency of ACARAT, ACAIMP, and ACAPairs questions.  And, ACA/HB also offers an additional step that allows you to calibrate the utilities to the Calibration Concepts (and the R-Sq is replaced by the R-Sq from that last calibration step...but I don't recommend this).  

The ACA/HB utilities (without calibration) should be able to be used within market simulators just like the ACA_OLS results...though they should be better.