Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Holdouts in ACBC

Dear Forum,

I'd like to add CBC-Holdout questions to my ACBC-Survey to test within-sample validity. I have the 9.4 Full Version.  I encountered the following problems:

1. The style of the CBC holdout tasks is completely different to style of the ACBC tournament tasks and I did not find any options to change it to ACBC-style (I did not even find a manual option).

2. When I test the survey and the first holdout tasks appears and I select a concept and click next, the survey jumps right to the end skipping all the other tournament choice tasks of the ACBC.

I would be more than happy if someone can help me out on these two issues as I did not find any help in the manual. Also does anyone know of a Best Practice of including Holdouts in ACBC, which explains the process step by step? Are there any other options to test within sample validity in ACBC which is simpler than using holdouts?

Thanks and best regards,
asked Oct 12 by Benedikt

1 Answer

0 votes

I would add the CBC holdouts AFTER your ACBC questions complete and I wouldn't worry that their formatting is a bit different.  If they're at all good,  your utility estimates should generalize well enough to questions with a different look.

What standards for in-sample validity are you trying to meet?  Are you assessing different model forms or are you just wanting confirmation that the ACBC predicts well to holdout CBC questions?
answered Oct 12 by Keith Chrzan Gold Sawtooth Software, Inc. (48,525 points)
Hi Keith,
thanks for the quick reply. Moving the CBC holdouts to the end solved the 'second' problem, thanks.
In essence, I'd like to calculate hit rates to predict how well the part-worth utilities for each person can predict which product concept the respondent would choose in the holdout sets.  Unfortunately I did not find any information on acceptable hit-rate levels for ACBC, so based on Orme's HB Priors paper, I am guessing that hit rates of around 70% are acceptable.
Moreover, I expect that I will end-up with two .CHO files, one is the actual ACBC survey and one is the CBC holdout tasks. Is there any paper/webinar on how to merge the .CHO files.    

Or is there any other easier way to calculate hit rates (for example by having holdout respondents)?

Sorry I am bit lost here. Until now the Lighthouse Studio Help pretty much  solved any issue for me, but I did not find anything on how to implement ACBC Holdout Tasks to calculate Hit Rates.    

Thanks for the help,

I don't see why hit rate targets should be different for ACBC than for CBC, so whatever target you like for the latter should work for the former.  

You can't get hit rates for holdout respondents, because you need the same respondents to complete the estimation utilities and the holdouts, which won't be the case if the respondents themselves don't also supply your estimation choice sets.   If you go the route of holdout respondents you'll want to measure the mean absolute error or the correlation of actual and predicted shares as your fit measures.  

I don't know how you'd merge the .cho files or why you would need to do so.
Dear Keith,
thanks for your reply and patience.  Based on the comment from Bryan Orme (https://sawtoothsoftware.com/forum/12907/holdout-cards-within-acbc) I thought you need to merge the two different .CHO files, which requires "excellent data processing skills". But maybe I am mistaken here.

Also I still struggle with the two completely different formats of the choice sets for utility estimation and the holdout choice sets (see: https://picload.org/view/dgipprar/unbenannt.jpg.html . I am afraid that respondents might stop the survey as they might think  the survey broke down. I just don't want to give any room for errors. Is there no way to change the format of the holdouts? I mean I should not be the first to have such kind of problems, right?

Thanks a lot for the great support!

When Bryan talks about merging the .cho files, he's describing what it would take specifically to add the extra CBC questions to the ACBC questions for utility estimation, NOT for purposes of analyzing holdouts - for the latter purpose that would be a wholly unnecessary task.

I suppose you could make free format questions to make your CBC tasks  look like the ACBC tournament questions, but that seems like a lot of work.  If your concern is that the format is different and that might change respondents' answers, well I'd hope the utilities we get from any kind of choice model are robust enough to survive modest differences in question format.  If your concern is that respondents will become confused, then add a page with a sentence of brief instruction, to the effect of "OK, two last questions and then we can move on."