Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

ACBC Test Design


I am currently building an ACBC survey (7 attributes, 5 levels each) to field to a relatively small sample of ~60 respondents. As I have a couple of prohibitions (At1L1 with At2L5 and At1L5 with At2L1), I tested the design of the module to make sure we'll be OK once in field. Our typical ACBCs of that size include:
- 8 screening tasks (4 concepts per task)
- 3 must haves and 4 unacceptables
- 16 tasks in choice tournament (3 concepts per task)
- 4 calibration concepts
- we do not include BYO in tournament.

Looking at the excel file from the testing (run for 60 simulated respondents), I have two respondents who did not see a particular level at all, although these were not the prohibited levels. All my standard errors are below 0.1 (but above 0.05, probably because of the small sample size, they tend to hover around 0.07-0.08), including those for the prohibited levels and the levels these two respondents didn't see.

Based on your documentation, it would appear that this is NOT a valid design, as two levels were never shown to two of the 60 respondents, however I do not fully understand why this might have happened, given the lack of prohibitions for these attributes/levels. How should one handle this?

asked Dec 2, 2014 by Orestis Mavroudis

1 Answer

+2 votes
Best answer
With ACBC studies, especially with relatively small sample sizes as you have, it is preferred if each level of each attribute appears at least 2 times and preferably 3 times per respondent.  However, recognize that ACBC purposefully oversamples the most preferred levels per respondent (those chosen in the BYO task), so it is expected that some levels will appear many fewer times than others.  (But, you are seeing some respondents see some levels zero times, which is not ideal.)

It’s interesting to note that even if a particular respondent never sees a level in the near-neighbor design (after the BYO question), HB estimation can still impute quite useful information from the other respondents regarding that level (especially for ordered attributes).   So, all is not lost!

I have tried to create a parallel design similar to yours here at the office to test. However, you have not let me know whether your attributes are ordered or not ordered, whether they have preference direction or not, and also (critically) the MIN and MAX number of attributes you are allowing ACBC to change from the BYO selections when creating near-neighbor concepts.

Our documentation suggests for 7-attribute problems to vary at MIN 2 attributes from the BYO choices and at MAX 3 attributes.  So, I have chosen those settings.  I also specified that all your attributes are ordered, along with your 2 prohibitions.

Another thing you didn’t specify, but that can affect your design balance a lot is whether you are asking ACBC to try to avoid Dominated Concepts.  I turned that off in my test, because I know that avoiding dominated concepts is a conflicting goal with achieving level balance.  Because I know you are struggling with level balance, I decided to turn Avoid Dominated Concepts off.

With my test, all sixty of my test respondents get each level at least 1 time.  But, 15 respondents have at least one situation where a level shows just once for them.  

Perhaps you aren’t allowing enough variation from BYO concepts when generating near-neighbor concepts?  Are you using something like MIN=2, MAX=3?

Here are my thoughts to help obtain more information for each level in your study under ACBC:

1.    See if you can reduce the number of levels for any of the attributes
2.    Turn off the “Avoid Dominated Concepts” setting
3.    Increase the variation from BYO concepts to MIN=3, MAX=3.
4.    See if you can increase the number of screening tasks to 9 instead of 8

If after reviewing this you are still stuck, you could send me your .SSI file to “bryan at saw tooth software dot com” then I could look specifically at your project setup and help you further.
answered Dec 2, 2014 by Bryan Orme Platinum Sawtooth Software, Inc. (148,140 points)
Thank you for the very thorough response Bryan. I've managed to resolve my issues using a combination of your suggestions.

All our attributes are ordinal, and we specify best/worse & worse/best relationshiops, and skip a BYO task upfront. Prior to my modifications, we set a Min of 2 and MAX of 4 for near-neighbour concepts.

It appears that the greatest benefit came from switching off the "avoid dominated concepts" setting, but the best results came from when I also increased the screener questions to 9, and the MIN/MAX to 3 & 5. The minimum number I now show a level is 4.

Thanks again,