Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

ACBC Test Design: Warnings

Hey guys :)

I prepared a ACBC and tested the design for 250 people. After an hour of processing, the level counts are all 4 and above and I get standard errors between 0.02 and 0.49 for my attributes.

However, I also receive a lot of warnings.

I received "Design optimize timed out. Proceeding with current concepts." about 254 times.

And 13 times warnings like:
>"Cannot create a new concept after 1000 iterations. Turning off avoid dominated concepts code."
>"Cannot create a new concept after 2000 iterations. Turning off balancing code."
>"Cannot create a new concept after 20000 iterations. Proceeding with 1 concepts."

What do these warnings mean? There is not a single line in the excel file without a warning.

Thanks a lot for your help!
asked Dec 10, 2015 by anonymous

1 Answer

+1 vote
Best answer
The .49 standard error...I hope that was on the "None" parameter and not on a level of an attribute.  I'm assuming so.  All standard errors for attribute levels should have 0.05 or less in typical applications with decent sample sizes (such as n=250).

It's extremely typical (with moderate to large-size attribute lists) to get a "timed out" message for every respondent.  That just means that all the possible relabelings and swaps that could have been done to further optimize the design were not done.  This doesn't mean that the design is bad, it just means that within the half second or so allotted for each respondent to find near-optimal designs, it couldn't do all the possible searching that the algorithm could have permitted--it had to break out early.  Typically means you got 99.5% of the way there, not 100% of the way there...which is splitting hairs and just doesn't matter.

The turning off "avoid dominated concepts mode" meant that the algorithm got stuck when trying to avoid dominated concepts within the set of near-neighbor cards, so it turned off the code preventing dominated concepts.  In the grand scheme of things this isn't such a big deal, since for years ACBC didn't even have an option for avoiding dominated concepts and it performed very well.  But, if it was really important to you that every respondent never sees a dominated concept, then this is a problem.

"Turning off balancing code" means that the algorithm was struggling so much to generate near-neighbor concepts that it turned off its checks to find decent level balance across non-BYO chosen levels.  Could mean some loss in efficiency for such respondents, so I'm glad this message occurred very few times.  

The last message "proceeding with 1 concept" would indicate that the algorithm just couldn't come up with near-neighbor concepts and proceeded with just one concept.  Do you have lots of attribute prohibitions?  This would be a bad thing to happen to a real respondent.  It means the ACBC survey couldn't proceed beyond the BYO question and would have skipped out of ACBC onto the next question in SSI Web.
answered Dec 10, 2015 by Bryan Orme Platinum Sawtooth Software, Inc. (148,140 points)
edited Dec 10, 2015 by Bryan Orme
Thanks a lot for your answer! That helped a lot.

I'm sorry - the standard error of 0.49 was an error. I meant 0.049 :-)

The "turning off avoid dominated concepts code" message occured only 4 times.
The "turning off balancing code" occured 5 times.
And the "proceeding with 1 concepts" occured 4 times. Two of which said 0 concepts and 1 said 3 concepts.

Actually I have no attribute prohibition. I included 6 attributes plus the price as an attribute. 5 of these attributes have only 2 levels and 1 attribute has 3 levels.

To be honest, I don't know if it is important whether every respondent never sees a dominated concept. It's the first time I'm using this software and doing this kind of analysis. But I supposed it is not important.

Also: Is a d-efficiency w/BYO of 1.0 and w/o BYO ranging between 0.71 and 0.84 ok?
Glad to hear the 0.49 was a mistype.  

I'm curious why 4 of your test respondents failed to be able to generate full designs.  But, I think I may know the answer:

You say your design is 2x2x2x2x2x3 plus "summed price" as the 7th attribute.  That means (aside from summed price), there are only 96 possible "cards" (product concepts) that could be generated.  And, that's without all the restrictions that ACBC is placing on the designs:

I don't know your survey settings, but you probably told the software that to generate a near-neighbor concept, it could only vary at most X attributes from the BYO-selected concept (I'm assuming you are asking a BYO question for probably all the attributes?  Or most of them?).  This dramatically reduces the pool of possible product concepts it could generate.  There are only 96 possible unique products (aside from price).

Avoiding dominated concepts also reduces the pool of possible products.

So, I think for 4 of your 250 respondents that the design algorithm got stuck and couldn't find enough concepts somehow?  It still seems strange to me that it quit after only finding 0 , 1, or 3 concepts.  

So, please review what your settings are for the maximum number of attributes allowed to vary to produce near-neighbor concepts.  I think if you increase this by 1 you will avoid having the possibility that some respondents (4 out of 250 in your test) cannot have a design generated for them.

As we say in our ACBC documentation, D-efficiency much lower than 1.0 is expected, due to the nature of our pivot (near-neighbor) designs which oversample greatly the BYO-selected levels for each respondent.  The most important thing with ACBC for individual-level designs is to see that each non-BYO selected level occurs at least 2x and preferably 3x per respondent.
Yes, I do a BYO with all attributes. And I used the manual's recommendation on the settings for the design.

So I have:
No. of screening tasks: 6
No. of concepts per screening task: 3
Min. attributes to vary from BYO selections: 1
Min. attributes to vary from BYO selections: 3
BYO-product modification strategy: mixed approach
Number of unacceptables: 3
Number of must haves: 2
Max. no. of product concepts brought into choice tournament: 18
No. of concepts per choice task: 3
No. of calibration concepts: 0
Avoid dominated concepts: yes
Include BYO in tournament: yes

Does that setting make sense for my number of attributes and attribute levels? Or shall I not avoid dominated concepts? I mean I don't have any preference order settings - so I wonder if not setting the check for "avoid dominated concepts" would change anything.