Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

CBC sample size NTA/C>1,000 rule of thumb versus simulation std err >.05

I have a design that gives me the following (150*12*4)/6 = 1,200 which IS greater than the more conservative 1,000 rule of thumb.  However, when I run an advanced simulation, my biggest attribute which has 6 levels is giving me standard errors in the .06 range...which is higher than that test's <.05 rule of thumb.  I would guess the simulation is more robust but .06 is not that much higher than .05.  I would need my sample size to be closer to 215 to get my errors to .05 or below.  

Curious about the perspective here?  Is it critical that we get the sample up to 215 (or higher) or a nice to have or somewhere in between?  Realizing that more is always better, from a practical standpoint, am wondering how much to push the add'l costs and my client to move it up - its always a matter of degree.

asked Sep 9, 2014 by stevetlg.com (425 points)
retagged Sep 9, 2014 by Walter Williams
Oops, I meant <.05

1 Answer

+1 vote
You're right that the simulation in the advanced test is more robust.  You're also right that 0.06 isn't that much greater than 0.05.  

The thing about sample size is that it's not like a cliff:  0.05 or less and you're always completely safe every time and, oh no, 0.06  you're over the cliff, your study crashes and burns.  

Sample size is certainly a diminishing returns sort of thing:  as you note, in your case you need to push it from 150 to 215 to get a small decrease in standard errors.  

So it's a judgment call, but the increase in precision in this case seems almost trivial compared to which I expect will be significant push back from the client when you try to increase sample size by over 40%!
answered Sep 9, 2014 by Keith Chrzan Platinum Sawtooth Software, Inc. (73,175 points)