Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

How can I know if an estimate is reliable?

Due to an error in preparing the design for a CBC exercise, I mistankely prohibited the appearance of one level of attribute to all brands, instead of excluding it from the exercise altogether. After data collection, I ran HB analysis for estimates and received the warning that the design was deficient and would not converge.
The result of the estimate showed an average RLH 0.604, which I consider rather high to the kind of data collected.
Is there any way to know if the estimates are really unreliable, due to the design being possibly deficient? Is there any way to treat the data post collection so it could make the data converge?
asked Jul 6 by prtlg (155 points)

1 Answer

+2 votes
 
Best answer
It may be a simple matter of (a) reconfiguring the design matrix and then (b) creating a new version of the .ssi file that isn't expecting the prohibited level and then (c) importing the response data before (d) re-running the analysis.  I would not use an analysis file that is giving you a message about the design being deficient.
answered Jul 6 by Keith Chrzan Gold Sawtooth Software, Inc. (48,200 points)
selected Jul 6 by prtlg
By the way, the RLH statistic may not be diagnostic - it is sensitive to the number of alternatives in the choice set.  With pairs or triples the 0.604 is in no way out of line and I think I've even done studies with quads where RLH was that high.  

If your prohibition is the ONLY problem with the design the analysis may be giving you the right answer, but I'm always more comfortable fixing the design and knowing for sure.
Keith,

regarding the design reconfiguration, by this, you are meaning that one must generate a new design without such prohibitions, is that correct? If yes, do we need to create a .cho file to estimate the utilities through CBC-HB Standalone?

Regards,
I was assuming that you were using our Lighthouse software.  In that case, you would need to recode the one attribute that had the never-seen level to have one level fewer.  Then you would import that design into a .ssi file that was expecting that level to appear with the correct one-level-fewer number of levels.  Then you could import the response data file and run the analysis.

If you didn't use Lighthouse and want to analyze this in CBC/HB, you could do that, too, but you would again want to recode the attribute with the missing level to have the right number of levels before you ran the analysis.  This will be a lot easier to do with a .csv formatted data file than with a .cho file.
Keith,

actually one has used Lighthouse Studio to run fielding and further estimations.
One thing that has been done is to save a new .ssi file, export the old design, exclude the level that was not shown in all concepts, re-import the design and run the analysis. Is that correct as well?
Mauricio, those are the steps, yes.
...