Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

Any statistical test for significance


After CBC/Latent Class estimation do I need to perform some reliability tests? And if so, any recommendations?

asked Nov 5, 2017 by Jenny

1 Answer

0 votes
If you are interested in whether individual respondents are reliable (consistent), then I recommend a combination of:

1.  Examine the responses to choice tasks for any evidence of straightlining (e.g., answering all 12 questions as the first concept in the questionnaire, when the information in the concepts is randomized such that no pattern would be expected).

2.  Examine the time to respond to the CBC questions.  If 12 questions are answered in 20 to 30 total seconds,  that doesn't seem to indicate much thought.

3.  Examine the fit statistic produced by individual-level CBC/HB estimation.  The software reports RLH for each respondent.  If respondents are answering randomly, their RLH should be no much greater than 1/k, where k is the number of concepts shown per task.  

4.  Responses to other non-CBC questions in the survey that might indicate a random-responder, such as straightlining or failure to complete reliability check questions correctly.

As for testing the reliability of a particular latent class solution, you should repeat the solution from multiple starting points.  If the same solution comes up nearly every time, no matter the random starting point, then this is some indication of stability.  But, a more formal test is to look at the BIC or CAIC (both measures of fit) for different solutions: such as 2-group, 3-group, 4-group, 5-group, etc.   You are looking for lower numbers for BIC and CAIC.  If the BIC or CAIC is minimized at 4 groups, then this is some statistical evidence about the justification of the 4-group solution relative to the other solutions you tested.
answered Nov 6, 2017 by Bryan Orme Platinum Sawtooth Software, Inc. (156,900 points)
Thanks a lot!

Do you think it's reasonable to limit the number of groups? I mean, I conducted the analysis with a 12-group solution, and 7-groups are the "best" with CAIC. Nevertheless, from a marketing point of view, 7-groups may be "too much" to interpret or let's say, a group with 4% ...how relevant may it be, if you're not into innovative products? How would you argue?
Often with strategic segmentation studies, usability for the client takes on just as much (or more) importance as minimizing CAIC.  If the 4-group solution is much more actionable and meaningful to the client, then it should be favored over a 7-group solution, even if the CAIC is a bit improved with the 7-group solution.