Have an idea?

Visit Sawtooth Software Feedback to share your ideas on how we can improve our products.

How come the min. recommended # of tasks in CVA is the same whether we use pairwise or single concept?

The literature indicates that we would like to present each level within an attribute a minimum of 3 times across a CVA survey. Yet, presumably we would achieve this twice as fast when using a pairwise comparison than when using a single concept evaluation, so why does SSI not change its recommended minimum # of tasks when we select pairwise or single concept?

I also have another related question regarding CVA: The CVA paper (https://www.sawtoothsoftware.com/download/techpap/cva3tech.pdf ) indicates on page 4 that pairwise comparison should be preferred over single concept evaluation, but that there is no difference in the estimated part worths between the 2 methods. Assuming that the min. # of concepts does not differ between the 2 methods, why should the pairwise comparison be the more preferred method, when it takes twice as long for the respondent to answer through (since they have to read through 2 concepts instead of 1)?

Thanks for the help!
asked Mar 28, 2014 by anonymous

1 Answer

0 votes
I don't know where you have obtained information that each level within each attribute should show a minimum of 3 times across a CVA survey.  Maybe you are mixing up a rule of thumb we have for MaxDiff with CVA?  Now, there is a rule of thumb (a "nice to have") in CVA that we should ask 3x as many questions as parameters to be estimated, but parameters to be estimated is not the same as levels per attribute.  Parameters to estimate is Total_Num_Levels - Total_Num_Attributes + 1.

The reason the minimum # of tasks remains the same under CVA whether using pairwise or single concept presentation is as follows: Under OLS (ordinary least squares regression), you need at least as many observations as parameters to estimate.  With pairwise coding, the number of parameters to estimate is still the same (compared to single concept presentation) and each CVA question still provides just 1 observation.  We code a pairwise task as still just one observation (not two), where levels shown on the "left" concept are coded in the design matrix as a -1 and levels shown on the "right" are coded as +1.  So, although there potentially is more information in the design matrix for each CVA question, there still needs to be at least as many observations as parameters to estimate for OLS to be able to compute.

Lastly, it doesn't take respondents twice as much time to answer a pairwise task as a single concept CVA question.  Respondents do not read all attribute levels on the screen.  As an illustration, sometimes, a level shown as part of a concept creates a "must avoid" concept, making it easy for the respondent to pick the other concept on the screen with an extreme rating.
answered Mar 31, 2014 by Bryan Orme Platinum Sawtooth Software, Inc. (169,815 points)
Thanks Bryan, sorry I meant # of parameters instead of # of attributes.... typed too fast.

What is the reasoning for preferring pairwise over single concept evaluation? Is it just a matter of forcing trade-offs?