I plan to launch an ACBC experiment to uncover drivers of hiring decisions. Participants have to select their preferred job candidates for a fictional analyst position based on six skills attributes, e.g., analytical skill, industry knowledge, interpersonal skills. As my sample is highly selective which leads to a rather small sample size (I can only survey experienced managers with hiring expertise contained in my university’s database) I have identified ACBC as method of choice.
However, I have two questions regarding the BYO task and selection of appropriate levels:
1) As higher skills are obviously better than lower skills, I fear that all respondents will select the highest level for all attributes in the BYO task. All attribute levels are ordinally-scaled from best to worst (very good, good, reasonable). Would that be problematic for the screening and choice tournament? I assume that the lower levels for any attribute won’t be used much across all participants and experiment phases. Is there some sort of loss in information richness in the data if that happens?
2) Right now I plan to use the three levels “very good”, “good” and "reasonable” for 4 of 6 attributes. I fear that any level lower than “reasonable” is unacceptable to any respondent (who wants to recruit an analyst with “poor” analytical skills?). However, I am also concerned that respondents will be biased to select “reasonable”, because it is the lowest level. Does anybody have experience with selecting appropriate levels for a comparable setting? I also considered using the levels “above average”, “average”, or “below average” (+ maybe exceptional). It would be very helpful to gather your opinion on that in case somebody has experience or a good feeling for my research setting.
As I understand that (A)CBC analysis was developed to study consumer preferences for products and services where most of the time attributes are nominal-scaled (Favorite restaurant: Italian, Chinese or Mexican?) I am just insecure about the specifics of my study setting and would warmly welcome any advices in that regard.
Thank you very much in advance and kind regards,