Exclusions, Validity and Realiability

Question

I want to conduct a CBC with 5 attributes with 2 levels each, so i have 32 possible profiles. Since in my master thesis I use conjoint analysis on schedules and not on products, I have to exclude 2 profiles, because these combinations have absolutely no sense. If I use HB for evaluation, do I have to take something into account because of the exclusions?

Then I have a few more questions about the holdout questions. I would like to add 2 holdout tasks per participant for validation (different profiles in both tasks but they are the same for all participants) and a third one which is identical to one of the two previous holdout tasks to check for reliability. Is this possible?

(1) Since my "normal" tasks all contain the None option, my holdout questions should probably do the same. How do I include the None option in the hit rate? If I calculate the individual utilities of a participant for both profiles of the holdout question and find out that the utility for concept A is better than for concept B and the respondent has chosen A (B), then the hit rate is 100% (0%) for the certain holdout task. But what if the respondent chose None, would the hit rate simply be 0%?

(2) I read in the forums that you propose a 50:30:20 preference for the design of holdout tasks with 3 concepts. I will do a pre-test with 10 subjects without Holdout tasks and therefore get an idea of how popular the individual profiles are. What do you suggest for the design of 2 concepts? A 30:70 distribution?

(3) Do I have to include unreliable participants (identified with the two identical holdout tasks) in the hit rate or do I have to exclude them before I calculate the hit rate?

(4) If I use 10 questions + 2 holdout tasks = 12 tasks, are positions 4 and 8 well suited for the holdout tasks?

Answer 1

Lots of good questions!  You had two questions before you started listing them so I will call them A and B in order.

A) You don't technically (shouldn't) need to take anything special into account with HB estimation if you exclude those profiles, but it kind of depends on how damaging it is to your design (depends which 2 you remove).  I recommend you run aggregate logit (MLE) as a gut check with your results, though (and tabulate your raw choices by attributes/levels too, i.e., "counts analysis").  The average utilities from HB and utilities from aggregate logit should be highly correlated (almost correlation of 1).  Inspect the utilities of the levels that were affected.  You may simply just need more iterations to achieve convergence.  If it's a masters thesis, run two sets of HB and compare the chains (the Gelman-Rubin diagnostic) too as an additional test to ensure convergence (overkill in the industry, but a good idea if you are deep diving into a single project as you are).  Last thing to say here: you can't simulate the combinations that you prohibited, of course.

B) Yes, that's fine and possible to include 3 holdouts with one duplicated

1) The "None" option gets treated as a third, fixed alternative.  You will have utility for "None" the same way that you would have utility for any fixed alternative (i.e., an alternative without any attributes and levels, just a single beta/ASC).  So in your hit rate calculation, you sum up the utility for concept A (first alternative) and concept B (second alternative) AND the None (third alternative).  Sometimes the None utility will be higher than the utility for concepts A and B so it is then predicted to be chosen and if respondent selected "None" in the holdout, then it is a hit.  Long story short, it is simply treated as a third alternative.

2) The 50:30:20 recommendation is so you don't have a dominated concept where it is an obvious choice (e.g., a very low priced alternative, and easy choice for many) and you don't have equally good options across the board where simple random data would yield a flat result too (for MAE/RMSE with an aggregate model, not as relevant for hit rates where you are calculating it at the individual level).  

3) I suppose this is your choice.  We often clean out "bad respondents" in our data (from speeding through the survey, or always picking the first option, etc), but this is for-profit industry, not academia and I'm not sure the academic approach here.  As far as I know, it's not a "rule of thumb" to simply remove them, although you do have a good argument to remove them.

4) Yes, it makes sense to space them out like that, especially if there is a repeat task.  Speaking of which, you seem to have 3 hold outs, so perhaps go with positions 4, 7, and 10.