I would like to explore differences across two groups of approximately 400 respondents each (i.e. respondents who already bought a product and respondents who did not).
I first estimated my model in CBC/HB (with uninformative hyperpriors) and computed the Mean Absolute Error (MAE) using my two holdouts and the simulation results from the SMRT package. I obtained a MAE of 0.6375.
I then estimated the same model using "purchase" as covariate and obtained a MAE of 0.64.
Finally, I estimated the same model on a split sample, merged the .hbu files, imported the merged .hbu in the SMRT package and computed again the MAE, which this time was 0.63.
At this point:
1) Are these differences in MAE relevant for model comparison? If not, which model is the best and why?
2) If the pooled model performs slightly better in predicting the holdouts than the model with mixture distributions, how can the model computed on a split sample perform the best? Is there anything I am overlooking?
Thanks for any hint!