Evaluating imputation methods methodology

[amirdol]

Hi,
I've been reading "Flexible Imputation of Missing Data" and there is a section about evaluation imputation methods. One of the main points is that RMSE is not a good evaluation metric since it doesn't capture the uncertainty of the missing data.

I do understand the concept but I have some followup questions:

1. What can we say about two imputation methods that are "uncertain" in nature that are being compared based on this metric when one reaches a lower RMSE then the other?
2. In the data I'm working with - imputation based on the mean reaches a lower RMSE compared to imputation based on cart algorithm (when using mice). This is consistent even when checking many different levels of missing rates. To me this is a bit peculiar and unexpected, and be happy to hear your thoughts

[stefvanbuuren]

The RSME tends to favour methods that ignore the inherent uncertainty of the prediction, i.e., methods that impute the most likely value. Using those methods leads to overly optimistic inferences. See https://stefvanbuuren.name/fimd/sec-simplesolutions.html#sec:regimp for more detail.

[amirdol]

@stefvanbuuren yes I read that chapter and I definitely understand that, but still - do you think it makes sense that mean imputation will yield lower RMSE compared to cart based imputation? When I saw similar comparisons in the literature that wasn't the case, so I started thinking maybe there is something wrong in my code or data..

[stefvanbuuren]

If A has lower RMSE than B, that may be because A is a superior method. But it may also be that A simply does not include enough variability. In order to decide between these two explanations, you'll need to look at the coverage. If A has adequate coverage, then we might say that it the better method of the two (at least in the case studied).