Why do deeper CNNs have better shift consistency?

[vadik6666]

With baseline CNNs with no anti-aliasing, we see better shift consistency if we increase the CNN's depth, e.g. VGG11 -> VGG19, Resnet18 -> Resnet152. Why is that so?

[richzhang]

Good question. Higher accuracy naturally lends itself to better shift consistency. A classifier that gets 100% accuracy will be consistent across shifts, even if has no shift-invariant inductive bias to begin with

[vadik6666]

Thank you @richzhang for a quick response!

1. Suppose we use an ensemble of very deep CNNs as a teacher and do knowledge distillation to a small student CNN e.g. MobileNetv2. All CNNs in here are not anti-aliased CNNs. Do you think such a student model will have shift consistency?
2. Same situation as in (1) but CNNs in the ensemble are anti-aliased versions, but student CNN isn't (it is again default MobileNetv2). What do you think about shift consistency of a student model?