There are multiple axes along which the current paper falls short of applying to realistic settings: 1) the assumption that one is given an oracle adversary, i.e. we have access to the worst-case perturbation (as opposed to a noisy gradient oracle, i.e. just doing PGD); 2) the results in section 4 apply only to shallow fully-connected ReLU networks; 3) the results hold only in a regime very close to initialization and it is assumed one has an early stopping criterion/oracle.
Weaknesses 2) and 3) are not unique to this work, and thus I heavily discount their severity when considering my overall recommendation.
2) the results in section 4 apply only to shallow fully-connected ReLU networks;