[7], it is shown that the comparison of performance between finite-and infinite-width networks is highly architecture-dependent.
it was found that infinite-width networks perform as well as or better than their finite-width counterparts for many fully-connected or locally-connected architectures.