One could equally well argue that mathematics arises from counting and measuring and so can't exist until after there is a universe of things to count and measure.
Mathematics might only exist in the context of a physical universe.
It cannot be argued that mathematics is an entirely abstract field.