I didn't know that the popes had issued directives like that, although I knew some bishops did.  I can understand why some clergy would resist such a directive, for the reason you suggest and for other reasons, too.
If I remember correctly, the clerical suit as street garb for priests did not become common in the United States until the last quarter of the nineteenth century.  A priest friend of mine told me that in the seminary he attended, there was a corridor where portraits of former rectors and faculty members were hung.  He said that one did not have to look too closely to see that the clerical collars on some of the reverend gentlemen had been painted over the secular suits, the shirts and ties as they were worn in the mid-1800s.  The paintings were defiled by order of a rector, ca 1930, who was scandalized, or was afraid the seminarians would be scandalized, by the secular attire of his predecessors.