The unmarried men who push the idea that contraception is immoral are in a tradition that said "sex is nasty". Augustine wrote that intercourse between a husband and wife was sinful, unless it was expressly with the intent to procreate. To him, and to many of the Church fathers, and the celibate churchmen who followed him, sex had to be "justified". You can't just have it because it's fun or because it makes you feel closer to your spouse. 

BTW, that's the main argument for the perpetual virginity of Mary: Sex is sinful, so the sinless Mary didn't engage in it. 

So, they said that "natural law" says that the proper reason for sex is procreation, and sex must be "open" to it.  That is the background to Humanae Vitae, vastly simplified.