The Fathers of the first three centuries did not have access to all four gospels, nor many of the letters of the NT. Matthew, by your logic, was closer to events than Iranaeus. He says they had marital relations. These Fathers have a lot to answer for. They oversaw a church which completely departed from Christ's teaching re war, Christians taking part in it, power, wealth, hierarchy etc.. 

It actually makes no difference to me whether Mary was a virgin or not. It has no bearing whatsoever on my faith. However, those who have insisted on it in history - based on zero evidence - have done a great disservice to ordinary Catholics, demoting sex and sexuality to a place lower than virginity/celibacy. It also reduces who Mary was. You won't find her name mentioned anywhere in the liturgy without the word 'virgin' lumped in front of it. It's almost obsessive. Why?

And people wonder why the clergy and hierarchy see themselves as 'higher' than mere mortals. There's one answer.