When faced with the obvious contradictions in the nativity narratives of Matthew and that of Luke, it helps to know that Scripture Scholars tell us that Matthew, especially, was using a 'Midrash' literary form common at the time.

In a Midrash writing, the author takes prophetic passages, and then tells his story in order to fulfill it. The point is to say that 'Jesus is the promised Messiah"!

So Matthew tells his story  from Joseph's perspective. Actually he does not really quote Isaiah well.
 Isaiah 7: 14 - 16 says: "The Lord will give you a sign in any case: It is this: the young woman is with child and will give birth to a son whom she will call Immanuel. 15. On curds and honey will he feed until he knows how to refuse the bad and choose the good. 16. Before the child knows how to refuse the bad and choose the good, the lands whose two kings are frightening you will be deserted."
    So the "sign" is that in a few years Ahaz will be safe, not the birth itself.