That is true but you need to remember that Joseph didn't find out Mary was pregnant until after she returned from Elizabeth's house and by that time was likely showing and Joseph was likely to have had her stoned per the law and she knew this on her way back.  Joseph gets the dream from God after he has  found out his fiance' is already pregnant and not with his child.  This is why she was not stoned.  After Joseph's dream, he accepted her and so the villagers likely thought, as you said, that he had impregnated her so this was reasonably acceptable still so no one desired to have her killed.