When did they get "married?"  And who married them?  Joseph may have taken Mary as a life partner, but no one in any church performed a ceremony, and the government never issued a license.  Marriage was instituted as a sacrament by the Roman-Catholic Church at the Council of Nicea.  Before that time, people just took life partners.  And they didn't have to get a divorce, either.