I like your phrase 'social contract'.  This issue is larger than 'what women choose to wear' or 'religious freedom'.  Seeing your fellow citizen's face in public seems like a basic underpinning of society.
Why change it for a small group who want to act differently?