It's not heartless, it's treating everyone the same.  Should the rest of us, male and female, be allowed to wear masks when we access government services?  That's the only issue here - should we be allowed to disguise our identity, to hide our faces, when we interact with public officials?  And, for that matter, should the officials themselves be allowed to wear masks or otherwise cover their faces when they serve us?