That's a good question, but an answer might be that what you are labelled at birth is not always what you are.  So when you say someone was "born male" it doesn't mean that person is male, even though they may share some physical characteristics of a male.