In general, I agree. Your gender (or lack thereof, if you identify as agender) is none of my business. There are some practical examples where your physical identity matters, of course, For example, if you commit a crime, society may be interested in your DNA, but still doesn't need to know if the DNA is from a "man" or a "woman" or an "other" to decide guilt or innocence.

Of course, this also means all humans could be sent to the same prisons, or sent to die in the same wars. Equality is simple, but it isn't always easy.