You're thinking of "sex", the physical equipment that's between your legs (and, notably, probably the least useful identifying feature around, since it's not something people regularly check). 

Gender is the individual's interpretation the culturally ingrained roles, and one needs to be mature enough to participate in culture in some capacity to exhibit it. An infant is oblivious to pronouns, let alone the implications of them.   Gender is a social construct which you signal to others by clothing, behaviour, and secondary traits, though none of those are immutable.  

 Yes, some people have a stronger sense of gender than others - I suspect apathy, those with no sense of gender, is pretty widespread, and people tend to subscribe to their sex simply for social convenience. That's certainly how I feel, anyway..