the other thing madelyn does n't know is all while sheila was growing up her mother worked for a white family who always called her 'little brown baby ' to her face and 'that nigger baby ' when they thought she could n't hear , because they did n't want to bother learning her name .