Ivey was born an only child to Boadman Nettles Ivey and Barbara Ivey. The family had been in Alabama for many generations, as her ancestor Ann Johnson Nettles Riley was one of several slave owning Nettles in the state. 