 Mary was the second daughter in a family of twelve —including Eleanor Conway Baldwin, Isaac Hite Baldwin, Ann Maury Baldwin, James Madison Baldwin, and Robert Stuart Baldwin.The family home was at Cedar Grove, which was near Belle Grove.All of the children in the family received their education from private tutors.