Elizabeth Ann Bayley was born on August 28, 1774, the second child of a socially prominent couple, a surgeon, Dr. Richard Bayley and Catherine Charlton of New York City. The Bayley and Charlton families were among the earliest European settlers in the New York area. Her father's parents were of French Huguenot and English descent and lived in New Rochelle, New York. As Chief Health Officer for the Port of New York, Dr. Bayley attended to immigrants disembarking from ships onto Staten Island and cared for New Yorkers when yellow fever swept through the city (for example, killing 700 in four months). Dr. Bayley later served as the first professor of anatomy at Columbia College. Elizabeth's mother was the daughter of a Church of England priest who was rector of St. Andrew's Church on Staten Island for 30 years. Elizabeth was raised in what would eventually become (in the years after the American Revolution) the Episcopal Church.