Born at 284 Marlborough Street in Boston's Back Bay, Isabel Weld Perkins was descended, on her mother Anna Weld Perkins' side, from a wealthy family of Boston merchants who traced their history back to Massachusetts Bay Colony. Isabel's father was Commodore George H. Perkins of Contoocook, New Hampshire, who was the commander of the during the American Civil War. The commodore's father, Judge Hamilton Eliot Perkins, was a prosperous businessman and attorney who built mills in Contoocook and for a short time ran a shipping firm in Boston that sailed clipper ships between the U.S. and West Africa.