Isabella was born at 24 Milk Street, Cheapside, London, England. Her father, Benjamin Mayson, died when she was young and her mother, Elizabeth Jerram later married Henry Dorling, who was a widower and he had four children of his own. They lived in Epsom, Surrey where Henry was Clerk of Epsom Racecourse. Isabella was sent to school in Heidelberg, Germany, where she became a very good pianist. When she finished school, she went back to Epsom.