Lady Mary died from a carriage accident in Parramatta Park, outside Government House, in 1847. Within a year of her death, rumours were circulated about the colony of New South Wales about FitzRoy's 'womanising' ways. In 1850, FitzRoy made a visit to Berrima, to inspect the Fitzroy IronWorks. The Governor stayed at the Surveyor General's Inn, operated by former boxing champion Edward "Ned" Chalker (sometimes Charker). Ned's step-daughter, Mary Ann Chalker, who was 18 at the time, worked there. Nine months later, she gave birth to a son, named Charles Augustus FitzRoy, after his father, the Governor. This boy was later adopted by ex-convict John Fitzsimons and his family.