Cavendish was born at Asthall Manor in Asthall, Oxfordshire. She had five sisters and a brother. She was married to Andrew Cavendish from 1941 until his death in 2004. They had seven children, three of which died at a very young age.