From 1956 until their divorce in 1977, she was married to Sir Hugh Fraser (1918–1984), a descendant of Scottish aristocracy 14 years her senior and a Roman Catholic Conservative Unionist MP in the House of Commons (sitting for Stafford), who was a friend of the American Kennedy family. They had six children: three sons, Benjamin, Damian, and Orlando; and three daughters, Rebecca Fitzgerald, wife of barrister Edward Fitzgerald, QC, Flora Fraser and Natasha Fraser-Cavassoni. All three daughters are writers and biographers. Benjamin Fraser works for JPMorgan, Damian Fraser is the managing director of the investment banking firm UBS AG (formerly S.G. Warburg) in Mexico, and Orlando Fraser is a barrister specializing in commercial law (Wroe). Antonia Fraser has 18 grandchildren.