Through his eldest son Simon, he was a grandfather of six, including Simon Fraser, Master of Lovat (1939–1994), Hon. Fiona Mary Fraser (b. 1941) (wife of Robin Richard Allen), Hon. Annabel Thérèse "Tessa" Fraser (b. 1942) (wife of Hugh Mackay, 14th Lord Reay and Sir Henry Keswick), Hon. Kim Ian Maurice Fraser (1946–2020), Hon. Hugh Alastair Joseph Fraser (1947–2011) (husband of Drusilla Jane Montgomerie), Hon. Andrew Roy Matthew Fraser (1952–1994) (husband of Lady Charlotte Anne Greville, a daughter of David Greville, 8th Earl of Warwick).