John was born as "John Wolrige-Gordon" in Ellon, Aberdeenshire, on 10 August 1935. He was the elder of the twin sons of Captain Robert Wolrige-Gordon, MC and his wife Joan Walter. His mother, Joan, was the daughter of Dame Flora MacLeod of MacLeod, the 28th Chief of Clan MacLeod. His younger twin brother, Patrick Wolrige-Gordon, would later become a Tory Member of Parliament. The twins had an older brother, Robert Wolrige-Gordon, who would later succeed their father as the 21st Laird of Hallhead, 10th Baron of Esslemont.