In January 1954 Prior married Jane Primrose Gifford Lywood, daughter of Air Vice-Marshal Oswyn George William Gifford Lywood, CB, CBE, a developer of the Typex cypher machines, of a landed gentry family of Woodlands, near Sevenoaks, Kent. They had four children. Prior's eldest son David Prior held the seat of North Norfolk between 1997 and 2001, and was appointed Parliamentary-Under Secretary of State for NHS Productivity; he was later, in May 2015, elevated to the peerage in his own right as Baron Prior of Brampton.