Gandon was born on 20 February 1742 in New Bond Street, London, at the house of his grandfather Peter Gandon, a French Huguenot refugee. He was the only son of Peter Gandon (b. 1713), a gunmaker, and Jane Burchall (possibly née Wynne). From 1749 he was educated at Shipley's Drawing Academy where he studied the classics, mathematics, arts and architecture. On leaving the drawing academy he was articled to study architecture in the office of Sir William Chambers. Chambers was an advocate of the neoclassical evolution of Palladian architecture, although he later made designs in the Gothic Revival style. However, it was Chambers's palladian and neoclassical concepts which most influenced the young Gandon.