Born in London, the son of the judge Edward Fry, he grew up in a wealthy Quaker family in Highgate. His siblings include Joan Mary Fry and Margery Fry, who became principal of Somerville College, Oxford. Fry was educated at Clifton College and King's College, Cambridge, where he was a member of the Conversazione Society, alongside freethinking men who would shape the foundation of his interest in the arts, including John McTaggart and Goldsworthy Lowes Dickinson. After taking a first in the Natural Science tripos, he went to Paris and then Italy to study art. Eventually he specialised in landscape painting.