Brown was educated at Aravon School – now closed but, at that time, a distinguished preparatory school in Bray, where he first studied Latin and French. Brown was educated at Shrewsbury School and New College, Oxford. At Shrewsbury, Brown initially expected to concentrate on sciences, but first studied Ancient Greek and turned in earnest to the study of history: "I myself had become a keen amateur astronomer, and had re-invented gunpowder to the detriment of my aunt's carpet. I fully intended to enter the Science stream of my new school. My housemaster summoned me to his study. In between puffs of his pipe, he announced with utter certainty: 'Brown, you did too well in Entrance [examination to do Science. You shall do ... Greek.' And Greek I did, if only for one year, before taking the Junior Certificate and then lapsing from the high calling of a classical scholar in the English Public School tradition into the study of mere History."