Thomas Graham was the third and only surviving son of Thomas Græme of Balgowan, in Perthshire and Lady Christian Hope, a daughter of the first Earl of Hopetoun. He was born in 1748, and was educated at home by the Reverend Fraser, minister of Moneydie, and afterwards by James Macpherson, the collector and translator of Ossian's poems. He went up to Christ Church, Oxford in 1766, and the following year he inherited the family estate following the death of his father.