Brown was born in Montrose on 21 December 1773, in a house that existed on the site where Montrose Library currently stands. He was the son of James Brown, a minister in the Scottish Episcopal Church with Jacobite convictions so strong that in 1788 he defied his church's decision to give allegiance to George III. His mother was Helen Brown "née" Taylor, the daughter of a Presbyterian minister. As a child Brown attended the local Grammar School (now called Montrose Academy), then Marischal College at Aberdeen, but withdrew in his fourth year when the family moved to Edinburgh in 1790. His father died late the following year.