Carswell is the son of two physicians. He lived in Uganda until his late teens. His father, Wilson Carswell, a respected Scottish doctor and Fellow of the Royal College of Surgeons, diagnosed the first confirmed Ugandan cases of HIV/AIDS in the early 1980s, and was one of a number of people engaged in drawing the world's attention to the unfolding pandemic. His father's experiences in Uganda were the inspiration for the character Dr Nicholas Garrigan in Giles Foden's novel "The Last King of Scotland". Carswell later attributed his libertarianism to his experiences of the "arbitrary rule" of Idi Amin.