Simon was born 26 May 1729, the son of Alexander Fraser of Balnain, son of Hugh Fraser of Balnain by his wife Katherine Chisholm, daughter of Alexander Chisholm 19th of Chisholm by his wife, eldest daughter of Roderick Mackenzie I of Applecross. Simon Fraser's mother was his father's second wife, Jean Mackintosh, daughter Angus Mackintosh 10th of Kyllachy by his wife Lucy, daughter of Sir Alexander Mackenzie, 2nd baronet of Coul. Simon's brother Thomas Fraser (born 8 June 1726), M.D. of Antigua's daughter Jean married Charles Grant, parents of Charles Grant, 1st Baron Glenelg