Born on Christmas Day 1728 at Anjengo, Travancore State, India, he was the second son of Alexander Orme, a physician and surgeon in the service of the East India Company. His mother's maiden name was Hill. He was sent when about two years old to the house of his aunt, Mrs. Robert Adams, in Cavendish Square, London. From about 1734 to 1741 he was educated at Harrow School under Dr James Cox, and was then placed for a year in the office of the accountant-general of the Royal African Company.