Latrobe was born on May 1, 1764, at the Fulneck Moravian Settlement, near Pudsey in the city of Leeds, in the West Riding of Yorkshire, England. His parents were the Reverend Benjamin Latrobe, a leader of the Moravian Church who was of Huguenot (French Protestant) ancestry, and Anna Margaretta Antes whose father was German and whose maternal line was Dutch. Antes was born in the American colony of Pennsylvania, but was sent to England by her father, a wealthy landowner, to attend a Moravian school at Fulneck.