A descendant of the last Gaelic Lords of Éile in Ireland, Charles Carroll was born in Annapolis, Maryland of a distinguished Roman Catholic family and was a distant cousin of Charles Carroll of Carrollton, (1737–1832), and Daniel Carroll (the First and Second), (1730–1796). His father, also Charles Carroll, took him to Europe in 1733 for his education. Young Charles spent six years at the English House school in Lisbon, Portugal. He then went to England to complete his education at Eton and Cambridge. After graduating Cambridge in 1746, Charles returned to Annapolis. He took up residence there. He busied himself learning to manage the family's farm and mills at Carrollton.