Andrew Ellicott was born in Buckingham Township, Bucks County, Pennsylvania as the first of nine children of Joseph Ellicott (1732–1780) and his wife Judith (née Blaker or Bleaker, 1729–1809). The Quaker family lived in modest conditions; his father was a miller and clockmaker. Young Andrew was educated at the local Quaker school, where Robert Patterson, who later became a professor and vice provost at the University of Pennsylvania in Philadelphia, was his teacher for some time. Andrew was a talented mechanic like many of the family and showed some mathematical talent, too.