Originally named "Vasseur", the family was descended from French Huguenots who arrived in England in the mid-eighteenth century. Thomas's family were Baptists, and he and his younger brother James (father of noted brewer and philanthropist Matthew Vassar) emigrated to the United States in 1796.
John Ellison Vassar was born January 13, 1813, and named for his maternal uncle. He was the son of Thomas Vassar of Norfolk, England.