Burt was the fifth of eight children and of Scottish and English ancestry. His European progenitor, Richard Burt, immigrated to America from England in 1638 or 1639 and settled in Taunton, Massachusetts. Burt's parents were Alvin Burt and Wealthy (Austin) Burt of Worcester, Massachusetts. He was born June 13, 1792, at his parents farm located in nearby Petersham, Massachusetts. When Burt was six years old the family moved to Montgomery County, New York. Burt's parents sold their family farm in 1802 because of poor economic times and moved to Freehold, New York. In 1803 the family moved to the town of Broadalbin, New York.