Scott was born to William Scott and Anna Mason in Dinwiddie County, Virginia, near Petersburg, Virginia, on June 13, 1786. He briefly attended College of William and Mary, studied law in the office of a private attorney, and served as a Virginia militia cavalry corporal near Petersburg in 1807.