Francis Lee was born on October 14, 1734, at a Lee family home at Machadoc, later known as Burnt House Field, in Hague, Westmoreland County, Virginia. He grew up at Stratford Hall, which his father completed in 1738. He was educated at home, where Lee pursued classical studies under Dr. Craig. He was of English descent, and was born into one of the First Families of Virginia. Lee was the fourth son of Thomas Lee and Hannah Harrison Ludwell (of the nearby Greenspring Plantation).