Martin was born in Kent in 1731, and was the grandson of Thomas Fairfax, 5th Lord Fairfax of Cameron (1657–1710) through his mother, Frances Fairfax Martin. Raised in humble surroundings in England, Martin relocated to Virginia in 1751 to assist his uncle, Thomas Fairfax, 6th Lord Fairfax of Cameron (known as Lord Fairfax), in administering the Northern Neck Proprietary, which encompassed up to . Martin resided with his uncle on their frontier Greenway Court estate in present-day Clarke County, Virginia. He earned the affections of Lord Fairfax on account of his energetic nature and loyalty, and through Martin's growing influence Lord Fairfax relocated the proprietary's base of operations to Greenway Court in 1762 and made Martin steward and land agent of the proprietary.