Madison married Nelly Conway (Port Conway, Caroline County, Virginia, January 9, 1731 – Montpelier, Orange County, Virginia, February 11, 1829), also of the planter class (her birthplace was named after her family). They had twelve children: