He was the son of Homer Virgil Milton (1781–1822), an officer who fought in the War of 1812, and the grandson of Revolutionary War hero and former Georgia Secretary of State, John Milton (1756–1804). Born near Louisville, Georgia, he married Susan Cobb († 1842) in about 1830, and they had three children. John and Susan Milton lived in Georgia and later in Alabama. John got remarried to a Caroline Howze (1826-1901) from Alabama in 1844; they had two sons and seven daughters.