Frémont was born on January 21, 1813 in Savannah, Georgia. He studied at the College of Charleston. Frémont was married to Jessie Benton from 1841 until his death in 1890. They had no children. Frémont died on July 13, 1890 in his home in New York City from peritonitis, aged 77.