Foster was born near Tiffin, Ohio, and was raised in Rome. This town would merge in 1854 with the town of Risdon to become Fostoria in honor of Foster's father. He was elected to the United States House of Representatives in 1870, serving from 1871 to 1879. In 1878, he failed getting re-elected, but was elected to the post of Governor of Ohio one year later. He served two two-year terms between 1880 and 1884. In 1890, Foster tried returning to the House, but was unsuccessful. In 1891, Benjamin Harrison made him Secretary of the Treasury due to the death of William Windom. Foster served till 1893 in that position.