McKim was born in Chester County, Pennsylvania. His parents were James Miller McKim, a Presbyterian minister, and Sarah Speakman McKim. They were active abolitionists and he was named after Charles Follen, another abolitionist and a Unitarian minister. After attending Harvard University, he studied architecture at the École des Beaux-Arts in Paris before joining the office of Henry Hobson Richardson in 1870. McKim formed his own firm in partnership with William Rutherford Mead, joined in 1877 by fellow Richardson protégé Stanford White.