Within five years of the house leaving the Taft family, the front veranda was removed, replaced by a one-story porch. Other modifications were the addition of a conservatory and the demolition of outbuildings, including a stable. Upon Thompson's death the house was sold by his widow to Colonel Ernest H. Ruffner in 1912. Upon Ruffner's death it was sold by his daughter. The William Howard Taft Memorial Association was formed on July 7, 1937, in hopes of buying the property, but went without support of the Taft family, as Robert Taft thought it would look too opportunistic to memorialize the house his father grew up in, and thus failed in acquiring the $12,000 to buy it. In the 1940s the building was used as apartments, with the new owner Elbert R. Bellinger once considering selling it to become a funeral parlor for local blacks. Taft family political fortunes faltered with Robert's death in 1953, and with Charles Phelps Taft II available to spearhead the movement, the William Howard Taft Memorial Association eventually acquired the house for $35,000 (the assessment price), instead of the $75,000 Bellinger was demanding for it.