Brady grew slowly from the 1920s through the 1950s, with population estimates reaching a peak of 6,800 in 1958. In 1959, the Gulf, Colorado and Santa Fe Railway abandoned the section of track between Brownwood and Brady, thereby reducing Brady's access to outside markets. The population fell to 5,338 by 1961, and subsequently stabilized. Brady Reservoir was completed in 1963 for flood control, municipal and industrial water needs, and recreation. The Atchison, Topeka and Santa Fe abandoned the track between Brady and Eden in 1972, leaving the town with only a branch track to connect it with the main line at Lometa, in Lampasas County.