The club suffered their first relegation in 1961 as they finished 23rd in the Third Division, but didn't have to wait long until their first Football League promotion, spending just one season in the Fourth Division as they ended the season second to Millwall by just one point. This trend continued over the next two decades as they were relegated to the Fourth Division in 1965 and promoted to the Third Division in 1966, then relegated in 1968 and promoted in 1974, relegated in 1976 and promoted in 1977 before a final relegation to the Fourth Division in 1981.