He underwent an operation to remove cartilage in his knee in November 1975, and his recovery was not entirely successful. He later admitted that after returning to match fitness from the injury he was only ever able to play at "half pace". His long-term replacement was young defender Alan Dodd. The club also faced a major decline following a gale which severely damaged the Victoria Ground's Butler Street stand, and a lack of adequate insurance cover left a significant repair bill meaning the club had to sell Alan Hudson, Jimmy Greenhoff, Mike Pejic, Sean Haslegrave and Ian Moores to fund the repair; Eric Skeels and John Ritchie also retired. Waddington filled in the gaps left by these players with youngsters, and it also meant that Smith could continue to be a regular for the club despite his reduced mobility. Waddington was dismissed in March 1976, and new manager George Eastham failed to prevent Stoke from being relegated at the end of 1976–77. Following relegation Smith formed another good partnership, this time with Mike Doyle, which helped Stoke gain promotion in 1978–79 under new manager Alan Durban. Smith claimed his two seasons in the Second Division were "boring", as even with his reduced mobility few attackers in the division posed too much of a threat for him.