The following season saw Rimmer lead a nomadic existence, as he moved on from Watford after only a couple more appearances for £200,000 to Notts County in Division Three. He was shortly off again, following manager John Barnwell to Walsall, who were playing in Division Two in another five–figure transfer. Rimmer made his debut in a 7–0 home thrashing by Chelsea but a week later scored a hat-trick in a shock 3–0 win at Sunderland. Relegation was already looming for the Saddlers, with a similar fate striking the next season despite Rimmer again finishing as the club's top scorer. He began the following season by scoring Walsall's first goal at their new Bescot Stadium.