Kavanagh began his career at Home Farm, before joining Middlesbrough in 1991 but struggled to ever hold down a first team place and was sold to Stoke City in 1996, after a short loan spell at the club, for £250,000. In 1996–97 he made 41 appearances scoring four goals including the final goal at the Victoria Ground against West Bromwich Albion. Kavanagh also had the honour of scoring Stoke's first goal at the Britannia Stadium in a League Cup match against Rochdale. He scored 10 goals in 1997–98 as Stoke suffered relegation to the third tier. In 1998–99 he top-scored with 13 goals as failed mount a sustained promotion challenge under Brian Little in 1998–99. In 1999–2000 Kavanagh scored 10 goals as Stoke reached the play-offs where they lost to Gillingham. He also played and scored in the 2000 Football League Trophy final as Stoke beat Bristol City 2–1. He was again a regular in 2000–01 as Stoke again failed in the play-offs this time losing to Walsall. He left Stoke in the summer of 2001 after making 244 appearances scoring 45 goals.