Initially playing for the reserves at Walsall, he went on to score on his first team debut on 31 January 2004 in a 3–2 defeat away at Stoke City. He went on to make a further ten first-team appearances for the club that season and was retained upon the season's conclusion, signing a one-year deal. Walsall having been relegated in 2003–04, Taylor could still not break into the team in League One. After three cup and three league appearances early in 2004–05 he was put on the transfer list by Paul Merson. Taylor went out on loan to Burton Albion in December, after he caught the eye of Nigel Clough during a Birmingham Senior Cup match with Burton. He impressed at Eton Park, making a total of six appearances and scoring four goals. He scored a memorable goal from 35 yards against Tamworth, and also scored against his future club Hereford in a 3–0 win. He scored four goals in six games for the Conference club, making a massive impact in his five weeks in Burton. On his return to the Bescot Stadium an offer from Carlisle United was rejected, and he signed a new two-year contract at the end of the season.