He was one of three players released by Carlisle, after which he signed a two-year contract for Portadown in August 2004. Kelly won the 2004–05 Irish Cup with Portadown, and scored in the 5–1 win over Larne in the final. He became Portadown captain for the 2005–06 season and by December 2005 was being linked with a move away from the club, with his contract expiring in the summer. He won the Harp player of the month award for February 2006. He launched a stinging attack on the "prima donnas" at the club and felt some of his teammates lacked "ambition and determination".