2003–04 was a difficult season, with champions Barry's first league win not coming until February 2004 when they beat fellow strugglers Welshpool Town 5–4 with a 98th-minute winning penalty from youngster Luke Sherbon. Manager Colin Addison was brought in resuscitate the team's ailing fortunes, yet the Dragons still ended up bottom of the division, four points off safety, and were relegated to the Welsh League Division One. Controversially, Addison was dismissed by Lovering on the eve of the new campaign, with assistant David Hughes replacing him; only to leave himself months later on finding his budget slashed. In the meantime, an independent district valuer had determined that the club should pay £42,000 in rent and rates each season for the remainder of the lease. Judging the figure to be unfairly based on the club's relinquished professional status, Lovering refused to pay this amount and instead moved the senior side to the White Tips Stadium in Treforest from January 2005 to May 2006. During the absence, a number of staunch supporters formed breakaway club Barry FC; the culmination of a series of disputes with chairman Lovering, who had banned them from fundraising at club. With the Town relegated to their lowest-ever league status at the end of the 2005–06 season, the future appeared bleak for this fallen giant of Welsh football.