Born in Cramlington, Northumberland, Brown joined Hartlepool's youth set-up from Cramlington Juniors football Club. He had joined too late to go through the Youth Scholarship system and was subsequently offered a temporary deal. He was part of the successful Hartlepool under 19s side that won the Dallas Cup XXV in 2004. James Brown was included on the substitutes bench for Hartlepool's match against Bournemouth on the last day of the 2004–05 season. He was expected to play a part in the Hartlepool's 2005–06 campaign and participated regularly during pre-season.