However, during pre-season, Brown ruptured his cruciate ligament against Hibernian and missed the majority of the subsequent season. On 6 May 2006 Brown made his comeback and debut for the club, coming off the bench to score in the 1–1 draw with Port Vale at Victoria Park. The game was bittersweet for Brown as Hartlepool's failure to win saw them relegated to League 2. James's finishing earned him his first professional contract with the club in 2006. During the 2006–07 campaign, he played nearly all of Hartlepool's games on the right wing.