Foster began his career as a trainee with the Sheffield Wednesday youth system. He joined Conference National club Scarborough on work experience in September 2004. Foster made his debut for Scarborough in a 1–0 defeat to Hereford United, coming on as a 76th minute substitute. He started the following match in the club's 2–1 loss at Carlisle United, and went on to make 10 appearances during his loan spell. Foster returned to his parent club, but failed to make any first-team appearances, and was subsequently loaned out to Conference North outfit Alfreton Town in February 2005 until the end of the season. He made his Alfreton debut in the club's 3–0 victory against Woking in the FA Trophy. He scored his first goal for the club in a 3–1 loss against Southport, and scored again a week later in Alfreton's 2–1 defeat against Kettering Town. His third and final goal for Alfreton came in the following game, scoring the first goal in the club's 3–0 win at Barrow. He played his last game for the club in the final game of the 2004–05 season, in a 3–1 loss at Stafford Rangers. Foster played in 19 games for the club, scoring three goals.