Born in Dundee, Wallace started his career with Celtic, signing for the club in May 2003 on a four-year deal. A pacy left winger, Wallace made only a handful of first-team appearances in his first year, before scoring his first goal for Celtic in a 5–0 win over Dunfermline on 8 November 2003. Whilst clearly a skillful player, Wallace made only a few more first team appearances that season, as Alan Thompson remained the first-choice wide-left player at Celtic. Wallace did however pick up his first silverware at the end of the season, coming on as a second-half substitute for Stephen Pearson in Celtic's 3–1 win over Dunfermline in the 2004 Scottish Cup Final.