Anderson was drafted by the Chicago White Sox in the first round (15th overall) of the 2003 Major League Baseball Draft. He made his major league debut on August 16, 2005, and appeared in 13 games for the White Sox. That offseason, Chicago traded Aaron Rowand to the Philadelphia Phillies in a deal that brought Jim Thome to Chicago, clearing the way for Anderson to become the starting center fielder. In 2006, Anderson showed himself to be a gifted defensive outfielder, though he struggled offensively, finishing with a .225 batting average despite signs of improvement in the second half of the season and some corrections in his swing mechanics. Anderson did not commit an error at the Major League level until August 18, 2006 against the Minnesota Twins, ending his streak of 110 errorless games. On May 20, 2006, Brian was ejected from an interleague game against the Chicago Cubs for fighting with Cubs first baseman John Mabry. Anderson was fined and served a five-game suspension.