Named for star college football player Albie Booth, Pearson grew up desiring to play baseball. Though he initially attended Mt. San Antonio College, he dropped out to sign with the Boston Red Sox after a psychology professor suggested he was more interested in baseball than his lectures. Pearson spent five years playing Minor League Baseball in Boston's organization but had not yet reached the major leagues upon his trade to the Senators in 1958. Pearson made Washington's roster out of spring training that season and batted .275, winning the American League (AL) Rookie of the Year and the "Sporting News" Rookie of the Year Awards. He struggled to start off the 1959 season, though, and was traded to Baltimore during the year; Pearson would bounce back and forth between the minor leagues and the majors in 1959 and 1960. Hearing that his native California was about to be awarded the expansion Los Angeles Angels, Pearson wrote Fred Haney, the Angels' general manager, asking the Angels to pick him in the expansion draft. Haney did draft Pearson–as the 30th and last pick.