Pearson only batted .214 in the 1964 season's first couple of months, and in June, he lost the starting centerfield job to Bob Perry. For the rest of the season, he was used as a pinch-hitter and pinch-runner, making occasional starts in left field and centerfield. On May 11, he had three hits, including a two-run home run against Moe Drabowsky in a 6–5 win over the Athletics. He also had three hits on May 22 in a 4–3 loss to the Yankees. In 107 games, Pearson had only 265 at bats, batting .223 with 59 hits and two home runs. His runs scored total dropped from 92 the previous season to 34.