However, Baker's velocity, which had once been in the 90's was now only in the mid to high 80's. He was cut from the Braves roster and started the season with AAA Richmond.
On May 15, 2006, he was traded back to the team who originally drafted him, the Boston Red Sox, and was assigned to AAA Pawtucket to complete a trade for reliever Franklin Núñez. Baker finished the season going 1-0 with a 2.40 ERA with the Richmond Braves but was 0-0 with a 6.07 ERA with Pawtucket.