Before the 2004 season began, the Houston Astros signed Andy Pettitte to boost their starting rotation. As a result, Robertson found himself battling with Tim Redding, Jared Fernandez, Carlos Hernández and Brandon Duckworth for the final spot in the rotation. After Roger Clemens was signed as well, the Astros decided before spring training began that Robertson would pitch out of the bullpen for the season. By the end of spring training, Robertson failed to crack the bullpen, and was to be demoted to the Zephyrs. On March 31, just before the season started, the Astros traded Robertson to the Cleveland Indians for Luke Scott and Willy Taveras.