Ramírez spent 2003 with the Class A Augusta GreenJackets, batting .275 in 111 games. In 2004, he split time across the GCL Red Sox (batting .400 in 6 games), the Class A-Advanced Sarasota Red Sox (.310 in 62 games), and the Double-A Portland Sea Dogs (.310 in 32 games). Ramírez was rated by "Baseball America" as the number ten prospect in baseball prior to the 2005 season, and with the Sea Dogs that year he was selected as an Eastern League All-Star while hitting .271 in 122 games.