His baseball skills drew the attention of the Red Sox, who signed him as an amateur free agent with a $12,000 signing bonus on December 1, 1961. He spent the next five years working his way through the Red Sox farm system, the first four as a shortstop. He excelled at the plate in his first three seasons, batting .299 with the Class D Olean (NY) Red Sox of the New York–Penn League in , .298 combined between the Waterloo Hawks and Winston-Salem Red Sox in and .295 with the Reading Red Sox in . However, he struggled with his defense, committing 74, 36 and 42 errors in those respective campaigns. He spent his last two years in the minors with the Toronto Maple Leafs, back-to-back Governors' Cup Champions managed by Dick Williams. After his batting average fell to .246 in , Andrews was shifted to second base and responded by hitting .267, with 14 homers and an International League-leading 97 runs scored in .