Zimmer began his career in 1949 with the Cambridge Dodgers of the Class-D Eastern Shore League. He then played with the Hornell Dodgers of the Class-D PONY League in 1950, the Elmira Pioneers of the Single-A Eastern League in 1951, the Mobile Bears of the Double-A Southern League in 1952, and the St. Paul Saints of the Triple-A American Association in 1953 and 1954. He made his major league debut with the Brooklyn Dodgers in 1954. Zimmer's big league career lasted 12 seasons, almost exclusively as a utility infielder. Notably, he played for the 1955 World Series champion Brooklyn Dodgers, and with the 1962 New York Mets, who lost a record 120 games.