Haller continued to platoon with Bailey through the 1963 season, finishing the year second to Johnny Edwards among National League catchers in fielding percentage. In December 1963, the Giants traded Bailey to the Milwaukee Braves for veteran catcher Del Crandall, and Haller became their undisputed starting catcher. He was a solid defensive catcher for the Giants from 1964 to 1967. In his book, "The Bill James Historical Baseball Abstract", baseball historian Bill James said the decision to give Joe Torre a National League Gold Glove Award was absurd, stating that Torre was given the award because of his offensive statistics and that either Haller or John Roseboro was more deserving of the award. Haller also helped offensively in 1965, hitting two home runs and driving in five runs during a game on September 27 to put the Giants in first place with one week left in the season. However, the Giants faltered and ended the season two games behind the Los Angeles Dodgers.