Sherman and his coaching staff coached three NFL Pro Bowl games, from 1961 to 1963. In 1965 and 1966, with the support of owner Wellington Mara, Sherman added two retired Giants to his staff, Emlen Tunnell and Rosey Brown. In addition to being future Hall of Famers, both were African American, the first black assistant coaches in the NFL. With much racial strife in the country at the time, this caused controversy in both the press and parts of the league, particularly in the still-segregated southern areas. In 1968 Sherman and his coaching staff were invited to coach the Senior Bowl's North Team, played in Mobile, Alabama. During the practice week, the Bowl organization held a big banquet at a local country club. Shortly beforehand, Sherman was told that Tunnell and Brown were not invited because of the club's segregation policy. Sherman quickly informed the Bowl committee that no Giants personnel would attend the banquet unless everyone was invited. When it was clear nothing could change his mind, the club relented. That was the first evening in the club's history that two African-Americans were seated and served in the dining room. Sherman remained close friends with both until their deaths.