Granger was bisexual, and had relationships with both women and men. Some of the people he was involved with were Shelley Winters, Ava Gardner, Leonard Bernstein, and screenwriter Arthur Laurents. His longtime romantic partner was television producer Robert Calhoun.