As a young man, he lived in Greenwich Village in the same apartment building as a young John Huston. The two men became good friends and remained so for life. Jaffe was later to star in two of Huston's films: "The Asphalt Jungle" and "The Barbarian and the Geisha". Jaffe's closest friends included Zero Mostel, Edward G. Robinson, Ray Bradbury, and Igor Stravinsky. In 1923 he appeared in the Broadway premiere of "God of Vengeance (Got fun Nekome)" by Sholem Asch, as Reb Ali. The production became notorious after the cast, producer, and theatre owner were indicted and found guilty on charges of indecency in May 1923. Jaffe began to work in film in 1934, rising to prominence with his very first role as the mad Tsar Peter III in "The Scarlet Empress". In 1938, Jaffe was forty-seven years old when he played the title role of water "boy" "Gunga Din".