Sarria was born in San Francisco to Julio Sarria and Maria Dolores Maldonado on December 22, 1922. His parents did not marry. His father showed no interest in raising him. José's mother tried to raise him on her own. This became too difficult. She placed him with another couple. Both they and his mother indulged his early interest in wearing girls' clothing. Sarria had a talent for languages. This led to his first serious romance with another man. Sarria tutored Paul Kolish. Kolish was an Austrian baron who had fled from the Nazis. Sarria and Kolish fell in love. Their relationship lasted until Kolish and his son were killed in a car accident in 1947.