Glass was born in Baltimore, Maryland, the son of Ida (née Gouline) and Benjamin Charles Glass. His family were Lithuanian-Jewish emigrants. His father owned a record store and his mother was a librarian. In his memoir, Glass recalls that at the end of World War II his mother aided Jewish Holocaust survivors, inviting recent arrivals to America to stay at their home until they could find a job and a place to live. She developed a plan to help them learn English and develop skills so they could find work. His sister, Sheppie, would later do similar work as an active member of the International Rescue Committee.