Beginning in the mid-seventies, Schwarz emerged as one of the South African Jewish community's foremost leaders. Schwarz often led the opposition to anti-semitic comments and movements by prominent public figures, and was often involved in heated clashes in parliament. In April 1982, Schwarz was ordered out of the parliamentary chamber after he described Nationalist MP Andre Fourie as "vuilgoed' (rubbish), in response to Fourie saying that he was "enough to make a Jew anti-Semitic". In August 1986, he took up the issue of anti-semitism within the far-right Conservative Party after its leader Clive Derby-Lewis responded to a question of whether Jews were "money and power grabbers" by asking "Is that anti-semitic or fact?". He also slammed the CP's supporters use of Nazi symbols and banners, and for burning the Israeli flags. While affirming that "the days of the Jews walking into the gas chambers are over", Schwarz also warned that the Jewish community should "not paint everybody with a broad brush", and that impulsive reactions should be avoided before "waging war". Furthermore, Schwarz would often use his access and relations with government ministers, police and the military establishment to investigate incidents of anti-semitism.