The law, only fully repealed in West Germany in 1994, dates to 1871 but was rarely enforced until the Nazi era.
An estimated 5,000 surviving victims will receive â‚¬3,000 (Â£2,630; $3,350) in compensation along with â‚¬1,500 per year spent in jail.
Justice Minister Heiko Maas called the new law a "belated act of justice".
It "created unimaginable suffering, which led to self-denial, sham marriages, harassment and blackmail", he said.
The law, known as Paragraph 175, outlawed "sexual acts contrary to nature... be it between people of the male gender or between people and animals".
Sex between women was never criminalised.
Under the Nazis, the offence was punishable with 10 years of forced labour, with tens of thousands of men sent to prison or concentration camps, where many perished.
Between 1949 and 1969, when the law was relaxed, 50,000 men were prosecuted and there were a further 14,000 cases until 1994.
But the law also wrecked additional lives, say historians. Some men living in fear of being discovered or convicted committed suicide, they say. Others lost jobs or were forced into sham marriages.
The vote by the lower house of parliament, the Bundestag, will need to be approved by the Bundesrat upper house, though it is expected to be uncontroversial.
The law was welcomed by Germany's Lesbian and Gay Association (LSVD).
"After long years of ignorance the victims of homophobic persecution are regaining their dignity. This is a historic step," it told the BBC.
"But the law also has serious gaps. The designated compensation payments are too low."
It also criticised a change to the draft bill due to pressure from the conservative Christian Democratic Union party, which restricted eligibility for compensation to those who had sex with over-16s only (from over 14), saying it was discriminatory.
Germany's parliament has voted to quash the convictions of tens of thousands of gay men criminalised under notorious historical anti-homosexual laws.