Alberta's Minister of Justice complained to the Canadian Judicial Council after Justice Robin Camp made "victim-blaming" statements during a 2014 rape trial.
Mr Camp acquitted the accused, but his verdict was overturned and a new trial ordered.
The judge will be able to make written submissions in his defence.
The council, which is in charge of investigating judicial misconduct, called an inquiry into Mr Camp's statements in 2015.
Mr Camp was presiding over a sexual assault trial when he asked the 19-year-old complainant "why couldn't you just keep your knees together?"
He also repeatedly referred to her, the complainant, as the accused, and told her that "pain and sex sometimes go together".
His comments drew sharp criticism from sexual assault victims and their advocates.
In its report, submitted Wednesday, the committee that led the inquiry found the judge's conduct "was so manifestly and profoundly destructive" he should no longer serve.
They found the judge "relied on discredited myths and stereotypes about women and victim-blaming" during the trial.
During the judicial inquiry, Mr Camp expressed remorse for his comments and said that he has been educating himself about sexual assault laws and stereotypes.
"I was not the good judge I thought I was. I struck the wrong tone in counsel submissions. I was rude and facetious," he said during the hearing.
"I didn't realize the implication came with those words."
During the inquiry, the complainant said his comments made her contemplate suicide.
Although the committee acknowledged his "significant efforts" to correct the situation, they found that no amount of training could repair the public confidence lost during the trial.
Mr Camp will have the opportunity to file written submissions on his behalf before the council makes its final recommendations to Alberta's Minister of Justice.
A verdict is expected in the new rape trial on 31 January.
A judicial committee is calling on the removal of a judge who told a woman she should "keep (her) knees together".