He began working backstage as a road agent and right-hand man to WWF promoter Vince McMahon, and is credited with inventing and booking the Royal Rumble match. In the late 1990s, he also worked in the talent-relations department. After his retirement, Patterson also worked as a WWF referee. He was selected as the in-ring referee for the main event at the first WrestleMania at Madison Square Garden on March 31, 1985, as well as the main event of WrestleMania XI. In 1992, Patterson was accused of sexual harassment by former announcer Murray Hodgson. He was briefly released from the company until the charges were dropped, when he was promptly rehired. This came after he had previously reportedly propositioned other wrestlers for sexual favors in exchange for a push in the WWF.