Phillips was fired on New Year's Eve 1980 by Oilers owner Bud Adams. Phillips was fired because he failed to report a player's in season recreational drug use to Adams until after the season ended. His 59 wins would be the most in franchise history until Jeff Fisher passed him in 2001 (by then, the team had become the Tennessee Titans). Soon afterward, he was hired by New Orleans Saints owner John Mecom Jr. as head coach and general manager of the Saints, serving from 1981 through the first 12 games of the 1985 season. As in his coaching tenure with the Oilers, Phillips took off his trademark Stetson inside the Louisiana Superdome. In 1983, his Saints went into the final week needing one more win to secure the first winning season and playoff berth in franchise history. The Rams beat the Saints for the final playoff spot in week 16, 26–24 on Mike Lansford's 42-yard field goal with 00:02 to play.