Anderson increasingly found himself at odds with conservatives in his home district and other members of the House. He was not always a faithful supporter of the Republican agenda, despite his high rank in the Republican caucus. He was very critical of the Vietnam War, and was a very controversial critic of Richard Nixon during Watergate. In 1974, despite his criticism of Nixon, he was nearly swept out by the strong anti-Republican tide in that year's election. He was re-elected with 55 percent of the vote, what would be the lowest percentage of his career. His spot as the chairman of the House Republican Committee was challenged three times after his election and, when Gerald Ford was defeated in the 1976 Presidential campaign, Anderson lost a key ally in Washington.