As Fairlie became better known, his personal life grew chaotic. He drank heavily and conducted a series of extramarital affairs, including one with the wife of his friend Kingsley Amis that nearly ended their marriage. Never responsible with money, he amassed thousands of pounds of debts. And in 1965, he insulted Lady Antonia Fraser on television, leading to a libel suit against him and the ITA. That year, he visited America for the first time, and fell immediately in love with the country. A few months later, he moved there for good.