By the early 1970s Butler's physical and mental powers were in unmistakable decline; he was, in the description of Charles Moore, then a student at Trinity, well into his "anecdotage". He scaled back his public appearances after an incident at the Booker Prize awards in London in December 1973, at which he told ill-judged anti-semitic jokes, causing grave offence to the publisher George Weidenfeld. As early as 1938, Chips Channon had called his clothes "truly tragic" and as he grew older, Butler acquired an ever more dishevelled appearance. He ate and drank copiously as Master of Trinity, causing him to put on weight and begin to suffer from heart problems. On a visit to Cambridge in 1975, the first time the two men had met in a decade, Macmillan commented on how fat Butler had become. Butler also suffered from a skin complaint from the 1950s, which grew progressively worse, to the point towards the end of his life that he would sometimes appear unshaven in public.