Lord Garvagh moved to Keswick at the beginning of the Second World War, and by 1942 various fighter squadrons at RAF Digby had taken over Ashby Hall as their officer's mess. One of the most notable was 609 Squadron (White Rose). By the end of the war the house became derelict and stood empty until the estate were broken up in the late 1950s. The park was cleared of its oak trees, the walled garden and surrounding farms sold separately, leaving the Hall itself to become the Lake Rendezvous Club, featuring acts including Diana Dors, Bob Monkhouse and Ken Dodd.