In 1920 Witley Court was sold by the 2nd Earl to Sir Herbert Smith, a Kidderminster carpet manufacturer. Sir Herbert maintained only a skeleton staff to manage the house whilst he and his family were away, and many areas were left unused. A major accidental fire broke out in September 1937, whilst Sir Herbert was at another of his houses. It started in the bakery situated in the basement room of the now least preserved tower. The staff tried to put the fire out with the ancient fire pump, which was connected to the fountain, but it failed to work as it had not been maintained for many years. Although only one wing of the house was gutted by the fire and the rest of it was almost intact, the insurance company declined to cover the major damage, so Sir Herbert resolved to sell the property.