In 1864 a fire burned everything except the stone walls and the Savoy Chapel, and the property sat empty until Richard D'Oyly Carte bought it in 1880 to build the Savoy Theatre specifically for the production of the Gilbert and Sullivan operas that he was producing. The new theatre was built speedily, and accounts noted that it "was situated on a site which, though rich in historical associations, was also rich in the olfactory sense, Mr Rimmel's scent factory being close by as was Burgess's Noted Fish-Sauce Shop."