In 1857, the Duke of Sutherland, who had owned Cliveden for eight years, commissioned George Devey to enlarge the existing building into a two-storey cottage. The subsequent alterations were in the vernacular style with brick and stucco walls, fish-scale pattern roof slates, a Gothick-style loggia and a turret above an exterior staircase leading to a balcony. Throughout the remainder of the 19th century the main purpose of the cottage was as a place of leisure, and it was frequently used by the Duke's wife Harriet to entertain guests, most notably her friend Queen Victoria.