The site was originally occupied by a small house named 'Gloucester House' (after Prince William Henry, Duke of Gloucester and Edinburgh, who owned it), with the front entrance on Upper Grosvenor Street. This house was purchased by Robert Grosvenor, 1st Marquess of Westminster, in 1805 for £20,000. He spent £17,000 on extending the house to make it more fashionable. In 1821, a large picture gallery long was added to the west of the house. It was here that many of the Grosvenor family's treasures were held.