By 1710 the castle was in such a bad state of repair that Richard, the 5th Viscount Fitzwilliam, selected 100 acres (0.4 km²) on which he built Mount Merrion House, surrounding the house by an granite wall. The house was completed in 1711 and served as a new seat by the 5th Viscount Fitzwilliam on the hill at Mount Merrion. The Fitzwilliam family left for England around 1726. Although the family no longer lived in Mount Merrion House, they retained possession of it, and rented the house out.