Feudal barony of Dunster Ryce, by whom he had no sons, only three daughters, Catherine, Dorothy and Mary, co-heiresses to 1/3 in total of his estate, the remaining 2/3rds going by entail to his younger brother Thomas Luttrell (died 1571). Mary survived him and remarried to James Godolphin of Cornwall. Thomas Luttrell (died 1571) of Dunster Castle (younger brother). In 1563 he was MP for the newly created Luttrell Pocket Borough of Minehead, two miles north-east of Dunster Castle. He sold the Devon and Somerset estates, excepting Dunster Castle, apparently to meet debts. These were however "amply replaced" by the large estate inherited from
Dunster Castle Conygar Tower, was constructed by architect Richard Phelps to improve the view from the castle, and a larger park of 141 hectares (348 acres) was built just to the south of the castle, requiring the eviction of a number of tenant farmers. Henry's son, John, inherited the castle in 1780, but when his son, also called John, inherited in 1816 he chose to live in London instead, opening up Dunster Castle to the public. By 1845 the castle appeared to visitors to be past its prime: with only two of John's sisters living there and no horses or hunting dogs