Feudal barony of Dunster Ryce, by whom he had no sons, only three daughters, Catherine, Dorothy and Mary, co-heiresses to 1/3 in total of his estate, the remaining 2/3rds going by entail to his younger brother Thomas Luttrell (died 1571). Mary survived him and remarried to James Godolphin of Cornwall. Thomas Luttrell (died 1571) of Dunster Castle (younger brother). In 1563 he was MP for the newly created Luttrell Pocket Borough of Minehead, two miles north-east of Dunster Castle. He sold the Devon and Somerset estates, excepting Dunster Castle, apparently to meet debts. These were however "amply replaced" by the large estate inherited from
Dunster Castle be slighted to prevent their further use. In the 1860s and 1870s, the architect Anthony Salvin was employed to remodel the castle to fit Victorian tastes; this work extensively changed the appearance of Dunster to make it appear more Gothic and Picturesque. Following the death of Alexander Luttrell in 1944, the family was unable to afford the death duties on his estate. The castle and surrounding lands were sold off to a property firm, the family continuing to live in the castle as tenants. The Luttrells bought back the castle in 1954, but in 1976 Colonel Walter Luttrell gave Dunster