Feudal barony of Dunster Ryce, by whom he had no sons, only three daughters, Catherine, Dorothy and Mary, co-heiresses to 1/3 in total of his estate, the remaining 2/3rds going by entail to his younger brother Thomas Luttrell (died 1571). Mary survived him and remarried to James Godolphin of Cornwall. Thomas Luttrell (died 1571) of Dunster Castle (younger brother). In 1563 he was MP for the newly created Luttrell Pocket Borough of Minehead, two miles north-east of Dunster Castle. He sold the Devon and Somerset estates, excepting Dunster Castle, apparently to meet debts. These were however "amply replaced" by the large estate inherited from
Dunster Castle Howards regiment. During this period the castle still kept an armoury of 43 muskets. Francis died heavily in debt in 1690, and his widow Mary moved the contents of the castle to London, where they were destroyed in a fire in 1696. At the start of the 18th century the Luttrells and Dunster Castle faced many financial challenges. Francis's son Alexander, inherited the castle when he came of age in 1704, but it was still mostly empty and carried large debts with it. Alexander died young in 1711 and his widow, Dorothy, spent almost twenty years paying off the debts.