De Burgh finally fell out of favour with Henry III; he was imprisoned and then finally stripped of Hadleigh Castle in 1239. For the rest of the century, Hadleigh was retained as a royal castle, as part of an estate containing of agricultural land, the park and the castle mill. By the 1250s, the castle had fallen into neglect and, despite some investment after it was given to Queen Eleanor in 1273, it remained in relatively poor condition. Only the mill, vital for the operation of the wider estate, appears to have been well-maintained. A new 17-metre-wide by 9-metre-long (56 foot by 30 foot) hall and an adjacent solar complex were built at the castle around 1290, but collapsed due to subsidence shortly afterwards. In 1299 the castle was given to Queen Margaret, who complained about the quality of the building and insisted that repairs were carried out. Her husband, Edward I, visited the castle twice, using it as a base for hunting in the area.