The property was inherited by a settlement made in 1748 by Bernard Ward's eldest son, Nicholas, 2nd Viscount Bangor, who was clearly insane. When his younger brother, Edward, died in 1812, leaving a young son, the youngest brother Robert took the opportunity to relocate the insane Nicholas into a smaller house in Downpatrick and strip Castle Ward of everything valuable. The house stood empty until the death of Nicholas in 1827, when it was inherited by Edward's son, now the 3rd Viscount. He and his descendants restored the building and its furnishings, but on the death of the 6th Viscount in 1950 the house and estate were given in lieu of death duties to the Government of Northern Ireland, who presented the house and its gardens to the National Trust in 1952.