Sir Lucius O'Brien was the eldest son of the first Sir Edward and was the 3rd Baronet. He died in 1794. His son, the second Sir Edward, was the 4th Baronet, and decided to rebuild the castle. In 1821, it was the birthplace of Edward's brother Robert's son, George O'Brien. Work began around 1822 and cost about Ј50,000 to complete. The Pain brothers submitted some classical designs but Edward O'Brien chose their neo-gothic designs, influenced by John Nash. James and George Richard Pain had been pupils of Nash in England. The building was completed in 1835.