Brown's first wife was Henrietta Hay, daughter of Edward Hay, a career diplomat, and granddaughter of George Hay, 8th Earl of Kinnoull. In 1802 she died and, the following year, he married Elizabeth, daughter of Thomas Boddington, a notorious apologist for the slave trade. When he died in 1818, he left a lifetime's interest in the hall to his wife, who lived for another 21 years. She continued Brown's benefactions, making sure the school continued. She also contributed the greater part of the cost of rebuilding the parish church, dedicated to St. Giles. In 1833, work began on the rebuilding, to a design by Francis Halley of Shifnal. The chancel and nave were reconstructed without division, under a single pitched roof, while a tower stood at the western end, above the entrance. The old materials were used where possible, although more sandstone was quarried on the estate to complete the work. Five years later, a new rectory completed the rebuilding. However, when William Smith died in 1837, Elizabeth nominated a relative, Thomas F. Boddington, as his successor. He lived for at least part of his incumbency at Shifnal.