Browne was also keen to ensure that the parish was better served spiritually. None of the rectors had actually lived in the parish for at least a century and communion was celebrated only four or five times a year. This was an issue that clearly troubled Browne for many years: one of his rare parliamentary speeches was in favour of compelling absent clergy to pay for replacement curates. Dr. James Chelsum, a minor scholar, was the rector from 1780. He contrived to combine his benefice at Badger with the rectory of Droxford in Hampshire from 1782, and a chaplaincy at Lathbury in Buckinghamshire. Although Browne must initially have trusted Chelsum, he clearly became disenchanted and arranged for his departure in 1795. Chelsum retained his other benefices until he died insane in 1801. In Chelsum's place Browne nominated William Smith, who proved a conscientious minister for 42 years. Smith was never absent from the parish for more than two weeks in the whole of his incumbency. Browne must have valued Smith greatly, as he bequeathed him the right to nominate his own successor. In the event, Smith sold the right back to Browne's widow in 1820 for £1200.