After moving to Dorking in around 1789, Labilliere often visited Box Hill to meditate. With old age he became increasingly eccentric and neglected his own personal hygiene to such an extent that he acquired the nickname "the walking dung-hill". He died on 6 June 1800. In accordance with his wishes he was buried head downwards, on 11 June on the western side of Box Hill above The Whites. In the presence of a crowd of thousands that included visitors from London as well as the local "quality gentry", Labilliere was buried without any religious ceremony, having reportedly said that the world was "topsy-turvey" and that it would be righted in the end if he were interred thus. However, in his "Book of Devotions", he wrote that he wished to emulate the example of St Peter, who was crucified upside-down according to tradition.