Whittaker’s plant collecting activities began to decline around 1863, around the time his botanical partner, Henry Harpur Crewe, moved away to become the Rector of Drayton Beauchamp in Buckinghamshire. In the following year they cooperated in the production of a manuscript list of the principal flowering plants and ferns of Derbyshire. His partnership with Crewe lasted for at least eighteen years and was very productive. The Harpur Crewe's family lived nearby Calke Abbey and they had funded the school at Breadsall.