Lord Spencer was active in the local politics of Northamptonshire as a Conservative councillor. He opened his ancestral home, Althorp, to the public and was a well-known art connoisseur, being a trustee of the Wallace Collection and chairman of the Royal School of Needlework. He was a Fellow of both the Society of Antiquaries of London and the Royal Society of Arts, and for eight years in the 1960s he was Chair of the Advisory Council of the Victoria and Albert Museum. He was Chairman of the Governors at Wellingborough School from 1946 to 1972.