Walker's education began with a governess at the family home, followed by spells at preparatory schools around the country. He attended Highgate School, gaining a Distinction in Divinity. While at Highgate he joined the School Bugles, learning to play the bugle. Walker was at a trials event with his father in Austria when the outbreak of the Second World War occurred in September 1939. Upon returning to Britain, the Highgate School governors became concerned about the possible extent of bombing raids on London, and so Walker and his fellow pupils were evacuated to Westward Ho! in Devon, staying there until 1941. During this time, Walker rose to the rank of Company Sergeant Major of the School Corps.