Walker was born at 214 Reddings Lane in Hall Green, Birmingham, England, on 10 October 1923. His family is of Scottish descent. His father Graham Walker was a despatch rider and works motorcyclist for the Norton Motorcycle Company who participated in the Isle of Man TT. His mother, Elsie Spratt, was the daughter of Harry Spratt, a businessman from Leighton Buzzard, Bedfordshire. Walker was an only child. In 1925 Walker and his family moved to Wolverhampton when Graham became the Competition Manager for Sunbeam. The family later moved to Coventry in 1928, when Graham worked as the Sales and Competition Director for Rudge-Whitworth.