Leaving St Peter's School, Yardley went to St John's College, Cambridge University, where he immediately began to show all-round ability at sports. He won the North of England Squash Championships every year between 1934 and 1939, and won his Blue in hockey, squash, and Rugby Fives. However, his main distinction came from cricket, where he was a Blue in each his four years at Cambridge. He played for the University team in his first year, the 1935 season, making ten first-class appearances without much success. His "Wisden" obituary noted that "class rather than performance guaranteed his place." He made his first-class debut against Sussex, scoring a duck in his first innings and 24 runs in the second. He passed fifty on just one occasion that season, scoring 319 runs at an average of 16.78, and bowled 69 balls without taking a wicket. Nevertheless, he played in the University Match, scoring just 19 and 36.