Yardley's batting form dipped in the 1950 season. He scored 1,082 runs at an average of 24.59, the final time he reached four figures in a season. With the ball he took 19 wickets at an average of 32.10. Yorkshire finished third in the County Championship behind joint winners Lancashire and Surrey. It took seven matches for Yardley to reach fifty runs in an innings, but he hit centuries against Surrey, Somerset and Scotland in the second half of the season. The West Indies toured England and Yardley resumed the England captaincy; he also captained MCC against the tourists and England against The Rest in a Test trial. However, neither Yardley nor Mann, the two likeliest candidates, were able to accept the captaincy of the MCC side in Australia that winter. The selectors spent much of the season assessing other players. Although Yardley represented the Gentlemen against the Players, the side was led by Brown, who scored a century and was appointed captain of the touring side. Brown also assumed the captaincy of England for the final Test against West Indies and Yardley was left out of the team. In the three Tests he played, Yardley scored 108 runs at an average of 18.00 with a top score of 41. He won the first Test but lost the next two, West Indies' first Test wins in England; the final Test was also lost by Brown. Following this series, Yardley did not play any more Tests, although his name was mentioned as a potential captain in 1953 before Hutton was appointed as England's first professional captain of the twentieth century. At the time, Yardley was still considered the best amateur candidate. In 20 Tests, Yardley scored 812 runs at an average of 25.37 and four fifties. With the ball, he took 21 wickets at an average of 33.66. On the fourteen occasions he was captain, he won four times, lost seven and drew three.