Cowdrey's father, Ernest Arthur Cowdrey, played for the Surrey County Cricket Club Second XI and Berkshire County Cricket Club in the Minor Counties, but lacked the talent to enter first-class cricket and his father made him join a bank. Ernest Cowdrey had been born in Calcutta, moved to India to run a tea plantation and played the 1926–27 MCC touring team for the Madras Europeans XI and top scored with 48. His mother, Molly Cowdrey (née Taylor), played tennis and hockey.