Travers was born in the London borough of Hendon, the elder son and the second of the three children of Walter Francis Travers, a merchant, and his wife, Margaret, "née" Burges. He was the great-grandson of Benjamin Travers, Surgeon Extraordinary to Queen Victoria. He was educated at the Abbey School, Beckenham, and at Charterhouse. He did not greatly enjoy his schooldays and later declared that he had been "a complete failure at school". The only thing he enjoyed there was cricket, for which he had a lifelong enthusiasm, later writing a memoir focusing on his passion for the game, "Ninety-four Declared: Cricket Reminiscences". When he was nine, his father took him to the Ashes match at the Oval. Eighty years later he recalled watching W G Grace and F S Jackson opening the batting for England with Ranjitsinhji coming in first wicket down: "I remember when Ranji came in to bat the crowd started singing; I think he only made 7; it was a very low scoring match."