In his personal life Brown played polo at Ham Polo Club in the summer and during the winter he spent most weekends hunting; he was joint Master of the South Oxford hounds. He also bred hunters and racehorses on his 700-acre farm in Buckinghamshire. His greatest success was his horse Linwell winning the 1957 Cheltenham Gold Cup. Brown was a qualified pilot, had his personal De Havilland Dove, which was normally flown by his personal pilot and also established his own airfield at Crosland Moor, to the south-west of Huddersfield.