Burge was born in Kangaroo Point, Queensland, a suburb of the city of Brisbane, into a cricketing family. His father Thomas John "Jack" Burge was a salesman who rose to be a departmental manager of D. & W. Murray, a retail outlet, before becoming a state representative for Nile Industries, a textile firm. Jack Burge represented Eastern Suburbs in Brisbane's grade cricket competition and later became a cricket administrator. The elder Burge served on the Queensland Cricket Association executive from 1945 until his death in 1957. He also represented Queensland on the Australian Board of Control from 1952 until 1957, and was a state selector from 1944 until 1949. Peter recalled "My dad was the best frustrated Test cricketer I ever knew. He always wanted to play, was involved in it for as long as I could remember."