Burge was inculcated with a love of cricket by his father, who reputedly gave him a rattle in the shape of a ball and bat when he was a baby. First holding a bat at the age of three, Burge infuriated his mother by striking a ball wrapped in a sock and tied to a rope, for hours on end. Aged five, he went to Buranda Boys' State School and due to his father's coaching, was by far the best cricketer at the school. He played his first competitive match when he was eight and a half, and scored his first century at nine. At the age of nine, Burge scored 223 for Buranda and then decided to retire because of extreme heat. His father berated him for handing the opposition his wicket. Burge said "It was good advice. I never did it again."