Roelfzema was born in the former Dutch East Indies in Surabaya, on Java, when it was under colonial rule. He was from a patrician family, which is listed in the . He was the second child of Siebren Erik Hazelhoff Roelfzema and his wife Cornelia Vreede. His older sister, Ellen, had been born two years before. His father managed rubber and coffee plantations. In the 1930s the family returned to the Netherlands so the children could be provided with a good education. They initially moved to The Hague, and later moved on to Wassenaar. He played soccer and was a member of the Hague Football Association.