Harris was born on 30 March 1930 in Bassendean, Perth, Western Australia, to Agnes Margaret (née Robbins) and Cromwell ("Crom") Harris, who had both emigrated from Cardiff, Wales. He grew up in Wembley, Western Australia. He was named after Rolf Boldrewood, the pseudonym of an Australian writer whom his mother admired. After his later fame, Harris was often referred to as "the boy from Bassendean" within Australia. As a child he owned a dog called Buster Fleabags, about whom he later wrote a book (for the UK Quick Reads Initiative).