Oliver was born on 23 April 1977 in Erdington, Birmingham, West Midlands, England, to Carole and Jim Oliver. His father was a school headmaster and social worker, and his mother was a music teacher. Both of his parents are originally from Liverpool, Merseyside. His uncle was the composer Stephen Oliver. His father's great-grandfather, William Boyd Carpenter was the Bishop of Ripon and court chaplain to Queen Victoria. Oliver learned to play the viola as a child.