D'Oliveira was born in Cape Town. His father, Basil (1931–2011), played for Worcestershire and England. His uncle, Ivan (born 1941), played for Leicestershire. His son, Brett (born 1992), plays for Worcestershire.