O'Brian was born in Chalfont St Peter, Buckinghamshire, England. He served during the Second World War. He was married to Elizabeth Jones until their divorce. They had two children, Jane and Richard. He was married Mary Tolstoy from 1945 until her death in 1998. They lived in Collioure, France.