By birth, Olivia was member of De Havilland family which belonged to landed gentry that originated from mainland Normandy. Her mother, Lilian Fontaine (née Ruse; 1886–1975), was educated at the Royal Academy of Dramatic Art in London and became a stage actress. Lilian also sang with the Master of the King's Music, Sir Walter Parratt, and toured England with the composer Ralph Vaughan Williams. Olivia's father, Walter de Havilland (1872–1968), served as an English professor at the Imperial University in Tokyo City before becoming a patent attorney. Her paternal cousin was Sir Geoffrey de Havilland (1882–1965), an aircraft designer and founder of the de Havilland aircraft company.