Miller grew up in St John's Wood, London, in a well-connected Jewish family. His father Emanuel (1892–1970), who was of Lithuanian descent and suffered from severe rheumatoid arthritis, was a military psychiatrist and subsequently a paediatric psychiatrist at Harley House. His mother, Betty Miller (née Spiro) (1910–1965), was a novelist and biographer who was originally from County Cork, Ireland. Miller's elder sister Sarah (died 2006) worked in television for many years and retained an involvement with Judaism that Miller, as an atheist, always eschewed.