Sassoon was a member of the prominent Jewish Sassoon family and Rothschild family. He was born in his mother's mansion on Avenue de Marigny, Paris. His father was Sir Edward Albert Sassoon, 2nd Baronet, MP, son of Albert Abdullah David Sassoon; his mother was Aline Caroline, daughter of Gustave Samuel de Rothschild. His sister was Sybil Sassoon, who married the Marquess of Cholmondeley. He was a cousin of the war poet Siegfried Sassoon. He was descended from the banking family of Frankfurt. When aged only nineteen years old his great-grandfather, James Rothschild was sent to Paris to set up the family business in France. James became wealthy. When he died in 1868 he was buried in Père Lachaise Cemetery. His branch of the Sassoon-Rothschild family kept the Jewish faith, donated to Jewish charities and founded synagogues.