The marriage was not a success. By the 1840s, Natalie Benjamin was living in Paris with the couple's only child, Ninette, whom she raised as a Catholic. Benjamin would visit them annually. While a senator, in the late 1850s he persuaded Natalie to rejoin him and expensively furnished a home in Washington for all three to live in. Natalie and their daughter soon embarked again for France. Benjamin, publicly humiliated by his failure to keep Natalie, consigned the household goods to auction. There were rumors, never substantiated, that Benjamin was impotent and that Natalie was unfaithful.