From 1777, she is known to have been the mistress of Prince Charles, Duke of Södermanland, brother to king Gustav III, who became the most known of all her lovers. Their affair continued on an on-an-off basis for about twenty years, was regarded a scandal and attracted much negative publicity to Duke Charles. Unlike her colleague in the Ballet, Sophie Hagman, who was the mistress of Charles' brother Prince Frederick, Charlotte Slottberg was never allowed to attend the royal court or have any position of an official mistress because of her reputation as a courtesan, and her affair with Charles was in contrast to that of Hagman and Prince Frederick never publicly acknowledged in any way. Their relationship was never exclusive in either part. In parallel to her, Duke Charles had a string of other lovers simultaneously - his affairs to Charlotte Eckerman and Françoise-Éléonore Villain, among others, all took place when Slottsberg was his mistress. Charlotte Slottsberg herself also had several acknowledged lovers in parallel to Charles. Nevertheless, Slottsberg was referred to as the "Favorite Sultaness" of his "harem" and as someone he always returned to in the end, and Johan Magnus af Nordin referred to her as "the Concubiness, the deputy duchess and regentess etc".