apparently , the ladies he brought home for a weekend fling were fair game for gossip and speculation while the duke himself was protected , with no one saying a mean or disrespectful comment about him .