most likely no one had ever dared say such a thing to his face , although the villagers , even his own servants , gossiped about his late wife 's many dalliances and anything else that had to do with him or his family .