though many people had claimed to see the ghost of the raven-haired woman floating around doing nasty things to guests and servants but never to family , unless that family happened to be sleeping with the man of the castle , as in , say , his wife ( which it was always his wife , only on one occasion was it his mistress ) , no one had proof that she actually killed anyone .