ernest found it difficult to ignore the more vicious of these rumours - the ones that accused miss foster of concocting a malevolent scheme to get the baby she 'd always wanted - and he 'd exchanged heated words , on several occasions , with a number of kingstonians in order to defend the young woman 's reputation .