Because, so long as the evidence against him was vague and intangible, it was very hard to disprove.
The evidence is circumstantial enough to make guilt believable, but he could also be completely innocent.
Proving your guilt or innocence is very easy to do when someone makes a charge against you.