It was nonsense to pretend that he was afraid of the scandal, as no possible scandal could attach to him. 
He could not have been afraid of any scandal.
Scandals could very easily attach themselves to him.