He had some disputes with the Audiencia. His opponents charged him with various offenses in the juicio de residencia (trial of grievances) that followed his administration. This was a nearly routine investigation at the end of a viceregal administration.
There were disagreements between him and the Audiencia. His adversaries brought up several charges during the trial of grievances, which happened after he served as viceregent. Such an inquiry was typical near the end of his term as viceroy.