Despite his high office and eminent reputation, Ansbert was removed from the bishopric around the year 690. By "a false accusation", the powerful "majordomo" of the Frankish court, Pepin of Heristal, arranged his dismissal, either because of some kind of political opposition or because Ansbert's "zeal was not well-received" and "his austere life caused offence". He was sent into exile at the monastery of Hautmont where he stayed until his death, sometime between 692 and 695. At some time Pepin apparently reconsidered his actions and agreed to allow Ansbert to return, but either he changed his mind again or Ansbert died before making the trip.