Despite the friendship and the inducement of Marcossay, Bishop of Geneva, young Allarmet retired to the Chartreuse of Dijon, where his merits soon became widely known. When Robert of Geneva was elected pope by the faction hostile to Pope Urban VI, Allarmet joined him at Avignon, either having been sent by the Duke of Burgundy or called by Robert himself.
Despite Marcossay's influence and Bishop of Geneva's persuasion, young Allarmet relocated to the Chartreuse of Dijon. His abilities quickly gained recognition. When Robert of Geneva was elected pope by a faction opposed to Pope Urban VI, Allarmet joined him in Avignon—either sent there by the Duke of Burgundy or directly called by Robert.