Robert was well known for his charity to the poor and destitute. His favorite form of charity was to redeem men from prison. For a time, Robert prospered, having four servants and keeping cattle, but he was soon in trouble again, this time with William de Stuteville, the constable of Knaresborough castle, who accused him of harbouring thieves and outlaws. Having his hermitage destroyed for the second time, this time by the forces of law and order under William de Stuteville, Robert returned to the cave at Knaresborough, where he stayed for the rest of his life.