Seeking a life of solitude, he visited a knight-hermit who lived in a cave by the River Nidd at Knaresborough, hiding from Richard I. On the death of the king, the knight returned home to his family leaving Robert on his own. The cave had a small chapel dedicated to St. Giles built around it. He continued to live there for some years, until a wealthy widow, Juliana, offered him a cell at St. Hilda’s Chapel in nearby Rudfarlington. There, he developed a reputation as a wise and holy man who cared for the poor. He stayed just a year before his hermitage was destroyed by bandits. Robert, dispossessed of his home, lived for a time under the church wall at Spofforth, and then he tried living with the monks at Hedley, near Tadcaster, but he found them far too easygoing for his style of life. By this time, the area had calmed down and he returned to Rudfarlington.