Earl Godwin's rebellion against the king in 1051 came as a blow to Ealdred, who was a supporter of the earl and his family. Ealdred was present at the royal council at London that banished Godwin's family. Later in 1051, when he was sent to intercept Harold Godwinson and his brothers as they fled England after their father's outlawing, Ealdred "could not, or would not" capture the brothers. The banishment of Ealdred's patron came shortly after the death of Ælfric Puttoc, the Archbishop of York. York and Worcester had long had close ties, and the two sees had often been held in plurality, or at the same time. Ealdred probably wanted to become Archbishop of York after Ælfric's death, but his patron's eclipse led to the king appointing Cynesige, a royal chaplain, instead. In September 1052, though, Godwin returned from exile and his family was restored to power. By late 1053 Ealdred was once more in royal favour. At some point, he was alleged to have accompanied Swein on a pilgrimage to the Holy Land, but proof is lacking.