Carew was popular with King Henry VIII, who sought his company, but was known in his youth for being something of a rake. He was one of a number of King Henry's companions whom Cardinal Wolsey believed had too much influence over the King. In 1518, Wolsey managed to have Carew sent away from court, replacing him with his own protégé, Richard Pace. He soon returned, but was removed again, to Ruysbank Tower, Calais, in 1519, when he was also High Sheriff of Surrey and Sussex. In 1521, he was made constable of Wallingford Castle, together with the stewardship of Wallingford. Wolsey finally engineered Carew's dismissal from the Privy chamber, when he presented the Eltham ordinances of 1526.