Cave-Brown had been on the books of Charlton Athletic but joined on a scholarship at Norwich City for the 2004–05 season. He made his full Norwich debut in the dying moments of the FA Cup third round defeat against West Ham United in January 2006, and made his first start in the League Cup victory over Torquay United in August 2006, although he was stretchered off after half an hour with a leg injury. He signed a professional contract in April 2006.