Caffrey succeeded Tommy Lyons as Dublin manager in November 2004. He worked with David Billings, Paul Clarke and Brian Talty as his management team for a three-year period. In his first year as manager he took Dublin to their 45th Leinster Senior Football Championship title in 2005. He repeated this feat against Offaly in the 2006 Leinster Senior Football Championship final and went on to take Dublin to the All-Ireland semi-final, where Dublin were narrowly beaten by Mayo. Caffrey received a one match ban for shouldering Mayo's John Morrison before the All-Ireland semi-final against Mayo in Croke Park. That meant that he had to watch Dublin's opening 2007 game in the Championship from the stands.