In May 2006, Dorsey was traded to the Cleveland Browns along with a 7th round pick in the 2007 NFL Draft for veteran quarterback Trent Dilfer, after having just re-signed with the 49ers. During training camp, Dorsey and Derek Anderson competed for the 2nd string position behind starter Charlie Frye. In the battle which ended in a virtual tie, Dorsey went on to serve as the third-string quarterback behind Anderson for all of the 2006 season. Dorsey was named the starter for the week 17 game against the Houston Texans but the previously injured Charlie Frye replaced him just minutes before the game. The Browns went on to lose to the Texans as Frye threw an interception and no touchdown passes.