Upon Tao Qian's death, the influential Mi clan strongly advocated the governorship be passed on to Liu Bei, to whom Mi Fang had his sister married. Thereafter, the Mi brothers joined Liu Bei's army to Xuyi and Huaiyin (淮陰; in Guangling, south of Xu Province) to counter-attack the warlord Yuan Shu in 196 CE. Zhang Fei, who was left behind by Liu Bei to guard Xiapi (capital of Xu Province at the time), killed Cao Bao, the chancellor of Xiapi when Tao Qian was still in charge of Xu Province, after an intense quarrel over some trivial things. Cao Bao's death caused unrest in the city, and the locals opened the city gate for Lü Bu, who then seized control of the city, capturing the families of Liu Bei and Mi Fang during the process. Upon hearing the news, Liu Bei's soldiers started to desert, and the remnant was defeated by the enemy commander Ji Ling. Liu Bei then retreated to Haixi (), Donghai Commandery (). Faced with enemies on both sides and a lack of supplies, the Mi brothers encouraged Liu Bei and used their personal wealth to support the army. Seeing no other viable option, Liu Bei requested for a truce with Lü Bu, who accepted and returned Liu Bei's family as an act of good faith. In fear of being isolated by Yuan Shu, Lü Bu obstructed further attempts by Yuan Shu to eliminate Liu Bei.