Branded a bridge burner, a capital offense, Ellis lived in exile in the east Tennessee mountains. He eventually became a pilot, leading Unionists, Confederate deserters, prison escapees, slaves, and all manner of fugitives through the mountains into Kentucky or wherever the Union lines would advance. At the same time Ellis recruited for several regiments, provided information on Confederate activities to federal authorities, and maintained a mail service between mountain Unionists and their men in the Union army. Possibly the most successful of all such pilots, and despite a bounty on his head, Ellis always (often narrowly) escaped capture, and became known as "The Old Red Fox".