She was born in Telford, Tennessee. She was the only child of John McCardle, a shoemaker, and Sarah Phillips-McCardle. Eliza lost her father when she was still a small child. She was raised by her widowed mother in Greeneville, Tennessee]. One day in September 1826, Eliza was chatting with classmates from Rhea Academy when she spotted Andrew Johnson and his family pull into town with all their belongings.