Rudolph was the oldest of two boys who were born to Plumie and Rethie Rudolph. As he got older, he began to help in the general store that his father owned. When he graduated from high school, he went to Paducah, Kentucky with his uncle. In 1933, Rudolph's uncle bought a doughnut shop and recipe for yeast-raised donuts. He bought them from a French federal employee named Joe La-beau. This all happened during the middle of the great depression, so they moved to Nashville, Tennessee to see if they could get better business there and secure their financial future.