Baker was born in Weakley County, Tennessee, a son of John Baker and his wife. Soon after that his family moved to Clarksville, Arkansas, and as Cullen Baker matured he spent much of his time in the saloons and bars in now Lafayette and Miller Counties. His father was an honest farmer, and owned cattle as well as working in the fields. Even when young Cullen Baker is said to have had a quick temper, which was revealed as he got older, as he drank heavily and often.