At the beginning of the Great Depression, John Brown lost his job and relocated the family to Palmer's Crossing, from Hattiesburg, where he worked at a turpentine factory until he was laid off in 1938. John Brown moved the family to Lux, Mississippi, where he worked as a sharecropper on a farm. During this time, Jesse Brown shared a bed with his brothers (as was common among many families) and attended a one-room school away. His parents were very strict about school attendance and homework, and Jesse Brown walked to school every day. The bros also were committed Baptists and Jesse, William, and Julia Brown sang in the church choir. In his spare time, Brown also worked in the fields of the farm harvesting corn and cotton.