However, young Barthé was beset with health problems, and after an attack of typhoid fever at age 14, he withdrew from school. Following this, he worked as a houseboy and handyman, but still spent his free time drawing. A wealthy family, the Ponds, who spent summers at Bay St. Louis, invited Barthé to work for them as a houseboy in New Orleans, Louisiana. Through his employment with the Ponds, Barthé broadened his cultural horizons and knowledge of art, and was introduced to Lyle Saxon, a local writer for the "Times Picayune". Saxon was fighting against the racist system of school segregation, and tried unsuccessfully to get Barthé registered in an art school in New Orleans.