Lufbery worked in a chocolate factory in Blois and Clermont-Ferrand until 1904. In 1906, Lufbery, along with his middle brother, Charles took a trans-Atlantic liner to the US to search for his father, not knowing that his father had left on an ocean liner heading the opposite direction to France. However, he managed to visit other family members in Connecticut. He ran away from his grandparents' home at 17, and travelled to such places as Egypt, Algeria, Tunisia, the Balkans, and Turkey.