Raoul Lufbery was born at Avenue de la Poudrière in Chamalières, Puy-de-Dôme, France to American Edward Lufbery and a French mother. Lufbery's paternal grandfather was Charles Samson Lufbery, who had emigrated to the United States from Great Britain in the mid-19th century and settled in New York. Lufbery's father, Edward moved to Chamalières in 1876, joining his elder brother, George and soon meet a local Frenchwoman, Anne Joséphine Vessière, who would later become his wife. Raoul was the youngest of their three sons. Edward was an American chemist working for a Parisian chocolate company. When Lufbery was one, his mother died and his father returned to Connecticut, United States, leaving him to be raised by his maternal grandmother, Madeline Vessière Greniere in France.