He was born as Raphael Schoar and both he and his identical twin brother, Moses, were born in Borisoglebsk, Tambov, a southern province of Russia on December 25, 1899. Their father, Abraham Shauer, a Hebrew scholar, writer and teacher, raised his six children in an intellectual environment in which much emphasis was placed on academic and artistic pursuits. Their mother, Bella, was an embroiderer. Their cousin was painter and meteorologist, Joshua Zalman Holland. Due to the many difficulties for the Jewish population in the late Russian Empire, the Soyer family was forced to emigrate in 1912 to the United States, where they ultimately settled in the Bronx. The family name changed from Schoar to Soyer during immigration.