Refregier was born in Moscow and emigrated to the United States in 1920. After working various odd jobs in New York City, he earned a scholarship to the Rhode Island School of Design in 1921. After finishing school, Refregier moved back to New York in 1925. To earn a living, Refregier worked for interior decorators, creating replicas of François Boucher and Jean-Honoré Fragonard paintings. He continued his creative development, and traveled to Munich in 1927. While there he studied under painter Hans Hofmann, who was creating abstract expressionism paintings.