Lavoe was born and raised in the Machuelo Abajo barrio of Ponce, Puerto Rico. Early in his life, he attended Escuela Libre de Música de Ponce, known today as the Instituto de Música Juan Morel Campos and, inspired by Jesús Sánchez Erazo, developed an interest in music. He moved to New York City on 3 May 1963, at the age of sixteen. Shortly after his arrival, he worked as the singer in a sextet formed by Roberto García. During this period, he performed with several other groups, including Orquesta New York, Kako All-Stars, and Johnny Pacheco's band.