Ferrer grew up on Fox Street in the Longwood section of the South Bronx and was raised by his mother and by his grandmother, who worked in the kitchen of the Waldorf-Astoria Hotel. Ferrer graduated from Catholic schools in the Bronx: St. Anselm Elementary School and Cardinal Spellman High School. As a high school student, he was a member of ASPIRA of New York, where he was elected to the post of Vice President of the citywide Aspira Clubs Federation (ACF), which included other future Puerto Rican leaders like Ninfa Segarra and Angelo Falcón. In 1968, as an Aspirante, he was part of a major student protest at the NYC Board of Education that resulted in such reforms as bilingual report cards and the recognition by the public schools of Puerto Rican Discovery Day (November 19). His family have Catalan origins.