Caro was born in New York City, the son of Celia (née Mendelow), also born in New York, and Benjamin Caro, born in Warsaw, Poland. He grew up on Central Park West at 94th Street. His father, a businessman, spoke Yiddish as well as English, but he didn't speak either very often. He was 'very silent,' Caro said, and became more so after Caro's mother died, after a long illness, when Robert was 12. It was his mother's deathbed wish that he should go to the Horace Mann School, an exclusive private school in the Riverdale section of The Bronx. As a student there, Caro translated an edition of his school newspaper into Russian and mailed 10,000 copies to students in the USSR. Graduated in 1953, he went on to Princeton University, where he majored in English. He became managing editor of "The Daily Princetonian", second to Johnny Apple, later a prominent editor at "The New York Times".