Unlike many other Ashkenazi Jewish immigrants, Victor Rostow always spoke to his children in English rather than Yiddish as he felt this was would improve their chances in life. Rostow's brother Eugene, who was named for Eugene V. Debs, became a legal scholar, and his brother Ralph, a department store manager. The American journalist Stanley Karnow described Rostow as extremely intelligent with a "brilliant" academic record that saw him graduate from high school at the age of 15. Rostow described his childhood as mostly happy with the only dark spots being that sometimes his classmates called him and his brothers "dirty Jews".