Jarislowsky was born in Berlin, Germany, the son of Kaethe (née Gassmann) and Alfred Jarislowsky. He is the stepson of a steel mill owner in Germany, who was ousted by the Nazis for harbouring Jews. Jarislowsky emigrated to the United States in 1941 after boarding schools in the Netherlands and France. He came of age and was deeply affected by the Great Depression. In Asheville, North Carolina, he attended the college preparatory school, Asheville School, and then studied mechanical engineering for two years at Cornell University. With the US entry into World War II, he served in the US Army. He finished basic training and studied Japanese at the University of Chicago before serving in counter-intelligence in Japan after the war.