He was born in St. Louis, Missouri to Jessie (née Crane) and Walker Evans. His father was an advertising director. Walker was raised in an affluent environment; he spent his youth in Toledo, Ohio; Chicago; and New York City. He attended the Loomis Institute and Mercersburg Academy, then graduated from Phillips Academy in Andover, Massachusetts in 1922. He studied French literature for a year at Williams College, spending much of his time in the school's library, before dropping out. He returned to New York and worked as a night attendant in the map room of the Public Library. After spending a year in Paris in 1926, he returned to the United States to join a literary and art crowd in New York City. John Cheever, Hart Crane, and Lincoln Kirstein were among his friends. He was a clerk for a stockbroker firm on Wall Street from 1927 to 1929.