Andersen was born in Chicago, Illinois. His mother, Jennie Olivia Johnson (1877–1925), was the daughter of a seaman from Luleå, in northern Sweden, who came to America as a young man and worked in the timber business. Elmer's father, Arne Kjelsberg Andersen (1866–1926), was an immigrant from Solør, Norway, who had settled in Chicago and became a streetcar motorman, operating on the Halsted streetcar line out of the Ashland Avenue car barns. "My earliest memory," Andersen wrote in his memoirs, "is of riding with him on the streetcar and being permitted to clang the bell as we came to street crossings." His parents separated when he was six years old. Andersen never understood why his parents separated and never questioned them. Elmer moved with his mother and infant sister, Caroline, to Muskegon, Michigan. His two older brothers, Arnold and Marvin, arrived in Muskegon later. The Andersens had only sporadic contact with their father after the separation. At the age of nine Elmer contracted a mild form of polio but, through exercise, regained his strength.