Bergen was born in Chicago, Illinois, one of five children and the younger of two sons of Swedish immigrants Nilla Svensdotter (née Osberg) and Johan Henriksson Berggren. He lived on a farm near Decatur, Michigan until he was four, when his family returned to Sweden, where he learned the language. He taught himself ventriloquism from a pamphlet called "The Wizard's Manual" when he was 11, after his family had returned to Chicago. He attended Lake View High School. After his father died, when Edgar was just 16, he went out to work as an apprentice accountant, a furnace stoker, a player-piano operator, and a projectionist in a silent-movie house.