Olsen was born in Bridgeport, Connecticut. He grew up in nearby Stratford, Connecticut. His father's parents came from Norway and his mother's parents came from Sweden. Olsen began his career working summers in a machine shop. Olsen fixed radios in his basement as a child.