Hagen was born to CEO Ragnar Hagen (1908–1969) and accountant Gerd Gamborg (1914-2008). He was named after his paternal grandfather, Carl, and his maternal grandfather, Ivar. He has two siblings, one younger, and one older sister. Hagen was before joining the Progress Party a passive member of the Young Conservatives, and according to him, both his parents voted for the Labour Party. According to Hagen himself and his secondary school classmates, he was relatively shy in his younger years. When he was seventeen years old, in 1961, he took work as an apprentice on the Norwegian America Line ship "MS Foldenfjord". He achieved Examen artium in 1963. In 1964, he was conscripted in the Norwegian Army, and served as an engineer soldier at Eggemoen near Hønefoss, and Maukstadmoen in Troms.