Neugebauer was born in Innsbruck, Austria. His father Rudolph Neugebauer was a railroad construction engineer and a collector and scholar of Oriental carpets. His parents died when he was quite young. During World War I, Neugebauer enlisted in the Austrian Army and served as an artillery lieutenant on the Italian front and then in an Italian prisoner-of-war camp alongside fellow countryman Ludwig Wittgenstein. In 1919, he entered the University of Graz in electrical engineering and physics and, in 1921, transferred to the University of Munich. From 1922 to 1924, he studied mathematics at the University of Göttingen under Richard Courant, Edmund Landau, and Emmy Noether. During 1924–1925, he was at the University of Copenhagen, where his interests changed to the history of Egyptian mathematics.