Wolters, who was born to a Catholic middle-class family in the northern German town of Coesfeld, obtained his degree and doctorate in architecture from the Technical University of Berlin, forging a close friendship with Speer while a student. After receiving his doctorate, he had difficulty finding employment prior to the Nazi rise to power. From 1933 to 1937, he worked for the Reichsbahn. In 1937, Speer hired him as a department head, and Wolters soon took major responsibility for Hitler's plan for the large scale reconstruction of Berlin. When Speer became Minister of Armaments and War Production in 1942, Wolters moved to his department, remaining his close associate.