Rau grew up in a suburb of Stuttgart, where early on he became active in socialist youth organizations. After military service in World War I, he participated in the German Revolution of 1918–19. From 1920 onward, he was a leading agricultural policy maker of the Communist Party of Germany (KPD). This ended in 1933, when Adolf Hitler came to power. Shortly afterward Rau was thrown in jail for two years. As an enemy of the Nazi regime in Germany he was imprisoned, in total, for more than half of the time of Hitler's rule. After his first imprisonment he emigrated in 1935 to the Soviet Union (USSR). From there, in 1937, he went on to Spain, where he participated in the Spanish Civil War as a leader of one of the International Brigades. In 1939, he was arrested in France, and was delivered by the Vichy regime back to Nazi Germany in 1942. After a few months in a Gestapo prison, he was transferred to the Mauthausen Concentration Camp in March 1943. While in the concentration camp he participated in conspiratorial prisoner activities, which led to a camp uprising in the final days before the end of World War II in Europe.