Media playback is not supported on this device
Moritz Furste scored from two penalty corners before Florian Fuchs scored the winner with no time left to cap a remarkable comeback from 2-0 down.
Elsewhere, favourites Australia were thumped 4-0 by the Netherlands.
They will now face neighbours Belgium, who beat India, while Germany will meet Argentina after they beat Spain.
In the women's pool stage China were sent home with a 3-0 defeat by New Zealand who will now play Australia in the quarter-final.
Great Britain play Spain in Monday's third quarter-final (22:00 BST).
Subscribe to the BBC Sport newsletter to get our pick of news, features and video sent to your inbox.
Defending Olympic champions Germany scored three times in the last five minutes to progress to the semi-finals of the men's hockey.