German captain Moritz Furste put the Germans ahead from a penalty corner but Eugene Magee blasted in an equaliser five minutes before half-time.
Goals from Furste and Martin Zwicker made it 3-1 before Mitch Darling's fine finish set up a tense finale.
Ireland must beat Canada and Argentina in their last two matches to keep hopes alive of making the quarter-finals.
The Irish have struggled to convert and defend penalty corners in Rio and they were again undone by a set-piece move for the opener as Furste hit the bottom corner late in the first quarter.
Germany forced the pace early in the second quarter but there was no way past impressive Irish keeper David Harte.
Magee made it level with a powerful reverse shot and Michael Watt fired just wide with a good chance early in the third quarter.
It was a costly miss as Furste netted from another penalty corner before Zwicker burst through to slip the ball past keeper Harte.
Ireland have shown determination and resilience in their pool games and they gave the Germans a scare when Darling fizzed into the corner with less than two minutes remaining.
The goal proved to be in vain as Germany held firm and Ireland now face must-win matches against Canada on Thursday and Argentina a day later.
"Our defence definitely improved - Germany are a good team and the stats will probably prove they had more circle entries than us but our composure was good and we ran hard in the heat," said Irish coach Craig Fulton.
"We did ourselves justice. You come off the back of a 5-0 loss and the character is determined by how the team responds and they definitely did. I am proud of the guys."
Ireland: D Harte, J Jackson, R Gormley, M Watt, J Jermyn, E Magee, K Shimmins, M Darling, K Good, P Gleghorne, C Harte
Subs: J Bell, C Cargo, A Sothern, P Caruth, S O'Donoghue
Germany: N Jacobi, M Muller, L Butt, M Haner, M Grambusch, C Wesley, T Hauke, C Ruhr, M Fuerste, F Fuchs, T Oruz
Subs: M Trompertz, T Herzbruch, T Grambusch, M Zwicker, N Wellen
A 3-2 defeat by Olympic champions Germany leaves Ireland without a point from their opening three Pool B games.