Eric Dier's stoppage-time header sealed a 3-2 win for Roy Hodgson's side in Berlin's Olympiastadion.
Toni Kroos and Mario Gomez had put Germany two goals up but Harry Kane and substitute Jamie Vardy levelled for England.
"We all need to take a look at ourselves and work hard now," said Juventus' Khedira.
"We did a lot right up until the second goal, then after that we were much worse - and England punished us.
"However, we played well for an hour. We need to play well for 90 minutes against Italy [in Tuesday's friendly] and in the European Championship."
Germany striker Gomez said the defeat was "inexcusable".
"We were playing very well until we were 2-0 up but we didn't use enough of our chances," he added.
"Afterwards, we stood off our opponents too much and England are very strong from set-pieces."
Saturday's friendly defeat by England was "unnecessary and stupid", according to Germany midfielder Sami Khedira.