The end result is either a waste of resources, as patients are sent for expensive and unnecessary scans.
Or worse, the full extent of the injury doesn't become apparent until hours later, often with fatal results.
That was exactly what happened in the tragic case of actor Natasha Richardson.
In 2009 she fell while skiing and initially laughed the incident off and refused any medical treatment.
But two days later she died from a brain injury sustained during the accident.
For doctors, a simple blood test to detect brain injury would be a huge deal.
It would reveal the patients who, despite all looking and acting perfectly fine, were actually seriously injured and, on top of that, save money on expensive scans for those that are fine.
And this is what researchers in Birmingham have achieved; the first blood test for brain damage.
It works by looking for tiny amounts of chemicals produced when the brain is injured.
A small drop of blood is placed on a silicon and gold chip and then it is placed in a special detector for analysis.
The team from the University of Birmingham and the Queen Elizabeth Hospital have used blood from real patients as part of their research.
They say their test reveals accurately which patients have damage to the brain and which are fine.
The aim is to reduce the size of the prototype detector by half so it could be carried by an ambulance and used at the scene of an accident.
At the moment the prototype detector is quite large and sits on a lab bench.
The finished device should be briefcase-sized and easy to keep in an ambulance, at a sports ground or even in an army vehicle on a battlefield.
This research has caught the eye of Alta Innovations Limited, a company set up by the university to get promising research like this to market.
They hope a real detector could be on sale in three years.
If they succeed, it would be an invention that saves money and lives.
Injuries to the brain are impossible to detect at the scene of an accident.