Heart problems, insomnia and panic attacks were all linked to excess caffeine consumption.
The EFSA said many people were consuming more than the safe level as they were unaware of all the different sources of caffeine.
Safe limits for children and pregnant women are much lower.
Caffeine is often a crutch to help us get through the day.
Tea, coffee, energy drinks, soft drinks and chocolate are all common sources of the drug.
The EFSA was tasked with assessing all the evidence to determine safe consumption levels.
Its report says 400mg a day, in healthy adults, has no health consequences.
This is in line with advice in many countries, including the UK, but the EFSA warns many people drink too much.
Around a third of people in Denmark, 17% in the Netherlands and 14% in Germany consume more than 400mg.
An EFSA spokesman told the BBC: "If you have a bar of dark chocolate at 11:00 and espresso with lunch, a tea at 16:00 and vodka-Red Bull in the evening - that's a lot of caffeine over the day."
Source: EFSA
The main health issues raised were: increased heart rate, higher blood pressure, irregular heartbeat, tremors, nervousness, insomnia and panic attacks.
The limit is 200mg a day in pregnant women due to the impact on the growing foetus.
The report acknowledged there was a limited amount of data on safe levels for children.
The EFSA is recommending 3mg per day for every kilogram the child weighs.
It also found no extra risk as a result of combining caffeine and alcohol.
Drinking more than five espressos worth of caffeine a day may be damaging to health, according to the European Food Safety Authority.