The athlete from Portaferry in County Down suffered a recurrence of a problem she has had with her right Achilles.
The 24-year-old said afterwards she felt she had to get off track immediately to see the Irish physio.
The race in Belgrade was won by Laura Muir of Great Britain who broke away from Germany's Konstanze Klosterhalfen.
It was a hugely disappointing end for Mageean, a bronze medallist outdoors at last year's European Athletics Championships.
"Going around there I just felt like I was losing power out of my right Achilles. I just wanted to get off and get to my physio," Mageean told RTE.
"Something is uncomfortable. Training had been going so well, something's up."
Ireland's John Travers was disappointed with his 11th place in the men's 1500m final in 3:53.11.
"Unfortunately I didn't put in the performance I wanted to put in," said Travers after the race which was won by Poland's Marcin Lewandowski.
Ciara Mageean hopes of a medal at the European Indoor Athletics Championships ended when she failed to finish the 1500m final because of injury.