Charles sealed victory in the final kilometre of the 21km run to beat Australia's Annabel Luxford in four hours, 13 minutes 59 seconds.
Brownlee led going into the run but was overtaken by eventual winner Lionel Sanders before retiring from the race.
The fourth leg of the ITU world series takes places in Leeds from 11-12 June.
Brownlee is due to compete alongside his brother Jonny in the event.
The 29-year-old said on Twitter: "Tough day from the first stroke of the swim. Hate not finishing but didn't have a choice today.
"Great event and well done to the other guys."
Commenting on her win, Charles said: "I can't believe it. I could see that I was closing the gap on the run, but I didn't expect to claw in the race win.
"Epic course, epic race and I really am over the moon."
The middle-distance triathlon in Samorin, Slovakia included a 1.9km swim, 90km cycle and 21km run.
The organisers implemented a 20-metre draft rule on the bike course - a rectangle three meters wide and 20 meters long - which athletes are only allowed to enter to overtake and then have 40 seconds to complete the pass.
Charles added: "I think the 20m draft rule is great, and certainly works in my favour."
Full list of results.
Britain's Lucy Charles won the inaugural Championship in Slovakia as double Olympic champion Alistair Brownlee retired from the men's race.