The 52-year-old, who has not been named, was taken to hospital by air ambulance following the incident during the event on Friday afternoon.
Police said the man, from Wallington, south London, had suffered a "medical emergency" and died later in hospital.
The three-day event is one of several open water swimming festivals held throughout the country.
Entrants chose to swim over several distances, including half-mile, one mile, two miles and 5k courses.
A Cumbria force spokeswoman said: "At 4.40pm police at the event were alerted to a man who was taken by air ambulance to the Cumberland Infirmary, as a result of a medical emergency.
"The man, aged 52, sadly died.
"The death is not being treated as suspicious and the coroner has been informed."
No-one from the event was available for comment.
A man has died after taking part in the Great North Swim on Windermere in the Lake District.