The university's Drug Discovery Unit (DDU) said it had discovered new compounds which had been shown to be successful when given orally to mice at very low doses.
The team is now working to improve their properties to the point where they can select a candidate drug.
DDU is working on the project with Medicines for Malaria Venture (MMV).
Professor Ian Gilbert, one of the leaders of the research effort, said: "This is tremendously exciting and amazing progress.
"We have discovered an entirely new class of compound that holds a great deal of anti-malarial promise."
The project started after the biology team at the DDU screened one of their collections of compounds against the malaria parasite.
Over the course of 18 months, two compound series were modified and refined before being tested.
MMV chief scientific officer Tim Wells commented: "Malaria control and elimination continues to face numerous challenges, not least of which is the threat of emerging resistance to the current effective treatment - artemisinin.
"In preparation for this eventuality, MMV and partners are researching over 50 projects in the largest-ever pipeline of anti-malarial medicines.
"DDU scientists have given us more compounds to work on that we hope to take through the research process.
"If successful, this class of compounds could well become a new source of much-needed alternatives to artemisinin, one day."
Following further studies, the candidate drug is expected to be ready to enter clinical trials.
Based on current progress, the scientists hope to have selected a candidate within a year.
Dr Kevin Read, another leader of the DDU team, said: "Malaria is a debilitating, often fatal parasitic disease that kills around one million people each year, mostly children under the age of five living in sub-Saharan Africa.
"Our compounds give hope that safe, affordable, new medicines to fight malaria will be ready to replace current drug treatments that are becoming ineffective due to the spread of drug resistance."
MMV is a foundation registered in Switzerland in 1999. Its mission is to discover, develop and facilitate delivery of new, effective and affordable anti-malaria drugs.
Dundee University researchers have reported "rapid progress" in efforts to develop a new treatment for malaria.