Being born before 34 weeks of pregnancy is linked to a host of health problems.
The study,published in the Lancet, showed that using a "cervical pessary" reduced the rate in the at-risk group.
Doctors said more studies were needed before the technique was used routinely.
The authors said 13 million babies were born prematurely every year.
In the trial, doctors were looking at women who had a cervix - part of the lower section of the uterus - which was shorter than 25mm. These pregnant women are thought to be at a higher risk of an early delivery.
The cervix was measured between 18 and 22 weeks into the pregnancy by an ultrasound scan. Of the 11,875 women who took part in the trial, 726 had a cervical length less than 25mm. Half of these women had a pessary, a small ring of silicone, inserted into their cervix.
In the group of women without the pessary, 27% of babies were born prematurely. The rate was six per cent among those fitted with a pessary.
Maria Goya, one of the researchers at the Vall d'Hebron Hospital, said: "Placement of a pessary is an affordable procedure, non-invasive, and easy to insert and remove as required."
The study concluded the pessary was a "reliable alternative for prevention of preterm birth" in a group of at-risk women.
Prof Steve Thornton of the University of Exeter, a spokesman for the Royal College of Obstetricians and Gynaecologists, said: "The difference in the two groups is pretty amazing."
He said more research was needed to prove that it worked, and to find out if it could help other women at risk of a premature birth.
"If this is borne out it could make a big difference," he added.
Drs Steve Caritis and Hyagriv Simhan, of the Magee Women's Hospital in Pittsburgh, said the findings had "renewed enthusiasm for the cervix as a therapeutic target" in preventing premature births.
However, they warned that few women had a short cervical length, which made "this screening approach fairly inefficient".
They added: "Additional well-designed studies are needed before pessary use can be validated as an effective treatment."
A cheap medical device can dramatically reduce the number of premature births in some at-risk women, according to a team of doctors in Spain.