Emily Diamond, Laviai Nielsen, Eilidh Doyle and Olympic gold medallist Christine Ohuruogu combined for a time of three minutes 24.36 seconds to finish fourth behind Jamaica.
USA (3:24.36) took gold with Poland finishing second in Sunday's race.
The GB men's team were sixth in their 4x400m final.
The British women's team were using the event to trial new tactics and give younger athletes opportunities before the World Championships in London in August.
"I think we can be excited about the team going into London," said Ohuruogu. "It shows what a good squad we have going forward."
Great Britain's 4x400m women missed out on a medal by 0.23 seconds on the final night of the World Relays in the Bahamas.