The city's council is in talks with Swansea City football club, Ospreys rugby region and Swansea University about launching the sporting hub.
It is understood a shortlist of possible locations has been drawn up but no decision has yet been made.
As well as bringing through future sporting stars, the public would also be able to use the facilities.
Sports will include football, rugby, swimming, cycling, hockey and boxing.
"We're talking about encouraging children to take up sports at an early age and excel in their abilities," said Swansea council leader Rob Stewart.
"It will be a full academy of sport.
"It will be linking into the sports excellence programme as well as the academies of the Ospreys and the Swans, bringing through the next generation heroes of the city.
"And it will have the state of the art facilities for the people of the city, and the people of Swansea would get the best access to the facilities."
Mr Stewart added the Swans and the Ospreys were "on board" and were keen to progress the scheme quickly.
Swansea could become the home for a new international sports academy and national athletes village.