Cafes will be introduced to the site as well as more green space which has been lacking since the square was created from gardens in the 1990s.
The city's council wants to breathe new life into the square and has listened to ideas from the public.
Leader Rob Stewart said it was important to get the scheme right.
"At the moment, there isn't really a reason to be in this square unless you're sitting here or walking through it," he said.
"What we want is make this a destination in itself because it's a really important part of the city."
Mr Stewart said the public made it known "loud and clear" they wanted gardens to form part of the revamp.
Provisional drawings show the square opened up with two levels to it, incorporating cafes, with grass surrounding much of the area.
The main fountains are due to be ripped out and replaced with a smaller water feature.
If approved, it is hoped work would be completed by 2018.
The square is in the middle of the city centre which is set to see other regeneration schemes including the St David's and civic sites, Kingsway and Parc Tawe retail park.
Chris Holley, leader of the opposition, said: "My biggest concern is that we lose so much public space.
"I think that's the important bit - that we've got to maintain public space."
Swansea's "tired" Castle Square is to be revamped with the aim of making it a key focal point in the wider city centre regeneration.