The square, which was developed from a gardens in the 1990s, is considered "tired" and councillors want to make changes to make it more vibrant.
Swansea council has now published a public open spaces notice giving people the chance to give their views.
Council leader Rob Stewart said it could become a "thriving, regenerated city centre".
The public can put their views forward until Thursday, 18 August.
People are being encouraged to have their say on plans to redevelop Swansea's Castle Square.