Councillors agreed to back the closure of Kingswood Civic Centre in advance of a formal consultation on the issue.
Labour has criticised the move and point to a 1,675-name petition opposing the closure.
A council spokesman said it was "exploring the possible options" for utilising its office accommodation.
"We are gathering the evidence of how the office space, training rooms and other facilities at the Kingswood Civic Centre are used, as well as other sites including Badminton Road. This work is continuing," he added.
But Labour's Pat Rooney said Tories had "prejudged the issue" and "many will now conclude that Kingswood's closure is a done deal".
The Civic Centre was built in the 1970s and refurbished in 2011.
The council said there is no suggestion all frontline services will be removed from the building and a final decision has yet to be made over its future.
Conservative-controlled South Gloucestershire Council has voted in favour of closing a building it refurbished in 2011 at a cost of £6m.