Wakefield Council said the city's market had "underperformed" and lost £193,000 a year since opening in 2008.
Proposals to sell the land at Union Street to the owners of nearby Trinity Walk shopping centre were approved at a cabinet meeting earlier.
Sovereign Land (Management) Ltd plans to develop the site with a cinema and restaurants.
The council hopes to relocate the outdoor market to the city's main precinct and the indoor market to The Ridings shopping centre.
It said the plan would "improve the vibrancy and viability of the city centre".
Councillor Denise Jeffery said: "We believe this is the right thing to do - for the market traders, for the city and for the people of the wider district.
"We are committed to creating a thriving and vibrant market to enhance Wakefield's shopping and leisure experience.
"Linking the two established shopping centres by relocating the market is a positive move."
The building of the market hall and its outdoor market was funded by the original developers of Trinity Walk at a cost of £3m.
The traders have been heavily subsidised since 2008.
A report to cabinet said selling the land and moving the markets would give the council a net capital receipt of £630,000.
Plans to knock down a multi-million pound market open for just six years have been approved.