Smith, 61, quit his role as head coach on Tuesday after 10 months in the job, despite signing a new deal in October.
His former assistant Stuart Dickens has been put in charge of the team for Friday's game against St Helens.
"By his manner you could tell this wasn't a decision that was going to get changed," Carter told BBC Radio Leeds.
"We've had a frosty relationship since Christmas and those reasons will stay with me. He was the highest paid coach we've ever had and it's the highest playing budget we've ever had here.
"Quitting is not an option for some of us and we have to live with the hand we were dealt."
Smith joined the club in May 2015 and helped them retain their Super League status by beating Bradford in the Million Pound Game.
But the Wildcats have won only one of their five Super League matches this term and are 10th in the table.
Smith criticised the effect the schedule was having on his squad before leaving his post.
"From my point of view I've tried to back him to the hilt with everything he's wanted, but I guess at the end of the day it was never enough," Carter added.
"I want somebody in as quickly as possible. We need stability and we need to start to focus on winning games."
Wakefield Trinity Wildcats chairman Michael Carter has admitted he had a "frosty" relationship with former coach Brian Smith.