Media playback is not supported on this device
Despite an encouraging first-half performance, Villa lost 1-0 at the Liberty Stadium to sit 12 points from safety with nine games remaining.
Federico Fernandez bundled the ball in to hand Villa a sixth defeat in a row.
"Once again it is frustrating to get nothing out of the game," the 49-year-old Frenchman said.
"Until mathematically it is not possible why should we give up?"
Off the field, Tom Fox stepped down from his role as chief executive at Villa Park this week, while sporting director Hendrik Almstadt also left the club.
There have been a number of changes recently, with ex-Football Association chairman David Bernstein becoming a director and former boss Brian Little taking an advisory role to the board.
"We showed as a team that we were not too much affected by the board stuff," added Garde. "We have to concentrate on our business and that is football and that's what we showed today.
"Today I'm quite pleased with the way that we played. When you've lost you're not happy, but the character that the players showed today we have to do every game."
Aston Villa will not give up on survival in the Premier League according to manager Remi Garde despite another defeat at Swansea.