The Addicks lost 4-3 at Shrewsbury on Tuesday, their third straight defeat, despite a Ricky Holmes hat-trick.
It left them 15th in League One, just six points above the bottom four.
"Too many people have got away with things for too long," said Robinson, who took charge in November after leaving former club MK Dons.
"For too long too many people have dodged bullets, and the manager has always been the one who gets it in the neck," the 36-year-old continued.
"I have to take responsibility as well, because I am the manger, and so do my staff, but ultimately we are not going on the pitch. We've not won enough games on the pitch.
"It can't be [down to] six, seven, managers. It can't be the lack of funds that have gone in. At some stage players have to stand up and be counted."
Robinson said the defeat confirmed what he already knew about his squad.
And he now wants to build a new team in his own identity, "with everyone singing from the same hymn sheet".
He added: "We are going through a transition period, we are looking forward to next season in some ways.
"It is not about short-term results, but long-term success."
Meanwhile, forward Stephy Mavididi has returned to parent club Arsenal after a scan revealed a hamstring injury will keep him out for three months.
Midfielder Jake Forster-Caskey could be sidelined for a month with a hamstring problem and defender Roger Johnson will miss the rest of the season with an injury picked up in training.
Charlton manager Karl Robinson says there are "deep" problems in the team, and that too many players "don't deserve" to wear the red shirt.