A Kyle Owens header which went in off Crues skipper Colin Coates for an own goal gave Ballymena the win which puts them just five points off the top.
Crusaders had led through a Jim Ervin own goal with Tony Kane levelling from the penalty spot.
Elsewhere, there were wins for Linfield, Cliftonville and Glentoran.
The match between Carrick Rangers and Glenavon ended in a 0-0 draw.
Linfield, now the only unbeaten team in the top flight now, are just two points behind Crusaders and will take over at the top if they win at Cliftonville on Tuesday night.
Ballymena's win was the eye-catching result of the day as it came against the team which hammered them 6-0 on the opening day of the season.
Former Ballymena winger David Cushley set up the opener as his low ball was eventually diverted in by defender Ervin for an own goal.
Ballymena, on a high after beating Linfield 4-1 in the League Cup on Tuesday night, hit back and equalised before half-time with a Kane penalty after Michael Gault's foul on Cathair Friel.
The hosts were reduced to 10 men when keeper Ross Glendinning was shown a straight red card for handling outside his area.
Instead of trying to hang on for a draw, David Jeffrey's United grabbed all three points as defender Coates could not keep out a 90th-minute header by Owens.
Linfield striker Andrew Waterworth scored his first league goal of the season to set the Blues on their way to a 4-0 home win over Ballinamallard United.
A free-kick by former Northern Ireland midfielder Sammy Clingan made it two with Mark Haughey and Paul Smyth adding further goals in the second half.
Gary Haveron celebrated his first win as Glentoran boss as a late Jonathan Smith penalty secured a 1-0 win away to Dungannon Swifts.
Carrick Rangers, under new manager Aaron Callaghan, ended a run of seven defeats in a row by drawing 0-0 at home to Glenavon who had defender Andrew Doyle red-carded for an off-the-ball incident 15 minutes from time.
Cliftonville made it four Premiership wins in five matches as goals by Daniel Hughes and skipper Jason McGuinness sealed a 2-0 home win over Ards.
In the late kick-off, Portadown lost for the seventh league game in a row as Brad Lyons of Coleraine scored the only goal at Shamrock Park.
Champions and leaders Crusaders suffered their first Premiership defeat of the season as 10-man Ballymena United snatched a late win.