The government announced compensation packages in December for season ticket holders hit by long-running disruption.
About 84,000 passengers are eligible for refunds compared to an estimated 590,000 who are not, according to the London Assembly's Labour group.
A Labour spokesperson said it was "damned insulting" to exclude passengers who pay as they go.
Transport spokesperson Florence Eshalomi said "Not only have Southern passengers endured the worst services in the country, they've seen their fares go up.
"If the government wants to avoid sticking the boot in anymore, it should be compensating all passengers."
The compensation scheme was announced after months of disruption to commuters on the Southern rail network in south London and southern England.
Annual season ticket holders will receive a payment equivalent to one month's travel, with quarterly, monthly and weekly season ticket holders able to claim an equivalent payment for the ticket type.
But those who pay as they travel with cash or cards will receive no payment under the scheme.
According to official performance data, 29.5% of Southern's mainline and coast services were more than five minutes late last year - almost three in every 10 services.
The Department for Transport said the disruption was caused by track failures, engineering works, "unacceptably poor performance" by the operator and a series of strikes.
Govia Thameslink Rail (GTR), which runs the rail line, apologised for "many months of disruption and misery".
Almost nine out of every 10 Southern rail passengers will not be compensated for poor service, it has been claimed.