The regulator said Vodafone had misled pay-as-you-go customers, charging them for top-up credit but "providing nothing in return".
It also found Vodafone had broken the rules on handling customer complaints.
Vodafone offered its "profound apologies" and said it was "determined to put everything right".
The fine stems from two earlier investigations into Vodafone, which has 20 million mobile customers in the UK.
One found that 10,452 pay-as-you-go customers lost out when Vodafone failed to credit their accounts after they paid to top up their mobile phone credit.
The affected customers collectively lost £150,000 over a 17-month period, Ofcom said.
The problems were caused by errors linked to the company's move to a new billing system.
However, Vodafone "failed to act quickly enough to identify or address these problems" and only moved to fix the issue after Ofcom intervened, the regulator said.
A second investigation found that Vodafone's customer service agents were not given "sufficiently clear guidance" on what constituted a customer complaint.
Moreover, poor processes meant some complaints were not handled "in a fair, timely manner".
The firm also failed to ensure customers were told, in writing, of their right to take an unresolved complaint to a third-party resolution scheme after eight weeks.
In a statement, Vodafone said it had "fully refunded or re-credited" 10,422 pay-as-you-go customers out of the 10,452 affected. It said it was unable to track down the remaining 30 affected.
It also said it had invested in better customer service and training.
"Everyone who works for us is expected to do their utmost to meet our customers' needs," it said.
"It is clear from Ofcom's findings that we did not do that often enough or well enough on a number of occasions. We offer our profound apologies to anyone affected by these errors."
Lindsey Fussell, Ofcom Consumer Group director, said: "Vodafone's failings were serious and unacceptable, and these fines send a clear warning to all telecoms companies.
"Phone services are a vital part of people's lives, and we expect all customers to be treated fairly and in good faith."
Regulator Ofcom has fined Vodafone £4.6m for "serious" breaches of consumer protection rules, its largest fine to date for a telecoms operator.