UK government borrowing fell to £10.13bn in May, the Office for National Statistics (ONS) said, down from £12.35bn a year earlier.
It was the lowest borrowing figure in May for eight years.
Public sector net debt excluding public sector banks now stands at £1.5 trillion, the ONS said, which is 80.8% of gross domestic product (GDP).
"While the deficit in the financial year ending 2015 has fallen by more than a third since its peak in the financial year ending 2010, public sector net debt has maintained a gradual upward trend," the ONS said in a statement.
Income tax receipts recorded their highest level for May in four years, rising £0.5bn, or 5.3%, from a year earlier to £10.8bn. VAT receipts rose by £0.6bn, or 5.6%, to £10.7bn.
The ONS also said that it now estimated total public sector borrowing in the financial year to March 2015 was £89.2bn, or 4.9% of GDP.
While this figure was slightly higher than the previous estimate, it was still £9.3bn lower than the previous year's total.
Analysts said the drop in government borrowing during May was good news for Chancellor George Osborne at the start of the new fiscal year.
"The chancellor is currently on track to significantly undershoot his fiscal targets for 2015-16", said Howard Archer, chief UK economist at IHG Global Insight.
But, he added: "Major questions remain ahead of the extra Budget on 8 July over the new Conservative government's ability to meet its ambitious fiscal targets over the longer term.
"In particular, the government is yet to clarify exactly where the planned cuts to departmental spending (£30bn) and welfare spending (£12bn) will be made."
Last week, the chancellor said he would attempt to bind future governments to maintaining a budget surplus when the economy is growing.
Experts say that the rise in public sector debt above £1.5 trillion will be troubling for Mr Osborne.
"The last public finance data before the Budget emphasised the scale of the challenges the chancellor still faces," said John Hawksworth, chief economist at PwC.
He estimated that the government would need to cut £40bn in unprotected departments over the next three years in order to achieve the chancellor's objective of eliminating the budget deficit by 2019.
A rise in income tax and VAT receipts helped to cut UK government borrowing in May, official figures have shown.