However, after the first full year since pension freedoms began, the Association of British Insurers (ABI) said most people were being sensible.
Well over half of those withdrawing money from their pots in the first three months of 2016 took out 1% or less of their value.
But at the same time 3,379 people took out more than 10% from their pensions.
Nevertheless that number is likely to include people who are investing the money elsewhere, or who may have multiple, small pension funds.
"Most people are taking a sensible approach," said Yvonne Braun, the ABI's director of policy, long term savings and protection.
"However, the data also suggests a minority are withdrawing too much too soon from their pension pot - 4% of pots are having a tenth or more withdrawn - and many other customers are taking their entire pot in one go."
The ABI wants the government to investigate why this is happening.
"This is a warning sign that requires further investigation. We need a full picture of these customers' circumstances and income," said Ms Braun.
A government spokesperson said keeping their annuity incomes was "the best option for the majority of people" but said that people should have the right to make the decision that is best for them.
"We are working with our partners, such as Pension Wise and the Department for Work and Pensions, to ensure consumers are protected and that there is clear information to help people understand their options," they added.
The so-called pension freedoms began in April 2015.
Since then, anyone over the age of 55 has been free to withdraw as much as they like from their pension pots, subject to income tax.
The pensions minister at the time, Steve Webb, said people would be free to spend it on a Lamborghini if they wanted too.
But many other experts have warned about spending pension savings too quickly.
Pensions: Could you run out of money before you die?
Pension choice: Annuity or Drawdown?
Over the full year, there were 300,000 lump sum withdrawals from pension pots, averaging Â£14,500 each, according to the ABI.
There were also 1.03m drawdown payments, averaging Â£3,800.
A minority of those aged 55 and over are withdrawing too much from their pension pots, the industry has warned.