It is the eighth largest fine ever imposed by the Financial Conduct Authority (FCA).
The FCA said the bank's London and international operations had failed to look after their clients' investments.
BNY Mellon said it regreted its failure, but insisted it had now improved its policies.
The rules, improved after the Lehman Brothers collapse in 2008, ring fence clients' assets if a firm becomes insolvent.
When Lehman's collapsed, it was difficult to sort out which money belonged to the bank, and which to customers.
In this case, BNY Mellon failed to account properly for shares, bonds and other investments, and, according to the FCA, would have found it difficult to return them quickly to the right clients.
"The firms' failure to comply with our rules was particularly serious given the systemically important nature of the firms, and the fact that safeguarding assets is core to their business," said Georgina Philippou,acting director of enforcement and market oversight at the FCA.
And she warned other banks and wealth management firms to take note.
"Other firms with responsibility for client assets should take this as a further warning that there is no excuse for failing to safeguard client assets."
The Bank of New York Mellon Group is the world's largest global custody bank.
The FCA investigated the bank's activities between 2007 and 2013, when the London and international operations of BNY Mellon held client funds worth up to £1.3 trillion and £236 billion respectively.
The FCA said that the risks taken by the bank had been compounded by the stress in the financial markets at the time.
In reaction, BNY Mellon said the fine would be fully covered by its reserves, and the company remained financially robust throughout the period in question.
It said no client lost out as a result of the issues identified by the FCA.
"We have taken clear steps to put in place a framework of new and improved policies and operational procedures, as well as enhance our specialist resources," said a spokesperson.
It said it had been trusted by clients to look after their assets for 230 years, a trust that could not have been earned without "robust regulatory compliance".
The Bank of New York Mellon's London branch has been fined £126m by the City regulator for failing to protect its customers' assets.