The Nikkei 225 closed down 5.4% or 918.86 points at 16,085.44. It was its worst one-day fall since mid-2013.
Earlier, US and European shares were pulled down by big falls among banking and technology stocks. Deutsche Bank led the fallers, dropping 9.5%.
The Dow Jones dropped 1.1%, London's FTSE 100 fell 2.7% and the Athens stock exchange dropped to a 25-year low.
Investors have been spooked by weak oil prices, with Brent crude hovering around $33 a barrel, concerns over slowing economic growth, particularly in China, the world's second largest economy, and the threat of a further interest rate rise in the US.
In Tokyo, the banks were the biggest fallers, with Mitsubishi UFJ dropping 8.3%, Sumitomo Mitsui down 8.7% and Mizuho Financial Group losing 5.8%.
The strengthening yen also hit big exporters, with the currency up 0.8% against the dollar.
Toyota dropped 5.9%, Honda lost 6.4% and Nissan fell 6.8%.
The Nikkei 225 is down more than 20% from its peak levels, reached last June.
"When the strong yen is a concern, you would buy domestic-demand sensitive stocks like banks, but we can't buy them now so we are really struggling what to buy on a day like this," said Masashi Oda, senior investment officer at Sumitomo Mitsui Trust Bank.
Meanwhile, in Australia the ASX index closed down 2.8% at 4,832, led by the country's banking sector.
There was little trading elsewhere in Asia, with China's mainland markets shut all week for Lunar New Year celebrations. Hong Kong and South Korea are both closed for three days.
Tokyo shares fell sharply in Tuesday trading, following tumbling markets across Europe and the US.