Rivers burst their banks and flood water destroyed buildings and roads after more than 50cm (20in) of torrential rain fell within 12 hours on Wednesday.
It happened on the south-western island Kyushu.
Fears are growing for 20 people who are missing, including a small child who was swept down a fast-flowing river.
More than 7,000 rescue workers, including police officers and soldiers, have been sent to the area. The government said 250 people so far had been rescued.
"We are in an extremely serious situation," said the country's Deputy Prime Minister Taro Aso, warning that hillsides might collapse.
About 400,000 people have been ordered to leave their homes, mainly in Fukuoka and Oita prefectures.
Two people have died in floods in Japan that have forced hundreds of thousands of people to leave their homes.