The country said it was planning to spend 19.5bn yen (£140m) to create the AI Bridging Cloud computer (AIBC).
It hopes the AIBC will run at a speed of 130 petaflops - outpacing China's Sunway TaihuLight machine, which is currently the world's fastest.
The machine would be used to carry out research by analysing massive datasets, said Japan.
The first applications could be medical research or work to improve software to control driverless cars and other robots.
The projected computing power of the AIBC would be far faster than the 93 petaflops that the Sunway TaihuLight machine hits regularly. That machine has a theoretical maximum processing power of 125 petaflops.
One petaflop is equal to one thousand trillion operations per second. A "flop" (floating point operation) can be thought of as a step in a calculation.
"As far as we know, there is nothing out there that is as fast," said Satoshi Sekiguchi, head of Japan's ‎National Institute of Advanced Industrial Science and Technology, in a statement. The AIBC will be overseen by the institute.
It is not clear yet what processors will form the core of the machine. China decided to develop and build its own chips for the Sunway machine.
Japanese businesses will be able to buy time on the machine to help their own internal research efforts.
The government hopes this will stop them turning to US companies such as Amazon, Microsoft and Google when they are looking for a ready source of supercomputer power.
The project to build the AIBC is part of a larger government programme that aims to help Japan restore its reputation in the field of science and technology.
Japan has started a project to build the world's fastest supercomputer by the end of 2017.