Prime Minister Narendra Modi said the train would link Mumbai and Ahmedabad, cutting travel time on the route from eight hours to two.
The deal was one of a raft of agreements reached after talks between the two sides in Delhi.
The leaders of Asia's second and third largest economies also announced other areas of co-operation.
These include working on defence technology, and agreeing a memorandum of understanding on the peaceful use of nuclear energy.
The latter agreement is expected to allow Japan to export nuclear plant technologies to India.
Last week Mr Modi's cabinet cleared the $14.7bn (Â£9.6bn) cost of building the bullet train system.
The agreements with Japan came during a three-day visit to India by the Japanese Prime Minister, Shinzo Abe, which began on Friday.
Both countries are in territorial disputes with China, and their new accords may be seen by some as a reaction against China's growing influence in the region.
India has agreed to buy a high-speed bullet train from Japan, in an attempt to transform its creaking rail system.