Shinzo Abe demanded the release of a second hostage, purported to be Kenji Goto, shown in the video. Japan says it is trying to verify the footage.
On Tuesday, Islamic State (IS) said it would kill the two men unless Japan paid $200m (Â£130m) within 72 hours.
But IS now appears to be demanding an exchange of captives instead.
Mr Goto appears to say IS would swap him for Sajida al-Rishawi - an Iraqi militant held in Jordan.
After the latest tape, Mr Abe said Japan would not bow to terrorism.
"This act of terrorism is an outrageous and unacceptable act of violence," he said.
"I again strongly demand the immediate release of Mr Kenji Goto unharmed."
In a statement, President Barack Obama said the US "strongly condemns the brutal murder of Japanese citizen Haruna Yukawa by the terrorist group ISIL [or IS]".
Mr Obama said the US would stand "shoulder to shoulder'' with Japan.
The video apparently shows Mr Goto holding a picture of what appears to be the body of Mr Yukawa.
US National Security Council official Patrick Ventrell said the US was also trying to establish the authenticity of the video and was closely co-ordinating with Japan.
Japan's chief cabinet secretary Yoshihide Suga said the video image "showing a killed Japanese person" was "an abominable and unforgivable act of violence".
There is some uncertainty over whether the video released on Saturday actually comes from Islamic State.
Unlike similar previous videos, the latest recording does not carry the logo of any IS media arms, and consists of an audio message with a still image. Other official videos have shown moving footage.
Haruna Yukawa, 42, was seized by militants in August after going to Syria to set up a private military contracting company, according to reports.
Kenji Goto, 47, is a well-known freelance journalist and documentary film-maker who went to Syria in October, reportedly to try to get Mr Yukawa released.
In a video released on Tuesday, a masked man was shown standing over kneeling hostages named as Mr Yukawa and Mr Goto and demanding the ransom.
The previously announced ransom is the same amount as that pledged in non-military aid for countries fighting IS by Prime Minister Abe during a recent tour of the Middle East.
Who are Islamic State (IS)?
Japan's prime minister has condemned as "outrageous and unacceptable" a video declaring the murder of Japanese hostage Haruna Yukawa by Islamic State.