Ceremonies were held in areas hit by the magnitude 9.0 earthquake, which devastated whole towns and badly damaged a nuclear power plant.
At a service in Tokyo, Emperor Akihito said that the situation for many survivors remained "severe".
Earlier, Prime Minister Shinzo Abe announced a new rebuilding plan.
At 2.46pm local time - the moment the earthquake struck underwater near the eastern Miyagi prefecture in 2011- tsunami warning alarms rang out again. A national minute's silence followed.
According to Japan's National Police Agency, 15,891 people died in the tragedy, and another 2,584 remain missing. Many were lost when a large tsunami swept the eastern coast after the earthquake.
Some 230,000 people have yet to return to their home towns since the disaster.
At a ceremony in Tokyo, Emperor Akihito said the situation remained "severe" for many survivors.
"Many of the people who were affected by the disasters are still living under a difficult environment today," he said.
"I'm especially concerned about the health of the people who are getting older every year. It is important for all Japanese to put their hearts together and stay close by."
A rebuilding programme initiated by Japan's government expires next March.
On Tuesday, Mr Abe said much work remained to be done, and that many towns were still unable to offer permanent housing or jobs to their residents.
Mr Abe said a new five-year rebuilding programme will be announced by the summer, but he did not indicate how much money was being allocated for reconstruction.
Cities across Japan have marked the fourth anniversary of the earthquake and tsunami that left more than 18,000 people dead or missing.