Mr Yoshida died on Tuesday in a Tokyo hospital, a spokesman for plant operator Tepco said.
He had been suffering from oesophagal cancer, which Tepco said was not linked to his work at the plant.
Former Prime Minister Naoto Kan praised his "leadership and decisiveness".
Masao Yoshida, a Tokyo-educated nuclear engineer, was chief of the nuclear plant at the time of the disaster.
The tsunami swamped the plant, knocking out cooling systems to the reactors, which subsequently melted down.
Mr Yoshida remained on site with a small team to try to bring the plant under control.
On 12 March, he ignored an order from top-level officials to stop pumping sea water into one of the reactors - actions that were subsequently seen as having prevented the situation from getting even worse.
He later said of the first few days after the earthquake: "There were several instances when I thought we were all going to die here. I feared the plant was getting out of control and we would be finished.''
He was criticised, however, for not installing adequate measures against a tsunami while he headed up Tepco's nuclear facility management division, before the disaster.
Because of his illness, Masao Yoshida stepped down as plant chief in December 2011 - the same month Japan declared the facility stabilised in a "cold shut-down" state.
"He literally put his life at risk in dealing with the accident,'' Tepco President Naomi Hirose said. "We keep his wishes to our heart and do utmost for the reconstruction of Fukushima, which he tried to save at all cost.''
Tepco has said it does not believe his illness was linked to radiation exposure, saying the condition would normally take between five and 10 years to develop if radiation were to blame.
The disaster forced tens of thousands of people to evacuate from the area around the plant and it remains unclear when they can return.
Since the disaster, workers have been cooling reactors by flushing sea water through them - but this has led to a build-up of contaminated water which must be stored.
There have also been a number of leaks of radioactive water at the plant.
On Tuesday, Tepco said that concentrations of caesium-134 in groundwater had soared over the weekend. The company said it was investigating the source of the leak.
Masao Yoshida, the Fukushima nuclear chief who led efforts to stabilise the crippled plant after the March 2011 earthquake and tsunami, has died at the age of 58.