He left Anjirō to lead the small congregation that they had managed to gather in Kagoshima. After Xavier returned from his unsuccessful endeavour in Kyoto, during which he realized both the Emperor of Japan and the Ashikaga shogun were powerless during the chaotic Sengoku period and their conversion would achieve nothing, he was recalled to Goa in 1552 and set his sights on China, dying on Shangchuan Island off its coast in the same year.
After Xavier's departure from Japan, Anjirō's mission in Kagoshima became deserted in just two years, and Anjirō was driven out by religious persecution. He descended into piracy and died a wakō pirate on a raid in China.