Although Tōyama remained a private citizen all his life, he was known as the "Shadow Shogun," "Spymaster," and "The Boss of Bosses," because of his tremendous covert influence on the nationalist politics and the "yakuza" crime syndicates. He also wrote an influential book on the "Three Shu" (Katsu Kaishu, Takahashi Deishu, and Yamaoka Tesshu). Despite his ultranationalism, Tōyama was paradoxically on good terms with Onisaburo Deguchi, Japan's most fervent pacifist. Tōyama was a charismatic, complex, and controversial figure in his lifetime, and remains so to this day.