Hirobumi ITO described Inoue as 'the most loyal person I've ever known' in a letter to Sanetsune TOKUDAIJI, and when Ito himself was doubled as the Minister of the Sovereign's Household to confront the conservatives in the imperial court, Ito had only Inoue from the close aides join the Imperial Household Ministry as the Chief Librarian.
伊藤博文は徳大寺実則あての書簡で井上を「忠実無二の者」と評し、宮中保守派との対決のために自ら宮内卿を兼ねた際にも自分の側近から井上だけを図書頭として宮内省入りさせる。