When Hisamitsu raised an army and went to Edo in 1862, he accused Saigo of going to Edo without permission and exiled him to the Tokunoshima Island, and when he pardoned Saigo as the result of his sympathizers' pleas in 1864, Hisamitsu left his teeth marks on his pipe due to his bitter decision, as this anecdotes show, there had been a large disparity between them later.
文久2年の率兵上京（後述）時には、西郷の無断東上を責めて徳之島、のち沖永良部島に配流し、藩内有志の嘆願により元治元年（1864年）に西郷を赦免する際も、苦渋の余りくわえていた銀のキセルの吸い口に歯形を残したなどの逸話があるように、のちのちまで両者のあいだには齟齬があった。