He seems to have been warned about his behavior by Kondo since he was a student at the Shieikan, and there is a description that Kondo gradually kept the ill-behaved Todo at a distance.
試衛館時代から近藤に品行の事でいつも注意されていたようで、近藤はだんだん品行の悪い藤堂を疎外していったとされている記述もある。