Although he is often described as a hopelessly violent villain in novels and TV dramas he also had open-hearted aspects which can be seen in such cases as: when Shinsen-gumi members had to borrow Montsuki kimonos from YAGI family upon visiting Katamori MATSUDAIRA, the Head of Aizu Clan, for a petition Gennnojo YAGI worried that they would all be bearing the same family crest (an extremely ridiculous situation at a public occasion), but SERIZAWA laughed, not minding at all; or when SERIZAWA came secretly to return a hibachi he borrowed from YAGI family, YAGI questioned him about cuts made by swords on the hibachi (when the members got drunk, they tested out their swords on randomly selected furniture that belonged to the YAGI family), but SERIZAWA said "it's me, it's me" and scratched his head and escaped.
小説やテレビドラマでは手のつけられない凶暴な悪漢のように描かれることが多いが、会津藩主松平容保へ嘆願に行く時に八木家から紋付を借りることになり、全員同じ家紋になってしまうと八木源之丞が心配すると（公式の場では、かなり滑稽）、芹沢はまったく意に介せず笑っていたり、八木家から借りた火鉢をこっそり返しに来て、火鉢に刀傷があったので問いただしたら（隊士たちは酔って八木家の家財を手当たり次第に試し切りの材料にしていた）、「俺だ、俺だ」と頭をかいて逃げてしまうなど気さくな一面もあった。