The way he is depicted in stories varies according to the era and people's sense of values; from a political point of view, he is regarded as a person with a fine character and distinguished in both literary and military arts, devoted himself to peace; from a view point of genuine swordsmanship, he is regarded as a skillful swordsman but a little inferior to other contemporary great swordsmen, his father, his son Jubei, or his nephew Toshiyoshi YAGYU (Hyogonosuke); and he is also described as an evil individual who distorted swordsmanship by politics.
また時代や価値観によって描かれ方が異なり、政治的な視点を通して見ると文武両道の太平を望み尽力した人格者として描かれることがあるが、純粋に剣術的な視点からの描き方だと達人ではあるが同時代の剣豪や父、息子・十兵衛、甥の柳生利厳（兵庫助）には一手及ばず、政を以って剣を歪めた悪役として描かれる事もある。