"Zatsu Zatsu Shui," a collection of Edo period anecdotesm mentions Yoriyuki as a bright child, and "Hosokawa Sansho Rykuden" includes a story of a competition in strength between Yoriyuki and his cousin Kiyouji HOSOKAWA.
江戸期の逸話集『雑々拾遺』に拠れば幼くして聡明さを見せ、また『細川三将略伝』に拠れば従兄弟の細川清氏と力比べをしたなどの幼少時の逸話がある。