It is definitely wrong that he was evaluated as an ordinary person, and I think that he just held back from the Hojo clan and concealed his talent or he was a late-boomer.
古来凡庸の人と評し来りしは必ず誤なるべく、北条氏を憚りて韜晦せし人か、さらずば大器晩成の人なりしかと覚え候。