It was originally a kakiage type castle (a simple and small castle on a stamped mound of soil and rocks) called Yokoyama-jo Castle built by Yorikatsu SHIOMI who is said to have been a descendant of Nagakiyo OGASAWARA of the late Muromachi period.
もとは小笠原長清の末裔とされる塩見頼勝が、室町時代後期に同地に築城した横山城という掻上城であった。