In 1398, the fort came under the control of the Tomars. The most distinguished of the Tomar rulers was Maan Singh, who commissioned several monuments within the fort. The Delhi Sultan Sikander Lodi tried to capture the fort in 1505 but was unsuccessful. Another attack, by his son Ibrahim Lodi in 1516, resulted in Maan Singh's death. The Tomars ultimately surrendered the fort to the Delhi Sultanate after a year-long siege. During the last period of the Afghan Lodi dynasty, the fort was under the control of Maharaja Kamdev Singh who was a Sakarwar Rajput king. After the defeat in Battle of Khanwa, Kamdev came to Ghazipur.