The 12th-century castle, around which grew the present town center, was founded by the occupying Moors. In 1361, Peter I of Castile and the dethroned Muhammed V signed the Pact of Casares, by which the Moorish King recuperated his throne, leaving Casares as part of the Nasrid dynasty. The town surrendered to the Catholic forces after the fall of Ronda in 1485 and was handed over to Rodrigo Ponce de León, Duke of Cádiz. Later during the Rebellion of the Moriscos, Rodrigo's descendant, the Duke of Arcos, accepted the surrender of the rebel Moriscos, the Moors who had "converted" to Christianity. Casares had taken an active part in the Morisco rebellion, put down by Don John of Austria. The town separated from Manilva in 1795, being granted the title of Villa. At a later period, Casares was the only town, apart from Cádiz, that the Napoleonic troops had not been able to take.