The settlement on the southeastern shore of Xiamen Island (now part of Siming District) developed as a seaport under the Song, although legal foreign trade was restricted to nearby Quanzhou, which administered the area. In 1387, attacks by the "Japanese" or "dwarf" pirates—many of them actually disaffected Chinese—prompted the Ming to protect the harbor with the fortress that gave Xiamen its name. The Portuguese first reached Xiamen in 1541. After the fall of the Ming to the Qing in 1644, Southern Ming loyalists including Koxinga used Xiamen as a base from which to launch attacks against the invading Qing-aligned Han Bannermen from 1650 to 1660. In 1661, Koxinga drove the Dutch from Taiwan and moved his operations there. His base on Xiamen fell to a combined Qing and Dutch invasion in 1663. The East India Company traded extensively with the port, constructing a factory there in 1678. It was raised to the status of a subprefecture in 1680, but the taxes and other restrictions placed on traders compelled the British to relocate to Canton and Fuzhou the next year. Trade resumed in 1685 and continued until the imposition of the Canton System. However, despite the Canton system which restricted Western trade to the port of Guangzhou, the Spanish were still allowed to trade in Xiamen. However, the Spanish rarely used this privilege as Chinese traders would ship their goods from Xiamen to Manila and vice versa, which was more profitable for both sides.