A recent survey that determined heavy metal concentrations in an abandoned Hg mine in Palawan, Philippines, found the occurrence of Hg with As, Ba, Cd, Co, Cr, Cu, Fe, Mn, Ni, Pb, Sb, Tl, V, and Zn. While the Hg originated from the mine waste calcines, the critical knowledge about the origin of the other heavy metals remains unknown. This study assessed the ecological and health risks from heavy metal pollution surrounding the abandoned Hg mine. Principal component analysis (PCA) showed that the abandoned mine and natural sources (i.e., local geology) are the two main contributors of heavy metal pollution. Historically, the mine waste calcines (retorted ore) were used as construction material for the wharf and as land filler for the adjacent communities. There is highly strong ecological risk associated with the heavy metals: Ni, Hg, Cr, and Mn contribute 44.3%, 29.5%, 10.7%, and 8.9% to the potential ecological risk index (RI), respectively. Hazard index (HI) exceeded 1 for both adults and children in all the sampling locations, implying non-carcinogenic adverse effects. The total cancer risk over a lifetime (LCR) also exceeded the threshold limit of 10−4for both adults and children, contributed mainly by Cr (91.8%) and As (8.1%). By combining the results of the PCA and risk assessments, a clear link between heavy metal source apportionment to ecological and health risks was established. It was estimated that the abandoned mine contributed to most of the ecological and health risks for people living near the wharf that was built using the calcine, as well as the nearby Honda Bay. The findings of this study are expected to help policy makers develop regulations that will safeguard the ecosystem and the general public from the damaging impacts of heavy metals from the abandoned mine.