Brain metastases (BM) are the most severe consequence of malignancy in the brain, resulting in substantial illness and death. The most common primary tumors that progress to BM are lung, breast, and melanoma. Historically, BM patients had poor clinical outcomes, with limited treatment options including surgery, stereotactic radiation therapy (SRS), whole brain radiation therapy (WBRT), systemic therapy, and symptom control alone. Magnetic Resonance Imaging (MRI) is a valuable tool for detecting cerebral tumors, though it is not infallible, as cerebral matter is interchangeable. This study offers a novel method for categorizing differing brain tumors in this context. This research additionally presents a combination of optimization algorithms called the Hybrid Whale and Water Waves Optimization Algorithm (HybWWoA), which is used to identify features by reducing the size of recovered features. This algorithm combines whale optimization and water waves optimization. The categorization procedure is consequently carried out using a DenseNet algorithm. The suggested cancer categorization method is evaluated on a number of factors, including precision, specificity, and sensitivity. The final assessment findings showed that the suggested approach exceeded the authors’ expectations, with an F1-score of 97% and accuracy, precision, memory, and recollection of 92.1%, 98.5%, and 92.1%, respectively.