Throughout the pandemic era, COVID-19 was one of the remarkable unexpected situations over the past few years, but with the decentralization and globalization of efforts and knowledge, a successful vaccine-based control strategy was efficiently designed and applied worldwide. On the other hand, excused confusion and hesitation have widely impacted public health. This paper aims to reduce COVID-19 vaccine hesitancy taking into consideration the patient’s medical history. The dataset used in this study is the Vaccine Adverse Event Reporting System (VAERS) dataset which was created as a corporation between the Food and Drug Administration (FDA) and Centers for Disease Control and Prevention (CDC) to gather reported side effects that may be caused by PFIEZER, JANSSEN, and MODERNA vaccines. In this paper, a Deep Learning (DL) model has been developed to identify the relationship between a certain type of COVID-19 vaccine (i.e. PFIEZER, JANSSEN, and MODERNA) and the adverse reactions that may occur in vaccinated patients. The adverse reactions under study are the recovery condition, possibility to be hospitalized, and death status. In the first phase of the proposed model, the dataset has been pre-proceesed, while in the second phase, the Pigeon swarm optimization algorithm is used to optimally select the most promising features that affect the performance of the proposed model. The patient’s status after vaccination dataset is grouped into three target classes (Death, Hospitalized, and Recovered). In the third phase, Recurrent Neural Network (RNN) is implemented for both each vaccine type and each target class. The results show that the proposed model gives the highest accuracy scores which are 96.031% for the Death target class in the case of PFIEZER vaccination. While in JANSSEN vaccination, the Hospitalized target class has shown the highest performance with an accuracy of 94.7%. Finally, the model has the best performance for the Recovered target class in MODERNA vaccination with an accuracy of 97.794%. Based on the accuracy and the Wilcoxon Signed Rank test, we can conclude that the proposed model is promising for identifying the relationship between the side effects of COVID-19 vaccines and the patient’s status after vaccination. The study displayed that certain side effects were increased in patients according to the type of COVID-19 vaccines. Side effects related to CNS and hemopoietic systems demonstrated high values in all studied COVID-19 vaccines. In the frame of precision medicine, these findings can support the medical staff to select the best COVID-19 vaccine based on the medical history of the patient.