To assess the prognostic value of convolutional neural networks (CNN) on coronary computed tomography angiography (CCTA) in comparison to conventional computed tomography (CT) reporting and clinical risk scores. 5468 patients who underwent CCTA with suspected coronary artery disease (CAD) were included. Primary endpoint was defined as a composite of all-cause death, myocardial infarction, unstable angina or late revascularization (> 90 days after CCTA). Early revascularization was additionally included as a training endpoint for the CNN algorithm. Cardiovascular risk stratification was based on Morise score and the extent of CAD (eoCAD) as assessed on CCTA. Semiautomatic post-processing was performed for vessel delineation and annotation of calcified and non-calcified plaque areas. Using a two-step training of a DenseNet-121 CNN the entire network was trained with the training endpoint, followed by training the feature layer with the primary endpoint. During a median follow-up of 7.2 years, the primary endpoint occurred in 334 patients. CNN showed an AUC of 0.631 ± 0.015 for prediction of the combined primary endpoint, while combining it with conventional CT and clinical risk scores showed an improvement of AUC from 0.646 ± 0.014 (based on eoCAD only) to 0.680 ± 0.015 (p < 0.0001) and from 0.619 ± 0.0149 (based on Morise Score only) to 0.6812 ± 0.0145 (p < 0.0001), respectively. In a stepwise model including all prediction methods, it was found an AUC of 0.680 ± 0.0148. CNN analysis showed to improve conventional CCTA-derived and clinical risk stratification when evaluating CCTA of patients with suspected CAD.