AbstractSerial MRI is an essential assessment tool in prostate cancer (PCa) patients enrolled on active surveillance (AS). However, it has only moderate sensitivity for predicting histopathological tumour progression at follow-up, which is in part due to the subjective nature of its clinical reporting and variation among centres and readers. In this study, we used a long short-term memory (LSTM) recurrent neural network (RNN) to develop a time series radiomics (TSR) predictive model that analysed longitudinal changes in tumour-derived radiomic features across 297 scans from 76 AS patients, 28 with histopathological PCa progression and 48 with stable disease. Using leave-one-out cross-validation (LOOCV), we found that an LSTM-based model combining TSR and serial PSA density (AUC 0.86 [95% CI: 0.78–0.94]) significantly outperformed a model combining conventional delta-radiomics and delta-PSA density (0.75 [0.64–0.87];p= 0.048) and achieved comparable performance to expert-performed serial MRI analysis using the Prostate Cancer Radiologic Estimation of Change in Sequential Evaluation (PRECISE) scoring system (0.84 [0.76–0.93];p= 0.710). The proposed TSR framework, therefore, offers a feasible quantitative tool for standardising serial MRI assessment in PCa AS. It also presents a novel methodological approach to serial image analysis that can be used to support clinical decision-making in multiple scenarios, from continuous disease monitoring to treatment response evaluation.Key Points•LSTM RNN can be used to predict the outcome of PCa AS using time series changes in tumour-derived radiomic features and PSA density.•Using all available TSR features and serial PSA density yields a significantly better predictive performance compared to using just two time points within the delta-radiomics framework.•The concept of TSR can be applied to other clinical scenarios involving serial imaging, setting out a new field in AI-driven radiology research.Supplementary InformationThe online version contains supplementary material available at 10.1007/s00330-023-09438-x.