Reading and arithmetic are core domains of academic achievement with marked impact on career opportunities and socioeconomic status. While associations between reading and arithmetic are well established, evidence on underlying mechanisms is inconclusive. The main goal of this study was to reevaluate the domain-specificity of established predictors and to enhance our understanding of the (co-)development of reading and arithmetic. In a sample of 885 German-speaking children, standard domain-specific predictors of reading and arithmetic were assessed before and/or at the onset of formal schooling. Reading and arithmetic skills were measured at the beginning and end of second grade. Latent variables were extracted for all relevant constructs: Grapheme-phoneme processing (phonological awareness, letter identification), RAN (RAN-objects, RAN-digits), number system knowledge (number identification, successor knowledge), and magnitude processing (non-symbolic and symbolic magnitude comparison), as well as the criterion measures reading and arithmetic. Four structural equation models tested distinct research questions. Grapheme-phoneme processing was a specific predictor of reading, and magnitude processing explained variance specific to arithmetic. RAN explained variance in both domains, and it explained variance in reading even after controlling for arithmetic. RAN and number system knowledge further explained variance in skills shared between reading and arithmetic. Reading and arithmetic entail domain-specific cognitive components, and they both require tight networks of visual, verbal, and semantic information, as reflected by RAN. This perspective provides a useful background to explain associations and dissociations between reading and arithmetic performance.