Title: MLM with Global Co-occurrence. Abstract: When pre-training models with the objective of MLM (masked language modeling) on multilingual corpora, the model learns to refine different language spaces to overlap each other for forming isomorphic spaces by understanding structural similarities from local bidirectional information. Global co-occurrence information is the primary source of information available to all methods, which potentially gives additional structural similarities to the model. In this work, we push MLM pre-training further to leverage global co-occurrence information. The result is MLM-GC (MLM with Global Co-occurrence) pre-training that the model learns local bidirectional information from masking and global co-occurrence information from a log-bilinear regression. In our experiments, MLM-GC pre-training substantially outperforms MLM pre-training for 4 downstream multilingual/cross-lingual tasks and 1 additional monolingual task, showing the advantages of capturing embedding analogies.