The OMG dataset: An Open MetaGenomic corpus for mixed-modality genomic language modeling
Under review as a conference paper at ICLR 2025

The OMG dataset: An Open MetaGenomic
corpus for mixed-modality genomic language
modeling

Anonymous authors
Paper under double-blind review

Abstract

Biological language model performance depends heavily on pretraining data
quality, diversity, and size. While metagenomic datasets feature enor-
mous biological diversity, their utilization as pretraining data has been
limited due to challenges in data accessibility, quality filtering and dedu-
plication. Here, we present the Open MetaGenomic (OMG) corpus, a ge-
nomic pretraining dataset totalling 3.1T base pairs and 3.3B protein cod-
ing sequences, obtained by combining two largest metagenomic dataset
repositories (JGI’s IMG and EMBL’s MGnify). We first document the
composition of the dataset and describe the quality filtering steps taken
to remove poor quality data. We make the OMG corpus available as a
mixed-modality genomic sequence dataset that represents multi-gene en-
coding genomic sequences with translated amino acids for protein cod-
ing sequences, and nucleic acids for intergenic sequences. We train the
first mixed-modality genomic language model (gLM2) that leverages ge-
nomic context information to learn robust functional representations, as
well as coevolutionary signals in protein-protein interfaces and genomic
regulatory syntax. Furthermore, we show that deduplication in embedding
space can be used to balance the corpus, demonstrating improved perfor-
mance on downstream tasks. The OMG dataset is publicly hosted on the
Hugging Face Hub at UrlHiddenForAnonymity and gLM2 is available at
UrlHiddenForAnonymity.

1

Introduction

Biological language models present an effective avenue for leveraging large amounts of un-
structured sequence data and learn functionally meaningful representations. Similar to
natural language processing (NLP) models (Touvron et al., 2023; Dodge et al., 2021), the
quality and diversity of pretraining data dictate the behavior and performance of biolog-
ical language models (Ding & Steinhardt, 2024). To date, the most widely used datasets
for biological language models (Hayes et al., 2024; Lin et al., 2023; Madani et al., 2023;
Nguyen et al., 2024) are derived from curated data repositories such as UniProt (UniProt
Consortium, 2019), UniRef (Suzek et al., 2007) and GTDB (Parks et al., 2022). However,
biological sequence diversity is immense and the above-mentioned data repositories cover
only a small fraction of the full sequence diversity found in nature. In order for biological
language models to improve, the size and diversity of pretraining data must also scale with
the size of the model.

Metagenomic sequences are partial genomic sequences derived from direct sequencing of
environmental (e.g.
soil, ocean) or biological samples (e.g. human skin, gut). Because
metagenomic sequencing circumvents the need for cultivation and isolation of biological
organisms, metagenomes typically feature sequences derived from uncultivated and novel
microorganisms (Tyson et al., 2004). These microbial genomes encode high levels of molec-
ular diversity and span previously unexplored branches of the tree of life (Hug et al., 2016).
Metagenomic datasets are unstructured by nature and a large fraction of the data is not
included in curated databases due to poor functional interpretability of these sequences. To

1

000

001
002
003
004
005

006
007
008
009
010
011

012
013
014
015
016
017

018
019
020
021
022
023

024
025
026
027
028

029
030
031
032
033
034

035
036
037
038
039
040

041
042
043
044
045
046

047
048
049
050
051

052
053

Under review as a conference paper at ICLR 2025

054

055
056
057
058
059

060
061
062
063
064
065

066
067
068
069
070
071

072
073
074
075
076
077

078
079
080
081
082

083
084
085
086
087
088

089
090
091
092
093
094

095
096
097
098
099
100

101
102
103
104
105

106
107

date, metagenomic sequences have not been fully utilized in biological language models due
to following limitations:

1. Metagenomic sequences are not readily downloadable in a single archive.
To date, the download of raw contigs (assembled genomic segments) from the two
main public repositories, Joint Genome Institute (JGI)’s IMG (Markowitz et al.,
2012) and European Molecular Biological Laboratory (EMBL)’s MGnify (Richard-
son et al., 2023), requires a large number of database queries and/or rate-limited
web API calls, as well as ad hoc approaches to robustly aggregate the results of
these queries into a single dataset.

2. Metagenomic sequences require extensive pre-processing. Raw metage-
nomically assembled contigs first undergo gene calling in order to identify protein
coding sequences and extract translated sequences. Additional quality filtering is
critical, as many metagenomes include poor or mis-assembled contigs.

3. Metagenomic sequences are difficult to deduplicate and balance. Like
most biological sequence datasets, metagenomes feature sampling biases (e.g. over-
representation of human gut microbiomes). Additionally, due to the lack of central-
ized databases for metagenomes, submissions of identical metagenomes to different
repositories result in duplicates. Unlike protein databases that can be deduplicated
and balanced using computationally efficient clustering algorithms (e.g. MMseqs2
(Steinegger & Söding, 2017)), clustering of a large dataset comprising genomic se-
quences of arbitrary region and length is computationally costly. Furthermore,
while curated genomic databases (e.g., GTDB (Parks et al., 2022) or BV-BRC (Ol-
son et al., 2023)) can be balanced with taxonomic labels, metagenomic sequences
rarely have taxonomic assignment, and ad-hoc assignment (e.g. Kraken (Wood &
Salzberg, 2014)) is computationally expensive and not always reliable.

Here, we document the collection and preprocessing steps of the OpenMetaGenome (OMG)
corpus. We then train the first mixed-modality genomic language model (gLM2) trained
on OMG, that leverages genomic context information to learn contextualized functional
representations of genomic elements. By training on mixed-modality data, gLM2 can per-
form both protein and DNA downstream tasks, and outperforms ESM2 (Lin et al., 2023)
on most protein tasks. Additionally, training on multi-protein contexts enables gLM2 to
predict protein-protein interfaces through co-evolutionary signal. Finally, we show that
embedding-based deduplication of the OMG dataset leads to improved functional represen-
tations, especially for underrepresented sequences.

2 Related Works

Pretraining corpora preprocessing in NLP. A number of previous studies have de-
veloped methods to improve the diversity and quality of pretraining corpora in NLP. For
instance, raw snapshots of Common Crawl (collection of webtext crawls) contain undesirable
data (e.g. hate speech, placeholder text). Studies have demonstrated that careful deduplica-
tion and rule-based filtering of Common Crawl (Dodge et al., 2021) improves overall model
performance (Penedo et al., 2024). More recently, efforts have been made to prune and bal-
ance pre-training data in semantic embedding space to achieve increased training efficiency
(Sorscher et al., 2022; Tirumala et al., 2023; Abbas et al., 2023). Dataset preprocessing
presents an important opportunity to minimize training resources, given the power-law na-
ture of LLM scaling (i.e. exponentially increasing compute requirement for diminishing
returns in performance improvement) (Hestness et al., 2017; Kaplan et al., 2020).

Biological sequence language models and their training datasets. Biological se-
quence language models are self-supervised models trained on discrete protein sequences or
genomic segments. Protein language models (pLMs) (Lin et al., 2023; Madani et al., 2023;
Elnaggar et al., 2022) are typically trained on high quality and curated publicly available
datasets such as UniRef (Suzek et al., 2007). UniRef is convenient for pLM training be-
cause it has been deduplicated using sequence similarity-based clustering (i.e. UniRef50
is deduplicated using 50% sequence identity). Previous efforts to increase the diversity of

2

Under review as a conference paper at ICLR 2025

108

109
110
111
112
113

114
115
116
117
118
119

120
121
122
123
124
125

126
127
128
129
130
131

132
133
134
135
136

137
138
139
140
141
142

143
144
145
146
147
148

149
150
151
152
153
154

155
156
157
158
159

160
161

Figure 1: (A) UMAP visualization of the OG dataset examples, colored by taxonomic
phylum, using embeddings from the 150M parameter gLM2 model. Distinct clusters form
for different phyla in embedding space. (B) Semantic deduplication of the OG dataset, with
pruned points highlighted in blue. Deduplication primarily removes samples from dense
clusters corresponding to overrepresented phyla. We visualize the semantic deduplication
on OG dataset to highlight taxonomic phyla most heavily pruned, and apply the same
pruning process to the OMG dataset for model training. (C) Comparison of the OG and
OMG datasets using a random 0.1% subset of each. Notably, the metagenomic data (OMG)
exhibits higher diversity.

the pretraining data includes cluster-balanced sampling (e.g. UniRef50/D for ESM models
(Rives et al., 2021) and sequence identity-based clustering of compiled protein databases
beyond curated databases (e.g. BFD (Steinegger et al., 2019; Elnaggar et al., 2022)). Ge-
nomic language models (gLMs) are trained on genomic sequences chunked at predefined
length thresholds. Diversification efforts for genomic datasets include pretraining on MG-
nify’s metagenomic contigs (Hwang et al., 2024) and balancing efforts in genomic pretraining
datasets include taxonomy-aware sampling (Dalla-Torre et al., 2023; Nguyen et al., 2024) of
curated genomic databases such as RefSeq (Pruitt et al., 2014), IMG/VR (Camargo et al.,
2022), IMG/PR (Camargo et al., 2024) and GTDB (Parks et al., 2022).

In this study, we define metagenomic datasets as collections
Metagenomic datasets.
of genomic contigs (contiguous genomic segments) computationally assembled from either
short-read or long-read raw sequence libraries. Typically, metagenomic datasets are se-
quenced from mixed community samples, which consist of multiple species, ranging from
hundreds to thousands of distinct species (Bahram et al., 2021). Complete genomes are
rarely obtained from metagenomic assemblies. Therefore, metagenomic assemblies require
extensive taxonomic profiling (Parks et al., 2021) and partial genome reconstruction through
contig clustering (i.e. binning). Because metagenomes are sequenced from diverse environ-
ments without the need for cultivation, their sequences feature the highest level of molecular
diversity amongst publicly available sequence datasets (Pavlopoulos et al., 2023). Metage-
nomic datasets also vary in quality depending on sequencing depth and sample type, where
low quality metagenomes feature computational assembly errors, short contig lengths, and
truncated protein sequences (Mende et al., 2012; Lai et al., 2022). Furthermore, while most
metagenomic datasets are predominantly analyzed with a focus on microbial (archaea, bac-
teria, viruses) communities, eukaryotic genomic material can comprise a substantial portion
of the raw library (West et al., 2018). Many standard metagenomic post-processing steps
(e.g. gene calling) fail on eukaryotic sequences, resulting in poor quality protein sequence
predictions. Critically, quality filtering and dataset deduplication of metagenomes require
domain-specific knowledge, yet there is little documentation of preprocessing steps needed
to make these datasets suitable for biological language model pretraining. While pretraining
on metagenomic datasets allows models to leverage rich molecular diversity and genomic
context, these models are most suitable for microbial genomes and may result in out-of-
distribution effects on eukaryotic sequences.

3

Under review as a conference paper at ICLR 2025

162

163
164
165
166
167

168
169
170
171
172
173

174
175
176
177
178
179

180
181
182
183
184
185

186
187
188
189
190

191
192
193
194
195
196

197
198
199
200
201
202

203
204
205
206
207
208

209
210
211
212
213

214
215

3 The Open MetaGenome corpus

Here, we document the construction of the OMG corpus. The OMG is a 3.1T base pair (bp)
pretraining dataset comprising EMBL’s MGnify database1 and JGI’s IMG database2. We
utilize the gene predictions conducted by the databases; the gene calling protocols for IMG
and MGnify are detailed in Huntemann et al. (2016) and Richardson et al. (2023) respec-
tively. The combined dataset is pre-processed into a mixed-modality dataset upon sequential
element-by-element quality-filtering (described in Section 3.1) . The mixed-modality dataset
of Open Metagenomes is made available as the OMG dataset (Fig. 1) containing 3.3 billion
protein coding sequences (CDS) (Tab. 1). We also make available a 10x smaller subset
of OMG that only consists of prokaryotic and viral genomes from INSDC3 as the Open
Genome mixed-modality dataset OG (Fig. 1, Appendix B). Finally, we make available a
protein-only dataset OMG_prot50, consisting of protein sequences derived from the OMG
dataset, clustered at 50% sequence identity (Appendix E). OMG_prot50 contains 207M
representative sequences from clusters with at least two members, representing >3-fold in-
crease in sequence diversity compared to UniRef50 (Suzek et al., 2007). All three datasets
are available for download from the Hugging Face Hub, and all dataset processing scripts
are available at UrlHiddenForAnonymity. As more metagenomic data becomes available,
we plan on regular updated releases of the corpus in the future.

Table 1: Statistics for the datasets made available in this study. CDS: Coding
sequences, IGS: Intergenic sequences. For reference, UniRef50 consists of 66M proteins.

# CDS # IGS Total # Contig Size Description

(bps)

(TB)

OMG

3.3B

2.8B 3.1T

271M 1.25

OG

0.4B

0.3B 0.4T

6.2M

0.16

OMG_prot50 207M

–

–

–

0.05

Filtered mixed-modality
ge-
nomic sequences featuring mul-
tiple protein coding genes (rep-
resented in AAs) interleaved
with intergenic sequences (rep-
resented in NAs).

the
IMG data
Fraction of
that
of prokaryotic
consist
genomes and associated taxo-
nomic metadata.

in
Protein coding sequences
AA,
50% se-
clustered at
quence identity. Singleton clus-
ters were removed from the
database. Clustering detail is
found in Appendix E

3.1 Dataset preprocessing

Multi-modal data processing. Metagenomic contigs often encode multiple genes on
either strand of the sequence. A genomic language model can be trained on raw nucleic
acid sequences (e.g. Evo (Nguyen et al., 2024), Nucleotide Transformers (Dalla-Torre et al.,
2023)) or by representing each genomic sequence as an order- and orientation-preserved list
of translated coding sequences in amino acids (e.g. (Hwang et al., 2024)). For the former
method, the context length needed to encode genomic sequences in nucleic acids can result

1Snapshot date 2022-11-23 (excluding all embargoed/restricted metagenomic samples, see

database statistics in Appendix A)

2Snapshot date 2023-08-27 (excluding all embargoed/restricted metagenomic samples and in-

cluding IMG genomes dataset derived from NCBI.)

3https://www.insdc.org, retrieved from IMG/M, metadata available in Appendix P

4

Under review as a conference paper at ICLR 2025

216

217
218
219
220
221

222
223
224
225
226
227

228
229
230
231
232
233

234
235
236
237
238
239

240
241
242
243
244

245
246
247
248
249
250

251
252
253
254
255
256

257
258
259
260
261
262

263
264
265
266
267

268
269

in unfeasibly large compute requirements. Furthermore, a recent study comparing nucleic
acid (NA) models against amino acid (AA) models on protein functional representations
demonstrated that NA may not be the most efficient input format for learning translated
protein functions (West-Roberts et al., 2024). The latter method, while benefiting from the
compressed sequence length and more expressive AA sequences for proteins, does not lever-
age the information stored in intergenic regions. These intergenic regions contain important,
yet, lesser characterized sequence patterns involved in transcription regulation and cellular
function such as ncRNA, microRNA, promoters, and transcription factor binding sites. We
developed a mixed-modality dataset that represents a genomic contig as a list of elements
where an element is either a coding sequence (CDS) or an intergenic sequence (IGS) (see
Fig. 2). CDS elements are represented in translated AA sequences and IGS elements are
represented in NA sequences. We also store the strand information (+/-) of CDS elements
and the order of all elements in the contig.

Edge-element removal. Metagenomic contigs are not complete genomic sequences,
therefore, both edges of the sequences are more likely to contain gene-calling errors.
In
our pre-processing, we remove edge CDS elements to address miscalled open reading frames
(ORFs) and fragmented protein sequences at the beginning and end of the metagenomic
contigs (Steinegger & Salzberg, 2020). Specifically, if a scaffold starts/ends with an inter-
rupted CDS, we remove that CDS element.
If a scaffold starts/ends with a non-coding
region, we remove the IGS element and the CDS adjacent to the IGS element. This filtering
step removes ~1.4B genomic elements likely to be poor quality, partial sequences with high
likelihood of assembly errors.

Contig length-based filtering and preprocessing. Assembly of shotgun metagenomic
libraries results in many short contigs that are often low in quality. To limit the impact of the
fragmented nature of metagenome assemblies, we first remove all metagenomic contigs that
are shorter than 2kb from the raw databases. Secondly, we enrich the corpus with contigs
that contain multiple genes by removing contigs that contain less than seven elements in total
or less than three CDS elements. Only contigs that meet the length requirement are added to
the dataset. In preprocessing these contigs into Hugging Face datasets (Lhoest et al., 2021),
we found that extremely large contigs resulted in process hanging errors and inefficient
storage. To address this issue, we chunk large contigs into 1000 elements. Appendix C
visualizes the distribution of contig length, as well as CDS and IGS element lengths.

Assembly quality (N/X-frequency) filtering. Due to the computational nature of
the metagenomic assembly, misassembled contigs comprise a nontrivial fraction of the data.
The quality of the assembly differs significantly across samples, depending on the biological
community composition, sample type, and sequencing depth (Vollmers et al., 2017; Lapidus
& Korobeynikov, 2021). Notably, the quality of assembly may vary across the contig, where
a section of the contig may contain assembly gaps due to shallow sequencing depth. One
way to determine poorly assembled sequences is by identifying the fraction of Ns (gaps or
ambiguous bases) in the raw DNA sequence (or Xs in the translated AA sequence). For
OMG, we process each contig sequentially element-by-element, and if an element comprises
>20% in invalid characters, we discard the element and start a new contig (Appendix. D).
Importantly, only contigs that meet the length requirement above are added to the dataset.
This sequential processing allows high quality regions of the contigs to be preserved, while
low quality stretches are discarded.

Element length-based filtering. A nontrivial portion of the metagenome can be eukary-
otic, however, most metagenomic gene-calling software tools are not optimized for eukaryotic
ORF prediction (Bruna et al., 2024). Additionally, metagenomes can contain sequences from
organisms that utilize alternative genetic codes (Borges et al., 2022; Cook et al., 2024), which
may not all be correctly predicted by common tools. A salient pattern observed for poor
gene prediction is low coding density, (i.e.
long stretches of IGS) or presence of very long
CDS sequences. To identify these, we process each contig sequentially element-by-element
and remove any CDS element >15,000 AAs or IGS element >4000 bps in length, and start
a new contig. These thresholds are designed to exclude regions of questionable gene calls,

5

Under review as a conference paper at ICLR 2025

such as long intergenic regions where no genes are predicted, and giant protein sequences,
which are prone to assembly errors and require careful curation to verify (West-Roberts
et al., 2023). This filtering step removes 2.5e-5% of CDS , and 1e-4% of IGS elements from
OMG.

Figure 2: Mixed-modality sequence processing and gLM2 masked language mod-
eling. A gene-called metagenomic contig is first preprocessed into a mixed-modality se-
quence consisting of CDS elements (blue) and IGS elements (grey). The mixed-modality
sequence then undergoes masking at 30% and gLM2 is trained with a masked token recon-
struction objective.

4 Experiments

4.1 GLM2: A Mixed-modality genomic language model

To showcase the efficacy of the OMG dataset for pretraining, we introduce gLM2: a
mixed-modality genomic language model pretrained on OMG. gLM2 learns contextualized
representations of genomic contigs, which are represented as sequences of CDS and IGS
elements. In order to tokenize the mixed-modality sequence, CDS elements are tokenized
using per-amino acid tokens, and IGS elements are tokenized using per-nucleotide tokens.
To distinguish strand orientation for CDS elements, we introduce two special tokens: <+>
and <->, which are prepended to each genomic element to indicate the positive and negative
strands, respectively.
gLM2 is trained using the masked language modeling objective,
where 30% of both CDS and IGS tokens are masked. Cross-entropy loss is applied only
on the masked tokens. gLM2 is trained at two scales: 150M and 650M parameters. Both
models are trained on the semantically deduplicated OMG dataset (Section 4.2) for 600k
steps. We train gLM2 using a context window of 4096 tokens to allow for multiple (9.7
± 3.3) CDS and IGS elements to appear in each example. For model architecture and
training hyperparameters, refer to Appendix F.

We benchmark gLM2 on the Diverse Genomic Embedding Benchmark (DGEB) (West-
Roberts et al., 2024). DGEB is a comprehensive benchmark that evaluates model represen-
tations across diverse taxa and 18 tasks representing multiple axes of biological function,
such as evolutionary distance similarity, remote homology prediction, enzyme classification,
and retrieval sensitivity.

6

270

271
272
273
274
275

276
277
278
279
280
281

282
283
284
285
286
287

288
289
290
291
292
293

294
295
296
297
298

299
300
301
302
303
304

305
306
307
308
309
310

311
312
313
314
315
316

317
318
319
320
321

322
323

Under review as a conference paper at ICLR 2025

4.2 OMG corpus balancing with genomic Semantic Deduplication

Biological datasets exhibit significant biases that can influence the performance and gener-
alizability of trained models (Ding & Steinhardt, 2024; West-Roberts et al., 2024). Unlike
protein databases, where short sequence lengths allow for clustering-based deduplication,
(meta)genomic sequences have highly variable lengths (Appendix C), making sequence-based
clustering challenging. To address this challenge, we perform deduplication in embedding
space by pruning examples with small cosine distance, following Semantic Deduplication
(SemDeDup) (Abbas et al., 2023). SemDeDup previously showed efficacy in removing se-
mantically similar examples over web-scale text and image datasets, demonstrating signifi-
cant speed up in convergence for downstream tasks.

For genomic semantic deduplication, we first trained a 150M gLM2 on the tokenized OMG
dataset for 600k steps. We then embed the entire OMG dataset, by extracting a mean-
pooled, per-example representation from the model’s last hidden layer. The example-level
embeddings correspond closely to the taxonomic classification available for the OG dataset
(Fig. 1A). This motivates embedding-based deduplication as a method for removing near
duplicates while balancing taxonomic bias. We prune the OMG dataset at 49% (i.e. 49%
of the original data is removed) at the deduplication threshold 2e-3 (where examples with
embeddings <2e-3 in cosine distance are deduplicated) (Appendix G). The pruned exam-
ples are saturated in highly dense clusters (Fig. 1B) which results in taxonomic balancing
(Appendix H) , measured by increased entropies of distribution across taxonomic levels (Ap-
pendix I). We then trained a 150M gLM2 on the pruned OMG dataset for an equal number of
steps, and compared its performance against the un-pruned version on DGEB. While prun-
ing results in a modest increase in the aggregate DGEB score (0.48 vs 0.47), we observe
improvements in tasks that feature underrepresented taxa (e.g. ArchRetrieval, RpoB Arch
phylogeny) (Appendix J). This improved performance for underrepresented taxa appears
to come at the cost of small regressions on tasks that are biased towards overrepresented
taxa. Genomic SemDeDup presents a tunable method for effectively pruning unstructured
genomic data without reliance on taxonomic labels.

Figure 3: Scaling performance on DGEB amino acid tasks for gLM2 and ESM2,
relative to pretraining floating point operations (FLOPs). gLM2_150M trained
with no data pruning is shown in black.

7

324

325
326
327
328
329

330
331
332
333
334
335

336
337
338
339
340
341

342
343
344
345
346
347

348
349
350
351
352

353
354
355
356
357
358

359
360
361
362
363
364

365
366
367
368
369
370

371
372
373
374
375

376
377

Under review as a conference paper at ICLR 2025

4.3 GLM2 performance on DGEB

We compare the performance of the 150M and 650M gLM2 models trained on the pruned
OMG dataset against the ESM2 series trained on the UniRef50/D dataset (Fig. 3). gLM2
In particular,
outperforms ESM2 on the overall DGEB score at each parameter scale.
gLM2’s performance scales with pretraining floating point operations (FLOPs) on protein
tasks where ESM2 plateaus in performance with scaling (i.e. Operon pair classification tasks,
ModAC paralogy task) (Appendix K). Such improved functional representation learning is
likely due to gLM2’s ability to leverage genomic context information, and thereby learn
relationships between genomic elements. gLM2, being a mixed-modality model, also learns
intergenic sequence representations. We compare gLM2’s performance on DGEB nucleic
acid (NA) tasks against the Nucleotide Transformer series (Appendix L). gLM2 performs
similarly on NA tasks when compared to Nucleotide Transformers, despite only a small
fraction of the training tokens consisting of DNA sequences.

Figure 4: gLM2 learns protein-protein interface co-evolutionary signal in the
2ONK (ModAC) complex. (A) ModA and ModC forms a structural complex with co-
evolutionary signal between residues (in yellow). (B) Co-evolutionary signal extracted from
multiple sequence alignment of 2ONK4(Ovchinnikov et al., 2014), calculated and visualized
using GREMLIN (PDB_benchmark_alignments/2ONK_A2ONK_C.fas). The region of inter-
protein co-evolutionary signals are highlighted with a red box. (C) Zoomed-in region of
inter-protein coevolutionary signal in B. (D) Categorical Jacobian calculated for Evo on
the DNA sequence encoding 2ONK_A and 2ONK_C (from 89,891 to 91,376 of genomic
sequence NC_000917.1). The L2 norm was computed over the (3,4,3,4) tensor for every
pair of codon positions to generate the contact map. (E) Categorical Jacobian calculated
for ESM2 650M on the concatenated 2ONK_A_2ONK_C sequence. No inter-protein co-
evolutionary signal is detected. (F) Categorical Jacobian calculated for gLM2_650M on
the concatenated 2ONK_A_2ONK_C sequence. (G) Zoomed-in region of inter-protein
coevolutionary signal in F.

4.4 GLM2 learns protein-protein interaction interfaces

We test gLM2’s ability to learn coevolutionary signals between proteins in protein-protein
interaction interfaces (Ovchinnikov et al., 2014). Previous studies have shown that pLMs
learn within-protein co-evolutionary information that can be extracted with a supervised
contact prediction head (Lin et al., 2023) using an unsupervised "categorical Jacobian" cal-
culation (Zhang et al., 2024). However, pLMs trained on individual proteins or protein
families cannot learn co-evolutionary information across proteins. We calculate the categor-

8

378

379
380
381
382
383

384
385
386
387
388
389

390
391
392
393
394
395

396
397
398
399
400
401

402
403
404
405
406

407
408
409
410
411
412

413
414
415
416
417
418

419
420
421
422
423
424

425
426
427
428
429

430
431

Under review as a conference paper at ICLR 2025

ical jacobian values from gLM2_650M on the concatenated sequence of 2ONK_A (ModA)
and 2ONK_C (ModC) (Appendix N). We demonstrate that gLM2 leverages multi-protein
context to learn protein-protein interfaces from a single concatenated sequence that closely
matches the co-evolutionary signal that can be learned from multiple sequence alignment
(MSA) based Potts model (GREMLIN (Kamisetty et al., 2013)) (Fig. 4). Such protein-
protein interface signals cannot be extracted in existing language model methods such as
ESM2 650M and Evo-1-8k-base (Fig. 4E and F). We validate the gLM2-predicted contacts
directly with the ground truth contacts from 2ONK PDB structure (Fig. 5), as well as 31
complexes previously described in (Ovchinnikov et al., 2014) (Appendix ??). The ability to
extract interacting residues without supervision nor MSA presents an opportunity to predict
novel protein-protein interactions from sequence information alone.

Figure 5: Ground truth comparisons of Jacobian-detected contacts against PDB
structures. (A) Left: Ground truth contacts derived from PDB structure (PDB: 2ONK;
ModAC complex) shown in Fig. 4, where contact is defined as residues that are within
<8Å. Middle: gLM2-predicted contacts using Categorical Jacobian. Right: Inter-protein
region highlighting top n highest scoring predicted contacts (red for true positive, blue for
false positive) overlaying ground truth contacts (gray), where n is the number of inter-
protein contacts identified in the ground truth. (B) Left: Ground truth contacts derived
from tRNA-Asp (PDB: 6UGG) shown in Fig. 6. Middle: gLM2-predicted contacts using
Categorical Jacobian. Right: Top n highest scoring contacts in gLM2 (red for true positive,
blue for false positive) overlaying ground truth contacts (gray), where n is the number of
contacts within tRNA identified in the PDB ground truth excluding the diagonal.

4.5 GLM2 learns regulatory syntax in intergenic DNA

We demonstrate gLM2’s ability to identify regulatory syntax and non protein-coding ele-
ments in IGS regions. We first validate gLM2’s ability to predict contacts in tRNA-Asp
against the ground truth 6UGG PDB structure (Fig. 5) We further demonstrate gLM2’s
ability to identify regulatory regions (sigma factor binding and terminator) in the genomic
context of tRNA-Asp (Fig. 6). We additionally observe a signal downstream of aspV and

4https://colab.research.google.com/github/sokrypton/GREMLIN_CPP/blob/master/

GREMLIN_TF.ipynb

9

432

433
434
435
436
437

438
439
440
441
442
443

444
445
446
447
448
449

450
451
452
453
454
455

456
457
458
459
460

461
462
463
464
465
466

467
468
469
470
471
472

473
474
475
476
477
478

479
480
481
482
483

484
485

Under review as a conference paper at ICLR 2025

uptream of the terminator region. This region lacks annotation in EcoCyc (Karp et al., 2023)
and presents the potential for gLM2-based unsupervised discovery of novel regulatory se-
quence motifs. We examined 23 additional intergenic regions in the E. coli K-12 genome that
contain at least one terminator and one promoter regions according to EcoCyc annotations.
We show conserved Categorical Jacobian patterns corresponding to previously validated
annotations across diverse regions of the genome (Appendix P). We further conducted a
similar analysis on B. subtilis 168 genomic region 119,848-120,978bp (5’->3’) containing a
L10 leader RNA gene and two ribosomal protein coding genes rplJ and rplL (Appendix O).
We observe putative contacts between the L10 leader RNA and ribosomal protein RplL,
an experimentally evidenced interaction (Johnsen et al., 1982). We also observe contacts
between RplJ and RplL, a known ribosomal protein complex. Furthermore, our analysis
highlights co-evolutionary signal between the Shine-Dalgarno sequences (ribosomal binding
site) upstream of rplJ and rplL, suggesting gLM2 understanding of genome-specific regula-
tory motifs.

Figure 6: gLM2 learns intergenic regulatory syntax and tRNA structure. We vi-
sualize co-evolutionary signal in E. coli K-12 substr. MG1655 chromosomal region 236,866-
237,087bp (5’->3’) containing aspV (tRNA-Asp encoding gene) using the Categorical Ja-
cobian. Structural signatures in tRNA-Asp sequence are visible. Other signals correspond
to known regulatory syntax including sigma factor binding sites (-35 and -10), transcription
initiation site (σ70 binding region), and rho-independent terminator sequence.

5 Conclusion

The OMG dataset is a large-scale mixed-modality biological pretraining corpus that lever-
ages the immense volume and diversity of unstructured metagenomic (primarily prokaryotic
and viral) sequences. We quality-filter and preprocess the raw metagenomic sequences into
a mixed-modality format ready for language model training. We showcase the efficacy of
mixed-modality input for genomic language modeling with gLM2. With genomic SemD-
eDup, we present an efficient method for reducing the bias and duplication in genomic
datasets. The gLM2 models trained on pruned OMG learn contextualized representations
for both CDS and IGS, and demonstrate efficient scaling and improved performance across
downstream tasks compared to uncontextualized protein language models trained on curated
databases. We further demonstrate the gLM2’s ability to learn protein-protein interfaces
at residue-level, paving the path towards unsupervised protein-protein complex prediction.
Finally, we show that gLM2 learns evolutionary couplings of regulatory motifs in the in-
tergenic DNA, indicating model understanding of both modalities of the data. The OMG
dataset and gLM2 models as well as the supporting code are publicly available for download.

10

486

487
488
489
490
491

492
493
494
495
496
497

498
499
500
501
502
503

504
505
506
507
508
509

510
511
512
513
514

515
516
517
518
519
520

521
522
523
524
525
526

527
528
529
530
531
532

533
534
535
536
537

538
539

Under review as a conference paper at ICLR 2025

Ethics Statement

This study aims to advance open science for genomics, by making the OMG corpus and gLM2
model publicly available on the HuggingFace Hub. The OMG corpus is constructed from
publicly available data within JGI’s IMG and EMBL’s MGnify repositories. We exclude
all embargoed and restricted data from the OMG corpus. As the data originates from
environmental samples, no personally identifiable information is associated with the dataset.

11

540

541
542
543
544
545

546
547
548
549
550
551

552
553
554
555
556
557

558
559
560
561
562
563

564
565
566
567
568

569
570
571
572
573
574

575
576
577
578
579
580

581
582
583
584
585
586

587
588
589
590
591

592
593

Under review as a conference paper at ICLR 2025

594

595
596
597
598
599

600
601
602
603
604
605

606
607
608
609
610
611

612
613
614
615
616
617

618
619
620
621
622

623
624
625
626
627
628

629
630
631
632
633
634

635
636
637
638
639
640

641
642
643
644
645

646
647

References

Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. SemDeDup:

Data-efficient learning at web-scale through semantic deduplication. March 2023.

Mohammad Bahram, Tarquin Netherway, Clémence Frioux, Pamela Ferretti, Luis Pedro
Coelho, Stefan Geisen, Peer Bork, and Falk Hildebrand. Metagenomic assessment of the
global diversity and distribution of bacteria and fungi. Environ. Microbiol., 23(1):316–326,
January 2021.

Adair L Borges, Yue Clare Lou, Rohan Sachdeva, Basem Al-Shayeb, Petar I Penev, Alexan-
der L Jaffe, Shufei Lei, Joanne M Santini, and Jillian F Banfield. Widespread stop-codon
recoding in bacteriophages may regulate translation of lytic genes. Nat Microbiol, 7(6):
918–927, June 2022.

Tomas Bruna, Alexandre Lomsadze, and Mark Borodovsky. A new gene finding tool
GeneMark-ETP significantly improves the accuracy of automatic annotation of large eu-
karyotic genomes. bioRxiv, April 2024.

Antonio Pedro Camargo, Stephen Nayfach, I-Min A Chen, Krishnaveni Palaniappan, Anna
Ratner, Ken Chu, Stephan J Ritter, T B K Reddy, Supratim Mukherjee, Frederik Schulz,
Lee Call, Russell Y Neches, Tanja Woyke, Natalia N Ivanova, Emiley A Eloe-Fadrosh,
Nikos C Kyrpides, and Simon Roux. IMG/VR v4: an expanded database of uncultivated
virus genomes within a framework of extensive functional, taxonomic, and ecological
metadata. Nucleic Acids Research, 51(D1):D733–D743, 11 2022. ISSN 0305-1048. doi:
10.1093/nar/gkac1037. URL https://doi.org/10.1093/nar/gkac1037.

Antonio Pedro Camargo, Lee Call, Simon Roux, Stephen Nayfach, Marcel Huntemann,
Krishnaveni Palaniappan, Anna Ratner, Ken Chu, Supratim Mukherjeep, T B K Reddy,
I-Min A Chen, Natalia N Ivanova, Emiley A Eloe-Fadrosh, Tanja Woyke, David A Baltrus,
Salvador Castañeda-Barba, Fernando de la Cruz, Barbara E Funnell, James P J Hall,
Aindrila Mukhopadhyay, Eduardo P C Rocha, Thibault Stalder, Eva Top, and Nikos C
Kyrpides. IMG/PR: a database of plasmids from genomes and metagenomes with rich
annotations and metadata. Nucleic Acids Res., 52(D1):D164–D173, January 2024.

Ryan Cook, Andrea Telatin, George Bouras, Antonio Pedro Camargo, Martin Larralde,
Robert A Edwards, and Evelien M Adriaenssens. Driving through stop signs: predict-
ing stop codon reassignment improves functional annotation of bacteriophages.
ISME
Commun, 4(1):ycae079, January 2024.

Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Revilla, Nicolas Lopez Carranza,
Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop,
Bernardo P de Almeida, Hassan Sirelkhatim, Guillaume Richard, Marcin Skwark, Karim
Beguir, Marie Lopez, and Thomas Pierrot. The nucleotide transformer: Building and
evaluating robust foundation models for human genomics. September 2023.

Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning,

2023. URL https://arxiv.org/abs/2307.08691.

Frances Ding and Jacob Steinhardt. Protein language models are biased by unequal sequence

sampling across the tree of life. March 2024.

Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groen-
eveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case
study on the colossal clean crawled corpus. April 2021.

Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion
Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu
Bhowmik, and Burkhard Rost. ProtTrans: Toward understanding the language of life
through Self-Supervised learning. IEEE Trans. Pattern Anal. Mach. Intell., 44(10):7112–
7127, October 2022.

12

Under review as a conference paper at ICLR 2025

648

649
650
651
652
653

654
655
656
657
658
659

660
661
662
663
664
665

666
667
668
669
670
671

672
673
674
675
676

677
678
679
680
681
682

683
684
685
686
687
688

689
690
691
692
693
694

695
696
697
698
699

700
701

Tomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin,
Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, Rohil Badkun-
dri, Irhum Shafkat, Jun Gong, Alexander Derry, Raúl Santiago Molina, Neil Thomas,
Yousuf A Khan, Chetan Mishra, Carolyn Kim, Liam J Bartie, Matthew Nemeth,
Patrick D Hsu, Tom Sercu, Salvatore Candido, and Alexander Rives. Simulating 500
million years of evolution with a language model. July 2024.

Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan
Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling
is predictable, empirically. December 2017.

Laura A Hug, Brett J Baker, Karthik Anantharaman, Christopher T Brown, Alexander J
Probst, Cindy J Castelle, Cristina N Butterfield, Alex W Hernsdorf, Yuki Amano, Kotaro
Ise, Yohey Suzuki, Natasha Dudek, David A Relman, Kari M Finstad, Ronald Amundson,
Brian C Thomas, and Jillian F Banfield. A new view of the tree of life. Nat Microbiol, 1:
16048, April 2016.

Marcel Huntemann, Natalia N Ivanova, Konstantinos Mavromatis, H James Tripp, David
Paez-Espino, Kristin Tennessen, Krishnaveni Palaniappan, Ernest Szeto, Manoj Pillay,
I-Min A Chen, Amrita Pati, Torben Nielsen, Victor M Markowitz, and Nikos C Kyrpi-
des. The standard operating procedure of the DOE-JGI metagenome annotation pipeline
(MAP v.4). Stand. Genomic Sci., 11:17, February 2016.

Yunha Hwang, Andre L Cornman, Elizabeth H Kellogg, Sergey Ovchinnikov, and Peter R
Girguis. Genomic language model predicts protein co-regulation and function. Nat.
Commun., 15(1):2880, April 2024.

M Johnsen, T Christensen, P P Dennis, and N P Fiil. Autogenous control: ribosomal protein
L10-L12 complex binds to the leader sequence of its mRNA. EMBO J., 1(8):999–1004,
1982.

Hetunandan Kamisetty, Sergey Ovchinnikov, and David Baker. Assessing the utility of
coevolution-based residue–residue contact predictions in a sequence- and structure-rich
era. Proceedings of the National Academy of Sciences, 110(39):15674–15679, 2013.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon
Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural
language models. January 2020.

Peter D. Karp, Suzanne Paley, Ron Caspi, Anamika Kothari, Markus Krummenacker, Pe-
ter E. Midford, Lisa R. Moore, Pallavi Subhraveti, Socorro Gama-Castro, Victor H. Tier-
rafria, Paloma Lara, Luis Muñiz-Rascado, César Bonavides-Martinez, Alberto Santos-
Zavaleta, Amanda Mackie, Gwanggyu Sun, Travis A. Ahn-Horst, Heejo Choi, Markus W.
Covert, Julio Collado-Vides, and Ian Paulsen. The ecocyc database (2023). EcoSal
Plus, 11(1):eesp–0002–2023, 2023. doi: 10.1128/ecosalplus.esp-0002-2023. URL https:
//journals.asm.org/doi/abs/10.1128/ecosalplus.esp-0002-2023.

Senying Lai, Shaojun Pan, Chuqing Sun, Luis Pedro Coelho, Wei-Hua Chen, and Xing-
Ming Zhao. metaMIC: reference-free misassembly identification and correction of de novo
metagenomic assemblies. Genome Biol., 23(1):242, November 2022.

Alla L Lapidus and Anton I Korobeynikov. Metagenomic data assembly - the way of de-

coding unknown microorganisms. Front. Microbiol., 12:613791, March 2021.

Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick
von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall,
Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven
Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp
Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas
Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexan-
der Rush, and Thomas Wolf. Datasets: A community library for natural language
In Proceedings of the 2021 Conference on Empirical Methods in Natural
processing.

13

Under review as a conference paper at ICLR 2025

702

703
704
705
706
707

708
709
710
711
712
713

714
715
716
717
718
719

720
721
722
723
724
725

726
727
728
729
730

731
732
733
734
735
736

737
738
739
740
741
742

743
744
745
746
747
748

749
750
751
752
753

754
755

Language Processing: System Demonstrations, pp. 175–184, Online and Punta Cana,
Dominican Republic, November 2021. Association for Computational Linguistics. URL
https://aclanthology.org/2021.emnlp-demo.21.

Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita
Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan Dos Santos Costa, Maryam
Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale
prediction of atomic-level protein structure with a language model. Science, 379(6637):
1123–1130, March 2023.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL

https://arxiv.org/abs/1711.05101.

Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin P Mohr, James M
Holton, Jose Luis Olmos, Jr, Caiming Xiong, Zachary Z Sun, Richard Socher, James S
Fraser, and Nikhil Naik. Large language models generate functional protein sequences
across diverse families. Nat. Biotechnol., 41(8):1099–1106, August 2023.

Victor M Markowitz, I-Min A Chen, Krishna Palaniappan, Ken Chu, Ernest Szeto, Yuri
Grechkin, Anna Ratner, Biju Jacob, Jinghua Huang, Peter Williams, Marcel Huntemann,
Iain Anderson, Konstantinos Mavromatis, Natalia N Ivanova, and Nikos C Kyrpides. IMG:
the integrated microbial genomes database and comparative analysis system. Nucleic
Acids Res., 40(Database issue):D115–22, January 2012.

Daniel R Mende, Alison S Waller, Shinichi Sunagawa, Aino I Järvelin, Michelle M Chan,
Manimozhiyan Arumugam, Jeroen Raes, and Peer Bork. Assessment of metagenomic as-
sembly using simulated next generation sequencing data. PLoS One, 7(2):e31386, Febru-
ary 2012.

Eric Nguyen, Michael Poli, Matthew G Durrant, Armin W Thomas, Brian Kang, Jeremy
Sullivan, Madelena Y Ng, Ashley Lewis, Aman Patel, Aaron Lou, Stefano Ermon,
Stephen A Baccus, Tina Hernandez-Boussard, Christopher Ré, Patrick D Hsu, and
Brian L Hie. Sequence modeling and design from molecular to genome scale with evo.
March 2024.

Pascal Notin, Aaron W. Kollasch, Daniel Ritter, Lood van Niekerk, Steffanie Paul, Hansen
Spinner, Nathan Rollins, Ada Shaw, Ruben Weitzman, Jonathan Frazer, Mafalda Dias,
Dinko Franceschi, Rose Orenbuch, Yarin Gal, and Debora S. Marks. Proteingym:
Large-scale benchmarks for protein design and fitness prediction. bioRxiv, 2023. doi:
10.1101/2023.12.07.570727. URL https://www.biorxiv.org/content/early/2023/12/
08/2023.12.07.570727.

Robert D Olson, Rida Assaf, Thomas Brettin, Neal Conrad, Clark Cucinell, James J Davis,
Donald M Dempsey, Allan Dickerman, Emily M Dietrich, Ronald W Kenyon, Mehmet
Kuscuoglu, Elliot J Lefkowitz, Jian Lu, Dustin Machi, Catherine Macken, Chunhong Mao,
Anna Niewiadomska, Marcus Nguyen, Gary J Olsen, Jamie C Overbeek, Bruce Parrello,
Victoria Parrello, Jacob S Porter, Gordon D Pusch, Maulik Shukla, Indresh Singh, Lucy
Stewart, Gene Tan, Chris Thomas, Margo VanOeffelen, Veronika Vonstein, Zachary S
Wallace, Andrew S Warren, Alice R Wattam, Fangfang Xia, Hyunseung Yoo, Yun Zhang,
Christian M Zmasek, Richard H Scheuermann, and Rick L Stevens. Introducing the bac-
terial and viral bioinformatics resource center (BV-BRC): a resource combining PATRIC,
IRD and ViPR. Nucleic Acids Res., 51(D1):D678–D689, January 2023.

Sergey Ovchinnikov, Hetunandan Kamisetty, and David Baker. Robust and accurate pre-
diction of residue-residue interactions across protein interfaces using evolutionary infor-
mation. Elife, 3:e02030, May 2014.

Donovan H Parks, Fabio Rigato, Patricia Vera-Wolf, Lutz Krause, Philip Hugenholtz,
Gene W Tyson, and David L A Wood. Evaluation of the microba community profiler
for taxonomic profiling of metagenomic datasets from the human gut microbiome. Front.
Microbiol., 12:643682, April 2021.

14

Under review as a conference paper at ICLR 2025

756

757
758
759
760
761

762
763
764
765
766
767

768
769
770
771
772
773

774
775
776
777
778
779

780
781
782
783
784

785
786
787
788
789
790

791
792
793
794
795
796

797
798
799
800
801
802

803
804
805
806
807

808
809

Donovan H Parks, Maria Chuvochina, Christian Rinke, Aaron J Mussig, Pierre-Alain
Chaumeil, and Philip Hugenholtz. GTDB: an ongoing census of bacterial and archaeal
diversity through a phylogenetically consistent, rank normalized and complete genome-
based taxonomy. Nucleic Acids Res., 50(D1):D785–D794, January 2022.

Georgios A Pavlopoulos, Fotis A Baltoumas, Sirui Liu, Oguz Selvitopi, Antonio Pedro
Camargo, Stephen Nayfach, Ariful Azad, Simon Roux, Lee Call, Natalia N Ivanova,
I Min Chen, David Paez-Espino, Evangelos Karatzas, Ioannis Iliopoulos, Konstantinos
Konstantinidis, James M Tiedje, Jennifer Pett-Ridge, David Baker, Axel Visel, Christos A
Ouzounis, Sergey Ovchinnikov, Aydin Buluç, and Nikos C Kyrpides. Unraveling the
functional dark matter through global metagenomics. Nature, 622(7983):594–602, October
2023.

Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell,
Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting
the web for the finest text data at scale. June 2024.

Kim D Pruitt, Garth R Brown, Susan M Hiatt, Françoise Thibaud-Nissen, Alexander As-
tashyn, Olga Ermolaeva, Catherine M Farrell, Jennifer Hart, Melissa J Landrum, Kelly M
McGarvey, Michael R Murphy, Nuala A O’Leary, Shashikant Pujar, Bhanu Rajput, San-
jida H Rangwala, Lillian D Riddick, Andrei Shkeda, Hanzhen Sun, Pamela Tamez, Ray-
mond E Tully, Craig Wallin, David Webb, Janet Weber, Wendy Wu, Michael DiCuccio,
Paul Kitts, Donna R Maglott, Terence D Murphy, and James M Ostell. RefSeq: an up-
date on mammalian reference sequences. Nucleic Acids Res., 42(Database issue):D756–63,
January 2014.

Lorna Richardson, Ben Allen, Germana Baldi, Martin Beracochea, Maxwell L Bileschi,
Tony Burdett, Josephine Burgin, Juan Caballero-Pérez, Guy Cochrane, Lucy J Colwell,
Tom Curtis, Alejandra Escobar-Zepeda, Tatiana A Gurbich, Varsha Kale, Anton Ko-
robeynikov, Shriya Raj, Alexander B Rogers, Ekaterina Sakharova, Santiago Sanchez,
Darren J Wilkinson, and Robert D Finn. MGnify: the microbiome sequence data analy-
sis resource in 2023. Nucleic Acids Res., 51(D1):D753–D759, January 2023.

Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi
Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure
and function emerge from scaling unsupervised learning to 250 million protein sequences.
Proc. Natl. Acad. Sci. U. S. A., 118(15), April 2021.

Noam Shazeer. Glu variants improve transformer, 2020. URL https://arxiv.org/abs/

2002.05202.

Ben Sorscher, Robert Geirhos, Shashank Shekhar, S Ganguli, and Ari S Morcos. Beyond
neural scaling laws: beating power law scaling via data pruning. Adv. Neural Inf. Process.
Syst., abs/2206.14486, June 2022.

Martin Steinegger and Steven L Salzberg. Terminating contamination:

large-scale search
identifies more than 2,000,000 contaminated entries in GenBank. Genome Biol., 21(1):
115, May 2020.

Martin Steinegger and Johannes Söding. MMseqs2 enables sensitive protein sequence search-
ing for the analysis of massive data sets. Nat. Biotechnol., 35(11):1026–1028, November
2017.

Martin Steinegger and Johannes Söding. Clustering huge protein sequence sets in linear

time. Nat. Commun., 9(1):2542, June 2018.

Martin Steinegger, Milot Mirdita, and Johannes Söding. Protein-level assembly increases
protein sequence recovery from metagenomic samples manyfold. Nat. Methods, 16(7):
603–606, July 2019.

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer:
Enhanced transformer with rotary position embedding, 2023. URL https://arxiv.org/
abs/2104.09864.

15

Under review as a conference paper at ICLR 2025

Baris E Suzek, Hongzhan Huang, Peter McGarvey, Raja Mazumder, and Cathy H Wu.
UniRef: comprehensive and non-redundant UniProt reference clusters. Bioinformatics,
23(10):1282–1288, May 2007.

Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari S Morcos. D4: Improving LLM
pretraining via document De-Duplication and diversification. Adv. Neural Inf. Process.
Syst., abs/2308.12284, August 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux,
Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien
Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and
efficient foundation language models. February 2023.

Gene W Tyson, Jarrod Chapman, Philip Hugenholtz, Eric E Allen, Rachna J Ram, Paul M
Richardson, Victor V Solovyev, Edward M Rubin, Daniel S Rokhsar, and Jillian F Ban-
field. Community structure and metabolism through reconstruction of microbial genomes
from the environment. Nature, 428(6978):37–43, March 2004.

UniProt Consortium. UniProt: a worldwide hub of protein knowledge. Nucleic Acids Res.,

47(D1):D506–D515, January 2019.

John Vollmers, Sandra Wiegand, and Anne-Kristin Kaster. Comparing and evaluating
metagenome assembly tools from a microbiologist’s perspective - not only size matters!
PLoS One, 12(1):e0169662, January 2017.

Patrick T West, Alexander J Probst, Igor V Grigoriev, Brian C Thomas, and Jillian F Ban-
field. Genome-reconstruction for eukaryotes from complex natural microbial communities.
Genome Res., 28(4):569–580, April 2018.

Jacob West-Roberts, Luis Valentin-Alvarado, Susan Mullen, Rohan Sachdeva, Justin Smith,
Laura A Hug, Daniel S Gregoire, Wentso Liu, Tzu-Yu Lin, Gabriel Husain, Yuki Amano,
Lynn Ly, and Jillian F Banfield. Giant genes are rare but implicated in cell wall degra-
dation by predatory bacteria. November 2023.

Jacob West-Roberts, Joshua Kravitz, Nishant Jha, Andre Cornman, and Yunha Hwang.
Diverse genomic embedding benchmark for functional evaluation across the tree of life.
July 2024.

Derrick E Wood and Steven L Salzberg. Kraken: ultrafast metagenomic sequence classifi-

cation using exact alignments. Genome Biol., 15(3):R46, March 2014.

Biao Zhang and Rico Sennrich. Root mean square layer normalization, 2019. URL https:

//arxiv.org/abs/1910.07467.

Zhidian Zhang, Hannah K Wayment-Steele, Garyk Brixi, Haobo Wang, Matteo Dal Peraro,
Dorothee Kern, and Sergey Ovchinnikov. Protein language models learn evolutionary
statistics of interacting sequence motifs. January 2024.

16

810

811
812
813
814
815

816
817
818
819
820
821

822
823
824
825
826
827

828
829
830
831
832
833

834
835
836
837
838

839
840
841
842
843
844

845
846
847
848
849
850

851
852
853
854
855
856

857
858
859
860
861

862
863

Under review as a conference paper at ICLR 2025

Appendix A Data sources

Type

Snapshot date # Samples # contigs* Total bps # CDS

IMG Metagenomes

Genomes

2023-08-27
2023-08-27

36,273
131,744

182M
6.2M

1.70T
0.4T

1.84B
0.4B

MGnify Metagenomes

2022-11-23

33,531

82M

1.03T

1.03B

*Number of contigs after filtering and preprocessing.

Appendix B Dataset Preprocessing

Sequences (purple) undergo filtering steps (green), yielding three Hugging Face datasets
(yellow) made available with this paper. ‘NA’ and ‘AA’ refer to nucleic acid and amino acid
data modalities respectively.

Appendix C Dataset Length Distributions

Length distributions of the OMG corpus. (A) Distribution of contig lengths in the number
of genomic elements (CDS and IGS). (B) Distribution of contig lengths in base pairs. (C)
Distribution of CDS lengths in amino acids. (D) Distribution of IGS lengths in base pairs.

17

864

865
866
867
868
869

870
871
872
873
874
875

876
877
878
879
880
881

882
883
884
885
886
887

888
889
890
891
892

893
894
895
896
897
898

899
900
901
902
903
904

905
906
907
908
909
910

911
912
913
914
915

916
917

Under review as a conference paper at ICLR 2025

918

919
920
921
922
923

924
925
926
927
928
929

930
931
932
933
934
935

936
937
938
939
940
941

942
943
944
945
946

947
948
949
950
951
952

953
954
955
956
957
958

959
960
961
962
963
964

965
966
967
968
969

970
971

Appendix D Invalid Character Distributions

Distribution of the percent of characters per genomic element considered as invalid ("X"
for amino acids and "N" for nucleotides) prior to applying the assembly quality filter from
Section 3.1. The assembly quality filter removes elements containing more than 20% invalid
characters, resulting in 0.004% of CDS and 0.2% of IGS being filtered from OMG. We show
the distribution for the subset of genomic elements containing at least 1 invalid character.

Appendix E OMG_prot50 clustering method

A total of 4.2B protein sequences were first clustered to remove fragments using MMseqs2
linclust (Steinegger & Söding, 2018) (commit f6c98, parameters:–min-seq-id 0.9 -c 0.9 –cov-
mode 1). Subsequently, the resulting sequences were clustered at 50% sequence id and 90%
sequence coverage using MMseqs2 linclust –min-seq-id 0.5 -c 0.9. Singleton clusters
(only one sequence in the cluster across the full dataset) were removed and remaining 207M
cluster representatives were uploaded as the Hugging Face dataset.

18

Under review as a conference paper at ICLR 2025

Appendix F GLM2 model parameters

gLM2 is a transformer encoder optimized using AdamW (Loshchilov & Hutter, 2019) and
trained in mixed precision bfloat16. We set the AdamW betas to (0.9, 0.95) and weight
decay of 0.1. We disable dropout throughout training. The learning rate is warmed up
for 1k steps, followed by a cosine decay to 10% of the maximum learning rate. gLM2 uses
RoPE (Su et al., 2023) position encoding, SwiGLU (Shazeer, 2020) feed-forward layers, and
RMS normalization (Zhang & Sennrich, 2019). We leverage Flash Attention 2 (Dao, 2023)
to speed up attention computation over the sequence length of 4096.

Dim

Num
heads

Num
layers

Context
length

Learning
rate

Batch
size

Pretraining
tokens

gLM2-150M 640

gLM2-650M 1280

10

20

30

33

4096

4096

1e-3

1e-3

128

128

315B

315B

972

973
974
975
976
977

978
979
980
981
982
983

984
985
986
987
988
989

990
991
992
993
994
995

996
997
998
999
1000

1001
1002
1003
1004
1005
1006

1007
1008
1009
1010
1011
1012

1013
1014
1015
1016
1017
1018

1019
1020
1021
1022
1023

1024
1025

19

Under review as a conference paper at ICLR 2025

Appendix G Semantic deduplication distance threshold

The percentage of remaining training examples as a function of the embedding distance
threshold. Examples within the distance threshold in embedding space are deduplicated.

Appendix H Taxonomic distribution of the OG dataset before

and after pruning

Data pruning through semantic deduplication reduces dataset bias toward overrepresented
phyla and orders.

1026

1027
1028
1029
1030
1031

1032
1033
1034
1035
1036
1037

1038
1039
1040
1041
1042
1043

1044
1045
1046
1047
1048
1049

1050
1051
1052
1053
1054

1055
1056
1057
1058
1059
1060

1061
1062
1063
1064
1065
1066

1067
1068
1069
1070
1071
1072

1073
1074
1075
1076
1077

1078
1079

20

Under review as a conference paper at ICLR 2025

Appendix I Taxonomic entropy of the OG dataset before and

after pruning

Semantic deduplication of the OG dataset consistently increases the taxonomic entropy
across all taxonomic ranks, indicating a more even distribution.

1080

1081
1082
1083
1084
1085

1086
1087
1088
1089
1090
1091

1092
1093
1094
1095
1096
1097

1098
1099
1100
1101
1102
1103

1104
1105
1106
1107
1108

1109
1110
1111
1112
1113
1114

1115
1116
1117
1118
1119
1120

1121
1122
1123
1124
1125
1126

1127
1128
1129
1130
1131

1132
1133

21

Under review as a conference paper at ICLR 2025

1134
1135
1136
1137
1138
1139

1140
1141
1142
1143
1144
1145

1146
1147
1148
1149
1150
1151

1152
1153
1154
1155
1156

1157
1158
1159
1160
1161
1162

1163
1164
1165
1166

B
E
G
D
e
h
t

n
o

d
e
t
a
u
l
a
v
e

e
r
a

s
l
e
d
o
m
h
t
o
B

.
s
p
e
t
s

k
0
0
6

r
o
f

h
c
a
e

,
t
e
s
a
t
a
d
G
M
O
d
e
n
u
r
p

d
n
a

l
a
n

i
g
i
r
o

e
h
t

n
o

s
l
e
d
o
m
2
M
L
g

r
e
t
e
m
a
r
a
p
M
0
5
1

o
w
t

n
i
a
r
t

e

W

.
s
e
c
n
e
u
q
e
s

d
e
t
n
e
s
e
r
p
e
r
-
r
e
d
n
u

h
t
i
w
s
k
s
a
t

r
o
f

y
l
l
a
i
c
e
p
s
e

,
e
c
n
a
m
r
o
f
r
e
p

s
e
v
o
r
p
m

i

g
n
i
n
u
r
P

.
k
r
a
m
h
c
n
e
b

n
o
i
t
a
c
i
l
p
u
d
e
d

c
i
t
n
a
m
e
s

f
o

n
o
i
t
a
l
b
A

J

i

x
d
n
e
p
p
A

B
E
G
D

l
a
i
r
e
t
c
a
B

B
o
p
R

e
r
o
c
S

-
o
l
y
h
P

y
n
e
g

o
i
r
b
i
V

c
i
n
o
r
e
p
O

r
i
a
P

B
p
o
M

-
t
s
u
l
C

g
n
i
r
e

-
g
r
e
v
n
o
C

s
e
m
y
z
n
E

-
i
r
t
e
R

t
n
e

k
u
E

-
s
s
a
l
C

n
o
i
t
a
c
fi
i

l
a
v
e

i
l
o
c
.
E

-
r
e
p
O

c
i
n
o

r
i
a
P

i

G
B
I
M

-
s
s
a
l
C

a
c
fi

i

n
o
i
t

C
E

-
i
s
s
a
l
C

n
o
i
t
a
c
fi

B
o
p
R

l
a
e
a
h
c
r
A

-
o
l
y
h
P

y
n
e
g

e
F
e
F

-
o
r
d
y
H

e
s
a
n
e
g

-
o
l
y
h
P

y
n
e
g

C
A
d
o
M

o
n
a
y
C

-
a
r
a
P

y
g
o
l

e
n
e
G
B

i

.
r
e
p
O

i
r
t
e
R

l
a
v
e

h
c
r
A

-
i
r
t
e
R

l
a
v
e

c
a
B

h
c
r
A

e
n
e
G
B

i

4
7
4
.
0

1
5
2
.
0

7
4
5
.
0

6
8
7
.
0

6
9
1
.
0

8
4
3
.
0

8
2
6
.
0

0
7
6
.
0

7
1
5
.
0

2
1
3
.
0

8
8
6
.
0

4
4
2
.
0

8
0
4
.
0

9
8
2
.
0

0
5
7
.
0

2
8
4
.
0

1
1
3
.
0

6
6
5
.
0

0
2
8
.
0

2
7
1
.
0

0
3
3
.
0

9
3
6
.
0

0
6
6
.
0

4
1
5
.
0

3
5
3
.
0

6
1
7
.
0

9
2
2
.
0

8
1
4
.
0

4
9
2
.
0

2
2
7
.
0

)
g
n
i
n
u
r
p

a
t
a
d

o
n
(

M
0
5
1
_
2
M
L
g

)
g
n
i
n
u
r
p

a
t
a
d

h
t
i
w
(

M
0
5
1
_
2
M
L
g

22

Under review as a conference paper at ICLR 2025

Appendix K Per task DGEB scaling with FLOPs for ESM2

and gLM2 models in amino acid tasks

Primary metric from the best scoring layer (between mid, and last) is reported for each
task. To account for model-specific patterns in learning task-relevant functional information
across different layers in the network (West-Roberts et al., 2024), DGEB calculates model
performance for both mid and last layer and reports the best score between the two.

1167

1168
1169
1170
1171
1172

1173
1174
1175
1176
1177
1178

1179
1180
1181
1182
1183
1184

1185
1186
1187
1188
1189
1190

1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206
1207

1208
1209
1210
1211
1212
1213

1214
1215
1216
1217
1218

1219
1220

23

Under review as a conference paper at ICLR 2025

Appendix L Per task DGEB scaling with FLOPs for

Nucleotide Transformers and gLM2 models in
nucleic acid tasks.

Primary metric from the best scoring layer (between mid, and last) is reported for each
task. To account for model-specific patterns in learning task-relevant functional information
across different layers in the network (West-Roberts et al., 2024), DGEB calculates model
performance for both mid and last layer and reports the best score between the two.

Appendix M GLM2 performance on ProteinGym

Model name

ESM2_650M
gLM2_650M_prot

Avg
Spearman
0.414
0.384

Activity Binding Expression Organismal Stability

0.425
0.406

0.337
0.327

0.415
0.412

Fitness
0.369
0.311

0.523
0.466

We evaluate gLM2 on the ProteinGym (Notin et al., 2023) Deep Mutational Scanning
(DMS) substitutions task. Because the DMS task is strictly a single-protein task (without
context), we benchmark gLM2_650M after finetuning for one epoch of OMG_prot50, the
single-protein dataset introduced in Table 1. While gLM2_650M_prot performs slightly
worse than ESM2_650M, we note that the ProteinGym benchmark includes eukaryotic
sequences, which are poorly represented in the OMG dataset.

Appendix N ModA and ModC sequence concatenation

This concatenated sequence was derived from the 2ONK_A_2ONK_C alignment used in
(Ovchinnikov et al., 2014).

MFLKVRAEKRLGNFRLNVDFEMGRDYCVLLGPTGAGKSVFLELIAGIVKPDRGEVRLNGADITPLPPERGIGFV
PQDYALFPHLSVYRNIAYGLRNVERVERDRRVREMAEKLGIAHLLDRKPARLSGGERQRVALARALVIQPRLLLLDEPLSAV
DLKTKGVLMEELRFVQREFDVPILHVTHDLIEAAMLADEVAVMLNGRIVEKGKLKELFSAKNGEVAEFLSARNLLLKVSKIL
DMRLLFSALLALLSSIILLFVLLPVAATVTLQLFNFDEFLKAASDPAVWKVVLTTYYAALISTLIAVIFGTPLAYILARKSF
PGKSVVEGIVDLPVVIPHTVAGIALLVVFGSSGLIGSFSPLKFVDALPGIVVAMLFVSVPIYINQAKEGFASVDVRLEHVAR
TLGSSPLRVFFTVSLPLSVRHIVAGAIMSWARGISEFGAVVVIAYYPMIAPTLIYERYLSEGLSAAMPVAAILILLSLAVFV
ALRIIVGREDVSEGQG

24

1221

1222
1223
1224
1225
1226

1227
1228
1229
1230
1231
1232

1233
1234
1235
1236
1237
1238

1239
1240
1241
1242
1243
1244

1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260
1261

1262
1263
1264
1265
1266
1267

1268
1269
1270
1271
1272

1273
1274

Under review as a conference paper at ICLR 2025

Appendix O Putative RNA-Protein-Protein interactions

We visualize a contiguous stretch (119,848-120,978bp, 5’->3’) of the B. sutilis 168 reference
genome. Putative residue-level interactions between L10 leader RNA (ldlJ ), proteins RplJ
and RplL are highlighted in gray boxes. Shine-Dalgarno sequences upstream of the two
protein-coding genes are highlighted and co-evolve.

Appendix P Additional Files

Additional Files are found in https://zenodo.org/records/14198868

Additional File 1. OG sample ID to original NCBI metadata. A JSON file mapping OG
sample ID (taxon_oid) to NCBI metadata (accessions, collection dates).

Additional File 2. DOIs for MGnify samples. DOIs for MGnify samples that were included
in this study, where available.

Additional File 3. DOIs for IMG samples, DOIs for IMG samples that were included in
this study, where available. Additional File 4. Comparison of gLM2 Jacobian Contacts
on 2ONK with (A) and without (B) the 2 basepair IGS sequence flanking ModA and ModC.
We show that the addition of IGS sequence does not change the results. Additional File 5.
A zip file containing all 32 evolutionary conserved complexes in PDB previously identified
in (Ovchinnikov et al., 2014), https://openseq.org/cplx.php?sort=prob&order=DESC&
mode=pdb. PDB contacts and gLM2 Jacobian Contacts are compared.
Additional File 6. A zip file containing Categorical Jacobian maps of 26 IGS regions in
E.coli K-12 str. MG1655 (Genome ID: U00096) with at least one promoter (highlighted in
red) and one terminator (highlighted in green) sites annotated in EcoCyc. File names and
figure title correspond to the start and end positions in the U00096 genome.

25

1275

1276
1277
1278
1279
1280

1281
1282
1283
1284
1285
1286

1287
1288
1289
1290
1291
1292

1293
1294
1295
1296
1297
1298

1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314
1315

1316
1317
1318
1319
1320
1321

1322
1323
1324
1325
1326

1327
1328

