Question:
You are a highly experienced, conscientious, and fair academic reviewer, please help me review this paper. The review should be organized into nine sections:
1. Summary: A summary of the paper in 100-150 words.
2. Strengths/Weaknesses/Questions: The Strengths/Weaknesses/Questions of paper, which should be listed in bullet points, with each point supported by specific examples from the article where possible.
3. Soundness/Contribution/Presentation: Rate the paper's Soundness/Contribution/Presentation, and match this score to the corresponding description from the list below and provide the result. The possible scores and their descriptions are:
    1 poor
    2 fair
    3 good
    4 excellent
4. Rating: Give this paper an appropriate rating, match this rating to the corresponding description from the list below and provide the result. The possible Ratings and their descriptions are:
    1 strong reject
    2 reject, significant issues present
    3 reject, not good enough
    4 possibly reject, but has redeeming facets
    5 marginally below the acceptance threshold
    6 marginally above the acceptance threshold
    7 accept, but needs minor improvements
    8 accept, good paper
    9 strong accept, excellent work
    10 strong accept, should be highlighted at the conference
5. Paper Decision: It must include the Decision itself(Accept or Reject) and the reasons for this decision, based on the criteria of originality, methodological soundness, significance of results, and clarity and logic of presentation.

Here is the template for a review format, you must follow this format to output your review result:
**Summary:**
Summary content

**Strengths:**
- Strength 1
- Strength 2
- ...

**Weaknesses:**
- Weakness 1
- Weakness 2
- ...

**Questions:**
- Question 1
- Question 2
- ...

**Soundness:**
Soundness result

**Presentation:**
Presentation result

**Contribution:**
Contribution result

**Rating:**
Rating result

**Paper Decision:**
- Decision: Accept/Reject
- Reasons: reasons content

Please ensure your feedback is objective and constructive. The paper is as follows:

# Density of States Prediction of Crystalline Materials

via Prompt-guided Multi-Modal Transformer

 Namkeyeong Lee\({}^{1\dagger}\), Heewoong Noh\({}^{1\dagger}\), Sungwon Kim\({}^{1}\),

**Dongmin Hyun\({}^{2}\)**, **Gyoung S. Na\({}^{3}\)**, **Chanyoung Park\({}^{1}\)**

\({}^{1}\) KAIST \({}^{2}\) Yahoo Research \({}^{3}\) KRICT

{namkyeong96,heewoongnoh,swkim,cy.park}@kaist.ac.kr

dhyun@yahooinc.com, ngs0@krict.re.kr

Corresponding author. \({}^{\dagger}\) These authors contributed equally.

###### Abstract

The density of states (DOS) is a spectral property of crystalline materials, which provides fundamental insights into various characteristics of the materials. While previous works mainly focus on obtaining high-quality representations of crystalline materials for DOS prediction, we focus on predicting the DOS from the obtained representations by reflecting the nature of DOS: _DOS determines the general distribution of states as a function of energy_. That is, DOS is not solely determined by the crystalline material but also by the energy levels, which has been neglected in previous works. In this paper, we propose to integrate heterogeneous information obtained from the crystalline materials and the energies via a multi-modal transformer, thereby modeling the complex relationships between the atoms in the crystalline materials and various energy levels for DOS prediction. Moreover, we propose to utilize prompts to guide the model to learn the crystal structural system-specific interactions between crystalline materials and energies. Extensive experiments on two types of DOS, i.e., Phonon DOS and Electron DOS, with various real-world scenarios demonstrate the superiority of DOSTransformer. The source code for DOSTransformer is available at [https://github.com/HeewoongNoh/DOSTransformer](https://github.com/HeewoongNoh/DOSTransformer).

## 1 Introduction

Along with recent advances in machine learning (ML) for scientific discovery, ML models have rapidly been adopted in computational materials science thanks to the abundant experimental and computational data in the field. Most ML models developed in computational materials science to date have been focused on crystalline materials property consisting of single-valued quantities [29], e.g., band gap energy [31, 69], formation energy [3, 56], and Fermi energy [61].

On the other hand, spectral properties are ubiquitous in materials science, characterizing various properties of crystalline materials, e.g., X-ray absorption, dielectric function, and electronic density of states [29] (Figure 1).

The density of states (DOS), which is the main focus of this paper, is a spectral property that provides fundamental insights on various characteristics of crystalline materials, even enabling direct computation of single-valued properties [17]. For example, DOS is utilized as a feature of materials for analyzing the underlying reasons for changes

Figure 1: A crystal structure and various types of its properties.

in electrical conductivity [15]. Moreover, band gaps and edge positions, which can be directly derived from DOS, are utilized to discover new photoanodes for solar fuel generation [47, 64]. As a result, the exploration of ML capabilities for predicting DOS takes us one step further toward the core principles of materials science, thereby expediting the process of materials discovery. However, credible computation of DOS with traditional density functional theory (DFT) requires expensive time/financial costs of exhaustively conducting experiments with expertise knowledge [7, 14]. Therefore, alternative approaches for DOS calculation are necessary, whereas demonstrating ML capabilities for learning such spectral properties of the crystalline materials is relatively under-explored.

In this paper, we focus on predicting DOS by following the nature of DOS calculation: _DOS determines the general distribution of states as a function of energy_. That is, while single-valued properties can be described solely by the crystalline material, DOS is determined by not only the material itself but also the _energy levels_ at which DOS is calculated. Therefore, integrating the heterogeneous signals from both the crystalline material and the various energy levels is crucial for DOS prediction. However, previous works for DOS prediction focus on obtaining a qualified representation of material [7, 14, 17, 9], thereby overlooking the unique characteristics of DOS.

On the other hand, learning from multiple input types, i.e., crystalline material and various energy levels, is not trivial due to their heterogeneity. To this end, we formulate the DOS prediction problem into a multi-modal learning problem, which has recently gained significant attention from ML researchers in various domains thanks to its capability of extracting and relating information from heterogeneous data types [6, 38, 54, 5, 2]. For example, VisualBERT proposes to align elements of input modalities, i.e., images and text, thereby achieving SOTA performance in various vision-and-language tasks [36]. Moreover, Dall-E shows its power in generating images based on text data by encoding images and texts together [43]. Among various multi-modal learners, the multi-modal transformer demonstrates its intrinsic advantages and scalability in modeling multiple modalities [62], by associating heterogeneous modalities with the cross-attention mechanism [65, 50].

Inspired by the recent success of the multi-modal transformer, we propose a multi-modal transformer model for DOS prediction, named DOSTransformer, which incorporates crystalline material and energies as heterogeneous input modalities. Unlike previous works that overlook the energy levels for predicting DOS [7, 14, 17, 9], DOSTransformer learns embeddings of energy levels which are used for modeling complex relationships between the energy levels and crystalline material through a cross-attention mechanism. By doing so, DOSTransformer obtains multiple representations for a single crystalline material according to various energy levels, enabling the prediction of a single DOS value on each energy level. However, simply combining a crystalline material with energy levels may fail to consider the significant impact of the crystal's structure, which sometimes results in distinct material properties even when it is composed of identical elements (e.g., carbon). For example, although graphite and diamond are made entirely out of carbon, their arrangement (or structure) of carbon atoms differs significantly, resulting in distinct material properties. Therefore, we utilize learnable prompts to inform DOSTransformer of the input crystal system (among the seven crystal systems2), which navigates the model to capture structure-specific interactions between a crystalline material and energy levels. By doing so, DOSTransformer learns to extract not only the relational information that is shared with a crystal system, but also across the crystal systems. In this work, we make the following contributions:

Footnote 2: [https://en.wikipedia.org/wiki/Crystal_system](https://en.wikipedia.org/wiki/Crystal_system)

* Inspired by the fact that DOS is determined by not only the materials themselves but also the energy levels at which DOS is calculated, we propose a multi-modal transformer model for DOS prediction, called DOSTransformer, which incorporates crystalline materials and energy as heterogeneous input modalities.
* To capture structure-specific interactions between a crystalline material and energy levels, thereby being able to predict DOS that significantly differs according to the crystal's structure, we utilize learnable prompts to inform DOSTransformer of the input crystal system.
* Our extensive experiments on two types of DOS, i.e., Phonon DOS and Electron DOS, demonstrate that DOSTransformer outperforms a wide range of state-of-the-art methods in various real-world scenarios, i.e., in-distribution scenarios and out-of-distribution scenarios.

To the best of our knowledge, this is the first work that considers various energy levels during DOS prediction and introduces prompts for crystal structural systems.

Related Works

### Machine Learning for Materials and Density of States

Traditional materials science research heavily relies on theory, experimentation, and computer simulations, but these approaches are time-consuming, costly, and inefficient, hindering their ability to keep up with the field's development [57]. As an alternative research tool, ML, which requires neither expensive trial and error experiments nor domain expert knowledge, has been attracting a surge of interest from materials scientists [67]. Consequently, various ML methods have been established to learn the relationship between materials and their properties which are usually obtained from ab-initio calculations [11]. Inspired by the recent success of GNNs in biochemistry [19; 48; 25; 33; 34; 32], CGCNN proposes a message-passing framework for crystal structure providing high-accuracy prediction for 8 different materials properties [61]. However, most studies have focused on single-valued properties [29], whereas various spectral properties, such as DOS, are also crucial.

The DOS describes the number of different states at a particular energy level that particles are allowed to occupy, determining various properties of crystalline materials. For example, phonon DOS has a crucial influence in determining the specific heat and vibrational entropy of crystalline materials as well as the interfacial thermal resistance [60], while electron DOS is used to derive the electronic contribution to heat capacity in metals [21] and the effective mass of electrons in charge carriers [18].

Despite the importance of DOS, how to incorporate ML capabilities for _decoding_ such spectral properties is under-explored since the vast majority of methods concentrate only on _encoding_ crystalline materials. Specifically, Chandrasekaran et al. [7] proposes to predict electron DOS based on the hand-crafted fingerprint of each grid point in the crystalline materials, while Del Rio et al. [14] leverages that of each atom in the crystalline materials. Chen et al. [9] predicts phonon DOS by learning the representations of the crystalline materials that are equivariant to 3D rotations, translations, and inversion with Euclidean neural networks [49; 28; 58], achieving high-quality prediction with a small number of training materials with over 64 atom types. In this work, we concentrate on decoding spectral properties regarding the complex relationship between the various energy levels and crystalline materials.

### Multi-modal Transformer, Prompt Tuning, and Positional Encoding

**Multi-modal Transformer.** In recent years, much progress in multi-modal learning has grown rapidly by extracting and aligning the rich information from heterogeneous modalities [2; 6]. Among the various multi-modal learning methods, the multi-modal transformer demonstrates its superior capability in learning multiple modalities [62], by associating the heterogeneous modalities with a cross-attention mechanism. Specifically, MuIT [50] learns the representations directly from unaligned multi-modal streams with multi-modal transformer, while Yao & Wan [65] improves machine translation quality by learning the image-aware text representations. In this paper, we investigate how to decode DOS of a crystalline material by learning the energy-aware crystal representations via a multi-modal transformer.

**Prompt Tuning.** On the other hand, prompt designing has gained significant attention from NLP researchers as an efficient method for fine-tuning large language models (LLMs) for various tasks [39]. There are two main approaches to prompt designing: discrete prompt designing [44] and continuous prompt tuning [37]. While discrete prompt designing involves manually creating task description tokens for LLMs through trial and error, continuous prompt tuning trains learnable prompts in a continuous latent space without requiring expert knowledge. Inspired by the success of continuous prompt tuning in NLP domain, it has been widely adopted in various domains such as computer vision [24] and vision-language [68]. While previous works utilize prompts for efficient fine-tuning of the models, we employ prompts to capture the heterogeneity of various crystal structures.

**Positional Encoding.** In the original proposal of the Transformer architecture, a sinusoidal function was suggested as a method for positional encoding. However, the sinusoidal function may have limitations in terms of learnability and flexibility, which can affect its effectiveness [53]. To address this issue, most pre-trained language models [16; 40] employ learnable vector embeddings as positional representations. By conceptualizing each energy value as the position of a word in a sentence, various energy embeddings in Section 4.2 can be analogous to the positional encoding used in traditional transformers.

## 3 Preliminaries

**Notations.** Let \(\mathcal{G}=(\mathcal{V},\mathcal{A})\) denote a crystalline material, where \(\mathcal{V}=\{v_{1},\ldots,v_{n}\}\) represents the set of atoms, and \(\mathcal{A}\subseteq\mathcal{V}\times\mathcal{V}\) represents the set of edges connecting the atoms in the crystalline material \(\mathcal{G}\). Moreover, \(\mathcal{G}\) is associated with a feature matrix \(\mathbf{X}\in\mathbb{R}^{n\times F}\) and an adjacency matrix \(\mathbf{A}\in\mathbb{R}^{n\times n}\) where \(\mathbf{A}_{ij}=1\) if and only if \((v_{i},v_{j})\in\mathcal{A}\) and \(\mathbf{A}_{ij}=0\) otherwise.

**Task: Density of States Prediction.** Given a set of crystalline materials \(\mathcal{D}_{\mathcal{G}}=\{\mathcal{G}_{1},\mathcal{G}_{2},\ldots,\mathcal{G }_{N}\}\) and a set of energies \(\mathcal{D}_{\mathcal{E}}=\{\mathcal{E}_{1},\mathcal{E}_{2},\ldots,\mathcal{E }_{M}\}\), our goal is to train a model \(\mathcal{M}\) that predicts the DOS of a crystalline material given a set of energies, i.e., \(\mathbf{Y}^{i}=\mathcal{M}(\mathcal{G}_{i},\mathcal{D}_{\mathcal{E}})\), where \(\mathbf{Y}^{i}\in\mathbb{R}^{M}\) is an \(M\) dimensional vector containing the DOS values of a crystalline material \(\mathcal{G}_{i}\) at each energy \(\mathcal{E}_{1},\ldots,\mathcal{E}_{M}\), and \(\mathbf{Y}_{j}^{i}\in\mathbb{R}\) is the DOS value of \(\mathcal{G}_{i}\) at energy level \(\mathcal{E}_{j}\).

## 4 Methodology: DOSTransformer

In this section, we introduce our proposed method, named DOSTransformer, a novel DOS prediction framework that learns the complex relationship between the atoms in the crystalline material and various energy levels by utilizing a cross-attention mechanism of the multi-modal transformer.

### Crystalline Material Encoder

Before modeling the pairwise interaction between the crystalline material and the energies, we first encode the crystalline material with GNNs to learn the representation of each atom, which contains not only the feature information but also the structural information. Formally, given a crystalline material \(\mathcal{G}=(\mathbf{X},\mathbf{A})\), we generate an atom embedding matrix for the crystalline material as follows:

\[\mathbf{H}=\text{GNN}(\mathbf{X},\mathbf{A}), \tag{1}\]

where \(\mathbf{H}\in\mathbb{R}^{n\times d}\) is an atom embedding matrix for \(\mathcal{G}\), whose \(i\)-th row indicates the representation of atom \(v_{i}\), and we stack \(L^{\prime}\) layers of GNNs. Among various GNNs, we adopt graph networks [4] as our crystalline material encoder, which is a generalized and extended version of various GNNs. We provide further details on the GNNs in Appendix C.

### Prompt-based Multi-modal Transformer

After obtaining the atom embedding matrix \(\mathbf{H}\), we aim to capture the relationship between a crystalline material and various energy levels by utilizing the cross-attention layers of a multi-modal transformer [62] with self-attention layers [52]. In a nutshell, we expect the cross-attention layers to integrate the heterogeneous signals from a crystalline material and various energy levels, while self-attention layers aim to integrate the information of material-specific energy representations.

**Cross-Attention Layers.** Specifically, we expect the cross-attention layers to generate the material-specific representation of the energies by repeatedly reinforcing the energy representation with the crystalline materials. To do so, we first introduce a learnable energy embedding matrix \(\mathbf{E}^{0}\in\mathbb{R}^{M\times d}\), whose \(j\)-th row, i.e., \(\mathbf{E}^{0}_{j}\), indicates the embedding of energy \(\mathcal{E}_{j}\in\mathcal{D}_{\mathcal{E}}\). Then, we present cross-modal attention for fusing the information from the crystal structure into various energy levels as follows:

\[\mathbf{E}^{l} =\text{Cross-Attention}(\mathbf{Q}_{\mathbf{E}^{l-1}},\mathbf{K} _{\mathbf{H}},\mathbf{V}_{\mathbf{H}})\in\mathbb{R}^{M\times d} \tag{2}\] \[=\text{Softmax}(\frac{\mathbf{E}^{l-1}\mathbf{H}^{\top}}{\sqrt{d} })\mathbf{H},\]

where \(l=1,\ldots,L_{1}\) indicates the index of the cross-attention layers. In contrast to the conventional Transformer, which introduces learnable weight matrices for query \(\mathbf{Q}\), key \(\mathbf{K}\), and value \(\mathbf{V}\), we directly employ the previously obtained energy embedding matrix \(\mathbf{E}^{l-1}\) as the query matrix, and the atom embedding matrix \(\mathbf{H}\) as the key and value matrices. Based on the above cross-attention mechanism, we obtain the material-specific energy embedding \(\mathbf{E}^{l}\in\mathbb{R}^{M\times d}\) by aggregating the information regarding the atoms in the crystalline material that were important at each energy level. The final material-specific energy embedding matrix \(\mathbf{E}^{L_{1}}\in\mathbb{R}^{M\times d}\) generated by the cross-attention layer reflects the relationship between the atoms in the crystalline material and various energy levels.

**Global Self-Attention.** In addition to cross-attention layers, we propose to enhance the material-specific energy embedding matrix \(\mathbf{E}^{L_{1}}\) by aggregating the information from other energy levels. To do so, we employ variants of self-attention layers in conventional Transformer [52] as follows:

\[\begin{split}\tilde{\mathbf{E}}^{p}&=\text{Self- Attention}(\mathbf{Q}_{\tilde{\mathbf{E}}^{p-1}},\mathbf{K}_{\tilde{\mathbf{E}}^{p-1}}, \mathbf{V}_{\tilde{\mathbf{E}}^{p-1}})\in\mathbb{R}^{M\times d}\\ &=\text{Softmax}(\frac{\tilde{\mathbf{E}}^{p-1}\tilde{\mathbf{E}}^{p -1\top}}{\sqrt{d}})\tilde{\mathbf{E}}^{p-1},\end{split} \tag{3}\]

where \(p=1,\dots,L_{2}\) indicates the index of the self-attention layer. We use an enhanced material-specific energy embedding \(\tilde{\mathbf{E}}^{0}\) as an input of the self-attention, which is obtained by concatenating the material-specific energy embedding \(\mathbf{E}^{L_{1}}\) with the material representation \(\mathbf{g}_{i}\in\mathbb{R}^{d}\), i.e., \(\tilde{\mathbf{E}}^{0}_{j}=\phi_{1}(\mathbf{E}^{glob}_{j})\), where \(\mathbf{E}^{glob}_{j}=(\mathbf{E}^{L_{1}}_{j}||\mathbf{g}_{i})\), \(\phi_{1}:\mathbb{R}^{2d}\rightarrow\mathbb{R}^{d}\), and \(||\) indicates the concatenation operation. Note that \(\mathbf{g}_{i}\) is a sum pooled representation of the material \(\mathcal{G}_{i}\), and \(\mathbf{E}^{L_{1}}_{j}\) indicates the \(j\)-th row of the energy embedding matrix \(\mathbf{E}^{L_{1}}\) computed by Equation 2 where \(j=0,\dots,M\). Although the output of cross-attention layers, i.e., \(\mathbf{E}^{L_{1}}\in\mathbb{R}^{M\times d}\), captures the local atom-level information regarding the material, we also recognize the importance of incorporating the global material-level information \(\mathbf{g}_{i}\) that may not have been fully encoded during the cross-attention phase. Note that the importance of considering the global contextual information for enhancing the learning of dependencies among neural representations has been widely studied in various domains such as computer vision and natural language processing [51, 55, 22]. By additionally considering the global material-level information, we aim to enhance the material-specific representation of the energies more comprehensively.

**System Self-Attention with Crystal System Prompts.** However, solely aggregating the information of material-specific energy embeddings may neglect the crucial influence of the structural properties in crystalline materials, whereas the structural properties play a significant role in determining the unique characteristics and properties of these materials [45]. To illustrate this point, let's examine the case of graphite and diamond, both composed entirely of carbon atoms. In graphite, carbon atoms are arranged in stacked layers, where each carbon atom is bonded to three neighboring carbon atoms in a hexagonal lattice. In contrast, the carbon atom in diamond forms strong covalent bonds with four neighboring carbon atoms, resulting in a tetrahedral arrangement. Even though graphite and diamond are made entirely out of carbon, due to the structural difference, graphite exhibits properties such as electrical conductivity and lubricity due to its layered structure [59, 26], while diamond is renowned for its hardness and thermal conductivity owing to its tightly bonded tetrahedral network [8, 27].

Despite the importance of structural information, effectively incorporating it into the model is not trivial. Naively concatenating the structural information as an input feature of materials is straightforward but may lead to interference, hindering the model's ability to learn and generalize knowledge across different crystal structures. To this end, we propose to provide additional information to the self-attention layers with learnable prompts, which indicate the structural information of the crystalline materials. Specifically, we adopt learnable prompts \(\mathbf{P}\in\mathbb{R}^{7\times d_{p}}\), whose \(k\)-th row \(\mathbf{P}_{k}\) represents one of the seven widely known crystal systems: Cubic, Hexagonal, Tetragonal, Trigonal, Orthorhombic, Monoclinic, and Triclinic. Then, given a crystalline material \(\mathcal{G}_{i}\), whose crystal system is given as \(k\), we incorporate the learnable prompts into the material-specific energy embedding \(\tilde{\mathbf{E}}^{0}\) as an input

Figure 2: Overall model architecture and attention layers in prompt-based multi-modal Transformer.

of the self-attention, which is obtained as follows: \(\widetilde{\mathbf{E}}_{j}^{0}=\phi_{2}(\mathbf{E}_{j}^{sys})\), where \(\mathbf{E}_{j}^{sys}=(\mathbf{E}_{j}^{L_{1}}||\mathbf{g}_{i}||\mathbf{P}_{k})\) and \(\phi_{2}:\mathbb{R}^{2d+d_{p}}\rightarrow\mathbb{R}^{d}\). By doing so, we obtain high-level energy features that contain the crystal structural system information (i.e., \(\widetilde{\mathbf{E}}^{L_{2}}\)), while keeping the low-level features to share the knowledge across all the materials (i.e., \(\mathbf{E}^{L_{1}}\)). In Section 5.4, we demonstrate the effectiveness of our prompt-based approach compared to other variants.

Besides the self-attention layers, we use additional cross-attention layers on top of the self-attention layers, whose input energy embedding \(\mathbf{E}^{0}\) in Equation 2 is given as \(\widetilde{\mathbf{E}}^{L_{2}}\), and output the final material-specific energy embeddings \(\mathbf{E}^{L_{3}}\), where \(L_{3}\) is the number of additional cross-attention layers. By doing so, the model extracts the enhanced relationships between the crystalline materials and energy levels regarding the crystal structural system information.

### Energy Decoder

After obtaining the final material-specific energy embedding matrix \(\mathbf{E}^{L_{3},i}\) of a crystalline material \(\mathcal{G}_{i}\), the DOS value at each energy level \(\mathcal{E}_{j}\), i.e., \(\hat{\mathbf{Y}}_{j}^{i}\), is given as follows: \(\hat{\mathbf{Y}}_{j}^{i}=\phi_{pred}(\mathbf{E}_{j}^{L_{3},i})\), where \(\phi_{pred}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{1}\) is a parameterized MLP for predicting DOS. While previous approaches directly predict DOS from the material representation \(\mathbf{g}_{i}\) using a function \(\phi_{pred}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{M}\), \(\mathsf{DOSTransformer}\) takes a different approach by making pointwise predictions that align with the nature of DOS calculation, i.e., DOS determines the general distribution of states as a function of energy To the best of our knowledge, \(\mathsf{DOSTransformer}\) is the first work that predicts DOS values in a pointwise manner at each energy level, and we later show in Section 5 that such an approach further enhances the performance of not only \(\mathsf{DOSTransformer}\) but also existing models.

### Model Training and Inference

\(\mathsf{DOSTransformer}\) is trained to minimize the root mean squared error (RMSE) loss \(\mathcal{L}\) between the predicted DOS value \(\hat{\mathbf{Y}}_{j}^{i}\) and the ground truth DOS value \(\mathbf{Y}_{j}^{i}\), i.e., \(\mathcal{L}=\frac{1}{N\cdot M}\sum_{i=1}^{N}\sum_{j=1}^{M}\sqrt{(\hat{\mathbf{ Y}}_{j}^{i}-\mathbf{Y}_{j}^{i})^{2}}\). More specifically, \(\mathsf{DOSTransformer}\) is trained by combining two distinct RMSE losses with balancing term \(\beta\), i.e., \(\mathcal{L}^{total}=\mathcal{L}^{glob}+\beta\cdot\mathcal{L}^{sys}\), where \(\mathcal{L}^{glob}\) and \(\mathcal{L}^{sys}\) are obtained by utilizing \(\mathbf{E}^{L_{3},glob}\) and \(\mathbf{E}^{L_{3},sys}\), respectively. By doing so, \(\mathsf{DOSTransformer}\) extracts the relationship information that is shared among the distinct crystal systems and within a crystal system, respectively. For inference, we utilize the model prediction obtained based on \(\mathbf{E}^{L_{3},sys}\). The overall model architecture is depicted in Figure 2.

## 5 Experiments

### Experimental Setup

**Datasets.** We use two datasets to comprehensively evaluate the performance of \(\mathsf{DOSTransformer}\), i.e., Phonon DOS and Electron DOS. We use the **Phonon DOS** dataset following the instructions of the official Github repository 3 of a previous work [9]. For **Electron DOS** dataset, we collect crystalline materials and their electron DOS data from Materials Project (MP) website 4. We provide further detailed preprocessing procedures and statistics on each dataset in Appendix A.

Footnote 3: [https://github.com/zhantaochen/phonondos_e3nn](https://github.com/zhantaochen/phonondos_e3nn)

Footnote 4: [https://materialsproject.org/](https://materialsproject.org/)

**Methods Compared.** We mainly compare \(\mathsf{DOSTransformer}\) to recently proposed state-of-the-art method, i.e., E3NN [9], which utilizes the Euclidean network for encoding material representation. We also compare \(\mathsf{DOSTransformer}\) to simple baseline methods, i.e., MLP and Graph Network [4], which predicts the entire DOS sequence directly from the learned representation of the materials without regarding the energies during training. Moreover, to evaluate the effectiveness of the transformer layer that considers the relationship between the atoms and various energy levels, we integrate energy embeddings into baseline methods for DOS prediction by concatenating the energy embeddings to the material representation. We provide more details on the implementation of \(\mathsf{DOSTransformer}\) and compared methods in Appendix C and D, respectively.

### In-Distribution Evaluation

**Evaluation Protocol.** In this section, we evaluate the model performances under in-distribution scenarios. While we evaluate \(\mathsf{DOSTransformer}\) with given data splits in a previous work [9] for **Phonon DOS** dataset, we randomly split the **Electron DOS** dataset into train/valid/test of 80/10/10%. Note that while all measures are reported in the original scale, we report MSE values for DOS prediction multiplied by a factor of 10 for clear interpretation during all experiments.

**Experimental Results.** In Table 1, we have the following observations: **1)** Comparing the baseline methods that overlook the energy levels for DOS prediction (i.e., Energy \(\mathbf{\mathcal{X}}\)) with their counterparts that incorporate both the energy levels and the crystal structure as heterogeneous input modalities through the energy embeddings (i.e., Energy \(\mathbf{\mathcal{missing}}\)), we find out that using the energy embeddings consistently enhances the model performance. This indicates that making pointwise predictions on each energy level is crucial for DOS prediction as discussed in Section 4.3, which also aligns with the domain knowledge of materials science, i.e., DOS determines the general distribution of states as a function of energy. **2)** However, we observe that E3NN shows a relatively small performance gain compared with other methods after incorporating the energy information. This is because the integration of the energy embedding may confound the model to learn the proper equivariance of the materials. **3)** On the other hand, \(\mathsf{DOSTransformer}\) outperforms previous methods that do not consider the complex relationships between the atoms in materials and various energy levels via the various attention mechanisms. This again implies that a naive integration of the energy information into previous models cannot fully benefit from the energy information. We also provide qualitative analysis on the obtained DOS in Section 5.5.

**Physical Validity of DOS.** In addition to the accuracy of DOS prediction, we assess the physical validity of the model-predicted DOS by deriving various physical properties, i.e., bulk modulus, band gap, and Fermi energy, of materials, based on the model-predicted DOS. To do so, given the DFT-calculated DOS, which is considered the ground-truth DOS based on which accurate derivation of various physical properties is possible, we train an MLP with a non-linearity in each layer to predict the properties of a crystal structure. Then, based on the obtained MLP weights, we predict the material properties given the model-predicted DOS as the input and calculate the MSE for each material property. We provide further details on the experimental setting in Appendix B. In Table 1, we observe that DOS predicted by \(\mathsf{DOSTransformer}\) shows the superiority of predicting physical properties of materials, indicating that \(\mathsf{DOSTransformer}\) not only achieves high accuracy but also produces physically valid predictions.

### Out-of-Distribution Evaluation

In this section, we evaluate the model performances under out-of-distribution (OOD) scenarios. It is well recognized that existing DFT calculation-based databases have limitations in terms of

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**Phonon DOS**} & \multicolumn{3}{c}{**Electron DOS**} & \multicolumn{3}{c}{**Physical Properties (MSE)**} \\ \cline{2-11}  & MSE & MAE & \(R^{2}\) & MSE & MAE & \(R^{2}\) & Bulk M. & Band G. & Ferm. E. \\ \hline \multicolumn{11}{l}{**Energy \(\mathbf{\mathcal{missing}}\)**} \\ \hline MLP & 0.309 & 0.106 & 0.576 & 0.347 & 0.130 & 0.487 & 0.695 & 1.597 & 4.650 \\  & (0.016) & (0.003) & (0.016) & (0.015) & (0.003) & (0.029) & (0.031) & (0.171) & (0.202) \\ Graph Network & 0.259 & 0.099 & 0.638 & 0.264 & 0.103 & 0.613 & 0.688 & 1.457 & 3.362 \\  & (0.009) & (0.001) & (0.013) & (0.005) & (0.000) & (0.008) & (0.060) & (0.051) & (0.189) \\ E3NN & 0.210 & 0.077 & 0.705 & 0.296 & 0.109 & 0.552 & 0.504 & 0.896 & 2.925 \\  & (0.004) & (0.001) & (0.007) & (0.005) & (0.001) & (0.013) & (0.033) & (0.093) & (0.111) \\ \hline \multicolumn{11}{l}{**Energy \(\mathbf{\mathcal{missing}}\)**} \\ \hline MLP & 0.251 & 0.099 & 0.652 & 0.341 & 0.128 & 0.499 & 0.521 & 1.409 & 4.372 \\  & (0.004) & (0.001) & (0.006) & (0.011) & (0.002) & (0.012) & (0.011) & (0.182) & (0.067) \\ Graph Network & 0.226 & 0.092 & 0.685 & 0.240 & 0.099 & 0.650 & 0.543 & 1.263 & 3.080 \\  & (0.007) & (0.002) & (0.011) & (0.005) & (0.001) & (0.005) & (0.085) & (0.107) & (0.100) \\ E3NN & 0.200 & 0.074 & 0.724 & 0.291 & 0.114 & 0.564 & 0.451 & 1.605 & 3.777 \\  & (0.001) & (0.001) & (0.002) & (0.000) & (0.000) & (0.008) & (0.023) & (0.231) & (0.175) \\ \hline \multirow{2}{*}{\(\mathsf{DOSTransformer}\)} & **0.191** & **0.071** & **0.733** & **0.221** & **0.089** & **0.679** & **0.427** & **0.461** & **2.337** \\  & (0.003) & (0.002) & (0.004) & (0.006) & (0.001) & (0.006) & (0.024) & (0.019) & (0.094) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Overall model performance under in-distribution scenarios (Bulk M. : Bulk Modulus / Band G. : Band Gap / Ferm. E. : Fermi Energy).

coverage, as they often focus on specific types of materials or structural archetypes, resulting in biased distributions [13, 30, 20]. Therefore, it is essential to evaluate the model's performance in OOD scenarios to assess its real-world applicability and generalizability beyond the limitations of the available databases [35].

**Evaluation Protocol.** To do so, we evaluate the model performance on the crystal structures that 1) contain a different number of atom species with the training set (i.e., # Atom species), and 2) belong to different crystal systems that were not included in the training set (i.e., Crystal System). In both scenarios, training data primarily consist of relatively simple structures compared to those in the test data. We provide further details on data split and evaluation in Appendix B.

**Experimental Results.** In Table 2, we again observe that utilizing energy embeddings consistently brings performance gain to all baseline models. Especially, Graph Network significantly benefits from energy embeddings compared to in-distribution scenarios (See Table 1). We attribute this to the inherent restrictive inductive biases in Graph Network [4], i.e., Graph Network heavily relies on the structural information of materials. Specifically, as the structure of the materials totally varies in the OOD scenarios, which makes the prediction of the whole DOS sequence challenging, incorporating the energy embeddings becomes especially helpful in the OOD scenarios for Graph Network. Moreover, DOSTransformer also alleviates the restrictive inductive bias of Graph Networks by elaborately modeling the complex relationships between energies and materials. It is worth noting that similar observations have been made by comparing Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) in the computer vision domain [1, 66]. In conclusion, we argue that incorporating energy information during model training is crucial not only for in-distribution scenarios but also for OOD scenarios, demonstrating the applicability of DOSTransformer in real-world applications. For a more detailed analysis, please refer to Appendix E.1.

**Fine-tuning on Few Materials from Complex Crystal Systems.** In addition to evaluating the performance under the OOD scenarios, we verify how well the models can make predictions for materials from complex crystal systems with only few training samples, which is a practical situation in reality. For this, we sampled a small subset of of materials (i.e., 10%) from the test set used in OOD scenarios of crystal systems, and fine-tune the models that are already trained on the training set. As for DOSTransformer, we fine-tune the model using two different approaches: i) fine-tuning all model parameters (referred to as "All"), and ii) tuning only the parameters of the prompts and the energy decoder while keeping all other model parameters frozen (referred to as "Only Prompt"). In Table 3, we have the following observations: **1)** As we expected, the additional fine-tuning step achieves performance gain for all models, while it was marginal due to a limited number of materials used for fine-tuning. **2)** Only fine-tuning the prompts of DOSTransformer achieved more performance gain compared to fine-tuning the whole model. This is because while fine-tuning the whole model on a small subset of materials may easily incur overfitting, fine-tuning only prompts enables the model to additionally learn from few new samples while maintaining the knowledge obtained previously. In conclusion, we believe that the proposed prompt tuning approach can have a significant impact in the

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{5}{c}{**Out-of-Distribution**} \\ \cline{2-6}  & \multicolumn{2}{c}{**\# Atom Species**} & \multicolumn{2}{c}{**Crystal System**} \\ \cline{2-6}  & MSE & MAE & \(R^{2}\) & MSE & MAE & \(R^{2}\) \\ \hline \hline
**Energy ✗** & & & & & \\ \hline MLP & 0.545 & 0.161 & 0.250 & 0.455 & 0.149 & 0.437 \\  & (0.005) & (0.001) & (0.003) & (0.006) & (0.002) & (0.007) \\ Graph Network & 0.460 & 0.144 & 0.381 & 0.407 & 0.132 & 0.505 \\  & (0.015) & (0.003) & (0.002) & (0.004) & (0.001) & (0.006) \\ E3NN & 0.541 & 0.154 & 0.231 & 0.421 & 0.134 & 0.483 \\  & (0.007) & (0.001) & (0.005) & (0.004) & (0.001) & (0.007) \\ \hline
**Energy ✓** & & & & & \\ \hline MLP & 0.545 & 0.159 & 0.260 & 0.455 & 0.149 & 0.445 \\  & (0.005) & (0.001) & (0.016) & (0.004) & (0.001) & (0.007) \\ Graph Network & 0.455 & 0.144 & 0.388 & 0.384 & 0.129 & 0.534 \\  & (0.002) & (0.002) & (0.004) & (0.012) & (0.004) & (0.014) \\  & 0.534 & 0.152 & 0.237 & 0.420 & 0.133 & 0.483 \\  & (0.013) & (0.001) & (0.017) & (0.007) & (0.001) & (0.008) \\ \hline \multirow{2}{*}{DOSTransformer} & **0.454** & **0.136** & **0.399** & **0.373** & **0.122** & **0.552** \\  & (0.008) & (0.001) & (0.017) & (0.006) & (0.001) & (0.006) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Overall model performance in OOD scenarios.

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Model** & MSE & MAE & \(R^{2}\) \\ \hline \hline
**Energy ✗** & & & \\ \hline MLP & 0.401 & 0.137 & 0.510 \\  & (0.017) & (0.003) & (0.021) \\ Graph Network & 0.394 & 0.131 & 0.519 \\  & (0.007) & (0.002) & (0.010) \\ E3NN & 0.414 & 0.133 & 0.490 \\  & (0.006) & (0.001) & (0.007) \\ \hline
**Energy ✓** & & & \\ \hline MLP & 0.394 & 0.136 & 0.519 \\  & (0.008) & (0.002) & (0.009) \\ Graph Network & 0.382 & 0.130 & 0.533 \\  & (0.003) & (0.000) & (0.002) \\ E3NN & 0.417 & 0.133 & 0.487 \\  & (0.004) & (0.001) & (0.007) \\ \hline \hline \multicolumn{3}{l}{**DOSTransformer**} \\ \hline All & 0.365 & **0.122** & 0.558 \\  & (0.005) & (0.001) & (0.008) \\ Only Prompt & **0.355** & **0.122** & **0.570** \\  & (0.008) & (0.001) & (0.010) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Fine-tuning on OOD systems.

field of materials science beyond simple DOS prediction, where the existing databases exhibit a bias towards dominant types of materials [13; 30; 20]. We further explore different proportions beyond the 10% of the materials used for fine-tuning, as detailed in the Appendix E.3.

### Model Analysis

**Ablation Studies.** To verify the benefit of each component of DOS-Transformer, we conduct ablation studies under in-distribution scenarios in Figure 3. We observe that utilizing only either one of the global and system losses deteriorates the model performance. This is because jointly optimizing the losses incentivizes the model to extract the relational information that is shared across the crystal systems and within a crystal system. On the other hand, it is worth noting that DOSTransformer with only one of the losses still outperforms baseline models shown in Table 1, indicating that modeling complex relationships between materials and various energies is crucial in DOS prediction.

**Sensitivity Analysis.** To verify the robustness of DOSTransformer, we measure the model performance by varying \(\beta\), which is introduced to control the effect of the crystal system loss, i.e. \(\mathcal{L}^{sys}\), in Section 4.4. In Figure 4, DOSTransformer shows robustness over various levels of \(\beta\), consistently outperforming baseline models in Table 1. This verifies that DOSTransformer can be easily trained without an expensive hyperparameter tuning, further demonstrating the practicality of DOSTransformer.

**Crystal System Prompts.** As discussed in Section 4.2, it is not trivial to effectively incorporate the crystal system information into the model. To evaluate the effectiveness of our proposed prompt-based approach, we explore different approaches for incorporating the crystal system information into the model. We make modifications to determine where and how to inject this information. In terms of the location, we consider two options: injecting the information into the input atoms (i.e., Atom Feat.) and injecting it before the self-attention layers (i.e., Before SA). Regarding the method of injection, we explore two strategies: one-hot encoding of crystal systems and the use of learnable crystal system prompts. In Table 4, we have following observations: **1)** Naively incorporating prompts into atom feature even perform worse than the model without using crystal system (see _"w/o \(\mathcal{L}^{sys}\)"_ in Figure 3), demonstrating the importance of an elaborate design choice for injecting crystal system information. **2)** We observe that our approach, i.e. Before SA and prompt, outperforms all other possible choices, indicating that DOSTransformer successfully incorporates crystal system information during training. We also examine the benefits of injecting crystal system information into baseline methods in Appendix E.2.

### Qualitative Analysis

In this section, we provide a qualitative analysis of the predicted DOS by mainly comparing it to the DFT-calculated (i.e., Ground Truth) DOS and our main baseline (i.e., E3NN). In Figure 5 (a), which represents the predicted DOS of materials not containing transitional metals, both E3NN and DOSTransformer successfully capture the overall trend of the DOS for several materials (e.g., mp-982366, mp-1009129, and mp-16378). However, DOSTransformer shows a much more precise prediction that closely aligns with the ground truth DOS, providing even more useful information beyond the shape of DOS. For example, peak points represent regions of high density and are likely to be strongly influenced when materials undergo changes in property, and thus represents the probabilistically important energy regions of the materials in the process of material discovery. Notably, our model better captures the peak points in the ground truth DOS compared to E3NN, demonstrating the applicability of DOSTransformer-predicted DOS for material discovery process.

\begin{table}
\begin{tabular}{c|c c c c} \hline \hline  & \multicolumn{2}{c}{**Method**} & \multicolumn{2}{c}{**Electron DOS**} \\ \cline{3-5}  & \multicolumn{2}{c}{One-hot} & Prompt & MSE & MAE \\ \hline \multirow{4}{*}{**Dataset**} & \multirow{2}{*}{Atom Feat.} & \multirow{2}{*}{✓} & \multirow{2}{*}{✓} & 0.222 & 0.089 \\  & & & & (0.004) & 0.001 \\  & & & & 0.227 & 0.090 \\  & & & & (0.005) & 0.001 \\ \cline{2-5}  & \multirow{2}{*}{Before SA} & \multirow{2}{*}{✓} & \multirow{2}{*}{✓} & 0.226 & 0.090 \\  & & & & (0.005) & 0.001 \\  & & & & **0.221** & **0.089** \\  & & & & (0.006) & 0.001 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparing various crystal system information injection approaches.

Figure 4: Sensitivity analysis.

Figure 3: Ablation studies.

On the other hand, Figure 5 (b) shows the DOS prediction for materials containing transition materials. Although \(\mathsf{DOSTransformer}\) provides more reliable prediction, we observe that the prediction errors of both models get larger compared to the materials that do not contain transition metals shown in Figure 5 (a). This can be attributed to the inherent complexity of physical properties in materials containing transition metals, as discussed in Section 6. Therefore, for our future work, we plan to design expert models in which each expert is responsible for materials with and without transition metals, to achieve more refined and accurate predictions of the DOS. This approach would enable a more comprehensive and elaborate analysis of the DOS in different material compositions.

## 6 Limitations & Future Work

It is widely known that materials containing transitional metals inherently have complex physical properties compared to other materials. As a result, the density functionals that yield optimal performance in DFT calculations differ between materials with and without transitional metals [12, 46]. However, despite the distinction, most ML-based DOS prediction research, including \(\mathsf{DOSTransformer}\), have attempted to encode the properties of both material types into a single model [9, 17], which may provide conflicting signals to the model. Therefore, as a future work, we plan to address this issue by developing specialized approaches that work well on both material types, e.g., utilizing expert models [41] in which an expert is assigned to each material type and later combined to perform well on both material types. We expect this approach to result in the robust and reliable prediction of DOS for both types of materials, enhancing the potential impact in the materials science field and broadening our understanding of material properties.

## 7 Conclusion

In this paper, we propose \(\mathsf{DOSTransformer}\), which predicts DOS of crystalline materials with various energy levels by following the nature of DOS calculation. Specifically, \(\mathsf{DOSTransformer}\) associates the heterogeneous information from materials and energy levels with cross-attention and self-attention layers. Moreover, the crystal system prompts are introduced to effectively train the model to learn the relational information that is shared across all the crystal systems and within a crystal system. By doing so, \(\mathsf{DOSTransformer}\) outperforms previous works in predicting two types of DOS, i.e., phonon DOS and electron DOS, in various scenarios, i.e., in-distribution scenarios and out-of-distribution scenarios. Extensive experiments verify that incorporating energy information is crucial in predicting the DOS of a crystal structure for real-world application and \(\mathsf{DOSTransformer}\) effectively utilizes the crystal structural information with crystal system prompts.

Figure 5: Qualitative Analysis.

## Acknowledgements

This work was supported by the core KRICT project from the Korea Research Institute of Chemical Technology (SI2051-10).

## References

* [1]Bai, Y., Mei, J., Yuille, A. L., and Xie, C. Are transformers more robust than cnns? _Advances in Neural Information Processing Systems_, 34:26831-26843, 2021.
* [2] Baltrusaitis, T., Ahuja, C., and Morency, L.-P. Multimodal machine learning: A survey and taxonomy. _IEEE transactions on pattern analysis and machine intelligence_, 41(2):423-443, 2018.
* [3] Banjade, H. R., Hauri, S., Zhang, S., Ricci, F., Gong, W., Hautier, G., Vucetic, S., and Yan, Q. Structure motif-centric learning framework for inorganic crystalline systems. _Science advances_, 7(17):eabf1754, 2021.
* [4] Battaglia, P. W., Hamrick, J. B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V., Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., et al. Relational inductive biases, deep learning, and graph networks. _arXiv preprint arXiv:1806.01261_, 2018.
* [5] Bayoudh, K., Hamdaoui, F., and Mtibaa, A. Hybrid-covid: a novel hybrid 2d/3d cnn based on cross-domain adaptation approach for covid-19 screening from chest x-ray images. _Physical and engineering sciences in medicine_, 43(4):1415-1431, 2020.
* [6] Bayoudh, K., Knani, R., Hamdaoui, F., and Mtibaa, A. A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets. _The Visual Computer_, 38(8):2939-2970, 2022.
* [7] Chandrasekaran, A., Kamal, D., Batra, R., Kim, C., Chen, L., and Ramprasad, R. Solving the electronic structure problem with machine learning. _npj Computational Materials_, 5(1):1-7, 2019.
* [8] Che, J., Cagin, T., Deng, W., and Goddard III, W. A. Thermal conductivity of diamond and related materials from molecular dynamics simulations. _The Journal of Chemical Physics_, 113(16):6888-6900, 2000.
* [9] Chen, Z., Andrejevic, N., Smidt, T., Ding, Z., Xu, Q., Chi, Y.-T., Nguyen, Q. T., Alatas, A., Kong, J., and Li, M. Direct prediction of phonon density of states with euclidean neural networks. _Advanced Science_, 8(12):2004214, 2021.
* [10] Choudhary, K. and DeCost, B. Atomistic line graph neural network for improved materials property predictions. _npj Computational Materials_, 7(1):185, 2021.
* [11] Choudhary, K., DeCost, B., Chen, C., Jain, A., Tavazza, F., Cohn, R., Park, C. W., Choudhary, A., Agrawal, A., Billinge, S. J., et al. Recent advances and applications of deep learning methods in materials science. _npj Computational Materials_, 8(1):1-26, 2022.
* [12] Cramer, C. J. and Truhlar, D. G. Density functional theory for transition metals and transition metal chemistry. _Physical Chemistry Chemical Physics_, 11(46):10757-10816, 2009.
* [13] De Breuck, P.-P., Evans, M. L., and Rignanese, G.-M. Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on modnet. _Journal of Physics: Condensed Matter_, 33(40):404002, 2021.
* [14] Del Rio, B. G., Kuenneth, C., Tran, H. D., and Ramprasad, R. An efficient deep learning scheme to predict the electronic structure of materials and molecules: The example of graphene-derived allotropes. _The Journal of Physical Chemistry A_, 124(45):9496-9502, 2020.
* [15] Deringer, V. L., Bernstein, N., Csanyi, G., Ben Mahmoud, C., Ceriotti, M., Wilson, M., Drabold, D. A., and Elliott, S. R. Origins of structural and electronic transitions in disordered silicon. _Nature_, 589(7840):59-64, 2021.

* [16] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_, 2018.
* [17] Fung, V., Ganesh, P., and Sumpter, B. G. Physically informed machine learning prediction of electronic density of states. _Chemistry of Materials_, 2022.
* [18] Garrity, K. F. First-principles search for n-type oxide, nitride, and sulfide thermoelectrics. _Physical Review B_, 94(4):045122, 2016.
* [19] Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. Neural message passing for quantum chemistry. In _International conference on machine learning_, pp. 1263-1272. PMLR, 2017.
* [20] Griffiths, R.-R., Schwaller, P., and Lee, A. A. Dataset bias in the natural sciences: a case study in chemical reaction prediction and synthesis design. _arXiv preprint arXiv:2105.02637_, 2021.
* [21] Hamaoui, G., Horny, N., Hua, Z., Zhu, T., Robillard, J.-F., Fleming, A., Ban, H., and Chirtoc, M. Electronic contribution in heat transfer at metal-semiconductor and metal silicide-semiconductor interfaces. _Scientific reports_, 8(1):1-9, 2018.
* [22] Hatamizadeh, A., Yin, H., Kautz, J., and Molchanov, P. Global context vision transformers. _arXiv preprint arXiv:2206.09959_, 2022.
* [23] Illas, F., de PR Moreira, I., Bofill, J., and Filatov, M. Extent and limitations of density-functional theory in describing magnetic systems. _Physical Review B_, 70(13):132414, 2004.
* [24] Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In _Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII_, pp. 709-727. Springer, 2022.
* [25] Jiang, D., Wu, Z., Hsieh, C.-Y., Chen, G., Liao, B., Wang, Z., Shen, C., Cao, D., Wu, J., and Hou, T. Could graph neural networks learn better molecular representation for drug discovery? a comparison study of descriptor-based and graph-based models. _Journal of cheminformatics_, 13(1):1-23, 2021.
* [26] Jorio, A., Dresselhaus, G., and Dresselhaus, M. S. _Carbon nanotubes: advanced topics in the synthesis, structure, properties and applications_, volume 111. Springer, 2008.
* [27] Kidalov, S. V. and Shakhov, F. M. Thermal conductivity of diamond composites. _Materials_, 2 (4):2467-2495, 2009.
* [28] Kondor, R., Lin, Z., and Trivedi, S. Clebsch-gordan nets: a fully fourier space spherical convolutional neural network. _Advances in Neural Information Processing Systems_, 31, 2018.
* [29] Kong, S., Ricci, F., Guevarra, D., Neaton, J. B., Gomes, C. P., and Gregoire, J. M. Density of states prediction for materials discovery via contrastive learning from probabilistic embeddings. _Nature communications_, 13(1):1-12, 2022.
* [30] Kumagai, M., Ando, Y., Tanaka, A., Tsuda, K., Katsura, Y., and Kurosaki, K. Effects of data bias on machine-learning-based material discovery using experimental property data. _Science and Technology of Advanced Materials: Methods_, 2(1):302-309, 2022.
* [31] Lee, J., Seko, A., Shitara, K., Nakayama, K., and Tanaka, I. Prediction model of band gap for inorganic compounds by combination of density functional theory calculations and machine learning techniques. _Physical Review B_, 93(11):115104, 2016.
* [32] Lee, J., Kim, S., Hyun, D., Lee, N., Kim, Y., and Park, C. Deep single-cell rna-seq data clustering with graph prototypical contrastive learning. _Bioinformatics_, 39(6):btd342, 2023.
* [33] Lee, N., Hyun, D., Na, G. S., Kim, S., Lee, J., and Park, C. Conditional graph information bottleneck for molecular relational learning. _arXiv preprint arXiv:2305.01520_, 2023.
* [34] Lee, N., Yoon, K., Na, G. S., Kim, S., and Park, C. Shift-robust molecular relational learning with causal substructure. _arXiv preprint arXiv:2305.18451_, 2023.

* [35] Li, K., DeCost, B., Choudhary, K., Greenwood, M., and Hattrick-Simpers, J. A critical examination of robustness and generalizability of machine learning prediction of materials properties. _npj Computational Materials_, 9(1):55, 2023.
* [36] Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., and Chang, K.-W. Visualbert: A simple and performant baseline for vision and language. _arXiv preprint arXiv:1908.03557_, 2019.
* [37] Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. _arXiv preprint arXiv:2101.00190_, 2021.
* [38] Lin, T.-Y., Cui, Y., Belongie, S., and Hays, J. Learning deep representations for ground-to-aerial geolocalization. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pp. 5007-5015, 2015.
* [39] Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. _ACM Computing Surveys_, 55(9):1-35, 2023.
* [40] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_, 2019.
* [41] Masoudnia, S. and Ebrahimpour, R. Mixture of experts: a literature survey. _The Artificial Intelligence Review_, 42(2):275, 2014.
* [42] Petretto, G., Dwaraknath, S., PC Miranda, H., Winston, D., Giantomassi, M., Van Setten, M. J., Gonze, X., Persson, K. A., Hautier, G., and Rignanese, G.-M. High-throughput density-functional perturbation theory phonons for inorganic materials. _Scientific data_, 5(1):1-12, 2018.
* [43] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. Zero-shot text-to-image generation. In _International Conference on Machine Learning_, pp. 8821-8831. PMLR, 2021.
* [44] Schick, T. and Schutze, H. Exploiting cloze questions for few shot text classification and natural language inference. _arXiv preprint arXiv:2001.07676_, 2020.
* [45] Shandiz, M. A. and Gauvin, R. Application of machine learning methods for the prediction of crystal system of cathode materials in lithium-ion batteries. _Computational Materials Science_, 117:270-278, 2016.
* [46] Sim, E., Song, S., and Burke, K. Quantifying density errors in dft. _The journal of physical chemistry letters_, 9(22):6385-6392, 2018.
* [47] Singh, A. K., Montoya, J. H., Gregoire, J. M., and Persson, K. A. Robust and synthesizable photocatalysts for co2 reduction: a data-driven materials discovery. _Nature communications_, 10 (1):1-9, 2019.
* [48] Stokes, J. M., Yang, K., Swanson, K., Jin, W., Cubillos-Ruiz, A., Donghia, N. M., MacNair, C. R., French, S., Carfrae, L. A., Bloom-Ackermann, Z., et al. A deep learning approach to antibiotic discovery. _Cell_, 180(4):688-702, 2020.
* [49] Thomas, N., Smidt, T., Kearnes, S., Yang, L., Li, L., Kohlhoff, K., and Riley, P. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. _arXiv preprint arXiv:1802.08219_, 2018.
* [50] Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., and Salakhutdinov, R. Multimodal transformer for unaligned multimodal language sequences. In _Proceedings of the conference. Association for Computational Linguistics. Meeting_, volume 2019, pp. 6558. NIH Public Access, 2019.
* [51] Tu, Z., Liu, Y., Lu, Z., Liu, X., and Li, H. Context gates for neural machine translation. _Transactions of the Association for Computational Linguistics_, 5:87-99, 2017.

* [52] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. _Advances in neural information processing systems_, 30, 2017.
* [53] Wang, G., Lu, Y., Cui, L., Lv, T., Florencio, D., and Zhang, C. A simple yet effective learnable positional encoding method for improving document transformer model. In _Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022_, pp. 453-463, 2022.
* [54] Wang, L., Li, Y., and Lazebnik, S. Learning deep structure-preserving image-text embeddings. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pp. 5005-5013, 2016.
* [55] Wang, L., Tu, Z., Way, A., and Liu, Q. Exploiting cross-sentence context for neural machine translation. _arXiv preprint arXiv:1704.04347_, 2017.
* [56] Ward, L., Agrawal, A., Choudhary, A., and Wolverton, C. A general-purpose machine learning framework for predicting properties of inorganic materials. _npj Computational Materials_, 2(1):1-7, 2016.
* [57] Wei, J., Chu, X., Sun, X.-Y., Xu, K., Deng, H.-X., Chen, J., Wei, Z., and Lei, M. Machine learning in materials science. _InfoMat_, 1(3):338-358, 2019.
* [58] Weiler, M., Geiger, M., Welling, M., Boomsma, W., and Cohen, T. S. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. _Advances in Neural Information Processing Systems_, 31, 2018.
* [59] Wissler, M. Graphite and carbon powders for electrochemical applications. _Journal of power sources_, 156(2):142-150, 2006.
* [60] Wu, Y.-J., Fang, L., and Xu, Y. Predicting interfacial thermal resistance by machine learning. _npj Computational Materials_, 5(1):1-8, 2019.
* [61] Xie, T. and Grossman, J. C. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. _Physical review letters_, 120(14):145301, 2018.
* [62] Xu, P., Zhu, X., and Clifton, D. A. Multimodal learning with transformers: a survey. _arXiv preprint arXiv:2206.06488_, 2022.
* [63] Yan, K., Liu, Y., Lin, Y., and Ji, S. Periodic graph transformers for crystal material property prediction. _Advances in Neural Information Processing Systems_, 35:15066-15080, 2022.
* [64] Yan, Q., Yu, J., Suram, S. K., Zhou, L., Shinde, A., Newhouse, P. F., Chen, W., Li, G., Persson, K. A., Gregoire, J. M., et al. Solar fuels photoanode materials discovery by integrating high-throughput theory and experiment. _Proceedings of the National Academy of Sciences_, 114(12):3040-3043, 2017.
* [65] Yao, S. and Wan, X. Multimodal transformer for multimodal machine translation. In _Proceedings of the 58th annual meeting of the association for computational linguistics_, pp. 4346-4350, 2020.
* [66] Zhang, C., Zhang, M., Zhang, S., Jin, D., Zhou, Q., Cai, Z., Zhao, H., Liu, X., and Liu, Z. Delving deep into the generalization of vision transformers under distribution shifts. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pp. 7277-7286, 2022.
* [67] Zhong, X., Gallagher, B., Liu, S., Kailkhura, B., Hiszpanski, A., and Han, T. Explainable machine learning in materials science. _npj Computational Materials_, 8(1):1-19, 2022.
* [68] Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to prompt for vision-language models. _International Journal of Computer Vision_, 130(9):2337-2348, 2022.
* [69] Zhuo, Y., Mansouri Tehrani, A., and Brgoch, J. Predicting the band gaps of inorganic solids by machine learning. _The journal of physical chemistry letters_, 9(7):1668-1673, 2018.

**Supplementary Material for**

**Density of States Prediction of Crystalline Materials**

**via Prompt-guided Multi-Modal Transformer**

## Appendix A Datasets

In this section, we provide further details on the dataset used for experiments.

### Phonon DOS

We use the **Phonon DOS** dataset following the instructions of the official Github repository5 of a previous work [9]. This dataset contains 1,522 crystalline materials whose phonon DOS is calculated from density functional perturbation theory (DFPT) by a previous work [42]. Since the provided dataset does not contain crystal system information, we additionally collect the information based on the Materials Project (MP) website 4 on the given each material's unique ID (MP-id).

Footnote 5: [https://github.com/zhantaochen/phonondos_e3nn](https://github.com/zhantaochen/phonondos_e3nn)

### Electron DOS

We also use **Electron DOS** dataset that contains 38,889 crystalline materials. The Electron DOS dataset consists of the materials and their electron DOS information that is collected from the MP website 4. Among the collected data, we exclude the materials that are tagged to include magnetism because the DOS of magnetism materials is not accurate to be directly used for training machine learning models [23]. We consider an energy grid of 201 points ranging from \(-5\) to \(5\) eV with respect to the band edges with \(50\) meV intervals and the Fermi energy is all set to \(0\) eV on this energy grid. Moreover, we normalize the DOS of each material to be in the range between 0 and 1. That is, the maximum and minimum value for each DOS is 1 and 0, respectively, for all materials. Moreover, we smooth the DOS values with the Savitzky-Golay filter with the window size of 17 and polyorder of 1 using scipy library following a previous work [9].

Footnote 4: [https://github.com/zhantaochen/phonondos_e3nn](https://github.com/zhantaochen/phonondos_e3nn)

### Data Statistics of Electron DOS dataset in OOD scenarios

As described in the main manuscript, we further evaluate the model performance in two out-of-distribution scenarios: **Scenario 1**: regarding the number of atom species, and **Scenario 2**: regarding the crystal systems. We provide detailed statistics of the number of crystalline materials for each scenario in Table 5 and Table 6.

\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & Cubic & Hexagonal & Tetragonal & Trigonal & Orthorhombic & Monoclinic & Triclinic & Total \\ \hline \# Materials & 8,385 & 3,983 & 5,772 & 3,964 & 8,108 & 6,576 & 2,101 & 38,889 \\ \hline \hline \end{tabular}
\end{table}
Table 6: The number of crystals according to different crystal systems (Scenario 2).

\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & Unary (1) & Binary (2) & Ternary (3) & Quaternary (4) & Quinary (5) & Senary (6) & Setpenary (7) & Total \\ \hline \# Materials & 386 & 9,034 & 21,794 & 5,612 & 1,750 & 279 & 34 & 38,889 \\ \hline \hline \end{tabular}
\end{table}
Table 5: The number of crystals according to the number of atom species (Scenario 1).

Evaluation Protocol

**Phonon DOS.** As described in the main manuscript, we evaluate the model performance based on the data splits given in a previous work [9].

**Electron DOS.** On the other hand, for the Electron DOS dataset, we use different dataset split strategies for each scenario. For the in-distribution setting, we randomly split the dataset into train/valid/test of 80/10/10%. On the other hand, for the out-of-distribution setting, we split the dataset regarding the structure of the crystals. For both scenarios, we generate training sets with simple crystal structures and a valid/test set with more complex crystal structures, because it is crucial to transfer the knowledge obtained from simple crystal structures to that from complex structures in real-world materials science. More specifically, in the scenario 1 (different number of atom species, i.e., # Atom species in Table 2), we use Binary and Ternary materials as training data and Unary, Quaternary, and Quinary materials as valid and test data. In the case of Unary, we exclude it from training data despite its simplicity due to the observed difficulty of the structure, as will be discussed in Section E.1. In the scenario 2 (different crystal systems, i.e., Crystal System in Table 2), we use Cubic, Hexagonal, Tetragonal, Trigonal, and Orthorhombic crystal systems as training set and Monoclinic and Triclinic as valid and test set. In this scenario, where no prompt is available for unseen crystal systems, we employ the mean-pooled representations of the trained prompts during testing, i.e., for the Monoclinic and Triclinic crystal systems. Please refer to Table 5 and Table 6 for detailed statistics of crystals in each scenario.

**Physical Properties.** In addition to evaluating the accuracy of the model's predictions of the DOS, it is crucial to assess the physical meaningfulness of the predicted DOS for real-world applications. To assess the physical meaningfulness of the predicted DOS, we utilize the predicted DOS to estimate a range of important material properties. Specifically, we evaluate three materials' properties: the bulk modulus for phonon DOS, and the band gap and Fermi energy for electron DOS (Table 1).

Bulk Modulus 6 is a thermodynamic quantity measuring the resistance of a substance to compression. It provides a measure of the material's ability to withstand changes in volume under applied pressure. In the context of elastic properties, the bulk modulus serves as a descriptor, as it indicates how well a material can recover its original volume after being subjected to compression.

Footnote 6: [https://en.wikipedia.org/wiki/Bulk_modulus](https://en.wikipedia.org/wiki/Bulk_modulus)

Another property we focus on is the Band Gap 7, which refers to the energy range in a material where no electronic states exist. It represents the energy difference between the top of the valence band and the bottom of the conduction band in insulators and semiconductors. Functional inorganic materials, such as those used in applications like LEDs, transistors, photovoltaics, or scintillators, require a comprehensive understanding of their band gap [69]. By accurately predicting the band gap based on DOS, we can accelerate the development of new materials for a wide range of applications.

Footnote 7: [https://en.wikipedia.org/wiki/Band_gap](https://en.wikipedia.org/wiki/Band_gap)

Additionally, we predict the Fermi Energy 8, which represents the highest energy level occupied by electrons at absolute zero temperature (0K). It can be used to determine the electrical and thermal characteristics of materials.

Footnote 8: [https://en.wikipedia.org/wiki/Fermi_energy](https://en.wikipedia.org/wiki/Fermi_energy)

## Appendix C Implementation Details

In this section, we provide implementation details of DOSTransformer.

**Graph Neural Networks.** Our graph neural networks consist of two parts, i.e., encoder and processor. Encoder learns the initial representation of atoms and bonds, while the processor learns to pass the messages across the crystal structure. More formally, given an atom \(v_{i}\) and the bond \(e_{ij}\) between atom \(v_{i}\) and \(v_{j}\), node encoder \(\phi_{node}\) and edge encoder \(\phi_{edge}\) outputs initial representations of atom \(v_{i}\) and bond \(e_{ij}\) as follows:

\[\mathbf{h}_{i}^{0}=\phi_{node}(\mathbf{X}_{i}),\ \ \mathbf{b}_{ij}^{0}=\phi_{ edge}(\mathbf{B}_{ij}), \tag{4}\]

where \(\mathbf{X}\) is the atom feature matrix whose \(i\)-th row indicates the input feature of atom \(v_{i}\), \(\mathbf{B}\in\mathbb{R}^{n\times n\times F_{e}}\) is the bond feature tensor with \(F_{e}\) features for each bond. With the initial representationsof atoms and bonds, the processor learns to pass messages across the crystal structure and update atoms and bonds representations as follows:

\[\mathbf{b}_{ij}^{l+1}=\psi_{edge}^{l}(\mathbf{h}_{i}^{l},\mathbf{h}_{j}^{l}, \mathbf{b}_{ij}^{l}),\ \ \ \mathbf{h}_{i}^{l+1}=\psi_{node}^{l}(\mathbf{h}_{i}^{l}, \sum_{j\in\mathcal{N}(i)}\mathbf{b}_{ij}^{l+1}), \tag{5}\]

where \(\mathcal{N}(i)\) is the neighboring atoms of atom \(v_{i}\), \(\psi\) is a two-layer MLP with non-linearity, and \(l=0,\ldots,L^{\prime}\). Note that \(\mathbf{h}_{i}^{L^{\prime}}\) is equivalent to the \(i\)-th row of the atom embedding matrix \(\mathbf{H}\) in Equation 1.

**Model Training.** In all our experiments, we use the AdamW optimizer for model optimization. For all the tasks, we train the model for 1,000 epochs with early stopping applied if the best validation loss does not change for 50 consecutive epochs.

**Hyperparameter Tuning.** Detailed hyperparameter specifications are given in Table 7. For the hyperparameters in DOSTransformer, we tune them in certain ranges as follows: number of message passing layers in GNN \(L^{\prime}\) in {2, 3, 4}, number of cross-attention layers \(L_{1}\), \(L_{3}\) in {2, 3, 4}, number of self-attention layers \(L_{2}\) in {2, 3, 4}, hidden dimension \(d\) in [64, 128, 256], learning rate \(\eta\) in {0.0001, 0.0005, 0.001}, and batch size \(B\) in {1, 4, 8}. We use the sum pooling to obtain the crystalline material \(i\)'s representation, i.e., \(\mathbf{g}_{i}\). We report the test performance when the performance on the validation set gives the best result.

## Appendix D Methods Compared

In this section, we provide further details on the methods that are compared with DOSTransformer in our experiments.

**MLP.** We first encode the atoms in a crystalline material with an MLP. Then, we obtain the representation of material \(i\), i.e., \(\mathbf{g}_{i}\), by sum pooling the representations of its constituent atoms. With the material representation, we predict DOS with an MLP predictor \(\phi^{\prime}\), i.e., \(\hat{\mathbf{Y}}^{i}=\phi^{\prime}(\mathbf{g}_{i})\), where \(\phi^{\prime}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{201}\).

On the other hand, when we incorporate energy embeddings into the MLP, we predict DOS for each energy \(j\) with a learnable energy embedding \(\mathbf{E}_{j}^{0}\) and obtained material representation \(\mathbf{g}_{i}\), i.e., \(\hat{\mathbf{Y}}_{j}^{i}=\phi(\mathbf{E}_{j}^{0}||\mathbf{g}_{i})\), where \(\phi:\mathbb{R}^{2d}\rightarrow\mathbb{R}^{1}\) is a parameterized MLP.

**Graph Network.** We first encode the atoms in a crystalline material with a graph network [4]. As done for MLP, we obtain the representation of material \(i\), i.e., \(\mathbf{g}_{i}\), by sum pooling the representations of its constituent atoms. With the material representation, we predict the DOS with an MLP predictor,

\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{**Hyperparameters**} & \multicolumn{2}{c}{**In-Distribution**} & \multicolumn{2}{c}{**Out-of-Distribution**} \\ \cline{2-5}  & **Phonon DOS** & **Electron DOS** & **\# Atom Species** & **Crystal Systems** \\ \hline \# Message Passing & & & & \\ Layers (\(L^{\prime}\)) & 3 & 3 & 3 & 3 \\ \# Cross-Attention & & & & \\ Layers (\(L_{1}\)) & 2 & 2 & 2 & 2 \\ \# Self-Attention & & & & \\ Layers (\(L_{2}\)) & 2 & 2 & 2 & 2 \\ \# Cross-Attention & & & & \\ Layers (\(L_{3}\)) & 2 & 2 & 2 & 2 \\ Hidden Dim. (\(d\)) & 256 & 256 & 256 & 256 \\ Learning Rate (\(\eta\)) & 0.0001 & 0.0001 & 0.0001 & 0.0001 \\ Batch Size (\(B\)) & 1 & 8 & 8 & 8 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Hyperparameter specifications of DOSTransformer.

i.e., \(\hat{\mathbf{Y}}^{i}=\phi^{\prime}(\mathbf{g}_{i})\), where \(\phi^{\prime}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{201}\). Note that the only difference with MLP is that the atom representations are obtained through the message passing scheme. We also compare the vanilla graph network that incorporates the energy information as we have done in MLP.

**E3NN.** For E3NN [9], we use the official code published by the authors9, which implements equivariant neural networks with E3NN python library10. By learning the equivariance, the model can generate high-quality representations with a small number of training materials. After obtaining the crystalline material representation \(\mathbf{g}_{i}\), all other procedures have been done in the same manner with other baseline models, i.e., MLP and Graph Network.

Footnote 9: [https://github.com/ninarina12/phononDoS_tutorial](https://github.com/ninarina12/phononDoS_tutorial)

Footnote 10: [https://docs.e3nn.org/en/latest/index.html](https://docs.e3nn.org/en/latest/index.html)

## Appendix E Additional Experiments

### Model Performance Analysis on Out-of-Distribution Scenarios

In this section, we conduct a comprehensive analysis of the model's predictions in the out-of-distribution scenarios presented in Table 2. In Table 8, we evaluate the performance of the model for each type of material, providing detailed insights into its predictive capabilities. We have following observations: **1)** We observe that DOSTransformer consistently outperforms in both out-of-distribution scenarios, which demonstrates the superiority of DOSTransformer. **2)** The performance of all the compared models generally degrades as the crystal structure gets more complex. That is, models perform worse in Quanrary crystals than in Quanrary crystals, and worse in Triclinic crystals than in Monoclinic crystals. **3)** On the other hand, it is not the case in Unary crystal. This is because in Unary crystal only one type of atom repeatedly appears in the crystal structure, which cannot give enough information to the model. However, DOSTransformer also makes comparably accurate predictions in the Unary materials by modeling the complex relationship between the atoms and various energy levels.

### Injecting Crystal System Information to Baseline Methods

In this section, we adopt our prompt-based crystal system information injection procedure to the baseline methods. We examine two approaches for injecting the information: 1) injecting the information into the input atoms (i.e., Position 1), and 2) injecting it before the DOS prediction layer (i.e., Position 2). In Table 9, we have the following observations: **1)** Compared to Table 1, all baseline models benefit from using crystal system information. This demonstrates the importance of

\begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{6}{c}{**\# Atom Species**} & \multicolumn{6}{c}{**Crystal System**} \\ \cline{2-11}  & \multicolumn{2}{c}{**Unary**} & \multicolumn{2}{c}{**Quarternary**} & \multicolumn{2}{c}{**Quinary**} & \multicolumn{2}{c}{**Monoclinic**} & \multicolumn{2}{c}{**Triclinic**} \\ \cline{2-11}  & MSE & MAE & MSE & MAE & MSE & MAE & MSE & MAE & MSE & MAE \\ \hline \multicolumn{11}{l}{**Energy ✗**} \\ \hline MLP & 0.578 & 0.180 & 0.500 & 0.153 & 0.673 & 0.182 & 0.370 & 0.132 & 0.470 & 0.146 \\  & (0.005) & (0.001) & (0.003) & (0.001) & (0.004) & (0.002) & (0.017) & (0.003) & (0.021) & (0.003) \\ Graph Network & 0.485 & 0.165 & 0.443 & 0.141 & 0.620 & 0.170 & 0.376 & 0.127 & 0.504 & 0.147 \\  & (0.013) & (0.003) & (0.003) & (0.001) & (0.007) & (0.003) & (0.003) & (0.001) & (0.007) & (0.001) \\ E3NN & 0.565 & 0.167 & 0.486 & 0.145 & 0.708 & 0.179 & 0.393 & 0.129 & 0.510 & 0.148 \\  & (0.025) & (0.002) & (0.001) & (0.000) & (0.013) & (0.002) & (0.003) & (0.001) & (0.006) & (0.002) \\ \hline \multicolumn{11}{l}{**Energy ✓**} \\ \hline MLP & 0.597 & 0.183 & 0.498 & 0.151 & 0.684 & 0.178 & 0.367 & 0.130 & 0.468 & 0.144 \\  & (0.034) & (0.005) & (0.003) & (0.002) & (0.015) & (0.002) & (0.012) & (0.002) & (0.002) & (0.003) \\ Graph Network & 0.471 & 0.160 & 0.416 & 0.137 & 0.571 & 0.165 & 0.359 & 0.125 & 0.476 & 0.144 \\  & (0.028) & (0.005) & (0.002) & (0.002) & (0.005) & (0.002) & (0.010) & (0.004) & (0.008) & (0.004) \\ E3NN & 0.567 & 0.166 & 0.481 & 0.143 & 0.689 & 0.175 & 0.393 & 0.128 & 0.516 & 0.147 \\  & (0.021) & (0.003) & (0.008) & (0.000) & (0.020) & (0.002) & (0.006) & (0.001) & (0.006) & (0.001) \\ \hline \multirow{2}{*}{**DOSTransformer**} & **0.417** & **0.145** & **0.413** & **0.128** & **0.570** & **0.157** & **0.343** & **0.117** & **0.466** & **0.137** \\  & (0.012) & (0.003) & (0.010) & (0.002) & (0.010) & (0.001) & (0.003) & (0.001) & (0.007) & (0.001) \\ \hline \hline \end{tabular}
\end{table}
Table 8: Model performance in Out-of-Distribution scenarios.

utilizing crystal structural systems information, which has been overlooked in previous works. **2)** However, DQSTransformer still outperforms all baseline methods with crystal system information (See DQSTransformer in Table 1), verifying the importance of an elaborate design of crystal system injection procedure. To be more specific, we notice a relatively significant performance gap between DQSTransformer and the best baseline model in Electron DOS, which comprises a broader range of crystalline materials than Phonon DOS. This finding highlights the importance of an intricate crystal system injection procedure when striving to learn the DOS of diverse crystalline materials.

### Various Training Data Ratio for Fine-Tuning

In this section, we additionally provide experimental results on various ratios of training data for fine-tuning in Table 3. That is, instead of sampling 10% of training data from the test set used in OOD scenarios in Section 5.3, we try various sampling ratios, i.e., 5%, 10%, 15%, and 20%, from the test set. We have the following observations: **1)** We notice a significant performance disparity between the "Only Prompt" and "All" approaches, particularly when the training dataset is limited. This phenomenon can be attributed to the challenge of overfitting when fine-tuning the entire model on a small subset of materials, as discussed in Section 5.3. **2)** Conversely, as the training data increases, we observe that the difference in performance between the "Only Prompt" and "All" methods diminishes. This is due to the enough amount of training data allowing for the adjustment of all model parameters, consistent with the analysis discussed in Section 5.3. However, it is important to note that the existing DFT calculation-based databases suffer from a highly biased distribution, which limits their coverage of different materials. This limitation emphasizes the significance of achieving good performance even with a small subset of training data. Therefore, we argue that the application of prompt tuning enhances the real-world applicability of DQSTransformer.

### Comparison with Crystal Neural Network

We conducted additional experiments on the state-of-the-art crystal neural networks, i.e., ALIGNN [10] and Matformer [63], by adjusting the output dimensions of these models (51 for Phonon DOS and 201 for Electron DOS) in Table 10. Although both ALIGNN and Matformer exhibited commendable performance, we observe that DQSTransformer consistently surpassed them significantly. These observations again demonstrate the importance of considering energy levels for accurate DOS prediction.

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{4}{c}{**Phonon DOS**} & \multicolumn{4}{c}{**Electron DOS**} \\ \cline{2-9}  & \multicolumn{2}{c}{**Position 1**} & \multicolumn{2}{c}{**Position 2**} & \multicolumn{2}{c}{**Position 1**} & \multicolumn{2}{c}{**Position 2**} \\ \cline{2-9}  & MSE & MAE & MSE & MAE & MSE & MAE & MSE & MAE \\ \hline \hline \multicolumn{9}{l}{**Energy ✗**} \\ \hline MLP & 0.308 & 0.105 & 0.324 & 0.109 & 0.343 & 0.128 & 0.340 & 0.127 \\  & (0.013) & (0.001) & (0.012) & (0.002) & (0.008) & (0.001) & (0.009) & (0.0031 \\ Graph Network & 0.260 & 0.097 & 0.244 & 0.095 & 0.260 & 0.102 & 0.260 & 0.103 \\  & (0.008) & (0.000) & (0.023) & (0.001) & (0.011) & (0.001) & (0.006) & (0.002) \\ E3NN & 0.210 & 0.077 & 0.209 & 0.079 & 0.292 & 0.108 & 0.299 & 0.110 \\  & (0.007) & (0.002) & (0.010) & (0.001) & (0.007) & (0.001) & (0.002) & (0.001) \\ \hline \multicolumn{9}{l}{**Energy ✓**} \\ \hline MLP & 0.251 & 0.100 & 0.245 & 0.098 & 0.332 & 0.125 & 0.336 & 0.127 \\  & (0.000) & (0.000) & (0.004) & (0.001) & (0.003) & (0.001) & (0.007) & (0.002) \\ Graph Network & 0.230 & 0.094 & 0.224 & 0.091 & 0.234 & 0.097 & 0.230 & 0.093 \\  & (0.009) & (0.002) & (0.009) & (0.002) & (0.001) & (0.004) & (0.009) & (0.002) \\ E3NN & 0.194 & 0.073 & 0.190 & 0.073 & 0.286 & 0.108 & 0.290 & 0.109 \\  & (0.004) & (0.000) & (0.000) & (0.001) & (0.002) & (0.000) & (0.003) & (0.000) \\ \hline \hline \end{tabular}
\end{table}
Table 9: Baseline model performance with crystal structural system prompts.

Figure 6: Various training data ratios for fine-tuning.

### Model Training and Inference Time

In this section, to verify the efficiency of DOSTransformer, we compare the training and inference time of the baseline methods in Table 11. We observe that DOSTransformer requires a longer training time per epoch on the Phonon DOS dataset compared to E3NN, which can be attributed to the two forward passes (i.e., system and global energy embeddings) during the training procedure. However, when it comes to the Electron DOS dataset, DOSTransformer demonstrates a shorter training time per epoch compared to E3NN. This is because the Electron DOS dataset has complex crystal structures, requiring more time for E3NN to learn equivariant representations. Furthermore, in terms of inference time, DOSTransformer demonstrates significantly faster computation per epoch compared to E3NN, particularly on the Electron DOS dataset. This is because we only utilize system prediction without global prediction during inference. As many predictive ML models are used for high-throughput screening in material discovery, inference time is a critical factor for ML models in materials science, demonstrating the practicality of DOSTransformer in real-world applications.

## Appendix F Broader Impacts

**Potential Positive Scientific Impacts.** In this work, we propose DOSTransformer, which is the first work that considers various energy levels during DOS prediction and introduces prompts for crystal structural system, demonstrating its applicability in real-world scenarios. For example, transferring the knowledge obtained from simple structured materials to complex structured materials is crucial because DFT calculation-based databases cover limited types of materials or structural archetypes. Therefore, we believe DOSTransformer has broad impacts on various fields of materials science.

**Potential Negative Societal Impacts.** This work explores the automation process for materials science without wet lab experiments. However, it is important to acknowledge that in the industry, there are skilled professionals dedicated to conducting such experiments for materials science.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**Phonon DOS**} & \multicolumn{3}{c}{**Electron DOS**} \\ \cline{2-7}  & MSE & MAE & \(R^{2}\) & MSE & MAE & \(R^{2}\) \\ \hline \multirow{2}{*}{ALIGNN} & 0.204 & 0.085 & 0.717 & 0.270 & 0.108 & 0.600 \\  & (0.005) & (0.001) & (0.007) & (0.000) & (0.001) & (0.001) \\ \multirow{2}{*}{Matformer} & 0.198 & 0.084 & 0.724 & 0.268 & 0.104 & **0.601** \\  & (0.008) & (0.002) & (0.013) & (0.004) & (0.001) & (0.002) \\ \hline \multirow{2}{*}{DOSTransformer} & **0.191** & **0.071** & **0.733** & **0.221** & **0.089** & **0.679** \\  & (0.003) & (0.002) & (0.004) & (0.006) & (0.001) & (0.006) \\ \hline \hline \end{tabular}
\end{table}
Table 10: Comparison to crystal neural networks.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**Training**} & \multicolumn{2}{c}{**Inference**} \\ \cline{2-5}  & **Phonon DOS** & **Electron DOS** & **Phonon DOS** & **Electron DOS** \\ \hline \hline \multirow{2}{*}{MLP} & 4.10 & 23.52 & 1.51 & 3.00 \\ Graph Network & 16.17 & 59.74 & 1.95 & 3.88 \\ E3NN & 21.21 & 141.02 & 3.72 & 9.49 \\ \hline \hline \multirow{2}{*}{**Energy ✓**} & & & & \\ \hline MLP & 4.67 & 27.88 & 1.66 & 3.10 \\ Graph Network & 17.45 & 66.83 & 2.16 & 4.28 \\ E3NN & 24.12 & 152.80 & 3.92 & 10.21 \\ \hline \hline \multirow{2}{*}{DOSTransformer} & 39.17 & 145.85 & 2.98 & 5.99 \\ \hline \hline \end{tabular}
\end{table}
Table 11: Training and inference time per epoch for each dataset (sec/epoch).

Therefore, it is important to proactively address these concerns by encouraging collaboration between automated methods and human experts.

## Appendix G Pseudo Code

Algorithm 1 shows the pseudocode of DOSTransformer.

```
Input : An input crystalline material \(\mathcal{G}=(\mathbf{X},\mathbf{A})\), Ground truth DOS \(\mathbf{Y}\), Number of attention layers \(L_{1},L_{2},L_{3}\), Initialized energy embeddings \(\mathbf{E}\), Initialized crystal system prompts \(\mathbf{P}\).
1\(\mathbf{H}\leftarrow\text{GNN}(\mathbf{X},\mathbf{A})\)
2\(\mathbf{E}^{L_{1}}\leftarrow\text{Cross-Attention}(\mathbf{H},\mathbf{E},L_{1})\)
3\(\mathbf{g}\leftarrow\text{Sum Pooling}(\mathbf{H})\)
4\(\mathbf{E}^{glob}\leftarrow(\mathbf{E}^{L_{1}}||\mathbf{g})\)
5\(\tilde{\mathbf{E}}^{0,glob}\leftarrow\phi_{1}(\mathbf{E}^{glob})\)
6\(\tilde{\mathbf{E}}^{L_{2},glob}\leftarrow\text{Self-Attention}(\tilde{ \mathbf{E}}^{0,glob},L_{2})\) // Global Self-Attention
7\(\mathbf{E}^{L_{3},glob}\leftarrow\text{Cross-Attention}(\mathbf{H},\tilde{ \mathbf{E}}^{L_{2},glob},L_{3})\)
8\(\hat{\mathbf{Y}}\leftarrow\phi_{pred}(\mathbf{E}^{L_{3},glob})\)
9\(\mathcal{L}^{glob}\leftarrow\text{RMSE}(\hat{\mathbf{Y}},\mathbf{Y})\)
10\(\mathbf{E}^{sys}\leftarrow(\mathbf{E}^{L_{1}}||\mathbf{g}||\mathbf{P})\)
11\(\tilde{\mathbf{E}}^{0,sys}\leftarrow\phi_{2}(\mathbf{E}^{sys})\)
12\(\tilde{\mathbf{E}}^{L_{2},sys}\leftarrow\text{Self-Attention}(\tilde{ \mathbf{E}}^{0,sys},L_{2})\) // System Self-Attention
13\(\mathbf{E}^{L_{3},sys}\leftarrow\text{Cross-Attention}(\mathbf{H},\tilde{ \mathbf{E}}^{L_{2},sys},L_{3})\)
14\(\hat{\mathbf{Y}}\leftarrow\phi_{pred}(\mathbf{E}^{L_{3},sys})\)
15\(\mathcal{L}^{sys}\leftarrow\text{RMSE}(\hat{\mathbf{Y}},\mathbf{Y})\)
16\(\mathcal{L}^{total}\leftarrow\mathcal{L}^{glob}+\beta\cdot\mathcal{L}^{sys}\) // Calculate total loss
17Function Cross-Attention\((\mathbf{H},\mathbf{E}^{0},L)\):
18for\(l=1,2,\ldots,L\)do
19\(\mathbf{E}^{l}\leftarrow\text{Softmax}(\frac{\mathbf{E}^{l-1}\mathbf{H}^{\top }}{\mathbf{H}})\)
20 end for return\(\mathbf{E}^{L}\)
21Function Self-Attention\((\tilde{\mathbf{E}}^{0},L)\):
22for\(p=1,2,\ldots,L\)do
23\(\tilde{\mathbf{E}}^{p}\leftarrow\text{Softmax}(\frac{\tilde{\mathbf{E}}^{p-1} \tilde{\mathbf{E}}^{p-1}}{\mathbf{E}^{p-1}})\)
24 end for return\(\tilde{\mathbf{E}}^{L}\)
```

**Algorithm 1**Pseudocode of DOSTransformer.

Response:
## Review

### Summary

This paper presents DOSTransformer, a novel transformer-based architecture designed for predicting the density of states (DOS) of crystalline materials. The model uniquely incorporates energy levels as an input modality alongside material configurations, enhancing its predictive capabilities compared to prior approaches. It employs a multi-modal transformer architecture with cross-attention and self-attention layers, demonstrating improved performance over existing methods such as graph networks and E3NN in various experimental settings. Comprehensive evaluations, including ablation studies and sensitivity analyses, validate the effectiveness of the proposed model. Overall, the work addresses an important problem in materials science with promising results.

### Strengths

- The DOSTransformer is evaluated on both in-domain and out-of-domain structures against relevant baseline models.
- The model design effectively utilizes domain-specific knowledge, such as crystal structure type and energy levels.
- The task of predicting DOS is enhanced by providing energy levels for targeted predictions, supported by thorough ablation studies.
- The approach is well-defined and uses state-of-the-art libraries for implementation.
- The results include comprehensive evaluations on two datasets, with empirical validation of predictions.
- The paper is well-written, making it easy to follow and understand.

### Weaknesses

- More detail is needed on the implications of DOS predictions for downstream applications, particularly for a non-materials science audience.
- Clarification on the necessary mean squared error (MSE) values for real-world applicability is required.
- The datasets used in the experiments are somewhat limited in diversity, potentially restricting the evaluation of the proposed method.
- Core ablation studies are missing, particularly regarding the novelty of taking energy levels as input.
- There is insufficient discussion on the relationship between learned prompts and the physical characteristics of materials.

### Questions

- What MSE values are considered acceptable for real-world applications?
- Are there other instances in the literature where transformers have been used to predict outputs of mathematical functions with fixed grid inputs?
- Could the model handle structural variations without relying solely on the input prompts during testing?
- What specific implementation details were used for E3NN, particularly regarding the irreducible representations?

### Soundness

**Score:** 3

**Description:** 3 = good. The foundational concepts and methodologies are sound, but there are areas that require further clarification and detail.

### Presentation

**Score:** 3

**Description:** 3 = good. The overall presentation of the paper is clear and coherent, though some sections could benefit from additional detail.

### Contribution

**Score:** 3

**Description:** 3 = good. The contribution of the research is significant, addressing a relevant problem in the field, although more comparisons with existing methods would enhance its impact.

### Rating

**Score:** 7

**Description:** 7 = accept, but needs minor improvements. The paper is technically solid, with moderate-to-high impact potential, and requires minor clarifications and enhancements.

### Paper Decision

**Decision:** Accept (poster)

**Reasons:** The paper presents a novel approach to an important problem in predicting the density of states in crystalline materials. It is grounded in sound methodology and provides valuable empirical results. While there are areas for improvement, particularly in clarifying the implications of its findings and the diversity of datasets, the strengths and overall contributions significantly outweigh the weaknesses, justifying an acceptance decision.