#  MPIrigen: MPI Code Generation through
Domain-Specific Language Models

Nadav Schneider<EMAIL_ADDRESS>Ben-Gurion University, IAECIsrael ,
Niranjan Hasabnis<EMAIL_ADDRESS>Intel LabsUnited States ,  Vy
A. Vo<EMAIL_ADDRESS>Intel LabsUnited States ,  Tal Kadosh
<EMAIL_ADDRESS>Ben-Gurion University, IAECIsrael ,  Neva Krien
<EMAIL_ADDRESS>Independent ResearcherIsrael ,  Mihai Capotă
<EMAIL_ADDRESS>Intel Labs, United StatesUnited States ,  Abdul Wasay
<EMAIL_ADDRESS>Intel LabsUnited States ,  Guy Tamir
<EMAIL_ADDRESS>IntelUnited States ,  Ted Willke<EMAIL_ADDRESS>Intel LabsUnited States ,  Nesreen Ahmed<EMAIL_ADDRESS>Intel
LabsUnited States ,  Yuval Pinter<EMAIL_ADDRESS>Ben-Gurion
UniversityIsrael ,  Timothy Mattson<EMAIL_ADDRESS>Independent
ResearcherUnited States and  Gal Oren<EMAIL_ADDRESS>Technion,
NRCNIsrael

(2023)

###### Abstract.

The imperative need to scale computation across numerous nodes highlights the
significance of efficient parallel computing, particularly in the realm of
Message Passing Interface (MPI) integration. While MPI serves as a cornerstone
for large-scale parallelism, its seamless integration into codebases,
especially concerning domain decomposition, has proven challenging. Static
tools aimed at addressing this challenge have exhibited limited effectiveness
and scalability. On the other hand, contemporary language models designed for
programming problems have demonstrated utility in parallel programming tasks
such as OpenMP pragma generation. However, the challenging parallel
programming task of generating MPI-based parallel programs has remained
unexplored.

This study first investigates the performance of state-of-the-art language
models in generating MPI-based parallel programs. Findings reveal that widely
used models such as GPT-3.5 and PolyCoder (specialized multi-lingual code
models) exhibit notable performance degradation, when generating MPI-based
programs compared to general-purpose programs. In contrast, domain-specific
models such as MonoCoder, which are pre-trained on MPI-related programming
languages of C and C++, outperform larger models. Subsequently, we introduce a
dedicated downstream task of MPI-based program generation by fine-tuning
MonoCoder on HPCorpusMPI. We call the resulting model as MPIrigen. We propose
an innovative pre-processing for completion only after observing the whole
code, thus enabling better completion with a wider context. Comparative
analysis against GPT-3.5 zero-shot performance, using a novel HPC-oriented
evaluation method, demonstrates that MPIrigen excels in generating accurate
MPI functions up to 0.8 accuracy in location and function predictions, and
with more than 0.9 accuracy for argument predictions. The success of this
tailored solution underscores the importance of domain-specific fine-tuning in
optimizing language models for parallel computing code generation, paving the
way for a new generation of automatic parallelization tools.

The sources of this work are available at our GitHub MPIrigen repository.

MPI, Domain Decomposition, MPI-rical, MPICodeCorpus, MonoCoder, Transformer,
LLM

††copyright: acmcopyright††journalyear: 2023††doi: XXXXXXX.XXXXXXX††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††copyright: none

##  1\. MPI: Source-to-Source Parallelization with Compilers

Efforts in the domain of source-to-source automatic parallelization have
primarily focused on transitioning from serial code to shared-memory
parallelization, especially with OpenMP. Such approaches have relied on
heuristics and rule-based methods (e.g., ComPar (Mosseri et al., 2020),
Par4all (Amini et al., 2012; par, [n. d.]), and Cetus (Bae et al., 2013))
employing AST generation and data dependence algorithms. However, these
methods often face challenges in handling diverse syntax and may yield sub-
optimal results, as noted in several studies (Harel et al., 2020; Milewicz et
al., 2021; Prema et al., 2017, 2019).

Other marginal efforts have been made to transform shared memory to
distributed memory parallelization, yet not directly from serial to
distributed memory parallelization. OMP2MPI (Saa-Garriga et al., 2015) is a
source-to-source compiler based on the BSCs Mercurium (Balart et al., 2004)
framework that automatically generates MPI source code from shared-memory
parallel OpenMP code. OMP2MPI gets parallel OpenMP code, and its AST as input,
detects and transforms pragma omp parallel for blocks, and then divides the
task into MPI master and slave processes. CATO (Squar et al., 2020) is another
static compiler that uses LLVM and Clang to transform OpenMP code to MPI. Its
main component is an LLVM transform pass, which transforms the original OpenMP
kernel using an intermediate representation (IR), an assembly-like
representation of code.

These tools have many limitations. First, given the final purpose is to
automatically convert serial code to a distributed one, converting serial code
to a shared memory through the process leads to performance degradation due to
imperfections in these compilers, including ones that originally related to
the serial to shared memory automatic parallelization. Imperfections like
increasing run time, using two-sided communication only, and even the
necessity of external information (Hamidouche et al., 2011) damage the utility
of this as a practical tool. Second, most of the static tools are source-to-
source compilers, which obligate their use on compiled codes only.

##  2\. MPI Parallelization with LLMs

Recently, there has been a shift towards data-driven methods, especially large
language models (LLMs). The basic approach to use LLMs for parallelization
tasks is by utilizing those as-is in a zeroshot performance fashion (Godoy et
al., 2023; Chen et al., 2023b; Valero-Lara et al., 2023; Nichols et al., 2023;
Mahmud et al., 2023; Nichols et al., 2024), while the more advanced approach
is built upon fine-tuning said models for the parallelization specific task
(Harel et al., 2023; Kadosh et al., 2023b, e; Nichols et al., 2023; Chen et
al., 2023a; Ding et al., 2023; Chen et al., 2023a, 2024b; Kadosh et al.,
2023d). In both of those cases, there has been success in utilizing
transformers to classify the need for OpenMP pragmas and MPI functions
(Schneider et al., 2023a) and even to generate single-lined OpenMP pragmas
(Kadosh et al., 2023c), introducing a novel approach for automatic
parallelization. However, the nuanced task of generating intricate, multi-
functional MPI codes across diverse locations in a dedicated fine-tuned model
has remained unexplored (Chen et al., 2024a). Nevertheless, recently,
MonoCoder (Kadosh et al., 2023c) has been introduced as a novel approach aimed
at enhancing LMs tailored for HPC tasks. The hypothesis posits that HPC-
specific LMs, such as those designed and trained specifically on HPC datasets,
would outperform existing LMs in HPC contexts. Two experiments are conducted
to validate this hypothesis. Initially, MonoCoder is constructed by reducing
the number of layers of PolyCoder code LM by a factor of 4 and pretraining it
solely on C and C++ codes. Despite its smaller size, MonoCoder achieves
comparable perplexity scores to PolyCoder on the HPCorpus dataset, indicating
its efficacy in understanding HPC-specific code structures. Subsequently,
MonoCoder’s performance on HPC tasks is evaluated, focusing on CodeBLEU
competence for high-performance and parallel code generations, particularly in
OpenMP parallelization tasks. Existing LMs struggle with capturing local
semantics, affecting their performance in predicting OpenMP clauses
accurately. To address this limitation, TokomPiler, a novel code pre-
processing scheme that eliminates local semantics, is introduced. MonoCoder
consistently outperforms existing LMs across various tests and tasks,
demonstrating its robustness and adaptability in HPC contexts, even under
semantic-less settings. Yet, no benchmark or adaptation of MonoCoder was
performed for the MPI code generation problem.

⬇

1int main(argc,argv)

2{

3 int done = 0, n = 10000, rank, numprocs, i;

4 double pi_total, pi, h, sum, x;

6 MPI_Init(&argc,&argv);

7 MPI_Comm_size(MPI_COMM_WORLD, &numprocs);

8 MPI_Comm_rank(MPI_COMM_WORLD, &rank);

10 while (!done)

11 {

12 MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD);

13 if (n == 0)

14 break;

15 h = 1.0 / (double) n;...

(a) MPI Source code

⬇

1int func_270(type_255,type_762)

2{

3 int var_289 = num_463, var_649 = num_199, var_635, var_257, var_165;

4 double var_663, var_792, var_305, var_610, var_55;

6 MPI_Init(&type_255,&type_762);

7 MPI_Comm_size(var_674,&var_257);

8 MPI_Comm_rank(var_674,&var_635);

10 while (!var_289)

11 {

12 MPI_Bcast(&var_649, num_700, var_167, num_463, var_674);

13 if (var_649 == num_463)

14 break;

15 var_305 = num_302 / (double) var_649;...

(b) TokomPiler MPI source code

⬇

1int main(argc,argv)

2{

3 int done = 0, n = 10000, rank, numprocs, i;

4 double pi_total, pi, h, sum, x;

6 while (!done)

7 {

8 if (n == 0)

9 break;

10 h = 1.0 / (double) n;...

12 (6, MPI_Init(&argc,&argv);)

13 (7, MPI_Comm_size(MPI_COMM_WORLD,&numprocs);)

14 (8, MPI_Comm_rank(MPI_COMM_WORLD,&rank);)

15 (12, MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD);)...

(c) MPI Pre-processed source code

Figure 1. The MPI functions in the source code (a) are removed and
concatenated with their corresponding line number to the last line (3). This
way, MPIrigen learns in a left-to-right fashion the relation between code and
its appropriate MPI functions. The TokomPiler version in (2), embeds AST
information while neglecting any semantic information. TokomPiler version is
used to demonstrate MonoCoder’s semantic information independence during
generation compared to other models.

##  3\. Research Objectives and Contributions

The gap in the existing methods for MPI source-to-source code parallelization
– either by compilers or LMs – has prompted our proposal for a data-driven
generative language model using a sequence-to-sequence transformer-based
approach. Such model’s aim is to automatically suggest MPI functions for MPI
codes with MPI functions excluded (semi-serial codes). This is by using a
designated database of semi-serial codes and fully MPI parallelized code
pairs, providing a new perspective on addressing the complexities of
distributed-memory parallelization. Thus, this study presents MPIrigen, a tool
developed to assist MPI programmers in automatically generating correct MPI
functions in an MPI-based domain decomposition parallel code. MonoCoder is the
base model, hence, to check its suitability and MPIrigen performance we would
like to answer the following research questions:

  1. (1)

Is MonoCoder capable of proper MPI code generation without fine-tuning, and in
the absence of local semantics?

  2. (2)

Is MPIrigen capable of inserting the calls to MPI functions in the right
locations?

  3. (3)

Is MPIrigen capable of generating calls to correct MPI functions?

  4. (4)

Is MPIrigen capable of generating correct arguments to the MPI functions?

Contributions. The main contributions of this paper are:

  * •

We create the first MPI codes only dataset — HPCorpusMPI which is based on
MPICodeCorpus and HPCorpus.

  * •

We have demonstrated MonoCoder understands MPI codes better than PolyCoder and
GPT3.5 by using a code completion task on HPCorpusMPI with TokomPiler source
code version, a code with AST information embedded while semantic information
is neglected.

  * •

We propose an innovative pre-process for completion tasks merely after
observing the whole code, thus enabling better completion with a wider
context.

  * •

We train and evaluate our approach, named MPIrigen by fine-tuning a domain-
specific model — MonoCoder, and find that our model performs well in
suggesting MPI functions for domain decomposition into an MPI-based parallel
code and is better than GPT3.5 zero-shot.

##  4\. HPCorpusMPI

While MPI programs have been already scraped from GitHub and gathered in
corpora such as MPICodeCorpus (Schneider et al., 2023a) or even HPCorpus
(Kadosh et al., 2023a), no corpus contains merely MPI domain decomposition
codes. HPCorpusMPI is a corpus consisting of both of the corpora above but
under the filter of MPI domain decomposition codes only and with duplication
removal resulting in a total of 16,384 programs.

Function | Amount | Function | Amount
---|---|---|---
MPI_Finalize | 19,183 | MPI_Comm_size | 14,387
MPI_Init | 16,135 | MPI_Recv | 13,783
MPI_Comm_rank | 16,096 | MPI_Bcast | 6,995
MPI_Send | 14,534 | MPI_Reduce | 3,600
Table 1. MPI Common Core functions distribution for HPCorpusMPI dataset.
context-100context-300context-600$0$$0.5$$1$$0.56$$0.68$$0.82$$0.5$$0.69$$0.79$$0.38$$0.54$$0.71$$0.47$$0.47$$0.48$$0.5$$0.59$$0.6$$0.52$$0.59$$0.6$CodeBLEUMonoCoderMonoCoder+TokompilerPolyCoderPolyCoder+TokompilerGPT3.5GPT3.5+Tokompiler
Figure 2. Performance of various models on Code Completion task over the
HPCorpusMPI. Models predict code continuation starting from token 100, 300,
and 600.

(a) MPIrigen (fine-tuned MonoCoder-0.7B over 16K MPI codes from HPCorpusMPI)

(b) GPT3.5 prompt engineering ”Generate the optimal MPI functions for the
provided code, and supply in the response the entire complete code with those
MPI functions: [CODE]”.

Figure 3. Comparative analysis of MPIrigen (fine-tuned MonoCoder-0.7B over
HPCorpusMPI for the MPI calls predictions) and GPT3.5 prompt engineering,
demonstrates that MPIrigen excels in generating accurate MPI codes in a
breakdown to locations, functions, and arguments. Functions accuracy is
measured for matched locations only and arguments accuracy for matched
functions only. The results are given as a function of the amount of original
MPI calls (1-20), showcasing that accuracy drops as complexity grows (besides
the ultra simplicity of one or two MPI function calls).

1234567891011121314151617181920$0$$1{,}000$$2{,}000$$3{,}000$Limit
functionsAmount of MPI
callsMPI_InitMPI_FinalizeMPI_Comm_rankMPI_Comm_sizeMPI_BcastMPI_ReduceMPI_SendMPI_Recv
(a) Ground truth

1234567891011121314151617181920$0$$1{,}000$$2{,}000$$3{,}000$Limit
functionsMPI_InitMPI_FinalizeMPI_Comm_rankMPI_Comm_sizeMPI_BcastMPI_ReduceMPI_SendMPI_Recv
(b) Prediction

Figure 4. Stacked bar chart of the ground truth and MPIrigen prediction
distribution of selected MPI functions under variance 2 (correct location and
function predictions are presented). The results are given as a function of
the amount of original MPI calls (1-20).

To create the dataset for training MPIrigen out of the corpus, several pre-
process stages have been done (Figure 1):

  1. (1)

Codes with both MPI_Init and MPI_Finalize and under the limit of 2048 tokens
(with BPE tokenizer) have been added.

  2. (2)

MPI functions through the code have been removed.

  3. (3)

Locations of the removed MPI functions and the functions themselves have been
written to the last line of the codes.

For convenience, the final dataset (with 13,322 programs) contains three
fields – ”Program”, ”Code”, ”MPI label” – and corresponds to the program’s
GitHub username, the code with removed MPI functions, and the label of the
locations and their MPI functions.

“MPI Common Core” functions, as defined in (Schneider et al., 2023b), are the
most prevalent MPI functions in domain decomposition. Therefore, it is
essential to validate that the distribution in the dataset is reasonable both
for model training and evaluation purposes. Towards that end, we analyzed
HPCorpusMPI for MPI Common Core functions. The distribution of these functions
is presented in Table 1.

##  5\. MPIrigen – Experimental Results

This study presents MPIrigen, a tool developed to assist MPI programmers in
automatically generating correct MPI functions in an MPI-based domain
decomposition parallel code. We introduce a dedicated downstream task: fine-
tuning MonoCoder, which is a PolyCoder model pre-trained for the C and C++
languages associated with MPI, on HPCorpusMPI, resulting in the creation of
MPIrigen.

Furthermore, we propose an innovative pre-process for the given completion
task and completion tasks in general. A number specifying its line number is
added for each line in the code. Then, all the MPI functions with their
locations are removed and concatenated to the last line (Figure 1). Training
input will be the resulting code and the test will be to predict the last
line. This enables training a language model in a left-to-right fashion,
outputting the MPI functions merely after observing the whole code, thus
enabling better completion with a wider context. MonoCoder is fine-tuned this
way resulting in MPIrigen.

we ran two experiments the first is flat code completion and the second is the
novel approach we are introducing.

MPI understanding. To develop MPIrigen and answer the first RQ, we first
investigated the performance of state-of-the-art pre-trained code language
models (PolyCoder and GPT3.5) in generating MPI codes using varied context
sizes for next-token predictions compared to MonoCoder. This code completion
task has been done over a thousand examples with up to 2048 tokens in
HPCorpusMPI and under an initial context of 100, 300, and 600 tokens (Figure
2). In addition, to check the resilience of these pre-trained models to
semantic information, the same test has been conducted through codes’
TokomPiler (Kadosh et al., 2023c) version. Any semantic information has been
replaced using TokomPiler (1(b)) and thus, a measure of reliance of these pre-
trained models on semantic information has been tested. Results show that as
context grows, MonoCoder is significantly outperforming PolyCoder and GPT-3.5
results, while also proving to not fix on local semantics for those results,
suggesting high generality capabilities.

Finetune task. Since we evaluate MPI function generation, prevalent generation
metrics such as BLUE, Meteor, and Rouge-l are not relevant. Thus, we propose
an HPC-oriented evaluation method. First, functions matching locations are to
be found. Matching toleration is determined by a variance component. The
variance is a flexibility measure for accuracy, as many times, the right
functions appear in one or two lines from the original locations, usually
without interfering with the original code structure (variance zero refers to
exact location predictions). MPI functions with correct locations will be
forwarded to calling evaluation, checking whether the right MPI function has
been called. Afterward, arguments will be checked out of the correct MPI
functions. Scoring is made by calculating the correct to total arguments
ratio. This evaluation was conducted with a variance of 0-2 and as a number of
MPI functions which reveals the accuracy distribution through simple MPI codes
to more complicated ones.

The model was developed using the Huggingface framework and was trained on two
Nvidia Tesla-V100 GPUs, having 32GB memory each. The training was carried out
with a batch size of 2, 2048 tokens and 3 epochs. The location accuracy as
well as functions under variance of 2 reach 80% and converge to 70%, while
arguments reach 95% and converge to 94%. GPT3.5 on the contrary, converged to
50% in locations and approximately 55% in functions, and 80% in arguments
(Figure 3).

High accuracy could be a misleading metric since the model might excel in
merely one frequent function, MPI_Init for example. Therefore, accuracy alone
can not guarantee a generalized operating model and matching function
distribution is necessary. This distribution is presented in Figure 4 with a
variance of two and showcasing that while there is a difference in scale
(i.e., the accuracy), the MPIrigen ability to predict multiple MPI functions
is in proper proportions to the dataset’s diversity.

##  6\. Conclusion

Our findings reveal that widely used models like GPT-3.5 and specialized
multi-lingual code models like PolyCoder exhibit notable performance
degradation when generating MPI codes compared to their outcomes for general-
purpose codes, as shown in (Kadosh et al., 2023c). In contrast, domain-
specific models like MonoCoder, pre-trained for the C and C++ languages
associated with MPI, outperform larger models, showcasing high generality
capabilities, especially when local misleading semantics are mitigated.
Comparative analysis of MPIrigen against GPT zero-shot performance, using the
above evaluation method for MPI functions generation, demonstrates that
MPIrigen excels in generating accurate MPI codes. The success of this tailored
solution underscores the importance of domain-specific fine-tuning in
optimizing language models for parallel computing code generation.

###### Acknowledgements.

This research was supported by the Israeli Council for Higher Education (CHE)
via the Data Science Research Center, Ben-Gurion University of the Negev,
Israel; Intel Corporation (oneAPI CoE program); and the Lynn and William
Frankel Center for Computer Science. Computational support was provided by the
NegevHPC project (neg, [n. d.]) and Intel Developer Cloud (Intel, 2023). The
authors thank Israel Hen and Gabi Dadush for their help and support.

## References

  * (1)
  * neg ([n. d.]) [n. d.].  NegevHPC Project.  https://www.negevhpc.com.  [Online].
  * par ([n. d.]) [n. d.].  Par4All homepage.  http://par4all.github.io/.  [Online].
  * Amini et al. (2012) Mehdi Amini, Béatrice Creusillet, Stéphanie Even, Ronan Keryell, Onig Goubier, Serge Guelton, Janice Onanian McMahon, François-Xavier Pasquier, Grégoire Péan, and Pierre Villalon. 2012.  Par4all: From convex array regions to heterogeneous computing. In _IMPACT 2012: Second International Workshop on Polyhedral Compilation Techniques HiPEAC 2012_.
  * Bae et al. (2013) Hansang Bae, Dheya Mustafa, Jae-Woo Lee, Hao Lin, Chirag Dave, Rudolf Eigenmann, and Samuel P Midkiff. 2013.  The cetus source-to-source compiler infrastructure: overview and evaluation.  _International Journal of Parallel Programming_ 41 (2013), 753–767.
  * Balart et al. (2004) Jairo Balart, Alejandro Duran, Marc Gonzàlez, Xavier Martorell, Eduard Ayguadé, and Jesús Labarta. 2004.  Nanos mercurium: a research compiler for openmp. In _Proceedings of the European Workshop on OpenMP_ , Vol. 8. 2004.
  * Chen et al. (2023b) Brian Chen, Nafis Mustakin, Alvin Hoang, Sakib Fuad, and Daniel Wong. 2023b.  VSCuda: LLM Based CUDA Extension for Visual Studio Code. In _Proceedings of the SC ’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis_ (¡conf-loc¿, ¡city¿Denver¡/city¿, ¡state¿CO¡/state¿, ¡country¿USA¡/country¿, ¡/conf-loc¿) _(SC-W ’23)_. Association for Computing Machinery, New York, NY, USA, 11–17.  https://doi.org/10.1145/3624062.3624064
  * Chen et al. (2024a) Le Chen, Nesreen K Ahmed, Akash Dutta, Arijit Bhattacharjee, Sixing Yu, Quazi Ishtiaque Mahmud, Waqwoya Abebe, Hung Phan, Aishwarya Sarkar, Branden Butler, et al. 2024a.  Position Paper: The Landscape and Challenges of HPC Research and LLMs.  _arXiv preprint arXiv:2402.02018_ (2024).
  * Chen et al. (2024b) Le Chen, Arijit Bhattacharjee, Nesreen Ahmed, Niranjan Hasabnis, Gal Oren, Vy Vo, and Ali Jannesari. 2024b.  OMPGPT: A Generative Pre-trained Transformer Model for OpenMP.  arXiv:2401.16445 [cs.SE]
  * Chen et al. (2023a) Le Chen, Pei-Hung Lin, Tristan Vanderbruggen, Chunhua Liao, Murali Emani, and Bronis de Supinski. 2023a.  LM4HPC: Towards Effective Language Model Application in High-Performance Computing.  _arXiv preprint arXiv:2306.14979_ (2023).
  * Ding et al. (2023) Xianzhong Ding, Le Chen, Murali Emani, Chunhua Liao, Pei-Hung Lin, Tristan Vanderbruggen, Zhen Xie, Alberto E Cerpa, and Wan Du. 2023.  HPC-GPT: Integrating Large Language Model for High-Performance Computing.  _arXiv preprint arXiv:2311.12833_ (2023).
  * Godoy et al. (2023) William Godoy, Pedro Valero-Lara, Keita Teranishi, Prasanna Balaprakash, and Jeffrey Vetter. 2023.  Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation. In _Proceedings of the 52nd International Conference on Parallel Processing Workshops_. 136–144.
  * Hamidouche et al. (2011) Khaled Hamidouche, Joel Falcou, and Daniel Etiemble. 2011.  A framework for an automatic hybrid MPI+ OpenMP code generation.. In _SpringSim (hpc)_. Citeseer, 48–55.
  * Harel et al. (2020) Re’em Harel, Idan Mosseri, Harel Levin, Lee-or Alon, Matan Rusanovsky, and Gal Oren. 2020.  Source-to-source parallelization compilers for scientific shared-memory multi-core and accelerated multiprocessing: analysis, pitfalls, enhancement and potential.  _International Journal of Parallel Programming_ 48, 1 (2020), 1–31.
  * Harel et al. (2023) Re’em Harel, Yuval Pinter, and Gal Oren. 2023.  Learning to parallelize in a shared-memory environment with transformers. In _Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming_. 450–452.
  * Intel (2023) Intel. 2023.  Intel Developer Cloud.  https://www.intel.com/content/www/us/en/developer/tools/devcloud/overview.html.  [Online].
  * Kadosh et al. (2023a) Tal Kadosh, Niranjan Hasabnis, Timothy Mattson, Yuval Pinter, and Gal Oren. 2023a.  Quantifying openmp: Statistical insights into usage and adoption. In _2023 IEEE High Performance Extreme Computing Conference (HPEC)_. IEEE, 1–7.
  * Kadosh et al. (2023b) Tal Kadosh, Niranjan Hasabnis, Timothy Mattson, Yuval Pinter, Gal Oren, et al. 2023b.  PragFormer: Data-driven Parallel Source Code Classification with Transformers.  (2023).
  * Kadosh et al. (2023c) Tal Kadosh, Niranjan Hasabnis, Vy A Vo, Nadav Schneider, Neva Krien, Mihai Capota, Abdul Wasay, Nesreen Ahmed, Ted Willke, Guy Tamir, et al. 2023c.  Domain-Specific Code Language Models: Unraveling the Potential for HPC Codes and Tasks.  _arXiv preprint arXiv:2312.13322_ (2023).
  * Kadosh et al. (2023d) Tal Kadosh, Niranjan Hasabnis, Vy A Vo, Nadav Schneider, Neva Krien, Abdul Wasay, Nesreen Ahmed, Ted Willke, Guy Tamir, Yuval Pinter, et al. 2023d.  Scope is all you need: Transforming LLMs for HPC Code.  _arXiv preprint arXiv:2308.09440_ (2023).
  * Kadosh et al. (2023e) Tal Kadosh, Nadav Schneider, Niranjan Hasabnis, Timothy Mattson, Yuval Pinter, and Gal Oren. 2023e.  Advising OpenMP Parallelization via a Graph-Based Approach with Transformers.  _arXiv preprint arXiv:2305.11999_ (2023).
  * Mahmud et al. (2023) Quazi Ishtiaque Mahmud, Ali TehraniJamsaz, Hung D Phan, Nesreen K Ahmed, and Ali Jannesari. 2023.  AUTOPARLLM: GNN-Guided Automatic Code Parallelization using Large Language Models.  _arXiv preprint arXiv:2310.04047_ (2023).
  * Milewicz et al. (2021) Reed Milewicz, Peter Pirkelbauer, Prema Soundararajan, Hadia Ahmed, and Tony Skjellum. 2021.  Negative Perceptions About the Applicability of Source-to-Source Compilers in HPC: A Literature Review. In _International Conference on High Performance Computing_. Springer, 233–246.
  * Mosseri et al. (2020) Idan Mosseri, Lee-or Alon, Re’Em Harel, and Gal Oren. 2020.  ComPar: optimized multi-compiler for automatic OpenMP S2S parallelization. In _OpenMP: Portable Multi-Level Parallelism on Modern Systems: 16th International Workshop on OpenMP, IWOMP 2020, Austin, TX, USA, September 22–24, 2020, Proceedings 16_. Springer, 247–262.
  * Nichols et al. (2024) Daniel Nichols, Joshua H Davis, Zhaojun Xie, Arjun Rajaram, and Abhinav Bhatele. 2024.  Can Large Language Models Write Parallel Code?  _arXiv preprint arXiv:2401.12554_ (2024).
  * Nichols et al. (2023) Daniel Nichols, Aniruddha Marathe, Harshitha Menon, Todd Gamblin, and Abhinav Bhatele. 2023.  Modeling Parallel Programs using Large Language Models.  _arXiv preprint arXiv:2306.17281_ (2023).
  * Prema et al. (2017) S Prema, R Jehadeesan, and BK Panigrahi. 2017.  Identifying pitfalls in automatic parallelization of NAS parallel benchmarks. In _Parallel Computing Technologies (PARCOMPTECH), 2017 National Conference on_. IEEE, 1–6.
  * Prema et al. (2019) S Prema, Rupesh Nasre, R Jehadeesan, and BK Panigrahi. 2019.  A study on popular auto-parallelization frameworks.  _Concurrency and Computation: Practice and Experience_ 31, 17 (2019), e5168.
  * Saa-Garriga et al. (2015) Albert Saa-Garriga, David Castells-Rufas, and Jordi Carrabina. 2015.  OMP2MPI: Automatic MPI code generation from OpenMP programs.  https://doi.org/10.48550/ARXIV.1502.02921
  * Schneider et al. (2023a) Nadav Schneider, Tal Kadosh, Niranjan Hasabnis, Timothy Mattson, Yuval Pinter, and Gal Oren. 2023a.  MPI-RICAL: Data-Driven MPI Distributed Parallelism Assistance with Transformers. In _Proceedings of the SC ’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis_ _(SC-W ’23)_. Association for Computing Machinery, New York, NY, USA, 2–10.  https://doi.org/10.1145/3624062.3624063
  * Schneider et al. (2023b) Nadav Schneider, Tal Kadosh, Niranjan Hasabnis, Timothy Mattson, Yuval Pinter, and Gal Oren. 2023b.  MPI-RICAL: Data-Driven MPI Distributed Parallelism Assistance with Transformers. In _Proceedings of the SC ’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis_ (Denver, CO, USA) _(SC-W ’23)_. Association for Computing Machinery, New York, NY, USA, 2–10.  https://doi.org/10.1145/3624062.3624063
  * Squar et al. (2020) Jannek Squar, Tim Jammer, Michael Blesel, Michael Kuhn, and Thomas Ludwig. 2020.  Compiler Assisted Source Transformation of OpenMP Kernels. In _2020 19th International Symposium on Parallel and Distributed Computing (ISPDC)_. 44–51.  https://doi.org/10.1109/ISPDC51135.2020.00016
  * Valero-Lara et al. (2023) Pedro Valero-Lara, Alexis Huante, Mustafa Al Lail, William F Godoy, Keita Teranishi, Prasanna Balaprakash, and Jeffrey S Vetter. 2023.  Comparing Llama-2 and GPT-3 LLMs for HPC kernels generation.  _arXiv preprint arXiv:2309.07103_ (2023).