CFG2VEC: Hierarchical Graph Neural Network for Cross-Architectural
  Software Reverse Engineering
Mission-critical embedded software is critical to our society's
infrastructure but can be subject to new security vulnerabilities as technology
advances. When security issues arise, Reverse Engineers (REs) use Software
Reverse Engineering (SRE) tools to analyze vulnerable binaries. However,
existing tools have limited support, and REs undergo a time-consuming, costly,
and error-prone process that requires experience and expertise to understand
the behaviors of software and vulnerabilities. To improve these tools, we
propose $\textit{cfg2vec}$, a Hierarchical Graph Neural Network (GNN) based
approach. To represent binary, we propose a novel Graph-of-Graph (GoG)
representation, combining the information of control-flow and function-call
graphs. Our $\textit{cfg2vec}$ learns how to represent each binary function
compiled from various CPU architectures, utilizing hierarchical GNN and the
siamese network-based supervised learning architecture. We evaluate
$\textit{cfg2vec}$'s capability of predicting function names from stripped
binaries. Our results show that $\textit{cfg2vec}$ outperforms the
state-of-the-art by $24.54\%$ in predicting function names and can even achieve
$51.84\%$ better given more training data. Additionally, $\textit{cfg2vec}$
consistently outperforms the state-of-the-art for all CPU architectures, while
the baseline requires multiple training to achieve similar performance. More
importantly, our results demonstrate that our $\textit{cfg2vec}$ could tackle
binaries built from unseen CPU architectures, thus indicating that our approach
can generalize the learned knowledge. Lastly, we demonstrate its practicability
by implementing it as a Ghidra plugin used during resolving DARPA Assured
MicroPatching (AMP) challenges.
Shih-Yuan Yu, Yonatan Gizachew Achamyeleh, Chonghan Wang, Anton Kocheturov, Patrick Eisen, Mohammad Abdullah Al Faruque
# CFG2VEC: Hierarchical Graph Neural Network for Cross-Architectural Software Reverse Engineering

###### Abstract

Mission-critical embedded software is critical to our society's infrastructure but can be subject to new security vulnerabilities as technology advances. When security issues arise, _Reverse Engineers_ (REs) use _Software Reverse Engineering_ (SRE) tools to analyze vulnerable binaries. However, existing tools have limited support, and REs undergo a time-consuming, costly, and error-prone process that requires experience and expertise to understand the behaviors of software and vulnerabilities. To improve these tools, we propose _cfg2vec_, a Hierarchical _Graph Neural Network_ (GNN) based approach. To represent binary, we propose a novel _Graph-of-Graph_ (GoG) representation, combining the information for control-flow and function-call graphs. Our _cfg2vec_ learns how to represent each binary function compiled from various CPU architectures, utilizing hierarchical GNN and the siamese network-based supervised learning architecture. We evaluate _cfg2vec_'s capability of predicting function names from stripped binaries. Our results show that _cfg2vec_ outperforms the state-of-the-art by 24.54% in predicting function names and can even achieve 51.84% better given more training data. Additionally, _cfg2vec_ consistently outperforms the state-of-the-art for all CPU architectures, while the baseline requires multiple training to achieve similar performance. More importantly, our results demonstrate that our _cfg2vec_ could tackle binaries built from unseen CPU architectures, thus indicating that our approach can generalize the learned knowledge. Lastly, we demonstrate its practicability by implementing it as a _Ghira_ plugin used during resolving DARPA _Assured MicroPatching_ (AMP) challenges.

 Software Reverse Engineering; Binary Analysis; Cross-Architecture; Machine Learning; Graph Neural Network;

## I Introduction

In mission-critical systems, embedded software is vital in manipulating physical processes and executing missions that could pose risks to human operators. Recently, the _Internet of Things_ (IoT) has created a market valued at 19 trillion dollars and drastically grown the number of connected devices to approximately 35 billion in 2025 [1, 2, 3]. However, while IoT brings technological growth, it unintendedly exposes mission-critical systems to novel vulnerabilities [4, 5, 6]. The reported number of IoT cyberattacks increased by 300% in 2019 [7], while the discovered software vulnerabilities rose from 1.6k to 100k [8]. The consequence can be detrimental, as indicated in [9], the _Hearthped_ bug [10] can lead to a leakage of up to 64K memory, threatening not only personal but also organizational information security. Besides, _Shellshock_ is a bash command-line interface shell bug, but it has existed for 30 years and remains a threat to enterprises today [11, 12]. For mission-critical systems, unexpected disruptions can incur millions of dollars even if they only last for a few hours or minutes [13]. As a result, timely analyzing these impacted software and patching vulnerabilities becomes critical.

However, mission-critical systems usually use software that can last for decades due to the criticality of the missions. Over time, these systems become legacy, and the number of newly-discovered threats can increase (as illustrated in Figure 1). Typically, for legacy software, the original development environment, maintenance support, or source code might no longer exist. To address vulnerabilities, vendors offer patches in the form of source code changes based on the current software version (e.g., ver 0.9). However, the only available data in the legacy system is binary based on its source code (e.g., ver 0.1). Such a version gap poses challenges in applying patches to the legacy binaries, leaving the only solution for applying patches to legacy software as direct binary analysis. Today, as Figure 2 shows, _Reverse Engineers_ (REs) have to leverage _Software Reverse Engineering_ (SRE)

Fig. 1: Legacy software life cycle.
tools such as _Ghidra_[14], _HexRays_[15], and _radare2_[16] to first disassemble and decompile binaries into higher-level representations (e.g., C or C++). Typically, these tools take the debugging information, strings, and the symbol-table and binary to reconstruct function names and variable names, allowing REs to rebuild a software's structure and functionality without access to source code [17]. For REs, these symbols encode the context of the source code and provide invaluable information that could help them to understand the program's logic as they work to patch vulnerable binaries. However, symbols are often excluded for optimizing the binary's footprint in mission-critical legacy systems where memory is limited. Because recovering symbols from _stripped binaries_ is not straightforward, most decompilers assign meaningless symbol names to coding elements. As for understanding the software semantics, REs have to leverage their experience and expertise to consume the information and then interpret the semantics of each coding element.

Recent works tackle these challenges with _Machine Learning_ (ML), aiming to recover the program's information from raw binaries. For example, [18], and [19] associate code features to function names and model the relationships between such code features and the corresponding source-level information (variable names in [19], variable & function names in [18]). Meanwhile, [20] and [21] use an encoder-decoder network structure to predict function names from stripped binary functions based on instruction sequences and control flows. However, none of them support cross-architectural debug information reconstruction. On the other side, there exist works focusing on the cross-platform in their ML models [22, 23, 24]. These works focus on modeling the binary code similarity, extracting a real-valued vector from each control-flow graph (CFG) with attributed features, and then computing the _Structural Similarity_ between the feature vectors of binary functions built from different CPU architectures.

In this paper, as part of a multi-industry-academia joint initiative between Siemens, the Johns Hopkins University Applied Physics Laboratory (JHU/APL), BAE Systems (BAE), and UCI, we propose _cfg2vec_, which utilizes a hierarchical _Graph Neural Network_ (GNN) for reconstructing the name of each binary function, aiming to develop the capacity for quick patching of legacy binaries in mission-critical systems. Our _Cfg2vec_ forms a _Graph-of-Graph_ (GoG) representation, combining CFG and FCG to model the relationship between binary functions' representation and their semantic names. Besides, _cfg2vec_ can tackle cross-architectural binaries thanks to the design of Siamese-based network architecture, as shown in Figure 3. One crucial use case of cross-architectural decompilation is _patching_, where the goal is to identify a known vulnerability or a bug and apply a patch. However, there can be architecture gaps when software with a bug can be compiled into many devices with diverse hardware architectures. For example, it is challenging to patch a stripped binary from an exotic embedded architecture compiled ten years ago that is vulnerable to a known attack such as _Heartbleed_[10]. While the reference patch is available in software, the reference architecture may not be readily available or documented, or the vendor may no longer exist. Under such circumstances, mapping code features across architectures is very helpful. It would allow for identifying similarities in code between a stripped binary that is vulnerable and its reference patch, even if the patch were built for a different type of CPU architecture. For _cfg2vec_, our targeted contributions are as follows:

* We propose representing binary functions in _Graph-of-Graph_ (GoG) and demonstrate its usefulness in reconstructing function names from stripped binaries.
* We propose a novel methodology, _cfg2vec_ that uses a hierarchical _Graph Neural Network_ (GNN) to model control-flow and function-calling relations in binaries.
* We propose using cross-architectural loss when training, allowing _cfg2vec_ to capture the architecture-agnostic representations of binaries.
* We release _cfg2vec_ in a GitHub repository: [https://github.com/AICPS/mindsight_cfg2vec](https://github.com/AICPS/mindsight_cfg2vec).
* We integrate our _cfg2vec_ into an experimental Ghidra plugin, assisting the realistic scenarios of patching DARPA _Assured MicroPatching_ (AMP) challenge binaries.

The paper is structured as follows: Section II discusses related works and fundamentals to provide a better understanding of the paper. Section III describes _cfg2vec_, including problem formulation, data preprocessing, and our main pipeline introduction. Section IV shows our experimental results. Lastly, we conclude the paper in Section V.

## II Related Work

This section introduces software reverse engineering backgrounds, discusses the related works using machine learning to improve reverse engineering, and ultimately covers graph learning for binary analysis.

### _Software Reverse Engineering_

_Software Reverse Engineering_ (SRE) aims at understanding the behavior of a program without having access to its source code, often being used in many applications such as detecting malware [25, 26], discovering vulnerabilities, and patching bugs in _legacy software_[27, 28]. One primary tool that _Reverse Engineers_ (REs) use to inspect programs is _disassembler_ which translates a binary into low-level assembly code. Examples of such tools include _GNU Binutils' objdump_[29], _IDA_[15], Binary Ninja [30], and Hopper [31]. However, even with these tools, reasoning at the assembly level still requires considerable cognitive effort from RE experts.

More recently, REs use _decompilers_ such as _Hex-Rays_[32], or _Ghidra_[14] to reverse the compiling process by further translating the output of disassemblers into the code that ensembles high-level programming languages such as C or C++ to reduce the burden of understanding assembly code. From assembly instructions, these decompilers can use program analysis and heuristics to reconstruct variables, types, functions, and control flow structure of a binary. However, the decompilation is incomplete even if these decompilers generate a higher-level output for better code understanding.
The reason is that the compilation process discards the source-level information and lowers its abstraction level in exchange for a smaller footprint size, faster execution time, or even security considerations. The source-level information such as comments, variable names, function names, and idiomatic structure can be essential for understanding a program but is typically unavailable in the output of these decompilers.

As Figure 2 demonstrated, REs use disassemblers or decompilers to generate high-level source code. Besides, [33] indicates REs will take notes and grant a name to those critical functions related to the vulnerabilities. This will create an annotated source code based on the high-level machine-generated source code. While annotating the source code, REs also analyze the significant part related to the vulnerability and ignore those general instructions or unrelated codes. At the same time, understanding the logic flow among functions is another major task they must focus on resolving their tasks. After classification, annotation, and understanding, REs experiment with several viable remedies to find the correct patch to fix the vulnerability.

### _Machine Learning for Reverse Engineering_

Software binary analysis is a straightforward first step to enhance security as developers usually deploy software in binaries [34]. Usually, experts conduct the patching process or vulnerability analysis by understanding the compilation source, function signatures, and variable information. However, after the compilation, such information is usually stripped or confuscated deliberately (e.g., _obfuscation_). Software binary analysis becomes more challenging in this case as developers have to recover the source-level information based on their experience and expertise. The early recovery work for binaries focuses on manual completion but suffers from low efficiency, high cost, and the error-prone nature of reverse engineering.

As _Machine Learning_ (ML) has significantly advanced in its reasoning capability, applying ML and reconstructing higher-level source code information as an alternative to manual-based approaches has attracted considerable research attention. For example, [35] was the first approach that used neural network-based and graph-based models, predicting the function types to assist the reverse engineer in understanding the binary. [36] also predicted function names with neural networks, aggregating the related features of sections of binary vectors. Then, it analyzes the connections between each function in the source code (e.g., Java) and their corresponding function names for function name prediction. [18], on the other hand, did not use a neural network. It combined a decision-tree-based classification algorithm and a structured prediction with a probabilistic graphical model, then matched the function name by analyzing symbol names, types, and locations. However, [18] can only predict from a predetermined closed set, incapable of generalizing to new names.

As the languages for naming functions are similar to natural language, recent research works start leaning toward the use of _Natural Language Processing_ (NLP) [20, 21, 37]. Precisely, these models predict semantic tokens based on the function names in the library, comprising the function name during inference. The underlying premise is that each token corresponds in some way to the attributes and functionality of the function. [20] uses _Control-Flow Graph_ (CFG) to predict function names. It combined static analysis with LSTM and transformer neural model to realize the name of functions. However, the dataset that consisted of unbalanced data and insufficient features was limited and hindered utter performance. [37] was designed to solve the limitation of the dataset. It provided _UbuntuDataset_ that contained more than 9 million functions in 22K software. [21] demonstrated the framework's effectiveness by building a large dataset. It considers the fine-grained sequence and structure information of assembly code when modeling and realizing function name prediction. Meanwhile, [21] reduced the diversity of data (instructions or words) while keeping the basic semantics unchanged, similar to word stemming and semantics in NLP. However, these works have low precision scores for prediction tasks, exampled by [21], only achieving around 41% in correctly predicting the function name subtokens. Moreover, the metrics for the inference of unknown functions are substantially lower [21], making it difficult for REs to find it helpful in practice.

Although many existing works can reconstruct source-level information, none of them supports reconstructing cross-platform debug information. Cross-compilation is becoming more popular in the development of software. Hardware manufacturers, for instance, often reuse the same firmware code base across several devices running on various architectures [38]. A tool that performs cross-architecture function name prediction/matching would be beneficial if we have a stripped binary compiled for one architecture and a binary of a comparable program compiled for another architecture with debug symbols. We may use the binary with the debug symbols to predict the names of functions in the stripped binary, which significantly aids debugging. A tool that could capture the architecture-agnostic characteristics of binaries

Fig. 2: The RE flow to solve security issues.
would also help in malware detection as the source code of malware can be compiled in different architectures [38, 39]. Comparing two binaries of different architectures becomes more complicated because they will have different instruction sets, calling conventions, register sets, etc. Furthermore, assembly instructions from different architectures cannot often be compared directly due to the slightly different behavior of different architectures [40]. Cross-architecture function name prediction will assist in finding a malicious function in a program compiled for different architectures by learning its features from a binary compiled for just one architecture. The tools mentioned above are not architecture-agnostic; thus, we cannot utilize them for such applications. To address the flaws mentioned above, aid in creating more efficient decompilers, and make reverse engineering more accessible, we propose _cfg2vec_. Incorporating the cross-architectural siamese network architecture, our _cfg2vec_ can learn to extract robust features that encompass platform-independent features, enhancing the state-of-the-art by achieving function name reconstruction across cross-architectural binaries.

### _Graph Learning for Binary Analysis_

Graph learning has become a practical approach across fields [41, 42, 43, 44]. Although conventional ML can effectively capture the features hidden in Euclidean data, such as images, text, or videos, our work focuses more on the application where the core data is graph-structured. Graphs can be irregular, and a graph may contain a variable size of unordered nodes; moreover, nodes can have a varying number of neighboring nodes, making deep learning mathematical operations (e.g., 2D Convolution) challenging to apply. The operations in conventional ML methods can only be applied by projecting non-Euclidean data into low-dimensional embedding space. In graph learning, _Graph Embeddings_ (GE) can transform a graph into a vector (embedding of a graph) or a set of vectors (embedding of nodes or edges) while preserving the relevant and structural information about the graph [41]. _Graph Neural Network_ (GNN) is a model aiming at addressing graph-related tasks in an end-to-end manner, where the main idea is to generate a node's representation by aggregating its representation and the representations of its neighbors [42]. GNN stacks multiple graph convolution layers, graph pooling layers, and a graph readout to generate a low-dimensional graph embedding from high-dimensional graph-structured data.

In software binary analysis, many approaches use _ControlFlow Graphs_ (CFGs) as the primary representations. For example, _Genius_ forms an _Attributed Control-Flow Graph_ (ACFG) representation for each binary function by extracting the raw attributes from each _Basic Block_ (BB), a straight-line code sequence with no branching in or out except at the entry and exit, in an ACFG [22]. _Genius_ measures the similarity of a pair of ACFGs through a bipartite graph matching algorithm, and the ACFGs are then clustered based on similarity. _Genius_ leverages a codebook for retrieving the embedding of an ACFG based on similarity. Another approach, _Gemini_, proposes a deep neural network-based model along with a Siamese architecture for modeling binary similarities with greater efficiency and accuracy than other state-of-the-art models of the time [23]. _Gemini_ takes in a pair of ACFGs extracted from raw binary functions generated from known vulnerability in code and then embeds them with a shared _Structure2vec_ model in their network architecture. Once embedded, _Gemini_ trains its model with a loss function that calculates the cosine similarities between two embedded representations. _Gemini_ outperforms models like _Genius_ or other approaches such as bipartite graph matching. In literature, there exist other works that consider the _Function Call Graph_ (FCG) as their primary data structures in binary analysis for malware detection [45]. Our _cfg2vec_ extracts relevant platform-independent features by combining the usage of CFG and FCG, resulting in a _Graph-of-Graph_ (GoG) representation for cross-architectural high-level information reconstruction tasks (e.g., function name).

## III CFG2vec Architecture

This section begins with problem formulation. Next, as Figure 4 shows, we depict how our _cfg2vec_ extracts the _Graph-of-Graph_ (GoG) representation from each software binary. Lastly, we describe the network architecture in _cfg2vec_.

### _Problem Formulation_

In our work, given a binary code, denoted as \(p\), compiled from different CPU architectures, we extract a graph-of-graph (GoG) representation, \(\mathcal{G}=(\mathcal{V},\mathcal{A})\) where \(\mathcal{V}\) is the set of nodes and \(\mathcal{A}\) is the adjacency matrix (As Figure 3 shows). The nodes in \(\mathcal{V}\) represent functions and the edges in \(\mathcal{A}\) indicate their cross-referencing relationships. That says, each of the node \(f_{i}\in\mathcal{V}\) is a CFG, and we denote it as \(f_{i}=(B,A,\phi)\) where the nodes in \(B\) represent the basic blocks and the edges in \(A\) denote their dependency relationships. \(\phi\) is a mapping function that maps each basic block in the assembly form to its corresponding extracted attributes \(\phi(v_{i})=C^{k}\) where C is a numeric value, and k is the number of attributes for the basic block (BB). Whereas the CFG structure is meant to provide more information at the lower BB level, the GoG structure is intended for recovering information at the overarching function level between the CFGs. Figure 3 is an example of a partial GoG structure with a closer inspection of one of its CFG nodes and another of a single CFG BB node, showing the set of features corresponding to that BB node. The goal is to design an efficient and effective graph embedding technique that can be used for reconstructing the function names for each function \(f_{i}\in\mathcal{V}\).

### _Ghidra Data ToolKit for Graph Extraction_

To extract the structured representation required for _cfg2vec_ we leverage the state-of-the-art decompiler _Ghidra_[14] and the _Ghidra Headless Analyzer_1. The _headless analyzer_ is a command-line version of _Ghidra_ allowing users to perform many tasks (such as analyzing a binary file) supported by _Ghidra_ via a command-line interface. For extracting GoG
from a binary, we developed our _Ghidra Data Toolkit_ (GDT); GDT is a set of Java-based metadata extraction scripts used for instrumenting _Ghidra Headless Analyzer_. First, GDT programmatically analyzes the given executable file and stores the extracted information in the internal Ghidra database. Ghidra provides a set of APIs to access the database and retrieve the information about the analyzed binary. GDT uses these APIs to export information such as Ghirda's PCode and call graph for each function. Specifically, the _FunctionManager_ API allows us to manipulate the information of each decompiled function in the binary and acquire the cross-calling dependencies between functions. For each function, we utilized another Ghidra API _DecompInterface2_ to extract 12 attributes associated with each basic block in a function. These attributes precisely correspond to the total number of instructions, including arithmetic, logic, transfer, call, data transfer, SSA, compare, and pointer instructions, as well as other instructions not falling within those categories and the total number of constants and strings within that BB. Lastly, by integrating all of the information, we form a GoG representation \(\mathcal{G}\) for each binary \(p\). We repeat this process until all binaries are converted to the GoG structure. We feed the resulting GoG representations to our model in batches, with the batch size denoted as B.

Footnote 2: Documentation of _Ghidra API DecompInterface_: [https://ghidra.re/ghidra_docs/api/ghidra/app/decompiler/DecompInterface.html](https://ghidra.re/ghidra_docs/api/ghidra/app/decompiler/DecompInterface.html)

### _Hierarchical Graph Neural Network_

Once \(\mathcal{G}\) is extracted from the GDT, we then feed it to our hierarchical network architecture (inspired from [46]) that contains both _CFG Graph Embedding_ layer and _GoG Graph Embedding Layer_ as Figure 4 shows. For each GoG structure, we denote it as \(\mathcal{G}=(\mathcal{V},\mathcal{A})\) where \(\mathcal{V}\) is a set of functions associated with \(\mathcal{G}\) and \(\mathcal{A}\) indicates the calling relationships between the functions in \(\mathcal{V}\). Each function in \(\mathcal{V}\) is in the form of CFG \(f_{i}=(B,A,\phi)\) where each node \(b\in B\) is a BB represented in a fixed-length attributed vector \(b\in R^{d}\), and \(d\) is the dimension that we have mentioned earlier. \(A\) encodes the pair-wise control-flow dependency relationships between these BBs.

#### Iii-C1 CFG Graph Embedding Layer

Our network architecture first feeds all functions in a batch of GoGs to the _CFG Graph Embedding Layer_ consisting of multiple graph convolutional layers and a graph readout operations. The input to this layer is a function \(f_{i}=(B,A,\phi)\) and the output is the fixed-dimensional vector representing a function. For each BB \(b_{k}\) we let \(b_{k}^{0}=b_{k}\), and we update \(b_{k}^{t}\) to \(b_{k}^{t+1}\) with the graph convolution operation shown as follows:

\[b_{k}^{t+1}=f_{G}(Wb_{k}^{t}+\sum_{b_{m}\in A_{k}}Mb_{m}^{t})\]

where \(f_{G}\) is a non-linear activation function such as ReLU, \(A_{k}\) is the list of adjacent BBs for \(b_{k}\), and \(W\in R^{d\times d}\) and \(M\in R^{d\times d}\) are the weights to be learned during the training. We run \(T\) iterations of such a convolution, which can be a tunable hyperparameter in our model. During the updates, each BB gradually aggregates the global information of the control-flow dependency relations into its representation, utilizing the representation of its neighbor. We obtain the final representation for each BB as \(b_{k}^{T}\). To acquire the representation for the function \(f_{i}\), we apply a graph readout operation such as _sum-readout_, described as follows,

\[g^{(T)}=\sum_{b_{k}\in B}b_{k}^{T} \tag{1}\]

We assign the value of \(g^{(T)}\) (a.k.a. CFG embedding) to \(f_{i}\). The graph readout operation can be replaced with _mean-readout_ or _max-readout_.

#### Iii-C2 GoG Graph Embedding Layer

Once all the functions have been converted to fixed-length graph embeddings, we then feed \(\mathcal{G}\) to the second layer of _cfg2vec_, the _GoG Embedding Layer_. Here, for each function \(f_{i}\) we apply another \(L\) iterations of graph convolution with \(\mathcal{F}\) and \(\mathcal{C}\). The updates can be illustrated as follows,

\[f_{k}^{(l+1)}=f_{GoG}(Uf_{k}^{l}+\sum_{f_{m}\in C_{k}}Vf_{m}^{(l)}) \tag{2}\]

where \(f_{GoG}\) is a non-linear activation function and \(C_{k}\) is the list of adjacent functions (calling) for the function \(f_{k}\) and \(U\in R^{d\times d}\) and \(V\in R^{d\times d}\) are the weights to be learned during the training. Lastly, we take the \(f_{k}^{(L)}\) as the representation that considers both CFG and GoG graph structures. We use these updated representations to perform cross-architecture function similarity learning.

Fig. 3: An example of a _Graph-of-Graph_ (GoG) of a binary compiled from a package Freccell with amd64 CPU architecture.
#### Iii-B3 Siamese-based Cross-Architectural Function Similarity

Given a batch of GoGs \(B=\{GoG_{1},GoG_{2},...,GoG_{B}\}\), we apply the hierarchical graph neural network to acquire the set of updated function embeddings, denoted as \(B_{F}=\{f_{1}^{(T)},f_{2}^{(T)},...,f_{K}^{(T)}\}\). We calculate the function similarity for each function pair with cosine similarity, denoted as \(\hat{y}\in[-1,1]\). The loss function \(J\) between \(\hat{Y}\) and a ground-truth label \(y\), which indicates whether a pair of functions have the same function or not, is calculated as follows,

\[J(\hat{y},y)=\left\{\begin{array}{ll}1-y,&\text{if y=1,}\\ MAX(0,\hat{y}-m),&\text{if y=-1,}\end{array}\right. \tag{3}\]

the final loss \(L\) is then calculated as follows,

\[L=H(Y,\hat{Y})=\sum_{i}(J(\hat{y_{i}},y_{i})), \tag{4}\]

where \(Y\) stands for ground-truth labels (either similarity or dissimilarity), and \(\hat{Y}\) represents the corresponding predictions. More specifically, we denote a pair of functions as similar if they are the same but compiled with different CPU architectures. The \(m\) is a constant to prevent the learned embeddings from becoming distorted (by default, \(0.5\)). To maintain the balance between positive and negative training samples, we developed a custom batching algorithm. The function leverages the knowledge gained by adding a binary of some package to a given batch to find and add a binary for the same package, built for a different architecture, to the provided batch as a positive sample. It will also include a binary from another package as a negative sample. This will give any batch a balanced proportion of positive and negative samples. Finally, we use the loss \(L\) to update all the associated weights in our neural networks with an _Adam_ optimizer. Once trained, we then use the model to perform function name reconstruction tasks.

## IV Experimental Results

In this section, we evaluate _cfg2vec_'s capability in predicting function names. We first describe the dataset preparation and the training setup processes. Then, we present the comparison of _cfg2vec_ against baseline in predicting function names. Although many baseline candidates tackle the same problem [18, 20, 37, 21], some require purchasing a paid version of IDA Pro to preprocess datasets, and some even do not open source their implementations. Therefore, [18] was the only feasible choice, as running other models using our datasets was almost impossible. Next, we also show the result of the ablation study over _cfg2vec_. Besides, we exhibit that our _cfg2vec_ can perform architecture-agnostic prediction better than the baseline. Lastly, we illustrate the real-world use case where our _cfg2vec_ is integrated as a _Ghidra_ plugin application for assisting in resolving challenging reverse engineering tasks. We conducted all experiments on a server equipped with Intel Core i7-7820X CPU @3.60GHz with 16GB RAM and two NVIDIA GeForce GTX Titan Xp GPUs.

### _Dataset Preparation_

Our evaluating data source is the ALLSTAR (_Assembled Labeled Library for Static Analysis Research_) dataset, hosted by _Applied Physics Laboratory_ (APL) [47]. It has over 30,000 Debian Jessie packages pre-built from i386, amd64, ARM, MIPS, PPC, and s390x CPU architectures for software reverse engineering research. The authors used a modified _Dockcross_ script in docker to build each package for each supported architecture. Then, they saved each resulting ELF with its symbols, the corresponding source code, header files, intermediate files (.o,.class,.gkd,.gimple), system headers, and system libraries altogether.

To form our datasets, we selected the packages that have ELF binaries built for the amd64, armel, i386, and mipsel CPU architectures. i386 and amd64 are widely used by general computers, especially in the Intel and AMD products, respectively. MIPS and ARM are crucial in embedded systems, smartphones, and other portable electronic devices [48]. In practice, we excluded the packages with only one CPU architecture in the ALLSTAR dataset. Additionally, due to our limited local computing resources, we eliminated packages that were too large to handle. We checked each selected binary on whether the ground-truth symbol information exists using the _Ghidra_ decompiler and Linux file command and removed the ones that do not have them. Lastly, we assembled our primary dataset, called the _AS-4cpu-30k-bin_ dataset, that consists of 27572 pre-built binaries from 1117 packages and 4 CPU architectures, as illustrated in Table I.

Our preliminary experiment revealed that the evaluation had a data leakage issue when splitting the dataset randomly. Therefore, we performed a non-random variant train-test split with a 4-to-1 ratio on the _AS-4cpu-30k-bin_ dataset, selecting roughly 80% of the binaries for the training dataset and leaving the rest for the testing dataset. We referenced [23] for their splitting methods, aiming to ensure that the binaries

Fig. 4: The architecture of _cfg2vec_ with a supervised hierarchical graph neural network approach.
that belong to the same packages stay in the same set, either the training or testing sets. Such a variant splitting method allows us to evaluate _cfg2vec_ truly.

Next, we converted binaries in the _AS-4cpu-30k-bin_ dataset into their _Graph-of-Graph_ (GoG) representations leveraging the GDT mentioned previously in Section III-B. Notably, we processed a batch of binaries related to one package at one time as developers might define user functions in different modules of the same package while putting prototype declarations in that package's main module. For this case, _Ghidra_ indeed recognizes two function instances while one only contains the function declaration and another has its actual function content. As these two instances correspond to the same function name and one contains only dummy instructions, they can thus create noise in our datasets, thus affecting our model's learning. To cope with this, our GDT also searches from other binaries of the same package for the function bodies. If found, our GDT associates that user function with the function graph node with the actual content data. Besides user functions, library function calls may exist, and searching their function bodies in the same package would fail for dynamically loaded binaries. Under such circumstances, _Ghidra_ would recognize these functions as _ThunkFunctions3_ which only contain one dummy instruction. As a workaround, we removed these _ThunkFunctions_ from our data as they might mislead the model's learning. Applying this workaround indicates that our model works in predicting function names for the user and statically linked functions.

Footnote 3: ThunkFunction Manual: [https://ghidra.re/ghidra_docs/api/ghidra/program/model/listing/ThunkFunction.html](https://ghidra.re/ghidra_docs/api/ghidra/program/model/listing/ThunkFunction.html)

We experimented [18] with our datasets, referencing to their implementation4. As [18] used a dataset with 3,000 binaries for experiments, we followed accordingly, preparing datasets with smaller but similar sizes. We achieved this by downsampling from our primary _AS-4cpu-30k-bin_ dataset, creating the _AS-3cpu-9k-bin_ dataset which has 9,000 binaries for i386, amd64, and armel CPU architectures. Furthermore, as [18] supports only one CPU architecture at a time, we then separated the _AS-3cpu-9k-bin_ dataset into different CPU architectures, generating three training datasets for testing [18]: _AS-i386-3k-bin_, _AS-amd64-3k-bin_, and _AS-armel-3k-bin_. For training, we utilized the strip Linux command, converting our original data into three: the original binaries (_debug_), stripped binaries with debug information (_stripped_), and stripped binaries without debug information (_stripped_wo_symtab_) to follow [18]'s required data format. For evaluation, we sampled 100 binaries from our primary dataset for each CPU architecture, labeled _AS-amd-100-bin_, _AS-i386-100-bin_, _AS-armel-100-bin_, and _AS-mipsel-100-bin_. We also have another evaluation dataset called _AS-noMipsel-300-bin_, which contains roughly 300 binaries produced for the amd64, i386, and armel platforms. Table I summarizes the data statistics for all these datasets, including the numbers of packages and binaries, the average number of function nodes, edges, and BB nodes. The following sections will detail how we utilized these datasets during our experiments.

Footnote 4: Debin’s [18] repository: [https://github.com/eth-sri/debin](https://github.com/eth-sri/debin)

### _Evaluation: Function Name Prediction_

Table II demonstrates the results of _cfg2vec_ in predicting function names. For the baseline, we followed [18]'s best setting where the feature dimension of register or stack offset are both 100 to train with our prepared datasets. For _cfg2vec_, we used three GCN layers and one GAT convolution layer in both graph embedding layers. For evaluation, we calculate the p@k (e.g., precision at k) metric, which refers to an average hit ratio over the top-k list of predicted function names. Specifically, we feed each binary represented in GoG into our trained model, converting each function \(f\in F\) and acquiring its function embedding \(h_{f}\). Then, we calculate pairwise cosine similarities between \(h_{f}\) and all the other function embeddings, forming a top-k list by selecting k names in which their embeddings are top-kth similar to \(h_{f}\). If the ground-truth function name is among the top-k list of function name predictions, we regard that as a hit; otherwise, it is a miss. During experiments, we set the top-k value to be 5, so our model can recommend the best five possible names for each function in a binary.

As shown in Table II, _cfg2vec_, trained with the _AS-3cpu-9k-bin_ dataset, can achieve a 69.75% prediction accuracy (e.g., p@1) in inferring function names. For [18], we had to train their models for each CPU architecture separately as it cannot train in a cross-architectural manner. Even so, for amd64 binaries, [18] only achieves 29.32% precision, while for i386 and armel, it performs 52.64% and 53.65%, respectively. This result indicates that in any case, our _cfg2vec_ outperforms [18]. Besides, while [18] only yields one prediction, our _cfg2vec_ suggests five choices, making it flexible for our users (e.g., REs) to select what they believe best fits the function among the best k predicted names. The p@2 to p@5 in Table II demonstrate that our _cfg2vec_ can provide enough hints of function names for users. For example, p@5 of _cfg2vec_ trained with our _AS-3cpu-9k-bin_ dataset can achieve 70.50% precision across all the CPU architecture binaries. We also experimented our _cfg2vec_ with larger datasets. From Table II, we can observe that _cfg2vec_ can have 5.04% performance gain in correctly predicting function names (e.g., p@1). Moreover, the gain increases to 28% when training _cfg2vec_ with the _AS-4cpu-30k-bin_ dataset. We believe training on a larger dataset implies training with a more diversified set of binaries. This allows our model to acquire more knowledge, thus being capable
of extracting more robust features for binary functions. In summary, this result indicates that compared to the baseline, our model can effectively provide contextually relevant names for functions in the decompiled code to our users.

We also experimented with various ablated network setups to study how each component of _cfg2vec_ contributes to performance. First, we simplified our _cfg2vec_ by stripping one GCN layer from the original experimental setup. As shown in Table III, we called this setup _2GCN-GAT_ which slightly decreased the performance by 0.75%. Then, from _2GCN-GAT_ setup, we further removed the GAT layer, calling it _2GCN_. We again observed a marginal performance decrease (\(<\)1%). Next, we eliminated another GCN layer from _2GCN-GAT_, constructing the _GCN-GAT_ setup. For _GCN-GAT_, we saw a drastic drop (4.2%) which highlights that the number of GCN layers can be an essential factor in the performance. Specifically, we found that going from 1 to 2 GCN layers improves prediction accuracy by more than 4%. However, we do not observe a significant performance gain when increasing the number of GCN layers to more than three. Therefore, we retained the original _cfg2vec_ model with its three GCN layers. All in all, as shown in Table III, all these ablated models, still outperform [18], which we attributed to the GoG representation we made for each binary in the dataset.

### _Evaluation: Architectural-agnostic Prediction_

Table IV demonstrates our _cfg2vec_'s capability in terms of cross-architecture support. As [18] supports training one CPU architecture at a time, we had to train it multiple times during experiments. Specifically, we trained [18] on three datasets: _AS-amd64-3k-bin_, _AS-i386-3k-bin_, and _AS-armel-3k-bin_, calling resulting trained models, [18]-amd64, [18]-i386, and [18]-armed, respectively. For these baseline models, we observe that they perform well when tested with the binaries built on the same CPU architecture but poorly with the ones built on different CPU architectures. For instance, [18]-amd64 achieves 29.3% accuracy for amd64 binaries, but performs worse for i386 and armel binaries (13.8% and 7.1%). Similarly, [18]-i386 achieves 52.6% accuracy for i386 binaries, but performs worse for amd64 and armel binaries (6.2% and 1.1%). Lastly, [18]-armel achieves 53.6% accuracy for armel binaries, but performs worse for amd64 and i386 binaries (11.8% and 8.9%). We used the top-1 prediction generated from _cfg2vec_ (a.k.a., p@1) as the comparing metric as [18] produces only one prediction per each function. From the results, we observe that _cfg2vec_ outperforms [18] across all three tested CPU architectures. The fact that _cfg2vec_ performs consistently well across all CPU architectures indicates that our _cfg2vec_ supports cross-architecture prediction.

To evaluate the capability of generalizing the learned knowledge, we tested all models with the _AS-mipsel-100-bin_ dataset, which has binaries built from another famous CPU architecture, mipsel, that our _cfg2vec_ does not train before. For [18], it has lower performance when testing on binaries built from the CPU architectures that it did not train before, exampled by the highest accuracy of [18] to be 13.84% when trained on _amd64_ binaries and evaluated on i386 binaries. In our work, as Table IV shows, our _cfg2vec_ achieves 36.69% accuracy when trained with amd64, i386, and armel binaries but tested on mipsel binaries. For [18], it does not even support analyzing mipsel binaries. In short, these results demonstrate that our _cfg2vec_ outperforms our baseline in the function name prediction task on cross-architectural binaries and generalizes better to the binaries built from unseen CPU architectures. To further investigate _cfg2vec_'s cross-architecture performance, we trained it on three datasets, each consisting of binaries built for two different architectures. We then gave the resulting trained models names that indicated the architectures from which the binaries were derived: _cfg2vec_-armel-i386, _cfg2vec_-amd64-i386, and _cfg2vec_-armel-amd64. These results show that our model performs well in the function name prediction job across all of these scenarios, including when tested on binaries compiled for unknown CPU architectures.
### _The Practical Usage of CFG2VEC_

In this section, we demonstrate how _cfg2vec_ assists REs in dealing with _Defense Advanced Research Projects Agency_ (DARPA) _Assured MicroPatching_ (AMP) challenges binaries. The AMP program aims at enabling fast patching of legacy mission-critical system binaries, enhancing decompilation and guiding it toward a particular goal of a _Reverse Engineer_ (RE) by integrating the existing source code samples, the original build process information, and historical software artifacts.

#### Iv-D1 The MINDSIGHT project

our multi-industry-academia initiative between Siemens, JHU/APL, BAE, and UCI jointly developed a project, _Making Intelligible Decompiled Source by Imposing Homomorphic Transforms_ (MINDSIGHT). Our team focused on building an automated toolchain integrated with _Ghidra_, aiming to enable the decompilation process with (1) a less granular identification of modular units, (2) an accurate reconstruction of symbol names, (3) the lifting of binaries to stylized C code, (4) a principled and scalable approach to reason about code similarity, and (5) the benchmarking of new decompilation techniques using state-of-the-art embedded software binary datasets. To date, our team has developed an open-source tool, _CodeCut5_, to improve the accuracy and completeness of _Ghidra_'s module identification, providing an automated script-based decompilation analysis toolchain to ease the RE's expert interpretation. Besides, we also developed a _Homomorphic Transform Language_ (HTL) to describe transformations on _Abstract Syntax Tree_ (AST) languages and the rules of their composition. Our tool, integrated with _ghidra_, allows developers to transform the decompiled code syntactically while keeping it semantically equivalent. The key idea is to use this HTL to morph a _Ghidra_ AST into a GCC AST to lift the decompiled binary to a high-level C representation. This process can make it easier for REs to comprehend the binary code. _cfg2vec_ is another tool developed in the MINDSIGHT project, enabling the reconstruction of function names, saving the manual guesswork from REs.

Footnote 5: _CodeCut_’s repository: [https://github.com/DARPAMINDSIGHT/CodeCut](https://github.com/DARPAMINDSIGHT/CodeCut)

#### Iv-D2 The cfg2vec plugin

In _MINDSIGHT_ project, we incorporated _cfg2vec_ into _Ghidra_ decompiler as a plugin application. Our _cfg2vec_ plugin assists REs in comprehending the binaries by providing a list of potential function names for each function without its name. Technically, like all _Ghidra_ plugins, our _cfg2vec_ plugin bases on Java with its core inference modules implemented as a REST API in Python 3.8. Once the metadata of a stripped binary is extracted from _Ghidra_ decompiler, it is then sent to the _cfg2vec_ end-point, which calculates and returns the inferred mappings for all the functions. Figure 5 demonstrates the user interface of our _cfg2vec_ plugin. In this scenario, the user must provide the vulnerable and the reference binary with extra debug information, such as function names. The "Match Functions" button triggers _cfg2vec_ functionality and displays the function mapping results in three tables:

* _Matched Table_: displays the mapping of similar functions.
* _Mismatched Table_: displays the mapping of _dissimilar_ functions and, therefore, candidates for patching.
* _Orphan Table_: displays the mapping of functions with a low confidence score.

The groupings reduce REs' workload. Rather than inspecting all functions, they can focus on patching candidate functions (mismatched functions) and the orphans. The "Explore Functions" button invokes Ghidra's function explorer, where the two functions can be compared side-by-side, as shown in Figure 5. This utility allows the user to switch between C and assembly language, thus assisting in confirming or modifying the mappings from the three tables. Regarding

Fig. 5: The plugin screenshot integrated into Ghidra.
_cfg2vec_'s function prediction, the "Rename Function" button takes the selected row from the tables and imposes the name from the patched binary in the vulnerable binary. When the "Match Functions" button fires, we invoke the FCG and CFG generators for the two programs (vulnerable and patched).

#### Iv-B3 The use-case for AMP challenge binaries

DARPA AMP challenges is about REs to patch a vulnerability regarding a weak encryption algorithm where the encryption of communication traffic was accomplished with a deprecated cipher suite, Triple DES or 3DES [49]. For this challenge, REs have to analyze the vulnerable binary, identify functions and instructions to be patched, _3DES cipher suite_ in this case, and patch 3DES-related function calls and instructions with the ones for AES [50]. All these steps happen at the decompiled binary level, and the vulnerable binaries are optimized by a compiler and stripped of the debugging information and function names. Furthermore, these binaries are sometimes statically linked against libraries such as GNU C Library [51] or OpenSSL, which introduce many extra functions to the binary (some of which will never be called/used). Given these complications, it becomes a non-trivial task for an RE to make sense of all these functions, find the problem, and successfully patch the problem. The direct usage of our _cfg2vec_ plugin was to pick a function of interest with stripped information and see predictions of potential function names or matching functions from the available reference binary to confirm that whether this function is in the critical path during RE's problem solving. As Figure 5 shows, our plugin allows users to see possible matches between functions from a stripped vulnerable binary and functions from a patched (reference) binary with extra information. REs may then leverage such information and make appropriate notes for that function, allowing them to complete their jobs more efficiently. The main feedback we received from REs who used the tool was that this is the functionality REs would like to have. However, the accuracy and usability of the tool were not high enough to truly utilize the tool's potential.

## V Conclusion

This paper presents _cfg2vec_, a Hierarchical Graph Neural Network-based approach for software reverse engineering. Building on top of _Ghidra_, our _cfg2vec_ plugin can extract a _Graph-of-Graph_ (GoG) representation for binary, combining the information from Control-Flow Graphs (CFG) and Function-Call Graphs (FCG). _Cfg2vec_ utilizes a hierarchical graph embedding framework to learn the representation for each function in binary code compiled into various architectures. Lastly, our _cfg2vec_ utilizes the learned function embeddings for function name prediction, outperforming the state-of-the-art [18] by an average of 24.54% across all tested binaries. By increasing the amount of data, our model achieved 51.84% better. While [18] requires training once for each CPU architecture, our _cfg2vec_ still can outperform consistently across all the architectures, only with one training. Besides, our model generalizes the learning better [18] to the binaries built from untrained CPU architectures. Lastly, we demonstrate that our _cfg2vec_ can assist the real-world REs in resolving _Darpa Assured MicroPatching_ (AMP) challenges.

## Acknowledgment

This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) and Naval Information Warfare Center Pacific (NIWC Pacific) under Contract Number N66001-20-C-4024. The views, opinions, and/or findings expressed are those of the author(s) and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government.
