11institutetext: Department of Data Science,
New Jersey Institute of Technology, Newark, NJ, USA 11email:
<EMAIL_ADDRESS>22institutetext: Accenture PLC 22email:
{bai.chien.kao<EMAIL_ADDRESS>
# Profile of Vulnerability Remediations in Dependencies Using Graph Analysis

Fernando Vera 11 Palina Pauliuchenka 11 Ethan Oh 11 Bai Chien Kao 22 Louis
DiValentin 22 David A. Bader 11

###### Abstract

This research introduces graph analysis methods and a modified Graph Attention
Convolutional Neural Network (GAT) to the critical challenge of open source
package vulnerability remediation by analyzing control flow graphs to profile
breaking changes in applications occurring from dependency upgrades intended
to remediate vulnerabilities. Our approach uniquely applies node centrality
metrics—degree, norm, and closeness centrality—to the GAT model, enabling a
detailed examination of package code interactions with a focus on identifying
and understanding vulnerable nodes, and when dependency package upgrades will
interfere with application workflow. The study’s application on a varied
dataset reveals an unexpected limited inter-connectivity of vulnerabilities in
core code, thus challenging established notions in software security. The
results demonstrate the effectiveness of the enhanced GAT model in offering
nuanced insights into the relational dynamics of code vulnerabilities, proving
its potential in advancing cybersecurity measures. This approach not only aids
in the strategic mitigation of vulnerabilities but also lays the groundwork
for the development of sophisticated, sustainable monitoring systems for the
evaluation of work effort for vulnerability remediation resulting from open
source software. The insights gained from this study mark a significant
advancement in the field of package vulnerability analysis and cybersecurity.

###### Keywords:

Graph Attention Convolutional Neural Network (GAT), Package Vulnerability
Analysis, open source, package upgrade, Knowledge Graph, Node Centrality
Metrics, Cybersecurity, Deep Learning Applications, Network Analysis, Code
Vulnerability Mitigation.

##  1 Introduction

Preventing and analyzing vulnerabilities is crucial for averting cyber attacks
and protecting programs and components [32]. In the face of the increasing
complexity and abstraction via dependencies and version management, the
likelihood of vulnerabilities residing within application code in this
dynamic, continuously updating world is heightened [15]. It’s clear that
during code development, it is not only essential to identify and address the
weakest link or the most apparent vulnerability, but also to pursue continuous
solutions with each code update [14]. To effectively protect yourself from
cyber attacks, three steps are proposed: 1) know the vulnerabilities, 2)
understand their impact on the application, and 3) act on them in depth.

Vulnerabilities remediation in open source packages is an essential process
for the open source community and the security of many users. This process
consists of identifying vulnerabilities, reporting them, and analyzing their
impact [2]. Open source maintainers then triage those vulnerabilities and bugs
and prioritize updates, with each update for the affected package fixing
affected functions and removing bugs. The new package is then updated to the
community without the targeted vulnerability.

Various tools are available to facilitate the discovery of vulnerabilities in
dependencies. CVEfixes is a repository of code that can be used to replicate
the comprehensive database of vulnerabilities curated from Common
Vulnerabilities and Exposures (CVE) records in the public U.S. National
Vulnerability Database (NVD). Static and dynamic code analysis capabilities
can be used to used to discover new vulnerabilities. Additional code analysis
techniques can be used beyond static analysis as well. There are a variety of
techniques used to perform vulnerability analysis on control flow graphs [26]
for the application. CodeQL [27] is a powerful tool used by the GitHub
community to identify and modify functions during their development.

Some scanning techniques and vendors have emerged to address vulnerabilities
residing in the package dependencies. Generally these solutions perform a
Software Composition Analysis (SCA) [12] technique in order to create a
Software Bill of Materials (SBOM). A typical SBOM lists (to varying degrees of
depth) the specific versions of dependencies used to compile the software
[30]. This dependency version list can be correlated to existing vulnerability
repositories to find the exact vulnerabilities present in each specific
dependency used to compile the software. Those specific dependencies with
vulnerabilities can be targeted for upgrade to newer versions where those
vulnerabilities are fixed by changing the version number and recompiling the
code with the newer version. Attempting to migrate to a newer version
sometimes causes errors however, and the application will refuse to compile,
or have erroneous functionality when run.

Understanding the impact of changes between dependency versions, along with
the functional interconnections and network structures is a task of that can
be used to estimate the likelihood of a specific package upgrade causing
breaking changes in application code. Given that all applications are distinct
and highly adaptable, there is a need for tools that generalize across
different applications and can provide insights into the complexity of the
upgrade. For this reason, we turn to knowledge graphs.[25, 13] Constructing a
graph enables us to comprehend the interactions between functions within the
code. In the context of this research, the aim is to represent functions in
the code as entities in a graph space. Constructing a Control Flow graph of
interactions between functions in the code allows examination of a
representation of the application code and dependencies in the graph space,
with the caller-callee relationship serving as the connecting link [33]. This
approach allows us to observe and analyze interactions and relationships
between functions. We propose that gaining a relational understanding of code
can assist in identifying the impact of changes in dependencies on the
workflow of the application and the downstream effects, allowing individuals
to move closer to full automation of package upgrades with minimal impacts to
functionality. This can directly translate to improved security by allowing
automated processes to upgrade vulnerable package versions and close
vulnerabilities when upgrades cause a minimal impact to the functionality of
the application.

Understanding the impact of changed function nodes between two versions of an
application using a graph is necessitates a profound comprehension of the
network’s topology and its connectivity attributes [34]. This knowledge is
proving indispensable in identifying the significance of the affected nodes to
the overall operation of the network and in making informed interventions on
the impact of an upgrade decision [4]. This process is involving an intricate
analysis of their centrality, density, connected components, degree
associativity coefficient, and the pathways traversing through them. Such
comprehensive evaluations are facilitating strategic decision-making for
optimally mitigating vulnerabilities, thereby ensuring a more robust and
secure network architecture. This aspect is particularly critical in software,
where functions can exhibit complex nested dependencies and feedback loops.

##  2 Background and Related Work

In the realm of package function analysis, traditional methods have primarily
revolved around static and dynamic analysis techniques. Static analysis, while
useful for a broad overview, often lacks the context-specific insights
garnered from dynamic runtime analysis [19]. Additionally, these techniques
typically deal with implementations of functions within the application, and
not vulnerabilities emerging from the dependencies of the application. Some
scanning techniques and vendors have emerged to address vulnerabilities
residing in the package dependencies. Dependency Vulnerabilities are
remediated in this paradigm by upgrading the package version to a version in
which the vulnerability is closed and recompiling the code. If the version of
the package with the remediation causes the software to no longer compile or
fail unit tests, then additional work will be necessary to update to
application code to be compatible with the remediated package version.
Understanding the effects of upgrading from the current package version to a
remediated package version is currently an opaque process.

Control flow graph analysis is a useful tool for understanding software
behavior, allowing developers to easily visualize the flow of execution within
a program. By reducing abstract and complex code into manageable graphics, it
provides information about the structure of the program and its possible
vulnerabilities and affected functions. This analysis is crucial to optimize
code performance and ensure strong cybersecurity measures. In our case, we
replace the control flow with a knowledge graph related to the use of
algorithms and measures focused on large-scale analysis.

Graph Attention Networks (GAT) have emerged as a promising tool in similar
analytical contexts. Originally conceptualized by Veličković et al.[29], GATs
leverage the attention mechanism to provide node-specific contextual insights,
enhancing the accuracy of feature representation in graph-structured data.
This attribute of GATs has been effectively utilized in various domains,
including but not limited to, bioinformatics, social network analysis, and
natural language processing [31]. In package function analysis, employing GAT
presents an innovative opportunity to address the existing limitations of
traditional methods. It offers a scalable approach to analyze the intricate
interrelationships and dependencies among package functions represented as
nodes in a graph. This method aligns with the recent trends in utilizing deep
learning techniques for software engineering challenges, as documented in
several contemporary studies [8].

##  3 Methodology

This section is outlining key definitions and delving into the critical
elements that underpin our methodological approach. A portion of our research
is the deployment of a modified Graph Attention Network (GAT), an advanced
model that is being specifically designed to analyze feature code interactions
and study the importance of changed functions represented as nodes in the
graph.

###  3.1 Modified Graph Attention Neural Network

The adaptation of the GAT model is motivated by the lack of current visibility
into the effects of upgrading package versions, which directly affects
developers ability to stay on top of package vulnerabilities originating from
dependencies. Unlike traditional GAT models based solely on norm [29], our
modified version incorporates essential node centrality metrics (degree
centrality, norm, and closeness) to evaluate the importance and
interrelationships of various functions within the code, with a particular
focus on identifying and evaluating vulnerable nodes. This enhancement
increases the ability of GATs to provide a more refined analysis of the
relational dynamics within the code, offering a comprehensive perspective on
the impact of changes in dependency versions and the potential risk of
breakage resulting from changed code.

### Rationale for Modification

Utilizing the Knowledge Graph for network analysis, particularly emphasizing
on graph attention Neural Network (GAT) as defined in the work of Veličković
et al. [29], is enabling us to train extensive datasets for the purpose of
evaluating and quantifying the significance of nodes within a network. This
base model is facilitating accurate detection of nodes based on both
transductive methods (such as Cora, Citeseer, Pubmed) and inductive
methodologies. Presented as a convolutional-like neural network operating on
knowledge graph-structured data, this model assigns an importance metric to
nodes within their neighborhoods.

The input to our layer consists of a set of node features,
$h=\\{\mathbf{h}_{1},\mathbf{h}_{2},\ldots,\mathbf{h}_{N}\\}$, with
$\mathbf{h}_{i}\in\mathbb{R}^{F}$, where $N$ denotes the number of nodes and
$F$ the number of features per node. This layer is producing a new set of node
features,
$h^{\prime}=\\{\mathbf{h}^{\prime}_{1},\mathbf{h}^{\prime}_{2},\ldots,\mathbf{h}^{\prime}_{N}\\}$,
with a potentially different cardinality $F^{\prime}$. A shared linear
transformation, parameterized by a weight matrix
$\mathbf{W}\in\mathbb{R}^{F^{\prime}\times F}$, is applied to each node. The
GAT then performs a self-attention mechanism $a$, yielding attention
coefficients $e_{ij}$ as:

$e_{ij}=a(\mathbf{W}\mathbf{h}_{i},\mathbf{W}\mathbf{h}_{j})$ (1)

These coefficients indicate the importance of node $j$’s features to node $i$.
The GAT model is assessing the first-order neighbors of $i$, necessitating a
normalization of coefficients across all node features $j$ using the softmax
function:

$\alpha_{ij}=\text{softmax}_{j}(e_{ij})=\frac{\exp(e_{ij})}{\sum_{k\in\mathcal{N}_{i}}\exp(e_{ik})}$
(2)

it is calculated the coefficient most relevant of each feture of node using
the norm. It identified as $\alpha_{ij}$. Replacing equation (1) in equation
(2)

$\alpha_{ij}=\frac{\exp\left(\text{LeakyReLU}\left(a^{\top}[\mathbf{W}{h}_{i}|\mathbf{W}{h}j]\right)\right)}{\sum{k\in\mathcal{N}(i)}\exp\left(\text{LeakyReLU}\left({a}^{\top}[\mathbf{W}{h}_{i}|\mathbf{W}{h}_{k}]\right)\right)}$
(3)

In line with the original GAT model [29], with the attention mechanism $a$ as
a single-layer feedforward neural network, we define a parameterized weight
matrix $\mathbf{a}\in\mathbb{R}^{2F^{\prime}}$ and apply the LeakyReLU
nonlinearity (negative input slope of 0.2). $a^{\top}$ is a learnable weight
vector in the single-layer feedforward neural network which constitutes the
attention mechanism. $\mathcal{N}(i)$ denotes the neighborhood of node $i$ in
the graph. $\|\|$ represents the concatenation operation.

$e_{ij}=a(\mathbf{W}\mathbf{h}_{i},\mathbf{W}\mathbf{h}_{j})$ (4)

This process calculates the most relevant coefficient for each feature of a
node using the norm, identified as $\alpha_{ij}$. To enhance the focus and
applicability of these coefficients, particularly in code analysis, we are
introducing a modification $\beta_{ij}$ to highlight the robustness and
criticality of all nodes. This modification encompasses functions such as
degree centrality, the norm, and closeness centrality metrics, relevant to
nodes $i$ and $j$. Consequently, the revised attention coefficient,
$\alpha^{\prime}_{ij}$, is defined as:

$\alpha^{\prime}_{ij}=\alpha_{ij}\cdot\beta_{ij}$ (5)

This novel approach is proving instrumental in evaluating the importance of
specific functions within their neighborhoods, offering insights into the
criticality of an affected function in a code structure. The modified model
enables the generation of a normalized score ranging from $0$ to $1$, which
reflects the importance of nodes based on metrics such as degree, norm, and
centrality. This scoring system facilitates a more nuanced understanding of
node significance within the graph structure, particularly in the context of
package code analysis.

###  3.2 Mapping and Mitigating Vulnerability in a Code Development

Figure 1: In this figure, we are providing a conceptual description of the
actors considered behind the open source. We are taking into account a generic
software ecosystem of code from key elements such as the operating system,
language, and software, packages or others that enable the code to function
and connect with the real world. A code under development typically begins
with the Open Source Base code (primary functions) that is interacting with
the repository, the operating system, and other components of the ecosystem.
This base code is undergoing updates $Nth$ times to enhance its functionality.
It is crucial to emphasize that within each component, there could exist
certain flaws which might potentially manifest as vulnerabilities within the
code.

Vulnerabilities are recorded in the NVD repositories from various sources
during the development process. These can be inherent from the base of the
code (fig.1) to the programming language used, integrated through specific
packages or software, or due to operating system flaws[7, 9]. These
vulnerabilities are remediated by standardizing community-driven coding
practices[24], like SAST and DAST scanning, and bug reporting. Nevertheless,
vulnerability mapping is continuously evolving, with new vulnerabilities
emerging or being discovered over time, as well as new tools and methods for
finding them[3, 28]. Thus, it is proving beneficial to maintain an updated,
dynamic, and comprehensive connection map, which serves as a guide to pinpoint
the source and address these vulnerabilities swiftly and precisely.

From the foundational layer of code, the language in which it is being written
is emerging[2], along with the environment and configurations is more correct
and the built-in packages[1]. From this foundation, the core code is
developing, forming the heart of the program with its principal functions. An
initial version of a code is committed, and subsequent new functionality and
features are added along with errors being resolved with updates. Each update
introduces a new layer of functions and packages that can improve the
structure of the code but also can also add complexity (fig. 1). A new update
may be integrating previous functions into the core (at the discretion of each
software developer) or may be leaving them as branches outside the core. Over
time, vulnerabilities can appear in any new update or may be stem all the way
back to the base code. A vulnerable code fragment, if reused across updates,
can extend its reach to critical branches, potentially impacting the entire
application’s core structure.Understanding the network of functions within the
code is crucial to mitigate such risks and ensuring that updates do not
inadvertently compromise the integrity of the application.

###  3.3 Knowledge Graph in Cybersecurity

According to Hogan [11], a debate is ongoing regarding the precise definition
of a Knowledge Graph, yet consensus exists about its remarkably high
adaptability. In the context of this discussion, the knowledge graph $G$ is
defined as $G=(E,R,S)$, where $E$ represents the entity (node), $R$ symbolizes
the relationship (edge), and $S$ denotes relationship facts (node-node
relationship). A triplet constitutes a typical form of knowledge
representation within this framework. Entities, serving as foundational
elements of the Knowledge Graph, encompass a wide range of classifications,
such as collections, categories, object types, and thing categories (e.g.,
domain, host, etc.). Relationships interconnect these entities to formulate
the graph’s structure, while attributes encapsulate features and parameters,
exemplified by entities like google.com, windows, and similar.

To construct a dataset, studying the relationships existing between functions
through their callee or caller interactions is necessary. A database is
generated considering the node entities $E_{i}$, their relations $R_{i}$, and
their relation $S_{i}$, which can be a call or a caller. This approach enables
us to generate a knowledge graph containing the identified entities.

The data structure is providing critical information, encompassing the
function’s path, its name, and whether it has been modified in the latest
update. Additionally, it indicates whether the function is vulnerable and
specifies its role as either a callee or a caller.

###  3.4 Building the Dataset

To build a comprehensive dataset, we compare subsequent versions of code where
upgraded package versions are intended to remediate vulnerabilities in the
open source packages [1]. We first use the open source software Syft to
generate the software bill of materials (SBOM) information for the current
version of the target application source code. Curl is used to fetch
vulnerability data for individual package versions in the SBOM from the Open
Source Vulnerabilities (OSV) database, which offers an accessible query
interface for all known dependency vulnerabilities. This information is mapped
to the SBOM to identify the existing vulnerabilities in the dependencies. If a
package with an existing vulnerability is detected, we search for the updated
version of the package and clone it. Using the CodeQL, a semantic code
analysis engine, we model the different versions of the code and construct a
control flow graph of the execution paths for the repository using the
vulnerable package and the fixed package. Tree-sitter is applied to perform
incremental analysis, construct and maintain a syntax tree, and build the
dataset. The impact of a single package upgrade can be measured by comparing
the control flow graphs of the application before and after the upgrade. The
updated version control flow graph is used as the base for the knowledge
graph, and changed or affected nodes are marked in the knowledge graph, along
with nodes that cause errors in compilation of the code. Multiple upgrades are
performed split into two sets for each each repository, one set consisting of
graphs introducing errors that result in code breakage, and another set of
graphs not impacting code functionality.

In this research, a preliminary approach is being used, initially focusing on
three application source repositories; The first case in a code base
comprising 9621 features with 27 vulnerabilities in its broken update; the
second case with 19,569 functions and 3 vulnerabilities. Subsequently, in
Cases 3 (15908 functions and 6 vulnerabilities) and 4 (16095 functions and no
vulnerabilities), an application is observed with a similar order of magnitude
but different dependencies regarding the base functions.

##  4 Analysis

In this section, we discuss the methods of data analysis and describe the
metrics we will use. Once the graph structure has been built, we can utilize
tools for small-scale analysis like NetworkX in Python [20] or for large-scale
projects: Arachne [21]. In this case, NetworkX is used exclusively to
facilitate calculation and analysis, given the volume of data (less than 20000
nodes). This analysis adopts a two-pronged approach. Firstly, it involves an
exploratory examination of graph connectivity, focusing on the distribution
and other metrics that provide insights into the difference between package
upgrades that break dataflow vs. those that do not and quantitative metrics on
the differences in interconnections within the code. This includes evaluating
the structural design and robustness of the network, ensuring that each
functional node and its connections contribute to the overall integrity and
efficiency of the system. Secondly, we analyze the modified GAT scores to give
us a normalized measure the interaction of vulnerabilities as nodes within the
graph. The model gives us an advanced mechanism to measure the importance of
nodes within a network. This approach allows for a targeted strategy for
classifying and assessing vulnerabilities, simplifying the assessment of
remediation effectiveness. The practical benefits of this approach are many
and include prioritizing development efforts, improving software integrity,
formulating strategic planning initiatives, and deepening knowledge about
software architecture. This comprehensive strategy not only improves the
immediate security posture of software systems, but also lays the foundation
for long-term sustainable software development and maintenance practices.

###  4.1 Graph analysis

It is crucial when analyzing the knowledge graph created with the data set to
measure the network and understand the meaning and relevance of the entities
and relationships within it [5]. It is necessary to quantify the connectivity,
the paths and their strength through the centrality analysis of each objective
element.

### Preliminar Graph Analysis

We are constructing a knowledge graph, denoted as
$G=(\hat{E},\hat{R},\hat{S})$, where $\hat{E}=V$ signifies the entities or
nodes of the graph. Our primary focus is on the callee-caller relationship
between these entities, which we are considering as a directional edge. This
approach enables us to simplify the triplet with the tuple
$(\hat{R},\hat{S})=E$ as the set of of edges $E$ into the graph structure
$G=(V,E)$ [5]. This representation allows us to prepare a knowledge graph in
relation to the GAT model [29].

This knowledge graph model is characterized by a dynamic representation of
functions, where the total number of vertices is indicated by $n$, and the
edges, denoted by $m$, are distinctly directed. Each edge $e\in E$ in this
graph is assigned a specific weight, denoted by $w(e)$, which is calculated to
reflect the significance and connectivity strength between various functions.
We are defining paths within this graph as sequences of edges, represented by
$\langle u_{i},u_{i+1}\rangle$, where $u_{0}=s$ and $u_{l}=t$ demarcate the
start and end vertices of the path, respectively. The aggregate length of
these paths is the peak of the weights of the constituent edges. Of particular
note is the definition of the distance between two vertices, $s$ and $t$,
represented as $d(s,t)$, with the shortest paths between them signified by
$\sigma_{st}$. Additionally, we are quantifying the number of these shortest
paths that traverse through a specific vertex $v$ using $\sigma_{st}(v)$, ,
being consistent with the notation of Bader et al [5].

Degree centrality: We are measuring the degree centrality of a vertex $v$,
denoted as $\text{deg}(v)$, to quantify the extent of interactions a node has
within its neighborhood. This metric reflects the node’s importance based on
the number of caller-callee connections it maintains [5].

The Norm:  By definition of the Euclidean norm of a vector used by velivckovic
et al [29] in GATs model, $\mathbf{x}\in\mathbb{R}^{n}$ as:
$\|\mathbf{x}\|_{2}=\sqrt{\sum_{i=1}^{n}x_{i}^{2}}$ where
$\mathbf{x}=(x_{1},x_{2},\ldots,x_{n})$ represents a vector in an
$n$-dimensional real space, and $x_{i}$ corresponds to the $i$-th element of
the vector.

Closeness centrality: is measuring the degree of proximity of a node to all
other nodes in the graph, rooted in the concept of distance. For any given
node $n$ its closeness centrality is being calculated as the average length of
the shortest path from $n$ to every other node in the graph. A node with a
higher closeness centrality is more centrally located in the network, thus
signifying its greater importance or influence within the network’s structure.
This metric is being determined by the inverse of the sum of the shortest
distances from the node in question to all other nodes [5, 18].
$CC(v)=1/\sum_{u\in V}d(v,u)$

In the context of our implementation, this is translating to an assessment of
how interconnected a specific function is in relation to the rest of the
functions within the code.

Betweenness centrality: is quantifying how frequently a node appears on the
shortest paths between other nodes, thereby acting as a critical bridge within
the network [5]. In the context of functions and vulnerabilities within
software systems, a function with high betweenness centrality is potentially
pivotal in the flow of information or processes. Such a function might have
the capacity to significantly influence or impact a multitude of other
functions. The pairwise dependency $\delta_{st}(v)$, representing the fraction
of shortest paths between nodes $s$ and $t$ that pass through node $v$, is
defined as follows: $\delta_{st}(v)=\frac{\sigma_{st}(v)}{\sigma_{st}}$

This leads to the formulation of betweenness centrality for a node $v$, where
$s,v,t\in V$, as:$BC(v)=\sum_{s\neq v\neq t}\delta_{st}(v)$

Betweenness centrality is a key metric in network analysis, reflecting a
node’s ability to control the flow of information or resources by acting as a
bridge on the shortest paths between nodes. Nodes with high betweenness
centrality are essential for network connectivity, highlighting their role in
facilitating communication and interactions. These strategically located nodes
ensure efficient dissemination of information by being part of numerous
shortest paths connecting various pairs of nodes within the network.

Connected components: are being defined as all subgraphs within which any two
vertices are connected to each other by paths, while simultaneously not being
connected to any additional vertices in the supergraph. [10] Our research
delineates a connected component in a graph as a set of nodes where each node
can access all other nodes within the same set, without external connections.
This notion is crucial for analyzing network structures, detecting isolated
clusters, and examining connectivity. The study further distinguishes between
undirected graphs, where a connected component comprises the largest set of
nodes interconnected by paths, and directed graphs, which introduce strongly
connected components defined by bidirectional paths between all pairs of
nodes.

Our approach is focusing on leveraging the concept of connected components to
gain insights into the network’s architecture. This analysis is pivotal in
identifying potential vulnerabilities or areas of improvement within the
network, especially in the context of cybersecurity and software engineering.
By understanding the formation and interaction of these components, we are
able to devise more effective strategies for network optimization and
vulnerability mitigation.

The clustering coefficient: for a node $v$, denoted as $C_{i}$, is currently
being assessed to determine the likelihood of connectivity between two
randomly chosen neighbors of this node. This measure is indicative of the
number of triangles in which the $i$-th node participates, normalized by the
maximum possible number of such triangles. We are computing the overall
average clustering coefficient by averaging these individual values across all
nodes in the graph. $C_{i}=2t_{i}/k_{i}(k_{i}-1)$As this average clustering
coefficient approaches 1, it is suggesting an increasing completeness of the
graph, characterized by a predominant, cohesive component. A higher
coefficient is typically indicative of triadic closure, as observed in denser
graphs where triangular formations are more prevalent [22].

The average clustering for a graph is now being calculated as the average of
the local clustering coefficients of all nodes:
$\bar{C}_{i}=\frac{1}{n}\sum_{i=1}^{n}C_{i}$ Therefore, the average clustering
coefficient is providing an overall indication of the degree of clustering
within the entire network. This metric reflects how closely nodes within a
graph tend to cluster together, offering insights into the network’s
structural density and connectivity patterns.

We are defining the quantity of triangles $t_{i}$ around each node $i$, where
$C_{i}=0$ indicates that the neighbors of a vertex are not connected, and
$C_{i}=1$ signifies that neighbors are fully connected. This approach allows
us to quantify the degree of local node interconnectivity within the graph,
contributing to our understanding of the network’s overall structure and
cohesiveness.

Degree Assortativity Coefficient: We are currently measuring the degree
assortativity coefficient, which quantifies the tendency of nodes in a network
to connect with other nodes that possess a similar degree. This metric is
providing insights into the network’s tendency towards assortative or
disassortative mixing [17]. Specifically, it determines whether nodes with a
high number of connections (high degree) are more likely to connect with other
highly connected nodes, or with nodes having fewer connections. The
coefficient varies from $-1$ to $1$, where values close to $1$ indicate
assortative mixing, implying high-degree nodes predominantly connect with
other high-degree nodes. Conversely, values close to $-1$ signify a
disassortative mixture, where high-degree nodes are more likely to connect
with low-degree nodes. A value around 0 indicates no particular preference in
the network’s connectivity pattern.

The mathematical definition of the Degree Assortativity Coefficient $r$ is
formulated as: $r={\sum_{jk}jk(e_{jk}-q_{j}q_{k})}/{\sigma_{q}^{2}}.$ In this
formula, $e_{jk}$ represents the fraction of edges in the network that connect
a node of degree $j$ to a node of degree $k$. The term $q_{j}$ denotes the
distribution of the remaining degrees of the nodes at the end of an edge, when
one end is attached to a node of degree $j$. Finally, $\sigma_{q}^{2}$ is the
variance of the distribution $q$. This coefficient is playing a crucial role
in understanding the structural tendencies of the network, providing valuable
insights into how nodes within the network preferentially form connections
based on their degrees.

Cyclomatic complexity is currently serving as a quantitative measure for
evaluating the number of linearly independent paths within a code, thus
providing an estimate of the program’s complexity [6]. This metric is
instrumental in understanding the intricacy and structural complexity of a
program [23]. Adapting McCabe’s original definition [16], in graph theory
terms, the cyclomatic complexity $V(G)$ of a control flow graph $G$ is being
defined as follows: $V(G)=E-N+2P$ In this equation, $E$ represents the number
of edges in the graph, $N$ signifies the number of nodes, and $P$ stands for
the number of connected components within the graph. This complexity measure
is providing a crucial insight into the potential paths and decision points in
a program’s structure. By quantifying the complexity, we are gaining a deeper
understanding of the software’s maintainability and potential areas for
refactoring. Cyclomatic complexity, therefore, is not just a theoretical
construct but a practical tool for guiding the development and maintenance of
robust and efficient software systems.

##  5 Dataset Results and Interpretation

This precursory study analyzes three distinct applications from the Python and
Java Source languages leveraging open source packages with control flow graphs
of: the respective base application, the broken application post-upgrade, and
the non-broken application post-upgrade versions, as applicable. For each
case, a specific number of functions have been marked as critical, in the
CodeQL analysis has marked the specific changes to these functions as the
reasons for causing the application to no longer compile. The objective is to
gain insights into how the structural and functional attributes of the
functions in the graph affect the likelihood of a code breakage, especially in
response to modifications and the introduction of new features or patches
intended to fix vulnerabilities.

CASE 1: (Table 1) We analyzed an application code base comprising 9626
functions, among which 27 functions were identified as critical, meaning the
remediation upgrade caused changes in these functions that would cause the
code to break. Upon examination of the resolved code, an increase in the
number of nodes and a rise in the average degree were observed following the
update. The density of the network remained constant, which aligns with
expectations. There was a increasing in the number of connected components,
and the degree assortativity coefficient stayed negative and constant.
Comparing both versions of the upgrades, the degree assortativity coefficient
for the Non Broken Upgrade was much lower than the metric from the Broken
Upgrade, while the closeness centrality was higher for Non Broken Upgrades.

Base Non Broken Upgrade Broken Upgrade Number of unique functions 9621 9621
9621 nodes kind Unchanged Changed Unchanged Changed Number of nodes 9621 9614
19 9613 23 Number of edges 15186 15178 19 15176 24 Average degree 3.1568
3.1575 2.0 3.1574 2.087 Density 0.0003281 0.0003285 0.1111 0.0003285 0.09486
Num. of connected components 327 327 2 327 2 Average clustering 0.06809
0.06815 0.0 0.06815 0.0 Degree assortativity coefficient -0.08515 -0.08519
-0.3996 -0.08521 -0.3464 Avg. betweenness centrality 0.00036 0.000361 1.63973
0.00036 2.13699 Avg. closeness centrality 0.15857 0.15864 0.11057 0.15871
0.0909 Cyclomatic Complexity 6219 6218 4 6217 5

Table 1: (Case1) Comparative analysis of code metrics from code base, non-
broken update and broken upgrade.

CASE 2: (Table 2)We analyzed an application code base comprising 19,569
functions with 37,615 caller-recipient interactions, among which 3 functions
were identified as critical in their failed update, causing the code to break.
Upon examination of the metrics, the number of functions affected for both
upgrades were significantly higher, however the number of functions remained
the same. There was a decrease in the cyclomatic complexity after both
upgrades, while the number of connected components rose. Comparing both
versions of the upgrade the degree assortativity coefficient for the Non
Broken Upgrade was again lower than the metric from the Broken Upgrade, along
with the density of the changed nodes, while the closeness centrality was
again higher for Non Broken Upgrades.

Base Non Broken Upgrade Broken Upgrade Number of unique functions 19569 19569
19569 nodes kind Unchanged Changed Unchanged Changed Number of nodes 19569
17635 2629 17255 3338 Number of edges 37615 33180 4586 32234 5850 Average
degree 3.8443 3.7630 3.4888 3.7362 3.5051 Density 0.00019646 0.00021339
0.00132754 0.00021654 0.00105037 Number of connected components 440 426 80 429
97 Average clustering 0.05552 0.05505 0.04415 0.05467 0.04482 Degree
assortativity coefficient -0.07772 -0.08047 -0.12601 -0.07853 -0.12377 Avg.
betweenness centrality 0.00018 0.00019 0.000124 0.00019 0.00012 Avg. closeness
centrality 0.17790 0.17686 0.18617 0.18716 0.17640 Cyclomatic Complexity 18926
16397 2117 15837 2706

Table 2: (CASE 2) Comparative analysis of code metrics from code base, non-
broken upgrade and broken upgrade.

CASE 3: (Table 3) We analyzed an application code base comprising 15908
functions with 29449 caller-recipient interactions, among which 6 functions of
the upgrade were critical. Upon examination of the metrics, there was a
decrease in the cyclomatic complexity after both upgrades, while the number of
connected components again rose. Comparing both versions of the upgrade the
Degree assortativity coefficient for the Non Broken Upgrade is this time
higher than the metric from the Broken Upgrade, however the density remains
lower in the changed nodes, while the closeness centrality was higher for Non
Broken Upgrades.

Base Non Broken Upgrade Broken Upgrade Number of unique functions 15908 15908
15908 nodes kind Unchanged Changed Unchanged Changed Number of nodes 15908
15553 603 15127 1015 Number of edges 29449 28616 912 27695 1816 Average degree
3.7024 3.6798 3.0249 3.6617 3.5783 Density 0.00023275 0.00023661 0.00502471
0.00024208 0.00352892 Number of connected components 337 341 19 335 28 Average
clustering 0.05004 0.04834 0.06402 0.04688 0.05890 Degree assortativity
coefficient -0.10958 -0.11123 -0.10286 -0.11203 -0.13705 Avg. betweenness
centrality 0.00022 0.00022 0.00018 0.00022 0.00022 Avg. closeness centrality
0.18028 0.18032 0.17876 0.18049 0.17684 Cyclomatic Complexity 14215 13745 347
13238 857

Table 3: (CASE 3) Comparative analysis of code metrics from code base, non-
broken update and broken upgrade

Statistical Differences between Base and Subgraphs: Given the consistent
direction of difference between the closeness centrality for the three cases
above, we decided to further explore those differences. The above analysis
that conducting a package upgrade generates two subgraphs of the control flow
graph, the functions affected by the upgrade and the functions not affected by
the upgrade. We posit that the Broken package upgrade subgraphs are
statistically different from a randomly drawn subgraph, and tested this using
the closeness centrality value for each node. We used a T-test an the
Kolmogorov-Smirnov (K-S) tests to evaluate differences in means and the
distributions, respectively. For Broken Case 1, we observe a T-statistic of
-4.881 with a p-value of 0.0012 and a K-S statistic of 0.6916 and a p-value of
8.16e-05, indicating a pronounced deviation in closeness centrality between
changed and all nodes. In contrast, Broken Case 2 has a a T-statistic of 8.399
with a p-value of 6.47e-17 and a lower K-S statistic of 0.1161 with a p-value
of 1.38e-27,suggesting less variation but still significant differences.
Broken Case 3 further corroborates the presence of significant differences
with a K-S statistic of 0.2176 and a p-value of 9.81e-34, complemented by a
T-statistic of -2.349 and a p-value of 0.019. In summary, the mean values for
closeness centrality and the distribution of the closeness centrality values
for the nodes appear to be statistically different between subgraphs of
changes resulting from upgrades that break functionality and the base
repositories.

The results from the three cases analyzed here exhibit a trend where
cyclomatic complexity tends to keep or decrease as upgrades are made to
package. This observation suggests a relationship between the resolution of
vulnerabilities and the simplification of code structure that may justify
further exploration. This ongoing analysis is crucial for understanding the
dynamic nature of package vulnerabilities and their impact on overall code
complexity.

Modified GAT Results The next step of the analysis attempts to analyze the
interconnectedness of the critical functions causing the package upgrades to
fail. When we apply the modified GAT model Fig. 2, the scores obtained. It is
necessary to see each case as a specific case. For this, an average GAT score
was obtained for each case. Having a high GAT score above average indicates
that the vulnerabilities are more critical and necessary since the network
depends more on them.

| Case 1 | Case 2 | Case 3
---|---|---|---
NC | 27 | 3 | 6
MSC | 0.5295 | 0.3662 | 0.5045
mSC | 0.2461 | 0.1254 | 0.1260
ASC | 0.4359 | 0.2185 | 0.3228
AGS | 0.4287 | 0.3785 | 0.3153

Figure 2: NC: number of Critical Function, MSFC:Max Score Critical, mSFC:Min
Score Critical, ASC:Avr Score Critical, and AG:Avr Gat Score for all nodes(Red
line)

Principal Component Analysis (PCA) and t-Distributed Stochastic Neighbor
Embedding (t-SNE) [29] are dimensionality reduction techniques applied
alongside GATs to enhance visualization and interpretability as we do in the
Case 2 Fig.5. While PCA projects data into a lower-dimensional space
preserving variance, t-SNE focuses on maintaining local relationships, making
it particularly useful for visualizing high-dimensional data generated by GATs
in a way that highlights patterns and relationships within the graph
structure.

The enhanced GAT Score significantly augments our ability to discern the
connectivity and importance of critical functions within a system’s context.
By employing a normalized approach, where scores closer to 1 denote higher
importance, we gain nuanced insights into the integral nature of specific
vulnerabilities. This is exemplified in the data for CASE 1, with an average
GAT score for the entire graph of 0.4287, and most of the 27 critical
functions are above this value (see Fig. 2). In CASE 2, the GAT scores for all
critical functions are less than the average, with a maximum GAT score for a
critical function of 0.3662, while the average GAT Score for the entire graph
is 0.3785. This implies that the criticality of these functions is lower. It’s
noteworthy that these scores are not solely reliant on the degree of
connectivity; rather, they integrate a weighted combination of degree, norm,
and centrality metrics. This holistic approach allows us to identify functions
that, despite having a lower degree of connectivity, hold substantial
significance within the network’s overall architecture. Such revelations
underscore the complexity of network dynamics and the crucial role of advanced
analytical tools in unveiling the intricate interplay of functions within a
software system.

Figure 3: Closeness Centrality Histogram, y-axes counts and x-axes Closeness
Centrality. Cases 1, 2, 3, and 4 with their respective base code, broken
upgrade, and non-broken upgrade as applicable. Light blue indicates the
centrality of all nodes, light red represents unchanged nodes, and green
denotes changed nodes.

Figure 4:  Normalized cases of clustering coefficient histogram. Blue: All
nodes, Red: Changed nodes, Green: Non changed nodes.

Figure 5: Visualization of communities for Case 2, using t-SNE and PCA applied
to a modified Graph Attention Network (GAT) as usually worked in GAT data
analysis,The red dot is the representation of the vulnerability in their
respective spaces. The third figure is a graphical representation of the
communities within the graph, each distinguished by a unique color.

##  6 Discussion

The results of our extensive analysis of several code bases reveal an
intricate dynamic between package vulnerabilities and the complexity of the
code. This study contributes to the current discourse in package development
by highlighting the nuanced relationship between the vulnerabilities after the
publication and evolution of the code.This requires not only immediate code
updates, but also a deeper understanding of the underlying dynamics of these
vulnerabilities.

Our analysis of CASES 1, 2, and 3 has led to several key observations about
the impact of vulnerability resolution on code structure. We noticed a trend
of decreased or consistent total cyclomatic complexity when comparing base
cases with broken and intact vulnerability resolutions. This pattern indicates
that effectively addressing vulnerabilities tends to simplify the code
structure, aligning with best practices for developing maintainable and less
error-prone packages. However, managing the complexity associated with package
vulnerabilities remains a formidable challenge. In particular, we observed an
increase in the number of connected components from the base case to the non-
broken update, suggesting that remediation efforts not only resolve
vulnerabilities but also improve the connectivity and robustness of the code
architecture. Despite this, even minor vulnerabilities can have far-reaching
effects across the entire application, underscoring the need for a
comprehensive approach to understanding and mitigating the impacts of package
updates. The intricate nature of this challenge is further highlighted in
cross-centrality graphs (3, 4), where significant variations in centrality
between nodes highlight the complex interdependencies within the code. These
findings emphasize the critical importance of detailed graph-based analyzes
for navigating the complexities of software vulnerability management and
improvement.

In addition, our analysis sheds light on the connectivity patterns of the
function within the code bases. The coefficient of negative degree supply
consistently observed in multiple studies indicates that the code functions
tend to connect with a diverse variety of other functions, instead of
predominantly linked to similar functionalities. This diversity in
connectivity patterns has deep implications to understand how vulnerabilities
could spread through a code base and affect their general integrity.

This ambiguity highlights a critical gap in current understanding and requires
more research. It emphasizes the importance of a strategic approach to code
updates, where essential functionalities such as configuration are refined and
maintained, instead of being completely eliminated.

The significance of the modified GAT score is further underscored by our
observations of the wide dispersion of nodes and low density within the graphs
analyzed. This dispersion necessitates a nuanced approach to understanding the
role of each node within the network, focusing not only on the number of
connections it can generate but also on the number of paths that traverse
through it. The modified GAT model offers a normalized view of how
interconnected a function is in relation to its degree, norm, and closeness
centrality metrics. This comprehensive perspective is crucial for effective
vulnerability management and software maintenance, as it highlights the
critical nodes that warrant prioritized attention. By integrating the insights
from the GAT score with observations of node propagation and density,
stakeholders are equipped with a powerful tool to identify and address the
most significant vulnerabilities, optimizing resource allocation and enhancing
the security and reliability of software systems. This blend of quantitative
measures and attention-based analysis signifies a significant advancement in
our ability to strategically mitigate vulnerabilities and strengthen software
infrastructure.

Our study underscores the dynamic nature of package vulnerabilities and their
impact on code complexity. It calls for continued vigilance and strategic
intervention in the package development lifecycle, stressing the need for
robust monitoring and adaptive strategies to manage the evolving landscape of
package vulnerabilities.

##  7 Conclusion

This paper embarks on a comprehensive exploration, leveraging the intricate
capabilities of knowledge graphs to delve into the dynamics of opensource
package function networks. Central to our investigation is the identification
and analysis of vulnerable functions, where we scrutinize their interactions
and assess the impact of their mitigation on the codebase. Our research
reveals a notable insight: targeted remediation of specific vulnerabilities
tends to preserve the overall network’s connectivity, underscoring the package
structure’s inherent resilience.

During our analysis, we noted a pattern of decreasing vulnerabilities
following successive package updates, prompting a pivotal inquiry: are these
vulnerabilities conclusively resolved, or do they subtly embed themselves into
subsequent features? This ambiguity signals a compelling need for further work
into the lifecycle of vulnerabilities within package development, promising to
enrich the discourse in this field significantly.

At the heart of our methodology is the odified Graph Attention Network (GAT),
especially its attention mechanism. Through the integration of node-centric
metrics—such as degree centrality, norm, and closeness centrality—our approach
refines the network’s ability to discern detailed aspects of the graph’s
architecture and the nuances of node characteristics. This methodological
advancement facilitates a nuanced portrayal of the network, yielding a
comprehensive understanding of node interrelations and their significance.

Furthermore, our investigation brings to light the existence of latent
vulnerabilities within the most critical segments of the code, initially
perceived as flawless. These covert vulnerabilities represent significant
security risks, with the potential to compromise vital components, including
databases and core functionalities. To address these issues, we advocate for
an in-depth and ongoing code analysis from its inception. Utilizing knowledge
graphs as both historical and dynamic monitoring tools enables proactive
surveillance of vulnerabilities.

For future works, our research direction will focus on methodological
improvements, particularly in dissecting the interconnectivity between
functions. The number of case analyzed was small with significant complexity,
necessitating additional application repositories for comprehensive analysis
to draw more robust conclusions. By evaluating various aspects such as
variable types, execution times, and functional dependencies, we aim to
unravel the importance of specific functions within the network. This
comprehensive strategy is designed to offer deeper insights into the
structural integrity and vulnerabilities of software systems, thereby making a
substantial contribution to enhancing package security and dependability. Our
endeavors are geared towards the development of robust software systems
capable of navigating the complexities of contemporary cyber threats, marking
a significant stride forward in the realm of package vulnerability analysis
and cybersecurity.

## Acknowledgment

This research was supported in part by NSF grant CCF-2109988.

## References

  * [1] Manar Alanazi, Abdun Mahmood, and Mohammad Jabed Morshed Chowdhury.  Scada vulnerabilities and attacks: A review of the state-of-the-art and open issues.  Computers & Security, page 103028, 2022.
  * [2] Mahmoud Alfadel, Diego Elias Costa, and Emad Shihab.  Empirical analysis of security vulnerabilities in python packages.  Empirical Software Engineering, 28(3):59, 2023.
  * [3] Raghavendra Rao Althar, Debabrata Samanta, Manjit Kaur, Dilbag Singh, and Heung-No Lee.  Automated risk management based software security vulnerabilities management.  IEEE Access, 10:90597–90608, 2022.
  * [4] Ashwin Arulselvan, Clayton W Commander, Lily Elefteriadou, and Panos M Pardalos.  Detecting critical nodes in sparse graphs.  Computers & Operations Research, 36(7):2193–2200, 2009.
  * [5] David A. Bader and Kamesh Madduri.  Parallel algorithms for evaluating centrality indices in real-world networks.  In 2006 International Conference on Parallel Processing (ICPP’06), pages 539–550, 2006.
  * [6] Christof Ebert, James Cain, Giuliano Antoniol, Steve Counsell, and Phillip Laplante.  Cyclomatic complexity.  IEEE software, 33(6):27–29, 2016.
  * [7] Jiahao Fan, Yi Li, Shaohua Wang, and Tien N. Nguyen.  A c/c++ code vulnerability dataset with code changes and cve summaries.  In Proceedings of the 17th International Conference on Mining Software Repositories, MSR ’20, page 508–512, New York, NY, USA, 2020. Association for Computing Machinery.
  * [8] Görkem Giray.  A software engineering perspective on engineering machine learning systems: State of the art and challenges.  Journal of Systems and Software, 180:111031, 2021.
  * [9] Katerina Goseva-Popstojanova and Andrei Perhinschi.  On the capability of static code analysis to detect security vulnerabilities.  Information and Software Technology, 68:18–33, 2015.
  * [10] Lifeng He, Xiwei Ren, Qihang Gao, Xiao Zhao, Bin Yao, and Yuyan Chao.  The connected-component labeling problem: A review of state-of-the-art algorithms.  Pattern Recognition, 70:25–43, 2017.
  * [11] Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia D’amato, Gerard De Melo, Claudio Gutierrez, Sabrina Kirrane, José Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, Axel-Cyrille Ngonga Ngomo, Axel Polleres, Sabbir M. Rashid, Anisa Rula, Lukas Schmelzeisen, Juan Sequeda, Steffen Staab, and Antoine Zimmermann.  Knowledge graphs.  ACM Computing Surveys, 54(4):1–37, jul 2021.
  * [12] Nasif Imtiaz, Seaver Thorn, and Laurie Williams.  A comparative study of vulnerability reporting by software composition analysis tools.  In Proceedings of the 15th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pages 1–11, 2021.
  * [13] Yan Jia, Yulu Qi, Huaijun Shang, Rong Jiang, and Aiping Li.  A practical approach to constructing a knowledge graph for cybersecurity.  Engineering, 4(1):53–60, 2018.
  * [14] Kai Liu, Fei Wang, Zhaoyun Ding, Sheng Liang, Zhengfei Yu, and Yun Zhou.  Recent progress of using knowledge graph for cybersecurity.  Electronics, 11(15):2287, 2022.
  * [15] Kai Liu, Fei Wang, Zhaoyun Ding, Sheng Liang, Zhengfei Yu, and Yun Zhou.  A review of knowledge graph application scenarios in cyber security.  arXiv preprint arXiv:2204.04769, 2022.
  * [16] Thomas J McCabe.  A complexity measure.  IEEE Transactions on software Engineering, (4):308–320, 1976.
  * [17] Mark EJ Newman.  Mixing patterns in networks.  Physical review E, 67(2):026126, 2003.
  * [18] UJ Nieminen.  On the centrality in a directed graph.  Social science research, 2(4):371–378, 1973.
  * [19] Namyong Park, Andrey Kan, Xin Luna Dong, Tong Zhao, and Christos Faloutsos.  Estimating node importance in knowledge graphs using graph neural networks.  In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 596–606, 2019.
  * [20] Edward L Platt.  Network science with Python and NetworkX quick start guide: explore and visualize network data effectively.  Packt Publishing Ltd, 2019.
  * [21] Oliver Alvarado Rodriguez, Zhihui Du, Joseph Patchett, Fuhuan Li, and David A Bader.  Arachne: An arkouda package for large-scale graph analytics.  In 2022 IEEE High Performance Extreme Computing Conference (HPEC), pages 1–7. IEEE, 2022.
  * [22] Jari Saramäki, Mikko Kivelä, Jukka-Pekka Onnela, Kimmo Kaski, and Janos Kertesz.  Generalizations of the clustering coefficient to weighted complex networks.  Physical Review E, 75(2):027105, 2007.
  * [23] Mir Muhammd Suleman Sarwar, Sara Shahzad, and Ibrar Ahmad.  Cyclomatic complexity: The nesting problem.  In Eighth International Conference on Digital Information Management (ICDIM 2013), pages 274–279. IEEE, 2013.
  * [24] Robert Seacord.  Secure coding to prevent vulnerabilities.  Carnegie Mellon University, Software Engineering Institute’s Insights (blog), May 2014.  Accessed: 2023-Nov-16.
  * [25] Leslie F Sikos.  Cybersecurity knowledge graphs.  Knowledge and Information Systems, pages 1–21, 2023.
  * [26] Sherri Sparks, Shawn Embleton, Ryan Cunningham, and Cliff Zou.  Automated vulnerability analysis: Leveraging control flow for evolutionary input crafting.  In Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007), pages 477–486. IEEE, 2007.
  * [27] Tamás Szabó.  Incrementalizing production codeql analyses.  arXiv preprint arXiv:2308.09660, 2023.
  * [28] Ángel Jesús Varela-Vaca, Diana Borrego, María Teresa Gómez-López, Rafael M Gasca, and A German Márquez.  Feature models to boost the vulnerability management process.  Journal of Systems and Software, 195:111541, 2023.
  * [29] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.  Graph attention networks.  arXiv preprint arXiv:1710.10903, 2017.
  * [30] Boming Xia, Tingting Bi, Zhenchang Xing, Qinghua Lu, and Liming Zhu.  An empirical study on software bill of materials: Where we stand and the road ahead.  arXiv preprint arXiv:2301.05362, 2023.
  * [31] Feng Xia, Xin Chen, Shuo Yu, Mingliang Hou, Mujie Liu, and Linlin You.  Coupled attention networks for multivariate time series anomaly detection.  IEEE Transactions on Emerging Topics in Computing, 2023.
  * [32] Zhihao Yan and Jingju Liu.  A review on application of knowledge graph in cybersecurity.  In 2020 International Signal Processing, Communications and Engineering Management Conference (ISPCEM), pages 240–243. IEEE, 2020.
  * [33] Kailong Zhu, Yuliang Lu, Hui Huang, Lu Yu, and Jiazhen Zhao.  Constructing more complete control flow graphs utilizing directed gray-box fuzzing.  Applied Sciences, 11(3):1351, 2021.
  * [34] Afra Zomorodian.  Computational topology.  Algorithms and theory of computation handbook, 2(3), 2009.


# Representation Learning For Speech Recognition Using Feedback Based
Relevance Weighting

###### Abstract

In this work, we propose an acoustic embedding based approach for
representation learning in speech recognition. The proposed approach involves
two stages comprising of acoustic filterbank learning from raw waveform,
followed by modulation filterbank learning. In each stage, a relevance
weighting operation is employed that acts as a feature selection module. In
particular, the relevance weighting network receives embeddings of the model
outputs from the previous time instants as feedback. The proposed relevance
weighting scheme allows the respective feature representations to be
adaptively selected before propagation to the higher layers. The application
of the proposed approach for the task of speech recognition on Aurora-4 and
CHiME-3 datasets gives significant performance improvements over baseline
systems on raw waveform signal as well as those based on mel representations
(average relative improvement of $15$% over the mel baseline on Aurora-4
dataset and $7$% on CHiME-3 dataset).

Index Terms—  Speech representation learning, feedback of acoustic embeddings,
raw speech waveform, 2-stage relevance weighting, speech recognition.

##  1 Introduction

Representation learning deals with the broad set of methods that enable the
learning of meaningful representations from raw data. Similar to machine
learning, representation learning can be carried out in an unsupervised
fashion like principal component analysis (PCA), t-stochastic neighborhood
embeddings (tSNE) proposed by [1] or in supervised fashion like linear
discriminant analysis (LDA). Recently, deep learning based representation
learning has drawn substantial interest. While a lot of success has been
reported for text and image domains (for eg., word2vec embeddings [2]),
representation learning for speech and audio is still challenging.

One of the research directions pursued for speech has been the learning of
filter banks operating directly on the raw waveform [3, 4, 5, 6, 7], mostly in
supervised setting. Other efforts attempting unsupervised learning of
filterbank have also been investigated. The work in [8] used restricted
Boltzmann machine while the efforts in [9] used variational autoencoders. The
wav2vec method recently proposed by [10] explores unsupervised pre-training
for speech recognition by learning representations of raw audio. There has
been some attempts to explore interpretability of acoustic filterbank
recently, for eg. SincNet filterbank by [11] and self-supervised learning by
[12]. However, compared to vector representations of text which have shown to
embed meaningful semantic properties, the interpretability of speech
representations from these approaches has often been limited.

Subsequent to acoustic filterbank processing, modulation filtering is the
process of filtering the 2-D spectrogram-like representation using 2-D filters
along the time (rate filtering) and frequency (scale filtering) dimension.
Several attempts have been made to learn the modulation filters also from
data. The earliest approaches using LDA explored the learning of the temporal
modulation filters in a supervised manner [13, 14]. Using deep learning, there
have been recent attempts to learn modulation filters in an unsupervised
manner [15, 16].

In this paper, we extend our previous work [17] on joint acoustic and
modulation filter learning in the first two layers of a convolutional neural
network (CNN) operating on raw speech waveform. The novel contribution of our
approach is the incorporation of acoustic embeddings as feedback in the
relevance weighting approach. In particular, the relevance weighting network
is driven by the acoustic/modulation filter outputs along with the embedding
of the previous one-hot targets. The output of the relevance network is a
relevance weight which multiplies the acoustic/modulation filter [17]. The
rest of the architecture performs the task of acoustic modeling for automatic
speech recognition (ASR). The approach of feeding the model outputs back to
the neural network is also previously reported as a form of recurrent neural
network (RNN) called the teacher forcing network [18]. However, in this work,
the embeddings of the model outputs are fed back only to the relevance
weighting network and not as a RNN architecture.

The ASR experiments are conducted on Aurora-4 (additive noise with channel
artifact) dataset [19], CHiME-3 (additive noise with reverberation) dataset
[20] and VOiCES (additive noise with reverberation) dataset [21]. The
experiments show that the learned representations from the proposed framework
provide considerable improvements in ASR results over the baseline methods.

##  2 Relevance Based Representation Learning

The block schematic of the senone embedding network is shown in Figure 1. The
entire acoustic model using the proposed relevance weighting model is shown in
Figure 3.

###  2.1 Step-0: Embedding network pre-training

The embedding network (Figure 1) is similar to the skip-gram network of
word2vec models as proposed in [2]. In this work, the one-hot encoded senone
(context dependent triphone hidden Markov model (HMM) states modeled in ASR )
target vector at frame $t$, denoted as $\boldsymbol{h}_{t}$, is fed to a
network whose first layer outputs the embedding denoted as
$\boldsymbol{e}_{t}$. This embedding predicts the one-hot target vectors for
the preceding and succeeding time frames $\boldsymbol{h}_{t-1}$ and
$\boldsymbol{h}_{t+1}$. This model is trained using the ASR labels for each
task before the acoustic model training. Once the model is trained, only the
embedding extraction part (first layer outputs) is used in the final ASR
model. We use embeddings of $200$ dimensions. During the ASR testing, the
embeddings are derived by feeding the softmax outputs from the acoustic model
(similar to teacher forcing network by [18]).

For the analysis, the TIMIT test set [22] consisting of $1344$ utterances is
used. The dataset is hand labelled for phonemes. The t-SNE visualization of
the embeddings is shown in Fig. 2 for phonemes from TIMIT test set for a group
of vowel phonemes {/ao/, /aa/, /ae/, /ey/, /uw/} and a group of plosives
{/t/}, fricatives {/sh/, /zh/}, and nasals {/em/, /eng/}. As seen in the t-SNE
plot of embeddings, the embeddings while being trained on one-hot senones,
provides segregation of different phoneme types such as vowels, nasals,
fricatives and plosives.

Fig. 1: Block schematic of senone embedding network used in the proposed
model. Fig. 2: t-SNE plot of the senone embeddings for TIMIT dataset. Fig. 3:
(a) Block diagram of the proposed representation learning approach from raw
waveform, (b) expanded acoustic FB relevance sub-network. Here,
$\boldsymbol{x}_{t}(f)$ denotes the sub-band trajectory of band $f$ for all
frames centered at time $t$, $\boldsymbol{e}_{t-1}$ denotes the acoustic
embedding vector for previous time step, (c) expanded modulation filterbank
relevance sub-network.

###  2.2 Step-1: Acoustic Filterbank representation [23]

The input to the neural network are raw samples windowed into $S$ samples per
frame with a contextual window of $T$ frames. Each block of $S$ samples is
referred to as a frame. This input of size $S\times 1$ raw audio samples are
processed with a 1-D convolution using $F$ kernels ($F$ denotes the number of
sub-bands in filterbank decomposition) each of size $L$. The kernels are
modeled as cosine-modulated Gaussian function [9, 23],

${{g}}_{i}(n)=\cos{2\pi\mu_{i}n}\times\exp{(-{n^{2}}\mu_{i}^{2}/{2})}$ (1)

where ${{g}}_{i}(n)$ is the $i$-th kernel ($i=1,..,F$) at time $n$, $\mu_{i}$
is the center frequency of the $i$th filter (in frequency domain). The mean
parameter ${\mu}_{i}$ is updated in a supervised manner for each dataset. The
convolution with the cosine-modulated Gaussian filters generates $F$ feature
maps which are squared, average pooled within each frame and log transformed.
This generates $\boldsymbol{x}$ as $F$ dimensional features for each of the
$T$ contextual frames, as shown in Figure 3. The $\boldsymbol{x}$ can be
interpreted as the “learned” time-frequency representation (spectrogram).

###  2.3 Acoustic FB relevance weighting

The relevance weighting paradigm for acoustic FB layer is implemented using a
relevance sub-network fed with the $F\times T$ time-frequency representation
$\boldsymbol{x}$ and embeddings $\boldsymbol{e}$ of the previous time step.
Let $\boldsymbol{x}_{t}(f)$ denote the vector containing sub-band trajectory
of band $f$ for all $T$ frames centered at $t$ (shown in Figure 3(b)). Then,
$\boldsymbol{x}_{t}(f)$ is concatenated with embeddings of the previous time
step $\boldsymbol{e}_{t-1}$ with $tanh()$ non-linearity. This is fed to a two
layer deep neural network (DNN) with a sigmoid non-linearity at the output. It
generates a scalar relevance weight $\boldsymbol{w}_{a}(t,f)$ as the relevance
weight corresponding to the input representation at time $t$ for sub-band $f$.
This operation is repeated for all the $F$ sub-bands which gives a $F$
dimensional weight vector $\boldsymbol{w}_{a}(t)$ for the input
$\boldsymbol{x}_{t}$.

The $F$ dimensional weights $\boldsymbol{w}_{a}(t)$ multiply each column of
the “learned” spectrogram representation $\boldsymbol{x}_{t}$ to obtain the
relevance weighted filterbank representation $\boldsymbol{y}_{t}$. The
relevance weights in the proposed framework are different from typical
attention mechanism [24]. In the proposed framework, relevance weighting is
applied on the representation as soft feature selection weights without
performing a linear combination. We also process the first layer outputs
($\boldsymbol{y}$) using instance norm [25, 26].

In our experiments, we use $T=101$ whose center frame is the senone target for
the acoustic model. We also use $F=80$ sub-bands and acoustic filter length
$L=129$. This value of $L$ corresponds to $8$ ms in time for a $16$ kHz
sampled signal. The value of $S$ is $400$ ($25$ ms window length) with frame
shift of $10$ms.

###  2.4 Step-2: Relevance Weighting of Modulation Filtered Representation

The representation $\boldsymbol{z}$ from acoustic filterbank layer is fed to
the second convolutional layer which is interpreted as modulation filtering
layer (shown in Figure 3). The kernels of this convolutional layer are 2-D
spectro-temporal modulation filters, learning the rate-scale characteristics
from the data. The modulation filtering layer generates $K$ parallel streams,
corresponding to $K$ modulation filters $\boldsymbol{w}_{K}$. The modulation
filtered representations $\boldsymbol{p}$ are max-pooled with window of
$3\times 1$, leading to feature maps of size $F^{\prime}\times T^{\prime}$.
These are weighted using a second relevance weighting sub-network (referred to
as the modulation filter relevance sub-network in Figure 3, expanded in Figure
3(c)).

The modulation relevance sub-network is fed with feature map
$\boldsymbol{p}_{k}$; where $k=1,2,...,K$, and embeddings $e$ of the previous
time step. The embedding $\boldsymbol{e}$ is linear transformed and
concatenated with the input feature map. This is fed to a two-layer DNN with
softmax non-linearity at the output. It generates a scalar relevance weight
$w_{m}(k)$ corresponding to the input representation at time $t$ ($t$ as
center frame) for $k$th feature map. The weights $\boldsymbol{w}_{m}$ are
multiplied with the representation $\boldsymbol{p}$ to obtain weighted
representation $\boldsymbol{q}$. The resultant weighted representation
$\boldsymbol{q}$ is fed to the batch normalization layer [27]. We use the
value of $K=40$ in the work. Following the acoustic filterbank layer and the
modulation filtering layer (including the relevance sub-networks), the
acoustic model consists of series of CNN and DNN layers with sigmoid
nonlinearity.

##  3 Experiments and Results

The speech recognition system is trained using PyTorch [28] while the Kaldi
toolkit [29] is used for decoding and language modeling. The models are
discriminatively trained using the training data with cross entropy loss and
Adam optimizer [30]. A hidden Markov model - Gaussian mixture model (HMM-GMM)
system is used to generate the senone alignments for training the CNN-DNN
based model. The ASR results are reported with a tri-gram language model or
using a recurrent neural network language model (RNN-LM).

For each dataset, we compare the ASR performance of the proposed approach of
learning acoustic representation from raw waveform with acoustic FB (A) with
relevance weighting (A-R) and modulation FB (M) with relevance weighting (M-R)
denoted as (A-R,M-R), traditional log mel filterbank energy (MFB) features (80
dimension), power normalized filterbank energy (PFB) features [31], mean
Hilbert envelope (MHE) features [32], and excitation based (EB) features [33].
We also compare performance with the SincNet method proposed in [11]. Note
that the modulation filtering layer (M) is part of the baseline model, and
hence notation M is not explicitly mentioned in the discussion. The neural
network architecture shown in Figure 3 (except for the acoustic filterbank
layer, the acoustic FB relevance sub-network and modulation filter relevance
sub-network) is used for all the baseline features.

Table 1: Word error rate (%) for different configurations of the proposed
model for the ASR task on Aurora-4 dataset.

Features | ASR (WER in %)
---|---
A | B | C | D | Avg.
Baseline Raw Waveform (A,M) | 4.1 | 6.8 | 7.3 | 16.2 | 10.7
Acoustic Relevance
A-R,M [Softmax, no embedding] [17] | 3.6 | 6.4 | 8.1 | 15.1 | 10.0
A-R,M [Sigmoid, no embedding] | 3.4 | 6.4 | 6.7 | 15.5 | 9.9
A-R,M [Sigmoid, with senone embedding] | 3.4 | 6.2 | 6.7 | 14.5 | 9.6
Acoustic Relevance & Mod. Relevance
A-R,M-R [Softmax, no embedding] [17] | 3.6 | 6.1 | 6.0 | 14.8 | 9.6
A-R,M-R [Sigmoid, no embedding] | 3.4 | 6.0 | 6.5 | 14.5 | 9.5
A-R,M-R [Sigmoid, with senone embeddings] | 3.0 | 5.8 | 6.2 | 14.4 | 9.1

Table 2: Word error rate (%) in Aurora-4 database with various feature
extraction schemes with decoding using trigram LM (and RNN-LM in paranthesis).

Cond | MFB | PFB | MHE |  EB | Sinc | MFB-R | S-R,M-R | A-R,M-R
---|---|---|---|---|---|---|---|---
A. Clean with same Mic
Clean | 4.2 | 4.0 | 3.8 | 3.7 | 4.0 | 3.9 | 3.8 | 3.0 (2.9)
B: Noisy with same Mic
Airport | 6.8 | 7.1 | 7.3 | - | 6.9 | 6.7 | 6.2 | 5.7
Babble | 6.6 | 7.4 | 7.4 | - | 6.7 | 6.5 | 6.1 | 5.7
Car | 4.0 | 4.5 | 4.3 | - | 4.0 | 4.1 | 3.9 | 3.6
Rest. | 9.4 | 9.6 | 9.1 | - | 9.4 | 9.6 | 8.4 | 7.0
Street | 8.1 | 8.1 | 7.6 | - | 8.4 | 8.4 | 7.5 | 6.3
Train | 8.4 | 8.6 | 8.6 | - | 8.3 | 8.2 | 7.4 | 6.8
Avg. | 7.2 | 7.5 | 7.4 | 6.0 | 7.3 | 7.2 | 6.6 | 5.8 (5.3)
C: Clean with diff. Mic
Clean | 7.2 | 7.3 | 7.3 | 5.0 | 7.3 | 7.1 | 6.8 | 6.2 (5.9)
D: Noisy with diff. Mic
Airport | 16.3 | 18.0 | 17.6 | - | 16.2 | 16.2 | 13.9 | 14.0
Babble | 16.7 | 18.9 | 18.6 | - | 17.6 | 16.9 | 16.0 | 15.0
Car | 8.6 | 11.2 | 9.6 | - | 9.0 | 8.9 | 7.9 | 8.0
Rest. | 18.8 | 21.0 | 20.1 | - | 19.0 | 18.8 | 19.2 | 18.5
Street | 17.3 | 19.5 | 18.8 | - | 17.3 | 17.8 | 16.6 | 15.8
Train | 17.6 | 18.8 | 18.7 | - | 18.1 | 17.9 | 16.6 | 15.3
Avg. | 15.9 | 17.9 | 17.3 | 15.8 | 16.2 | 16.1 | 15.1 | 14.4 (13.7)
Avg. of all conditions
Avg. | 10.7 | 11.7 | 11.4 | 9.9 | 10.8 | 10.8 | 9.9 | 9.1 (8.7)

###  3.1 Aurora-4 ASR

This database consists of read speech recordings of $5000$ words corpus,
recorded under clean and noisy conditions (street, train, car, babble,
restaurant, and airport) at $10-20$ dB SNR. The training data has $7138$ multi
condition recordings ($84$ speakers) with total $15$ hours of training data.
The validation data has $1206$ recordings for multi condition setup. The test
data has $330$ recordings ($8$ speakers) for each of the $14$ clean and noise
conditions. The test data are classified into group A - clean data, B - noisy
data, C - clean data with channel distortion, and D - noisy data with channel
distortion.

The ASR performance on the Aurora-4 dataset is shown in Table 1 for various
configurations of the proposed approach and Table 2 for different baseline
features. In order to observe the impact of different components of the
proposed model, we tease apart the components and measure the ASR performance
(Table 1). The fifth row (A-R,M-R, softmax with no-embedding) refers to the
previous attempt using the 2-stage filter learning reported in [17]. In this
paper, we explore the variants of the proposed model such as use of softmax
nonlinearity instead of sigmoid in both relevance weighting sub-networks,
sigmoid in both relevance weighting sub-networks, without and with senone
embedding, and the 2-stage approach (both relevance weighting sub-networks).
Among the variants with acoustic relevance weighting alone, the A-R [sigmoid
with senone embeddings] improves over the softmax nonlinearity. With joint
A-R,M-R case, again the sigmoid with senone embeddings provides the best
result.

While comparing with different baseline features in Table 2, it can be
observed that most of the noise robust front-ends do not improve over the
baseline mel filterbank (MFB) performance. The raw waveform acoustic FB
performs similar to MFB baseline features on average while performing better
than the baseline for Cond. A and B. The ASR system with MFB-R features, which
denote the application of the acoustic FB relevance weighting over the fixed
mel filterbank features, also does not yield improvements over the system with
baseline MFB features. We hypothesize that the learning of the relevance
weighting with learnable filters allows more freedom in learning the model
compared to learning with fixed mel filters. The proposed (A-R,M-R)
representation learning (two-stage relevance weighting) provides considerable
improvements in ASR performance over the baseline system with average relative
improvements of $15$% over the baseline MFB features. Furthermore, the
improvements in ASR performance are consistently seen across all the noisy
test conditions and with a sophisticated RNN-LM. In addition, the performance
achieved is also considerably better than the results such as excitation based
features (EB) reported by [33].

For comparison with the SincNet method by [11], our cosine modulated Gaussian
filterbank is replaced with the sinc filterbank as kernels in first
convolutional layer (acoustic FB layer in Fig. 3). The ASR system with sinc FB
(Sinc) is trained jointly without any relevance weighting keeping rest of the
architecture same as shown in Fig. 3. From results, it can be observed that
the parametric sinc FB (without relevance weighting) performs similar to MFB
and also our learned filterbank A. In addition, the relevance weighting with
Sinc filterbank (S-R,M-R) results show that the relevance weighting is also
applicable to other prior works on learnable front-ends.

Table 3: Word error rate (%) in CHiME-3 Challenge database for multi-condition
training.

Test Cond | MFB | PFB | RAS | MHE | A-R | A-R,M-R
---|---|---|---|---|---|---
Sim_dev | 12.9 | 13.3 | 14.7 | 13.0 | 12.4 | 11.9
Real_dev | 9.9 | 10.7 | 11.4 | 10.2 | 9.9 | 9.5
Avg. | 11.4 | 12.0 | 13.0 | 11.6 | 11.2 | 10.7
Sim_eval | 19.8 | 19.4 | 22.7 | 19.7 | 19.0 | 18.7
Real_eval | 18.3 | 19.2 | 20.5 | 18.5 | 17.2 | 17.0
Avg. | 19.1 | 19.3 | 21.6 | 19.1 | 18.1 | 17.8

Table 4: WER (%) for cross-domain ASR experiments.

Filters | ASR Trained and Tested on
---|---
Learned on | Aurora-4 | CHiME-3
Aurora-4 | 9.1 | 14.3
CHiME-3 | 9.2 | 14.2

Fig. 4: ASR performance in WER (%) for VOiCES database.

###  3.2 CHiME-3 ASR

The CHiME-3 corpus for ASR contains multi-microphone tablet device recordings
from everyday environments, released as a part of 3rd CHiME challenge [20].
Four varied environments are present - cafe (CAF), street junction (STR),
public transport (BUS) and pedestrian area (PED). For each environment, two
types of noisy speech data are present - real and simulated. The real data
consists of $6$-channel recordings of sentences from the WSJ$0$ corpus spoken
in the environments listed above. The simulated data was constructed by
artificially mixing clean utterances with environment noises. The training
data has $1600$ (real) noisy recordings and $7138$ simulated noisy utterances,
constituting a total of $18$ hours of training data. We use the beamformed
audio in our ASR training and testing. The development (dev) and evaluation
(eval) data consists of $410$ and $330$ utterances respectively. For each set,
the sentences are read by four different talkers in the four CHiME-3
environments. This results in $1640$ ($410\times 4$) and $1320$ ($330\times
4$) real development and evaluation utterances.

The results for the CHiME-3 dataset are reported in Table 3. The ASR system
with SincNet performs similar to baseline MFB features. The initial approach
of raw waveform filter learning with acoustic FB relevance weighting (A-R)
improves over the baseline system as well as the other multiple noise robust
front-ends considered here. The proposed approach of 2-stage relevance
weighting over learned acoustic and modulation representations provides
significant improvements over baseline features (average relative improvements
of $7$% over MFB features in the eval set).

###  3.3 Representation transfer across tasks

In a subsequent analysis, we perform a cross-domain ASR experiment, i.e., we
use the acoustic filterbank learned from one of the datasets (either Aurora-4
or CHiME-3 challenge) to train/test ASR on the other dataset. The results of
these cross-domain filter learning experiments are reported in Table 4. The
rows in the table show the database used to learn the acoustic FB and the
columns show the dataset used to train and test the ASR (all other layers in
Figure 3 are learned in the ASR task). The performance reported in this table
are the average WER on each of the datasets. The results shown in Table 4
illustrate that the filter learning process is relatively robust to the domain
of the training data, suggesting that the proposed approach can be generalized
for other “matched” tasks.

###  3.4 VOiCES ASR

The Voices Obscured in Complex Environmental Settings (VOiCES) corpus is a
creative commons speech dataset being used as part of VOiCES Challenge [21].
The training data set of $80$ hours has $22,741$ utterances sampled at $16$kHz
from $202$ speakers, with each utterance having $~{}12-15$s segments of read
speech. We performed a 1-fold reverberation and noise augmentation of the data
using Kaldi [29]. The ASR development set consists of $20$ hours of distant
recordings from the $200$ VOiCES dev speakers. It contains recordings from $6$
microphones. The evaluation set consists of $20$ hours of distant recordings
from the $100$ VOiCES eval speakers and contains recordings from $10$
microphones. The ASR performance on VOiCES dataset with baseline MFB features
and our proposed approach (A-R,M-R) of 2-step relevance weighting is reported
in Figure 4. These results suggest that the proposed model is also scalable to
relatively larger ASR tasks where consistent improvements can be obtained with
the proposed approach.

##  4 Summary

The summary of the work is as follows.

  * •

Extending the previous efforts in 2-stage relevance weighting approach with
the use of embeddings feedback from past prediction.

  * •

Incorporating the feedback in the form of word2vec style senone embedding for
the task of learning representations.

  * •

Performance gains in terms of word error rates for multiple ASR tasks.

## References

  * [1] Laurens van der Maaten and Geoffrey Hinton,  “Visualizing data using t-sne,”  Journal of machine learning research, vol. 9, no. Nov, pp. 2579–2605, 2008.
  * [2] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean,  “Efficient estimation of word representations in vector space,”  Proc. of International Conference on Learning Representations (ICLR), arXiv preprint arXiv:1301.3781, 2013.
  * [3] Dimitri Palaz, Ronan Collobert, and Mathew Magimai Doss,  “Estimating phoneme class conditional probabilities from raw speech signal using convolutional neural networks,”  Proceedings of Interspeech, pp. 1766–1770, 2013.
  * [4] Tara N Sainath, Brian Kingsbury, Abdel Rahman Mohamed, and Bhuvana Ramabhadran,  “Learning filter banks within a deep neural network framework,”  in IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), 2013, pp. 297–302.
  * [5] Zoltán Tüske, Pavel Golik, Ralf Schlüter, and Hermann Ney,  “Acoustic modeling with deep neural networks using raw time signal for LVCSR,”  in Proc. of Interspeech, 2014, pp. 890–894.
  * [6] Yedid Hoshen, Ron J Weiss, and Kevin W Wilson,  “Speech acoustic modeling from raw multichannel waveforms,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 4624–4628.
  * [7] Tara N Sainath, Ron J Weiss, Andrew Senior, Kevin W Wilson, and Oriol Vinyals,  “Learning the speech front-end with raw waveform CLDNNs,”  in Proc. of Interspeech, 2015, pp. 1–5.
  * [8] Hardik B Sailor and Hemant A Patil,  “Filterbank learning using convolutional restricted boltzmann machine for speech recognition.,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016, pp. 5895–5899.
  * [9] Purvi Agrawal and Sriram Ganapathy,  “Unsupervised raw waveform representation learning for ASR,”  Proc. of Interspeech 2019, pp. 3451–3455, 2019.
  * [10] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli,  “wav2vec: Unsupervised pre-training for speech recognition,”  Proc. of Interspeech, pp. 3465–3469, 2019.
  * [11] Mirco Ravanelli and Yoshua Bengio,  “Interpretable convolutional filters with SincNet,”  in Proc. of Neural Information Processing Systems (NIPS), 2018.
  * [12] Santiago Pascual, Mirco Ravanelli, Joan Serrà, Antonio Bonafonte, and Yoshua Bengio,  “Learning problem-agnostic speech representations from multiple self-supervised tasks,”  Proc. of Interspeech, pp. 161–165, 2019.
  * [13] Sarel Van Vuuren and Hynek Hermansky,  “Data-driven design of RASTA-like filters.,”  in Eurospeech, 1997, vol. 1, pp. 1607–1610.
  * [14] Jeih-Weih Hung and Lin-Shan Lee,  “Optimization of temporal filters for constructing robust features in speech recognition,”  IEEE Transactions on Audio, Speech, and Language Processing, vol. 14, no. 3, pp. 808–832, 2006.
  * [15] Hardik B Sailor and Hemant A Patil,  “Unsupervised learning of temporal receptive fields using convolutional rbm for asr task,”  in European Signal Processing Conference (EUSIPCO). IEEE, 2016, pp. 873–877.
  * [16] Purvi Agrawal and Sriram Ganapathy,  “Modulation filter learning using deep variational networks for robust speech recognition,”  IEEE Journal of Selected Topics in Signal Processing, vol. 13, no. 2, pp. 244–253, 2019.
  * [17] Purvi Agrawal and Sriram Ganapathy,  “Interpretable representation learning for speech and audio signals based on relevance weighting,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2823–2836, 2020.
  * [18] Ronald J Williams and David Zipser,  “A learning algorithm for continually running fully recurrent neural networks,”  Neural computation, vol. 1, no. 2, pp. 270–280, 1989.
  * [19] Hans-Günter Hirsch and David Pearce,  “The Aurora experimental framework for the performance evaluation of speech recognition systems under noisy conditions,”  in ASR2000-Automatic Speech Recognition: Challenges for the new Millenium ISCA Tutorial and Research Workshop (ITRW), 2000.
  * [20] Jon Barker, Ricard Marxer, Emmanuel Vincent, and Shinji Watanabe,  “The third ‘CHiME’ speech separation and recognition challenge: Dataset, task and baselines,”  in IEEE Workshop on ASRU, 2015, pp. 504–511.
  * [21] Mahesh Kumar Nandwana, Julien Van Hout, Colleen Richey, Mitchell McLaren, Maria Alejandra Barrios, and Aaron Lawson,  “The VOiCES from a distance challenge 2019,”  Proc. of Interspeech, pp. 2438–2442, 2019.
  * [22] John S Garofolo, Lori F Lamel, William M Fisher, Jonathan G Fiscus, and David S Pallett,  “DARPA TIMIT acoustic-phonetic continous speech corpus,”  NASA STI/Recon technical report, vol. 93, 1993.
  * [23] Purvi Agrawal and Sriram Ganapathy,  “Robust raw waveform speech recognition using relevance weighted representations,”  in Proc. of Interspeech, 2020, pp. 1649–1653.
  * [24] Yu Zhang, Pengyuan Zhang, and Yonghong Yan,  “Attention-based LSTM with multi-task learning for distant speech recognition,”  Proc. of Interspeech, pp. 3857–3861, 2017.
  * [25] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams,  “Learning representations by back-propagating errors,”  Nature, vol. 323, no. 6088, pp. 533, 1986.
  * [26] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky,  “Instance normalization: The missing ingredient for fast stylization,”  arXiv preprint arXiv:1607.08022, 2016.
  * [27] Sergey Ioffe and Christian Szegedy,  “Batch normalization: Accelerating deep network training by reducing internal covariate shift,”  Proc. of ICML, pp. 448–456, 2015.
  * [28] Adam Paszke, Sam Gross, Soumith Chintala, and Gregory Chanan,  “PyTorch,”  Computer software. Vers. 0.3, vol. 1, 2017.
  * [29] Daniel Povey et al.,  “The KALDI speech recognition toolkit,”  in IEEE ASRU. IEEE Signal Processing Society, 2011, number EPFL-CONF-192584.
  * [30] Diederik P Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  Proc. of ICLR, arXiv preprint arXiv:1412.6980, 2015.
  * [31] Chanwoo Kim and Richard M Stern,  “Power-normalized cepstral coefficients (PNCC) for robust speech recognition,”  in ICASSP, 2012, pp. 4101–4104.
  * [32] Seyed Omid Sadjadi, Taufiq Hasan, and John HL Hansen,  “Mean Hilbert envelope coefficients (MHEC) for robust speaker recognition,”  in Proc. of Interspeech, 2012.
  * [33] Thomas Drugman, Yannis Stylianou, Langzhou Chen, Xie Chen, and Mark JF Gales,  “Robust excitation-based features for automatic speech recognition,”  in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 4664–4668.