TFix^+: Self-configuring Hybrid Timeout Bug Fixing for Cloud Systems

Jingzhu He,
Ting Dai, Member, IEEE,
and Xiaohui Gu, Senior Member, IEEE
Jingzhu He is with ShanghaiTech University.

E-mail<EMAIL_ADDRESS>
Ting Dai is with IBM Research.

E-mail<EMAIL_ADDRESS>
Xiaohui Gu is with North Carolina State University.

E-mail<EMAIL_ADDRESS>
IEEE Transactions on Dependable and Secure Computing (TDSC)

Timeout bugs can cause serious availability and performance issues which are often difficult to fix due to the lack of diagnostic information. Previous work proposed solutions for fixing specific type of timeout-related performance bugs.
In this paper, we present TFix$^+$, a self-configuring timeout bug fixing framework for automatically correcting two major kinds of timeout bugs (i.e., misused timeout bugs and missing timeout bugs) with dynamic timeout value predictions.
TFix$^+$ provides two new hybrid schemes for fixing misused and missing timeout bugs, respectively.
TFix$^+$ further provides prediction-driven timeout variable configuration based on runtime function tracing.
We have implemented a prototype of TFix$^+$ and conducted experiments on 16 real world timeout bugs.
Our experimental results show that TFix$^+$ can effectively fix 15 out of tested 16 timeout bugs.

Reliability, availability, and serviceability; Distributed debugging; Automatic bug fixing; Diagnostics; Performance

§ INTRODUCTION

Timeout is commonly used to handle unexpected failures in complex distributed systems. For example, when a server $A$ sends a request to another server $B$, $A$ can use the timeout mechanism to avoid endless waiting in case $B$ fails to respond.
Timeout bugs can severely impact system availability and performance, causing system hang and performance degradation [15, 20, 12].
Our previous bug study [12] shows that 78% of real world timeout bugs are caused by missing timeout mechanisms or mis-using timeout schemes (e.g., setting a too small or a too large timeout value).

For example, a misused timeout bug caused Amazon DynamoDB to experience a five-hour service outage in 2015 [4]. The root cause of this bug is an improper timeout value setting under unexpected workload increase.
Timeout bugs are often difficult to fix because of the lack of diagnostic information and the correct timeout value often depends on the runtime execution environments and application workloads.

Previous work [17, 18] has proposed bug fixing schemes for specific bug types (e.g., hang bugs, data corruption bugs) using static code analysis methods. However, an effective generic timeout bug fixing solution requires both runtime knowledge (e.g., network bandwidth for setting a proper RPC timeout variable) and root cause function localization [17, 18]. In this paper, we propose a hybrid approach to fixing both misused and missing timeout bugs. We introduce prediction-driven timeout configuration scheme to realize self-configuring fully automated timeout bug fixing.

§.§ A Motivating Example

To better understand how real-world timeout bugs happen, and how they can affect cloud services,
we use the Yarn-1630 (v2.2.0) [We use “system name-bug #" to denote different bugs.] bug as one example shown by Figure <ref>.
This bug is caused by missing timeout settings on asynchronous polling operations in YarnClientImpl.
When submitting an application, the Yarn client periodically polls the ResourceManager to get the application status (line #154), until the application is submitted successfully (line #155).

With a slow submission process in the ResourceManager end, the Yarn client gets stuck, causing system hang indefinitely.

Figure <ref> also shows the patch generated by TFix$^+$.

To break the Yarn client from the infinite waiting while still guaranteeing that the asynchronous communications between the Yarn client and the ResourceManager can succeed during normal runs, TFix$^+$ inserts a timeout mechanism in the loop body (added lines between line #156 to #170) with user configurable timeout variables (added lines before line #142). The automatically generated path by TFix$^+$ saves users diagnosing and development effort to fix the bug.

To further ease the users' testing work and better accommodate their runtime environments, TFix$^+$ provides dynamic timeout value predictions based on runtime monitoring data. In this example, we configure the timeout value as 1130 ms based on the runtime available resources and our workloads.

§.§ Contribution

In this paper, we present TFix$^+$, a self-configuring hybrid timeout bug fixing system.
Compared with its preliminary version TFix [17], which focuses only on fixing misused timeout bugs using static analysis methods,
TFix$^+$ provides self-configuring holistic fixing solutions for both misused and missing timeout bugs. TFix$^+$ proposes dynamic timeout value prediction scheme using both runtime tracing and static taint analysis.

When a timeout bug is detected by a runtime bug detection tool such as TScope [16], TFix$^+$ executes a drill-down bug analysis protocol to automatically narrow down the root cause of the detected bug and produce bug fixing patches with sound timeout value recommendations.
Specifically, TFix$^+$ first determines whether the detected timeout bug is caused by misused timeout bugs (i.e., incorrectly used timeout variables) or missing timeout bugs (i.e., lacking timeout mechanisms).

For misused timeout bugs, TFix$^+$ identifies timeout affected functions from an application performance trace and then uses those functions to pinpoint misused timeout variables using taint analysis.

For missing timeout bugs, TFix$^+$ identifies the root cause as either an infinite loop or a blocking function call via stack trace analysis, and inserts different timeout mechanisms with configurable timeout variables, respectively.
To achieve high efficiency and obtain strong robustness,
TFix$^+$ recommends a proper timeout value to adapt to the user runtime environments based on the historical execution time of the pinpointed timeout affected functions during normal runs.
TFix$^+$ validates the generated patch by checking whether the bug still occurs under the same workload and all the test suites are passed after adopting the patch.
Specifically, our paper makes the following contributions.

* We describe a holistic drill-down bug analysis framework which can automatically narrow down the root cause of a timeout bug and provide a complete bug fixing patch.

* We describe a hybrid scheme that combines dynamic application performance tracing and static taint analysis to find the timeout affected function and localize the misused timeout variable when the bug is caused by mis-using a timeout mechanism.

* We propose a dynamic timeout value prediction scheme to provide proper timeout value recommendation based on the historical execution time of the timeout affected function.

* We have conducted an empirical study over 91 real production timeout bugs to quantify the coverage of our fixing strategy. TFix$^+$ can fix 79% bugs completely. We have implemented a prototype of TFix$^+$ and conducted experimental evaluation over 16 real world reproduced timeout bugs. The results show that TFix$^+$ can successfully fix 15 out of 16 tested bugs and provide correct timeout configurations in seconds.

The rest of the paper is organized as follows. Section <ref> describes design details.
Section <ref> presents the experimental evaluation.
Section <ref> discusses the limitation of TFix$^+$.
Section <ref> discusses related work.
Finally, the paper concludes in Section <ref>.

§ SYSTEM DESIGN

The architecture of TFix$^+$.

In this section, we present the design details of the TFix$^+$ system.
We first provide an overview of TFix$^+$, followed by the fixing strategies of misused and missing timeout bugs, respectively.

Next, we describe our prediction-driven timeout value configuration and the patch validation.

§.§ Approach Overview

As shown in Figure <ref>,
when a server system experiences software hang or performance slowdown, TFix$^+$ leverages
TScope [16] to identify whether the anomaly is caused by a timeout bug after analyzing
a window of system call trace collected by the kernel tracing module LTTng [13].
If TScope confirms that the performance anomaly is caused by a timeout bug, TFix$^+$ is triggered to conduct further drill-down analysis. TFix$^+$ first performs timeout bug classification to determine whether the timeout bug is caused by missing timeout check or incorrectly using timeout variables, which is described in our previous work [17].

If a misused timeout bug is confirmed, TFix$^+$ identifies those timeout-affected functions by checking the abnormality of their execution time and frequency from application function traces (Section <ref>).
TFix$^+$ further checks whether the timeout-affected functions use a too large or a too small timeout value when the bug happens. (Section <ref>).
For both cases, TFix$^+$ uses corresponding checking conditions to pinpoint the misused timeout variables associated the timeout-affected functions through static taint analysis (Section <ref>).

If a missing timeout bug is confirmed, TFix$^+$ identifies the root cause as either an infinite loop or a blocking function call by examining the stack traces (Section <ref>). TFix$^+$ then adopts different patching strategies to add timeout mechanisms, either inside the loop body or surrounding the blocking function call (Section <ref>).

Lastly, TFix$^+$ produces prediction-driven timeout value configuration to adapt to the user runtime environment automatically based on historical execution data and performs dynamic validations over the generated patch (Section <ref>).
The whole drill-down bug diagnosis protocol is executed automatically without requiring any human intervention.
We will describe each component in detail in the following subsections.

§.§ Misused Timeout Bug Classification

TFix leverages TScope [16] to determine whether a detected system anomaly is caused by a timeout bug. Timeout bugs can be broadly classified into two groups: 1) misused timeout bug where the system anomaly is caused by some incorrectly used timeout variables; and 2) missing timeout bug where the system anomaly is caused by lack of timeout mechanisms. In this paper, TFix focuses on fixing misused timeout bug by identifying root cause timeout variables and suggesting proper timeout values for fixing the bug. To achieve this goal, TFix first needs to classify a detected timeout bug as a misused timeout bug. Intuitively, a misused timeout bug is triggered when a certain timeout related function (e.g., URL.openConnection, ServerSocketChannel.open, ReentrantLock.tryLock) is executed.
Thus, TFix performs misused timeout bug classification by checking whether timeout related functions are invoked when the bug is triggered.

TFix first provides an offline comparative analysis to extract timeout related functions for each server system. We observe that different server systems often employ different timeout related classes. However, although each system has multiple timeout variables to guard connections,
those timeout mechanisms are usually configured by common timeout configuration classes. For example, Flume's timeout mechanisms are built on top of MonitorCounterGroup inside the instrumentation class,
which is used for monitoring system state and building timers. To identify those timeout configuration classes, we employ a dual testing scheme. For each system, we produce a set of test cases each of which consists of two dual parts: one part uses timeout and the other part does not employ timeout. For example, we build a socket connection between the client and HDFS server to write data into HDFS.
The difference between the two counterparts is that one has socket write timeout while the other does not.
We use HProf <cit.> to trace the invoked Java functions during the execution of those dual test cases.
We compare the lists of the Java functions produced by the two dual test cases in order to extract those functions which only appear in the profiling result of those test cases with timeout mechanisms. To further narrow down the scope of timeout related functions, we only keep those functions that are related to timeout configuration, network connection and synchronization since timeout mechanisms need timers to monitor the elapsed time and timeout mechanisms are often used in network connection and synchronization operations.

After identifying those timeout related functions, TFix needs to employ an efficient scheme to match with those functions during production run. To avoid expensive application function instrumentation, TFix employs a system call frequent episode mining scheme <cit.> to match with those timeout related functions. The basic idea is to extract unique system call sequences produced by those timeout related functions during the offline analysis. During production run, TFix performs the frequent episode mining over runtime system call sequences and checks whether the frequent system call sequences produced by those timeout related functions exist in the runtime trace. If we find one or more timeout related function matching, TFix classifies the detected bug as a misused timeout bug.

§.§ Misused Timeout Bug Fixing

In this subsection, we discuss how TFix$^+$ identifies the timeout variable for a misused timeout bug. TFix$^+$ first identifies the timeout affected function, then differentiates whether the bug is caused by a too large or a too small timeout value. For each case, TFix$^+$ takes different matching criteria to identify the misused timeout variable via static taint analysis.

§.§.§ Timeout Affected Function Identification

A web search example to illustrate how Dapper's trace is generated.

The Dapper trace of the web search example.

After classifying a detected bug as a misused timeout bug [17],

TFix$^+$ identifies the timeout affected functions by checking the abnormality of the functions' execution time and frequency. To achieve this goal, TFix$^+$ leverages a commonly used application performance tracing tool, i.e., Google's Dapper framework [28]. Dapper allows us to trace the beginning and ending timestamps of all function calls and the control flow graph for the diagnosed bug.
We choose Dapper tracing tool because it supports distributed systems and incurs low runtime overhead to production systems. The existing implementations of Dapper tracing can only be applied to RPC related functions.
TFix$^+$ augments the Dapper tracing tool to support all timeout related functions, such as IPC functions and synchronization functions [17].

Dapper's tracing can be modeled as a tree. The tree nodes are called spans and the edges indicate the control flow between spans.
Each span contains a span id and a parent id.
The root span does not have a parent id.

All spans in the tree share the same trace id.
A span represents a RPC connection or a function call,
containing the information of both the caller and callee (or the client and server).

A span also contains a set of activities including the beginning timestamps, ending timestamps, process name, thread name, and the messages embedded in a RPC or function call.

For example, in Figure <ref>, a user issues a web search request to the local server A.
Server A receives the request and sends the request to the remote server B and C to retrieve the results.
Server B stores the data locally and sends the result back to Server A.
Server C does not contain the data, thus it sends a request to the remote server D to get the result before responding to Server A.
In this case, a simple web search contains four RPC calls between the user and the local server or between servers.

When we apply the Dapper framework to trace the control flow of the example in Figure <ref>,
we get a RPC tree, shown in Figure <ref>.
The root span (i.e., Span 0) represents the RPC request and response between the user and Server A.

Span 1 indicates the RPC connection between Server A and Server B.
Span 2 represents the RPC connection between Server A and Server C.
Span 3 illustrates the RPC connection between Server C and Server D.

The edge between each span indicates a control flow.
For example, Server A receives the user's request and issues RPC call 1 and 2 to Server B and C, respectively.
Thus, Span 1 and Span 2 share the same parent span (i.e., Span 0).

[frame=single, style=baseC, basicstyle=]
"i":"1b1bdfddac521ce8", "s":"df4646ae00070999", "b":1543260568612,
"e":1543260568654, "r":"RunJar", "p":["84d19776da97fe78"],

A trace example of Dapper.

After retrieving a Dapper trace for a target bug,
we first extract the execution time and frequency of all the functions invoked when the bug happens.
we calculate the frequency of each function by simply counting how many times it is invoked in the Dapper trace.
We calculate the execution time of each function by subtracting the beginning time from the ending time.
Figure <ref> shows a Dapper trace example with

various labels indicating different information.
Among them, “b” and “e” indicate the beginning timestamp and the ending timestamp of a function, respectively.
“d” represents the function name and “r” represents the process name. We identify the timeout affected functions by
checking the abnormality in the functions' execution time
and frequency.

§.§.§ Misused Timeout Bug Differentiation

TFix$^+$ checks whether timeout affection functions use a too large or a too small timeout value based on the following rationales.

If the timeout value is set to be too large, the execution time of the timeout-affected functions become longer than their normal execution time, causing system hang or slowdown.
In contrast, if the timeout value is set to be too small, the system will experience repeated failures due to frequent timeout.
Therefore, the frequency of the timeout-affected functions become higher than their normal execution frequency.

The HBase-13647 and HBase-6684 bugs show the examples where the timeout value is set too large, i.e., the RPC connection timeout is misconfigured as Integer.MAX_VALUE.

Under normal state,
the HBase client can successfully exchange messages with the HBase server (e.g., HMaster, RegionServer) within tens of seconds.
However, when the HBase server fails, the HBase client hangs for about 24 days,
significantly increasing the execution time of the HBase client's RPC function.
TFix$^+$ identified the RPC function as the timeout affected function which uses a too large timeout value.

The HDFS-4301 bug shows the example where the timeout value is set too small.
The timeout value is set to 60 seconds. During normal run, NameNode transfers file system image to the Secondary NameNode within 60 seconds. However, when the image size is too large or network congestion occurs, the transmission process cannot be finished in 60 seconds. Under this circumstance, the system experiences repeated timeout failures because the system retries the image transfer but fails each time. TFix$^+$ identified the doGetUrl function as the timeout affected function because their invocation frequencies significantly increase. Besides, the execution time of doGetUrl function is nearly the same as the timeout variable value setting (i.e., 60 seconds).

§.§.§ Misused Timeout Variable Identification

TFix$^+$ correlates the pinpointed functions with different timeout variables included in the system configuration files in order to identify
the specific misused timeout variables that are attributing to the timeout bug.

To localize which timeout variable is used when the bug happens,
we first retrieve all the timeout variables in the target system.
In large scale distributed systems,
timeout variables along with other configurable parameters
are often stored in specific configuration files [34].

For example, in a Hadoop system, all the configurable variables are defined with default values in configuration files, such as HConstant and DFSConfigKeys classes.
These variables' value can be overridden and customized by users in .xml configuration files.
Thus, all the variables appear in systems' configuration files and contain “timeout” keyword in their names are potentially related to misused timeout bugs.
Next, we taint all these timeout variables using static taint analysis tools [2] and conduct data flow dependency analysis on them to extract all related variables.
We then check whether the timeout affected functions use the timeout related variables.
If a timeout affected function $f$ uses a timeout variable $v_t$,
we consider $v_t$ as a misused timeout variable candidate.

We then compare the execution time of $f$ with the value of $v_t$ to identify the misused timeout values.

If the bug is caused by a too large value, the execution time of $f$ should be no larger than the value of $v_t$, considering the system experiences a long delay and we may terminate the system tracing earlier. If the bug is caused by a too small value, we consider $v_t$ as the misused timeout variable if the execution time of $f$ matches the value of $v_t$.

For example, Figure <ref> shows how TFix$^+$ uses the static taint analysis to identify the misused timeout variable for the HDFS-4301 bug.

In this bug, the default timeout value is set to 60 seconds in DFS_IMAGE_TRANSFER_TIMEOUT_DEFAULT in DFSConfigKeys.java. If users configure the timeout variable dfs.image.transfer.timeout in hdfs-site.xml, the system uses the configured value. Otherwise, the system uses the default value.
We annotate both dfs.image.transfer.timeout and DFS_IMAGE_TRANSFER_TIMEOUT_DEFAULT as tainted.
After applying static taint analysis, we find that the timeout affected function doGetUrl uses both tainted variables at line #271-273. Since the user configures the value of dfs.image.transfer.timeout in hdfs-site.xml,
we determine that the misused timeout variable is dfs.image.transfer.timeout.
We also perform cross validation between the timeout variable value and the execution time of the timeout affected function to confirm whether our timeout variable identification is accurate.

§.§ Missing Timeout Bug Fixing

In this subsection, we describe how TFix$^+$ identifies the root cause of missing timeout bugs as either an infinite loop or a blocking function call and inserts proper timeout mechanisms to fix the missing timeout bugs.

§.§.§ Root Cause Function Pinpointing

[frame=single, style=baseJava, basicstyle=]
//Stack trace dump 1 at time 12:11:30
"main" #1 prio=5 os_prio=0 tid=... nid=... runnable [...]
java.lang.Thread.State: RUNNABLE
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at YarnClientImpl.(*@aLeft@*)@BG@submitApplication@BG@(*@aRight@*)(buggycode.java:(*@bLeft@*)@BG@71@BG@(*@bRight@*))
at testcode.main(testcode.java:51)
/*(*@eLeft@*)same function(*@eRight@*)*/ /*(*@fLeft@*)different lines(*@fRight@*)*/
//Stack trace dump 2 at time 12:11:31
"main" #1 prio=5 os_prio=0 tid=... nid=... runnable [...]
java.lang.Thread.State: RUNNABLE
at java.lang.Thread.sleep(Native Method)
at YarnClientImpl.(*@cLeft@*)@BG@submitApplication@BG@(*@cRight@*)(buggycode.java:(*@dLeft@*)@BG@75@BG@(*@dRight@*))
at testcode.main(testcode.java:51)

[overlay,remember picture]

A stack trace snippet of an infinite loop bug, i.e., Yarn-1630 (v2.2.0) bug.

[frame=single, style=baseJava, basicstyle=]
//Stack trace dump 1 at time 10:11:30
"main" #1 prio=5 os_prio=0 tid=... nid=... runnable [...]
java.lang.Thread.State: RUNNABLE
at JobEndNotifier.(*@aLeft@*)@BG@httpNotification@BG@(*@aRight@*)(JobEndNotifier.java:(*@bLeft@*)@BG@138@BG@(*@bRight@*))
at JobEndNotifier.localRunnerNotification(JobEndNotifier.java:148)
at TestJobEndNotifier.main(TestJobEndNotifier.java:139)
/*(*@eLeft@*)same function(*@eRight@*)*/      /*(*@fLeft@*)same line(*@fRight@*)*/
//Stack trace dump 2 at time 10:11:31
"main" #1 prio=5 os_prio=0 tid=... nid=... runnable [...]
java.lang.Thread.State: RUNNABLE
at JobEndNotifier.(*@cLeft@*)@BG@httpNotification@BG@(*@cRight@*)(JobEndNotifier.java:(*@dLeft@*)@BG@138@BG@(*@dRight@*))
at JobEndNotifier.localRunnerNotification(JobEndNotifier.java:148)
at TestJobEndNotifier.main(TestJobEndNotifier.java:139)

[overlay,remember picture]

A stack trace snippet of a blocking bug, i.e., Mapreduce-5066 bug.

Our previous study [12] shows that missing timeout usually causes system stuck at a certain state, i.e, repeating on executing a block of code inside of an infinite loop or endlessly waiting for an asynchronous function call to finish.

We use the stack trace analysis to pinpoint the hanging functions with infinite loops and the blocking function calls.

After a timeout bug is classified as missing timeout, TFix$^+$ dumps the stack trace of the running system continuously.

Figure <ref> shows an example of the dumped stack trace on a process.
The trace contains each thread's status, (e.g., runnable or waiting) and the detailed information of each Java function, (e.g., function name and the line number). The per-thread trace also shows the explicit control flow that for the two contingent functions, the top function is the callee of the bottom function.

To localize the hang/blocking functions, TFix$^+$ first extracts the common function(s) in multiple traces.
This is because hang and blocking are long-term stable status. The same function call appears in all the dumped traces, indicating the manifestation of the hang or blocking.

TFix$^+$ then extracts the inner-most common functions as the candidate root cause functions.

TFix$^+$ pinpoints the root cause functions after filtering out those candidate functions triggered by long-running background processes, e.g., the heartbeat process,
by analyzing an interrupted application performance trace using Dapper [28].

Specifically, TFix$^+$ forces the hang/blocking function to terminate abruptly after a period of tracing time.
TFix$^+$ then localizes the root cause function based on the following criteria:
1) the function's execution time is larger than its normal execution time;
2) the function's start time coincides with the time when the performance alert is reported by the timeout bug detection tool such as TScope [16],
and 3) the function's ending time is the termination time forced by TFix$^+$.

For example, in the MapReduce-5066 bug, the candidate root cause functions are Listener.run(), Responder.doRunLoop(), Reader.run(), and JobEndNotifier.httpNotification(). However, the first three functions are triggered by background processes, thus are not root cause functions.

TFix$^+$ excludes those three functions because their execution time is not longer than that in normal runs and their start time is much earlier than the bug detection time.
After pruning, TFix$^+$ identifies the JobEndNotifier.httpNotification() function as the root cause function.

Next, TFix$^+$ compares the line numbers of the inner-most common functions in different traces to identify the bugs as infinite loops or blocking function calls.
For infinite loops where a block of code executes repeatedly, the line numbers of the candidate root cause function in each trace should be different, as shown in Figure <ref>.
For blocking function calls, the line numbers of the candidate root cause function are always the same, as shown in Figure <ref>.

§.§.§ Adding Timeout Mechanism

TFix$^+$ adopts different strategies to add the timeout mechanisms for the system hangs on

an infinite loop or a blocking function call.

For missing timeout induced infinite loops,

we add a timeout checking in the loop body to ensure the system jump out the loop after the pre-defined time elapses.

Specifically, TFix$^+$ adds a configurable timeout variable and marks down the starting time point before the loop is executed. In every loop iteration, TFix$^+$ checks whether the elapsed time exceeds the configured timeout value using an if branch. If the checking result is yes, TFix$^+$ breaks out of the loop by throwing an exception.
For example, the Yarn-1630 (v2.2.0) bug, shown in Figure <ref>, and described in Section <ref> falls into this category.

For missing timeout induced blocking function calls,
TFix$^+$ reuses existing timeout settings if applicable,

or adds new timeout mechanisms surrounding the blocking function call.

Existing timeout mechanisms can be provided by overloaded functions or existing methods inside the same class. Overloaded methods are easily to identify.
For example, if the function is stuck on Socket.connect(SocketAddress endpoint), TFix$^+$ examines all the methods in Socket class and finds the overloaded function Socket.connect(SocketAddress endpoint, int timeout) with timeout settings. TFix$^+$ replaces the original one with the supplementary method to add a timeout setting. Besides the overloaded functions, TFix$^+$ examines all the methods of the invoked class and searches the keywords “set” and “timeout” in all the methods' names. If the method name contains both keywords, we consider it as a candidate timeout setting. TFix$^+$ calls the method to add the timeout setting to the function call. For example, for HDFS-3180 bug, the system hangs on URLConnection.connect() function. We search all the methods inside the URLConnection and find connection.setConnectTimeout() and connection.setReadTimeout() methods. TFix$^+$ leverages the two existing methods to add timeout mechanisms.

If there is no existing configured timeout mechanisms, TFix$^+$ puts the function in a callable thread and leverages ExecutorService and Future interfaces provided by standard Java library to monitor and manager the execution of callable threads. For example, Figure <ref> shows the patch of HDFS-4176 bug. When the bug occurs, the system hangs on the getActiveNodeProxy().rollEditLog() function. TFix$^+$ does not find any pre-configured timeout mechanisms, so TFix$^+$ replaces the getActiveNodeProxy().rollEditLog() function with rollEditLogWithTimeout() function and puts the getActiveNodeProxy().rollEditLog() function into a callable thread. TFix$^+$ configures the timeout mechanism using future.get(timeout, TimeUnit.MILLISECONDS) method to terminate the callable thread when the elapsed time exceeds the value of timeout.

§.§ Prediction-driven Timeout Value Configuration and Patch Validation

After pinpointing the misused timeout variable or adding a configurable timeout variable, TFix$^+$ recommends a proper timeout variable value, to fix the timeout bug. During the normal run, a correct timeout value should be consistent with the expected execution time of the timeout affected function. This runtime execution time depends on the application workload (e.g., input file size) and the runtime available resources (e.g., CPU availability and network bandwidth). TFix$^+$ employs an online timeout value prediction scheme to adapt to dynamic runtime environments.

To achieve model-driven prediction, we first perform continuous collection of normal execution time of the identified function when the timeout bug is not triggered. We also collect the corresponding runtime system metrics and function input parameters when the function is invoked.
We then employ polynomial regression schemes to establish the mapping from the application workload and runtime environment parameters $S_1, S_2,..., S_k,..., S_N$ to function's execution time $T$.
The polynomial fitting function is calculated by Equation <ref>, where $\beta_j$ is the coefficient for the combinatorial term ${S_1}^{p_1}{S_2}^{p_2}...{S_i}^{p_i}...{S_N}^{p_N}$. Each input parameter $S_i$ has the polynomial degree $p_i$. The sum of the degree $p_i$ should be no larger than $P$ which is the highest polynomial degree. This equation contains all the combinatorial terms constructed by all the input parameters when the polynomial degree is no greater than $P$.

\begin{equation}
\begin{aligned}
T_r = \sum(\beta_j{S_1}^{p_1}{S_2}^{p_2}...{S_i}^{p_i}...{S_N}^{p_N})  \\
where \quad p_i \geq 0 \quad and \quad \sum_{i=1}^{N}(p_i) \leq P
\end{aligned}
\label{eq:prediction}
\end{equation}

Based on our experimental evaluation results, we set the highest polynomial degree $P$ to 3 to both avoid overfitting and achieve fast online prediction. TFix$^+$ changes the $P$ from 1 to 3 and selects the best fitting model using the least-square error.

In practice, any prediction model can hardly achieve perfect prediction. In order to achieve effective timeout bug fixing, TFix$^+$ includes built-in prediction error handling mechanisms. Particularly, under-estimation error (i.e., predicted timeout value is less than expected timeout value) of the prediction model has serious impact to our solution, which causes our timeout bug fix to fail. Thus, we introduce a padding to avoid the under-estimation error. We calculate the estimated timeout values $T_{est}$ produced by the regression model and measure the exact historical execution time $T_{hist}$ during the normal run. We calculate the relative fitting error, which is $(T_{est}-T_{hist})$ over the exact execution time $T_{hist}$, i.e., $\frac{T_{est}-T_{hist}}{T_{hist}}$. Suppose there are $M$ measured data points, the padding ratio is calculated by Equation <ref>.

\begin{equation}
T_{ratio} = 2\times \max_{i \in \{ 1, 2,..., M\}}(\frac{T_{est}^i-T_{hist}^i}{T_{hist}^i})
\label{eq:ratio}
\end{equation}

Our experimental results show that our padding scheme can avoid all under-estimations without imposing too much timeout delay. Finally, TFix$^+$ predicts the timeout value $T_{predict}$ using Equation <ref>.

\begin{equation}
T_{predict} = T_r\times (1 + T_{ratio})
\label{eq:padding}
\end{equation}

Timeout value prediction based on the function's input (i.e., the transferred image size).

Timeout value prediction based on the network traffic.

Timeout value prediction for the distributed HDFS-4301 bug. The bug occurs when setting improper timeout value on the image transfer between the Primary NameNode and the Secondary NameNode. The Primary NameNode is set up on our lab machine and the Secondary NameNode is set up on an AWS machine.

The timeout value recommendation considers three different cases: 1) the timeout value is set too large, 2) the timeout value is set too small, and 3) timeout mechanism is missing.
For the bugs caused by too large timeout values, the regression model is an interpolation model, since the predicted timeout value is within the range of the historical execution time of the timeout affected functions during normal runs.

For the bugs caused by too small timeout values, the predicted timeout value $T_{predict}$ is beyond the range of the function's execution time $T_{hist}$ in normal runs. In this case, TFix$^+$ builds an extrapolation prediction model to suggest a timeout value.

For the missing timeout cases, the regression model can be either interpolation or extrapolation model, which depends on the current workload and system environment. For example,
Figure <ref> shows the mapping model from the transferred fsimage size and the network traffic to the execution of the function doGetUrl() for HDFS-4301 bug. Figure <ref> shows the mapping model of the transferred image size and execution time when we fix the network traffic. Figure <ref> shows the mapping model of the file size and network traffic when we fix the image size to 20 megabytes. Figure <ref> shows an extrapolation model and Figure <ref> shows an interpolation model, respectively.

Our online prediction scheme can adapt to different applications, because the predicted value is calculated from the normal run statistics on the specific applications. Our previous work [17] adopts a profiling-based heuristic method to provide timeout value recommendation, which might take a long time and high profiling cost to derive the right timeout value configuration.

After the patch is generated, we validate whether the patch is correct or not. A correct patch should fix the bug without introducing new bugs. To check whether the bug is fixed, we apply the patch to the program and re-run the bug triggering test cases. We trigger the bug under the same triggering condition and workload to observe whether the bug occurs again. To check whether our patch introduces new bugs, we run all the test suites provided by the release package (e.g., [3]) to check whether all of them are passed. Normally, a mature system is packaged with test cases to test all the functionalities. For example, Hadoop system is equipped with its JUnit tests to test each class's functions. We determine that TFix$^+$'s patch is valid when it does not influence any other functionalities, that is, all the tests are passed.

§ EVALUATION

In this section, we present our evaluation details.
We first introduce our evaluation methodology.
We then present our empirical study results on 91 real world misused or missing timeout bugs.
we then present the experimental evaluation results on 16 reproduced timeout bugs.
Finally we present several case studies.

All the experiments are conducted in our research lab in a cluster of hosts which are equipped with quad-core Xeon 2.53Ghz CPUs and 16GB memories and running 64-bit Ubuntu v16.04.
The system call trace is collected using LTTng v2.0.1.
The function call trace is collected using Google's Dapper framework. We adopt existing static taint tracking framework, i.e., Checker [2], to perform misused timeout variable identification.
We use jstack to dump the stack traces for all the Java processes.
TFix$^+$ is implemented on top of Soot compiler [6] to patch binary code.
We use Python's scikit-learn [5] package to build the polynomial regression model to predict timeout values as described in Equation <ref> to <ref>.

§.§ Evaluation Methodology

System description.

System     Setup Mode      1c|Description
Hadoop              Distributed     c]@l@The utilities and libraries for Hadoop
HDFS                Distributed     Hadoop distributed file system
MapReduce           Distributed     c]@l@Hadoop big data processing framework
Yarn          Distributed     c]@l@Hadoop resource management platform
HBase               Standalone     c]@l@Non-relational, distributed database
Flume               Standalone     c]@l@Log data collection/aggregation
/movement service

We first conduct an empirical study on 91 real world misused or missing timeout bugs which are collected by our previous study [12] to manually evaluate the fixing strategies of TFix$^+$.

We check each bug to determine whether TFix$^+$ can fix the bug without introducing new problems.
We should note that, our benchmarks do not contain concurrency bugs, e.g., race conditions and deadlocks. Even though TFix$^+$ can fix some of them by adding timeout mechanisms, in most cases, fixing concurrency bugs requires specific techniques, e.g., order checking [23], atomicity checking [22] and etc.

To the best of our effort, we have reproduced 16 timeout bugs from six commonly-used open-source systems, listed in Table <ref>.
We set up four systems in the distributed mode to investigate timeout issues occurring on the communications among different nodes.
The 16 reproduced bugs are listed in Table <ref>, including 11 misused and 5 missing timeout bugs.
They cover timeout problems in both network communications and intra-node coordinations, representing common real-world missing and misused timeout bugs.

Workload: To replay real-word scenarios, each system runs a specific workload, listed in Table <ref>.

for the Hadoop, HDFS, MapReduce and Yarn systems, we run word count job on a 765MB text file.
For the HBase system, we use the YCSB workload generator to make insertion, query and update operations on a table.
For the Flume system, we write log events to the log collection tool and distribute the logs repeatedly.
These workloads invoke the timeout affected functions all of our tested timeout bugs.

Expected timeout value measurement: In order to validate the prediction accuracy, we measure the exact expected timeout value during buggy run. We run the same workload under same runtime computing environment without triggering the bug and retrieve the timeout affected function's execution time in the function call trace, which is the exact expected timeout value.

§.§ Empirical Study Results

Fixing coverage of TFix$^+$.

Figure <ref> shows the fixing coverage of TFix$^+$ on 91 studied timeout bugs.
TFix$^+$ fixes 72 bugs (i.e., 79%) completely. TFix$^+$ fixes 12 (i.e., 13%) bugs partially because their fixes require not only proper timeout mechanisms but also retry schemes and program semantic repairs. For the rest seven (i.e., 8%) bugs which cannot be fixed by TFix$^+$, they are all caused by improper hard-coded timeout values.

The seven misused timeout bugs cannot be fixed by TFix$^+$ are caused by hard-coded timeout values, i.e., no configurable timeout variables.
Those timeout bugs often occur in early versions of a system, such as Hadoop 0.x version and HBase 0.x version.
TFix$^+$ cannot identify the misconfigured timeout variables in our taint analysis since the timeout affected functions do not use any configurable parameter or configuration constant.
Although TFix$^+$ cannot localize misused timeout variables, TFix$^+$ identifies the bug as a misused timeout bug and pinpoint the timeout affected function. The hard-coded timeout value should be located in the timeout affected function or its callee functions. Developers can perform a cross validation to check whether any hard-coded value equals to the timeout affected function's execution time. Then developers can expose a configurable timeout variable for users.

§.§ Experimental Results

In this subsection, we present the experimental results on 16 reproduced timeout bugs.
we first present the results of fixing misused timeout bugs and fixing missing timeout bugs,
and then we describe the results of timeout value prediction and patch validation. Finally, we present the negative case study to show why TFix$^+$ cannot fix one timeout bug completely.

§.§.§ Missing Timeout Bugs Fixing Results

Results of adding timeout mechanisms.

Bug ID        Hang Type     c]@l@Missing Timeout Bug
Fixing Strategy
HDFS-3180              Blocking func          c]@l@Use existing timeout setting

HDFS-4176              Blocking func          c]@l@Leverage future.get() method
of a callable thread
MapReduce-5066         Blocking func          c]@l@Leverage future.get() method
of a callable thread

Yarn-1630 (v2.2.0)     Infinite loop                   c]@l@Check whether the elapsed time
exceeds the pre-defined timeout
Flume-1819     Infinite loop                  c]@l@Check whether the elapsed time
exceeds the pre-defined timeout

Table <ref> shows the results of adding timeout mechanisms on the 5 missing timeout bugs.
TFix$^+$ fixes the HDFS-3180 bug by first identifying that it hangs on a blocking function call and then adding timeout mechanisms using existing timeout-setting functions, i.e., connection.setConnectTimeout() and connection.setReadTimeout().
For the HDFS-4176 and MapReduce-5066 bugs, there are no existing timeout mechanisms that can be set on corresponding blocking function calls.
TFix$^+$ fixes them by isolating the blocking functions in callable threads with timeout settings on the future.get() method.

TFix$^+$ fixes the Yarn-1630 (v2.2.0) and Flume-1819 bugs by first identifying that they hang on infinite loops and then adding timeout checking on the loop body. Specifically, TFix$^+$ checks whether the elapsed time exceeds the pre-defined timeout value in each loop iteration.

§.§.§ Prediction-driven Timeout Value Configuration and Patch Validation Results

Table <ref> shows the bug fixing results by TFix$^+$.
We list the expected timeout value under current workload for each bug. After adopting TFix$^+$'s patch with the predicted timeout value in the system, we find that 15 out of 16 bugs do not occur anymore under the same workload with all the test suites passed successfully.

We list the timeout values in the bugs' patch files in Table <ref>.
We observe that the timeout values in the manual patches are not always correct. 11 out of 16 patches of the bugs need user inputs to set manual thresholds. It means that when patching misused timeout bugs, developers usually make the timeout variable configurable for users and set a default value which requires the user to set the timeout value based on his or her own application workloads.
However, it is challenging to make the correct configurations for different production workloads and computing environments, even for experienced engineers.
For example, in the patch of Yarn-1630 (v2.3.0) bug,
the default value of the yarn.client.application-client-protocol
.poll-timeout-ms variable is configured to be -1 milliseconds, which equals infinity. Developers expose the variable for users to configure. If users do not properly configure the timeout variable, the timeout bug still happens in the patched version.

We also observed that TFix$^+$'s fixing schemes sometimes are different from the manual patches. We use HDFS-4301 bug as an example. In the patch of HDFS-4301 bug, the default value of dfs.image.transfer.timeout is still set to 60 seconds, which is identical with the timeout value before patching. However, the patch partitions the fsimage file into multiple chunks of a equal chunk size. The timeout variable limits the maximum transferring time for each chunk. In contrast, TFix$^+$ changes the timeout value to 138.68 seconds, that successfully fix the problem.

We should note that, the predicted timeout value by TFix$^+$ might be different under different workloads.
This is our design choice, because a fixed timeout setting cannot handle unexpected workload changes or environment fluctuations.
For example, in HBase-15645 bug, the misused timeout variable hbase.client.operation.timeout defines the time to block a certain table to prevent concurrency issues.
Since the table size is small for YCSB workload in our evaluation, the predicted value by TFix$^+$ is only 3.47 seconds.
If we use a large timeout value under the same YCSB workload, the user will still experience a noticeable delay in the system. On the other hand, if we increase the table size, the predicted value will increase correspondingly.

Although TFix$^+$ introduces timeout value over-estimations to avoid too small timeout bugs, the differences between the predicted values and the expected values are no more than 20 seconds for all the tested bugs.

We also list the diagnosis time for each bug in Table <ref>, which is the total execution time for all the fixing components of TFix$^+$. The diagnosis time is within ten seconds, which makes it practical to apply TFix$^+$ in real-world systems. The runtime overhead comes from three tracing modules, i.e., system call tracing, function call tracing and system metric tracing. As discussed in our previous work [17], function call tracing incurs low overhead because TFix$^+$ only traces a small number of functions (functions related to timeout configuration, network connection and synchronization). Our prototype implementation experiments show that TFix$^+$ incurs less than 3% runtime overhead to the monitored system.

§.§.§ Negative Case Study

As shown in Table <ref>, Flume-1819 bug cannot be fixed by TFix$^+$ since not all the test suites are passed after adopting TFix$^+$'s patch. Flume-1819 bug is triggered when the system hangs on an infinite loop to read the data into channels. To fix the bug, a timeout mechanism is needed to flush the cache data into disk when the bug is triggered. Otherwise, the system experiences data loss.
TFix$^+$ can partially fix the bug by enabling the system to jump out of the infinite loop. However, in order to fix the bug completely, we need to trigger cache data flushing, which requires application specific knowledge.
In addition, TFix$^+$ provides a proper value for the timeout mechanism, which complements the manual patch. As shown in Table <ref>, the default value of the manual patch is not applicable under current workload. The expected timeout value (i.e., 30 seconds) is significantly larger than the one (i.e., 3 seconds) in the manual patch.

§ LIMITATION

TFix$^+$ fixes both misused timeout bugs and missing timeout bugs which cover 78% timeout bugs in total. When fixing misused timeout bugs, TFix$^+$ can localize the misused timeout variable if the cloud system uses timeout variables in timeout handling operations. However, we observe that some timeout bugs are caused by hard-coded timeout values. The hard-coded cases are discussed in Section <ref>. Although TFix$^+$ cannot localize misused timeout variables, TFix$^+$ can correctly identify them as the misused timeout bugs and pinpoint the timeout affected functions. TFix$^+$ provides useful hints to localize the hard-coded values.

TFix$^+$ currently cannot cover the other three categories of timeout bugs, i.e., improper handling, unnecessary timeout, and clock drifting. For the bugs caused by improper handling, we need to add or revise application-specific functions to fix them.
For the bugs caused by unnecessary timeout, TFix$^+$ can identify the timeout affected functions when the bugs cause performance issues.

However, whether removing the unnecessary timeout alters the control flow and brings any unwanted effects is out of the scope of this paper.
For the bugs caused by clock drifting, the fixing strategy is to enhance the synchronization of distributed systems, rather than to modify any timeout mechanisms.

TFix$^+$ currently only supports cloud systems written in Java since the implemation of fixing misused and missing timeout bugs only works on Java platforms now.

However, our approach is agnostic to programming languages. TFix$^+$ can be easily extended to support other programming languages by replacing Java specific Dapper, static taint analysis, timeout mechanism instrumentation components with other programming language counterparts.

§ RELATED WORK

Automatic bug fixing.
Work has also been done to automatically fix different bugs.
For example,
AFix [22] and CFix [23] proposed automatic patching strategies for concurrency bugs.

ClearView [27] identified violated invariants from erroneous executions and generated candidate repair patches to change the invariants.
Tian et al. [30] presented an automatic bug fixing patch identification tool to maintain older stable versions.
Tufano et al. [31] applied an Encoder-Decoder model based on neural networks to mine the existing patches and automatically generate new ones.
Narya [24] predicted a host failure by adopting domain knowledge and machine learning models and took appropriate actions to mitigate failures.
In comparison, our work fixes timeout bugs with the drill-down bug analysis approach that can not only identify bug root causes but also apply proper timeout fixing strategies.

Our previous work HangFix [18] focused on fixing hang bugs caused by infinite loops and blocking function calls.
HangFix proposed to resolve some hang problems by inserting a timeout mechanism with a pre-set timeout variable. In contrast, TFix$^+$ focuses on fixing both misued and missing timeout bugs which could cause performance slowdown, job failures, or software hang.
In HangFix, the newly introduced timeout variable is heuristically configured, thus cannot handle dynamic workloads.
TFix$^+$ provides the prediction-driven timeout variable configuration scheme which can adapt to user runtime environment and dynamic workloads.
TFix [17] is the early version of TFix$^+$, which focuses on only fixing misused timeout bugs covering 47% timeout bugs. In contrast, TFix$^+$ fixes both misused and missing timeout bugs that cover 78% timeout bugs. We have conducted extensive empirical study on the fixing coverage of TFix$^+$. Moreover, TFix$^+$ provides a new prediction-driven approach to achieving self-configuring timeout bug fixing.

Tracing-based bug detection and diagnosis.
Previous work has extensively used tracing techniques to detect and diagnose bugs.
For example, X-ray [7] dynamically instrumented binaries to trace the inputs and outputs of different components and inferred the traces to diagnose performance bugs.

Chopstix [10] collected low-level OS events such as CPU utilization and I/O operations online and reconstructed these events offline for troubleshooting intermittent and non-reproducible bugs.

REPT [11] utilized hardware traces to reconstruct the program's execution and employed record-and-replay techniques for debugging.
Magpie [9] instrumented middleware and packet transfer
points to record fine-grained system events and correlated
these events to capture the control-flow and resource consumption of each request for debugging.
CrashTuner [25] detected crash-recovery bugs by injecting faults in the test suites and exposing the bugs via log-based program analysis.
Gong et al. [14] conducted a study on applying machine learning techniques to detecting malwares from a major Android App market.
Different from TFix$^+$, these tracing techniques do not apply for timeout bug diagnosis or fixing.

Tracing techniques have also been used by previous work to detect timeout bugs specifically.

For example,
TScope [16] detected timeout bugs using timeout related feature selection and machine-learning based anomaly detection on system call traces.

SafeTimer [26] checked whether a timeout bug is caused by network delay or packet delay at the OS level.

Those tracing-based timeout detection techniques are complementary to and can be used with TFix$^+$ as an automated system for timeout detection, diagnosis and fixing. In our experiment, we use TScope as the front-end to identify whether an anomaly is caused by timeout.

Configuration bug detection and diagnosis.

Different techniques have been used to detect and diagnose misconfiguration issues.

Violet [19] adopted symbolic execution to detect specious configurations.
Ctests [29] generated new test cases based on original test suites to detect errors caused by configuration changes.
SPEX [33] studied configuration constraints and exposed potential configuration errors by injecting errors that violate the constraints.
ConfValley [21] introduced a new language to define system validation rules and checked configurations against those rules before the application was deployed in production.
PCheck [32] analyzed the application source code and automatically emulated the late execution that used configuration values to detect latent configuration errors.
CODE [35] detected configuration bugs by identifying the abnormal program executions using invariant configuration access rules.
ConfAid [8] adopted dynamic taint tracking methods to instrument the binary application and analyzed the information flow to pinpoint the root causes of configuration errors.
ConfDiagnoser [37] extracted the control flow of configuration options, instrumented the application code for profiling and analyzed the configuration deviation to detect the erroneous configuration options.
EnCore [36] applied machine learning techniques to model the correlation between the configuration settings and the executing environment and correlations between configuration entries, in order to learn and detect configuration bugs.
In summary, existing approaches focused on detecting misconfiguration issues that caused functional bugs. They cannot be readily applied to detecting performance issues caused by misconfigured variables that are triggered during system runtime due to input data or environment changes. Moreover, existing tools cannot recommend applicable configurations to fix the bugs. In comparison, TFix$^+$ adopts the prediction-driven scheme to suggest proper timeout variable configurations.

§ CONCLUSION

In this paper, we have presented TFix$^+$, a self-configuring hybrid timeout bug fixing system. TFix$^+$ proposes two hybrid schemes to fix misused and missing timeout bugs, respectively.
TFix$^+$ adopts a prediction-driven timeout configuration recommendation scheme based on runtime tracing and static taint analysis to provide end-to-end bug fixing.
Our empirical study shows TFix$^+$ can fix 79% of 91 real world misused or missing timeout bugs.
The experimental results on 16 reproduced timeout bugs show that TFix$^+$ can produce effective fixes for 15 out of 16 tested timeout bugs.

§ ACKNOWLEDGMENTS

§ ACKNOWLEDGMENT

This research is sponsored in part by NSF CNS1513942 grant, and NSF CNS1149445 grant.
Any opinions expressed in this paper are those of the authors and do not necessarily reflect the views of NSF.

[2]
Checker framework.

[3]
Hadoop test suites.

[4]
Irreversible Failures: Lessons from the DynamoDB Outage.

[5]

[6]
Soot: A framework for analyzing and transforming java and android applications.

[7]
M. Attariyan, M. Chow, and J. Flinn.
X-ray: Automating root-cause diagnosis of performance anomalies in
production software.
In OSDI, 2012.

[8]
M. Attariyan and J. Flinn.
Automating configuration troubleshooting with dynamic information
flow analysis.
In OSDI, 2010.

[9]
P. Barham, A. Donnelly, R. Isaacs, and R. Mortier.
Using magpie for request extraction and workload modelling.
In OSDI, 2004.

[10]
S. Bhatia, A. Kumar, M. E. Fiuczynski, and L. L. Peterson.
Lightweight, high-resolution monitoring for troubleshooting
production systems.
In OSDI, 2008.

[11]
W. Cui, X. Ge, B. Kasikci, B. Niu, U. Sharma, R. Wang, and I. Yun.
Rept: Reverse debugging of failures in deployed software.
In OSDI, 2018.

[12]
T. Dai, J. He, X. Gu, and S. Lu.
Understanding real world timeout problems in cloud server systems.
In IC2E, 2018.

[13]
M. Desnoyers and M. R. Dagenais.
The lttng tracer: A low impact performance and behavior monitor for
In OLS (Ottawa Linux Symposium). Citeseer, 2006.

[14]
L. Gong, Z. Li, F. Qian, Z. Zhang, Q. A. Chen, Z. Qian, H. Lin, and Y. Liu.
Experiences of landing machine learning onto market-scale mobile
malware detection.
In EuroSys, 2020.

[15]
H. S. Gunawi, M. Hao, T. Leesatapornwongsa, T. Patana-anake, T. Do,
J. Adityatama, K. J. Eliazar, and et al.
What bugs live in the cloud? a study of 3000+ issues in cloud
In SoCC, 2014.

[16]
J. He, T. Dai, and X. Gu.
Tscope: Automatic timeout bug identification for server systems.
In ICAC, 2018.

[17]
J. He, T. Dai, and X. Gu.
Tfix: Automatic timeout bug fixing in production server systems.
In ICDCS, 2019.

[18]
J. He, T. Dai, X. Gu, and G. Jin.
Hangfix: automatically fixing software hang bugs for production cloud
In SoCC, 2020.

[19]
Y. Hu, G. Huang, and P. Huang.
Automated reasoning and detection of specious configuration in large
systems with symbolic execution.
In OSDI, 2020.

[20]
J. Huang, X. Zhang, and K. Schwan.
Understanding issue correlations: a case study of the hadoop system.
In SoCC, 2015.

[21]
P. Huang, W. J. Bolosky, A. Singh, and Y. Zhou.
Confvalley: a systematic configuration validation framework for cloud
In EuroSys, 2015.

[22]
G. Jin, L. Song, W. Zhang, S. Lu, and B. Liblit.
Automated atomicity-violation fixing.
In PLDI, 2011.

[23]
G. Jin, W. Zhang, D. Deng, B. Liblit, and S. Lu.
Automated concurrency-bug fixing.
In OSDI, 2012.

[24]
S. Levy, R. Yao, Y. Wu, Y. Dang, P. Huang, Z. Mu, P. Zhao, and et al.
Predictive and adaptive failure mitigation to avert production cloud
vm interruptions.
In OSDI, 2020.

[25]
J. Lu, C. Liu, L. Li, X. Feng, F. Tan, J. Yang, and L. You.
Crashtuner: detecting crash-recovery bugs in cloud systems via
meta-info analysis.
In SOSP, 2019.

[26]
S. Ma and Y. Wang.
Accurate timeout detection despite arbitrary processing delays.
In USENIX ATC, 2018.

[27]
J. H. Perkins, S. Kim, S. Larsen, S. Amarasinghe, J. Bachrach, M. Carbin,
C. Pacheco, and et al.
Automatically patching errors in deployed software.
In SOSP, 2009.

[28]
B. H. Sigelman, L. A. Barroso, M. Burrows, P. Stephenson, M. Plakal, D. Beaver,
S. Jaspan, and C. Shanbhag.
Dapper, a large-scale distributed systems tracing infrastructure.
Technical report, Technical report, Google Inc, 2010.

[29]
X. Sun, R. Cheng, J. Chen, E. Ang, O. Legunsen, and T. Xu.
Testing configuration changes in context to prevent production
In OSDI, 2020.

[30]
Y. Tian, J. Lawall, and D. Lo.
Identifying linux bug fixing patches.
In ICSE, 2012.

[31]
M. Tufano, C. Watson, G. Bavota, M. Di Penta, M. White, and D. Poshyvanyk.
An empirical investigation into learning bug-fixing patches in the
wild via neural machine translation.
In ASE, 2018.

[32]
T. Xu, X. Jin, P. Huang, Y. Zhou, S. Lu, L. Jin, and S. Pasupathy.
Early detection of configuration errors to reduce failure damage.
In OSDI, 2016.

[33]
T. Xu, J. Zhang, P. Huang, J. Zheng, T. Sheng, D. Yuan, Y. Zhou, and
S. Pasupathy.
Do not blame users for misconfigurations.
In SOSP, 2013.

[34]
T. Xu and Y. Zhou.
Systems approaches to tackling configuration errors: A survey.
ACM Computing Surveys (CSUR), 47(4):1–41, 2015.

[35]
D. Yuan, Y. Xie, R. Panigrahy, J. Yang, C. Verbowski, and A. Kumar.
Context-based online configuration-error detection.
In USENIX ATC, 2011.

[36]
J. Zhang, L. Renganarayana, X. Zhang, N. Ge, V. Bala, T. Xu, and Y. Zhou.
Encore: Exploiting system environment and correlation information for
misconfiguration detection.
In ASPLOS, 2014.

[37]
S. Zhang and M. D. Ernst.
Automated diagnosis of software configuration errors.
In ICSE, 2013.

[]Jingzhu He
is an assistant professor in the School of Information Science and Technology at the ShanghaiTech University. She received her PhD degree in 2021 from the Department of Computer Science at the North Carolina State University. She received an MPhil degree in computer science from Hong Kong Baptist University in 2016, and her BS degree in electronic and information science from Nanjing University, China in 2013.

[]Ting Dai
is a research staff member at IBM Research. He received his PhD degree in 2019 from the Department of Computer Science at North Carolina State University. He received a BS in information security and MS in computer software and theory from Nanjing University of Posts and Telecommunications, China in 2011 and 2014 respectively. He has interned with IBM Research in the summer of 2018 and InsightFinder Inc. in the summer of 2016. Ting is a member of the IEEE.

[]Xiaohui Gu
is a professor in the Department of Computer Science at the North Carolina State University. She received her PhD degree in 2004 and MS degree in 2001 from the Department of Computer Science, University of Illinois at Urbana-Champaign. She received her BS degree in computer science from Peking University, Beijing, China in 1999. She was a research staff member at IBM T. J. Watson Research Center, Hawthorne, New York, between 2004 and 2007. She received ILLIAC fellowship, David J. Kuck Best Master Thesis Award, and Saburo Muroga Fellowship from University of Illinois at Urbana-Champaign. She also received the IBM Invention Achievement Awards in 2004, 2006, and 2007.  She has filed nine patents, and has published more than 60 research papers in international journals and major peer-reviewed conference proceedings. She is a recipient of NSF Career Award, four IBM Faculty Awards 2008, 2009, 2010, 2011, and two Google Research Awards 2009, 2011, best paper awards from ICDCS 2012 and CNSM 2010, and NCSU Faculty Research and Professional Development Award. She is a Senior Member of IEEE.