parameters), the longer it takes to optimize PPO surrogate loss Schulman et
al. (2017).
Statistical analysis: Statistically, the difference between both generalist
agents is not significant in terms of makespan values on the 1% and 2% data
budgets (see Table 25). On the zero-shot, the MGDT agents perform best with
CLES values equal to 1. In terms of training time the IMPALA configurations
perform significantly better than the other training methods in the ($6\times
6$) instance (as reported in Table 26) on the 1% and 2% data budgets. In terms
of testing time, IMPALA configurations perform significantly better than MGDT
configurations on the ($6\times 6$) instance. In terms of cumulative reward,
the two generalist agents do not show significantly different performance
against each other on all fine-tuning data budgets.

On the $30\times 20$ instance of the PDR task, on average, all IMPALA agents
perform similarly in terms of makespan, and cumulative reward earned on all
fine-tuning data budgets. Similarly, each of the MGDT agents has the same
average performance when applied to the $30\times 20$ instance of the PDR task
in terms of makespan and cumulative reward earned. MGDT-PPO has the longest
training time on the 1% and 2% fine-tuning data budgets. IMPALA-V_TRACE and
IMPALA-PPO have the longest testing time.
Statistical analysis: Statistically, the difference between both generalist
agents is not significant in terms of makespan values on the zero-shot and 2%
fine-tuning (see Table 29). On the 1% fine-tuning data budgets, in terms of
makespan, MGDT-PPO performs worse than MGDT-MAENT with CLES values close to 0.
In terms of training time, the IMPALA configurations perform significantly
better than the other training methods in the ($30\times 20$) instance (Table
30) on the 1% and 2% data budgets. In terms of testing time, IMPALA
configurations perform worse than MGDT on the ($30\times 20$) instance. In
terms of cumulative reward, the observed differences between the two
generalist agents on all fine-tuning data budgets are not significant.

Finding 6: Both generalist agents easily adapt to the PDR task on both small
and medium instances. In this case, pre-training on other tasks (compared to
training from scratch) indeed helps with rapid adaptation to the new task
(PDR). Further, in the case of PDR tasks, the feature extractor is a GNN,
suggesting that feature extraction representation is important for good
performance transfer.

Regarding MsPacman game, among the model-free algorithms, IMPALA agents
perform best on average in terms of bugs detected and testing time at all
fine-tuning data budgets.In terms of training time, at the 1% fine-tuning data
budget, the MGDT agent configurations performed on average the lowest fine-
tuning time. In terms of reward earned, IMPALA-V_TRACE performs best on
average at the zero-shot fine-tuning data budget.

Finding 7: IMPALA agent configurations achieved the lowest testing time across
all fine-tuning data budgets on the MsPacman game. The IMPALA agent takes
advantage of the distributed nature of its architecture as well as multiple
actors stepping into the game environment increasing the chance of detecting
bugs.

Statistical analysis: Tables 33, 34, 35, and 38 show the results of the post-
hoc test analysis of model-free algorithms regarding their performance on the
MsPacman game.

Table 38: Results of post-hoc tests analysis of cumulative reward performance
by each model-free algorithm on the MsPacman game (in bold are DRL
configurations where p-value is $<$ 0.05 and have greater performance w.r.t
the effect size).

Data budgets | A | B | mean(A) | mean(B) | pval | CLES
---|---|---|---|---|---|---
zero-shot | IMPALA-PPO | IMPALA-V_TRACE | 8.84e+1 | 1.96e+2 | 1.79e-2 | 3.69e-3
IMPALA-PPO | MGDT-DQN | 8.84e+1 | 1.52e+2 | 8.90e-6 | 6.40e-56
IMPALA-PPO | MGDT-MAENT | 8.84e+1 | 1.93e+2 | 3.48e-2 | 1.34e-2
IMPALA-PPO | MGDT-PPO | 8.84e+1 | 4.50e+1 | 1.32e-7 | 1.00e+0
IMPALA-V_TRACE | MGDT-DQN | 1.96e+2 | 1.52e+2 | 2.60e-1 | 8.65e-1
IMPALA-V_TRACE | MGDT-MAENT | 1.96e+2 | 1.93e+2 | 1.00e+0 | 5.21e-1
IMPALA-V_TRACE | MGDT-PPO | 1.96e+2 | 4.50e+1 | 5.16e-3 | 1.00e+0
MGDT-DQN | MGDT-MAENT | 1.52e+2 | 1.93e+2 | 4.17e-1 | 1.91e-1
MGDT-DQN | MGDT-PPO | 1.52e+2 | 4.50e+1 | 2.16e-6 | 1.00e+0
MGDT-MAENT | MGDT-PPO | 1.93e+2 | 4.50e+1 | 1.01e-2 | 9.99e-1
1% | IMPALA-PPO | IMPALA-V_TRACE | 8.84e+1 | 8.79e+1 | 9.27e-1 | 6.38e-1
IMPALA-PPO | MGDT-DQN | 8.84e+1 | 1.51e+2 | 5.86e-7 | 5.21e-90
IMPALA-PPO | MGDT-MAENT | 8.84e+1 | 1.69e+2 | 2.42e-2 | 6.98e-3
IMPALA-PPO | MGDT-PPO | 8.84e+1 | 4.50e+1 | 3.86e-7 | 1.00e+0
IMPALA-V_TRACE | MGDT-DQN | 8.79e+1 | 1.51e+2 | 4.79e-7 | 9.54e-91
IMPALA-V_TRACE | MGDT-MAENT | 8.79e+1 | 1.69e+2 | 2.37e-2 | 6.67e-3
IMPALA-V_TRACE | MGDT-PPO | 8.79e+1 | 4.50e+1 | 4.69e-7 | 1.00e+0
MGDT-DQN | MGDT-MAENT | 1.51e+2 | 1.69e+2 | 7.39e-1 | 2.91e-1
MGDT-DQN | MGDT-PPO | 1.51e+2 | 4.50e+1 | 6.84e-7 | 1.00e+0
MGDT-MAENT | MGDT-PPO | 1.69e+2 | 4.50e+1 | 5.05e-3 | 1.00e+0
2% | IMPALA-PPO | IMPALA-V_TRACE | 1.82e+2 | 8.90e+1 | 2.00e-3 | 1.00e+0
IMPALA-PPO | MGDT-DQN | 1.82e+2 | 1.50e+2 | 9.48e-2 | 9.46e-1
IMPALA-PPO | MGDT-MAENT | 1.82e+2 | 2.00e+2 | 9.32e-1 | 3.67e-1
IMPALA-PPO | MGDT-PPO | 1.82e+2 | 4.50e+1 | 4.77e-4 | 1.00e+0
IMPALA-V_TRACE | MGDT-DQN | 8.90e+1 | 1.50e+2 | 4.14e-10 | 1.00e-93
IMPALA-V_TRACE | MGDT-MAENT | 8.90e+1 | 2.00e+2 | 3.13e-2 | 1.13e-2
IMPALA-V_TRACE | MGDT-PPO | 8.90e+1 | 4.50e+1 | 5.71e-6 | 1.00e+0
MGDT-DQN | MGDT-MAENT | 1.50e+2 | 2.00e+2 | 3.08e-1 | 1.53e-1
MGDT-DQN | MGDT-PPO | 1.50e+2 | 4.50e+1 | 1.98e-7 | 1.00e+0
MGDT-MAENT | MGDT-PPO | 2.00e+2 | 4.50e+1 | 9.52e-3 | 9.99e-1

  * •

mean(A) and mean(B) refer to cumulative reward values.

Statistically, among the model-free DRL algorithms, IMPALA agent
configurations perform best with a CLE value of 100% in terms of detected
bugs. MGDT agent configurations achieve the lowest training time compared to
the IMPALA agent configurations with CLE values close to 100% at the 1% fine-
tuning data budget (Table 34). IMPALA agent configurations have the lowest
testing time in comparison to MGDT agent configurations with CLE values close
to 0% at all fine-tuning data budgets (Table 35). In terms of cumulative
reward earned, among the MGDT agent configurations, MGDT-DQN and MGDT-MAENT
perform best at all fine-tuning data budgets with CLE values of 100% (Table
38). Among the IMPALA agent configurations, IMPALA-PPO performs worst in terms
of cumulative reward earned at zero-shot fine-tuning data budget with CLE
value close to 0%, nevertheless, it performs best at 2% fine-tuning data
budget. The PPO algorithm features a smaller number of random actions during
training, thanks to its update rule. As a result, it leverages rewards that
have already been discovered. This characteristic means that the PPO
algorithm’s performance increases or decreases in terms of accumulated reward
throughout training, based on past data. Furthermore, this property explains
its improved reward performance over IMPALA-V_TRACE at 2% fine-tuning data
budget, given that IMPALA-PPO achieves positive rewards at zero-shot and 1%
fine-tuning data budgets.

Finding 8: IMPALA-V_TRACE shows a statistically significant improved
performance in comparison to the other model-free DRL algorithms in finding
bugs in the MsPacman game. Our results also provide insights regarding the
strengths and weaknesses of different model-free DRL algorithms on the
MsPacman game. The choice of the model-free DRL algorithm depends on specific
metrics such as bug detection, training efficiency or cumulative reward
optimization. IMPALA-PPO outperforms IMPALA-V_TRACE at the 2% fine-tuning data
budget in terms of cumulative reward obtained, given its ability to achieve
positive rewards at zero-shot and 1% fine-tuning data budgets.

##  5 Recommendations about the selection of generalist agents

In this section, we discuss our recommendations regarding selecting generalist
agents by researchers and practitioners.

The exploration capability of the MGDT generalist agent has led it to detect
more bugs in part of the Blockmaze game environment where IMPALA could not,
across all fine-tuning data budgets.

Recommendation 1: For exploration-intensive tasks, we recommend generalist
agents that offer sequence modeling representation of data for rapid
adaptation and exploration of the unknown in an environment.

MGDT and IMPALA generalist agents have shown good performance on the studied
PDR-based scheduling in both evaluated scheduling instances across all fine-
tuning data budgets. Compared to the Blockmaze and MsPacman games, the PDR-
based scheduling solves the problem of sparse rewards by often rewarding the
agent according to its makespan during an episode.

Recommendation 2: We recommend using generalist agents on tasks where the
distribution of rewards is dense.

The scalability property of IMPALA has led it to outperform MGDT on the
MsPacman game. Moreover, in the MsPacman game, we evaluated IMPALA bug
detection capability during a continuous learning mechanism and it achieved
the desired outcome.

Recommendation 3: We recommend using scalable generalist agents in a
continuous learning setting for data efficiency.

Our results also indicate that the performance of the generalist agents varies
among the model-free DRL algorithms used for fine-tuning. In the MsPacman
game, V_TRACE algorithm outperforms the classic model-free algorithm PPO.
Similarly, MAENT outperforms DQN and PPO.

Recommendation 4: When fine-tuning a generalist on a task, we recommend using
a model-free DRL algorithm adapted to the generalist agent’s architecture for
sample-efficient policy optimization.

Our results indicate that the generalist agents significantly outperform the
specialist ones on the studied PDR-based scheduling tasks across all fine-
tuning data budgets, by performing the lowest makespan time.

Recommendation 5: We recommend using generalist agents for solving scheduling
problems.

##  6 Related work

Recently, researchers have started using pre-trained deep learning models in
SE for code-related tasks. Wang et al. Wang et al. (2022) exploited natural
language pre-trained models in code-related tasks. Specifically, the natural
language models are first augmented with semantic-preserving transformation
sequences for pre-training purposes, then fine-tuned on two downstream tasks:
code clone detection and code search tasks. Kanade et al. Kanade et al. (2020)
proposed cuBERT a contextual embedding of source code derived from BERT Devlin
et al. (2018) model. CuBERT is pre-trained on a massive corpus of python codes
and then fine-tuned on downstream code-related tasks. The results show that by
pre-training with cuBERT, the fine-tuned models outperformed the baseline
model trained from scratch. Similarly, Feng et al. Feng et al. (2020) proposed
codeBERT, a BERT transformer-based model pre-trained on code from github and
fine-tuned for code search and documentation generation tasks. In this work,
similar to these previous studies, we adopt BERT as the model’s architecture
of the MGDT agent. Moreover, we pre-train our generalist agents to leverage
prior skills during fine-tuning.

DRL algorithms have been leveraged by researchers to improve the performance
of SE tasks Singh and Sharma (2013); Bahrpeyma et al. (2015); Chen et al.
(2020); Vuong and Takada (2018). Zheng et al. Zheng et al. (2019), Tufano et
al. Tufano et al. (2022) our baselines studies, trained specialist agents to
detect game bugs. Bagherzadeh et al. Bagherzadeh et al. (2021) leveraged DRL
algorithms in continuous improvement regression testing. They investigated
ranking models as DRL problems to find the optimal prioritized test cases.
Zhang et al. Zhang et al. (2020) another one of our baseline studies, trained
a specialist agent to solve the JSSP on scheduling instances. These studies
are similar to our work as we examine specialist agents’ performance on SE
tasks. However, we compare specialist agents’ performance against pre-trained
generalist agents’ performance.

Researchers have developed DRL generalist agents, to solve multiple tasks
simultaneously. Mendonca et al. Mendonca et al. (2021) proposed Latent
Explorer Achiever (LEXA), a DRL-based agent capable of learning general skills
in an environment where some states are estimated based on the uncertainty of
the agent policy. LEXA is evaluated on robotic locomotion and manipulation
tasks and outperforms prior studies. Similarly, Kalashnikov et al. Kalashnikov
et al. (2021) proposed a multi-robot learning system where multiple robots can
learn a variety of tasks. The authors show that their proposed approach
enables the robots to generalize to never-before-seen tasks and achieve high
performance. Other prior works include Lee et al. Lee et al. (2022), Espeholt
et al. Espeholt et al. (2018), our baseline studies where authors developed
generalist agents trained on the Atari suite environments and capable of
achieving performance close to or better than the ones of specialist agents.
In this paper, we leverage generalist agents on SE tasks, which none of these
studies had explored.

##  7 Threats to validity

Conclusion validity. The conclusion’s limitations concern the degree of
accuracy of the statistical findings on the best-performing fine-tuned general
agents. We use Tukey and Welch’s ANOVA post-hoc tests as statistical tests.
The significance level is set to 0.05 which is standard across the literature
as shown by Welch et al. Welch (1947), Games et al. Games and Howell (1976).
The non-deterministic nature of DRL algorithms can also threaten the
conclusions made in this work. We address this by collecting results from 5
independent runs for all our experiments.

Internal validity. A potential limitation is the number of pre-trained
generalist agents used. We have chosen to evaluate only two generalist agents
because of the availability of their source code. Moreover, the goal of our
study was to show that generalist agents can be leveraged in SE tasks for
resource efficiency, thus using any other generalist agents does not
invalidate our findings.

Construct validity. A potential threat to construct validity stems from our
evaluation criteria. However, these criteria are standard in the literature.
We use these measures( the number of bugs detected, the average cumulative
reward, the makespan, the training and testing time) to make a fair comparison
between the generalist agents in identical circumstances. We discussed how
they should be interpreted in Section 3.2.8

External validity. Since our goal is to investigate whether or not pre-trained
generalist agents can be leveraged for SE tasks, a potential limitation is the
choice of the tasks used to evaluate the pre-trained generalist agents. We
address this threat by choosing bug detection and scheduling tasks that are
two different types of SE tasks, in order to achieve diversity. While in
scheduling we aim to minimize the makespan, in the bug detection task we aim
to find bugs in a game as early as possible.

Reliability validity. To allow other researchers to replicate or build on our
research, we provide a detailed replication package rep (2023) including the
code and obtained results.

##  8 Conclusion and Discussions

In this paper, we investigated the applicability of pre-trained generalist
agents to two important SE tasks: the detection of bugs in two games and the
minimization of makespan on two instances of the task scheduling problem. We
compare the efficiency of pre-trained generalist agents against that of three
baseline approaches in terms of (i) the detection of bugs, and (ii) the
minimization of the makespan of scheduling instances. Our results show that
the fine-tuned generalist agents can perform close to or better than the
specialist agents on some SE tasks. Our results also highlight the
difficulties of some model-free DRL algorithms in handling the complexity of
generalist agents, which suggests further investigation. We formulate
recommendations to help SE practitioners make an informed decision when
leveraging generalist agents to develop SE tasks. Our experiments also show
that generalist agents achieve good transfer performance on the PDR-based
scheduling on both evaluated scheduling instances, which have the higher
action space sets. Recall that the generalist agents were pre-trained on the
Atari suite environments (Section 3.2.1 and Section 3.2.2) with the full Atari
action set consisting of 18 actions. This makes us wonder whether, for good
transfer learning performance, the full action set of the generalist agents
has to be reached. In the future, we plan to investigate this in more detail.
Moreover, we plan to expand our study to investigate larger models and more SE
tasks. We also plan to use some SE tasks to pre-train the generalist agents.

##  9 Conflict of interest

The authors declared that they have no conflict of interest.

## References

  * Car (2016) (2016) Cartpole. https://gym.openai.com/envs/CartPole-v0/
  * MsP (2018) (2018) Mspacman. https://www.gymlibrary.dev/environments/atari/ms_pacman/
  * rep (2023) (2023) Replication package. https://github.com/npaulinastevia/Harnessing_generalist_agents_on_SE
  * Agarwal et al. (2020) Agarwal R, Schuurmans D, Norouzi M (2020) An optimistic perspective on offline reinforcement learning. In: International Conference on Machine Learning, PMLR, pp 104–114
  * Ahmadi et al. (2022) Ahmadi H, Ashtiani M, Azgomi MA, Saheb-Nassagh R (2022) A dqn-based agent for automatic software refactoring. Information and Software Technology 147:106893
  * Arcuri and Briand (2014) Arcuri A, Briand L (2014) A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing, Verification and Reliability 24(3):219–250
  * Argall et al. (2009) Argall BD, Chernova S, Veloso M, Browning B (2009) A survey of robot learning from demonstration. Robotics and autonomous systems 57(5):469–483
  * Arnab et al. (2021) Arnab A, Dehghani M, Heigold G, Sun C, Lučić M, Schmid C (2021) Vivit: A video vision transformer. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 6836–6846
  * Bagherzadeh et al. (2021) Bagherzadeh M, Kahani N, Briand L (2021) Reinforcement learning for test case prioritization. IEEE Transactions on Software Engineering
  * Bahrpeyma et al. (2015) Bahrpeyma F, Haghighi H, Zakerolhosseini A (2015) An adaptive rl based approach for dynamic resource provisioning in cloud virtualized data centers. Computing 97(12):1209–1234
  * Bellemare et al. (2013) Bellemare MG, Naddaf Y, Veness J, Bowling M (2013) The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research 47:253–279
  * Blazewicz et al. (2000) Blazewicz J, Pesch E, Sterna M (2000) The disjunctive graph machine representation of the job shop scheduling problem. European Journal of Operational Research 127(2):317–331
  * Brockman et al. (2016) Brockman G, Cheung V, Pettersson L, Schneider J, Schulman J, Tang J, Zaremba W (2016) Openai gym. arXiv:1606.01540
  * Brown (2005) Brown AM (2005) A new software for carrying out one-way anova post hoc tests. Computer methods and programs in biomedicine 79(1):89–95
  * Chen et al. (2020) Chen J, Ma H, Zhang L (2020) Enhanced compiler bug isolation via memoized search. In: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, pp 78–89
  * Chen et al. (2021) Chen L, Lu K, Rajeswaran A, Lee K, Grover A, Laskin M, Abbeel P, Srinivas A, Mordatch I (2021) Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems 34:15084–15097
  * Devlin et al. (2018) Devlin J, Chang MW, Lee K, Toutanova K (2018) Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:181004805
  * Espeholt et al. (2018) Espeholt L, Soyer H, Munos R, Simonyan K, Mnih V, Ward T, Doron Y, Firoiu V, Harley T, Dunning I, et al. (2018) Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In: International conference on machine learning, PMLR, pp 1407–1416
  * Feng et al. (2020) Feng Z, Guo D, Tang D, Duan N, Feng X, Gong M, Shou L, Qin B, Liu T, Jiang D, et al. (2020) Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:200208155
  * Games and Howell (1976) Games PA, Howell JF (1976) Pairwise multiple comparison procedures with unequal n’s and/or variances: a monte carlo study. Journal of Educational Statistics 1(2):113–125
  * Gao et al. (2007) Gao J, Gen M, Sun L, Zhao X (2007) A hybrid of genetic algorithm and bottleneck shifting for multiobjective flexible job shop scheduling problems. Computers & Industrial Engineering 53(1):149–162
  * Gromicho et al. (2012) Gromicho JA, Van Hoorn JJ, Saldanha-da Gama F, Timmer GT (2012) Solving the job-shop scheduling problem optimally by dynamic programming. Computers & Operations Research 39(12):2968–2977
  * Guo et al. (2018) Guo S, Liu J, Yang Y, Xiao B, Li Z (2018) Energy-efficient dynamic computation offloading and cooperative task scheduling in mobile cloud computing. IEEE Transactions on Mobile Computing 18(2):319–333
  * Harman et al. (2015) Harman M, Jia Y, Zhang Y (2015) Achievements, open problems and challenges for search based software testing. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), IEEE, pp 1–12
  * Hochreiter and Schmidhuber (1997) Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural computation 9(8):1735–1780
  * Kalashnikov et al. (2021) Kalashnikov D, Varley J, Chebotar Y, Swanson B, Jonschkowski R, Finn C, Levine S, Hausman K (2021) Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:210408212
  * Kan (2012) Kan AR (2012) Machine scheduling problems: classification, complexity and computations. Springer Science & Business Media
  * Kanade et al. (2020) Kanade A, Maniatis P, Balakrishnan G, Shi K (2020) Learning and evaluating contextual embedding of source code. In: International conference on machine learning, PMLR, pp 5110–5121
  * Kaur and Verma (2012) Kaur S, Verma A (2012) An efficient approach to genetic algorithm for task scheduling in cloud computing environment. International Journal of Information Technology and Computer Science (IJITCS) 4(10):74–79
  * Lazaric (2012) Lazaric A (2012) Transfer in reinforcement learning: a framework and a survey. In: Reinforcement Learning: State-of-the-Art, Springer, pp 143–173
  * Lee et al. (2022) Lee KH, Nachum O, Yang MS, Lee L, Freeman D, Guadarrama S, Fischer I, Xu W, Jang E, Michalewski H, et al. (2022) Multi-game decision transformers. Advances in Neural Information Processing Systems 35:27921–27936
  * McCarthy et al. (2006) McCarthy J, Minsky ML, Rochester N, Shannon CE (2006) A proposal for the dartmouth summer research project on artificial intelligence, august 31, 1955\. AI magazine 27(4):12–12
  * McGraw and Wong (1992) McGraw KO, Wong SP (1992) A common language effect size statistic. Psychological bulletin 111(2):361
  * Mendonca et al. (2021) Mendonca R, Rybkin O, Daniilidis K, Hafner D, Pathak D (2021) Discovering and achieving goals via world models. Advances in Neural Information Processing Systems 34:24379–24391
  * Mnih et al. (2013) Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M (2013) Playing atari with deep reinforcement learning. arXiv preprint arXiv:13125602
  * Mnih et al. (2015) Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, et al. (2015) Human-level control through deep reinforcement learning. nature 518(7540):529–533
  * Mnih et al. (2016) Mnih V, Badia AP, Mirza M, Graves A, Lillicrap T, Harley T, Silver D, Kavukcuoglu K (2016) Asynchronous methods for deep reinforcement learning. In: International conference on machine learning, PMLR, pp 1928–1937
  * Munro (2005) Munro MJ (2005) Product metrics for automatic identification of” bad smell” design problems in java source-code. In: 11th IEEE International Software Metrics Symposium (METRICS’05), IEEE, pp 15–15
  * Nouwou Mindom et al. (2023) Nouwou Mindom PS, Nikanjam A, Khomh F (2023) A comparison of reinforcement learning frameworks for software testing tasks. Empirical Software Engineering 28(5):111
  * Pfau et al. (2017) Pfau J, Smeddinck JD, Malaka R (2017) Automated game testing with icarus: Intelligent completion of adventure riddles via unsupervised solving. In: Extended abstracts publication of the annual symposium on computer-human interaction in play, pp 153–164
  * Radford et al. (2018) Radford A, Narasimhan K, Salimans T, Sutskever I, et al. (2018) Improving language understanding by generative pre-training
  * Rohlfshagen et al. (2017) Rohlfshagen P, Liu J, Perez-Liebana D, Lucas SM (2017) Pac-man conquers academia: Two decades of research using a classic arcade game. IEEE Transactions on Games 10(3):233–256
  * Santos et al. (2018) Santos RES, Magalhães CVC, Capretz LF, Correia-Neto JS, da Silva FQB, Saher A (2018) Computer games are serious business and so is their quality: Particularities of software testing in game development from the perspective of practitioners. In: Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, Association for Computing Machinery, New York, NY, USA, ESEM ’18, DOI 10.1145/3239235.3268923, URL https://doi.org/10.1145/3239235.3268923
  * Schulman et al. (2017) Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:170706347
  * Singh and Sharma (2013) Singh L, Sharma DK (2013) An architecture for extracting information from hidden web databases using intelligent agent technology through reinforcement learning. In: 2013 IEEE conference on Information & Communication Technologies, IEEE, pp 292–297
  * Singh and Kahlon (2011) Singh S, Kahlon KS (2011) Effectiveness of encapsulation and object-oriented metrics to refactor code and identify error prone classes using bad smells. ACM SIGSOFT Software Engineering Notes 36(5):1–10
  * Taillard (1993) Taillard E (1993) Benchmarks for basic scheduling problems. european journal of operational research 64(2):278–285
  * Tong et al. (2020) Tong Z, Chen H, Deng X, Li K, Li K (2020) A scheduling scheme in the cloud computing environment using deep q-learning. Information Sciences 512:1170–1191
  * Tufano et al. (2022) Tufano R, Scalabrino S, Pascarella L, Aghajani E, Oliveto R, Bavota G (2022) Using reinforcement learning for load testing of video games. In: Proceedings of the 44th International Conference on Software Engineering, pp 2303–2314
  * Vaswani et al. (2017) Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Advances in neural information processing systems 30
  * Vuong and Takada (2018) Vuong TAT, Takada S (2018) A reinforcement learning based approach to automated testing of android applications. In: Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation, pp 31–37
  * Wang et al. (2022) Wang D, Jia Z, Li S, Yu Y, Xiong Y, Dong W, Liao X (2022) Bridging pre-trained models and downstream tasks for source code understanding. In: Proceedings of the 44th International Conference on Software Engineering, pp 287–298
  * Wang et al. (2019) Wang H, Zheng S, Xiong C, Socher R (2019) On the generalization gap in reparameterizable reinforcement learning. In: International Conference on Machine Learning, PMLR, pp 6648–6658
  * Welch (1947) Welch BL (1947) The generalization of ‘student’s’problem when several different population varlances are involved. Biometrika 34(1-2):28–35
  * Zhang et al. (2008) Zhang C, Rao Y, Li P (2008) An effective hybrid genetic algorithm for the job shop scheduling problem. The International Journal of Advanced Manufacturing Technology 39:965–974
  * Zhang et al. (2020) Zhang C, Song W, Cao Z, Zhang J, Tan PS, Chi X (2020) Learning to dispatch for job shop scheduling via deep reinforcement learning. Advances in Neural Information Processing Systems 33:1621–1632
  * Zheng et al. (2022) Zheng Q, Zhang A, Grover A (2022) Online decision transformer. In: International Conference on Machine Learning, PMLR, pp 27042–27059
  * Zheng et al. (2019) Zheng Y, Xie X, Su T, Ma L, Hao J, Meng Z, Liu Y, Shen R, Chen Y, Fan C (2019) Wuji: Automatic online combat game testing using evolutionary deep reinforcement learning. In: 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), IEEE, pp 772–784
  * Zhu et al. (1997) Zhu H, Hall PA, May JH (1997) Software unit test coverage and adequacy. Acm computing surveys (csur) 29(4):366–427
  * Zhu et al. (2023) Zhu Z, Lin K, Jain AK, Zhou J (2023) Transfer learning in deep reinforcement learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence