# Evolutionary Self-Replication as a Mechanism for Producing Artificial
Intelligence

Samuel Schmidgall
U.S. Naval Research Laboratory
<EMAIL_ADDRESS>
Joe Hays
U.S. Naval Research Laboratory
<EMAIL_ADDRESS>
###### Abstract

Can reproduction alone in the context of survival produce intelligence in our
machines? In this work, self-replication is explored as a mechanism for the
emergence of intelligent behavior in robotic learning environments. By
focusing purely on survival, while undergoing natural selection, evolved
organisms are shown to produce meaningful, complex, and intelligent behavior,
demonstrating creative solutions to challenging problems without any notion of
reward or objectives. Atari and robotic learning environments are re-defined
in terms of natural selection, and the behavior which emerged in self-
replicating organisms during these experiments is described in detail.

##  1 Introduction

For decades it has been the primary strategy of Artificial Intelligence (AI)
research to hand-craft rules, logic, architectures, and objectives for our
machines to closely follow, making it difficult for open-endedness to emerge
([1, 2, 3, 4, 5]). Major progress in AI has followed a pattern of lessening
manual design, and allowing the machine to discover its own solutions – to
explore. Perhaps evolution is successful precisely because there is so much
room for what is defined as fit through the lens of life and death. Why is it
that we are not harnessing this approach to bring life, creativity, and
intelligence to our machines?

Exploration seems to be the nature of innovation; in fact, unguided
exploration, independent of objectives has shown to be an effective means of
producing intelligence itself ([6, 7, 8, 9]). It is well known that objective-
based optimization can be deceptive, ironically preventing the optimal
solution according to the same objective being optimized from being reached
([10, 11, 12, 13]). Significant manual effort goes into the design of reward,
hyper-parameters, architectures, and learning algorithms, whereas,
realistically, such nuances could be discovered by an algorithm itself. The
paradigm of AI-generating algorithms (AI-GAs) argues that a more effective
method of producing AI is to introduce a process whereby the AI learns how to
design and improve itself ([14]). In reality, this is precisely the process
with which biological organisms have developed.

It is easy to neglect the idea that evolution emerges as a property of
reproducing organisms under the pressure of survival; as in, it may look as if
there is an overarching algorithm being applied to the organisms, whereas, it
is rather that the reproductive organisms themselves, under the pressure of
natural selection, undergo evolution. While this may seem like just a matter
of semantics, conceptualizing evolution in this way alters the way in which
algorithms seeking to harness the power of evolution are executed. Evolution
is not simply applied to organisms, rather, the organisms themselves are
designed in such a way that they evolve naturally. This begs the question, how
can evolution be implemented in this way to produce organisms which are
interesting and of use for us? This work answers that question by abstracting
the properties of biological self-replication and formulating modern learning
problems in terms of natural selection: life and death. In this way,
interesting behavior on a series of robotic learning problems, as well as
several Atari games, is observed to emerge based solely on life, death, and
replication in the absence of reward.

##  2 Related Work

The division between evolutionary computation and evolutionary self-
replication is important to recognize. Evolutionary computation operates as
follows: (1) generate an initial population of random individuals, (2) assign
each individual a fitness based on a defined reward or objective function, (3)
use the obtained fitness evaluations to perform selection or parameter
optimization, (4) create a new population by varying the previous population,
(5) repeat from 2. Whereas, evolutionary self-replication typically operate as
follows: (1) generate initial organism(s), (2) organisms must meet conditions
for survive or cease to exist, and (3) organisms which survive longer tend to
replicate more often.

From a biological perspective, evolutionary algorithms have been argued to
lack crucial components of the evolutionary process ([15]). One primary
difference is that the execution of evolutionary algorithms is centralized,
with synchronized manual selection periods in the form of generations, unlike
the parallel and decentralized process of natural evolution. There is also no
notion of spatial relationships between individuals, and population sizes are
kept constant through highly synchronous timing and organization, whereas
natural evolution is spatially organized, and population sizes vary according
to birth and death events, occasionally going extinct. Additionally,
evolutionary algorithms have no self-referential notion of life, death, or
reproduction – such processes are applied to individuals. Evolutionary self-
replicators more closely resemble the process of natural evolution. The
fundamental difference between evolutionary algorithms and evolutionary self-
replicators, is that evolutionary algorithm methods undergo a structured form
of artificial selection, whereas evolutionary self-replicators undergo natural
selection ([16, 17, 18]).

Evolutionary self-replicating programs have a rich history in the field of
Artificial Life (ALife) and Artificial Intelligence (AI). The first organisms
to undergo digital evolution were from the program Darwin, created in Bell
Labs in 1961. In the game Darwin, computer programs compete with each other by
trying to stop the execution of other programs. Further advancements includes
work on Tierra ([16, 19, 20]) where it was observed for the first time that
evolutionary self-replicating programs evolve meaningfully, and toward higher
complexity. Work in Tierra found that the emergence of tens of thousands of
unique programs is possible from a single ancestral organism. Avida extends
work in Tierra under a set of different dynamics ([21, 22, 23, 24]). Avida
experiments demonstrated a capability of digital organisms to evolve necessary
structure to solve logic puzzles in order to gain energy in the form of
computation time ([24]). These discoveries show that evolutionary self-
replicators are creative, surprising, and most of all, intelligent.

While these worlds are rich in potential and discovery, it is often the case
that we’d like our organisms to solve more concrete, applicable, or
interesting problems. Toward this end, we present three scenarios of
environment reformulation based on survival, foraging, and survival-foraging,
which address these expectations.

##  3 Evolutionary Self-Replication

###  3.1 An evolutionary self-replicator

An evolutionary self-replicator is introduced whose survival is defined by
maintaining an equilibrium and whose self-reproductive success is defined
probabilistically over the course of the replicator’s lifetime111To allow for
more descriptive commentary of the results in this section the implementation
and code is elsewhere discussed in significant depth at
https://github.com/AnonymousSubmission/AnonymousSubmission and in the
supplementary material.. Initially, a finite grid of empty 1-dimensional cells
are instantiated. From here, a "basic organism" emerges, from which the
specific qualities of this basic organism depends on the environment and
definition of the self-replicator; in the simplest case, a small neural
network which initially produces random behavior emerges. To retain the utmost
simplicity, the actual process of replication is abstracted222As in, for
simplicity, the process of replication is not evolved with the organism but is
a built in mechanism., and mutations are introduced through random noise. The
environment that the organism finds itself in defines survival, life, and
death, and to retain existence, the organism must be adapted for survival. If
the organism encounters death, then it ceases to exist and is removed from the
1-dimensional grid.

Algorithm 1 Evolutionary Self-Replication

$n\geq 2$

$\texttt{ReproductiveProb}>0$

$D_{Grid}\leftarrow n$ $\triangleright$ Grid dimensionality

$\texttt{Grid}\leftarrow\texttt{Replicator}[D_{grid}]$ $\triangleright$ An
empty $D_{grid}$-dimensional array of replicators

while True do

if GridEmpty(Grid) then

$\texttt{Grid}[\texttt{Random}(D_{Grid})]\leftarrow\texttt{Replicator}()$

$\texttt{OrganismOrder}\leftarrow\texttt{RandomList}(\texttt{Grid})$
$\triangleright$ Random organism ordering

for orgIndex in OrganismOrder do $\triangleright$ Iterate through random
ordering

$\texttt{DeathState}\leftarrow\texttt{EnvironmentInteract(orgIndex)}$

if Uniform(0, 1) $\leq$ ReproductiveProb then $\triangleright$ Reproduction

$\texttt{Neighbors}\leftarrow\texttt{FreeNeighbors}(\texttt{orgIndex, Grid})$

if Neighbors then

$\texttt{Grid}[\texttt{Random(Neighbors)}]\leftarrow\texttt{Mutate}(\texttt{Grid}[\texttt{orgIndex}])$

if DeathState then $\triangleright$ Death

RemoveFromGrid(orgIndex, Grid)

Replication on the 1-dimensional grid works as follows: a) at every
environment timestep each organism has a probability of replication, b) if an
organism successfully replicates, a mutated copy of itself is randomly
produced in either one of the two neighboring cells, c) if the selected cell
is occupied then replication does not occur. In this way, once the cells
become increasingly occupied, replication on the 1-dimensional grid becomes a
competition for longevity. There is no reward: either the organism lives or it
dies. The longer an organism is able to survive, the more likely it is to
replicate; the copies replicated from organisms with longer lifespans are
expected to be more likely to survive than those derived from organisms with
shorter lifespans, containing similar, but not identical, properties to their
ancestral organisms. Evolution occurs without guidance and without explicit
selection, instead, natural selection.

###  3.2 Re-defining robotic learning

To solve the familiar suite of modern robotic-learning tasks, first, the tasks
must be re-defined in a way which is suitable for the emergence of natural
selection: life and death. Three approaches toward this re-definition are
considered in this section: survival, foraging, and a survival-foraging.

Experimental Setup
Each task uses a single hidden-layer fully-connected recurrent neural network
with 32 neurons in the hidden-layer. Mutations occur on each weight between
neurons sampled from a zero-mean Gaussian distribution with a standard
deviation of $0.001$ when an organism replicates. In environment domains with
discrete actions, such as the Atari games, an output neuron exists for every
action command in the game and the neuron index with the maximum value is
selected for the current action (argmax). In continuous action spaces, an
output neuron exists for every controllable joint, and the action for each
joint is set to the value of its corresponding neuron.

####  3.2.1 Survival

Survival-based environments require the maintenance of some sort of state
which is defined by the environment. Once a certain condition is met, a death-
state, then the organism is terminated and ceases to exist, preventing further
reproduction. In survival-based environments, it is the sole avoidance of
external death-states that enable an organisms to survive, independent of
additional criterion. As is seen in the following survival experiments, this
sole focus on survival may, though not always, lead to organisms performing
clever environment exploitation, where death-states are avoided by
manipulating flaws in the environment’s design.

For the Atari environments presented, Random Access Memory (RAM) states were
selected for use as sensory input because the natural environment of digital
organisms is digital data, and hence it is useful to show that RAM states may
be used meaningfully. RAM state training is much less studied compared to
image-state information, perhaps as a result for being more difficult to learn
because of their discrete nature ([25]). Nonetheless, the solutions to the
Atari environments presented here utilize RAM.

Boxing-survival: The Boxing-survival environment has been adapted from an
existing Atari game ([26]), in this case, Boxing. In this environment, two
players, our organism and a pre-defined AI, are placed in a boxing ring where
actions include moving around and punching with either fist. The first
individual boxer to 100 points wins, where points are defined by how many
punches are landed on the opposing player. The fight has a 2-minute timer, and
the original environment ends either when an individual reaches 100 points or
when the timer runs out. Converting this environment opened several
possibilities in defining death. Two possible definitions of varying
difficulty are considered in this experiment; the first environment, Boxing-
survival-fight, defines the death-state as the moment the opponent obtains
more points, and, among the more difficult possibilities, the second
environment, Boxing-survival-avoidance, defines getting punched a single time
as the death-state.

In Boxing-survival-fight, the strategy for continued existence is non-trivial.
Consider the case of if the organism were to significantly out-box the
opposing side. In this scenario, the organism would reach 100 points before
the timer runs out, and hence, organisms which did not score quite as quickly,
but still managed to do better than the opponent would be more fit for
survival. Early successful replicators, which were the most efficient at
defeating the opponent quickly, served as a critical catalyst for obtaining a
larger population lifespan, but eventually dwindled in favor of more reserved
organisms, which did not rush to 100 points; rather, these organisms only
obtained points when necessary in favor of dodging punches and waiting for the
clock to run out. This ultimately led to a population of organisms which
survived the maximum amount of time possible – the entire 2-minute timer.

The Boxing-survival-avoidance environment is not as forgiving as the survival-
fight environment was. This environment does not allow the organism to receive
a single hit from the opponent, which creates only a single, and quite
challenging, possibility for continued existence: to avoid the enemy’s punches
completely. The organism discovers that if it stands in one of the boxing-ring
corners and rapidly punches its arm opposite the corner, then its opponent
will walk right into the punches and automatically back up, move around, and
come back, without ever throwing a punch – an exploitation of the enemy AI
design. This way, using a simple yet clever strategy, the organism
successfully avoids being hit by its opponent until the timer runs out. Upon
initially discovering this strategy, right-arm punches in the corner were
chaotic and relentless, without much consideration for timing; however, over
time the punches became more well-timed and deliberate as survival time
increased.

Space-Invaders-survival: Space-Invaders-survival presents a significantly
higher degree of complexity compared to the Boxing-survival environments. In
this environment, the organism is represented by a spaceship at the bottom of
the screen moving along the x-axis. Aliens stack on top of each other in a
row, firing down at the organism, and the organism may destroy enemies by
returning fire. There are three destructable defences which the organism can
temporarily cover under until the missile attack ultimately destroys them. The
goal in the original environment, based on the defined objective function, is
to destroy all of the aliens before losing the three allotted lives. This
environment naturally translates to survival, since there is already a
definition of life and death built into the game.

Like the other environments, the solution to this problem is not trivial, and
is quite surprisingly creative. One would imagine the clear strategy is to
minimize enemy bombardment by eliminating opposing aliens. However, once the
final alien is destroyed, then the game ends, and so does the organism’s life.
Much like in the Pong-survival environment, it is not in the organism’s best
interest to actually win the game, rather finding an exploit for an extended
lifespan is more desirable. This remains consistent with the organism
solution. To obtain a lifespan beyond completing the game, the organism
obtains a solution so as to destroy every alien except for one and keeps the
alien alive while skillfully avoiding its attacks.

####  3.2.2 Foraging

Not all desirable human-constructed environments lend themselves toward being
defined immediately in terms of life and death. Additionally, many practical
considerations may require routinely performing some task. Foraging-based
environments introduce the notion of a metabolism, whereby the organism must
meet some requirement at regular intervals in order to survive. This
requirement may be defined by a wide-variety of possibilities, for example:
obtaining resources, scoring points in a game, or progressing toward some
location.

Foraging tasks tend to be less prone to environment exploitation, as routinely
fulfilling some energy-requirement becomes necessary. The introduction of
foraging enables behavior which is desirable in the real world, considering
physical robots will likely require obtaining energy, resources, or
maintenance.

Reacher-forager: The Reacher-forager environment adapts the original Reacher
reinforcement learning benchmark, where a two-link arm resting on a plane is
controlled with the task of reaching the tip of the second link onto a goal
position. This task may naturally be extended to the foraging domain, by
simply requiring that the goal position be reached to obtain energy. Once
energy runs out, the organism dies. When a goal position is reached, energy is
deposited into the organisms energy storage, which slowly depletes over time,
and the goal position is reset.

The solution to this task is quite clever, and goes beyond what would be
expected of the organism. Instead of carefully placing the second link in the
correct location, the arm develops a more prosperous strategy of rapidly
spinning the arm in a circle while expertly compressing and expanding the
second link such that all possible goal positions are reached before energy is
depleted. Using this solution, the Reacher-forager organism population becomes
capable of indefinite existence. By adding an energy penalty to the Reacher’s
movement, this behavior returns to the expected strategy of precise and slow
movement toward goal positions.

####  3.2.3 Survival-foraging

Often times it is useful to incorporate both death-states and metabolism into
an environment definition. Both real-world and simulated applications of
learning often seek to avoid undesirable states, while simultaneously
fulfilling periodic requirements. In this section, an example of an
environment which benefits from both survival and forager environment traits
is presented.

BattleZone-survival-forager: BattleZone is an Atari game where the organism, a
tank, is placed in an environment filled with other enemy tanks. The tank
begins with 5-lives and the game score is based on the number of enemy
vehicles destroyed before all lives are lost. In converting this to a
survival-forager environment, the survival component includes death after all
5-lives are lost, and the forager component replenishes hunger after
destroying an enemy vehicle. When foraging is removed, the organism strategy
defaults to avoiding all tanks and driving around aimlessly, since the more
vehicles that get destroyed, the higher the game difficulty gets.

While it was indeed not the intention of these organisms to perform
competitively with state of the art optimization, and while these organisms
were not optimized for the defined objective function, the reward obtained by
the BattleZone-survival-forager organism was comparable to many of the highest
recorded to the best of our knowledge ([27, 28, 29, 30, 31]), though several
still outperform this ([32, 33]). With the destruction of each enemy vehicle
producing a reward of 1000, the resulting organism obtained a median reward of
102000 translating to 102 vehicles destroyed over all five lives. The strategy
for obtaining such performance is relatively simple to observe, as the tank
skillfully dodges enemy fire while returning fire and averaging 20 enemy
eliminations per life lost. The network structure used to obtain this
performance was a small fixed-weight structure with no additional
modifications. It is likely that significant improvements in performance could
be realized through additional structural changes.

####  3.2.4 A comparison of methods

One important question that is beneficial to consider is how does the
performance of evolutionary self-replicators compare with alternative
evolutionary approaches? While other methods are not immediately comparable
for all of the tasks presented in this work, as infinite lifespan solutions do
not naturally extend themselves to synchronized generation cycles, it is
feasible to demonstrate a comparison on several of the presented finite-
lifespan environments. Two methods are considered for comparison: Evolutionary
Algorithms (EAs) ([salimans2017evolution]) and Genetic Algorithms (GAs)
([such2018deep]).

Figure 1: Change in lifespan measured in the number of organisms for
Evolutionary Self-Replicators (orange), Genetic Algorithms (blue), and
Evolutionary Algorithms (green) for several finite-lifespan tasks. The number
of organisms per generation cycle for EAs and GAs is determined by the
population size, and are organized according to the timestep in which death
occurred.

The optimization of EAs and GAs differ from evolutionary self-replicators in
that the process of selection or optimization requires an explicit fitness
function. To determine this, reward for the EA and GA methods is defined as
the lifespan of the organism. For both methods, the network structure and
mutation scale is set identical to the evolutionary self-replicators
experiments; additional hyperparameters are defined in the Appendix.

Median Organism Lifespan (EvoReplicators, GAs, and EAs)
---
Environment |  EvoReplicator |  GA |  EA
Boxing-survival-avoidance |  7141 (Optimal) |  4152 |  2789
Boxing-survival-fight |  7141 (Optimal) |  7141 (Optimal) |  3234
Pong-survival |  23099 |  6788 |  865
Pong-survival-forager |  6637 |  567 |  324
Table 1: Median organism lifespan on the four tasks shown in Figure 1 across
EvoReplicators, GAs, and EAs. Performance was measured across ten independent
runs for each algorithm on each task.

Based on the results from Figure 1, EAs and GAs demonstrate stabler lifespan
growth over time in comparison to evolutionary self-replicators, which seems
to discover new solutions in the form of discrete innovations. These
discoveries are demonstrated by the large jumps in median lifespan, as opposed
to the smooth growth in lifespan seen in EAs and GAs. EAs are demonstrated to
struggle reaching the optimal lifespan, whereas GAs are shown to be closer in
performance when compared with evolutionary self-replicators on the first two
tasks. However, while GAs are closer in performance, evolutionary self-
replicators are demonstrated to more quickly find the optimal lifespan
solution for every task in Figure 1. The median organism lifespan shown in
Table 1 demonstrates that, on the four tasks presented, evolutionary self-
replicators more consistently arrive at large-lifespan solutions compared with
the EA and GA methods used in this work.

##  4 Discussion

Evolutionary self-replication presents itself as a powerful process toward
developing intelligence. It is shown that creative, meaningful, and
intelligent behavior emerges through the simple process of life, death, and
replication. It was shown that this capability can be harnessed in modern
robotic learning environments to obtain both practical and interesting
behavior by re-framing problems in terms of survival by natural selection.

The simplicity of the approach presented in this work is intentional, so as to
lay a foundation and allow significant room for future research. In the work
on Tierra, the self-replication process is the central focus, and Avida, both
replication and logic functions, whereas in this work, the process of survival
in complex learning environments is the central focus, with replication being
abstracted to a probabilistic process. An interesting possibility of future
work could incorporate the dynamics of Avida ([22, 23, 24]) or Tierra ([19,
20]) together with the survival dynamics and environments presented in this
work. Perhaps the concept of reproduction affects the way intelligence is
produced in evolutionary self-replication.

Some of the environments demonstrated in Section 3 were difficult to solve and
did not translate well toward being defined in terms of natural selection.
Perhaps this is a function of time, a product of using a small neural network,
a small 1-dimensional grid, or the parameters did not work well. It was
noticed that in some environments, small changes in parameters produced
drastic changes in behavior even with small mutation sizes, and hence random
noise mutations across generations struggled to retain desirable traits.
Perhaps this was a consequence of the discrete Atari RAM states, or the way in
which action-selection was treated. It was also observed that environments
with a high degree of randomness together with short average lifespans made it
challenging for advantageous traits to propagate across generations. It should
also be noted that not all of the solutions arrived at by evolutionary self-
replication, while still valid according to the survival definition, are
necessarily useful for an AI researcher looking to solve a particular task.
For example, the solution presented in the Pong-survival environment led to a
solution with no winning or losing, with the paddle fixated on a single point.
If the opposing challenger were changed it might lead to a dramatic change in
survivability. In this way, environment design becomes a more substantial
aspect toward obtaining desirable solutions since the self-replicators are
prone to finding exploitative solutions.

An interesting question to ask is whether evolutionary self-replication is a
learning algorithm in the traditional sense. In some ways, it resembles other
learning algorithms, and in others, it differs drastically. It seems as if the
emergence of intelligence in the case of evolutionary self-replication is a
matter of phenomenon rather than that of design; a phenomenon of great
intrigue to the development of artificial intelligence and life.

## References

  * [1] Nils J. Nilsson.  Logic and artificial intelligence.  Artificial Intelligence, 47(1):31–56, 1991.
  * [2] David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton.  Reward is enough.  Artificial Intelligence, 299:103535, 2021.
  * [3] Frederick Hayes-Roth.  Rule-based systems.  Communications of the ACM, 28(9):921–932, 1985.
  * [4] Paul Resnick and Hal R Varian.  Recommender systems.  Communications of the ACM, 40(3):56–58, 1997.
  * [5] Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf.  Support vector machines.  IEEE Intelligent Systems and their applications, 13(4):18–28, 1998\.
  * [6] Sebastian Risi, Sandy Vanderbleek, Charles Hughes, and Kenneth Stanley.  How novelty search escapes the deceptive trap of learning to learn.  pages 153–160, 01 2009.
  * [7] Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O. Stanley, and Jeff Clune.  Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents.  CoRR, abs/1712.06560, 2017.
  * [8] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine.  Diversity is all you need: Learning skills without a reward function, 2018\.
  * [9] Joel Lehman and Kenneth O. Stanley.  Novelty Search and the Problem with Objectives, pages 37–56.  Springer New York, New York, NY, 2011.
  * [10] Kenneth O Stanley and Joel Lehman.  Why greatness cannot be planned: The myth of the objective.  Springer, 2015.
  * [11] Joel Lehman and Kenneth Stanley.  Abandoning objectives: Evolution through the search for novelty alone.  Evolutionary computation, 19:189–223, 06 2011.
  * [12] Jean-Baptiste Mouret and Jeff Clune.  Illuminating search spaces by mapping elites, 2015.
  * [13] Justin K. Pugh, Lisa B. Soros, and Kenneth O. Stanley.  Quality diversity: A new frontier for evolutionary computation.  Frontiers in Robotics and AI, 3:40, 2016.
  * [14] Jeff Clune.  Ai-gas: Ai-generating algorithms, an alternate paradigm for producing general artificial intelligence, 2020.
  * [15] Agoston E Eiben and Jim Smith.  From evolutionary computation to the evolution of things.  Nature, 521(7553):476–482, 2015.
  * [16] Thomas S Ray.  Evolution, ecology and optimization of digital organisms.  Santa Fe, 1992.
  * [17] Alastair Channon and Bob Damper.  The artificial evolution of real intelligence by natural selection.  Image, Speech & Intelligent Systems Research Group, University of Southampton. http://www. soton. ac. uk/~ adc96r, 1997.
  * [18] AD Channon, RI Damper, et al.  Perpetuating evolutionary emergence.  FROM ANIMALS TO ANIMATS 5, pages 534–539, 1998.
  * [19] Thomas S Ray and Chenmei Xu.  Measures of evolvability in tierra.  Artificial Life and Robotics, 5(4):211–214, 2001.
  * [20] Thomas S Ray.  An approach to the synthesis of life.  Artificial life II, pages 371–408, 1991.
  * [21] Charles Ofria and Claus O Wilke.  Avida: A software platform for research in computational evolutionary biology.  Artificial life, 10(2):191–229, 2004.
  * [22] Richard E Lenski, Charles Ofria, Robert T Pennock, and Christoph Adami.  The evolutionary origin of complex features.  Nature, 423(6936):139–144, 2003.
  * [23] Christoph Adami, Charles Ofria, and Travis C Collier.  Evolution of biological complexity.  Proceedings of the National Academy of Sciences, 97(9):4463–4468, 2000.
  * [24] Richard E Lenski, Charles Ofria, Travis C Collier, and Christoph Adami.  Genome complexity, robustness and genetic interactions in digital organisms.  Nature, 400(6745):661–664, 1999.
  * [25] Jakub Sygnowski and Henryk Michalewski.  Learning from the memory of atari 2600, 2016.
  * [26] Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling.  The arcade learning environment: An evaluation platform for general agents.  CoRR, abs/1207.4708, 2012.
  * [27] Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver.  Distributed prioritized experience replay.  arXiv preprint arXiv:1803.00933, 2018.
  * [28] Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al.  Noisy networks for exploration.  arXiv preprint arXiv:1706.10295, 2017.
  * [29] Audrunas Gruslys, Will Dabney, Mohammad Gheshlaghi Azar, Bilal Piot, Marc Bellemare, and Remi Munos.  The reactor: A fast and sample-efficient actor-critic agent for reinforcement learning.  arXiv preprint arXiv:1704.04651, 2017.
  * [30] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling.  The arcade learning environment: An evaluation platform for general agents.  Journal of Artificial Intelligence Research, 47:253–279, 2013.
  * [31] Derek Yang, Li Zhao, Zichuan Lin, Tao Qin, Jiang Bian, and Tie-Yan Liu.  Fully parameterized quantile function for distributional reinforcement learning.  Advances in neural information processing systems, 32:6193–6202, 2019.
  * [32] Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell.  Agent57: Outperforming the atari human benchmark.  In International Conference on Machine Learning, pages 507–517. PMLR, 2020.
  * [33] Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney.  Recurrent experience replay in distributed reinforcement learning.  In International conference on learning representations, 2018.
  * [34] Larry Yaeger.  Computational genetics, physiology, metabolism, neural systems, learning, vision, and behavior or poly world: Life in a new context.  In SANTA FE INSTITUTE STUDIES IN THE SCIENCES OF COMPLEXITY-PROCEEDINGS VOLUME-, volume 17, pages 263–263. Citeseer, 1994.
  * [35] Peter J Bentley.  Is evolution creative.  In Proceedings of the AISB, volume 99, pages 28–34. Citeseer, 1999\.
  * [36] Charles Darwin.  The origin of species.  PF Collier & son New York, 1909.
  * [37] Tim Taylor and Alan Dorin.  Rise of the self-replicators.  2020\.
  * [38] Russell J Garwood, Alan RT Spencer, and Mark D Sutton.  Re vosim: Organism-level simulation of macro and microevolution.  Palaeontology, 62(3):339–355, 2019.
  * [39] Andrew N. Pargellis.  Digital life behavior in the amoeba world.  Artificial Life, 7(1):63–75, 2001.
  * [40] John D. Co-Reyes, Suvansh Sanjeev, Glen Berseth, Abhishek Gupta, and Sergey Levine.  Ecological reinforcement learning.  CoRR, abs/2006.12478, 2020.

##  Appendix A Appendix

Code for the experiments performed in this work are available at:
https://github.com/AnonymousSubmission/AnonymousSubmission.

###  A.1 Evolutionary Self-Replicator Hyperparameters

Hyperparameter Table
---
Hidden Layers |  1
Hidden Neurons |  32
Grid Dimension |  32
Mutation Probability |  0.5
Nonlinearity |  tanh
Mutation Scale |  0.001
Hidden Layer Type |  Recurrent

###  A.2 EA Hyperparameters

Hyperparameter Table
---
Hidden Layers |  1
Hidden Neurons |  32
Nonlinearity |  tanh
Hidden Layer Type |  Recurrent
Mutation Scale |  0.1
Mutation Scale Decay |  0.999
Learning Rate |  0.1
Learning Rate Decay |  0.999
Population Size |  48
Weight Decay Constant |  0.01

###  A.3 GA Hyperparameters

Hyperparameter Table
---
Hidden Layers |  1
Hidden Neurons |  32
Nonlinearity |  tanh
Hidden Layer Type |  Recurrent
Mutation Scale |  0.005
Selection Type |  Elite Selection
Elite Size |  8
Population Size |  128
Weight Decay Constant |  0.01

###  A.4 Environment Usage

Figure 2: Usage of the EvoGym library in Python.

Included in the code for this work is a custom Evolutionary Gym environment,
EvoGym, which has definitions of each environment defined in this work. The
structure of this library is set up to operate identically to the original Gym
environment without the inclusion of reward definitions.
https://github.com/AnonymousSubmission/AnonymousSubmission.