# Machine learning applications for electricity market agent-based models: A
systematic literature review

Alexander J. M. Kell<EMAIL_ADDRESS>Imperial College
LondonLondonGreater LondonUK43017-6221 ,  A. Stephen McGough  Newcastle
UniversityNewcastleUK and  Matthew Forshaw  Newcastle UniversityUK
<EMAIL_ADDRESS>
###### Abstract.

The electricity market has a vital role to play in the decarbonisation of the
energy system. However, the electricity market is made up of many different
variables and data inputs. These variables and data inputs behave in sometimes
unpredictable ways which can not be predicted a-priori. It has therefore been
suggested that agent-based simulations are used to better understand the
dynamics of the electricity market. Agent-based models provide the opportunity
to integrate machine learning and artificial intelligence to add intelligence,
make better forecasts and control the power market in better and more
efficient ways. In this systematic literature review, we review 55 papers
published between 2016 and 2021 which focus on machine learning applied to
agent-based electricity market models. We find that research clusters around
popular topics, such as bidding strategies. However, there exists a long-tail
of different research applications that could benefit from the high intensity
research from the more investigated applications.

machine learning, multi-agent system, literature review, electricity, markets,
forecasting

††ccs: Computing methodologies Modeling and simulation††ccs: Computing
methodologies Machine learning††ccs: Computing methodologies Artificial
intelligence

##  1\. Introduction

It is now almost universally acknowledged that we are having a devastating
impact on our environment (May, 2015). To help mitigate these impacts, a move
towards a low-carbon energy supply will be required. A clear understanding of
the mechanisms and their outcomes in the electricity generation markets is
becoming increasingly important as the first step towards mitigating the
possible devastating impacts of climate change. Due to the falling prices of
low-carbon electricity generation technologies, such as wind and solar power
(IRENA, 2018), electricity is increasingly being used to decarbonise energy
demands such as heating, automotive and in the industrial sector.

With the advent of renewable energy supply technologies, novel methodologies
are required to manage an electricity system with increased fluctuation due to
the nature of many renewable intermittent generation processes. This evolution
in the electricity system raises important questions, such as how to balance
supply and demand, how to bid in an uncertain market and how to forecast
prices. Electricity market models are an important tool used by policy makers,
researchers and industry to investigate questions such as these.

Traditional methods for analysing electricity systems, such as centralised
optimisation models have certain limitations such as the requirement for
perfect foresight, homogeneity of actors within the electricity system and a
cost-optimal system (Peng et al., 2021; Pfenninger et al., 2014; Ringler et
al., 2016). Agent-based models (ABMs) have been proposed as a novel
methodology to mitigate these limitations. Specifically, agent-based models
can model imperfect foresight, heterogeneity of actors with differing
objectives and individual level actions can be simulated.

ABMs are increasingly incoporating artificial intelligence (AI) and machine
learning (ML) to provide answers to complex questions such as electricity
frequency control, demand response and bidding strategies. As this is a
growing field of research, this paper attempts to provide a review of AI and
ML applied to ABMs for electricity markets. The goal of this paper is to
analyze the current literature on how AI is being used with ABMs in the area
of electricity markets. This paper makes three separate contributions to the
field:

  1. (1)

A classification of problem domains used by ABMs with ML and AI

  2. (2)

An overview of the relevant literature which uses these techniques

  3. (3)

New avenues of research to use ABMs with AI and ML.

This paper is structured as follows: Section 2 introduces the concept of ABMs
with AI and ML applied to electricity markets in more detail. Section 3
introduces the methodology undertaken to review the literature. Section 4, the
presented literature is discussed, placed in context and compared. Section 5
explores a possible future research direction.

##  2\. Background

###  2.1. Simulation of complex systems

Many real-world complex systems feature heterogenous, interacting and adaptive
units as well as exhibiting emergent properties (Ringler et al., 2016). The
analysis, forecasting and control of these systems can prove challenging due
to the many different features. In addition, the behaviour of these systems
can be non-linear and feature emergent properties that can be difficult to
predict. However, it remains important to perform analyses and better
understand these systems to achieve the intended goals (Kell et al., 2019).

Simulation and modelling can be used to perform “what-if” analyses “in-
silico”. That is, through the replication of salient features of the real-
world system into computer code, a model of this system can be designed. Once
the model has been created, multiple different analyses can be undertaken to
investigate the system, which allows practitioners to build knowledge of the
complex systems which they are studying.

Analyses can be carried out where real-world experimentation would be
prohibitive. This could be due to the requirement of high costs or because the
risk of experimentation would be excessively high. In addition, many
experimentations can be undertaken in parallel allowing users to find the best
parameter set or action to reach a particular objective.

These models can be used to investigate the effect of policies, shocks such as
sudden price rises or other scenarios on the system. In many cases, these
investigations would not have been possible without a simulation approach.
Another use of these simulations is to generate theories and for verification.
However, these models always have limitations, which should be improved on.
This could be by improving implementation, calibration, verification,
validation, interpretation and visualisation of results (Ringler et al.,
2016). These are key challenges for any model or computer simulation.

###  2.2. Agent-based models and simulation of electricity systems

Agent-based modelling is a framework to design a particular type of a computer
simulation. These models contain a number of agents with certain decision
rules and relationships with other agents. The behaviour of the individual
agents aggregates and leads to emergent system behaviour. This emergent system
behaviour can be difficult to define a-priori, and is why these simulations
can be useful.

Electricity systems are increasingly being modelled with an agent-based
modelling framework (Kell et al., 2020; Ringler, 2012). This is likely a
consequence of the transition from a homogenous central actor to the
heterogenous nature of multiple actors within a decentralised electricity
system. These heterogenous actors may have different objectives, risk
tolerances and budgets, among other factors which may have a significant
impact on the final electricity system. Agent-based modelling is well suited
to a system such as the one described here.

Agent-based models offer a methodology that can be used in international or
national electricity systems, local energy markets or in microgrids. In this
review, we present work which have used agent-based models for these three
different scopes, underlying the flexibility of this approach.

###  2.3. Machine learning and artificial intelligence

Machine learning (ML) is the study of computer algorithms that improve
automatically with the use of data. A computer program is said to learn from
experience E with respect to some class of tasks T and performance measure P,
if its performance at tasks in T, as measured by P, improves with experience E
(Mitchell, 1997).

Machine learning methods can be split into three different categories: (1)
supervised learning, (2) unsupervised learning and (3) reinforcement learning.
Each of these methods can be used in the following cases:

  1. (1)

Supervised learning is used where the data has labels, such as predicting the
energy used by a sub-station where we have measurements of the known, real
values.

  2. (2)

Unsupervised learning is where there are no labels associated with the data.
The model must, therefore, infer useful patterns from the data without
knowledge of meaning or significance. For example, a user may have data on two
species of animals and must distinguish between the two without having data on
which animal belongs to which species a-priori.

  3. (3)

Reinforcement learning is the case where an agent is placed within an
environment and must determine the best action to take to maximise some
reward. For example, an agent bidding into a market needs to know how much it
should bid to maximise long-term profit.

In addition to the aforementioned machine learning methods, there exists an
additional paradigm: optimisation methods. These methods explore a
mathematical or software function to find a minimum or maximum value of an
objective. These can be used to minimise an expected error, minimise total
cost or maximise total return from a system, for example.

In the following subsections we explore an overview of the different machine
learning techniques that are applicable to the problems addressed in this
review:

####  2.3.1. Supervised Learning

Supervised learning uses training data, which contains both the inputs and
desired outputs, to predict the output associated with new inputs. A
functioning supervised learning model, will therefore be able to correctly
predict the output for inputs that were not part of the original training
data. Supervised learning can be used for both regression and classification.
Regression is where a continuous output is returned, whereas classification is
where a discrete value is returned.

A select number of widely used supervised learning algorithms are support
vector machines (Cortes and Vapnik, 1995), linear regression (Seal, 1967),
logistic regression (Walker and Duncan, 1967), decision trees (Quinlan, 1987)
and neural networks (Hill et al., 1994).

####  2.3.2. Unsupervised Learning

As previously discussed, unsupervised learning infers patterns from unlabelled
data. Unsupervised learning can therefore exhibit self-organisation that can
capture patterns in data that were previously unknown.

Some of the most common algorithms include hierarchical clustering (Murtagh
and Contreras, 2012), k-means clustering (Hartigan and Wong, 1979) and self-
organizing maps (Kohonen, 1990).

Unsupervised learning is not the preferred option when labelled data is
available. Therefore it is used less in simulations and agent-based models due
to the high availability of data. However, in the real-world unsupervised
learning can be a powerful tool, where labelled data is lacking.

####  2.3.3. Reinforcement Learning

Reinforcement learning (RL) is an algorithm type which allows intelligent
agents to take actions in an environment in order to maximise a cumulative
reward. Reinforcement learning does not require labelled input/output data.
The basis of RL is to find a balance between exploration and exploitation.

A large amount of research has been dedicated in recent times to improving the
performance of reinforcement learning algorithms (Arulkumaran et al., 2017;
Lillicrap et al., 2016). Various different methodologies have been tried and
tested, from the use of neural networks in deep reinforcement learning to
updating a lookup table.

RL is well suited to simulation environments. This is due to the high
availability of observation data, the ability to craft rewards seen by the
agents and the high number of simulation iterations that can be run to train
the agents. They are also powerful at dealing with uncertainty and can perform
actions for a wide variety of problems. It is for these reasons that RL is
used a lot in simulations, and particularly agent-based models.

##  3\. Analysis

Artificial intelligence (AI) and machine learning (ML) have been integrated
with agent-based models to model the electricity sector with increasing
frequency over the last years. This is due to the ability of AI to optimise
agent behaviour, system parameters and add functionality to agent-based models
(ABMs). This study, therefore, reviewed recent papers regarding applications
of AI and ML in this space. We categorised these papers into four different
criteria: market type, application, algorithm type and algorithm used. To do
this, we used different search terms on Scopus and reviewed all 55 articles in
the field over the past five years. It was found that the majority of papers
used reinforcement learning applied to bidding strategies. However, a range of
applications were investigated through a wide variety of means. This included
price forecasting, demand forecasting, microgrid management and risk
management. We highlight the major findings from this study in the following
sections.

In this section, we review the literature that investigates how artificial
intelligence and machine learning can be integrated into agent-based models
for the electricity sector. To select the related articles to review, we
conduct a systematic literature review of relevant research in the field. We
limit our search to literature published in the five most recent years
(2016-2021). As a result of this, we provide a comprehensive status of the
applications of ML and AI in agent-based models for the electricity sector.
For this purpose we used the Elsevier Scopus database. To find the articles,
we used the following set of search terms to select our articles:

  1. (1)

Machine Learning, Artificial Intelligence, Deep Learning, Neural Networks,
Decision Tree, Support Vector Machine, Clustering, Bayesian Networks,
Reinforcement Learning, Genetic Algorithm, Online Learning, Linear regression.

  2. (2)

Agent-based modelling.

  3. (3)

Electricity.

We searched using each of the keywords in each of the bullet points. For
instance, the first keyword search was: Machine Learning, Agent-Based
Modelling and Electricity. The second was: Artificial Intelligence, Agent-
based modelling and Electricity. We selected these search terms to focus this
review on agent-based models applied to the electricity sector and machine
learning, which is the focus of this review.

These search terms resulted in 149 research articles. However, not all of
these were related to our research focus. For instance, a number of electric
vehicle, buildings and biological papers were returned. After reading the
titles, followed by the abstracts, these 149 papers were reduced to 55 papers
which were specifically related to agent-based modelling, electricity,
artificial intelligence and machine learning.

Figure 1. Number of articles published per year which work on AI in
electricity focused ABMs.

Figure 1 shows the amount of articles published each year between 2016 and the
present date. Whilst the number of articles published in this field has
increased per year since 2018 to 2020, the number of papers published in 2017
was lower when compared to the other years, with a large number published in
2016. We reviewed these 55 papers systematically in the following sections.

###  3.1. Market Type

In this literature review, we make three different market type distinctions:
international/national energy market, local energy market and a microgrid. The
international/national energy market typically considers a country, multiple
countries or the world. A local energy market is a smaller region than the
international/national energy market, for instance, a city or region. Whereas
a microgrid serves a discrete geographic footprint, such as a university
campus, business centre or neighbourhood. Whilst there is some cross-over
between a local energy market and microgrid, a microgrid can be disconnected
from the traditional grid and operate autonomously.

Tables 5, 6 and 7, 8 and 9 categorise the papers to their respective market
types. The papers have been displayed in chronological order and categorise
the market type, machine learning (ML) type used, the application in which it
was used and the algorithm used. These different criteria are explored in the
following subsections.

###  3.2. Machine Learning Types

Within this work, we have covered five different types of artificial
intelligence paradigms. These are: supervised learning, unsupervised learning,
reinforcement learning, optimisation and game theory.

Each of these techniques have been utilised in the papers surveyed. However, a
particular prevalence has been placed on reinforcement learning within the
research community. As shown by Figure 2, 67% of papers used a reinforcement
learning algorithm. This greatly outweighs the other machine learning types.
The second most used machine learning type was supervised learning, used by
eight papers. The fact that reinforcement learning has been used so
extensively within the agent-based modelling community for electricity
highlights the usefulness of this technique within this field, as previously
discussed.

Figure 2. Number of machine learning categories surveyed in papers.

Within each of the different machine learning categories there exist many
algorithms. The algorithms used in the papers surveyed are presented in Figure
3. Within reinforcement learning the most commonly used algorithms are
Q-Learning (Watkins and Dayan, 1992), Roth-Erev and Variant Roth-Erev (Roth AE
and Erev I, 1995).

Within supervised learning, the most commonly used algorithms are artificial
neural networks (ANN) (Hill et al., 1994), linear regression (Seal, 1967) and
support vector machines (SVM) (Cortes and Vapnik, 1995). Fewer algorithms were
used for both unsupervised learning and optimisation. For unsupervised
learning, only the following algorithms were used: Bayesian classifier
(Tschiatschek et al., 2014), K-Means Clustering (Hartigan and Wong, 1979) and
Naive Bayes classifier (Tschiatschek et al., 2014). For optimisation the
following algorithms were trialled: Bi-level coordination optimisation (Dempe,
2002), Genetic Algorithm. (Mitchell, 1995), Iterative algorithm (Kelley, 1999)
and Particle Swarm Optimisation (Kennedy and Eberhart, 1995). For the game
theory method, a game theoretic algorithm was used (Myerson, 1997).

Figure 3. Machine learning algorithms used per machine learning category.

###  3.3. Applications

Within this work, we classified each paper by the problem domain which they
are trying to solve, or the application. The applications are: agent
behaviour, bidding strategies, bilateral trading, demand forecasting, demand
response, electricity grid control, expansion planning, forecasting carbon
emissions, load scheduling, market investigation, microgrid management, peer
to peer trading, price forecasting, risk management, scheduling of
flexibility, secure demand side management and tariff design. Table
(table:application_defs)

Table 1. Application definitions. Application | Definition
---|---
Agent behaviour | Use of AI to control decisions made by agents
Bidding strategies | Bid within an electricity market
Bilateral trading | Inform trading decisions based on two trading entities
Demand forecasting | Forecasting of future electricity (or other) demand
Demand response | Reduction or increase of electricity demand to fit supply
Electricity grid control | Control of electricity flow around an electricity grid
Expansion planning | Decisions made upon which generators should be invested in
Forecasting carbon emissions | Forecasting of carbon emissions using AI
Load scheduling | Logistical planning of physical power flows by an operator
Market investigation | Analysis of the wider power market
Microgrid management | Management of electricity flows within a microgrid
Peer to peer trading | Strategies for trading between two or more entities within a power market
Price forecasting | Forecasting of electricity (or other) prices
Risk management | Management and reduction of risk
Scheduling of flexibility | Flexibility management of power system to ensure that supply meets demand
Secure demand side management | A resilient method to control demand to fit supply
Tariff design | Design of tariffs to charge or pay within a power market

Figure 4 displays the number of applications used by each paper. The most
utilised application was bidding strategies, with price forecasting and tariff
design following behind. However, the bidding strategies application was
investigated 49% times, with price forecasting investigated only 8 times. This
demonstrates a considerable research effort in this area.

Figure 4. Application number per paper.

Figure 5 displays the number of applications per machine learning type area.
We can see that bidding strategies are highly used within the reinforcement
learning type. The reinforcement learning algorithm, however, is shown to be
highly versatile, with different applications investigated, from demand
response, flexibility scheduling to expansion planning. This is due to the
ability for reinforcement learning to learn different policies based upon
solely the reward and observations within an environment.

Within supervised learning, a large amount of research effort has been put
into price forecasting. This is likely due to the strong ability of supervised
learning techniques at making predictions. However, outside of classification
and making predictions, supervised learning is not so versatile, when compared
to reinforcement learning.

Optimisation is used for five different applications. This is because
optimisation requires a problem domain to be maximised or minimised. This may
not be the case for all applications.

Unsupervised learning and the game theoretic machine learning types are used
less than the other machine learning types. This is because unsupervised
learning is preferential when there is no labelled data. However, with
labelled data, supervised learning can yield more accurate results. Within
simulations it is often the case that data is available, and so supervised
learning is used in preference to unsupervised learning. The game theoretic
approach has been used in a single application in the papers surveyed: bidding
strategies. The application of game theory is possible for the problem of
bidding strategies, however, the assumptions of a Nash equilibrium and perfect
information may not always exist in an electricity market.

Figure 5. Applications per application type.

###  3.4. Market Type

Table 2 displays the frequency of market types for the papers reviewed. Whilst
Table 3 shows the number of different applications per market type.

  1. (1)

68.9% of the papers reviewed relate to an international/national electricity
market. The availability of data and the relative importance of the subject of
whole system transitions in current affairs may explain why such research
effort has been dedicated to this.

  2. (2)

32.8% of the papers for the international/national focus on bidding
strategies, as is shown by Table 3. This is due to the ability of
reinforcement learning to make strategic decisions without full knowledge of
the system. This is true within electricity markets, where bidding strategies
must be formulated without the knowledge of the behaviour of other actors. In
addition, the ability to model strategic bidding is of significance importance
for international/national energy models due to the appearance of oligopolies
in national energy markets. However, whilst many studies have explored on
strategic bidding and a few have focused on oligopolistic bidding strategies,
a study on the impact on the wider market does not exist.

  3. (3)

Price forecasting is the second largest category which is explored, and is
investigated in 13.1% of papers. This is due to the ability for supervised
learning to make predictions in time-series data. It is also of importance for
agents to make realistic predictions of future prices. However, there exists a
gap in the literature on the long-term effects of the different accuracies of
these forecasts.

  4. (4)

18% of papers focus on the local energy market. A significant reduction when
compared to the international/national electricity market. This is because of
the limited availability of publicly available data for these local energy
markets. ABMs require a high amount of data to inform the behaviour of the
agents and environment, and so data collection for local energy markets can be
expensive and difficult to obtain. Microgrids are explored in 13.1% of papers.
Similarly to local energy markets this is because of a smaller amount of
publicly available data. It is also possible that researchers place an
increasing focus on international/national electricity markets due to the
availability of these models and perceived impact from a large system.

Market Type | Percentage (%)
---|---
International/National | 68.9
Local energy market | 18.0
Microgrid | 13.1
Table 2. Frequency of market type for papers reviewed. Market Type | Application | Percentage (%)
---|---|---
Microgrid | Bidding strategies | 4.9
Microgrid | Scheduling of flexibility | 3.3
Microgrid | Electricity grid control | 1.6
Microgrid | Load scheduling | 1.6
Microgrid | Microgrid management | 1.6
Local energy market | Bidding strategies | 6.6
Local energy market | Tariff design | 4.9
Local energy market | Demand response | 3.3
Local energy market | Forecasting carbon emissions | 1.6
Local energy market | Peer to peer trading | 1.6
International/National | Bidding strategies | 32.8
International/National | Price forecasting | 13.1
International/National | Agent behaviour | 3.3
International/National | Expansion planning | 3.3
International/National | Tariff design | 3.3
International/National | Bilateral trading | 1.6
International/National | Demand forecasting | 1.6
International/National | Demand response | 1.6
International/National | Load scheduling | 1.6
International/National | Market investigation | 1.6
International/National | Peer to peer trading | 1.6
International/National | Risk management | 1.6
International/National | Secure demand side management | 1.6
Table 3. Frequency of market type and application for papers reviewed.

###  3.5. Applications

  1. (1)

A significant proportion of papers have focused on bidding strategies, with
44.3% of papers investigating this. Bidding strategies is top for each of the
market types, indicating the versatility and applicability of this technique
in electricity markets.

  2. (2)

The application of control, for instance grid control, load scheduling and
demand response has seen a significant amount of promising research.
Particular in the application of decentralised control (Najafi et al., 2018;
Shafie-Khah et al., 2020; Čaušević et al., 2017). With the advent of
distributed data collection and internet of things, it has become possible to
achieve various welfare objectives through a distributed control algorithm.
That is, to optimise for individuals or distributed generators from the
perspective of those respective individuals and generators. Studies have shown
efficiencies close to centralised algorithms. However, centralised algorithms
may reduce the agency of individuals and therefore must overcome an additional
barrier in uptake if the individuals do not perceive that the algorithm is
working in their own best interest.

  3. (3)

Various applications have been explored within the research community. In
total, 17 different applications were explored. This demonstrates the
versatility of ML applied to ABMs. This, however, highlights a significant gap
in the literature as the majority of applications have only been explored by
one or two papers. For instance, the ability to optimise the electricity
system parameters in question has not been explored to the same level of
detail as forecasting or trading behaviour.

  4. (4)

Whilst there are many studies which look at optimising certain aspects of an
electricity market, there is no study which looks at optimising the whole
electricity system as a whole.

###  3.6. Algorithm Type

Table 4 displays the frequency of each of the algorithms used. Q-Learning is
the most used algorithm, with 29% of papers using this algorithm. Second is
the Roth-Erev algorithm, which is used by 9.7% of papers. Whilst this shows
the versatility of these algorithms, further research could be placed into the
use of deep reinforcement learning (DRL) to improve results. DRL uses deep
neural networks to act as function approximators. DDPG, DQN and DPG are
examples of DRL algorithms.

The majority of the algorithms have only been used in a single paper, and so,
there remains a significant gap in the literature to apply these algorithms to
the different applications to investigate with results can be improved through
other algorithms.

Algorithm Used | Percentage (%)
---|---
Q-Learning | 29.0
Roth-Erev | 9.7
ANN | 6.5
Linear regression | 4.8
Genetic Algorithm | 4.8
DDPG | 3.2
Variant Roth-Erev | 3.2
DPG | 1.6
DQN | 1.6
SVM | 1.6
SARIMAX | 1.6
Bayesian networks | 1.6
Bi-level coordination optimization | 1.6
Probe and adjust | 1.6
Policy Iteration (PI) | 1.6
Particle Swarm Optimisation | 1.6
Novel WoLF-PHC | 1.6
Naive Bayes classifier | 1.6
MDP | 1.6
MADDPG | 1.6
Classification trees | 1.6
Lasso regression | 1.6
LSTM | 1.6
Bayesian classifier | 1.6
Iterative algorithm | 1.6
Clustering | 1.6
Game theory | 1.6
Fitted Q-iteration (FQI) | 1.6
Extreme Machine Learning | 1.6
Deep Q-Learning | 1.6
K-Means Clustering | 1.6
Table 4. Frequency of algorithms for papers reviewed.

##  4\. Literature review

###  4.1. Reinforcement Learning

In this section we review the papers that utilised reinforcement learning for
the applications shown in Figure 5. Firstly, we cover the papers which
consider the bidding strategies problem.

Liu et al.(Liu et al., 2020) establish non-cooperative and cooperative game
models between thermal power companies. They show that, compared with other RL
algorithms, the MADDPG algorithm (Lowe et al., [n.d.]) is more accurate, with
an increase in revenue of 5.2%. They show that thermal companies are more
inclined to use physical retention methods to make profits in the medium and
long-term power market.

Liang et al.(Liang et al., 2020) use non multi-agent algorithm, called the
DDPG algorithm (Lillicrap et al., 2016) to model the bidding strategies of
GenCos. They show that the DDPG algorithm is able to converge to a Nash
equilibrium even with imperfect information. They find that they are able to
reflect collusion through adjusting the GenCos’ patience parameter.

Purushothaman et al.(Purushothaman and Chandrakala, 2020) model the learning
capabilities of power generators through the use of the Roth-Erev RL
algorithm. They find that the agents are able to exhibit market power through
this approach. Kiran et al.(Kiran and Vijaya Chandrakala, 2020) use a variant
of the Roth-Erev algorithm to investigate the ability for a generator to bid
strategically within a market. They find that agents have the ability to bid
strategically, and increase their net earnings. However, the impact on the
wider market and how to limit this ability was not explored in these papers.

Sousa et al.(Sousa and Tome Saraiva, 2020) use an ABM to model the Iberian
electricity market, with a focus on hydropower plants. Differently to the
previously mentioned papers, they use a state-table based approach, the
Q-Learning algorithm to bid into the day-ahead market. Poplavskaya et
al.(Poplavskaya et al., 2020) model the balancing services market, and
investigate the effect of different market structures on price. They find that
in an oligopoly, prices can deviate from the competitive benchmark by a factor
of 4-5. They conclude that changing market type would not solve this issue.

Nunna et al.(Nunna et al., 2020) also use a Q-learning algorithm to develop
bidding strategies for energy storage systems, however their algorithm focuses
on a simulated-annealing-based approach. They observe that they can
effectively reinforce the balance between supply and demand in the microgrids
using a mixture of local and global energy storage systems.

Differently to the previously mentioned papers, Lin et al.(Lin et al., 2019)
investigate herding behaviours of electricity retailers on the monthly
electricity market. Herding behaviours are where individuals act collectively
as part of a group. They model herding behaviours mathematically based on the
relationship network of electricity retailers which are imbedded in an ABM.
They use the Roth-Erev RL to model these behaviours. They find that the
herding behaviours might bring positive or negative effects to electricity
retailers depending on the differing bidding strategies.

Wang et al.(Wang et al., 2019) investigate the bidding behaviour of all
players in the electricity market. They propose a hybrid simulation model and
integrate RL to bid. They find that with the hybrid simulation model, the
dynamics of the entire market remain stable, the market clearing prices
converge, and the market share is relatively uniform.

Ye et al.(Ye et al., 2019) propose a novel multi-agent deep RL algorithm,
where they combine the DPG (Silver et al., 2014) algorithm with LSTM
(Hochreiter and Schmidhuber, 1997) for multi-agent intelligence. Their
algorithm achieves a significantly higher profit than other RL methods for a
GenCo.

Pinto et al.(Pinto et al., 2019) investigate the ability for collaborative RL
models to optimise energy contract negotiation strategies. The results show
that the collaborative learning process enables players’ to correctly identify
which negotiation strategy to apply in each moment, context and opponent.

Machado et al.(Machado et al., 2019) take a different approach, and
investigate how the energy price is affected when a government intervention is
observed through the increase in number of public companies participating in
auctions. They find that they are able to model the impact of public companies
on the overall electricity market, rather than trying to improve the bidding
strategy of a single company.

Feng et al.(Feng et al., 2018) follow a similar, governmental policy approach
to Machado et al. and explore the effect of a transition from a monopoly to a
competitive electricity market. They simulate a day-ahead market and use RL to
bid strategically. They find that they can characterise the risk
characteristics and decision-making objectives of market participants.

Calabria et al.(Calabria et al., 2018) simulate the Brazilian power system,
using 3 years worth of data which covers 98% of the total hydro capacity of
Brazil. They use RL to simulate the behaviour of virtual reservoirs in this
market. They find that through the management of these virtual reservoirs,
they can save water, while maintaining current efficiency and security levels.

Gaivoronskaia et al.(Gaivoronskaia and Tsyplakov, 2018) present a modification
of the classical Roth-Erev algorithm to represent agents’ learning in an ABM
of the Russian wholesale electricity market, in a day-ahead market. Staudt et
al.(Staudt et al., 2018) investigate the effect of the number of energy
suppliers needed for a competitive market. They find that several suppliers
are required to ensure a welfare optimal pricing. Where welfare is defined as
a benefit to the community.

Esmaeili Aliabadi et al.(Esmaeili Aliabadi et al., 2017) study the effect of
risk aversion on GenCos’ bidding behaviour in an oligopolistic electricity
market. They find that the change in the risk aversion level of even one GenCo
can significantly impact on all GenCo bids and profits.

Rashedi et al.(Rashedi et al., 2016) use Q-Learning to learn optimal bidding
strategies of suppliers in a day-ahead market. They compare the competitive
behaviour of players in both the multi-agent and single-agent case.
Chrysanthopoulos et al.(Chrysanthopoulos and Papavassilopoulos, 2016) take a
similar approach, by modelling the day-ahead market as a stochastic game. They
use an ABM to model GenCos to maximise their profit using RL.

Skiba et al.(Skiba et al., 2016) model the bidding behaviour on the day-ahead
and control reserve power markets using an RL algorithm in an ABM. They
investigate the resulting market prices. Tang et al.(Tang et al., 2021)
investigate the bidding strategies of generators under three pricing
mechanisms. They find they are able to achieve market equilibrium results
through their novel RL approach.

Bakhshandeh et al.(Bakhshandeh and Foroud, 2016) assess the ability for GenCos
to withhold capacity to increase the market price using RL. The results
demonstrate the emergence of capacity withholding by GenCos, which have an
effect on the market price.

Xu et al.(Xu et al., 2019) simulate a proactive residential demand response in
a day-ahead market using RL. This is in contrast to the general electricity
markets modelled in the previously mentioned papers. They use residential data
in China, and test a case with 30,000 households. They find that a proactive
residential demand response may yield significant benefits for both the supply
and demand side.

Deng et al.(Deng and Wu, 2021) use a DDPG algorithm to also model residential
demand response. They find that the goal of peak load-cutting and valley
filling can be achieved with this method.

Shafie-Khah et al.(Shafie-Khah et al., 2020) develop a novel decentralised
demand response model. In their model, each agent submits bids according to
the consumption urgency and a set of parameters by the RL algorithm,
Q-Learning. Their studies show that their decentralised model drops the
electricity cost dramatically, which was nearly as optimal as a centralised
approach. Najafi et al.(Najafi et al., 2018) also propose a decentralised
demand response model. They use Q-Learning and consider small scale GenCos.
Similarly to Shafie-Khah et al., they show the effectiveness of this
technique.

Tomin et al.(Tomin et al., 2020) propose an RL method to interact with the
electricity grid for active grid management, as opposed to bidding strategies.
They demonstrate the effectiveness of this approach on a test 77-node scheme
and a real 17-node network diagram of the Akademgorodok microdistrict
(Irkutsk).

Huang et al.(Huang and Guo, 2016) propose a generation investment planning
model for GenCos. They use a genetic algorithm and Q-learning to improve their
optimisation ability, and show that the model is effective and could provide
support for plant expansion planning. Manabe et al.(Manabe et al., 2016) also
consider the generation expansion planning problem. They find through their
simulation that their agents can generate higher profit using RL.

Foruzan et al.(Foruzan et al., 2018) use an ABM to study distributed energy
management in a microgrid. They use an RL algorithm to allow generation
resources, distributed storages and customers to develop optimal strategies
for energy management and load scheduling. They are able to reach a Nash
equilibrium, where all agents benefit through this approach.

Viehmann et al.(Viehmann et al., 2021) analyse the different markets: Uniform
Pricing (UP) and Discriminatory Pricing. UP is where all agents pay the
maximum accepted price, and DP pay their accepted bid. Through the use of RL,
they find that UPs lead to higher prices in all analysed market settings.

Pinto et al.(Pinto and Vale, 2019) introduce a learning model to enable
players to identify the expected prices of bilateral agreements as opposed to
the multilateral markets surveyed by the previously mentioned papers. For this
they use a Q-Learning algorithm, and they use real data from the Iberian
electricity market.

Mbuwir et al.(Mbuwir et al., 2020) explore two model-free RL techniques:
policy iteration (PI) (Lagoudakis and Parr, 2003) and fitted Q-iteration (FQI)
(Riedmiller, 2005) for scheduling the operation of flexibility providers -
battery and heat pump in a residential microgrid. Their simulation results
show that PI outperforms FQI with a 7.2% increase in photovoltaic self-
consumption in the multi-agent setting and a 3.7% increase in the single-agent
setting.

Naseri et al.(Naseri et al., 2021) propose an autonomous trading agent which
aims to maximise profit by developing tariffs. They find that designing
tariffs with usage-based charges and fixed periodic charges can help the agent
segment the retail market resulting in lower peak demand and capacity charges.

Bose et al.(Bose et al., 2021) simulate a local energy market as a multi-agent
simulation of 100 households. Through the use of the Roth-Erev reinforcement
learning algorithm to control trading, and demand response of electricity.
They are able to achieve self-sufficiency of up to 30% with trading, and 41.4%
with trading and demand response. Kim et al. (Kim and Lee, 2020) consider a
prosumer that consumes and produces electric energy with an energy storage
system. They use the DQN reinforcement learning algorithm to maximise profit
in peer-to-peer energy trading. Liu et al.(Liu et al., 2021) also investigate
the peer-to-peer trading problem in a local energy market. They find that the
community modelled are able to increase profitability through the use of DQN
RL.

Year | First Author |  Market Type |  Application |  Algorithm Used
---|---|---|---|---
2021 | Bose S. (Bose et al., 2021) |  Local energy market |  Peer to peer trading |  Roth-Erev
2021 | Naseri N. (Naseri et al., 2021) |  Local energy market |  Tariff design |  SARIMAX, MDP
2021 | Tang C. (Tang et al., 2021) |  International/National |  Tariff design |  Novel WoLF-PHC
2021 | Liu D. (Liu et al., 2021) |  International/National |  Bidding strategies, Peer-to-Peer |  MADDPG
2021 | Deng C. (Deng and Wu, 2021) |  International/National |  Demand response |  DDPG
2021 | Viehmann J. (Viehmann et al., 2021) |  International/National |  Market investigation |  Q-Learning
2020 | Tomin N. (Tomin et al., 2020) |  Microgrid |  Electricity grid control |  Q-Learning
2020 | Liang Y. (Liang et al., 2020) |  International/National |  Bidding strategies |  DDPG
2020 | Kim J.-G. (Kim and Lee, 2020) |  International/National |  Peer to peer trading |  Deep Q-Learning
2020 | Shafie-Khah M. (Shafie-Khah et al., 2020) |  Local energy market |  Demand response |  Q-Learning
2020 | Sousa J.C. (Sousa and Tome Saraiva, 2020) |  International/National |  Bidding strategies |  Q-Learning
2020 | Poplavskaya K. (Poplavskaya et al., 2020) |  International/National |  Bidding strategies |  Q-Learning
2020 | Liu Y. (Liu et al., 2020) |  Local energy market |  Bidding strategies |  DQN
2020 | Nunna H.S.V.S.K. (Nunna et al., 2020) |  Microgrid |  Bidding strategies |  Q-Learning
2020 | Purushothaman K. (Purushothaman and Chandrakala, 2020) |  International/National |  Bidding strategies |  Roth-Erev
2020 | Mbuwir B.V. (Mbuwir et al., 2020) |  Microgrid |  Scheduling of flexibility |  Policy Iteration (PI), Fitted Q-iteration (FQI)
2020 | Kiran P. (Kiran and Vijaya Chandrakala, 2020) |  International/National |  Bidding strategies |  Variant Roth-Erev
2019 | Lin F. (Lin et al., 2019) |  International/National |  Bidding strategies |  Roth-Erev
2019 | Wang J. (Wang et al., 2019) |  International/National |  Bidding strategies |  Roth-Erev
2019 | Machado M.R. (Machado et al., 2019) |  International/National |  Bidding strategies |  Q-Learning
2019 | Ye Y. (Ye et al., 2019) |  International/National |  Bidding strategies |  DPG,LSTM
2019 | Pinto T. (Pinto and Vale, 2019) |  International/National |  Price forecasting |  Q-Learning
2019 | Pinto T. (Pinto et al., 2019) |  International/National |  Bidding strategies |  Q-Learning
2018 | Feng H. (Feng et al., 2018) |  Local energy market |  Bidding strategies |  Variant Roth-Erev
2018 | Foruzan E. (Foruzan et al., 2018) |  Microgrid |  Load scheduling |  Q-Learning
2018 | Calabria F.A. (Calabria et al., 2018) |  International/National |  Bidding strategies |  Q-Learning
2018 | Gaivoronskaia E.A. (Gaivoronskaia and Tsyplakov, 2018) |  International/National |  Bidding strategies |  Roth-Erev
2018 | Staudt P. (Staudt et al., 2018) |  Microgrid |  Bidding strategies |  Probe and adjust
2018 | Najafi S. (Najafi et al., 2018) |  Local energy market |  Demand response, Bidding strategies |  Q-Learning
2017 | Esmaeili Aliabadi D. (Esmaeili Aliabadi et al., 2017) |  International/National |  Bidding strategies |  Q-Learning
2016 | Huang X. (Huang and Guo, 2016) |  International/National |  Expansion planning |  Q-Learning
2016 | Rashedi N. (Rashedi et al., 2016) |  International/National |  Bidding strategies |  Q-Learning
2016 | Skiba L. (Skiba et al., 2016) |  International/National |  Bidding strategies |  Roth-Erev
2016 | Bakhshandeh H. (Bakhshandeh and Foroud, 2016) |  International/National |  Bidding strategies |  Q-Learning
2016 | Chrysanthopoulos N. (Chrysanthopoulos and Papavassilopoulos, 2016) |  Local energy market |  Bidding strategies |  Q-Learning
Table 5. Articles relating to reinforcement learning algorithm type.

###  4.2. Supervised Learning

In this section we review the papers that used a supervised learning approach
with their agent-based models, which focus on electricity. First we consider
papers which focus on price forecasting.

Fraunholz et al.(Fraunholz et al., 2021) use ANNs to forecast electricity
price endogenously within the long-term energy model, PowerACE. They find that
the ANN method outperforms the linear regression method and that this
endogenous method has a significant impact on simulated electricity prices.
This is of importance since these are major results for electricity market
models. Pinto et al.(Pinto et al., 2016) uses SVMs and ANNs for price
forecasting using real data from MIBEL, the Iberian market operator. They show
an ability to return promising results in a fast execution time.

Goncalves et al.(Goncalves et al., 2019) use multiple different methods to
understand the main drivers of electricity prices. These include lasso and
standard regression (Tibshirani, 1996), and causal analysis such as Bayesian
networks and classification trees. Their results are coherent and show the
impact of different generators on final electricity price.

Opalinski et al.(Opalinski and Dudek, 2016) propose a hybrid prediction model,
where the best results from a possible large set of different short-term load
forecasting models are automatically selected based on their past performance
by a multi-agent system. They show an increase in prediction accuracy with
their approach. This is in contrast to the previous papers which trial
different approaches and manually select algorithms based on the results.

Bouziane et al.(Bouziane et al., 2021) forecast carbon emissions using a
hybrid ANN and ABM approach from different energy sources from a city. They
forecast energy production using agents and calculate the benefits of using
renewable energy as an alternative way of meeting electricity demand. They
find they are able to reduce emissions by 3% per day using this approach.

Pinto et al.(Pinto and Falcão-Reis, 2019) propose an approach to addressing
the adaptation of players’ behaviour according to participation risk. To do
this, they combine the two most commonly used approaches of forecasting:
internal data analysis and sectorial data analysis. They show that their
proposed approach is able to outperform most market participation strategies
and reach a higher accumulated profit by adapting players’ actions according
to the participation risk.

Maqbool et al.(Maqbool et al., 2019) investigate the impact of feed-in tariffs
and the installed capacity of wind power on electricity consumption from
renewables. They use linear regression to understand the outputs from an
agent-based model to achieve this. They find that the effect of increasing
installed capacity of wind power is more significant than the effect of feed-
in tariffs.

El Bourakadi et al.(El Bourakadi et al., 2019) propose the use of an Extreme
Machine Learning (EML) (Huang et al., 2006) algorithm to make decisions about
selling/purchasing electricity from the main grid and charging and discharging
batteries from an ABM. The EML algorithm predicts wind and photovoltaic power
output from weather data and then makes a classification decision on whether
to buy or sell power.

Babar et al.(Babar et al., 2020) propose a secure demand-side management
engine to preserve the efficient utilisation of energy based on a set of
priorities. A model is developed to control intrusions into the smart grid
using a naive Bayes classifier. Simulation is used to test the efficiency of
the system, and the result reveal that the engine is less vulnerable to
intrusion.

Year | First Author |  Market Type | Application |  Algorithm Used
---|---|---|---|---
2021 | Fraunholz C. (Fraunholz et al., 2021) |  International/National | Price forecasting |  ANN
2021 | Bouziane S.E. (Bouziane et al., 2021) |  Local energy market | Forecasting carbon emissions |  ANN
2020 | Babar M. (Babar et al., 2020) |  International/National | Secure demand side management |  Naive Bayes classifier
2019 | Maqbool A.S. (Maqbool et al., 2019) |  International/National | Tariff design |  Linear regression
2019 | Goncalves C. (Goncalves et al., 2019) |  International/National | Price forecasting |  Linear regression, Lasso regression, Bayesian networks, Classification trees
2019 | Pinto T. (Pinto and Falcão-Reis, 2019) |  International/National | Risk management |  ANN
2019 | El Bourakadi D. (El Bourakadi et al., 2019) |  Microgrid | Bidding strategies |  Extreme Machine Learning
2016 | Opalinski A. (Opalinski and Dudek, 2016) |  International/National | Demand forecasting |  Linear regression
2016 | Pinto T. (Pinto et al., 2016) |  International/National | Price forecasting |  ANN, SVM
Table 6. Articles relating to supervised learning algorithm type.

###  4.3. Unsupervised learning

In this section we discuss the papers which use an unsupervised learning
approach with agent-based models.

Imran et al.(Imran et al., 2020) develop a novel strategy for bilateral
negotiations. They enable each GenCo to estimate the reservation price of its
opponent using Bayesian learning. They show that the agents which use Bayesian
learning gain an advantage over non-learning agents.

Čaušević et al.(Čaušević et al., 2017) propose a novel clustering algorithm to
cluster agents into virtual cluster members for the application of load
scheduling. They show that large-scale centralised energy systems can operate
in a decentralised fashion when only local information is available.

Gomes et al.(Gomes et al., 2021) propose a management system for the operation
of a microgrid by an electricity market agent. They use K-means clustering for
scenario reduction and a stochastic mixed-integer linear programming problem
to manage the system.

Year | First Author |  Market Type | Application |  Algorithm Used
---|---|---|---|---
2021 | Gomes I.L.R. (Gomes et al., 2021) |  Microgrid | Microgrid management |  K-Means Clustering
2020 | Imran K. (Imran et al., 2020) |  International/National | Bilateral trading |  Bayesian classifier
2017 | Čaušević S. (Čaušević et al., 2017) |  International/National | Load scheduling |  Novel clustering algorithm
Table 7. Articles relating to unsupervised learning algorithm type.

###  4.4. Optimisation

In this subsection we review the papers which use optimisation as a basis for
investigation.

Bevilacqua et al.(Bevilacqua et al., 2019) compare three optimisation methods
to implement agent rationality in the Italian electricity market with an ABM.
Through this, they observe that the moddel exhibits a very good fit to real
data.

Duan et al.(Duan et al., 2018) propose a bi-level coordination optimisation
integrated resource strategy to unify supply-side and demand side resources
across China. They do this for mid-long term planning.

Gao et al.(Gao et al., 2018) use a genetic algorithm to determine an optimal
bidding strategy. They verify their approach by modelling a 30-bus system as
an example. Meng et al.(Meng et al., 2018) also use a genetic algorithm
optimisation approach to make dynamic pricing decisions. They model the day-
ahead market and propose a two-level optimisation model. They model the price
responsiveness of different customers using the optimisation algorithm, and
then optimise the dynamic prices that the retailer sets to maximise its
profit. They confirm the feasibility and effectiveness of their technique
through simulation.

Table 8. Articles relating to optimisation algorithm type. Year | First Author |  Market Type | Application |  Algorithm Used
---|---|---|---|---
2019 | Bevilacqua S. (Bevilacqua et al., 2019) |  International/National | Agent behaviour |  Genetic Algorithm,Particle Swarm Optimisation
2018 | Duan W. (Duan et al., 2018) |  International/National | Expansion planning |  Bi-level coordination optimization
2018 | Gao Y. (Gao et al., 2018) |  International/National | Bidding strategies |  Genetic Algorithm
2018 | Meng F. (Meng et al., 2018) |  Local energy market | Tariff design |  Genetic Algorithm

###  4.5. Game theory

Filho et al.(Filho et al., 2016) is the only paper in this review which takes
a game theoretic approach. They deal with a comparative analysis of individual
strategies of generating units in auctions, using non-cooperative game theory
approach. They find that their method is best suited for second-price auctions
and can be extended to more complicated networks with high precision. However,
they find that it is not possible to use this methodology in some, more
complex, systems. For example, bidding within a stochastic and uncertain
environment, where a predictions are required for the behaviour of other
actors.

Table 9. Articles relating to game theoretic algorithm type. Year | First Author |  Market Type | Application |  Algorithm Used
---|---|---|---|---
2016 | Filho N.S.C. (Filho et al., 2016) |  International/National | Bidding strategies |  Game theory

##  5\. Conclusion

This paper provides a systematic review of artificial intellgigence (AI)
applied to agent-based models within the electricity sector. Through this
analysis, various research gaps have been identified. This is because, the
majority of papers focus on similar themes and algorithms. This, therefore,
highlights a wide range of areas in which the research community can focus on
in future work.

Significant and increasing research interest has been placed into agent-based
models applied to the electricity sector. However, this work has shown that a
lot of research has clustered around similar subjects. For instance, the top
application investigated for each market type is bidding strategies, largely
exploring whether it is possible to use reinforcement learning to bid into an
electricity market. However, the impact of these bidding strategies on the
wider market has been investigated to a lesser extent. Through classification
of the different paper types, a long-tail was identified in research
undertaken into different topics. For instance, a small amount of papers
carried out work on the forecasting of carbon emissions, electricity grid
control or bilateral trading. This, therefore, suggests that further work
could be carried out to build upon this work in the future.

Another aspect that has been explored to a lesser extent is the optimisation
of the electricity system parameters as a whole as applied to policy. Much of
the work is focused on micro-behaviours within the energy system. For example,
improving on the demand response application. Kell et al. optimise the whole
energy system by modifying selected parameters(Kell et al., 2020) showing that
this work can be carried out to great effect.

It was found that only 18% of papers focus on the local electricity market. A
significantly smaller proportion when compared to the international/national
electricity market. This may be due to the lack of publicly available data for
these local electricity markets. There therefore exists a high impact
opportunity to collect and make local electricity market data available. This
is because local electricity markets will become increasingly important for
future systems with more small-scale renewable energy projects.

However, the wide-range of applications that were investigated shows the large
impact that artificial intelligence can have on the electricity sector when
combined with agent-based models. Due to the generalisable nature of these
techniques findings from one application can be utilised to inform other
applications. For instance, the highly researched bidding strategies domain
can be used to inform microgrid management, which was only investigated by a
single paper. Through this approach, it is suggested that high efficiencies
can be gained within the power sector.

## Acknowledgements

This work was supported by the Engineering and Physical Sciences Research
Council, Centre for Doctoral Training in Cloud Computing for Big Data [grant
number EP/L015358/1].

## References

  * (1)
  * May (2015) 2015\.  Paris Agreement.  _United Nations_ 21 (2015), 1–23.
  * Arulkumaran et al. (2017) Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. 2017\.  A Brief Survey of Deep Reinforcement Learning.  _IEEE Signal Processing Magazine_ (2017), 1–16.  arXiv:arXiv:1708.05866v2
  * Babar et al. (2020) M. Babar, M. U. Tariq, and M. A. Jan. 2020.  Secure and resilient demand side management engine using machine learning for IoT-enabled smart grid.  _Sustainable Cities and Society_ 62 (2020).  www.scopus.com Cited By :14.
  * Bakhshandeh and Foroud (2016) H. Bakhshandeh and A. A. Foroud. 2016.  Development of reinforcement learning algorithm to study the capacity withholding in electricity energy markets.  _Iranian Journal of Electrical and Electronic Engineering_ 12, 1 (2016), 42–51.  www.scopus.com Cited By :1.
  * Bevilacqua et al. (2019) S. Bevilacqua, C. da Costa Pereira, E. Guerci, F. Precioso, and C. Sartori. 2019.  _Analysing the Impact of Rationality on the Italian Electricity Market_. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 11676 LNAI.  236–247 pages.  www.scopus.com Cited By :1.
  * Bose et al. (2021) S. Bose, E. Kremers, E. M. Mengelkamp, J. Eberbach, and C. Weinhardt. 2021.  Reinforcement learning in local energy markets.  _Energy Informatics_ 4, 1 (2021).  www.scopus.com Cited By :1.
  * Bouziane et al. (2021) S. E. Bouziane, M. T. Khadir, and J. Dugdale. 2021\.  A collaborative predictive multi-agent system for forecasting carbon emissions related to energy consumption.  _Multiagent and Grid Systems_ 17, 1 (2021), 39–58.  www.scopus.com
  * Calabria et al. (2018) F. A. Calabria, J. T. Saraiva, and A. P. Rocha. 2018\.  Improving the Brazilian electricity market: How to replace the centralized dispatch by decentralized market-based bidding.  _Journal of Energy Markets_ 11, 2 (2018), 83–106.  www.scopus.com Cited By :1.
  * Chrysanthopoulos and Papavassilopoulos (2016) N. Chrysanthopoulos and G. P. Papavassilopoulos. 2016.  Learning optimal strategies in a stochastic game with partial information applied to electricity markets. In _IET Conference Publications_ , Vol. 2016.  www.scopus.com Cited By :1.
  * Cortes and Vapnik (1995) Corinna Cortes and Vladimir Vapnik. 1995.  Support-Vector Networks.  _Machine Learning_ 20, 3 (1995), 273–297.  https://doi.org/10.1023/A:1022627411411 arXiv:arXiv:1011.1669v3
  * Dempe (2002) Stephan Dempe. 2002\.  _Foundations of bilevel programming_.  Springer Science & Business Media.
  * Deng and Wu (2021) C. Deng and K. Wu. 2021\.  Residential demand response strategy based on deep deterministic policy gradient.  _Processes_ 9, 4 (2021).  www.scopus.com
  * Duan et al. (2018) W. Duan, Y. Zheng, and T. Zhang. 2018.  Agent-based Integrated Resource Strategic Planning Model. In _2nd IEEE Conference on Energy Internet and Energy System Integration, EI2 2018 - Proceedings_.  www.scopus.com
  * El Bourakadi et al. (2019) D. El Bourakadi, A. Yahyaouy, and J. Boumhidi. 2019\.  _Extreme learning machine based multi-agent system for microgrid energy management_. Advances in Intelligent Systems and Computing, Vol. 912.  30–39 pages.  www.scopus.com
  * Esmaeili Aliabadi et al. (2017) D. Esmaeili Aliabadi, M. Kaya, and G. Sahin. 2017.  Competition, risk and learning in electricity markets: An agent-based simulation study.  _Applied Energy_ 195 (2017), 1000–1011.  www.scopus.com Cited By :28.
  * Feng et al. (2018) H. Feng, Z. Yang, Y. Zheng, F. Ye, X. Zhang, and X. Shi. 2018.  Intelligent Agent Based Bidding Simulation Method for Multi-input Decision Factors of Power Suppliers.  _Dianli Xitong Zidonghua/Automation of Electric Power Systems_ 42, 23 (2018), 72–77.  www.scopus.com Cited By :4.
  * Filho et al. (2016) N. S. C. Filho, F. O. S. Saraiva, and V. L. Paucar. 2016\.  Comparative analysis of game theory application to various types of auctions in electric markets. In _CACIDI 2016 - Congreso Aergentino de Ciencias de la Informatica y Desarrollos de Investigacion_.  www.scopus.com Cited By :1.
  * Foruzan et al. (2018) E. Foruzan, L. . Soh, and S. Asgarpoor. 2018.  Reinforcement Learning Approach for Optimal Distributed Energy Management in a Microgrid.  _IEEE Transactions on Power Systems_ 33, 5 (2018), 5749–5758.  www.scopus.com Cited By :61.
  * Fraunholz et al. (2021) C. Fraunholz, E. Kraft, D. Keles, and W. Fichtner. 2021\.  Advanced price forecasting in agent-based electricity market simulation.  _Applied Energy_ 290 (2021).  www.scopus.com Cited By :1.
  * Gaivoronskaia and Tsyplakov (2018) E. A. Gaivoronskaia and A. A. Tsyplakov. 2018.  Using a modified EREV—roth algorithm in an agent-based electricity market Model 2.  _Zhournal Novoi Ekonomicheskoi Associacii_ 39, 3 (2018), 55–83.  www.scopus.com Cited By :1.
  * Gao et al. (2018) Y. Gao, X. Zhou, J. Ren, X. Wang, and D. Li. 2018.  Double layer dynamic game bidding mechanism based on multi-agent technology for virtual power plant and internal distributed energy resource.  _Energies_ 11, 11 (2018).  www.scopus.com Cited By :5.
  * Gomes et al. (2021) I. L. R. Gomes, R. Melicio, and V. M. F. Mendes. 2021\.  A novel microgrid support management system based on stochastic mixed-integer linear programming.  _Energy_ 223 (2021).  www.scopus.com Cited By :3.
  * Goncalves et al. (2019) C. Goncalves, M. Ribeiro, J. Viana, R. Fernandes, J. Villar, R. Bessa, G. Correia, J. Sousa, V. Mendes, and A. C. Nunes. 2019.  Explanatory and causal analysis of the MIBEL electricity market spot price. In _2019 IEEE Milan PowerTech 2019_.  www.scopus.com
  * Hartigan and Wong (1979) J. A. Hartigan and M. A. Wong. 1979.  Algorithm AS 136: A K-Means Clustering Algorithm.  _Journal of the Royal Statistical Society C_ 28, 1 (1979), 100–108.  https://doi.org/10.2307/2346830
  * Hill et al. (1994) Tim Hill, Leorey Marquez, Marcus O’Connor, and William Remus. 1994\.  Artificial neural network models for forecasting and decision making.  _International Journal of Forecasting_ 10, 1 (1994), 5–15.  https://doi.org/10.1016/0169-2070(94)90045-0 arXiv:arXiv:1011.1669v3
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long Short-Term Memory.  _Neural Computation_ 9, 8 (1997), 1735–1780.  https://doi.org/10.1162/neco.1997.9.8.1735
  * Huang et al. (2006) Guang-Bin Huang, Qin-Yu Zhu, and Chee-Kheong Siew. 2006\.  Extreme learning machine: theory and applications.  _Neurocomputing_ 70, 1-3 (2006), 489–501.
  * Huang and Guo (2016) X. Huang and R. Guo. 2016\.  A multi-agent model of generation expansion planning in electricity market.  _Dianli Xitong Baohu yu Kongzhi/Power System Protection and Control_ 44, 24 (2016), 1–8.  www.scopus.com Cited By :3.
  * Imran et al. (2020) K. Imran, J. Zhang, A. Pal, A. Khattak, K. Ullah, and S. M. Baig. 2020.  Bilateral negotiations for electricity market by adaptive agent-tracking strategy.  _Electric Power Systems Research_ 186 (2020).  www.scopus.com Cited By :3.
  * IRENA (2018) IRENA. 2018.  _Renewable Power Generation Costs in 2017. IRENA - International Renewable Energy Agency_.  160 pages.
  * Kell et al. (2019) Alexander J.M. Kell, Matthew Forshaw, and A. Stephen McGough. 2019.  Optimising energy and overhead for large parameter space simulations.  _2019 10th International Green and Sustainable Computing Conference, IGSC 2019_ (2019).  https://doi.org/10.1109/IGSC48788.2019.8957205
  * Kell et al. (2020) Alexander J. M. Kell, Matthew Forshaw, and A. Stephen McGough. 2020.  Long-term electricity market agent based model validation using genetic algorithm based optimization.  (2020), 1–13.  arXiv:2005.10346 http://arxiv.org/abs/2005.10346
  * Kelley (1999) Carl T Kelley. 1999\.  _Iterative methods for optimization_.  SIAM.
  * Kennedy and Eberhart (1995) James Kennedy and Russell Eberhart. 1995.  Particle swarm optimization. In _Proceedings of ICNN’95-international conference on neural networks_ , Vol. 4. IEEE, 1942–1948.
  * Kim and Lee (2020) J. . Kim and B. Lee. 2020\.  Automatic P2P energy trading model based on reinforcement learning using long short-term delayed reward.  _Energies_ 13, 20 (2020).  www.scopus.com Cited By :2.
  * Kiran and Vijaya Chandrakala (2020) P. Kiran and K. R. M. Vijaya Chandrakala. 2020.  _Variant Roth-Erev Reinforcement Learning Algorithm-Based Smart Generator Bidding as Agents in Electricity Market_. Advances in Intelligent Systems and Computing, Vol. 1048.  981–989 pages.  www.scopus.com Cited By :2.
  * Kohonen (1990) T. Kohonen. 1990\.  The self-organizing map.  _Proc. IEEE_ 78, 9 (1990), 1464–1480.  https://doi.org/10.1109/5.58325
  * Lagoudakis and Parr (2003) Michail G Lagoudakis and Ronald Parr. 2003.  Least-squares policy iteration.  _The Journal of Machine Learning Research_ 4 (2003), 1107–1149.
  * Liang et al. (2020) Y. Liang, C. Guo, Z. Ding, and H. Hua. 2020\.  Agent-Based Modeling in Electricity Market Using Deep Deterministic Policy Gradient Algorithm.  _IEEE Transactions on Power Systems_ 35, 6 (2020), 4180–4192.  www.scopus.com Cited By :4.
  * Lillicrap et al. (2016) Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2016.  Continuous control with deep reinforcement learning.  _4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings_ (2016).  arXiv:1509.02971
  * Lin et al. (2019) F. Lin, J. Wei, X. Ren, Y. Yao, Y. Wang, and K. Zhang. 2019.  Herding Behavior of Electricity Retailers in the Monthly Electricity Market: Model and Simulation. In _Asia-Pacific Power and Energy Engineering Conference, APPEEC_ , Vol. 2019-December.  www.scopus.com
  * Liu et al. (2021) D. Liu, Y. Gao, W. Wang, and Z. Dong. 2021\.  Research on bidding strategy of thermal power companies in electricity market based on Multi-Agent Deep Deterministic Policy Gradient.  _IEEE Access_ (2021).  www.scopus.com
  * Liu et al. (2020) Y. Liu, D. Zhang, C. Deng, and X. Wang. 2020\.  Deep Reinforcement Learning Approach for Autonomous Agents in Consumer-centric Electricity Market. In _2020 5th IEEE International Conference on Big Data Analytics, ICBDA 2020_. 37–41.  www.scopus.com Cited By :1.
  * Lowe et al. ([n.d.]) Ryan Lowe, Aviv Tamar, L G Jan, Jean Harb, Pieter Abbeel, and Igor Mordatch. [n.d.].  Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments.  ([n. d.]).  arXiv:arXiv:1706.02275v3
  * Machado et al. (2019) M. R. Machado, M. Kenichi Fujii, C. De Oliveira Ribeiro, and E. E. Rego. 2019.  An Agent-Based Model Applied to Brazilian Wind Energy Auctions.  _IEEE Latin America Transactions_ 17, 5 (2019), 865–874.  www.scopus.com
  * Manabe et al. (2016) Y. Manabe, T. Funabashi, T. Kato, M. Kurimoto, and Y. Suzuoki. 2016.  Probabilistic investment strategy modeling for generation expansion planning. In _19th Power Systems Computation Conference, PSCC 2016_.  www.scopus.com Cited By :1.
  * Maqbool et al. (2019) A. S. Maqbool, J. Baetens, S. Lotfi, L. Vandevelde, and G. van Eetvelde. 2019.  Assessing financial and flexibility incentives for integrating wind energy in the grid via agent-based modeling.  _Energies_ 12, 22 (2019).  www.scopus.com Cited By :2.
  * Mbuwir et al. (2020) B. V. Mbuwir, D. Geysen, F. Spiessens, and G. Deconinck. 2020\.  Reinforcement learning for control of flexibility providers in a residential microgrid.  _IET Smart Grid_ 3, 1 (2020), 98–107.  www.scopus.com Cited By :4.
  * Meng et al. (2018) F. Meng, X. . Zeng, Y. Zhang, C. J. Dent, and D. Gong. 2018.  An integrated optimization + learning approach to optimal dynamic pricing for the retailer with multi-type customers in smart grids.  _Information Sciences_ 448-449 (2018), 215–232.  www.scopus.com Cited By :10.
  * Mitchell (1995) M. Mitchell. 1995\.  Genetic Algorithms: An Overview.  (1995).
  * Mitchell (1997) Tom Mitchell. 1997\.  Machine learning.  (1997).
  * Murtagh and Contreras (2012) Fionn Murtagh and Pedro Contreras. 2012.  Algorithms for hierarchical clustering: an overview.  _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ 2, 1 (2012), 86–97.
  * Myerson (1997) Roger B Myerson. 1997\.  _Game theory: analysis of conflict_.  Harvard university press.
  * Najafi et al. (2018) S. Najafi, S. Talari, A. S. Gazafroudi, M. Shafie-Khah, J. M. Corchado, and J. P. S. Catalão. 2018\.  _Decentralized control of DR using a multi-agent method_. Studies in Systems, Decision and Control, Vol. 145.  233–249 pages.  www.scopus.com Cited By :11.
  * Naseri et al. (2021) N. Naseri, S. Talari, W. Ketter, and J. Collins. 2021\.  Sustainable Smart Electricity Markets: Designing Tariffs Using Multi-Agent Simulation: Poster. In _e-Energy 2021 - Proceedings of the 2021 12th ACM International Conference on Future Energy Systems_. 290–291.  www.scopus.com
  * Nunna et al. (2020) H. S. V. S. K. Nunna, A. Sesetti, A. K. Rathore, and S. Doolla. 2020.  Multiagent-Based Energy Trading Platform for Energy Storage Systems in Distribution Systems with Interconnected Microgrids.  _IEEE Transactions on Industry Applications_ 56, 3 (2020), 3207–3217.  www.scopus.com Cited By :13.
  * Opalinski and Dudek (2016) A. Opalinski and G. Dudek. 2016.  Electricity demand prediction by multi-agent system with history-based weighting. In _Proceedings - 2016 17th International Scientific Conference on Electric Power Engineering, EPE 2016_.  www.scopus.com
  * Peng et al. (2021) Wei Peng, Gokul Iyer, Valentina Bosetti, Vaibhav Chaturvedi, James Edmonds, Allen A. Fawcett, Stéphane Hallegatte, David G. Victor, Detlef van Vuuren, and John Weyant. 2021.  Climate policy models need to get real about people - here’s how.  _Nature_ 594, 7862 (2021), 174–176.  https://doi.org/10.1038/d41586-021-01500-2
  * Pfenninger et al. (2014) Stefan Pfenninger, Adam Hawkes, and James Keirstead. 2014\.  Energy systems modeling for twenty- first century energy challenges.  _Renewable and Sustainable Energy Reviews_ 33 (2014), 74–86.
  * Pinto and Falcão-Reis (2019) T. Pinto and F. Falcão-Reis. 2019.  Strategic participation in competitive electricity markets: Internal versus sectorial data analysis.  _International Journal of Electrical Power and Energy Systems_ 108 (2019), 432–444.  www.scopus.com Cited By :3.
  * Pinto et al. (2019) T. Pinto, I. Praça, Z. Vale, and C. Santos. 2019\.  _Collaborative reinforcement learning of energy contracts negotiation strategies_. Communications in Computer and Information Science, Vol. 1047.  202–210 pages.  www.scopus.com
  * Pinto et al. (2016) T. Pinto, T. M. Sousa, I. Praça, Z. Vale, and H. Morais. 2016.  Support Vector Machines for decision support in electricity markets’ strategic bidding.  _Neurocomputing_ 172 (2016), 438–445.  www.scopus.com Cited By :30.
  * Pinto and Vale (2019) T. Pinto and Z. Vale. 2019\.  _Contextual simulated annealing q-learning for pre-negotiation of agent-based bilateral negotiations_. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 11804 LNAI.  519–531 pages.  www.scopus.com Cited By :1.
  * Poplavskaya et al. (2020) K. Poplavskaya, J. Lago, and L. de Vries. 2020.  Effect of market design on strategic bidding behavior: Model-based analysis of European electricity balancing markets.  _Applied Energy_ 270 (2020).  www.scopus.com Cited By :5.
  * Purushothaman and Chandrakala (2020) K. Purushothaman and V. Chandrakala. 2020.  Roth-Erev Reinforcement Learning Approach for Smart Generator Bidding towards Long Term Electricity Market Operation Using Agent Based Dynamic Modeling.  _Electric Power Components and Systems_ 48, 3 (2020), 256–267.  www.scopus.com Cited By :1.
  * Quinlan (1987) J. Ross Quinlan. 1987\.  Simplifying decision trees.  _International journal of man-machine studies_ 27, 3 (1987), 221–234.
  * Rashedi et al. (2016) N. Rashedi, M. A. Tajeddini, and H. Kebriaei. 2016\.  Markov game approach for multi-agent competitive bidding strategies in electricity market.  _IET Generation, Transmission and Distribution_ 10, 15 (2016), 3756–3763.  www.scopus.com Cited By :16.
  * Riedmiller (2005) Martin Riedmiller. 2005\.  Neural fitted Q iteration–first experiences with a data efficient neural reinforcement learning method. In _European conference on machine learning_. Springer, 317–328.
  * Ringler (2012) Philipp Ringler. 2012\.  PowerACE Agent-based simulation of electricity markets.  _MOCAP Workshop on Modelling Carbon Prices - Interacting agent networks & Strategies under risk_ (2012).
  * Ringler et al. (2016) Philipp Ringler, Dogan Keles, and Wolf Fichtner. 2016\.  Agent-based modelling and simulation of smart electricity grids and markets - A literature review.  _Renewable and Sustainable Energy Reviews_ 57 (2016), 205–215.
  * Roth AE and Erev I (1995) Roth AE and Erev I. 1995.  Learning in extensive-form games: Experimental data and simple dynamic models in the intermediate term.  _Games and Economic Behavior_ 8, 1 (1995), 164–212.
  * Seal (1967) Hilary L Seal. 1967\.  Studies in the History of Probability and Statistics. XV The historical development of the Gauss linear model.  _Biometrika_ 54, 1-2 (1967), 1–24.
  * Shafie-Khah et al. (2020) M. Shafie-Khah, S. Talari, F. Wang, and J. P. S. Catalão. 2020.  Decentralised demand response market model based on reinforcement learning.  _IET Smart Grid_ 3, 5 (2020), 713–721.  www.scopus.com
  * Silver et al. (2014) David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. 2014\.  Deterministic policy gradient algorithms. In _International conference on machine learning_. PMLR, 387–395.
  * Skiba et al. (2016) L. Skiba, A. Maaz, and A. Moser. 2016.  Agent-based price simulation of the day-ahead- spot-market and markets for control reserve power. In _International Conference on the European Energy Market, EEM_ , Vol. 2016-July.  www.scopus.com Cited By :3.
  * Sousa and Tome Saraiva (2020) J. C. Sousa and J. Tome Saraiva. 2020.  Simulation of Hydro Power Plants in the Iberian Market using an Agent-Based Model and Q-Learning. In _International Conference on the European Energy Market, EEM_ , Vol. 2020-September.  www.scopus.com
  * Staudt et al. (2018) P. Staudt, J. Gärttner, and C. Weinhardt. 2018.  Assessment of market power in local electricity markets with regards to competition and tacit collusion. In _MKWI 2018 - Multikonferenz Wirtschaftsinformatik_ , Vol. 2018-March. 912–923.  www.scopus.com Cited By :3.
  * Tang et al. (2021) C. Tang, L. Zhang, F. Liu, and Y. Li. 2021\.  Research on Pricing Mechanism of Electricity Spot Market Based on Multi-agent Reinforcement Learning (Part I): Bi-level Optimization Model for Generators Under Different Pricing Mechanisms.  _Zhongguo Dianji Gongcheng Xuebao/Proceedings of the Chinese Society of Electrical Engineering_ 41, 2 (2021), 536–552.  www.scopus.com
  * Tibshirani (1996) Robert Tibshirani. 1996\.  Regression Shrinkage and Selection Via the Lasso.  _Journal of the Royal Statistical Society: Series B (Methodological)_ 58, 1 (1996), 267–288.  https://doi.org/10.1111/j.2517-6161.1996.tb02080.x
  * Tomin et al. (2020) N. Tomin, V. Kurbatsky, V. Borisov, and S. Musalev. 2020\.  Development of Digital Twin for Load Center on the Example of Distribution Network of an Urban District. In _E3S Web of Conferences_ , Vol. 209.  www.scopus.com
  * Tschiatschek et al. (2014) Sebastian Tschiatschek, Karin Paul, and Franz Pernkopf. 2014\.  Integer Bayesian network classifiers.  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ 8726 LNAI, PART 3 (2014), 209–224.  https://doi.org/10.1007/978-3-662-44845-8_14
  * Viehmann et al. (2021) J. Viehmann, S. Lorenczik, and R. Malischek. 2021\.  Multi-unit multiple bid auctions in balancing markets: An agent-based Q-learning approach.  _Energy Economics_ 93 (2021).  www.scopus.com
  * Walker and Duncan (1967) Strother H Walker and David B Duncan. 1967.  Estimation of the probability of an event as a function of several independent variables.  _Biometrika_ 54, 1-2 (1967), 167–179.
  * Wang et al. (2019) J. Wang, J. Wu, and Y. Che. 2019.  Agent and system dynamics-based hybrid modeling and simulation for multilateral bidding in electricity market.  _Energy_ 180 (2019), 444–456.  www.scopus.com Cited By :8.
  * Watkins and Dayan (1992) Christopher J. C. H. Watkins and Peter Dayan. 1992.  Q-Learning.  _Machine Learning_ 292 (1992), 179–184.  https://doi.org/10.4018/978-1-59140-993-9.ch026
  * Xu et al. (2019) S. Xu, X. Chen, J. Xie, S. Rahman, J. Wang, H. Hui, and T. Chen. 2019\.  Agent-based modeling and simulation of the electricity market with residential demand response.  _CSEE Journal of Power and Energy Systems_ PP, 99 (2019).  www.scopus.com
  * Ye et al. (2019) Y. Ye, D. Qiu, J. Li, and G. Strbac. 2019\.  Multi-Period and Multi-Spatial Equilibrium Analysis in Imperfect Electricity Markets: A Novel Multi-Agent Deep Reinforcement Learning Approach.  _IEEE Access_ 7 (2019), 130515–130529.  www.scopus.com Cited By :11.
  * Čaušević et al. (2017) S. Čaušević, M. Warnier, and F. M. T. Brazier. 2017\.  Dynamic, self-organized clusters as a means to supply and demand matching in large-scale energy systems. In _Proceedings of the 2017 IEEE 14th International Conference on Networking, Sensing and Control, ICNSC 2017_. 568–573.  www.scopus.com Cited By :5.