# Experiences from the MediaEval Predicting Media Memorability Task

Alba G. Seco de Herrera
University of Essex
Colchester, UK CO4 3SQ
<EMAIL_ADDRESS>
Mihai Gabriel Constantin
University Politehnica
of Bucharest, Romania
Claire-Hélène Demarty
InterDigital
France
Camilo Fosco
Massachusetts Institute of
Technology Cambridge, USA
Sebastian Halder
University of Essex
Colchester, UK CO4 3SQ
Graham Healy
Dublin City University
Ireland
Bogdan Ionescu
University Politehnica
of Bucharest, Romania
Ana Matran-Fernandez
University of Essex
Colchester, UK CO4 3SQ
Alan F. Smeaton
Dublin City University
Ireland
Mushfika Sultana
University of Essex
Colchester, UK CO4 3SQ
Lorin Sweeney
Dublin City University
Ireland

###### Abstract

The Predicting Media Memorability task in the MediaEval evaluation campaign
has been running annually since 2018 and several different tasks and data sets
have been used in this time. This has allowed us to compare the performance of
many memorability prediction techniques on the same data and in a reproducible
way and to refine and improve on those techniques. The resources created to
compute media memorability are now being used by researchers well beyond the
actual evaluation campaign. In this paper we present a summary of the task,
including the collective lessons we have learned for the research community.

##  1 Introduction

The Benchmarking Initiative for Multimedia Evaluation
(MediaEval)111https://multimediaeval.github.io/ offers challenges related to
multimedia analysis, retrieval and exploration. Since 2018, the “Predicting
Media Memorability” task has been organised annually as part of MediaEval.
This task aims to understand what makes a video more, or less, memorable.
Several articles describe the overall evolution of the set of sub-tasks and
the various approaches that were used to create the resources [14, 16, 8, 6,
4, 3, 9, 2]. The data sets used and the sub-tasks in the Predicting Media
Memorability task have evolved over the years and are described in Sections
2–4. An overview of the main techniques that achieved best results is given in
Section 5. The main lessons learned are described in Section 6 and conclusions
are given in Section 7.

##  2 Sub-tasks over the years

Table 1: Overview of the sub-tasks performed over the years. Sub-task | 2018 | 2019 | 2020 | 2021 | 2022
---|---|---|---|---|---
Short-term video-based prediction | ✓ | ✓ | ✓ | ✓ | ✓
Long-term video-based prediction | ✓ | ✓ | ✓ | ✓ |
Generalisation |  |  |  | ✓ | ✓
EEG-based prediction |  |  |  | ✓ | ✓

Table 1 summarises the sub-tasks which have run each year in MediaEval. The
_video-based prediction_ sub-tasks require participants to automatically
predict memorability scores for videos, i.e., the probability for a video to
be remembered in the short- or long-term. The _generalisation_ sub-task aims
to test if the prediction approaches can be generalised to other sources of
video data. Specifically, participants are asked to train their systems on one
dataset and test them on another dataset. Finally, the _electroencephalogram
(EEG) based prediction_ sub-task involves the use of EEG signals from users
who watched the videos, to predict whether a viewer will remember a video or
not (with or without augmenting the EEG data with features extracted from the
videos). Task participants are provided with an extensive data set of videos
with memorability annotations, related information, pre-extracted state-of-
the-art visual features, and EEG recordings (see Section 4).

Figure 1: Annual participation in the MediaEval Predicting Media Memorability
task. Numbers for 2022 are temporary as the deadline for participation is
still open.

##  3 Participation over the years

In Figure 1 the number of groups that registered for the task and that
participated are shown across time. Besides the larger availability of
benchmarks and data sets nowadays, the task remains attractive for
participants. This shows the impact that building data collections and tasks
and sharing these resources with other researchers can have on research
challenges like computing memorability.

##  4 Data over the years

This section describes the evolution of the data released over the years,
followed by a description on how memorability scores have been calculated.

###  4.1 Data sets

Three video data sets and two EEG-based data sets have been released as part
of the MediaEval Predicting Media Memorability task, including pre-extracted
features for each.

Figure 2: A sample of frames (and corresponding memorability scores) from
videos in the VideoMem data set.

##### VideoMem

[2] is composed of 10,000 7-second long soundless videos shared under a
license allowing their use and redistribution in the context of MediaEval.
They were extracted from high quality cinematic raw stock footage used by
professionals when creating content. They are varied and contain different
scenes types. This data set was used in the 2018, 2019 and 2022 editions of
the task. In the 2018 and 2019 editions, the data set was split into 80% for
training and development and 20% for testing. In the 2022 edition, 70% of the
videos were released as part of the training set, 15% for validation and 15%
for testing. Figure 2 shows a sample of frames from some of the videos in this
data set and their corresponding memorability scores.

##### TRECVid 2019 Video-to-Text

[1] contains 6,000 videos with sound, shared under Creative Commons licenses
that allow their redistribution. This data was used in the 2020 and 2021
MediaEval Predicting Media Memorability tasks and was divided into three sets.
In the 2020 edition, the training set contained 590 videos, the development
set 410, and the test set 500. In the 2021 edition, the training set contained
588 videos, the development set 1,116, and the test set 500. The data contains
Twitter Vine videos where various actions are performed. These videos have
much more action happening compared to those in the _Memorability 2018_ data
set, and thus they correspond to more generic use cases. Each video consists
of a coherent unit in terms of meaning.

##### Memento10K

[11] contains 10,000 3-second long videos depicting in-the-wild scenes, with
their associated short-term memorability scores, memorability decay values,
action labels, and 5 accompanying captions. 7,000 videos were released as a
training set and 1,500 were provided for validation. The last 1,500 videos
were used as the test set for scoring submissions. This data set was used in
the 2021 and 2022 editions of the MediaEval Predicting Media Memorability
task.

##### 2021 EEG data set

[16] was collected as part of a pilot study to check the feasibility of
finding neural correlates of memory encoding in a short-term memorability
task. The videos belong to the TRECVid data set. Both encoding and decoding
were recorded in a short-term memorability experiment, but it resulted in a
highly imbalanced data set in which the majority of the video clips were
correctly remembered, limiting possible analyses and prompting a refinement in
the protocol for the 2022 edition of the EEG task.

##### EEGMem

[14] comprises pre-extracted features from EEG recordings for 12 subjects
captured while they watched a subset of the Memento10k [11] videos. Volunteers
watched the same videos a second time through a custom-built online portal
between 24–72 hours after the video-EEG recording session, where they were
required to indicate for each video whether or not they recognised it,
providing binary ground truth annotations of memory encoding for long-term
memorability. The EEG subtask prompts participants to predict whether a given
participant will remember a specific video based on their EEG signals, and
includes a cross-subject/transfer learning prompt with data from three
subjects being left out of the training set.

###  4.2 Measuring video memorability

Two slightly different protocols were used to collect memorability scores
depending on the data set [2, 11]. In both cases, participants watched a
sequence of videos and were asked to press the space bar when they recall
seeing a repeated video. For the long-term memorability, participants viewed a
new sequence of videos after 24 to 72 hours.

For the VideoMem collection, the memorability scores were computed from 9,402
participants for short-term, and 3,246 participants for long-term
memorability. On average, 38 (at least 30) annotations were collected for each
video in the short-term task, whereas in the long-term task, a mean of 13
annotations (at least 9) were collected. For the TRECVid 2019 Video-to-Text
data set, a minimum of 14 annotations per video in the short-term memorability
step were collected with a mean of 22. For long-term memorability a minimum of
3 and an average of 7 annotations/video were collected. For the Memento10K
only short-term memorability annotations were acquired. The memorability
scores were computed with 90 annotations/video on average.

####  4.2.1 Memorability scores over the years

In the first four editions of the task, each memorability score was calculated
as the percentage of correct recognitions among participants (termed _raw
scores_). In the 2018 and 2019 editions, a linear correction was applied to
take into account the different memory retention durations for the short-term
scores. The reason for this was that each video repetition happened after
different time intervals (between 45–100 videos after the first occurrence)
and it is well known that memory decreases linearly with the increase in
retention duration. This normalisation aimed to maximise the
representativeness of the memory performance after the maximal interval of 100
videos. This linear correction was not applied for long-term scores as no
specific relationship was observed between the retention durations and the raw
scores. The normalisation was not used after the 2020 edition.

In 2021, a second memorability score was introduced based on a different
normalisation. This normalised score was based on the assumption that the hit
rate for recollection should decay linearly over short timeframes. Also, each
video’s decay rate may be different, and this is taken into account by using
an alpha parameter. For more details regarding this process see [11]. In the
last edition of the task, in 2022, this normalisation was established.

##  5 Techniques used

Details regarding the methods employed by participants and the results from
those methods can be found in the proceedings of the MediaEval
workshop222http://ceur-ws.org/. With over 30 published papers during the 4
completed editions of the Predicting Media Memorability task and several
different methods used by each participating team, some general trends can be
noticed with regards to the methods employed by participants.

We observe that fusion methods (whether they process the same modality with
more than one system or make use of multi-modal information), and in
particular late fusion methods, tend to lead to the top-performing results
amongst our participants, regardless of the scale of memorability (short- or
long-term) or data set analysed. We conclude that this may be due to better
modality representation when creating individual systems for each modality, or
even when analysing different aspects of the same modality. This would allow
single end-to-end systems to better contribute to a final fusion scheme, and
to create better fusion schemes.

Additionally, some participating teams showed an interest in exploring
memorability by analysing and predicting correlated concepts. This is an
useful approach when studying subjective concepts like memorability, and in
the current literature some authors center their study around finding the
positive or negative correlation between memorability and other concepts like
interestingness or aesthetic appeal [7]. While the results are varied and this
type of approach was not always successful, we still consider these types of
studies of great value to understand what makes an audiovisual clip more or
less memorable. Some examples in this case would be represented by the study
of perplexity [13], exploring video semantics [10] or using aesthetic-based
features [5].

Finally, it is encouraging to see that over time there has been an increase of
performance of the methods developed over the editions of this competition. We
can observe this by analysing the results on VideoMem and TRECVid, which ran
for consecutive editions from 2018 to 2021. For example, for the VideoMem
data, used during the 2018 and 2019 editions of the task, all participating
teams from 2019 scored above the average performance from 2018. Furthermore,
two teams had better performance than the top performance from the previous
year, in both short- and long-term memorability. A similar trend can be
noticed for the TRECVid data, when all teams from 2021 had better results
compared with the average from 2020 for short-term memorability, and only one
team being behind the average for long-term memorability. Regarding the top
results, three teams outperformed the top results from the previous year for
short-term and one team for long-term memorability. We will continue
monitoring these trends for the upcoming 2022 edition and beyond.

##  6 Lessons Learned

In the first two editions of the task, VideoMem was used. This data set
contains different scene types, but the videos have limited motion. We found
that including motion information is crucial when studying videos compared to
only studying images. Therefore the TRECVid 2019 Video-to-Text data set was
used in subsequent editions of the task, since it contains more action-rich
video content than VideoMem. Furthermore, for this data set, sound and pre-
computed sound features were also provided to task participants. While no
conclusive findings were established, the results from some participants [15,
12] suggest that the audio modality provides a degree of useful information
during video memorability prediction.

Independently of the source of data, prediction results are significantly
lower for long-term memorability than for short-term. One explanation might be
the lower number of annotations for long-term scores, but there are other
factors at play as well. Attempts to predict long-term scores from the short-
term ones were not conclusive, leading to the conclusion that short-term and
long-term memories are induced by different characteristics.

Not only does the source of the data influence the quality of the results;
memorability annotation quality has a strong impact too. An analysis of the
VideoMem data set shows a bias in the distribution of memorability scores
towards higher score values. This might be a clue that the task of recognising
videos is rather simple as they contain different kinds of information (i.e.,
motion, color, objects…). Furthermore, the number of annotations is really
relevant. For the VideoMem data set, annotation consistency was computed as
explained in [2]. It reaches 0.616 (resp. 0.364) for the short-term (resp.
long-term) task (higher is better). These values correspond to respectively
strong and moderate correlations according to the usual Spearman scale of
interpretation, validating that the number of annotations for the short-term
task is sufficient and that a larger number of annotations for the long-term
task would have been beneficial.

The TRECVid 2019 Video-to-Text data set has a lower number of annotations and
therefore it is very difficult to achieve a strong correlations in the scores.
Due to this, it was excluded from the 2022 edition of the task. Unfortunately,
annotation acquisition is costly, tedious, and time–consuming. Instead, the
Memento10k collection was released since it contains the largest number of
annotations/video (an average of 90), although unfortunately it does not
contain long-term annotations and the short-term annotations were created
without using the sound of the videos. Therefore, in the interest of clarity,
standardisation, and the facilitation of more directed inquiry, in the 2022
edition, the scope of the tasks was narrowed, forgoing raw and long-term
memorability scores in favour of normalised short-term memorability scores.

As for the EEG data sets, an error in the data acquisition protocol in the
2021 edition resulted in a heavily imbalanced data set that did not allow for
it to be used to make predictions, as most participants had very high recall
for the videos (some of them reaching 100% recall). However, an analysis of
the EEG data using time-frequency analyses showed (by comparing brain
responses from first vs. second presentation of the videos that were repeated)
that there are differences that can be used for successful prediction of
memorability, thus motivating the acquisition of EEGMem for 2022.

Finally, providing pre-computed features has favoured the participation of
more researchers in the task because it helps approach the challenges from
different perspectives giving room for a deeper study of the task. The
MediaEval workshop is organised annually and supports open discussion to plan
tasks and also evaluate procedures for the future. It has improved motivation
across research groups which has turned into many international groups being
involved on the organisation of the task favouring its success.

##  7 Conclusions

This paper gives a brief overview of the five years of challenges of the
MediaEval Predicting Media Memorability Task. It mainly focuses on overviews
of how the data, the sub-tasks and the techniques evolved over the years. We
also highlight some lessons learned through the organisation of this benchmark
over the years and the support of the organisers and the participants.
Although there have been some significant advances, the prediction of media
memorability remains an open scientific challenge.

## Acknowledgments and Disclosure of Funding

Financial support was provided by the University of Essex Faculty of Science
and Health Research Innovation and Support Fund and by Science Foundation
Ireland under Grant Number SFI/12/RC/2289_P2, cofunded by the European
Regional Development Fund. Financial support was also provided under project
AI4Media, a European Excellence Centre for Media, Society and Democracy, H2020
ICT-48-2020, grant #951911.

## References

  * [1] G. Awad, A. A. Butt, K. Curtis, Y. Lee, J. Fiscus, A. Godil, A. Delgado, J. Zhang, E. Godard, L. Diduch, et al.  TRECVID 2019: An evaluation campaign to benchmark video activity detection, video captioning and matching, and video search & retrieval.  In Proceedings of TRECVID 2019. NIST, USA, 2019.
  * [2] R. Cohendet, C.-H. Demarty, N. Q. Duong, and M. Engilberge.  VideoMem: Constructing, analyzing, predicting short-term and long-term video memorability.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2531–2540, 2019.
  * [3] R. Cohendet, C.-H. Demarty, N. Q. Duong, M. Sjöberg, B. Ionescu, and T.-T. Do.  MediaEval 2018: Predicting media memorability task.  In MediaEval Multimedia Benchmark Workshop Working Notes. CEUR, 2018\.
  * [4] M. G. Constantin, B. Ionescu, C.-H. Demarty, N. Q. Duong, X. Alameda-Pineda, and M. Sjóberg.  The predicting media memorability task at MediaEval 2019.  In MediaEval Multimedia Benchmark Workshop Working Notes. CEUR, 2019\.
  * [5] M. G. Constantin, C. Kang, G. Dinu, F. Dufaux, G. Valenzise, and B. Ionescu.  Using aesthetics and action recognition-based networks for the prediction of media memorability.  In MediaEval Multimedia Benchmark Workshop Working Notes, 2019.
  * [6] A. Garcia Seco de Herrera, R. Savran Kiziltepe, J. Chamberlain, M. G. Constantin, C.-H. Demarty, F. Doctor, B. Ionescu, and A. F. Smeaton.  Overview of MediaEval 2020 predicting media memorability task: What makes a video memorable?  In Working Notes Proceedings of the MediaEval 2020 Workshop. CEUR, December 2020.
  * [7] M. Gygli, H. Grabner, H. Riemenschneider, F. Nater, and L. Van Gool.  The interestingness of images.  In Proceedings of the IEEE international conference on computer vision, pages 1633–1640, 2013.
  * [8] R. S. Kiziltepe, M. G. Constantin, C.-H. Demarty, G. Healy, C. Fosco, A. García Seco de Herrera, S. Halder, B. Ionescu, A. Matran-Fernandez, A. F. Smeaton, and L. Sweeney.  Overview of the MediaEval 2021 predicting media memorability task.  In MediaEval Multimedia Benchmark Workshop Working Notes. CEUR, December 2021.
  * [9] R. S. Kiziltepe, L. Sweeney, M. G. Constantin, F. Doctor, A. G. S. de Herrera, C.-H. Demarty, G. Healy, B. Ionescu, and A. F. Smeaton.  An annotated video dataset for computing video memorability.  Data in Brief, 39:107671, 2021.
  * [10] R. Kleinlein, C. Luna-Jiménez, and F. Fernández-Martínez.  THAU-UPM at MediaEval 2021: From Video Semantics To Memorability Using Pretrained Transformers.  In MediaEval Multimedia Benchmark Workshop Working Notes, 2021.
  * [11] A. Newman, C. Fosco, V. Casser, A. Lee, B. McNamara, and A. Oliva.  Multimodal memorability: Modeling effects of semantics and decay on video memorability.  In A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, editors, Computer Vision – ECCV 2020, pages 223–240, Cham, 2020. Springer International Publishing.
  * [12] A. Reboud, I. Harrando, J. Laaksonen, R. Troncy, et al.  Predicting media memorability with audio, video, and text representations.  In Proceedings of the MediaEval 2020 Workshop, December 2020.
  * [13] A. Reboud, I. Harrando, J. Laaksonen, R. Troncy, et al.  Exploring multimodality, perplexity and explainability for memorability prediction.  In MediaEval Multimedia Benchmark Workshop Working Notes, 2021.
  * [14] L. Sweeney, M. G. Constantin, C.-H. Demarty, C. Fosco, A. García Seco de Herrera, G. Healy, S. Halder, B. Ionescu, A. Matran-Fernandez, A. F. Smeaton, and M. Sultana.  Overview of the MediaEval 2022 predicting video memorability task.  In MediaEval Multimedia Benchmark Workshop Working Notes. CEUR, 2022\.
  * [15] L. Sweeney, G. Healy, and A. F. Smeaton.  Predicting media memorability: comparing visual, textual and auditory features.  In MediaEval Multimedia Benchmark Workshop Working Notes. CEUR Workshop Proceedings, October 2021.
  * [16] L. Sweeney, A. Matran-Fernandez, S. Halder, A. G. S. de Herrera, A. F. Smeaton, and G. Healy.  Overview of the EEG pilot subtask at MediaEval 2021: predicting media memorability.  In MediaEval Multimedia Benchmark Workshop Working Notes. CEUR, December 2021.


# Market mechanism to enable grid-aware dispatch of Aggregators in radial
distribution networks

Nawaf Nazir  Energy and Environment Directorate
Pacific Northwest National Laboratory
Richland, USA
<EMAIL_ADDRESS>Mads Almassalkhi This work was supported by the U.S.
Department of Energy’s Advanced Research Projects Agency—Energy (ARPA-E) Award
DE-AR0000694 and NSF Award ECCS-2047306 is gratefully acknowledged. Department
of Electrical and Biomedical Engineering
University of Vermont
Burlington, USA
<EMAIL_ADDRESS>
###### Abstract

This paper presents a market-based optimization framework wherein Aggregators
can compete for nodal capacity across a distribution feeder and guarantee that
allocated flexible capacity cannot cause overloads or congestion. This
mechanism, thus, allows Aggregators with allocated capacity to pursue a number
of services at the whole-sale market level to maximize revenue of flexible
resources. Based on Aggregator bids of capacity (MW) and network access price
($/MW), the distribution system operator (DSO) formulates an optimization
problem that prioritizes capacity to the different Aggregators across the
network while implicitly considering AC network constraints. This grid-aware
allocation is obtained by incorporating a convex inner approximation into the
optimization framework that prioritizes hosting capacity to different
Aggregators. We adapt concepts from transmission-level capacity market
clearing, utility demand charges, and Internet-like bandwidth allocation rules
to distribution system operations by incorporating nodal voltage and
transformer constraints into the optimization framework. Simulation based
results on IEEE distribution networks showcase the effectiveness of the
approach.

###### Index Terms:

Aggregator, DER, DSO, flexibility, markets

##  I Introduction

The addition of large-scale renewable energy in power systems has led to
increased volatility in supply that can negatively affect the reliability of
the distribution grid. To counter this volatility in renewable generation,
flexibility offered by demand-side resources (such as electric vehicles and
controllable loads) has shown promise [1, 2]. Unlike traditional MW-scale
thermal generators, flexibility in medium- and low-voltage networks will be
composed of thousands of kW-scale customer-owned devices that are distributed
throughout the medium- and low-voltage networks and will require careful
coordination and communication. Many of these customer-owned devices can be
controlled as an aggregate resource by an Aggregator. Uncoordinated operation
of these Aggregators can lead to reliability issues and congestion within the
distribution network [3]. This motivates the need to coordinate Aggregators
across the network in a manner that guarantees satisfaction of AC network
constraints (i.e., a grid-aware allocation of Aggregator flexibility). Based
on this, we cast the grid-aware allocation of Aggregators as a market-based,
convex optimization problem, which metes out the effective hosting capacity of
a radial feeder based on Aggregators’ capacity bids (MW, $/MW).

(a)

(b)

Figure 1: (a) Day-ahead net-demand at a single primary distribution node, (b)
Day-ahead aggregator flexibility capcity to meet the net-demand.

The concept of hosting capacity (HC) is not new and has been used by utilities
to site and size solar PV and EV charging station installations in a grid-
aware manner. However, HC methods tend to be piecemeal and static and consider
worst-case annual demand profiles, which can lead to overly conservative HC
values and an under-utilization of distribution network capacity. Demand-side
flexibility (e.g., from controlled behind-the-meter loads, EVs chargers, and
batteries) can also be thought of as capacity, however, such flexible assets
represent dynamic resources and require a re-thinking of the conventional
(static) HC concept [4]. Thus, our aim in this work is to determine a
distribution feeder’s dynamic hosting capacity and how to prioritize nodal
access in a grid-aware manner when multiple Aggregators offer flexibility at
different locations in the network. The authors’ recent work on determining
nodal hosting capacity developed a novel convex inner approximation (CIA) of
the set of admissible active power injections [5] that embeds network voltage
and line flow constraints within the nodal (net) injection bounds.
Conventional OPF methods can also dispatch flexibility in a grid-aware manner
but require full network and device data, which is only reasonable when the
Distribution network operator (DNO) is also the aggregator (i.e., vertically
integrated) [6]. In this paper, we aim to utilize the CIA formulation to
determine the dynamic hosting capacity of Aggregators by formulating it as a
market-clearing mechanism.

Such an approach can provide a grid-aware interface between Aggregators (that
manage behind-the-meter devices) and a distribution network operator (DNO) or
Utility (that is responsible for maintaining system reliability and security).
Another key benefit of the presented approach is that it establishes a clear
separation between Aggregators’ customer-owned devices and their associated
private device data and the Utility’s sensitive network data. That is, the
grid-aware allocation is achieved without Aggregators needing any grid data
nor Utility needing any device or personal usage data. Finally, since the
DSO’s role focuses on grid-aware allocation of Aggregator flexibility, it
allows the Aggregators to quantify their dispatchable flexibility a priori to
maximize revenues from participating in multiple market opportunities on
different timescales (e.g., ancillary services, capacity, frequency
regulation, arbitrage).

###  I-A Literature survey

Grid-aware coordination of distributed resources has previously been studied
in several works in literature, such as [7] that provides a certificate test
whether grid-aware control is required, [3] that provides various strategies
for network safe coordination of Aggregators, [8] that describes local control
laws that respond to real-time changes in voltage levels and [4] that develops
network constraint dependent import/export envelopes for Aggregators to
operate under. Related to the optimal power flow based methods, several works
in literature such as [9, 6] have developed centralized optimization based
frameworks to dispatch flexible resources in three-phase networks. In this
manuscript, we develop a method for a DSO to optimally allocate network
capacity between competing Aggregators, such that no Aggregator dispatch will
negatively impact grid reliability. Thus, the DSO’s allocation of Aggregators’
flexibility is grid-aware and enables Aggregators to participate with their
allocated flexibility in any whole-sale TSO markets without the potential to
cause network congestion in the radial distribution feeder. We achieve this by
formulating the optimization problem as a day-ahead market-clearing mechanism.

Day ahead market-clearing mechanisms have been studied in the literature with
[10] proposing a distribution locational marginal pricing (DLMP) for active
and reactive powers. These distribution-level market mechanism generally
employ DLMPs within transactive energy schemes that broadcast variable DMLPs
to devices to incentivize participation [11]. Combining grid-aware DER control
with market-clearing methods has also been studied with [12] proposing a
network constrained transactive energy framework to coordinate the
flexibility. These works, however, either neglect the distribution network
constraints, or utilize a simplified linear model, that does not provide
guarantees on system reliability. In this paper, we present a grid-aware
market-clearing mechanism that allocates and prioritizes flexibility to
competing Aggregators while avoiding network congestion. Based on the above,
this manuscript contributes to the literature as follows:

  1. 1.

We adapt the CIA formulation within a market mechanism to allocate and
prioritize flexibility amongst competing Aggregators located across a
distribution network.

  2. 2.

We introduce a two-step market mechanism where step 1 checks the feasibility
of hosting the available Aggregator flexibility and in case of expected
congestion, step 2 allocates flexibility to Aggregators based on their bids.

  3. 3.

We extend this mechanism to account for uncertainty in background demand with
a robust formulation.

##  II Problem definition and System Model

The proposed market architecture is illustrated in Fig. 2, showing the
different actors within the distribution system (DSO, DNO, and Aggregators)
and highlighting their information exchanges. The DSO is assumed to be an
independent authority that manages Aggregators’ bids (MW, $/MW) and DNO’s
network data and clears the flexibility market when congestion is expected.
The DNO owns and maintains the infrastructure and is compensated by the
Aggregators to use the network during periods of congestion. The Aggregators
pay a fee (set by the DSO’s market clearing mechanism) to the DNO in order to
access the distribution network capacity at their cleared allocation, called
the market-clearing price.

######  Definition II.1 (Market-clearing price).

A uniform price auction equilibrium solution at the intersection of the
distribution network capacity and the Aggregators’ bids.

The Aggregators then use their allocation to actively participate in different
valuable wholesale markets. As part of the bid structure, the Aggregators
provide the maximum capacity charges ($/MW) that they are willing to pay for
access to the distribution network, which could either be a single price per
feeder or nodal-level prices. These prices are realized in the same manner as
demand charges but within the context of a network’s hosting capacity. In
addition, the network capacity allocation concept is analogous to the ones
utilized in wireless network bandwidth allocation, where ISPs provide Mbps
capacity allocations to its users [13].

Thus, based on internal costs of deploying demand-side flexibility, including
the costs to acquire, deploy, and operate connected devices and expected
revenue from delivering whole-sale market services, the Aggregators need to
compute their own prices. The problem considered in this paper is to allocate
flexible hosting capacity across a distribution network to multiple
Aggregators while accounting for the nonlinear AC physics and operational
constraints. Specifically, we break down the problem into two steps: i)
feasibility check ii) prioritization of flexibility. In Step 1, the DSO
receives each Aggregators’ capacity bid (MW) and the DNO’s network model to
check if any Aggregator actions can cause network congestion. If Step 1 is
feasible, then all flexibility is accepted and Aggregators can dispatch freely
while the DNO is guaranteed that Aggregator actions will not impact
reliability. If Step 1 is infeasible, the DNO’s network cannot host all
Aggregators’ flexible capacity and the DSO uses the prices ($/MW) from the
Aggregator’s bid to prioritize and allocate Aggregators at each node in the
network. In the next section, we provide the mathematical model of the
distribution grid that will be utilized in solving this market optimization
problem.

Figure 2:  Illustration of market architecture showing information exchange
between different entities.

###  II-A Nonlinear AC model

Consider a balanced, radial distribution network, shown in Fig. 3, as an
undirected graph $\mathcal{G}=\\{\mathcal{N}\cup\\{0\\},\mathcal{L}\\}$
consisting of a set of $N+1$ nodes with $\mathcal{N}:=\\{1,\ldots,N\\}$ and a
set of $N$ branches
$\mathcal{L}:=\\{1,\ldots,N\\}\subseteq\mathcal{N}\times\mathcal{N}$, such
that $(i,j)\in\mathcal{L}$, if nodes $i,j$ are connected. Node $0$ is assumed
to be the substation node with a fixed voltage $V_{0}$. If $V_{i}$ and $V_{j}$
are the voltage phasors at nodes $i$ and $j$ and $I_{ij}$ is the current
phasor in branch $(i,j)\in\mathcal{L}$, then define $v_{i}:=|V_{i}|^{2}$,
$v_{j}:=|V_{j}|^{2}$ and $l_{ij}:=|I_{ij}|^{2}$. Let $P_{ij}$ ($Q_{ij}$) be
the active (reactive) power flow from node $j$ to $i$, let $p_{j}$ ($q_{j}$)
be the active (reactive) power injections into node $j$, and let $r_{ij}$
($x_{ij}$) be the resistance (reactance) of branch $(i,j)\in\mathcal{L}$,
which means that the branch impedance is given by
$z_{ij}:=r_{ij}+\mathbf{i}x_{ij}$. Then, for a radial network, the relation
between node voltages and power flows is given by the DistFlow equations
$\forall(i,j)\in\mathcal{L}$:

$\displaystyle v_{j}=$ $\displaystyle
v_{i}+2r_{ij}P_{ij}+2x_{ij}Q_{ij}-|z_{ij}|^{2}l_{ij}$ (1a) $\displaystyle
P_{ij}=$ $\displaystyle p_{j}+\sum_{h:h\rightarrow j}(P_{jh}-r_{jh}l_{jh})$
(1b) $\displaystyle Q_{ij}=$ $\displaystyle q_{j}+\sum_{h:h\rightarrow
j}(Q_{jh}-x_{jh}l_{jh})$ (1c) $\displaystyle l_{ij}(P_{ij},Q_{ij},v_{j})=$
$\displaystyle\frac{P_{ij}^{2}+Q_{ij}^{2}}{v_{j}},$ (1d)

where nodal power injections are $p_{j}:=p_{\text{g},j}-P_{\text{L},j}$ and
$q_{j}:=q_{\text{g},j}-Q_{\text{L},j}$ with $p_{\text{g},j}$
($q_{\text{g},j}$) as the controllable active (reactive) flexibility and
$P_{\text{L},j}$ ($Q_{\text{L},j}$) is the uncontrollable active (reactive)
demand.

Figure 3:  Nomenclature for a radial distribution network [14].

We seek to determine if the network’s available flexible capacity, e.g.,
$p_{g}^{+}\in\mathbb{R}^{N}$, can be hosted by the distribution network. That
is, for all $p_{g}\in[0,p_{g}^{+}]$, do the corresponding voltages and
currents remain within limits, i.e.,
$v_{j}(p_{g})\in[\underline{v_{j}},\overline{v_{j}}]\quad\forall
j\in\mathcal{N}$ and
$l_{ij}(p_{g})\in[\underline{l_{ij}},\overline{l_{ij}}]\quad\forall(i,j)\in\mathcal{L}$.
We can formulate the problem as:

$\displaystyle\text{(NLP)}\quad s_{i}^{*}=\arg\min_{s_{i}^{+}}\
\sum_{i=1}^{N}s_{i}^{+}$ (2)
$\displaystyle\text{s.t.}\quad\eqref{eq:volt_rel}-\eqref{eq:curr_rel}$ (3)
$\displaystyle p_{i}=p_{\text{g},i}^{+}-P_{\text{L},i}-s_{i}^{+}$
$\displaystyle\quad\forall i\in\mathcal{N}$ (4)
$\displaystyle\underline{v_{i}}\leq v_{i}\leq\overline{v_{i}}$
$\displaystyle\quad\forall i\in\mathcal{N}$ (5)
$\displaystyle\underline{l_{ij}}\leq l_{ij}\leq\overline{l_{ij}}$
$\displaystyle\quad\forall(i,j)\in\mathcal{L}$ (6) $\displaystyle
s_{i}^{+}\geq 0$ $\displaystyle\quad\forall i\in\mathcal{N}$ (7)

where $p_{\text{g},i}^{+}$ is the available upper flexibility at node $i$ and
$s_{i}^{+}$ is the slack variable when considering upper flexibility at node
$i$. A similar optimization problem can be formulated to determine the slack
variable $s_{i}^{-}$ when considering the lower flexibility
$p_{\text{g},i}^{-}$. In either case, if the flexibility $p_{\text{g},i}$ can
be accommodated by the network without violating any constraints, then
$s_{i}^{*}=0\quad\forall i\in\mathcal{N}$, and there is no possibility of
network congestion and a market mechanism is not required to prioritize
flexibility in the network and hence the DSO does not have to worry about
network reliability. If however, $\exists i\in\mathcal{N}$, s.t.,
$s_{i}^{*}>0$, then uncoordinated dispatch of flexible resources can lead to
network congestion and violation of network constraints. In order to avoid
such a situation, we need to prioritize the available flexibility.

However, solving (NLP) in general represents a technical challenge, as the
DistFlow equations governing the distribution system are non-linear due to
(1d), making the optimization problem (NLP) non-convex. In order to overcome
this challenge, many works in literature have considered approximations and
convex relaxation techniques [15]. Amongst the approximation methods is the
so-called linear DistFlow model or LinDist [16], whereas the convex relaxation
techniques include a host of different methods [17]. In the next section, we
will discuss the shortcomings of these methods, particularly as it relates to
the original (NLP), and then provide an alternative method that overcomes
those shortcomings.

###  II-B Linear and relaxation based models

This section presents the effects of using linear and convex relaxation based
models when determining maximum active power injections. First, consider a
linear model.

####  II-B1 Shortcomings of LinDist

The LinDist model is obtained by neglecting the $l_{ij}$ term in (1a)-(1d).
Consider a simple two-node system, where the active power injection, $p_{j}$,
is varied at the load node and then observe the corresponding changes in
voltage magnitude, $|V_{j}|$. As can be seen from Fig. 4a, the voltages
obtained from the DistFlow model (that represents the actual non-linear
physics) and the LinDist model (that sets $l_{ij}=0\ \forall ij$) are
identical at $p_{j}=0$. However, as we move $p_{j}$ away from this point,
LinDist becomes less accurate and this is especially the case towards the edge
of the allowable power injections range (when voltages are near their upper
and lower limits). Correspondingly, the models arrive at different conclusions
when it comes to estimating the maximum and minimum active power injections as
shown in Fig. 4b. Specifically, the LinDist model overestimates the lower
bound of the active power injection, which can in practice lead to voltage
violations due to the model mismatch between LinDist and DistFlow models.
Thus, linear models can clearly overestimate a network’s capacity for
flexibility and are hence not well-suited for grid-aware resource allocation
problems.

(a)

(b)

Figure 4: (a) Comparison in change in node voltage with change in real power
set-points between DistFlow and LinDist for two node system [18]. The two
models result in different voltages and if design is based on linear model
then network voltages limit will be violated if $p_{j}$ is operated at its
lower limit, (b) Comparison of feasible region obtained from LinDist and
DistFlow models for two node model.

####  II-B2 Shortcomings of convex relaxations

In this section, we expand on the above discussion to illustrate how
approaches based on convex relaxations also can overestimate flexible capacity
of a network, which can negatively impact reliability. Specifically, we
consider the conventional SOCP relaxation of DistFlow, obtained by replacing
the nonlinear equality constraint in (1d) with a convex inequality constraint.
We also impose an upper bound on the maximum hosting capacity to determine the
hosting capacity for which the AC network limits are satisfied. Solving the
SOCP for three different distribution networks (13-node, 37-node and 123-node)
results in Fig. 5, which shows the maximum network voltage magnitude for
different values of $P_{\text{max}}$. The examples show that the hosting
capacity obtained from an SOCP solution will exceed the voltage limits unless
$P_{\text{max}}$ is reduced significantly. For example, the SOCP formulation
for the 37-node feeder will erroneously predict that the total allowable
active power injections in the feeder exceeds 15MW. However, total allowable
injections beyond 2.5MW can exceed the voltage limits at one or more nodes in
the network. This is because a convex relaxation represents an outer
approximation of the original DistFlow model and, hence, solutions obtained
are not guaranteed to be physically realizable, i.e., they may violate network
constraints despite being feasible in convex SOCP. For further discussion on
convex OPF techniques, please see [19, 20, 5].

These shortcomings of the linear and the convex relaxation based models
highlight the need for an inner approximation that provides admissibility
guarantees. A convex form of an inner approximation would result in a scalable
implementation that inherits the benefits of convex optimization with network
admissibility guarantees. Recent works in literature have developed such a
convex inner approximation of the power flow equations [5], which can be used
to determine a feeder’s maximum, network-admissible active power nodal
injections (i.e., nodal hosting capacity). A summary of the convex inner
approximation formulation from [5] will be provided next.

Figure 5:  Plot of maximum nodal voltage with $P_{\text{max}}$ shows that the
predicted hosting capacity obtained with a convex relaxation in (CR) is much
higher than actually achievable. The figure shows the solutions obtained from
the convex relaxation method (SOCP) are infeasible at the determined hosting
capacity. As the maximum hosting capacity, $P_{\text{max}}$, is reduced, a
feasible solution that satisfies all network constraints is found.

.

##  III Convex inner approximations preliminaries

In order to obtain a convex inner approximation (CIA) formulation first we
define the vector form of (1a)-(1c) as:

$\displaystyle V=$ $\displaystyle
v_{\text{0}}\mathbf{1}_{N}+M_{\text{p}}p+M_{\text{q}}q-Hl,$ (8) $\displaystyle
P=$ $\displaystyle Cp-D_{\text{R}}l,\qquad Q=Cq-D_{\text{X}}l,$ (9)

where $P:=[P_{ij}]_{(i,j)\in\mathcal{L}}\in\mathbb{R}^{N}$,
$Q:=[Q_{ij}]_{(i,j)\in\mathcal{L}}\in\mathbb{R}^{N}$,
$V:=[v_{i}]_{i\in\mathcal{N}}\in\mathbb{R}^{N}$,
$p:=[p_{i}]_{i\in\mathcal{N}}\in\mathbb{R}^{N}$,
$p_{\text{g}}:=[p_{\text{g},i}]_{i\in\mathcal{N}}\in\mathbb{R}^{N}$,
$P_{\text{L}}:=[P_{\text{L},i}]_{i\in\mathcal{N}}\in\mathbb{R}^{N}$,
$q:=[q_{i}]_{i\in\mathcal{N}}\in\mathbb{R}^{N}$,
$Q_{\text{L}}:=[Q_{\text{L},i}]_{i\in\mathcal{N}}\in\mathbb{R}^{N}$, and
$l:=[l_{ij}]_{(i,j)\in\mathcal{L}}\in\mathbb{R}^{N}$ and matrices
$R:=\text{diag}\\{r_{ij}\\}_{(i,j)\in\mathcal{L}}\in\mathbb{R}^{N\times N}$,
$X:=\text{diag}\\{x_{ij}\\}_{(i,j)\in\mathcal{L}}\in\mathbb{R}^{N\times N}$,
$Z^{2}:=\text{diag}\\{z_{ij}^{2}\\}_{(i,j)\in\mathcal{L}}\in\mathbb{R}^{N\times
N}$, and $A:=[0_{N}\quad I_{N}]B-I_{N}$ and $B\in\mathbb{R}^{(N+1)\times N}$
is the incidence matrix of $\mathcal{G}$ relating the branches in
$\mathcal{L}$ to the nodes in $\mathcal{N}\cup\\{0\\}$, where $I_{N}$ is the
$N\times N$ identity matrix and $0_{N}$ is a column vector of $N$ rows. Also
matrices $M_{\text{p}}:=2C^{T}RC$, $M_{\text{q}}:=2C^{T}XC$,
$H:=C^{T}(2(RD_{\text{R}}+XD_{\text{X}})+Z^{2})$ and $C:=(I_{N}-A)^{-1}$,
$D_{\text{R}}:=(I_{N}-A)^{-1}AR$, and $D_{\text{X}}:=(I_{N}-A)^{-1}AX$
describe the network topology and impedance parameters. Further details about
this formulation of the DistFlow equations can be found in [14, 21, 5].

Based on the linear model of the DistFlow equations in (8)-(9) and considering
$l_{\text{lb}}$ and $l_{\text{ub}}$ as the new proxy variables for lower and
upper bounds on $l$. Then, we can define the corresponding upper $(.)^{+}$ and
lower $(.)^{-}$ bounds of $P$, $Q$ and $V$ as follows:

$\displaystyle P^{+}:=$ $\displaystyle Cp-D_{\text{R}}l_{\text{lb}}$ (10a)
$\displaystyle P^{-}:=$ $\displaystyle Cp-D_{\text{R}}l_{\text{ub}}$ (10b)
$\displaystyle Q^{+}:=$ $\displaystyle Cq-
D_{\text{X}_{+}}l_{\text{lb}}-D_{\text{X}_{-}}l_{\text{ub}}$ (10c)
$\displaystyle Q^{-}:=$ $\displaystyle Cq-
D_{\text{X}_{+}}l_{\text{ub}}-D_{\text{X}_{-}}l_{\text{lb}}$ (10d)
$\displaystyle V^{+}:=$ $\displaystyle
v_{\text{0}}\mathbf{1}_{n}+M_{\text{p}}p+M_{\text{q}}q-H_{+}l_{\text{lb}}-H_{-}l_{\text{ub}}$
(10e) $\displaystyle V^{-}:=$ $\displaystyle
v_{\text{0}}\mathbf{1}_{n}+M_{\text{p}}p+M_{\text{q}}q-H_{+}l_{\text{ub}}-H_{-}l_{\text{lb}},$
(10f)

where $D_{\text{X}_{+}}$ and $H_{+}$ include the non-negative elements of
$D_{\text{X}}$ and $H$, respectively, and $D_{\text{X}_{-}}$ and $H_{-}$ are
the corresponding negative elements. These bounds
$l_{\text{lb}},l_{\text{ub}}$ allow us to neglect the non-linear term in (1d).
Next we will obtain the expression for these bounds on $l$.

Based on any nominal operating point
$x^{0}_{ij}:=\text{col}\\{P_{ij}^{0},Q_{ij}^{0},v_{j}^{0}\\}\in\mathbb{R}^{3}$,
the second-order approximation for (1d) can be expressed as

$\displaystyle l_{ij}$ $\displaystyle\approx
l_{ij}^{0}+\mathbf{J}_{ij}^{\top}\mathbf{\delta}_{ij}+\frac{1}{2}\mathbf{\delta}_{ij}^{\top}\mathbf{H}_{\text{e},ij}\mathbf{\delta}_{ij},$
(11)

where $l_{ij}^{0}:=l_{ij}(x_{ij}^{0})$ are squared branch currents,
$\mathbf{\delta}_{ij}:=[P_{ij}-P_{ij}^{0},\,Q_{ij}-Q_{ij}^{0},\,v_{j}-v_{j}^{0}]^{\top}$
and Jacobian $\mathbf{J}_{ij}\in\mathbb{R}^{3\times 1}$ and Hessian
$\mathbf{H}_{\text{e},ij}\in\mathbb{R}^{3\times 3}$ are defined for each
branch $(i,j)$ as detailed in [5]. Furthermore, since
$\mathbf{H}_{\text{e},ij}$ is provably positive semi-definite per [21] the
second-order approximation in (11) can be used to derive lower and upper
bounds of $l_{ij}$ for all $(i,j)\in\mathcal{L}$. This is described as
follows:

$\displaystyle l_{ij}=|l_{ij}|$
$\displaystyle\approx|l_{ij}^{0}+\mathbf{J}_{ij}^{\top}\mathbf{\delta}_{ij}+\frac{1}{2}\mathbf{\delta}_{ij}^{\top}\mathbf{H}_{\text{e},ij}\mathbf{\delta}_{ij}|$
(12)
$\displaystyle\leq|l_{ij}^{0}|+|\mathbf{J}_{ij}^{\top}\mathbf{\delta}_{ij}|+|\frac{1}{2}\mathbf{\delta}_{ij}^{\top}\mathbf{H}_{\text{e},ij}\mathbf{\delta}_{ij}|$
(13) $\displaystyle\leq
l_{ij}^{0}+\max\\{2|\mathbf{J}_{ij}^{\top}\mathbf{\delta}_{ij}|,|\mathbf{\delta}_{ij}^{\top}\mathbf{H}_{\text{e},ij}\mathbf{\delta}_{ij}|\\}$
(14) $\displaystyle\Rightarrow l_{ij}$ $\displaystyle\leq
l_{ij}^{0}+\max\\{2|\mathbf{J}_{ij+}^{\top}\mathbf{\delta}_{ij}^{+}+\mathbf{J}_{ij-}^{\top}\mathbf{\delta}_{ij}^{-}|,\mathbf{\psi}_{ij}\\}\leq
l_{\text{ub},ij},$ (15) $\displaystyle\text{and }l_{ij}$ $\displaystyle\geq
l_{ij}^{0}+\mathbf{J}_{ij+}^{\top}\mathbf{\delta}_{ij}^{-}+\mathbf{J}_{ij-}^{\top}\mathbf{\delta}_{ij}^{+}=:l_{\text{lb},ij},$
(16)

where $\mathbf{J}_{ij+}$ and $\mathbf{J}_{ij-}$ includes the positive and
negative elements of $\mathbf{J}_{ij}$,
$\mathbf{\delta}_{ij}^{+}:=\mathbf{\delta}_{ij}(P_{ij}^{+},Q_{ij}^{+},v_{j}^{+},x_{ij}^{0})$
and
$\mathbf{\delta}_{ij}^{-}:=\mathbf{\delta}_{ij}(P_{ij}^{-},Q_{ij}^{-},v_{j}^{-},x_{ij}^{0})$,
and
$\mathbf{\psi}_{ij}:=\max\\{(\mathbf{\delta}_{ij}^{+,-})^{\top}\mathbf{H}_{\text{e},ij}(\mathbf{\delta}_{ij}^{+,-})\\}$,
which represents the largest of eight possible combinations of $P/Q/v$ terms
in $\mathbf{\delta}_{ij}$ with mixed $+,-$ superscripts. A detailed
description of these terms together with an analysis on the accuracy of this
second-order Taylor series approximation can be found in [21, 5].

This inner approximation of the DistFlow equations (10a)-(10f),(15),(16) is
provably convex and will henceforth be referred to as the convex inner
approximation (CIA). This CIA can be employed to obtain a hyper-rectangle
characterization of the set of active power nodal injections where the corner
points represent the feeders nodal capacity limits for the flexibility.
Further analysis in [5] also proves the admissibility of the range $\Delta
p_{\text{g}}:=[p_{\text{g}}^{-},p_{\text{g}}^{+}]$ obtained through the CIA
formulation and also presents an iterative algorithm that increases the size
of the admissible set. The next section will lay out the market clearing
problem to allocate flexibility amongst Aggregators based on the CIA
formulation.

##  IV Market-based grid-aware formulation

To formulate the market-clearing problem, the DSO receives capacity and price
bids from the Aggregators. The capacity bids represent an upper bound on the
Aggregator’s available flexibility at each node and the price bid represents
an upper bound on acceptable capacity charges at each nodes. Based on these
bids, the DSO allocates flexible capacity to Aggregators at each node in the
distribution network. The DSO’s market-clearing problem is implemented as a
two-step market optimization problem. The flowchart in Fig. 6 illustrates the
two-step approach. In Step 1, the DSO checks feasibility of hosting all
Aggregators’ flexible capacity. If congestion is expected to result from any
Aggregator dispatch, Step 2 is used to prioritize allocated flexibility based
on the Aggregators’ price bid, which represent their maximum access charges.
This two-step approach is detailed next.

Figure 6:  Illustration of the DSO’s market clearing mechanism for grid-aware
allocation of Aggregator flexibility.

###  IV-A Step 1: Network feasibility certificate

In Step 1, the DSO first checks the feasibility of network to host the total
Aggregators’ flexibility (MW) and does not require prioritizing any node or
Aggregator and, thus, does not require the price element from the Aggregators’
bids. To determine the impact of active power injections, the DSO leverages
the CIA presented above to formulate an inner approximation of the non-convex
(NLP) problem. This is illustrated below in (P1). That is, in Step 1, the DSO
solves (P1) and, if no slack variables are activated (i.e.,
$s_{i}^{*}=0\,\,\forall i$), the DSO informs the Aggregators that all flexible
capacity can be accommodated by the network and no premium access charge are
transferred to the Aggregators. This optimization problem can be formulated
as:

$\displaystyle\text{(P1)}\quad s_{i}^{*}=\arg\min_{s_{i}^{+}}\
\sum_{i=1}^{N}s_{i}^{+}$ (17a)
$\displaystyle\text{s.t.}\quad\eqref{eq:P_relation_1}-\eqref{eq:V_relation_2},\eqref{eq:l_upper},\eqref{eq:l_lower}$
(17b) $\displaystyle p_{i}=p_{\text{g},i}^{+}-P_{\text{L},i}-s_{i}^{+}$
$\displaystyle\quad\forall i\in\mathcal{N}$ (17c)
$\displaystyle\underline{v_{i}}\leq v_{i}\leq\overline{v_{i}}$
$\displaystyle\quad\forall i\in\mathcal{N}$ (17d) $\displaystyle
l_{\text{ub},ij}\leq\overline{l_{ij}}$
$\displaystyle\quad\forall(i,j)\in\mathcal{L}$ (17e) $\displaystyle
s_{i}^{+}\geq 0$ $\displaystyle\quad\forall i\in\mathcal{N}$ (17f)

where $p_{\text{g},i}^{+}:=\sum_{m}p^{\text{bid}}_{m,i}$, with
$p^{\text{bid}}_{m,i}$ being the flexible capacity bid (MW) of the $mth$
Aggregator at node $i$, where $m\in\mathcal{M}$, with $\mathcal{M}$ being the
set of Aggregators. Also, let $k_{m,i}$ be the price bid ($/MW) provided by
the $m$-th Aggregator to the DSO to define its upper bound on acceptable
network access charges at node $i$. Now, if (P1) predicts that any slack
variables will become active (i.e., any $s_{i}^{*}>0$), then some Aggregator
flexibility must be curtailed and, thus, we require a grid-aware allocation
that prioritizes nodal flexibility in certain locations and from certain
Aggregators (who are willing to pay higher access fees). Thus, we need to
leverage the available pricing information from each Aggregator. This
prioritization is described in Step 2 in the following section.

###  IV-B Step 2: Network-aware optimal allocation

If the DSO predicts network congestion by solving (P1), then the DSO needs to
allocate flexible capacity optimally based on price information. This is the
goal of Step 2, where we solve for an optimal capacity allocation of the
Aggregators based on both the capacity (MW) and price bids ($/MW). Thus, Step
2 determines the maximum DNO access charge that an Aggregator is willing to
pay. This represents a generalization of the author’s prior work on CIA [5],
where the formulation now support multiple Aggregators competing for nodal
hosting capacity via flexibility capacity bids (MW and $/MW). Specifically,
the Aggregator capacities feature in the energy balance constraints while the
price bids enter in the objective function to prioritize aggregator
flexibility across the network.

Based on the price bids and flexibility offered by various Agregators, the DSO
solves a centralized market optimization problem that determines the
admissible flexibility range and the market clearing price for each Aggregator
at each node, i.e., $\Delta p_{m,i}=p^{+}_{m,i}-p^{-}_{m,i}$. The optimization
problem is formulated as:

$\displaystyle\text{(P2)}\quad
p^{+}_{m,i}=\arg\max_{p_{m,i}}\sum_{m}\sum_{i}k_{m,i}p_{m,i}$ (18a)
$\displaystyle s.t.\ p^{\text{bid}}_{m,i}-p_{m,i}\geq 0\quad\forall
m\in\mathcal{M},\forall i\in\mathcal{N}$ (18b) $\displaystyle
p_{i}=\sum_{m}p_{m,i}-P_{\text{L},i}\quad\forall i\in\mathcal{N}$ (18c)
$\displaystyle\eqref{eq:P_relation_1}-\eqref{eq:V_relation_2},\eqref{eq:l_upper},\eqref{eq:l_lower}$
(18d) $\displaystyle\underline{V}\leq V^{-}(p,q)\quad V^{+}(p,q)\leq$
$\displaystyle\overline{V}$ (18e) $\displaystyle
l_{\text{ub}}\leq\overline{l}.$ (18f)

A similar optimization problem needs to be solved for the lower ranges,
$p_{\text{m},i}^{-}$. To illustrate this two-step market mechanism, next we
present case studies on the modified IEEE-37 node distribution network with
multiple Aggregators.

####  IV-B1 Case study 1: Feeder-level prices per Aggregator

To showcase the market-clearing problem (P1) and (P2), we utilize a modified
IEEE-37 node system from [8] and consider a scenario where each Aggregator’s
bid consists of nodal capacities (MW) and a fixed capacity price ($\$/MW$) for
all nodes as shown in Fig. 7a. The price bid values are adopted from
commercial demand charges based on available cost-optimization models [22]. In
this case study we only consider the upper range of flexibility as depicted in
Fig. 7b. Based on the Aggregator bids, the DSO first solves (P1) and checks if
$||s_{i}^{*}||_{\infty}>\epsilon$, where tolerance $\epsilon$ chosen to be 10
W. In this case study, (P1) yields $||s_{i}^{*}||_{\infty}=0.9$ MW, which
indicates that the Aggregators’ flexibility could cause network congestion.
Hence, the DSO proceeds to Step 2 and solves (P2), which uses the capacity
prices to prioritize Aggregators access and satisfy network constraints.

(a)

(b)

Figure 7: (a) Day-ahead Aggregator prices in a distribution feeder (assuming
Aggregators have same prices at different nodes), (b) Day-ahead Aggregator
flexibility bids into the the DSO clearing market.

Based on predicted congestion, DSO solves (P2) and the resulting flexibility
allocation is shown in Fig. 8a and highlights that Aggregators willing to pay
less for hosting capacity are allocated less flexibility. Here, Aggregator 1
is prioritized due to its willingness to pay a higher price to access the
network’s capacity. This point is further illustrated in Table I which shows
what fraction of Aggregator and nodal flexibility is allocated by the DSO. The
results show that nodes close to the substation get priority in capacity
allocation.

After the Aggregators’ flexible capacities are allocated by the DSO, we
utilize a price-clearing scheme where the lowest price at each node decides
the market clearing price [23] as depicted in Fig. 8b. The figure shows three
different clearing prices, with the nodes closer to the substation having a
lower price because of less congestion at those nodes. The revenue earned by
the DNO in this case is calculated according to (19) and comes out to
$\$72,870$.

$\displaystyle\text{Total DNO Revenue}=\sum_{i}\sum_{m}k_{\text{c},i}p_{m,i}$
(19)

where $k_{\text{c},i}$ is the clearing price at node $i$.

This case study employed feeder-level prices per Aggregator, which means that
all nodes are priced the same by each Aggregator, which could represent some
average customer or flexible load acquisition costs. In the next case study,
we explore nodal-level capacity prices for the Aggregators, which is similar
to the concept of distribution locational marginal prices (DLMPs) [10].
However, unlike much of the literature on DLMPs, this work focuses on pricing
access to network capacity (similar to demand charges) rather than
incentivizing DER participation by broadcasting (variable) prices in the
distribution network.

(a)

(b)

Figure 8: Case Study 1 results. (a) Allocation of Aggregator flexibility after market clearing, (b) Clearing price after market allocation based on cheapest allocation at each node.  TABLE I: Fraction of flexible capacity allocated across nodes and aggregators Item | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
---|---|---|---|---|---|---|---|---
Nodes | 0.99 | 0.73 | 0.3 | 0.36 | 0.69 | 0.71 | 0.36 | 0.22
Agg. | 1 | 0.52 | 0.47 | 0.05 |  |  |  |

####  IV-B2 Case study 2: Nodal Aggregator pricing

In this case study, we consider a nodal-level Aggregator pricing structure as
shown in Table II for the four Aggregators considered. The price bid values
are chosen randomly but are again based on the concept of commercial demand
charges based on available cost-optimization models [22]. Based on these
prices, we solve the (P2) market optimization problem to determine the
capacity allocations which are shown in Fig. 9a. Based on this allocation, we
again compute the clearing price based on the least-price allocation of an
Aggregator at each node, which is shown in Fig. 9b. The DNO revenue in this
scenario is calculated to be $\$41,460$.

TABLE II: Data for nodal-level aggregator capacity prices ($/MW) Node | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
---|---|---|---|---|---|---|---|---
Agg. 1 | 9.8 | 10.9 | 1.2 | 2.5 | 12.9 | 13.5 | 13.4 | 4.1
Agg. 2 | 4.6 | 6.1 | 5.6 | 1.6 | 3.60 | 7.0 | 4.6 | 2.6
Agg. 3 | 29.1 | 9.0 | 6.8 | 14.0 | 21.0 | 2.7 | 0.9 | 15.6
Agg. 4 | 13.2 | 1.6 | 11.4 | 12.7 | 1.3 | 4.8 | 2.5 | 12.7

(a)

(b)

Figure 9: Case Study 2 results. (a) Allocation of Aggregator flexibility after
market clearing under nodal Aggregator pricing, (b) Clearing price after
market allocation based on cheapest allocation at each node

##  V Robust allocation of flexibility

The analysis provided up to this point has been based on the assumption that
the background demand is known in advance. However, in practice that may not
be the case and it becomes important for the DSO to consider the uncertainties
in forecasting background demand. To address this problem, we augment
optimization formulations (P1) and (P2).

###  V-A Robust formulation

In this method, we consider the uncertainty in background demand to be bounded
and obtain a formulation that is robust to the worst-case of the bounded
uncertainty. The robust versions of (P1) is formulated as shown below:

$\displaystyle\text{(ROB-P1)}\quad s_{i}^{*}=\arg\min_{s_{i}^{+}}\
\sum_{i=1}^{N}s_{i}^{+}$ (20a)
$\displaystyle\text{s.t.}\quad\eqref{eq:P_relation_1}-\eqref{eq:V_relation_2},\eqref{eq:l_upper},\eqref{eq:l_lower}$
(20b) $\displaystyle
p_{i}=p_{\text{g},i}^{+}-P_{\text{L},i}-s_{i}^{+}+D_{i}^{+}$
$\displaystyle\quad\forall i\in\mathcal{N}$ (20c)
$\displaystyle\underline{v_{i}}\leq v_{i}\leq\overline{v_{i}}$
$\displaystyle\quad\forall i\in\mathcal{N}$ (20d) $\displaystyle
l_{\text{ub},ij}\leq\overline{l_{ij}}$
$\displaystyle\quad\forall(i,j)\in\mathcal{L}$ (20e) $\displaystyle
s_{i}^{+}\geq 0$ $\displaystyle\quad\forall i\in\mathcal{N}$ (20f)

where $D_{i}^{+}$ in (20c) is the upper robust bound on the uncertainty in
background demand at node $n$, which is considered alongside the upper
flexibility range $p_{\text{g},i}^{+}$, respectively. A similar problem would
consider the lower robust bound on uncertainty $D_{i}^{-}$ in order to
determine the feasibility of the lower flexibility range $p_{\text{g},i}^{-}$.
In a similar fashion, the robust (P2) is formulated as follows:

$\displaystyle\text{(ROB-P2)}\quad
p^{+}_{m,i}=\arg\max_{p_{m,i}}\sum_{m}\sum_{i}k_{m,i}p_{m,i}$ (21a)
$\displaystyle s.t.\ p^{\text{bid}}_{m,i}-p_{m,i}\geq 0\quad\forall
m\in\mathcal{M},\forall i\in\mathcal{N}$ (21b) $\displaystyle
p_{i}=\sum_{m}p_{m,i}-P_{\text{L},i}+D_{i}^{+}(D_{i}^{-})\quad\forall
i\in\mathcal{N}$ (21c)
$\displaystyle\eqref{eq:P_relation_1}-\eqref{eq:V_relation_2},\eqref{eq:l_upper},\eqref{eq:l_lower}$
(21d) $\displaystyle\underline{V}\leq V^{-}(p,q)\quad V^{+}(p,q)\leq$
$\displaystyle\overline{V}$ (21e) $\displaystyle
l_{\text{ub}}\leq\overline{l}$ (21f)

A similar optimization problem would consider the lower robust bound on
uncertainty $D_{i}^{-}$ in order to determine the lower flexibility allocation
$p_{\text{m},i}^{-}$. This formulation ensures that the market allocation
$p_{\text{m},i}^{+}(p_{\text{m},i}^{-})$ accounts for the uncertainty
$D_{i}^{+}(D_{i}^{-})$.

####  V-A1 Simulations under uncertainty

In the previous sections, the simulations assumed a known background demand at
each node. However, in forecasting the background demand at each node, we need
to account for prediction errors to ensure a grid-aware allocation of
flexibility. In this section, we consider a robust formulation by accounting
for the uncertainty at each flexible node as shown in Table III. Based on this
demand uncertainty, we solve (ROB-P1) and (ROB-P2) to obtain the grid-aware
Aggregator capacity allocation, which is shown in Fig. 10a. From Fig. 10a, the
total allocation is reduced by $0.44$ MW in this case (compared to the
deterministic case) as (ROB-P1) and (ROB-P2) need to reserve certain amount of
network capacity to account for the uncertainty in background demand. Based on
this allocation, we again determine the clearing price based on the least
price Aggregator at each flexible node (which is again higher than the price
in the deterministic case) and this clearing price for the robust case is
depicted in Fig. 10b. Since robustness raises the cleared price but reduces
flexible capacity, there is a trade-off when calculating DNO revenue. In this
case the DNO revenue is calculated as $\$39,540$, which is 4.8% less than the
deterministic case. Even though the simulation results in this paper utilized
a modified IEEE-37 node system, the CIA formulation is convex and has been
shown to scale to larger networks [5].

TABLE III: Uncertainty in background demand at flexible nodes (MW) Node | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
---|---|---|---|---|---|---|---|---
Uncertainty | 0.2 | 0.1 | 0.02 | 0.1 | 0.2 | 0.03 | 0.02 | 0.03

(a)

(b)

Figure 10: (a) Allocation of Aggregator flexibility after market clearing
under varying Aggregator prices and demand uncertainty, (b) Clearing price
based on cheapest allocation at each node under uncertainty in demand.

##  VI Conclusions and future work

Leveraging recent work on convex inner approximation, we have now formulated a
novel grid-aware allocation market-clearing mechanism which is executed in two
steps. Step 1 considers a feasibility check to determine whether the available
Aggregator flexibility can be safely hosted on the network. In Step 2,
flexibility is prioritized based on Aggregator’s bids and AC network physics
and constraints to provide grid-aware allocation across nodes in the network.

Future work will extend the work on uncertainty in background (net) demand,
especially at non-flexible nodes, to consider DNO margins and Aggregator
flexibility reserves. Recent work by [24] offers an interesting and systematic
approach to develop a robust convex inner approximation that can be employed
in (P1) and (P2). Finally, the authors are pursuing a hybrid solution to
manage uncertainty that considers that the Utility can share real-time grid
measurements with the Aggregators, which can then adapt its cleared grid-aware
allocations dynamically. This approach raises the prospect that Aggregators
can then offer the DNO flexibility reserves to incentivize improved forecasts
of background demand.

## References

  * [1] J. L. Mathieu, S. Koch, and D. S. Callaway, “State estimation and control of electric loads to manage real-time energy imbalance,” _IEEE Transactions on power systems_ , vol. 28, no. 1, pp. 430–440, 2012.
  * [2] K. Desrochers, V. Hines, F. Wallace, J. Slinkman, A. Giroux, A. Khurram, M. Amini, M. Almassalkhi, and P. D. Hines, “Real-world, full-scale validation of power balancing services from packetized virtual batteries,” in _2019 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT)_. IEEE, 2019, pp. 1–5.
  * [3] S. C. Ross and J. L. Mathieu, “Strategies for network-safe load control with a third-party aggregator and a distribution operator,” _IEEE Transactions on Power Systems_ , 2021.
  * [4] M. Z. Liu, L. N. Ochoa, S. Riaz, P. Mancarella, T. Ting, J. San, and J. Theunissen, “Grid and market services from the edge: Using operating envelopes to unlock network-aware bottom-up flexibility,” _IEEE Power and Energy Magazine_ , vol. 19, no. 4, pp. 52–62, 2021.
  * [5] N. Nazir and M. Almassalkhi, “Grid-aware aggregation and realtime disaggregation of distributed energy resources in radial networks,” _IEEE Transactions on Power Systems_ , 2021.
  * [6] M. Almassalkhi, S. Brahma, N. Nazir, H. Ossareh, P. Racherla, S. Kundu, S. P. Nandanoori, T. Ramachandran, A. Singhal, D. Gayme, C. Ji, E. Mallada, Y. Shen, P. You, and D. Anand, “Hierarchical, grid-aware, and economically optimal coordination of distributed energy resources in realistic distribution systems,” _Energies_ , vol. 13, no. 23, 2020. [Online]. Available: https://www.mdpi.com/1996-1073/13/23/6399
  * [7] D. Molzahn and L. A. Roald, “Grid-aware versus grid-agnostic distribution system control: A method for certifying engineering constraint satisfaction,” in _Proceedings of the 52nd Hawaii International Conference on System Sciences_ , 2019.
  * [8] K. Baker, A. Bernstein, E. Dall’Anese, and C. Zhao, “Network-cognizant voltage droop control for distribution grids,” _IEEE Transactions on Power Systems_ , vol. 33, no. 2, pp. 2098–2108, 2018.
  * [9] N. Nazir, P. Racherla, and M. Almassalkhi, “Optimal multi-period dispatch of distributed energy resources in unbalanced distribution feeders,” _IEEE Transactions on Power Systems_ , vol. 35, no. 4, pp. 2683–2692, 2020.
  * [10] L. Bai, J. Wang, C. Wang, C. Chen, and F. Li, “Distribution locational marginal pricing (dlmp) for congestion management and voltage support,” _IEEE Transactions on Power Systems_ , vol. 33, no. 4, pp. 4061–4073, 2018\.
  * [11] D. J. Hammerstrom, S. E. Widergren, and C. Irwin, “Evaluating transactive systems: Historical and current u.s. doe research and development activities,” _IEEE Electrification Magazine_ , vol. 4, no. 4, pp. 30–36, 2016.
  * [12] J. Hu, G. Yang, C. Ziras, and K. Kok, “Aggregator operation in the balancing market through network-constrained transactive energy,” _IEEE Transactions on Power Systems_ , vol. 34, no. 5, pp. 4071–4080, 2018.
  * [13] J. Frolik, “Qos control for random access wireless sensor networks,” in _2004 IEEE Wireless Communications and Networking Conference (IEEE Cat. No. 04TH8733)_ , vol. 3. IEEE, 2004, pp. 1522–1527.
  * [14] R. Heidari, M. M. Seron, and J. H. Braslavsky, “Non-local approximation of power flow equations with guaranteed error bounds,” in _Control Conference (ANZCC), 2017 Australian and New Zealand_ , 2017, pp. 83–88.
  * [15] D. K. Molzahn, I. A. Hiskens _et al._ , “A survey of relaxations and approximations of the power flow equations,” 2019.
  * [16] M. Baran and F. F. Wu, “Optimal sizing of capacitors placed on a radial distribution system,” _IEEE Transactions on power Delivery_ , vol. 4, no. 1, pp. 735–743, 1989.
  * [17] J. A. Taylor, _Convex optimization of power systems_. Cambridge University Press, 2015.
  * [18] N. Nazir and M. Almassalkhi, “Convex inner approximation of the feeder hosting capacity limits on dispatchable demand,” in _2019 IEEE 58th Conference on Decision and Control (CDC)_. IEEE, 2019, pp. 4858–4864.
  * [19] L. Gan, N. Li, U. Topcu, and S. H. Low, “Exact convex relaxation of optimal power flow in radial networks,” _IEEE Transactions on Automatic Control_ , vol. 60, no. 1, pp. 72–87, 2014.
  * [20] M. Nick, R. Cherkaoui, J.-Y. Le Boudec, and M. Paolone, “An exact convex formulation of the optimal power flow in radial distribution networks including transverse components,” _IEEE Transactions on Automatic Control_ , vol. 63, no. 3, pp. 682–697, 2017.
  * [21] N. Nazir and M. Almassalkhi, “Voltage positioning using co-optimization of controllable grid assets in radial networks,” _IEEE Transactions on Power Systems_ , vol. 36, no. 4, pp. 2761–2770, 2021.
  * [22] J. A. McLaren, P. J. Gagnon, and S. Mullendore, “Identifying potential markets for behind-the-meter battery energy storage: A survey of us demand charges,” National Renewable Energy Lab.(NREL), Golden, CO (United States), Tech. Rep., 2017\.
  * [23] A. Sinha, A. Basu, R. Lahiri, S. Chowdhury, S. Chowdhury, and P. A. Crossley, “Setting of market clearing price (mcp) in microgrid power scenario,” in _2008 IEEE Power and Energy Society General Meeting-Conversion and Delivery of Electrical Energy in the 21st Century_. IEEE, 2008, pp. 1–8.
  * [24] D. Lee, K. Turitsyn, D. K. Molzahn, and L. Roald, “Robust ac optimal power flow with robust convex restriction,” _IEEE Transactions on Power Systems_ , pp. 1–1, 2021.