# Crowdsourcing the State of the Art(ifacts)

Maria Teresa Baldassarre  0000-0001-8589-2850 University of Bari, Italy
<EMAIL_ADDRESS>,  Neil Ernst  0000-0001-5992-2366 University
of Victoria, Canada<EMAIL_ADDRESS>,  Ben Hermann  0000-0001-9848-2017
Technische Universität Dortmund, Germany<EMAIL_ADDRESS>and
Tim Menzies, Rahul Yedida  0000-0002-5040-3196 NC State University, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS>
(Date: Received: date / Accepted: date)

###### Abstract.

In any field, finding the “leading edge” of research is an on-going challenge.
Researchers cannot appease reviewers and educators cannot teach to the leading
edge of their field if no one agrees on what is the state-of-the-art.

Using a novel crowdsourced “reuse graph” approach, we propose here a new
method to learn this state-of-the-art. Our reuse graphs are less effort to
build and verify than other community monitoring methods (e.g. artifact tracks
or citation-based searches). Based on a study of 170 papers from software
engineering (SE) conferences in 2020, we have found over 1,600 instances of
reuse; i.e., reuse is rampant in SE research. Prior pessimism about a lack of
reuse in SE research may have been a result of using the wrong methods to
measure the wrong things.

Artifact Evaluation and Open Science and Reuse

## Introduction

According to Popper (Popper, 2014), the ideas we can most trust are those that
have been most tried and most tested. For that reason, many of us are involved
in the process called “Science” that produces trusted knowledge by sharing
one’s ideas, and trying out and testing others’ ideas. Science and scientists
form communities where people do each other the courtesy of curating,
clarifying, critiquing and improving a large pool of ideas.

Prior to this study, the standard conclusion was that researchers in the field
of software engineering are rarely reusing research results (e.g., da Silva et
al. reported that from 1994-2010, only 72 studies had been replicated by 96
new studies (da Silva et al., 2012)). If true, this is a significant problem
since not knowing the state of the art complicates both research and graduate
education.

We argue in this paper that, at least in the area of software engineering,
this “reuse problem” is more apparent, than real. We describe a successful
experiment where teams of researchers from around the world read 170 recent
(2020) conference papers from software engineering. This work generated the
“reuse graph” of Figure 1. In that figure, each edge connects papers to the
prior work that they are (re-)using. As discussed below, when compared to
other community monitoring methods (e.g. artifact tracks or bibliometric
searches (Mathew et al., 2018; Baldassarre et al., 2019)), these reuse graphs
are less effort to build and verify. For example, it took around 12 minutes
per paper for our team from Hong Kong, Canada, the United States, Italy,
Sweden, Finland, and Australia to apply this reuse graph methodology to
software engineering111That team included the authors of this paper plus Jacky
Keung from City University (Hong Kong); Greg Gay from Chalmers University
(Sweden); Burak Turhan from Oulu University (Finland); and Aldeida Aleti from
Monash University (Australia). We gratefully acknowledge their work, and that
of their graduate students. In particular, we especially call out the work of
Afonso Fontes from Chalmers University (Sweden)..

Figure 1. From the web-site https://reuse-dept.org. The 1,635 arrows in this
diagram connects reuser to reused. Blue dots denote the 714 sources found with
with a digital object identifier (a DOI); e.g. any paper from a peer-reviewed
source. Red dots denote the 48 papers we found without DOIs (e.g. those from
arxiv.org). Gray and green denote the 297 websites and 57 Github repositories
(respectively) reused in this sample. The black squares pull out four examples
where a paper has reused material from dozens of other sources. Data
collection for this graph is on-going and at the time of this writing, that
data comes from 40% of the papers published in the 2020 technical program of
ICSE, ASE, FSE, ICMSE, MSR, and ESEM.

There are many other methods to map the structure of SE research such as (a)
manual or automatic citations searchers or (b) “artifact evaluation
committees” that aim to foster the generation and sharing of research products
(for more on artifact evaluation committees, see below). Such studies can lag
significantly behind current work. For example, our citation analysis of SE
(Mathew et al., 2018) only went up to 2016, the study itself was conducted in
2017, but not fully published till 2018. Given the enormous effort required
for that work, we have vowed never to do it again. Reuse graphs, on the other
hand, are faster to keep up to date since the work of any one individual
working on these graphs is minimal.

Another reason to favor reuse graphs is that they are community
comprehensible, community verifiable, and community correctable. All the data
used for our reuse graphs is community-collected. All the data can be audited
at https://reuse-dept.org and if errors are detected, issue reports can be
raised in our GitHub repository (and the error corrected). The same may not be
true for studies based on citation servers run by professional bodies and for-
profit organizations (e.g., see Table 1).

EXAMPLE #1: At the time of this writing, one of us has an entry in Google
Scholar metrics software systems saying that their paper “How to” has 80
citations in the last five years at IEEE Transactions on Software Engineering.
That link connects to some other paper, not written by any of us. Yet there is
no ”help” button at Google Scholar where this error can be reported. This is
disappointing since it is suspected that this mysterious “How to” paper
references work that might be the most cited from IEEE TSE in the last five
years. That would be a significant achievement, if we could document it (but
using Google Scholar, we cannot).
---
EXAMPLE #2: In our recent large scale text-mining studies of 30,000+ SE papers
(Mathew et al., 2018) , it was found that papers can appear in our citation
server, but not in others. Also, examples were found where some papers had
twenty times the citations in one server than in another. Here again, we were
unable to contact anyone working on those citation servers to fix those
errors.
EXAMPLE #3: Sometimes it is possible to contact the owners of these citations
sites, but even then they may not fix errors. For example, there is no
accepted convention for how to typeset a hypen, so different venues add in
zero to one space before or after (and some even typeset the hyphen as two
dashes). Hence, Zhou et al. found that (a) papers with hyphens in the title
get reported as different papers in different venues; which means that (b)
those papers get fewer citations (Zhou et al., 2021). Zhou et al. report that
when they contacted the owners of these citation servers, rather than fix the
errors, those owners started lobbying for the Zhou et al. paper not to be
published.
Table 1. Examples of errors in citation servers.

R2.4 What is the value of a verified, continually updated, snapshot of some
current research area? Once our reuse graph covers several years (and not just
2020 publications), we foresee several applications:

  1. (1)

Graduate students could direct their attention to research areas that are both
very new (nodes from recent years) and very productive (nodes with an
unusually large number of edges attached);

  2. (2)

The organizers of industrial and research conferences could select their
keynote speakers from that space of new and productive artifacts.

  3. (3)

When applying for promotion or hiring, research faculty or industrial workers
could document the impact of their work beyond papers, including tools,
datasets, and innovative methods;

  4. (4)

Growth patterns might guide federal government funding priorities or
departmental hiring plans.

  5. (5)

Venture capitalists could use these graphs to detect emergent technologies,
perhaps even funding some of those.

  6. (6)

Conference organizers could check if their program committees have enough
members from currently hot topics.

  7. (7)

Further, those same organizers could create new conference tracks and journals
sections in order to service active research communities that are under-
represented in current publication venues.

  8. (8)

Journal editors could find reviewers with relevant experience.

  9. (9)

Educators can use the graphs to guide their teaching plan.

Further to the last point, we are planning an immediate application of Figure
1 graph for our Fall’21 graduate SE classes. There, we will tell students that
understanding the current state of the art will be their challenge for the
rest of their career. But, using reuse graphs, it is possible for a community
to find and maintain a shared understanding of that state-of-the-art. To
demonstrate this, in our Fall’21 classes, we are leaving the second half of
the lecture plan blank. We will let students find and define what cutting edge
techniques will be discussed there. To do so, their homework for the initial
three weeks of class is to, at first, learn this reuse graph approach by
performing our standard “reuse graph 101”
exercise222https://github.com/bhermann/DoR/blob/main/workflow/training.md.
Then in week 2, read some papers to find their reuse (if any); Finally, in
week 3, they should check someone else’s reuse findings from other papers.

## Studying Reuse

In our reuse study, we targeted papers from the 2020 technical programs of six
major international SE conferences: Software Engineering (ICSE), Automated
Software Engineering (ASE), Joint European Software Engineering Conference /
Foundations of Software Engineering (ESEC/FSE), Software Maintenance and
Engineering (ICSME), Mining Software Repositories (MSR), and Empirical
Software Engineering and Measurement (ESEM). These conferences were selected
using advice from (Mathew et al., 2018), but our vision is to expand; for
example, by looking at all top-ranked SE conferences. GitHub issues were used
to divide up the hundreds of papers from those conferences into “work packets”
of ten papers each. Reading teams were set up from software engineering
research teams from around the globe in Hong Kong, Istanbul (Turkey), Victoria
(Canada), Gothenburg (Sweden), Oulu (Finland), Melbourne (Australia), and
Raleigh (USA). Team members would assign themselves work packets and then read
the papers looking for the kinds of reuse enumerated below. Once completed, a
second person (from any of our teams) would do the same and check for
consistency. Fleiss Kappa statistics are then computed to track the level of
reader disagreement. All interaction was done via the GitHub issue system (see
Figure 2).

Figure 2. Controlling data collection for building the reuse graphs.

Teams were asked to record six kinds of reuse:

  1. (1)

Most papers have to benchmark their new ideas against some prior recent state-
of-the-art paper. That is, they reuse old papers as stepping stones towards
new results.

  2. (2)

Another thing that is often reused are statistical methods. Here we do not
mean “we use a two-tailed t-test” or some other decades-old widely-used
statistical method. Rather, we refer instead to statistical methods for recent
papers that propose statistical guidance for the kinds of analysis seen in SE
. Perhaps because this kind of analysis is very rare, these people are
exceedingly high cited; e.g.

     * •

A 2008 paper Benchmarking Classification Models for Software Defect Prediction
(Lessmann et al., 2008) currently has 1,178 citations;

     * •

A 2011 paper A practical guide for using statistical tests to assess
randomized algorithms in software engineering (Arcuri and Briand, 2011)
currently has 778 citations.

  3. (3)

Metrics and Method descriptions (which may be guidelines, with no tools);

  4. (4)

R3.2a Data sets;

  5. (5)

Sanity checks (justification for why a particular approach works or is
reasonable to avoid bad data);

  6. (6)

and, indeed, the software packages of the kind currently being reviewed by SE
conference artifact evaluation committees (tools and replications).

We can report that it is not difficult to read papers in order to detect these
kinds of reuse:

  * •

It is fast to find the above six kinds of reuse. Our graduate students report
that reading their first paper might take up to an hour. But after two or
three papers, the median reading time drops to around 12 minutes (see Figure
3(a)).

  * •

When we compare the reuse reported by different readers, we get Figure 3(b).
In our current results, the median Fleiss Kappa score (for reviewer agreement)
is one (i.e. very good).

(a) Reading times

(b) Fleiss Kappa scores

(c) Years that the works were created. R3.1 .

Figure 3. Reading time results (left). Agreement scores (center). Yearly
prevalence of reused papers (right).

Of course, there any many other items being reused than the six listed
above333See https://pasteboard.co/Ke4tKgO.png. It is an open question, worthy
of future work, to check if those other items can be collected in this way.

## Related Work

Apart from software engineering, many other disciplines are actively engaged
in artifact creation, sharing, and re-use. Artifacts are useful for building a
culture of replication and reproducibility, already acknowledged as important
in SE (Santos et al., 2021). Fields such as psychology have had many early
results thrown into doubt because of a failure to replicate the original
findings (Schimmack, 2020). Sharing research protocols and data allows for
other research teams to conduct severe tests of the original studies (Mayo,
2018), strengthening (or rejecting) these initial findings. In medicine, drug
companies are mandated to share the research protocols and outcomes of their
drug trials, something that has become of vital recent importance (albeit not
without challenges (DeVito et al., 2020)). In physics and astronomy, artifact
sharing is so commonplace that large community infrastructures exist solely to
ensure data sharing, not least because governments which fund these costly
experiments insist on it.

Furthermore, software plays such a vital role in this enterprise that many
fields have begun training and developing science-specific research software
engineers, for example, at the UK’s Software Sustainability
Institute444https://www.software.ac.uk, or the USA’s NSF-funded Molecular
Science Software Institute555https://molssi.org. Indeed, a founding impetus
for the World Wide Web was the need for CERN (Center for Nuclear Research) to
facilitate knowledge sharing (Berners-Lee, 1990)666And Tim Berners-Lee himself
has said “Had the technology been proprietary, and in my total control, it
would probably not have taken off.” Another argument for artifacts!—a mission
now continued by Zenodo, also operated by CERN.

In more theoretical areas of CS, pioneering use of preprint servers has
enabled ‘reuse’ of proofs, essential to progress. In machine learning,
replication is focused on stepping-stones, enabled by highly successful
benchmarks such as ImageNet (Russakovsky et al., 2015). However, recent
advances with extremely costly training regimens have called replicability
into
question777https://www.technologyreview.com/2020/11/12/1011944/artificial-
intelligence-replication-crisis-science-big-tech-google-deepmind-facebook-
openai/.

In the specific case of software engineering research, prior to this paper,
there was little recorded and verified evidence of reuse. Many researchers
have conducted citation studies that find links to highly cited papers (e.g.
(Mathew et al., 2018)). As stated in our introduction, such studies can lag
behind the latest results. Also, recalling Table 1, we have cause to doubt the
conclusions from such citation studies.

### What about Artifact Evaluation Committees?

R1.1 Another practice that is becoming increasingly common is for conferences
to run artifact evaluation committees. The authors of accepted conference
papers submit software packages that, in theory, let others re-execute that
work. These evaluation committees award “badges” as shown in Table 2.

Table 2. Badges currently awarded at ACM conferences (Association for Computing Machinery, 2020). This table is for the ACM and analogous tables are used at other conferences. Available |  Functional |  Reusable |  Reproduced |  Replicated
---|---|---|---|---
|  |  |  |
Archived in a public repository with a long-term retention policy. A DOI needs to be provided. |  Artifacts need to be documented, consistent, complete, exercisable, and include appropriate evidence of verification and validation. |  Only available to artifacts already qualifying for the functional badge. Needs to significantly exceed minimal functionality. |  Results of this paper have been reproduced by a different team using the original artifact. |  Results of this paper have been replicated by a different team without the original artifact.
Figure 4. Artifact evaluation committee sizes 2011-2019. From Hermann et al.
(Hermann et al., 2020)

Artifact evaluation is something of a “growth industry” in SE (and programming
languages, or PL, community). Figure 4 shows the increasing number of people
evaluating artifacts 2011 to 2019. As to more recent data: (a) at PLDI’20, 61
of 77 papers offered artifacts (Donaldson and Torlak, 2020), and at ECOOP’21,
all 20 research papers offered software artifacts (Møller and Sridharan,
2021); (b) at ASE’21, they have a 60 person artifact evaluation track.

The question has to be asked: are all the people of Figure 4 making the best
use of their time? Perhaps not. We note that most artifacts are assigned the
badges requested by the authors. Given that, it might be safe to ask some of
the personnel from Figure 4 to (e.g.) spend less time on evaluating conference
artifacts and spend more time working on Figure 1.

Also, we suspect that the badges of Table 2 need refinement since much time
can be squandered on minor issues with little practical effect. For example,
it can be hard to distinguish “functional” from “reusable” (in fact, some
artifact evaluation committees just ignore the “functional” badge; e.g. see
the artifact evaluation process at ICSE’21, ICMSE’21, and ASE’21).

Further, as to checking for “functional” and “reusable”, that requires
downloading then installing then running the software. This can be a very long
process (taking hours to days), especially for quirky research prototypes
where (e.g.) the scripts have one letter typos and/or the install instructions
are missing small, but crucial pieces of information.

But most importantly, it is not clear that the artifact evaluation process is
creating reused artifacts. If we query ACM Portal for “software engineering”
and “artifacts” in the range 2015 to 2020, we find that most of the recorded
artifacts are R1.4 not reused in replications or reproductions888 As of
December 10 2020, that search returns 2,535 SE papers with artifact badge. Of
these, 43%, 30%, 20%, 5%, 2% are available, functional, reusable, reproduced
and replicated artifacts (respectively).. Specifically, only $\frac{1}{20}$th
are reproduced and only $\frac{1}{50}$th are replicated. R1.2 One possibility
for these results, which we can quickly discount, is that SE artifacts are a
new idea that will take a while to catch on. If this were the case, we would
expect that older SE artifacts are reproduced more, since they have been
around for longer. But this is not the case. Look at Figure 3c, we see that
most of the reused artifacts were created very recently.

Perhaps we need to change the definition of the badges and say an artifact is
“reusable” if it is reused (and not before). Also, it might be useful to
reflect more on what is actually being reused (as we have done, above).

## Next Steps for Reuse Graphs

While Figure 1 is a promising start, to scale up from here, we need to
organize a larger reading population. Our goal is to analyze 200, 2000, 5000
papers in 2021, 2022, 2023 (respectively) by which time we would have covered
most of the major SE conferences in the last five years. After that, our
maintenance goal would be to read 500 (ish) papers per year to keep up to date
with the conferences. R3.6 Based on Figure 3a, then assuming each paper is
read by two people, then maintenance goal would be achievable by a team of
twenty people working two hours per month on this task. To organize this work,
we have created the “ROSE initiative”, see Table 3.

Table 3. The ROSE initiative: Recognizing and Rewarding Open Science in
Software Engineering: an international multi-conference workshop that will
continually report updates to the SE reuse graphs. Researchers that reuse the
most from other papers will be applauded and awarded a “R-index” (reuse
index).
---
Researchers that build the artifacts that are most reused will be applauded
(even louder) and be awarded an “R+-index” indicating that they are the people
producing the artifacts that are most used by the rest of the community.
In between each conference, the ROSE initiative will co-ordinate an
international team of volunteers incrementally updating the SE reuse graph.
This reuse graph will be displayed at a publicly available web site (reuse-
dept.org). Individual researchers can browse that site, check their entries,
and propose corrections and extensions.
All reports of reuse will be double checked and disputed claims will be then
be tripled-checked.
All the tools used to create that web site will be freely available for
download. Hence, if the SE community does not like how we are running these
reuse graphs, they can take all our code and data and do something else.
Also, researchers from other disciplines can take our tools and apply them to
their own community.

If that work interests you, then there are many ways you can get involved:

  * •

If you are a researcher and wish to check that we have accurately recorded
your contribution, please visit https://reuse-dept.org.

  * •

If you want to apply reuse graphs to your community, please use our tools at
https://github.com/bhermann/DoR/.

  * •

If you are interested in joining this initiative and contributing to an up-to-
minute snapshot of SE research, then please (a) take our how-to-read-for-reuse
tutorial999https://github.com/bhermann/DoR/blob/main/workflow/training.md; (b)
then visit the dashboard at Figure 2, find an issue with no one’s face on it,
and assign yourself a task.

  * •

Better yet, if you are an educator teaching a graduate SE class, then get your
students to do the three week reading assignments shown in the introduction.
As a result, students will join an international team exploring reuse in SE
that will keep them informed and updated about the state-of-the-art in SE for
many years to come. Also, as a side-effect, they will also see first hand the
benefit of open source tools that can be shared by teams working around the
globe.

We see this effort as one part of the broader open science effort, in addition
to helping the community identify the state of the art (e.g., patterns of
growth in the reuse graph). Among the goals of open science are to increase
confidence in published results, and to acknowledge that science produces more
types of artifacts than just publications: researchers also produce method
innovations, new datasets, and improved tools. If we take an agile view of SE
science, then as researchers we should focus on generating these artifacts and
rapidly securing critique, curation, and clarification from our peers and the
public.

## References

  * (1)
  * Arcuri and Briand (2011) Andrea Arcuri and Lionel Briand. 2011.  A Practical Guide for Using Statistical Tests to Assess Randomized Algorithms in Software Engineering. In _Proceedings of the 33rd International Conference on Software Engineering_ (Waikiki, Honolulu, HI, USA) _(ICSE ’11)_. Association for Computing Machinery, New York, NY, USA, 1–10.  https://doi.org/10.1145/1985793.1985795
  * Association for Computing Machinery (2020) Association for Computing Machinery. 2020\.  Artifact Review and Badging.  https://www.acm.org/publications/policies/artifact-review-and-badging-current.  Accessed: 2020-12-08.
  * Baldassarre et al. (2019) Maria Teresa Baldassarre, Danilo Caivano, Simone Romano, and Giuseppe Scanniello. 2019.  Software models for source code maintainability: A systematic literature review. In _2019 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)_. IEEE, 252–259.
  * Berners-Lee (1990) Tim Berners-Lee. 1990\.  _Information Management: A Proposal_.  https://www.w3.org/History/1989/proposal.html
  * da Silva et al. (2012) Fabio Q. B. da Silva, Marcos Suassuna, A. César C. França, Alicia M. Grubb, Tatiana B. Gouveia, Cleviton V. F. Monteiro, and Igor Ebrahim dos Santos. 2012.  Replication of empirical studies in software engineering research: a systematic mapping study.  _Empirical Software Engineering_ (Sept. 2012).  https://doi.org/10.1007/s10664-012-9227-7
  * DeVito et al. (2020) Nicholas J DeVito, Seb Bacon, and Ben Goldacre. 2020\.  Compliance with legal requirement to report clinical trial results on ClinicalTrials.gov: a cohort study.  _The Lancet_ 395, 10221 (feb 2020), 361–369.  https://doi.org/10.1016/s0140-6736(19)33220-9
  * Donaldson and Torlak (2020) Alastair F. Donaldson and Emina Torlak (Eds.). 2020.  _PLDI 2020: Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation_ (London, UK). Association for Computing Machinery, New York, NY, USA.  https://doi.org/10.1145/3385412
  * Hermann et al. (2020) Ben Hermann, Stefan Winter, and Janet Siegmund. 2020\.  Community Expectations for Research Artifacts and Evaluation Processes. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Virtual Event, USA) _(ESEC/FSE 2020)_. Association for Computing Machinery, New York, NY, USA, 469–480.  https://doi.org/10.1145/3368089.3409767
  * Lessmann et al. (2008) Stefan Lessmann, Bart Baesens, Christophe Mues, and Swantje Pietsch. 2008.  Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings.  _IEEE Transactions on Software Engineering_ 34, 4 (2008), 485–496.  https://doi.org/10.1109/TSE.2008.35
  * Mathew et al. (2018) George Mathew, Amritanshu Agrawal, and Tim Menzies. 2018\.  Finding Trends in Software Research.  _IEEE Transactions on Software Engineering_ (2018), 1–1.  https://doi.org/10.1109/TSE.2018.2870388
  * Mayo (2018) D. G. Mayo. 2018\.  _Statistical inference as severe testing: How to get beyond the statistics wars_.  Cambridge University Press.
  * Møller and Sridharan (2021) Anders Møller and Manu Sridharan. 2021.  Front Matter, Table of Contents, Preface, Conference Organization. In _35th European Conference on Object-Oriented Programming (ECOOP 2021)_ _(Leibniz International Proceedings in Informatics (LIPIcs), Vol. 194)_ , Anders Møller and Manu Sridharan (Eds.). Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl, Germany, 0:i–0:xxiv.  https://doi.org/10.4230/LIPIcs.ECOOP.2021.0
  * Popper (2014) Karl Popper. 2014\.  _Conjectures and refutations: The growth of scientific knowledge_.  Routledge.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015\.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision (IJCV)_ 115, 3 (2015), 211–252.  https://doi.org/10.1007/s11263-015-0816-y
  * Santos et al. (2021) Adrian Santos, Sira Vegas, Markku Oivo, and Natalia Juristo. 2021.  Comparing the results of replications in software engineering.  _Empirical Software Engineering_ 26, 2 (Feb. 2021).  https://doi.org/10.1007/s10664-020-09907-7
  * Schimmack (2020) Ulrich Schimmack. 2020\.  A meta-psychological perspective on the decade of replication failures in social psychology.  _Canadian Psychology/Psychologie canadienne_ 61, 4 (nov 2020), 364–376.  https://doi.org/10.1037/cap0000246
  * Zhou et al. (2021) Zhi Quan Zhou, T. H. Tse, and Matt Witheridge. 2021.  Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors.  _IEEE Transactions on Software Engineering_ 47, 6 (2021), 1164–1183.  https://doi.org/10.1109/TSE.2019.2915065


# Uncertainty Assessment of Dynamic Thermal Line Rating for Operational Use at
Transmission System Operators

Aleksandra Rashkovska, Mitja Jančič, Matjaž Depolli, Janko Kosmač, and Gregor
Kosec This work was supported by ELES d.o.o. and the Slovenian Research Agency
(ARRS) under Grant No. P2-0095.A. Rashkovska, M. Jančič, M. Depolli and G.
Kosec are with the Department of Communication Systems, Jožef Stefan
Institute, Ljubljana 1000, Slovenia (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>gregor.kosec@ijs.si).M. Jančič is
also with the Jožef Stefan International Postgraduate School, Ljubljana 1000,
Slovenia.J. Kosmač is with ELES d.o.o. – the operator of the electric power
transmission network of the Republic of Slovenia, Ljubljana, Slovenia (e-mail:
janko.kosmac@eles.si).

###### Abstract

Transmission system operators (TSOs) in recent years have faced challenges in
order to ensure maximum transmission capacity of the system to satisfy market
needs, while maintaining operational safety and permissible impact on the
environment. A great help in the decision-making process was introduced with
the Dynamic Thermal Rating (DTR) – an instrument to monitor and predict the
maximal allowed ampacity of the power grid based on weather measurements and
forecast. However, the introduction of DTR raises a number of questions
related to the accuracy and uncertainty of the results of thermal assessment
and the level of acceptable risk and its management. In this paper, we present
a solution for estimating DTR uncertainty, appropriate for operational use at
TSOs. With the help of conductor surface temperature measurements, weather
measurements and predicted weather data, we also estimate the error of the
weather forecast and the DTR itself. Following the results of the data
analyses, we build an operative solution for estimating the ampacity
uncertainty based on Monte Carlo random simulations and integrate it into the
operational environment of ELES – the operator of the Slovenian electric power
transmission network.

###### Index Terms:

Dynamic Thermal Rating, Transmission line, Transmission System Operator,
Ampacity, Uncertainty, Forecast, Probability Distribution Function, Monte
Carlo.

##  I Introduction

In the last years, we have witnessed an extremely rapid development of the
electricity market and services, mainly due to the inclusion of renewable
energy resources in the network, which can cause extensive and rapid changes
in the electrical power load. Combined with growing power consumption, the
existing power transmission lines are utilized to much higher extent than they
were in the past. This has already led to critical bottlenecks and, in the
worst scenarios, also to system-wide instabilities, resulting in blackouts,
e.g. the disturbance in 2006 [1], or the Italian blackout in 2003 [2].

The occurrence of instabilities in the power grid encouraged the transmission
system operators (TSOs) to re-evaluate the measures for maintaining system
stability and to find a way to better utilize the existing infrastructure. A
great help in this process was introduced with the Dynamic Thermal Rating
(DTR) [3] – an instrument for predicting the temperature of a transmission
line based on the load and weather conditions, measured or predicted. The
temperature of a transmission line is often a limiting factor of the power
transmission capacity. However, instead in conductor temperature, TSOs are
often interested in ampacity – the maximal allowed current before the power
line reaches critical temperature, which is fully determined by the weather
data and the material properties of the power line.

TSOs, for the most part, use DTR to alleviate the infrastructural deficiencies
that emerge due to the extensive financial burden and vast societal consensus
for environmental care when acquiring new transmission corridors. Reliable DTR
is also an important part in congestion management of power systems [4],
especially in operation and planning of power systems where renewable energy
intermittencies exist [5, 6]. DTR models have been gradually improved in the
last few decades by including more and more refined physical models for
various phenomena, ranging from rain impinging to different parametrisation of
convective cooling [7, 8, 9, 10]. The advancements in the DTR research
community are periodically collected and presented in comprehensive DTR
guidelines provided by the CIGRE [3], IEEE [11] and IEC [12] standards. The
evolution of DTR physical models and methods, and their integration into
operational use, are thoroughly discussed in recent review papers [13, 14].

The introduction of DTR raised a number of questions related to the
uncertainty of the computed ampacity, level of acceptable risk and its
management. Traditionally, TSOs allow power lines to be utilized up to a
static thermal limit, which is calculated based on relatively conservative
weather parameters (high temperature and sun radiation, and low wind), and
provides safe and secure operation of the power line. With the advent of DTR
systems, which use an indirect approach to the calculation of the dynamic
thermal value, i.e. use of weather parameters along the power line, the
question on how accurate is the dynamic line rating popped-up. Therefore, in
the last years, a growing interest has been shown in applying probabilistic
methods to weather-based DTR procedures, in order to account for the
uncertainty of forecasting ambient conditions. Although recently several
machine learning approaches have been also applied for probabilistic DTR
forecasting [15, 16, 17], the Monte Carlo method is still the most widely used
approach to draw possible deviations in DTR with respect to weather forecasts
[18, 19, 20].

By using probabilistic approaches, the ampacity of a line is described,
instead by a scalar value as in the standard DTR approach, with a Probability
Distribution Function (PDF). By means of Monte Carlo (MC) simulations [21],
the uncertainty of the input parameters to the DTR model, i.e. the weather
forecast uncertainty, is transformed into uncertainty of the predicted
ampacity. This approach was proposed for the first time in [22] and then
further developed by other researchers [18, 19, 23, 24]. Nevertheless, in
these papers, the meteorological data was assumed to follow _a priori_
Gaussian distribution, which is a strong assumption not supported by real data
observations. In a recent paper [20], the authors propose that the PDFs of
weather parameters, which are employed to feed the thermal model of the line
in order to assess its uncertainty, are tuned to the actual weather
forecasting errors at a certain location.

In this paper, we also use the weather data measurements and the weather
forecast data in the proximity of a transmission line, and combine them with
the conductor surface temperature measurements, to determine the expected
uncertainty of DTR prediction. What separates our study, compared to previous
related work, is that we present a concrete solution procedure on how to put a
DTR uncertainty assessment module in operation, calibrated for a specific
location, i.e. operating point. The presented solution procedure could be
useful for any TSO. Moreover, compared to previous studies, we examine the
dependence of the ampacity distribution not only on the weather conditions
(air temperature, solar irradiance, and wind speed and direction), but also on
the conductor type and emissivity. We also assess the weather uncertainty,
examining the dependence of the forecast error on the forecast horizon.
Finally, we assess the DTR uncertainty introduced by the the DTR model itself.

The rest of the paper is organized as follows. In Section II, we present and
analyse the weather data provided by the TSO. Section III describes the
proposed solution procedure for the assessment of the DTR uncertainty using MC
simulations. Before presenting the implementation of the proposed solution for
operational use at TSOs in Section V, some prior analyses, aimed at
simplifying the implementation, are made in Section IV. The paper finalizes
with the conclusions presented in Section VI.

##  II Weather data

As it will be demonstrated in Section III, the main source of ampacity
prediction uncertainty stems from uncertainty of weather data. Therefore, to
understand the uncertainty of ampacity prediction, we have to first analyse
the uncertainty of weather data, which is the goal in this section.

The data for the study has been provided by the Slovenian TSO ELES for two
locations in Slovenia: Podlog (Obersielach-Podlog transmission line) and
Bevkov vrh (Cerkno-Idrija transmission line). The weather data was measured
with two weather stations at the mentioned locations with a time resolution of
5 minutes. The time span covers almost all of year 2019 and a part of year
2020. From the data, we have extracted the following measured weather
parameters:

  * •

$T$ $[$\mathrm{\SIUnitSymbolCelsius}$]$: ambient temperature,

  * •

$S$ $[$\mathrm{W}\text{/}{\mathrm{m}}^{2}$]$: solar irradiance,

  * •

$v$ $[$\mathrm{m}\text{/}\mathrm{s}$]$: wind speed,

  * •

$\phi$ $[$\mathrm{\SIUnitSymbolDegree}$]$: wind direction referencing the
north.

In the following, these data will be referred to as _measured_ weather data.
On the other hand, _predicted_ weather data will refer to the data coming from
the weather forecast model of the TSO, which is derivative of the Aladin model
provided by the Slovenian Environment Agency. The predicted weather data is
consisted of two sets:

  * •

Assessment of current weather conditions or _nowcast_.

  * •

Predicted weather data for different time horizons, in the range between 0 and
72 hours in the future.

Such a data structure allows the relationship between the error distributions
for the weather parameters and the time horizon of the weather forecast to be
investigated.

The goal here is to prepare the input data for the MC simulations, i.e., the
error distributions of the weather parameters computed as the difference
between the measured and the predicted values. We start with the analysis of
the error distributions for the nowcast, given in Fig. 1. For the error
distribution of the solar irradiance, we can observe a large spike at the
value of zero. Such a spike is expected since these examples belong to the
solar irradiance during the night, which is close to zero and is predominantly
correctly predicted. Therefore, we omit the examples where the predicted solar
irradiance is zero and recalculate the error distribution for the solar
irradiance. The resulting PDF is given in Fig. 2. In the following, we refer
to this distribution as the solar error distribution.

At a first glance, the error distributions for all weather parameters do not
seem to belong to a normal Gaussian distribution, as usually assumed in the
literature [18, 19, 23, 24]. We have tested this hypothesis using the
Kolmogorov-Smirnov normality test. The test rejected the hypothesis for all
weather parameters with a high significance level ($p$ value $<0.0001$). This
is also visually demonstrated by fitting normal distribution (red curve) over
the error distributions shown in Fig. 1 and Fig. 2. Therefore, the goal would
be to fit the error data to a custom kernel distribution for each variable
separately.

|
---|---
|
Figure 1: Error distributions for the weather parameters, compared to normal
distribution (red curve), for Bevkov vrh. Figure 2: Error distribution for the
solar irradiance, omitting the zero values, compared to normal distribution
(red curve), for Bevkov vrh.

Next, we look into the dependence of the weather forecast error on the
forecast horizon. The question asked here is: “ _How does the weather forecast
error increase with time?_ ”. We compare the error distributions of the
weather nowcast and forecast for different time horizons for each of the
weather parameters. Two error distributions for the weather forecast for
different time horizons are analysed:

  * •

error distribution for the weather forecast for the time horizon 0-12 hours,
referred to as _short-term_ forecast, and

  * •

error distribution for the weather forecast for the time horizon 12-72 hours,
referred to as _medium-term_ forecast.

Fig. 3 shows the error distributions of the nowcast, and for the short- and
medium-term forecasts, for each weather parameter for Bevkov vrh. We can
observe that, as expected, the nowcast is significantly more accurate than the
forecast, while the difference between the short- and medium-term forecasts
are not that expressive. For each weather parameter, a two-sample Kolmogorov-
Smirnov test was used to test the hypotheses that each pair of error data for
different time horizons come from the same distribution. The tests rejected
all hypotheses, except the ones for wind speed and direction for the
comparison between short- and medium-term forecast distributions. Effectively,
this means that the weather forecast spread is wide enough at a given forecast
time span, so different error distributions should be used for different
forecast horizons.

|
---|---
|
Figure 3: Error distributions for the weather parameters for different
forecast horizons for Bevkov vrh.

##  III Concept of the ampacity uncertainty assessment

In this section, we describe the procedure for obtaining the uncertainty of
the DTR. The core of the solution procedure are MC simulations. The first step
in solving uncertainty problems with the MC method is to build or provide the
model between the output and the inputs, which, in our case is DTR model – a
physical model that transforms the input weather parameters to the output
ampacity, i.e. the maximal load at which the power line temperature will not
exceed a predefined critical temperature. In this study, we use the extended
CIGRE [3] DTR model – DiTeR [7], which is in operative use at the Slovenian
TSO ELES since 2019. The physical model comprises Joule heating, convective
cooling, solar heating, cooling due to evaporation, and radiation. In its
essence, the model solves the heat transport equation (second-order partial
differential equation) with non-linear boundary conditions, describing
different heat terms due to the weather conditions [7]. By solving the heat
transport, DiTeR computes the temperature profile within the conductor,
assuming the line load is known, or the ampacity.

The next step in the uncertainty assessment with the MC method is the
preparation of the input probability distributions, i.e. the PDFs of the
weather parameters. Once the PDFs for all input parameters are known, the MC
procedure samples them by the inverse transformation methods and transforms
the input PDFs to the output PDF, namely the distribution of ampacity. If we
suppose that the confidence interval corresponding to the output confidence of
$100p$% is required, where $p$ is the value of confidence level, then the
number of MC repeated calculations is $M$ times, where $M$ satisfies
$M\geq{\frac{1}{10p}}10^{4}$ [24].

The solution procedure is shown schematically in Fig. 4. It starts by using
the measured data to determine the errors of the input variables. To obtain a
PDF from the samples of weather forecasting errors, a kernel distribution is
fitted over the error data for each variable separately using the Epanechnikov
kernel that optimizes the mean square error between the data and the fit. In
the next step, the weather forecast error distributions are offset to the
given weather conditions $T_{0}$, $S_{0}$, $v_{0}$ and $\phi_{0}$, which gives
us the spread of the given weather conditions, namely the PDFs for temperature
$\mathcal{T}(T)$, solar irradiance $\mathcal{S}(S)$, wind speed
$\mathcal{V}(v)$ and wind direction $\Phi(\phi)$. The constructed PDFs are
truncated at unrealistic values, namely negative wind speed, and negative
solar irradiance or solar irradiance above the solar constant. Then, using the
MC method, the PDFs prepared in the previous step are transformed into
ampacity distribution $\mathcal{I}_{th}$ normalized with the _nominal
ampacity_ $I_{th_{0}}$, i.e. the ampacity computed with DTR at $T_{0}$,
$S_{0}$, $v_{0}$ and $\phi_{0}$. Finally, the lower $I_{th}^{lo}$ and upper
$I_{th}^{hi}$ limits of ampacity are computed by integrating
$\mathcal{I}_{th}$ from the Cumulative Distribution Function (CDF) using
cumulative trapezoidal numerical integration, and then searching for the
values of $I_{th}$ for which CDF($I_{th}$) equals $\frac{p}{2}$ and
$1-\frac{p}{2}$, respectively.

Figure 4: Conceptual representation of the DTR uncertainty estimation.

The uncertainty of DTR stems not only from the uncertainty of the weather
forecast but also from the error of the DTR model itself. The latter can be
assessed using conductor skin temperature ($T_{s}$) measurements provided from
the TSO for the transmission lines on both investigated locations. In Fig. 5
(left), the measured skin temperature of the conductor is compared to the
modelled temperature computed using the measured instead of modelled
meteorological weather data. This way, the generated error includes only the
error of the DTR model and the measurement error of the used sensors, but it
no longer includes the error of the weather nowcast/forecast. In addition to
the DTR calculation based on the measured weather data, we also compute the
conductor temperature with the nowcast meteorological data. Introducing the
DTR error as a difference between computed and measured conductor skin
temperature, we can prepare DTR error distributions (Fig. 5 (right)). We can
see that the error induced by the weather uncertainty dominates, even for the
nowcast.

Figure 5: Time evolution of conductor temperature: measured, computed with
measured weather data and computed with weather forecast data (left). Error
distribution of conductor temperature computed with measured and forecasted
weather parameters (right).

##  IV Impact of weather uncertainty and conductor properties on ampacity
uncertainty

In operational use, the ampacity for all spans in the power grid is typically
computed every time new data becomes available. In the case of the Slovenian
network, this happens every minute. Using MC for uncertainty assessment for
each ampacity prediction would result in a large computational burden.
Therefore, before implementing the proposed procedure in operational use by
the TSO, some prior analyses have been made to optimize the actual
implementation by understanding the relationships between the uncertainty of
ambient factors, material properties and the uncertainty of the ampacity.
Foremost, prior analyses have been made to identify which weather parameters
and material properties of the conductor influence the ampacity probability
distribution the most.

First, we examine the impact of the weather parameters on the ampacity
uncertainty. According to our preliminary analysis of weather parameters, we
calculate normalized ampacity PDFs for each combination (sub-space) of wind
speed $v\in\\{0.15;0.5;2;5;15\\}\ $\mathrm{m}\text{/}\mathrm{s}$$ and wind
direction $\phi\in\\{0,45,90\\}$\mathrm{\SIUnitSymbolDegree}$$, for the
conductor type 243-AL1/39 and emissivity of 0.9. Furthermore, for each wind
speed and angle combination, 9 more combinations for ambient temperature
$T\in\\{0,15,30\\}\ $\mathrm{\SIUnitSymbolCelsius}$$ and solar irradiance
$S\in\\{100,500,1000\\}\ $\mathrm{W}\text{/}{\mathrm{m}}^{2}$$ are used to
compute normalized ampacity distributions. These distributions are represented
in Fig. 6 with blue colour, while their average is represented with red
colour. The spread of normalized ampacity PDFs is relatively low for all sub-
spaces, which means that the ambient temperature and the solar irradiance have
relatively low impact on the normalized ampacity PDF. Therefore, the average
of the corresponding distributions will be used as a representative PDF of a
given sub-space of weather variables.

Figure 6: Normalized ampacity distributions for different combinations of
weather parameters. All plots are for conductor type 243-AL1/39 with
emissivity of 0.9.

Next, we examine the impact of conductor type and emissivity on the ampacity
uncertainty. We compare the ampacity distributions for the two most commonly
used conductors in the Slovenian network: 243-AL1/39 and 490-AL1/64, and for
three values of the emissivity: 0.2, 0.5 and 0.9. Fig. 7 shows the normalized
ampacity distributions for both conductor types and all emissivity values when
varying a single weather parameter at a time. In the first row, distributions
are shown at ambient temperatures of $0\ $\mathrm{\SIUnitSymbolCelsius}$$,
$15\ $\mathrm{\SIUnitSymbolCelsius}$$, and $30\
$\mathrm{\SIUnitSymbolCelsius}$$; in the second row, for solar irradiance of
$100\ $\mathrm{W}\text{/}{\mathrm{m}}^{2}$$, $500\
$\mathrm{W}\text{/}{\mathrm{m}}^{2}$$, and $1000\
$\mathrm{W}\text{/}{\mathrm{m}}^{2}$$; and in the last row, for wind angles of
$0$\mathrm{\SIUnitSymbolDegree}$$, $45$\mathrm{\SIUnitSymbolDegree}$$, and
$90$\mathrm{\SIUnitSymbolDegree}$$. In all calculations, the wind speed is $5\
$\mathrm{m}\text{/}\mathrm{s}$$. The results demonstrate that, for the given
ranges of weather parameters, the distributions are approximately the same for
both conductor types. This was also confirmed by the Kolmogorov-Smirnov test
for all the cases in Fig. 7. While we observe no major differences in the
shape of the ampacity distributions, they however have different peaks due to
different emissivity. The higher the peak, the higher is the emissivity of the
conductor, as it allows higher ampacity.

Figure 7: Normalized ampacity distributions for two conductor types and three
values of emissivity: 0.2, 0.5 and 0.9, and for different values of predicted
weather parameters: air temperature (first row), solar irradiance (second
row), and wind direction (third row). In all calculations, the wind speed is
$5\ $\mathrm{m}\text{/}\mathrm{s}$$. In each graph, the highest peak is for
emissivity of 0.9, the middle for emissivity of 0.5, and the lowest for
emissivity of 0.2.

We also examine the normalized ampacity distribution when the predicted wind
speed varies (Fig. 8). We observe that for wind speeds of $5\
$\mathrm{m}\text{/}\mathrm{s}$$ and $15\ $\mathrm{m}\text{/}\mathrm{s}$$,
similarly to other before-mentioned weather parameters, the distributions are
still practically the same (in shape) for both conductor types. This statement
is again confirmed by Kolmogorov-Smirnov tests. We again observe small
differences in the peaks due to different emissivities. However, the ampacity
distributions for wind speeds of around $2\ $\mathrm{m}\text{/}\mathrm{s}$$
and lower are visibly different in shape and peak height. Nevertheless, the
Kolmogorov-Smirnov test again does not reject the hypothesis of equal
distributions for different conductor types.

Figure 8: Normalized ampacity distributions for two conductor types and three
values of emissivity: 0.2, 0.5 and 0.9, and for different values of predicted
wind speed. The values for the other parameters are: ambient temperature $15\
$\mathrm{\SIUnitSymbolCelsius}$$, solar irradiance $500\
$\mathrm{W}\text{/}{\mathrm{m}}^{2}$$, and wind direction
$90$\mathrm{\SIUnitSymbolDegree}$$.

Besides the most commonly used conductors for the transmission lines,
243-AL1/39 and 490-AL1/64, we have also examined the ampacity distribution for
other conductors used by the TSO: 243-ZTAL/39-HACIN, 149-AL1/24, 149-AL1/24
HACIN, and Cu80. Fig. 9 shows the normalized ampacity distributions for all
listed conductors for different wind speeds, and for constant emissivity of
0.9 and wind direction of $90$\mathrm{\SIUnitSymbolDegree}$$. We can conclude
that in the regimes of wind speed below $2\ $\mathrm{m}\text{/}\mathrm{s}$$,
we can expect for the ampacity distributions to differ with conductor type and
emissivity.

Figure 9: Normalized ampacity distributions for all conductor types used by
the TSO for different values of predicted wind speed. The values for the other
parameters are: ambient temperature $15\ $\mathrm{\SIUnitSymbolCelsius}$$,
solar irradiance $500\ $\mathrm{W}\text{/}{\mathrm{m}}^{2}$$, wind direction
$90$\mathrm{\SIUnitSymbolDegree}$$, and emissivity 0.9.

##  V Implementation of operative DTR uncertainty assessment module

A schematic sketch of the operative DTR uncertainty assessment environment is
shown in Fig. 10. The pre-computed part - preparation of the ampacity
distribution database - is on the right. Because the calculation of the
ampacity probability distribution for a single combination of weather
conditions and conduction properties would be computationally too demanding to
be performed in real-time, the probability distributions are rather pre-
computed for various combinations of input data and stored in a database.
Based on the results of the pre-analysis, we have identified that the time
horizon of the forecast, the wind speed and direction (from the weather
parameters), and the conductor type and emissivity (from the material
properties of the conductor), influence the normalized probability
distribution of the ampacity the most. In accordance with those findings,
therefore, in the space of weather and material variables, only a finite but
large enough number of ampacity distributions are calculated for a finite
number of their combinations. There is a total of 972 combinations stored in
the database, which are made for: 3 values of forecast horizon (nowcast,
short- and medium-term forecast), 6 wind speeds $v\in\\{0.15;0.5;2;5;10;15\\}\
$\mathrm{m}\text{/}\mathrm{s}$$, 3 relative wind directions
$\phi\in\\{0,45,90\\}$\mathrm{\SIUnitSymbolDegree}$$, 6 conductor types, and 3
conductor emissivities $\epsilon\in\\{0.2,0.5,0.9\\}$.

The database is stored in a custom-made binary form, optimized for reading
speed and minimal size. It is also accompanied by a set of Python tools for
editing and viewing the database files. These tools are used when the database
is generated by MC simulations, and can also be used to generate a new
database from a new MC run on another set of weather data. It is important to
note that it would take weeks for the database to be generated on a single
computer. However, the MC approach is highly scalable on parallel computer
architectures, so a computer cluster is best used to reduce the execution
time. A new database has to be generated only when the input data changes
significantly, e.g. when a new line is integrated into the system or after an
additional year of weather data is available.

Figure 10: Schematic representation of the operative DTR uncertainty
estimation.

The left part of Fig. 10 shows how the real-time calculations are performed.
The module implemented in C++ is compiled into a Linux shared library and is
executed periodically to compute ampacity $I_{th}$ with uncertainty from the
supplied weather conditions, forecast horizon, conductor properties, the
nominal ampacity $I_{th_{0}}$ computed with DTR for the given weather
conditions, and the desired confidence level. Ampacity CDFs for the data
points nearest to the input of the algorithm are looked up from the database
and transformed into the resulting ampacity CDF for the input using homotopy -
a smooth transition from one distribution to another. Then the result is
normalized with $I_{th_{0}}$. It should be noted that all the CDFs in the
database and the resulting CDF from the procedure above are in a discrete
form. A continuous form of CDF-1 is taken as the linearly interpolated inverse
of the CDF, which is then finally used to calculate the confidence interval of
the resulting ampacity as described in Section III.

We will demonstrate the homotopic rectilinear routing of the probability
distributions for one weather variable. Note that in real use, the
interpolation is multidimensional, as the weather variable space is. Let us
assume that for some values of the temperature, the solar irradiance, the wind
angle, and the conductor emissivity, we know the normalized ampacity
distribution for wind speeds of $5\ $\mathrm{m}\text{/}\mathrm{s}$$ and $15\
$\mathrm{m}\text{/}\mathrm{s}$$. If we need to obtain the ampacity
distribution for a value of wind speed between $5\
$\mathrm{m}\text{/}\mathrm{s}$$ and $15\ $\mathrm{m}\text{/}\mathrm{s}$$, we
use homotopy between the probability distribution of the two nearest
neighbors, i.e. the distributions at $5\ $\mathrm{m}\text{/}\mathrm{s}$$ and
$15\ $\mathrm{m}\text{/}\mathrm{s}$$. This is demonstrated in Fig. 11, where
the red curve represents the distribution for wind speed of $15\
$\mathrm{m}\text{/}\mathrm{s}$$, and the blue curve represents the
distribution for wind speed of $5\ $\mathrm{m}\text{/}\mathrm{s}$$, while the
green curve is the result of homotopic rectilinear routing for two wind speed
values between $5\ $\mathrm{m}\text{/}\mathrm{s}$$ and $15\
$\mathrm{m}\text{/}\mathrm{s}$$.

Figure 11: Homotopic rectilinear routing from a normalized thermal flow
distribution at wind speed of $5\ $\mathrm{m}\text{/}\mathrm{s}$$ (blue curve)
to a thermal flow distribution at wind speed of $15\
$\mathrm{m}\text{/}\mathrm{s}$$ (red curve). The green curve is their linear
combination or the result of homotopic rectilinear routing.

The execution time of the real-time module is dominated by the database lookup
time. It is measured in tens to hundreds of milliseconds for a single system
data point, which is currently efficient enough for the TSO. Lookups of
multiple system data points, for example, for multiple lines or multiple time
horizons, could be performed in parallel though, if required with future
expansion of the TSO’s system.

The underlying DTR software (DiTeR) at the TSO can run in two modes: as
standalone software and as an embedded system within the SUMO framework –
TSO’s heterogeneous collection of subsystems from different vendors developed
to increase safety and security, as well as the capacity of the existing
transmission network [25]. For efficient integration into the SUMO framework,
the developed DTR uncertainty assessment module uses the same API as the main
DTR module and it is triggered after each DTR calculation. The results are
communicated to the TSO’s software infrastructure by a SOAP-based protocol.
The main computational load is represented by the real-time calculations,
which are triggered once every minute, and forecast calculations, which are
repeated every 5 minutes, for all spans. Currently, the considered
transmission system consists of 29 power lines with more than 2000 spans (Fig.
12).

Figure 12: An overview of the considered transmission system. The two lines
investigated in this paper are plotted in black, while the others in red.

The module for DTR uncertainty assessment was successfully incorporated into
the Slovenian TSO’s operational environment. An example of a result given by
the module is given in Fig. 13. In the plot, the ampacity calculated by the
current DTR model is presented with green line, while the lower and the upper
limit of the ampacity, calculated by the new uncertainty assessment module,
are presented with a yellow and blue line, respectively. In this example, we
observe a situation when both the lower and the upper limit of the ampacity
are greater than the nominal ampacity calculated with DTR. In scenarios with
extremely low forecasted wind speeds, as in this case, the MC method certainly
always samples more favorable cooling conditions, as the wind speed can only
be positive. In an extreme case, the nominal ampacity computed on zero
predicted wind speed is also the minimal ampacity, so all MC trials will give
higher ampacity. Furthermore, on Fig. 1, we can also observe that the wind
speed forecast error is centered around 1 $\mathrm{m}\text{/}\mathrm{s}$, i.e.
in the data we used, the measured wind speed is in average higher than the
forecasted. Therefore, it is expected that in low wind regimes, the nominal
ampacity will be in the lower part (less than 1) of the normalized ampacity
PDF. In another words, it is expected that the actual ampacity in low or no
wind prediction will be higher than the predicted one, as the low wind
prediction stands for the worst case scenario.

As another example of using the implemented module, in Fig. 14 we present the
statistics of the dynamic ampacity, taking into account its uncertainty, with
respect to the static ampacity ($645\ $\mathrm{A}$$ for the 243-AL1/39
conductor) and the measured load. An interesting conclusion from Fig. 14 is
that, when taking into account realistic weather data, even the conservative
static limit can be too high. We observed that, for the confidence level of
99%, the lower limit of the thermal current is lower than the static current
for 5.21% of the time. Nevertheless, as expected, most of the time the dynamic
ampacity is well above the static limit, and the actual load is always below
the predicted ampacity.

Figure 13: Example plot from the module for DTR uncertainty assessment
presenting the calculated ampacity with a given lower and upper limit for the
Obersielach-Podlog transmission line on 12 July, 2020. Figure 14: Comparison
of static, dynamic and measured current distributions for the Bevkov vrh
location.

##  VI Conclusions

In this paper, we presented a solution for estimating the uncertainty of the
ampacity as a result of the DTR. First, we examined the weather forecast and
measurement data, together with on-site measurements of conductor skin
temperature, on two locations in Slovenia: Podlog and Bevkov vrh. We analyzed
the dependence of the weather forecast error from the forecast horizon:
nowcast, short-term forecast and medium-term forecast, with conclusion that
there is no significant difference in quality between short-term and medium-
term wind forecast. With the help of conductor surface temperature
measurements, weather data measurements and predicted weather data, we
estimated also the error of the DTR itself, with a clear conclusion that the
majority of the ampacity forecast uncertainty originates from the weather
forecast uncertainty. Furthermore, we demonstrated that the influence of the
ambient temperature and the solar irradiance forecast error on the normalized
ampacity distribution is much less important than the influence of the wind
velocity forecast error. We also investigated the influence of different
conductor types and their emissivity and have concluded that they are not
negligible either.

Following the results of the data analyses, we designed and implemented a
complete operational solution for estimating the DTR uncertainty. Using the
Monte Carlo method, we randomly sampled the fitted-to-location error
distributions of the input weather conditions and used the DTR model to
calculate the ampacity distribution. In accordance with the findings, in the
space of weather and material variables (two weather parameters and two
material properties that significantly influence the normalized ampacity
probability distribution), we calculated only a finite number of ampacity
distributions for a finite number of input combinations and used homotopic
rectilinear routing for computing intermediate values. When compared to
previous related studies, this type of solution procedure for estimating the
DTR uncertainty is presented for the first time in this paper. The developed
solution was implemented as a stand-alone module and integrated into the
environment of the operator of the Slovenian electric power transmission
network – ELES, where it is in operational use since August 2020.

The main limitation of this work is that the final result greatly depends on
the available measured data. Nevertheless, the TSOs have great interest in
installing new weather stations along the lines, and with steadily increasing
the amount of available on-site weather data, the system will provide more
reliable predictions, as it is designed to periodically update the
distributions as more data becomes available. Future work will be focused on
including ensemble weather predictions that inherently provide uncertainty for
each specific weather parameter, which could be used on locations where
measurements are not available.

## References

  * [1] J. W. Bialek, “Why has it happened again? Comparison between the UCTE blackout in 2006 and the blackouts of 2003,” in _2007 IEEE Lausanne Power Tech_. IEEE, 2007, pp. 51–56.
  * [2] A. Berizzi, “The Italian 2003 blackout,” in _IEEE Power Engineering Society General Meeting, 2004._ IEEE, 2004, pp. 1673–1679.
  * [3] “Guide for Thermal Rating Calculations of Overhead Lines,” CIGRE, Technical brochure No. 601, Dec. 2014.
  * [4] M. Mahmoudian Esfahani and G. R. Yousefi, “Real Time Congestion Management in Power Systems Considering Quasi-Dynamic Thermal Rating and Congestion Clearing Time,” _IEEE Transactions on Industrial Informatics_ , vol. 12, no. 2, pp. 745–754, 2016.
  * [5] M. Dabbaghjamanesh, A. Kavousi-Fard, and S. Mehraeen, “Effective Scheduling of Reconfigurable Microgrids With Dynamic Thermal Line Rating,” _IEEE Transactions on Industrial Electronics_ , vol. 66, no. 2, pp. 1552–1564, 2019.
  * [6] M. Dabbaghjamanesh, A. Kavousi-Fard, S. Mehraeen, J. Zhang, and Z. Y. Dong, “Sensitivity Analysis of Renewable Energy Integration on Stochastic Energy Management of Automated Reconfigurable Hybrid AC–DC Microgrid Considering DLR Security Constraint,” _IEEE Transactions on Industrial Informatics_ , vol. 16, no. 1, pp. 120–131, 2020.
  * [7] G. Kosec, M. Maksić, and V. Djurica, “Dynamic thermal rating of power lines–model and measurements in rainy conditions,” _International Journal of Electrical Power & Energy Systems_, vol. 91, pp. 222–229, 2017.
  * [8] P. Pytlak, P. Musilek, E. Lozowski, and J. Toth, “Modelling precipitation cooling of overhead conductors,” _Electric Power Systems Research_ , vol. 81, no. 12, pp. 2147–2154, 2011.
  * [9] S. Karimi, P. Musilek, and A. M. Knight, “Dynamic thermal rating of transmission lines: A review,” _Renewable and Sustainable Energy Reviews_ , vol. 91, pp. 600–612, 2018.
  * [10] M. Maksić, V. Djurica, A. Souvent, J. Slak, M. Depolli, and G. Kosec, “Cooling of overhead power lines due to the natural convection,” _International Journal of Electrical Power & Energy Systems_, vol. 113, pp. 333–343, 2019.
  * [11] “IEEE Standard for Calculating the Current-Temperature Relationship of Bare Overhead Conductors,” IEEE, Std 738-2012, Dec. 2013.
  * [12] “Overhead electrical conductors - Calculation methods for stranded bare conductors,” International Electrotechnical Commission (IEC), Technical Report 61597, Oct. 2016, document 7/661/DC.
  * [13] F. G. Erdinç, O. Erdinç, R. Yumurtacı, and J. P. Catalão, “A comprehensive overview of dynamic line rating combined with other flexibility options from an operational point of view,” _Energies_ , vol. 13, no. 24, p. 6563, 2020.
  * [14] D. A. Douglass, J. Gentle, H.-M. Nguyen, W. Chisholm, C. Xu, T. Goodwin, H. Chen, S. Nuthalapati, N. Hurst, I. Grant _et al._ , “A review of dynamic thermal line rating methods with forecasting,” _IEEE Transactions on Power Delivery_ , vol. 34, no. 6, pp. 2100–2109, 2019.
  * [15] J. L. Aznarte and N. Siebert, “Dynamic Line Rating Using Numerical Weather Predictions and Machine Learning: A Case Study,” _IEEE Transactions on Power Delivery_ , vol. 32, no. 1, pp. 335–343, 2017.
  * [16] F. Fan, K. Bell, and D. Infield, “Probabilistic Real-Time Thermal Rating Forecasting for Overhead Lines by Conditionally Heteroscedastic Auto-Regressive Models,” _IEEE Transactions on Power Delivery_ , vol. 32, no. 4, pp. 1881–1890, 2017.
  * [17] J. Zhan, C. Y. Chung, and E. Demeter, “Time Series Modeling for Dynamic Thermal Rating of Overhead Lines,” _IEEE Transactions on Power Systems_ , vol. 32, no. 3, pp. 2172–2182, 2017.
  * [18] A. Michiorri and P. C. Taylor, “Forecasting real-time ratings for electricity distribution networks using weather forecast data,” in _Proceedings of the 20th International Conference and Exhibition on Electricity Distribution (CIRED)_ , 2009.
  * [19] T. Ringelband, P. Schäfer, and A. Moser, “Probabilistic ampacity forecasting for overhead lines using weather forecast ensembles,” _Electrical Engineering_ , vol. 95, no. 2, pp. 99–107, 2013.
  * [20] D. Poli, P. Pelacchi, G. Lutzemberger, T. B. Scirocco, F. Bassi, and G. Bruno, “The possible impact of weather uncertainty on the dynamic thermal rating of transmission power lines: A monte carlo error-based approach,” _Electric Power Systems Research_ , vol. 170, pp. 338–347, 2019.
  * [21] G. Fishman, _Monte Carlo: Concepts, Algorithms and Applications_. New York: Springer-Verlag, 1996.
  * [22] E. Siwy, “Risk Analysis in Dynamic Thermal Overhead Line Rating,” in _Proceedings of the International Conference on Probabilistic Methods Applied to Power Systems_ , 2006.
  * [23] S. Karimi, A. M. Knight, P. Musilek, and J. Heckenbergerova, “A probabilistic estimation for dynamic thermal rating of transmission lines,” in _Proceedings of the IEEE 16th International Conference on Environment and Electrical Engineering (EEEIC)_ , 2016.
  * [24] Y. Wang, W. Tao, Z. Yan, and R. Wei, “Uncertainty analysis of dynamic thermal rating based on environmental parameter estimation,” _EURASIP Journal on Wireless Communications and Networking_ , 2018, article No. 167.
  * [25] G. Lakota, J. Kosmač, J. Kostevc, A. Souvent, and T. Fatur, “Real-time and short-term forecast assessment of power grid operating Limits-SUMO,” in _Actual trends in development of Power System Relay Protection and Automation_ , 2015.

|  Aleksandra Rashkovska (M’14) received B.Sc. degree in Electrical
Engineering from the University Ss. Cyril and Methodius, Skopje, Macedonia, in
2006, and Ph.D. degree in Computer Science from the Jožef Stefan International
Postgraduate School, Ljubljana, Slovenia, in 2013. She is currently a Research
Fellow at the Department of Communication Systems, Jožef Stefan Institute,
Ljubljana, Slovenia. Her research interest includes advanced bio-signal
analysis, computer simulations in biomedicine, biomedical applications of data
mining and control theory, and data mining in sensor networks.
---|---
|  Mitja Jančič received the B.S. in physics continued to M.Sc. in mechanical
engineering, both at the University of Ljubljana in Slovenia. He is currently
in pursuit of a PhD at the Jožef Stefan international postgraduate school. He
also holds the position of a research fellow at the Department of
Communication Systems at the Jožef Stefan Institute in Ljubljana. His research
interests include meshless methods, numerical procedures for solving PDE
systems and generic programming algorithms.
---|---
|  Matjaž Depolli received a PhD in computer and information science from
Jožef Stefan International Postgraduate School Ljubljana in 2010. He currently
holds the position of research fellow at the Department of Communication
Systems at the Jožef Stefan Institute in Ljubljana. His research interests
include evolutionary computation, computer simulation of physical phenomena,
parallel computing, and ECG analysis. He’s been involved in the development of
wireless body sensors, software for ECG analysis, cluster management software,
and cloud services.
---|---
|  Janko Kosmač studied at the University of Ljubljana, Slovenia, where he
received his BSc (1990), MSc (1993) and PhD (1996) degrees. In 1996, he has
been employed at the Elektroinštitut Milan Vidmar (EIMV), where he was leading
the development of the Slovenian lightning localization system. In 2007, he
was promoted to the head of the electric power system control and operation
department at EIMV. He joined ELES, the Slovenian transmission system
operator, in 2013. He is head of process system department and project manager
of the dynamic thermal rating project at ELES.
---|---
|  Gregor Kosec (M’19) graduated at University of Ljubljana, Faculty of
Mathematics and Physics in 2006 and obtained PhD in 2011 at the University of
Nova Gorica. In 2011 he became a member of Parallel and Distributed Systems
Laboratory at Jožef Stefan Institute. In 2020 he became head of the Parallel
and Distributed Systems Laboratory. His main research interest covers
computational modelling, meshless methods, and generic programming.
---|---