In recent years, Low-Earth Orbit (LEO) mega-constellations have emerged as a promising network technology and have ushered in a new era for democratizing Internet access.
The Starlink network from SpaceX stands out as the only consumer-facing LEO network with over 2M+ customers and more than 4000 operational satellites.
In this paper, we conduct the first-of-its-kind extensive multi-faceted analysis of Starlink network performance leveraging several measurement sources.
First, based on 19.2M crowdsourced M-Lab speed test measurements from 34 countries since 2021, we analyze Starlink global performance relative to terrestrial cellular networks.
Second, we examine Starlink's ability to support real-time web-based latency and bandwidth-critical applications by analyzing the performance of (i) Zoom video conferencing, and (ii) Luna cloud gaming, comparing it to 5G and terrestrial fiber.
Third, we orchestrate targeted measurements from Starlink-enabled RIPE Atlas probes to shed light on the last-mile Starlink access and other factors affecting its performance globally.
Finally, we conduct controlled experiments from Starlink dishes in two countries and analyze the impact of globally synchronized “15-second reconfiguration intervals” of the links that cause substantial latency and throughput variations.
Our unique analysis provides revealing insights on global Starlink functionality and paints the most comprehensive picture of the LEO network's operation to date.

§ INTRODUCTION

Over the past two decades, the Internet's reach has grown rapidly, driven by innovations and investments in wireless access <cit.> (both cellular and WiFi) and fiber backhaul deployment that has interconnected the globe <cit.>.
Yet, the emergence of Low-Earth Orbit (LEO) satellite networking, spearheaded by ventures like Starlink <cit.>, OneWeb <cit.>, and Kuiper <cit.>, is poised to revolutionize global connectivity.
LEO networks consist of mega-constellations with thousands of satellites orbiting at 300–2000 km altitudes, offering ubiquitous low latency coverage worldwide.
Consequently, these networks are morphing into “global ISPs” capable of challenging existing Internet monopolies <cit.>, bridging connectivity gaps in remote regions <cit.>, and providing support in disaster-struck regions with impaired terrestrial infrastructure <cit.>.

Starlink from SpaceX stands out with its expansive fleet of 4000 satellites catering to 2M+ subscribers across 63 countries <cit.>.
The LEO operator plans to further amplify its coverage and quality of service (QoS) by launching $\approx$ 42,000 additional satellites in the coming years <cit.>.
However, despite significant global interest and the potential to impact the existing Internet ecosystem, only limited explorations have been made within the research community to understand Starlink's performance.
The challenge stems from a lack of global vantage points required to accurately gauge the network's performance since factors such as orbital coverage, density of ground infrastructure, etc., can impact connectivity across regions.
Initial studies have resorted to measurements from a handful of geographical locations <cit.> or extrapolated global performance through simulations <cit.> and emulations <cit.>.
However, the community agrees on the limited scope of such studies and has made open calls to establish a global LEO measurement testbed to address this challenge <cit.>.
Some researchers have navigated around this hurdle by exploring alternative measurement methods, e.g., by targeting exposed services behind user terminals <cit.> or by mining speed test reports shared on social media platforms, such as Reddit <cit.>.
While innovative, we argue that these techniques are insufficient to uncover the intricacies affecting the network, specifically its capability to support web applications.

This paper addresses this knowledge gap and provides the first comprehensive multi-faceted measurement study on Starlink.
Our work is distinct from previous works in several ways.
we examine the global evolution of the network since 2021 by analyzing the M-Lab speed test measurements <cit.> from 34 countries (largest so far).
We complement our investigation
through active measurements over 98 RIPE Atlas <cit.> probes in 21 countries and
conduct high-resolution experiments over controlled terminals in two European countries to investigate real-time web application performance and factors impacting Starlink's last-mile access.
Specifically, we make the following contributions.

(1) We present a longitudinal study of global Starlink latency and throughput performance from M-Lab users
in <Ref>.
Our analysis, incorporating $\approx$ 19.2 M samples, reveals that Starlink performs competitively to
terrestrial cellular networks.
However, its performance varies globally due to infrastructure deployment differences, and is dependent on
the density and closeness of ground stations and Point-of-Presence (PoP).
We also observe signs of bufferbloating as Starlink's latency increases by several factors under traffic load.

(2) We assess and compare the
performance of real-time web applications, specifically Zoom video conferencing and Amazon Luna cloud gaming, to terrestrial networks (<Ref>).
We find that, under optimal conditions, Starlink is capable of supporting such applications, matching the performance over cellular; however, we do observe some artifacts due to the network's periodic reconfigurations.

(3) We perform targeted measurements from Starlink RIPE Atlas <cit.> probes and leverage their diverse locations to characterize the satellite last-mile “bent-pipe” performance (<Ref>).
We find that the “bent-pipe” latency within the dense 53$\degree$ shell remains consistent worldwide ($\approx$ 40 ms), and is significantly lower to yet incomplete 70$\degree$ and 97.6$\degree$ orbits.
We also find evidence of Starlink inter-satellite links (ISLs) connecting remote regions, showcasing superior performance to terrestrial paths in our case study.

(4) Our high-frequency measurements from terminals in two European countries
confirm that Starlink performs network reconfigurations every 15s, leading to noticeable latency and throughput degradations at sub-second granularity.
By correlating data from our terminals, one covered by 53$\degree$ and the other restricted to 70$\degree$ and 97.6$\degree$ connectivity, we find that the reconfigurations are globally synchronized events and likely independent of satellite handovers.

Leveraging multi-dimensional, global, and controlled high resolution measurements, our findings distinctively advance the state-of-the-art by illuminating Starlink's global performance and the influence of internal network operations on real-time web applications.

§ BACKGROUND

Orbits of three Starlink inclinations and crowdsourced Ground Station (GS) and Point-of-Presence (PoP) locations <cit.>. Shaded regions depict Starlink's service area.

Starlink is a LEO satellite network operated by SpaceX that aims to provide global Internet coverage through a fleet of satellites flying at $\approx$ 500 km above the Earth's surface.
The majority of Starlink's operational 4000 satellites lie within the 53$\degree$ shell, which only covers parts of the globe (see <Ref>).
The 70$\degree$ and 97.6$\degree$ orbits allow serving regions near the poles.
These other shells however have fewer satellites
(see <Ref>, <Ref> for constellation details).

<Ref> shows the cross-section of Starlink end-to-end connectivity.
To access the Internet over the Starlink network, end-users require a dish, a.k.a. “Dishy”[We use “Dishy” and “user terminal” interchangeably in the paper.], that communicates with satellites visible above 25$\degree$ of elevation through phased-array antennas using Ku-band (shown as User Link (UL)).
Starlink satellites, equipped with multiple antennas subdivided into beams, can connect to multiple terminals simultaneously <cit.> and relay all connections to a ground station (GS) on a Ka-band link (shown in green).
The connection forms a direct “bent-pipe” in case the terminal and GS lie within a single satellite's coverage cone; otherwise, the satellites can relay within space to reach far-off GSs via laser inter-satellite links (ISLs), forming an “extended bent-pipe”.
Note that not all Starlink satellites are ISL-capable and it is difficult to effectively estimate ISL usage as Starlink satellites have no user visibility at IP layer and, therefore, do not show up in s.

Finally, the GSs relay traffic from satellites to Starlink point-of-presence (PoP) through a wired connection, which routes it to the destination server via terrestrial Internet <cit.>.
The public availability of GS deployment information differs across countries.
No official source exists, so we rely on crowdsourced data for the geolocations of GSs and PoPs <cit.>, which is also shown in <Ref>.

Starlink follows “bent-pipe” connectivity as traffic traverses the client-side terminal, one or more satellites via inter-sat links (ISLs), nearest ground station (GS), ingressing with the terrestrial Internet via a point-of-presence (PoP).

§ GLOBAL STARLINK PERFORMANCE

Median of minimum RTT (in ms) of devices connected via Starlink (left) and top-3 serving ISPs (right) in the same country to the nearest M-Lab server.

Starlink latency distribution from select cities in each continent.

§ REAL-TIME APPLICATION PERFORMANCE

While the global Starlink performance in <Ref> is promising for supporting web-based applications, it does not accurately capture the potential impact of minute network changes caused by routing, satellite switches, bufferbloating, etc., on application performance.
Real-time web applications are known to be sensitive to such fluctuations <cit.>.
In this section, we examine the performance of Zoom and Amazon Luna cloud gaming over Starlink
(see <Ref> for details).
This allows us to assess the suitability of the LEO network to meet the requirements of the majority of real-time Internet-based applications, as both applications impose a strict latency control loop.
Cloud gaming necessitates high downlink bandwidth, while Zoom utilizes uplink and downlink capacity simultaneously.

Distribution of median (a) download and (b) upload goodput over Starlink from selected cities globally.

Zoom Video Conferencing
<Ref> shows samples from Zoom calls conducted over a high-speed terrestrial network and over Starlink.
The total uplink throughput over Starlink is slightly higher,
which we trace to FEC (Forward Error Correction) packets that are frequently sent in addition to raw video data (on average 14699 Kbps vs. 22 Kbps over terrestrial).
The frame rate, inferred from the packets received by the Zoom peer, does not meaningfully differ between the two networks ($\approx$ 27 FPS).
Note that, since Zoom does not saturate the available uplink and downlink capacity, it should not be impacted by bufferbloating.
Yet, we observe a slightly higher loss rate over LEO, which the application combats by proactively utilizing FEC.
The uplink one-way delay (OWD) over Starlink is higher and more variable compared to the terrestrial connection (on average 5214 ms vs. 277 ms).
All observations also apply to the downlink except that Starlink's downlink latency (3511 ms) is similar to the terrestrial connection (327 ms).
Our analysis broadly agrees with <cit.> but
our packet-level insight reveals bitrate fluctuations partly caused by FEC.
Further, our Starlink connection was more reliable and we did not experience second-long outages.

Interestingly, we observe that the Starlink OWD often noticeably shifts at interval points that occur at 15 s increments.
Further investigation reveals the cause to be the Starlink reconfiguration interval, which, as reported in FCC filings <cit.>, is the time-step at which the satellite paths are reallocated to the users.
Other recent work also reports periodic link degradations at 15 s boundaries in their experiments, with RTT spikes and packet losses of several orders <cit.>.
We explore the impact of reconfiguration intervals and other Starlink-internal actions on network performance in <Ref>.

Terrestrial              Cellular                 Starlink
Idle RTT (ms)            9                        46                       40

Throughput (Mbps)        1000                     150                      220

Frames-per-second        591.51        591.68       591.63

Bitrate (Mbps)           23.080.38       22.824.24       22.812.16

Time at 1080p (%)       100                      94.11                    99.45

Freezes (ms/min)         00              0220.34      0119.74

Inter-frame (ms)         173.65       1811.1       166.76

Game delay (ms)          133.5319.79     165.8223.55     167.1323.12

RTT (ms)                 1113.41      3917.06      5016.28

Jitter buffer (ms)       153.27       121.33        153.35

The game metrics are aggregated over 150 minutes of playtime per connection.
Values denote medianSD and the worst performer is highlighted.

Uplink Zoom video traffic over a terrestrial network (left) and Starlink (right). Vertical dashed lines show Starlink reconfiguration intervals.

Amazon Luna Cloud Gaming
<Ref> shows 150 minutes of cloud gaming performance over terrestrial, 5G cellular, and Starlink networks.
Overall, all networks realized close to 60 FPS playback rate at consistently high bitrate ($\approx$ 20 Mbps).
Starlink lies in between the better-performing terrestrial and cellular in terms of
bitrate fluctuations, frame drops and freezes
[Freeze is when the inter-frame delay (IFD) is larger than $\text{max}(3\times$$\text{IFD}, \text{IFD} + 150)$.].
Starlink exhibits the highest game delay, i.e., the delay experienced by the player between issuing a command and witnessing its effect.
Specifically, the wired network delivers the visual response about 2 frames ($\approx$ 33 ms) earlier than both 5G and Starlink.
While examining the gaming performance over time, we observe occasional drops to < 20 FPS over Starlink (see <Ref>), that coincide with Starlink's reconfiguration interval.
These fluctuations are only visible at sub-second granularity and, hence, are not reflected in global performance analysis (<Ref>).

Despite these variations, Starlink's performance remains competitive with 5G, highlighting its potential to deliver real-time application support, especially in regions with less mature cellular infrastructure.
Note, however, that our Starlink terminal was set up without obstructions and the weather conditions during measurements were favorable to its operation <cit.>.
Different conditions, especially mobility, may change the relative performance of Starlink and cellular, which we plan to explore further in the near future.

Cloud gaming over 5G (left) and Starlink (right). Vertical dashed lines show Starlink reconfiguration intervals.

[title=Takeaway #2, enhanced, breakable]
Starlink is competitive with the current 5G deployment for supporting demanding real-time applications.
We also observe that Starlink experiences regular performance changes every 15s linked to its reconfiguration interval period.
While these internal black-box parameters do influence performance to a certain extent, application-specific corrective measures, like FEC, are effective in mitigating these artifacts.

§ DISSECTING THE BENT-PIPE

We now attempt to uncover Starlink's behind-the-scenes operations and their impact on network performance.
We follow a two-pronged approach to undertake this challenge.
Our longitudinal  measurements over RIPE Atlas accurately isolate the bent-pipe (terminal-to-PoP) global performance, allowing us to correlate it with parameters like ground station deployment, satellite availability, etc. (<ref>).
We then perform high-frequency, high-resolution experiments over Starlink terminals deployed in two EU countries to zoom in on bent-pipe operation and highlight traffic engineering signatures that may impact application performance (<Ref>).

§ RELATED WORK

LEO satellites have become a subject of extensive research in recent years, with a particular focus on advancing the performance of various systems and technologies.
Starlink, the posterchild of LEO networks, continues to grow in its maturity and reach with $>$ 2M subscribers as of September 2023 <cit.>.
Despite its growing popularity, there has been limited exploration into measuring Starlink's performance so far.
Existing studies either have a narrow scope, employing only a few vantage points <cit.> or focus on broad application-level operation <cit.> without investigating root-causes.
<cit.> embarked on a journey across Canada with four dishes to scrutinize various factors, such as temperature and weather, that might influence Starlink's performance.

A few endeavors have attempted to unveil the operations of Starlink's black-box network.
<cit.> revealed the operator's internal network topology from s, whereas <cit.> spotlighted a potential global network controller.
The absence of global measurement sites poses a predominant challenge hampering a comprehensive understanding of Starlink's performance. As we show in this work, Starlink's performance varies geographically due to differing internal configurations and ground infrastructure availability.
Some researchers have devised innovative methods to combat this.
For example, <cit.> conducted measurements towards exposed services behind the Starlink user terminal, while <cit.> mined social media platforms like Reddit to gauge the LEO network's performance.
Our study not only corroborates and extends existing findings but also stands as the most extensive examination to date.
Our approach – anchored in detailed insights from 34 countries, leveraging 19.2 million crowdsourced M-Lab measurements, 2.9 million active RIPE Atlas measurements, and two controlled terminals connecting to different Starlink orbits – provides a deeper understanding of the Starlink “bent-pipe” and overall performance.

§ CONCLUSIONS

Despite its potential as a “global ISP” capable of challenging the state of global Internet connectivity, there have been limited performance evaluations of Starlink to date.
We conducted a multi-faceted investigation of Starlink, providing insights from a global perspective down to internal network operations.
Globally, our analysis showed that Starlink is comparable to cellular for supporting real-time applications (in our case Zoom and Luna cloud gaming), though this varies based on proximity to ground infrastructure.
Our case study shows Starlink inter-satellite connections helping remote users achieve better Internet service than terrestrial networks.
However, at sub-second granularity, Starlink exhibits performance variations, likely due to periodic internal network reconfigurations at 15s intervals.
We find that the reconfigurations are synchronized globally and are not caused only by satellite handovers.
As such, this first-of-its-kind study is a step towards a clearer understanding of Starlink’s operations and performance as it continues to evolve.

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

authorpersonRichelle Adams.
Active queue management: A survey.
journalIEEE communications surveys & tutorials
volume15, number3 (year2012),

authorpersonAmazon. year2023.
titleLuna Cloud Gaming.

[Arnold et al(2020)]
authorpersonTodd Arnold, personEge
Gürmeriçliler, personGeorgia Essig, personArpit
Gupta, personMatt Calder, personVasileios Giotsas,
and personEthan Katz-Bassett. year2020.
(How Much) Does a Private WAN Improve Cloud
Performance?. In booktitleIEEE INFOCOM 2020 - IEEE
Conference on Computer Communications. pages79–88.

[Baltaci et al(2022)]
authorpersonAygün Baltaci, personHendrik
Cech, personNitinder Mohan, personFabien Geyer,
personVaibhav Bajpai, personJörg Ott, and
personDominic Schupke. year2022.
Analyzing Real-Time Video Delivery over Cellular
Networks for Remote Piloting Aerial Vehicles (seriesIMC
'22). publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages98–112.

[Baltaci et al(2023)]
authorpersonAygun Baltaci, personKaushik
Chavali, personMike Kosek, personNitinder Mohan,
personDominic Schupke, and personJörg Ott.
Multipath Transport Analysis over Cellular and LEO
Access for Aerial Vehicles.
journalIEEE Access (year2023).
Accepted, to appear in the next issue.

authorpersonAlan Boyle.
titleAmazon to offer broadband access from orbit with
3,236-satellite `Project Kuiper' constellation.
Accessed: 2023-05-26.

authorpersonCAIDA. year2023.
titleASRank Website.
Accessed: 2023-10.

authorpersonShkelzen Cakaj.
The parameters comparison of the “Starlink” LEO
satellites constellation for different orbital shells.
journalFrontiers in Communications and Networks
volume2 (year2021), pages643095.

[Case No. 2021-00002(2021)]
authorpersonCase No. 2021-00002.
booktitleApplication of Starlink Services.

Accessed on 2023-05-24.

[Chang et al(2021)]
authorpersonHyunseok Chang, personMatteo
Varvello, personFang Hao, and personSarit
Mukherjee. year2021.
Can You See Me Now? A Measurement Study of Zoom,
Webex, and Meet. In booktitleProceedings of the 21st ACM
Internet Measurement Conference (Virtual Event) (seriesIMC
'21). publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages216–228.

[Chiu et al(2015)]
authorpersonYi-Ching Chiu, personBrandon
Schlinker, personAbhishek Balaji Radhakrishnan,
personEthan Katz-Bassett, and personRamesh
Govindan. year2015.
Are We One Hop Away from a Better Internet?. In
booktitleProceedings of the 2015 Internet Measurement
Conference (Tokyo, Japan) (seriesIMC '15).
publisherAssociation for Computing Machinery,
addressNew York, NY, USA, numpages7 pages.

[Corneo et al(2021a)]
authorpersonLorenzo Corneo, personMaximilian
Eder, personNitinder Mohan, personAleksandr
Zavodovski, personSuzan Bayhan, personWalter Wong,
personPer Gunningberg, personJussi Kangasharju, and
personJörg Ott. year2021a.
Surrounded by the Clouds: A Comprehensive Cloud
Reachability Study. In booktitleProceedings of the Web
Conference 2021 (Ljubljana, Slovenia) (seriesWWW '21).
publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages295–304.

[Corneo et al(2021b)]
authorpersonLorenzo Corneo, personNitinder
Mohan, personAleksandr Zavodovski, personWalter
Wong, personChristian Rohner, personPer Gunningberg,
and personJussi Kangasharju. year2021b.
(How Much) Can Edge Computing Change Network
Latency?. In booktitle2021 IFIP Networking Conference (IFIP
Networking). pages1–9.

[Dang et al(2021)]
authorpersonThe Khang Dang, personNitinder
Mohan, personLorenzo Corneo, personAleksandr
Zavodovski, personJörg Ott, and personJussi
Kangasharju. year2021.
Cloudy with a Chance of Short RTTs: Analyzing Cloud
Connectivity in the Internet. In booktitleProceedings of
the 21st ACM Internet Measurement Conference (Virtual Event)
(seriesIMC '21). publisherAssociation for
Computing Machinery, addressNew York, NY, USA,

[Feldmann et al(2020)]
authorpersonAnja Feldmann, personOliver
Gasser, personFranziska Lichtblau, personEnric
Pujol, personIngmar Poese, personChristoph Dietzel,
personDaniel Wagner, personMatthias Wichtlhuber,
personJuan Tapiador, personNarseo Vallina-Rodriguez,
personOliver Hohlfeld, and personGeorgios
Smaragdakis. year2020.
The Lockdown Effect: Implications of the COVID-19
Pandemic on Internet Traffic. In booktitleProceedings of
the ACM Internet Measurement Conference (Virtual Event, USA)
(seriesIMC '20). publisherAssociation for
Computing Machinery, addressNew York, NY, USA,

[Garcia et al(2023)]
authorpersonJohan Garcia, personSimon
Sundberg, personGiuseppe Caso, and personAnna
Brunstrom. year2023.
Multi-Timescale Evaluation of Starlink Throughput.
In booktitleProceedings of the 1st ACM Workshop on LEO
Networking and Communication (Madrid, Spain)
(seriesLEO-NET '23). publisherAssociation for
Computing Machinery, addressNew York, NY, USA,

[Gill et al(2022)]
authorpersonPhillipa Gill, personChristophe
Diot, personLai Yi Ohlsen, personMatt Mathis, and
personStephen Soltesz. year2022.
M-Lab: User Initiated Internet Data for the
Research Community.
journalSIGCOMM Comput. Commun. Rev.
volume52, number1 (datemar
year2022), pages34–37.

[Grazia et al(2017)]
authorpersonCarlo Augusto Grazia,
personNatale Patriciello, personMartin Klapez, and
personMaurizio Casoni. year2017.
Mitigating congestion and bufferbloat on satellite
networks through a rate-based AQM. In booktitle2017 IEEE
International Conference on Communications (ICC). pages1–6.

authorpersonBusiness Insider.
booktitleEverything we know about Elon Musk's
Starlink satellites and future internet plans.

Accessed: 2023-10-12.

authorpersoniperf. year2023.
titleiperf / iperf3.

[Iqbal et al(2021)]
authorpersonHassan Iqbal, personAyesha
Khalid, and personMuhammad Shahzad.
Dissecting Cloud Gaming Performance with DECAF.
journalProc. ACM Meas. Anal. Comput. Syst.
volume5, number3, Article articleno31
(datedec year2021), numpages27 pages.

[Iyer et al(2022)]
authorpersonJayasuryan V Iyer, personKhasim
Shaheed Shaik Mahammad, personYashodhan Dandekar,
personRamakrishna Akella, personChen Chen,
personPhillip E Barber, and personPeter J Worters.
titleSystem and method of providing a medium access
control scheduler.
US Patent 11,540,301.

[Izhikevich et al(2023)]
authorpersonLiz Izhikevich, personManda
Tran, personKatherine Izhikevich, personGautam
Akiwate, and personZakir Durumeric.
titleDemocratizing LEO Satellite Network Measurement.
[arxiv]2306.07469 [cs.NI]

[Jain et al(2022)]
authorpersonAkshath Jain, personDeepayan
Patra, personPeijing Xu, personJustine Sherry, and
personPhillipa Gill. year2022.
The Ukrainian Internet under Attack: An NDT
Perspective. In booktitleProceedings of the 22nd ACM
Internet Measurement Conference (Nice, France) (seriesIMC
'22). publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages166–178.

authorpersonAmritha Jayanti.
titleStarlink and the Russia-Ukraine War: A Case of
Commercial Technology and Public Purpose?

[Jian et al(2020)]
authorpersonYubing Jian, personMohit
Agarwal, personShyam Krishnan Venkateswaran,
personYuchen Liu, personDouglas M. Blough, and
personRaghupathy Sivakumar. year2020.
WiMove: Toward Infrastructure Mobility in MmWave
WiFi. In booktitleProceedings of the 18th ACM Symposium on
Mobility Management and Wireless Access (Alicante, Spain)
(seriesMobiWac '20). publisherAssociation for
Computing Machinery, addressNew York, NY, USA,

[Jiang et al(2012)]
authorpersonHaiqing Jiang, personZeyu Liu,
personYaogong Wang, personKyunghan Lee, and
personInjong Rhee. year2012.
Understanding Bufferbloat in Cellular Networks. In
booktitleProceedings of the 2012 ACM SIGCOMM Workshop on
Cellular Networks: Operations, Challenges, and Future Design (Helsinki,
Finland) (seriesCellNet '12).
publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages1–6.

[Jin et al(2019)]
authorpersonYuchen Jin, personSundararajan
Renganathan, personGanesh Ananthanarayanan,
personJunchen Jiang, personVenkata N. Padmanabhan,
personManuel Schroder, personMatt Calder, and
personArvind Krishnamurthy. year2019.
Zooming in on Wide-Area Latencies to a Global
Cloud Provider. In booktitleProceedings of the ACM Special
Interest Group on Data Communication (Beijing, China)
(seriesSIGCOMM '19). publisherAssociation for
Computing Machinery, addressNew York, NY, USA,
numpages13 pages.

[Kassem et al(2022)]
authorpersonMohamed M. Kassem,
personAravindh Raman, personDiego Perino, and
personNishanth Sastry. year2022.
A Browser-Side View of Starlink Connectivity. In
booktitleProceedings of the 22nd ACM Internet Measurement
Conference (Nice, France) (seriesIMC '22).
publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages151–158.

[Kassing et al(2020)]
authorpersonSimon Kassing, personDebopam
Bhattacherjee, personAndré Baptista Águas,
personJens Eirik Saethre, and personAnkit Singla.
Exploring the "Internet from Space" with Hypatia.
In booktitleProceedings of the ACM Internet Measurement
Conference (Virtual Event, USA) (seriesIMC '20).
publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages214–229.

authorpersonDr. T.S. Kelso.

Accessed: 2023-10-12.

authorpersonGunther Krebs.
titleStarlink Block v1.0.

authorpersonMeasurement Lab.
titleM-Lab Locate API v2.

authorpersonMeasurement Lab.
titleM-Lab to the Cloud.

authorpersonMeasurement Lab.
titleMeasurement Lab Infrastructure.

[Lai et al(2023)]
authorpersonZeqi Lai, personHewu Li,
personYangtao Deng, personQian Wu,
personJun Liu, personYuanjie Li,
personJihao Li, personLixin Liu,
personWeisen Liu, and personJianping Wu.
StarryNet: Empowering Researchers to Evaluate
Futuristic Integrated Space and Terrestrial Networks. In
booktitle20th USENIX Symposium on Networked Systems Design
and Implementation (NSDI 23). publisherUSENIX Association,
addressBoston, MA, pages1309–1324.

[Li et al(2010)]
authorpersonAng Li, personXiaowei Yang,
personSrikanth Kandula, and personMing Zhang.
CloudCmp: Comparing Public Cloud Providers. In
booktitleProceedings of the 10th ACM SIGCOMM Conference on
Internet Measurement (Melbourne, Australia) (seriesIMC
'10). publisherAssociation for Computing Machinery,
addressNew York, NY, USA, numpages14 pages.

[Luckie et al(2021)]
authorpersonMatthew Luckie, personBradley
Huffaker, personAlexander Marder, personZachary
Bischof, personMarianne Fletcher, and personK
Claffy. year2021.
Learning to Extract Geographic Information from
Internet Router Hostnames. In booktitleProceedings of the
17th International Conference on Emerging Networking EXperiments and
Technologies (Virtual Event, Germany) (seriesCoNEXT
'21). publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages440–453.

[López et al(2 12)]
authorpersonMelisa López,
personSebastian Bro Damsgaard, personIgnacio
Rodríguez, and personPreben Mogensen.
An Empirical Analysis of Multi-Connectivity between
5G Terrestrial and LEO Satellite Networks. In booktitle2022
IEEE Globecom Workshops (GC Wkshps). publisherIEEE,

[Ma et al(2 28)]
authorpersonSami Ma, personYi Ching Chou,
personHaoyuan Zhao, personLong Chen,
personXiaoqiang Ma, and personJiangchuan Liu.
Network Characteristics of LEO Satellite
Constellations: A Starlink-Based Measurement from End Users.
[arXiv]2212.13697 [cs.NI]

[Mangla et al(2022)]
authorpersonTarun Mangla, personUdit Paul,
personArpit Gupta, personNicole P. Marwell, and
personNick Feamster. year2022.
Internet Inequity in Chicago: Adoption,
Affordability, and Availability. In booktitleElsevier

authorpersonJonathan C. McDowell.
titleJonathan's Space Pages: StarlinkStatistics.
Accessed: 2023-10.

[Measurement Lab(2 21)]
authorpersonMeasurement Lab.
year(2009-02-11 – 2015-12-21).
titleThe M-Lab NDT Data Set.

[Michel et al(2022)]
authorpersonFrançois Michel,
personMartino Trevisan, personDanilo Giordano, and
personOlivier Bonaventure. year2022.
A First Look at Starlink Performance. In
booktitleProceedings of the 22nd ACM Internet Measurement
Conference (Nice, France) (seriesIMC '22).
publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages130–136.

[Mohan et al(2020)]
authorpersonNitinder Mohan, personLorenzo
Corneo, personAleksandr Zavodovski, personSuzan
Bayhan, personWalter Wong, and personJussi
Kangasharju. year2020.
Pruning Edge Research with Latency Shears. In
booktitleProceedings of the 19th ACM Workshop on Hot Topics
in Networks (Virtual Event, USA) (seriesHotNets '20).
publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages182–189.

[Narayanan et al(2020)]
authorpersonArvind Narayanan, personEman
Ramadan, personJason Carpenter, personQingxu Liu,
personYu Liu, personFeng Qian, and
personZhi-Li Zhang. year2020.
A First Look at Commercial 5G Performance on
Smartphones. In booktitleProceedings of The Web Conference
2020 (Taipei, Taiwan) (seriesWWW '20).
publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages894–905.

[Narayanan et al(2021)]
authorpersonArvind Narayanan, personXumiao
Zhang, personRuiyang Zhu, personAhmad Hassan,
personShuowei Jin, personXiao Zhu,
personXiaoxuan Zhang, personDenis Rybkin,
personZhengxuan Yang, personZhuoqing Morley Mao,
personFeng Qian, and personZhi-Li Zhang.
A Variegated Look at 5G in the Wild: Performance,
Power, and QoE Implications. In booktitleProceedings of the
2021 ACM SIGCOMM 2021 Conference (Virtual Event, USA)
(seriesSIGCOMM '21). publisherAssociation for
Computing Machinery, addressNew York, NY, USA,

[Noordally et al(2016)]
authorpersonRehan Noordally, personXavier
Nicolay, personPascal Anelli, personRichard Lorion,
and personPierre Ugo Tournoux.
Analysis of Internet Latency: The Reunion Island
Case. In booktitleProceedings of the 12th Asian Internet
Engineering Conference (Bangkok, Thailand) (seriesAINTEC
'16). publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages49–56.

authorpersonOneWeb. year2023.
titleOneWeb: Connect with Ease.
Accessed: 2023-05-26.

authorpersonNathan Owens.
titleUnofficial Crowdsourced Starlink Global Gateways &
Accessed: 2023-05-03.

[Pan et al(2023)]
authorpersonJianping Pan, personJinwei
Zhao, and personLin Cai. year2023.
titleMeasuring a Low-Earth-Orbit Satellite Network.
[arxiv]2307.06863 [cs.NI]

[Paul et al(2022a)]
authorpersonUdit Paul, personJiamo Liu,
personDavid Farias-llerenas, personVivek Adarsh,
personArpit Gupta, and personElizabeth Belding.
Characterizing Internet Access and Quality
Inequities in California M-Lab Measurements. In
booktitleACM SIGCAS/SIGCHI Conference on Computing and
Sustainable Societies (COMPASS) (Seattle, WA, USA)
(seriesCOMPASS '22). publisherAssociation for
Computing Machinery, addressNew York, NY, USA,

[Paul et al(2022b)]
authorpersonUdit Paul, personJiamo Liu,
personMengyang Gu, personArpit Gupta, and
personElizabeth Belding. year2022b.
The Importance of Contextualization of Crowdsourced
Active Speed Test Measurements. In booktitleProceedings of
the 22nd ACM Internet Measurement Conference (Nice, France)
(seriesIMC '22). publisherAssociation for
Computing Machinery, addressNew York, NY, USA,

[Pete Heist(2023)]
authorpersonPete Heist.
titleirtt (Isochronous Round-Trip Tester).

[Pratt et al(1999)]
authorpersonStephen R. Pratt,
personRichard A. Raines, personCarl E. Fossa, and
personMichael A. Temple. year1999.
An operational and performance overview of the
IRIDIUM low earth orbit satellite system.
journalIEEE Communications Surveys
volume2, number2 (year1999),

Conference Name: IEEE Communications Surveys.

authorpersonMike Puchol.
booktitleStarlink Coverage Tracker.

Accessed: 2023-10-12.

authorpersonRIPE. year2023.
titleRIPE Atlas.

authorpersonVia Satellite.
titleStarlink Surpasses 2 Million Subscribers.

authorpersonAnkit Singla.
SatNetLab: A Call to Arms for the next Global
Internet Testbed.
journalSIGCOMM Comput. Commun. Rev.
volume51, number2 (datemay
year2021), pages28–30.

authorpersonSpaceX. year2016.
titleSPACEX NON-GEOSTATIONARY SATELLITE SYSTEM.

authorpersonSpaceX. year2019.
titleSPACEX NON-GEOSTATIONARY SATELLITE SYSTEM.

authorpersonSpaceX. year2021.
titlePETITION OF STARLINK SERVICES, LLC FOR DESIGNATION AS
AN ELIGIBLE TELECOMMUNICATIONS CARRIER.

Accessed: 2023-05-26.

booktitleStarlink Direct to Cell.

Accessed: 2023-10-12.

titleStarlink Fair Use Policy.

titleStarlink maritime.

[Starlink Services(2021)]
authorpersonLLC Starlink Services.
titlePetition of Starlink Services, LLC for Designation as
an Eligible Telecommunications Carrier.
Submitted to the Department of Telecommunications and Cable,
Commonwealth of Massachusetts.

[Taneja et al(2023)]
authorpersonAryan Taneja, personDebopam
Bhattacherjee, personSaikat Guha, and
personVenkata N. Padmanabhan. year2023.
titleOn viewing SpaceX Starlink through the Social Media
[arxiv]2307.13441 [cs.NI]

[Tanveer et al(2023)]
authorpersonHammas Bin Tanveer, personMike
Puchol, personRachee Singh, personAntonio Bianchi,
and personRishab Nithyanand. year2023.
Making Sense of Constellations: Methodologies for
Understanding Starlink's Scheduling Algorithms.
journalarXiv preprint arXiv:2307.00402

authorpersonElon Musk Twitter.
titleStarlink Inter-Satellite Links.

authorpersonStarlink Twitter.
titleStarlink Twitter inter-satellite link announcement.

[Yeganeh et al(2020)]
authorpersonBahador Yeganeh,
personRamakrishnan Durairajan, personReza Rejaie,
and personWalter Willinger. year2020.
A First Comparative Characterization of
Multi-cloud Connectivity in Today’s Internet. In
booktitleInternational Conference on Passive and Active
Network Measurement. Springer, pages193–210.

[Zhao et al(2023)]
authorpersonHaoyuan Zhao, personHao Fang,
personFeng Wang, and personJiangchuan Liu.
Realtime Multimedia Services over Starlink: A
Reality Check. In booktitleProceedings of the 33rd Workshop
on Network and Operating System Support for Digital Audio and Video
(Vancouver, BC, Canada) (seriesNOSSDAV '23).
publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages43–49.

authorpersonZoom. year2023.
titleZoom Video Communications.

§ STARLINK ORBITAL INFORMATION

2*# Planes
3c# Satellites

Position     Launched     Filed <cit.>

53°       72     550     1401                                                               1665                  1584

53.2°     72     540     1542                                                               1637                  1584

70°       36     570     301                                                                 408                   720

97.6°     10     560     230                                                                230                   508

Starlink orbital shell design and number of operational satellites as of early October 2023 <cit.>.

Starlink and other emerging satellite constellations, such as OneWeb and Kuiper <cit.>, are called megaconstellations
due to combining multiple orbital shells.
Early satellite constellations for telephone services only consisted of a single shell <cit.>.
<Ref> shows the parameters of Starlink's orbital shells.
The eccentricity parameters of Starlink orbits are negligible due to the satellites' positions in Low Earth Orbit (LEO).
While discussing Starlink's constellation design, we simplify the orbit into circular orbits.

§ DATA CENTER ENDPOINTS

2*                    6cData centers per continent
EU              NA              SA             AS              AF             OC
Amazon EC2 (AMZN)           6               6               1              6               1              1

Google Cloud (GCP)          6               10              1              8               -              1

Microsoft Azure (MSFT)      14              9               1              10              2              3

Digital Ocean (DO)          4               6               -              1               -              -

Alibaba (BABA)              2               1               -              2               -              1

Amazon Lightsail (LTSL)     3               2               -              2               -              1

Oracle (ORCL)               4               4               1              7               -              2
Total                       39     38     4     36     3     9

Global density of data center endpoints of different cloud providers.

<Ref> shows the distribution of the data centers in our dataset by cloud provider and deployed continent.
Each endpoint is a VM established in a compute-capable cloud data center location.
Our selection is influenced by previous studies that have found that significant end-to-end performance differences may appear while measuring different cloud networks due to private WANs, peering agreements, etc. <cit.>.
We believe that our resulting endpoint selection reduces such biases in Internet traffic steering/routing that may affect our aggregated analysis.

§ TOP-3 MNOS PER COUNTRY

<Ref> lists the top-3 Mobile Network Operators (MNOs) of the countries that are part of our M-Lab dataset.
The selection is based on a combination of each AS's rank and the number of M-Lab measurements originating from that AS.

§ GLOBAL VIEW OF BENT-PIPE OPERATION

Global latencies from GSs to PoPs as measured from RIPE Atlas probes.

<Ref> provides additional information about the Starlink last-mile performance analysis presented in <Ref>.
The figure shows the latency between Starlink ground stations (GSs) and Points of Presences (PoPs) on a world map grouped by country.
It is apparent that the latencies are similar all over the world ($\leq$ 6 ms) except in North America ($\geq$ 6 ms).
The anomaly correlates with the dense deployment of GSs and PoPs.

§ TARGETED MEASUREMENT CHALLENGES

Some of the measurements required for Section <ref> required collection of data during time when Dishy received only patchy connectivity. We discovered that both   and   did not handle the interrupted nature of the connection well:  in particular relies on a separate TCP connection to act as the control plane. Both behaved unpredictably and unreliably on a link that has connectivity only for brief windows. Accordingly, for these experiments, we replaced   with  which we set to send only a single ICMP packet with a 200 ms timeout. We then ran it in a loop.

We were unable to find a suitable replacement for  , and therefore relied upon a manual approach. When we detected the start of a connectivity window   was started. Once the connectivity window had passed, we stopped the experiment and restarted the   server. Restarting the   server was necessary to ensure that subsequent   tests could connect ( permits only a single active connection to a server, and a loss of connectivity mid-way through an experiment can leave the server in a state where it believes an experiment is ongoing when it has in fact concluded). Automatically restarting the   server at the end of each connectivity window was not possible because the Starlink-connected computer, now without an Internet connection, could not signal to the remote   server.

An additional challenge caused by the interrupted nature of the connection was not discovered until towards the end of the targetted measurement period. The unstable connection prevented the clock on the computer connected to the Dishy from synchronising over NTP, resulting in it drifting by several seconds duration of the experiment setup. Accordingly, when the absolute timestamps of the recorded data have been analysed, they have first been adjusted to account for the time slip. The gRPC data was collected by a separate computer that did not suffer from clock drift.

minimum RTT

Download Goodput

Upload Goodput

Evolution of Starlink aggregate goodput ((a), (b)) and minimum RTT (c) during download measurements from cities in South America, North America, Europe, and Australia in the last 12 months.

Country                 Mobile Operator                 ASN     AS Rank <cit.>
3*Australia                      Telstra     AS1221     63

Optus     AS4804     4514

Vodafone     AS133612     7242
3*Austria                        A1 Telekom Austria     AS8447     152

T-Mobile Austria     AS8412     511

Magenta Telekom     AS25255     512
3*Belgium                        Proximus     AS5432     975

Telenet     AS6848     1231

Orange     AS47377     3458
3*Brazil                         TIM     AS26615     90

Claro     AS28573     7178

Vivo     AS18881     11754
3*Canada                         Bell     AS577     89

Telus     AS852     200

Rogers     AS812     241
3*Chile                          Movistar     AS7418     3893

Claro     AS27995     7275

Entel     AS27651     11848
3*Colombia                       Tigo Colombia     AS13489     1353

Claro     AS10620     11770

Colombia Móvil     AS27831     11775
3*Czechia                        Vodafone     AS16019     272

O2 Czech Republic     AS5610     828

T-Mobile     AS5588     2308
3*Dominican Republic             Claro     AS6400     2636

Altice Dominicana     AS28118     3118

Tricom     AS27887     3539
3*France                         Orange     AS3215     204

SFR     AS15557     413

Bouygues Telecom     AS5410     1316
3*Germany                        Deutsche Telekom     AS3320     20

Vodafone     AS3209     237

O2     AS6805     1917
3*Greece                         Vodafone Greece     AS3329     719

Forthnet GR     AS1241     915

Cosmote     AS29247     11995
Guadeloupe     Dauphin Telecom                    AS33392     6315
3*Ireland                        Eir     AS5466     1747

Vodafone     AS15502     3471

Three     AS13280     7364
3*Italy                          TIM     AS3269     244

Wind Tre     AS1267     434

Vodafone     AS30722     1049
3*Japan                          KDDI     AS2516     91

SoftBank     AS17676     111

NTT Docomo     AS9605     11764

Country                 Mobile Operator                 ASN     AS Rank <cit.>
3*Kenya                          Safaricom     AS33771     841

Airtel Kenya     AS36926     976

Telkom Kenya     AS12455     4566
Martinique                                      Digicel     AS48252     3508
2*Mexico                         AT&T     AS28469     5558

Telcel     AS28403     15443
2*Mozambique                     Movitel     AS37342     12294

mCel     AS36945     21048
3*Netherlands                    KPN     AS1136     747

T-Mobile     AS50266     1268

Tele2     AS13127     1413
3*New Zealand                    2degrees     AS9790     250

Vodafone     AS9500     1924

Spark     AS4771     3913
3*Nigeria                        MTN     AS29465     885

Airtel     AS36873     1167

Glo     AS328309     23694
3*Norway                         Telenor     AS2119     255

NextGenTel AS     AS15659     4570

TELIA NORGE AS     AS12929     7314
3*Peru                           Movistar     AS6147     1735

Claro     AS12252     1741

Entel     AS21575     2276
2*Philippines                    Globe Telecom     AS132199     11910

Smart Communications     AS10139     12101
2*Poland                         Orange Polska     AS5617     136

T-Mobile Poland     AS12912     175
3*Portugal                       NOS     AS2860     809

MEO     AS15525     1329

Vodafone     AS12353     1592
3*Puerto Rico                    Claro     AS10396     1088

Liberty     AS14638     1242

T-Mobile     AS21928     5435
Saint Barthélemy                                Digicel     AS3215     204
3*Spain                          Orange     AS12479     332

Vodafone     AS12430     358

Movistar     AS3352     382
3*Sweden                         Tele2 Sweden     AS1257     195

Telia Company     AS3301     387

Telenor Sweden     AS8642     65529
3*United Kingdom                 O2     AS5089     505

Vodafone     AS5378     5443

EE     AS12576     11745
3*United States                  T-Mobile     AS21928     5435

AT&T Mobility LLC     AS20057     7191

Verizon     AS22394     11784

The selection of top-3 terrestrial MNOs (mobile network operators) for countries with Starlink M-Lab measurements.


# Uncertainty Estimation in Contrast-Enhanced MR Image Translation with Multi-
Axis Fusion

###### Abstract

In recent years, deep learning has been applied to a wide range of medical
imaging and image processing tasks. In this work, we focus on the estimation
of epistemic uncertainty for 3D medical image-to-image translation. We propose
a novel model uncertainty quantification method, Multi-Axis Fusion (MAF),
which relies on the integration of complementary information derived from
multiple views on volumetric image data. The proposed approach is applied to
the task of synthesizing contrast enhanced T1-weighted images based on native
T1, T2 and T2-FLAIR scans. The quantitative findings indicate a strong
correlation ($\rho_{\text{h}ealthy}=0.89$) between the mean absolute image
synthetization error and the mean uncertainty score for our MAF method. Hence,
we consider MAF as a promising approach to solve the highly relevant task of
detecting synthetization failures at inference time.

##  1 Introduction

Deep learning is evolving at light speed and is being applied to a wide range
of medical imaging and image processing tasks. However, deep learning models
are often considered black boxes, and no measure of uncertainty accompanies
their predictions. This is especially problematic in medical image-to-image
translation, where the consequences of a wrong target domain image
synthetization can be as severe as tumor removal or hallucination [1].
Therefore, the development of methods to estimate the uncertainty of deep
learning models is crucial. Uncertainty in deep learning can be categorized
into two types: epistemic and aleatoric uncertainty. Aleatoric uncertainty
characterizes inherent noise in observations (i.e., data). Epistemic
uncertainty refers to a lack of knowledge of the model (i.e., domain shift,
underfitting, etc.) [2]. In this work, we focus on the estimation of epistemic
uncertainty.

Bayesian neural networks facilitate the quantification of epistemic
uncertainty, but are computationally intensive and complex to train, limiting
their practical use [3, 4]. Due to this, multiple methods have been proposed
to estimate epistemic uncertainty in deep learning models. Two of the most
popular methods are Monte Carlo Dropout (MC-Dropout) [5] and Deep Ensemble
[6]. MC-Dropout is a computationally feasible approximation to Bayesian neural
networks, providing uncertainty estimates for models’ confidence scores. Deep
Ensemble is a non-Bayesian solution that yields well-calibrated predictive
uncertainty estimates with minimal hyperparameter tuning at the cost of
requiring multiple models training. MC-Dropout and Deep Ensemble techniques
have shown promising results in uncertainty estimation for image
classification and regression tasks[2]. However, the application of MC-Dropout
and Deep Ensemble in image-to-image translation for contrast-enhanced MRI is
still relatively unexplored. In [7], a Bayesian deep learning framework that
combines both uncertainty types was proposed and had inspired several works in
medical image-to-image translation [8, 9, 10]. They model the predictive
distributions using the Gaussian or a generalized Gaussian distribution.
Often, those methods require extensive hyperparameter tuning to balance the
estimation of the distribution parameters, e.g., the predictive variance
versus the predictive mean.

The closest to our method is test-time data augmentation for uncertainty
estimation. In [11], Ayhan et al. proposed to generate augmented examples per
test case using simple and randomized transformations and feeding them to a
neural network. In this way, samples of an approximate predictive distribution
are obtained which can then be used to evaluate the predictive uncertainty.

We propose Multi-Axis Fusion [12] to estimate the uncertainty for 3D medical
image-to-image translation. This method uses the unique properties of $2$D or
$2.5$D slice-wise processing of $3$D medical images. Multi-axis fusion was
proposed for segmentation and combines predictions of multiple axes [12].
Here, we also use this technique for $3$D medical image-to-image translation
and have adapted it for uncertainty estimation.

In this work, we investigate the efficacy of MC-Dropout, Deep Ensemble, and
our method to capture uncertainty in medical image-to-image translation using
a U-Net [13] style GAN model [14] (see Figure 1). A good uncertainty
estimation aims to find a proxy for the prediction error, which is not
available at the inference time. We evaluated the different uncertainty
estimation techniques by comparing the mean uncertainty score with the mean
absolute error (MAE) between the synthesized and ground truth images.

Fig. 1: Comparison of three uncertainty methods. (a) MC-Dropout employs a
single model $f_{MC}$ with different parameters
$\theta_{1},\theta_{2},\dots,\theta_{M}$ at inference, (b) Deep Ensemble
utilizes multiple independent models $f_{1},f_{2},\dots,f_{M}$, and (c) Multi-
axis fusion uses information from various slicing sets
$S_{1},S_{2},\dots,S_{M}$ and a single model $f_{1}$. (Adapted from [2])

##  2 Methods

Let $\mathbf{V}\in\mathbb{R}^{W\times H\times D}$ be a $3$D volume of width
$W$, height $H$, and depth $D$ and let $\mathbf{I_{d}}$ be the $d^{th}$ $2$D
slice of size $W\times H$ sliced along $\mathbf{V}$ axial plane (view). We
denote the set of all slices of $\mathbf{V}$ along axial plane as
$S=\\{\mathbf{I}_{1},\mathbf{I}_{2},\dots,\mathbf{I}_{D}\\}$. To perform
image-to-image translation of $\mathbf{V}$, we synthesize a new volume
$\mathbf{V}^{\prime}$ of the same size as $\mathbf{V}$ by applying a trained
model $f$ with parameters $\theta$ to each slice $\mathbf{I_{d}}$ of
$\mathbf{V}$ independently. Denoting each translated slice as
$\mathbf{I^{\prime}_{d}}=f(\mathbf{I_{d}},\theta)$ and translated set as
$S^{\prime}=\\{\mathbf{I}^{\prime}_{1},\mathbf{I}^{\prime}_{2},\dots,\mathbf{I}^{\prime}_{D}\\}$,
the translated volume $\mathbf{V}^{\prime}$ is created by stacking the slices
in $S^{\prime}$ (along axial plane), i.e.,
$\mathbf{V}^{\prime}=\text{stack}(S^{\prime})$

###  2.1 MC-Dropout

MC-Dropout [5] uses a single model $f_{MC}$ that was trained with dropout. At
inference time dropout is applied as well to stochastically sample subgraphs
of the model, each uniquely described by a parameter set
$\theta_{1},\theta_{2},\dots,\theta_{M}$. For a given input $\mathbf{I_{d}}$,
individual predictions
$f_{MC}(\mathbf{I_{d}},\theta_{1}),\dots,f_{MC}(\mathbf{I_{d}},\theta_{M})$
are generated for each parameter set at inference time to give $M$ samples of
predictive distribution for the test case. The sample mean of these $M$
samples, i.e,

$\mathbf{I^{\prime}_{d}}=\frac{1}{M}\sum_{m=1}^{M}f_{MC}(\mathbf{I_{d}},\theta_{m})$
(1)

is used to obtain the final prediction. To estimate the associated uncertainty
map $\mathbf{I_{d}}^{\sigma}$, the sample variance is calculated:

$\mathbf{I}^{\sigma}_{d}=\frac{1}{M}\sum_{m=1}^{M}\left(f_{MC}(\mathbf{I_{d}},\theta_{m})-\mathbf{I_{d}}^{\prime}\right)^{2}\text{
.}$ (2)

For our experiments we set $M=9$. By stacking image slices from (1) and
uncertainty maps from (2) we obtain the final synthetic $3$D volume and its
uncertainty map, respectively.

###  2.2 Deep Ensemble

Ensembling in machine learning involves combining multiple individual models
to create a more robust predictive model. Following [6], we trained $M$ models
$f_{1},f_{2},\dots,f_{M}$ on the entire dataset with different random seeds to
initialize the model weights, while the model architecture and the training
procedure are identical. At inference time, we computed for a given test input
$\mathbf{I_{d}}$ the forward pass of each model $f_{m}$ to obtain $M$
predictive distribution samples for the test case. Similarly to Section 2.1,
we obtained $M=9$ samples using $9$ trained models. The sample mean of the
individual predictions $f_{1}(\mathbf{I_{d}}),...,f_{M}(\mathbf{I_{d}})$ is
computed to obtain the final synthetic image $\mathbf{I}^{\prime}_{d}$, i.e.,

$\mathbf{I^{\prime}_{d}}=\frac{1}{M}\sum_{m=1}^{M}f_{m}(\mathbf{I_{d}})\text{
.}$ (3)

Similarly to Section 2.1, the associated uncertainty map
$\mathbf{I}_{d}^{\sigma}$ was obtained by computing the sample variance, and
similarly stacking is used to obtain the final synthetic $3$D volume and its
$3$D uncertainty map.

###  2.3 Multi-Axis Fusion

We adopted the multi-axis prediction fusing from [12] to estimate the
uncertainty of the synthesized image. The method is based on the observation
that the segmentation of a 3D volume can be improved by fusing the predictions
of multiple 2D slicing sets of the 3D volume along multiple planes (views).
The slicing planes include the principal planes (axial, sagittal, and coronal)
along with their corresponding oblique planes with a specific acute angle (cf.
Figure 1 (c)). We denote each set of slices along a particular plane $m$ as
$S^{m}=\\{\mathbf{I}_{1}^{m},\mathbf{I}_{2}^{m},\dots,\mathbf{I}_{D_{m}}^{m}\\}$.
At inference time, we evaluated each test slice in $S^{m}$ using a single
model $f$, i.e., $\mathbf{I^{\prime m}_{d}}=f(\mathbf{I^{m}_{d}})$ to produce
$M$ predicted slice sets, i.e., $S^{\prime 1},\dots,S^{\prime M}$. By stacking
each predicted slice set along its slicing plane, $M$ samples of predictive
distribution of the translated $3$D volume are obtained, i.e.,
$\mathbf{V}^{\prime m}=\text{stack}(S^{\prime m})$. The final voxel-wise
prediction is computed by averaging the predictions of the individual slicing
sets, i.e.,

$\mathbf{V}^{\prime}_{MAF}=\frac{1}{M}\sum_{m=1}^{M}\mathbf{V}^{\prime
m}\text{ .}$ (4)

The associated $3$D uncertainty map is also computed by taking the voxel-wise
variance of computed $\mathbf{V}^{\prime 1},\dots,\mathbf{V}^{\prime M}$.

In our experiments, we used $M=9$ slicing sets. Slicing sets are obtained by
slicing the 3D volume along the three principal axial, sagittal, and coronal
planes. We obtained $6$ additional slicing sets with unique slicing planes and
unique local context after rotating the original volume by $45$ degrees along
each of the three principal axes.

##  3 Experiments and Results

We conduct a quantitative and qualitative analysis to evaluate the uncertainty
estimation of the three methods. The quantitative analysis is done by
correlation measurement between mean uncertainty score (MU) and mean absolute
error (MAE). We qualitatively compare the uncertainty map to the mean absolute
map to assess if both maps spatially correlate.

In our experiments, we test each method for the translation task of having
T1N, T2W, and T2F sequences as input and T1C as target. We used the BraTS 2023
dataset [15]. The dataset contains 1,251 exams with all four MRI scans, each
with dimensions of $240\times 240\times 155$ voxels. We split the dataset into
1,125 for training and 126 for validation/testing. The scans have been
resampled to a 1mm isotropic resolution, co-registered to a template brain,
and skull-stripped. We utilize the provided segmentation masks to calculate
each metric for two regions of interest (ROI), i.e., healthy and tumor. The
healthy ROI is defined as the brain region excluding the tumor. The model
architecture and training procedure are described in the following sections.

###  3.1 Implementation

Full implementation details can be found in our public code repository
anonymous.

Model: Due to promising results of U-net [13] style networks, we adapted an
architecture from Baltruschat et al. [14]. Our U-net style generator consists
of $8$ downsample steps where each downsampling was achieved by convolution
with stride $2$ [16]. Each step contains two consecutive blocks of
convolution, instance normalization, and Mish[17] activation function. For
upsampling, we used transposed convolution. The output layer has a CeLU[18]
activation function. For the $f_{MC}$ model, we added a dropout layer after
each convolution layer with a dropout rate of $0.1$. The generator is
optimized with a combined loss function based on adversarial loss [19],
perceptual loss [20], and frequency loss [14]. The input to our model is a
2.5D image stack [21] for each sequence, and all three sequences are combined
in the channel dimension. For the 2.5D, we take two neighboring slices. Hence,
in total, the input has nine channels. The output has one channel since we
only predict the slice of interest for T1C. We used a patch-wise discriminator
from [22] with $5$ convolution layers (conv-layers) and added spectral
normalization [23] to each conv-layer to stabilize the training. The
discriminator is optimized with a least square loss [19].

Dataset Preprocessing: We used three steps for data preprocessing. First,
histogram standardization was applied to all scans as proposed by Nyul et al.
[24]. The training set was used to determine intensity landmarks for each
sequence, and this standardization was applied to both training and validation
datasets. Secondly, MinMax intensity normalization was performed on each input
and target volume set. While MinMax values were computed using only the input
sequences (i.e., T1N, T2W, T2F), the normalization was applied to both input
and target volumes. This was done to model the relationship of contrast
uptake, resulting in an input data range of $[0,1]$ and a possible target data
range of $[0,\inf]$. Thirdly, we shift the range by linear scaling to $[-1,1]$
and $[-1,\inf]$ for input and target, respectively.

Training: To augment the training data, we sliced the scans in three basic
planes which, after removing all zero slices, resulted in $502,971$ and
$56,270$ slices for training and validation, respectively. Furthermore, we
also applied online random cropping to $256\times 256$ (after initially zero-
padding all slices to $288\times 288$), random horizontal flipping
(probability of $p=0.5$) and random rotations in the range of $[-15,15]$
degrees ($p=0.5$). Our models are trained using AMSGrad optimizer with an
initial learning rate of $0.0001$ and a batch size of $64$. We reduced the
learning rate by a factor of two every $10$ epochs. The models are trained for
$100$ epochs, where $400,000$ training images are used in each epoch.

Table 1: Pearson correlation coefficient $\rho$ and Kendall rank correlation $\tau$ between MAE and MU score in brain tumor region and healthy brain region of the image. | MC-Dropout | Deep Ens. | MAF
---|---|---|---
Pearson $\rho_{\text{healthy}}$ | -0.11 | 0.38 | 0.89
Pearson $\rho_{\text{tumor}}$ | 0.19 | 0.45 | 0.61
Kendall $\tau_{\text{healthy}}$ | -0.16 | 0.31 | 0.63
Kendall $\tau_{\text{tumor}}$ | 0.21 | 0.43 | 0.43

###  3.2 Results and Discussion

Quantitative: Table 1 shows the Pearson correlation $\rho$ and the Kendall
$\tau$ rank correlation between the MAE and the MU. The values are calculated
separately for the healthy ROI and the tumor ROI. For MC-Dropout, the
Pearson’s correlation coefficients of both ROIs ($\rho_{\text{tumor}}=0.19$
and $\rho_{\text{healthy}}=-0.11$) indicate that there is no linear
correlation between MAE and MU. Furthermore, the associated rank correlations
suggest that there is no statistically significant ordinal association between
MAE and MU. This is visibly in Figure 2 top row. For Deep Ensemble, we
observed a moderate Pearson correlation of MU and MAE with a higher
correlation coefficient for the tumor region than for the healthy region
(i.e., $\rho_{\text{tumor}}=0.45$ vs. $\rho_{\text{healthy}}=0.38$). This
behavior is different for MAF, where the correlation coefficient
$\rho_{\text{healthy}}=0.89$ is substantially greater than
$\rho_{\text{tumor}}=0.61$. Nevertheless, both show a high linear correlation
between the mean uncertainty score and the mean absolute error, which exceed
the scores of all other models.

Fig. 2: Correlation between MAE and MU score in brain tumor region (left
column) and healthy brain region of the image (right column) along with the
OLS linear fit (black).

Qualitative: Figure 3 shows all three methods’ absolute error maps and
uncertainty maps. We selected this example because all three methods
synthesized a false positive enhanced tumor in the right temporal lobe. The
error is less prominent for Deep Ensembles but still visible in the error map.
The corresponding uncertainty map for Deep Ensemble does not capture this
region; hence, it is not helpful as a proxy. MC-Dropout’s uncertainty map
captures this area sightly, but also some other structures not shown in the
error map. The uncertainty map from our proposed method shows a strong
uncertainty in the area of the false positive tumor.

Fig. 3: An exemplary absolute error and uncertainty maps. The first column
shows the three input sequences: T1-Native (T1N), T2-W, T2-Flair. The second
column shows: ground truth T1-CE (T1C), absolute contrast changes
($\left\lvert\text{T1C}-\text{T1N}\right\rvert$). The synthetic T1-CE image
(T1sC), the error map ($\left\lvert\text{T1sC}-\text{T1C}\right\rvert$), and
the uncertainty map (see Section 2) are shown for each method in the third to
the fifth column.

##  4 Conclusion

We proposed a novel uncertainty estimation method, Multi-Axis Fusion, which
relies on the integration of complementary information derived from multiple
views on volumetric image data. The proposed approach is applied to the task
of synthesizing contrast-enhanced T1-weighted images based on native T1,
T2-weighted, and T2-FLAIR sequences. We compare MAF to the well-established
baseline methods MC-Dropout and Deep Ensembles. Our quantitative findings
indicate a strong correlation ($\rho_{\text{healthy}}=0.89$,
$\rho_{\text{tumor}}=0.61$) between the mean absolute image synthetization
error and the mean uncertainty score for our MAF method, outperforming both
MC-Dropout and Deep Ensemble models. Observing a high correlation particularly
in healthy regions, suggests that the MAF confidence estimates provide a
meaningful proxy for the synthetization error at the test time.

## References

  * [1] J. P. Cohen et al.  Distribution matching losses can hallucinate features in medical image translation.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part I, pp. 529–536. Springer, 2018.
  * [2] K. Zou et al.  A review of uncertainty estimation and its application in medical imaging.  arXiv, 2023.
  * [3] M. Magris and A. Iosifidis.  Bayesian learning for neural networks: an algorithmic survey.  Artificial Intelligence Review, pp. 1–51, 2023.
  * [4] R. M. Neal.  Bayesian learning for neural networks, volume 118.  Springer Science & Business Media, 2012.
  * [5] Y. Gal and Z. Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In international conference on machine learning, pp. 1050–1059. PMLR, 2016.
  * [6] B. Lakshminarayanan et al.  Simple and scalable predictive uncertainty estimation using deep ensembles.  Advances in neural information processing systems, 30, 2017.
  * [7] A. Kendall and Y. Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  Advances in neural information processing systems, 30, 2017.
  * [8] Z. Zhang et al.  Reducing uncertainty in undersampled mri reconstruction with active acquisition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2049–2058, 2019.
  * [9] J. Kleesiek et al.  Can virtual contrast enhancement in brain mri replace gadolinium?: a feasibility study.  Investigative radiology, 54(10):653–660, 2019.
  * [10] U. Upadhyay et al.  Uncertainty-aware gan with adaptive loss for robust mri image enhancement.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3255–3264, 2021.
  * [11] M. S. Ayhan and P. Berens.  Test-time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks.  In Medical Imaging with Deep Learning, 2022.
  * [12] I. M. Baltruschat et al.  Scaling the u-net: segmentation of biodegradable bone implants in high-resolution synchrotron radiation microtomograms.  Scientific reports, 11(1):24237, 2021.
  * [13] O. Ronneberger et al.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234–241. Springer, 2015.
  * [14] I. M. Baltruschat et al.  freggan with k-space loss regularization for medical image translation.  arXiv preprint arXiv:2303.15938, 2023.
  * [15] B. H. Menze et al.  The multimodal brain tumor image segmentation benchmark (brats).  IEEE transactions on medical imaging, 34(10):1993–2024, 2014.
  * [16] J. T. Springenberg et al.  Striving for simplicity: The all convolutional net.  arXiv preprint arXiv:1412.6806, 2014.
  * [17] D. Misra.  Mish: A self regularized non-monotonic activation function.  arXiv preprint arXiv:1908.08681, 2019.
  * [18] J. T. Barron.  Continuously differentiable exponential linear units.  arXiv preprint arXiv:1704.07483, 2017.
  * [19] X. Mao et al.  Least squares generative adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pp. 2794–2802, 2017.
  * [20] J. Johnson et al.  Perceptual losses for real-time style transfer and super-resolution.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 694–711. Springer, 2016.
  * [21] S. Pasumarthi et al.  A generic deep learning model for reduced gadolinium dose in contrast-enhanced brain mri.  Magnetic Resonance in Medicine, 86(3):1687–1700, 2021.
  * [22] P. Isola et al.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125–1134, 2017.
  * [23] T. Miyato et al.  Spectral normalization for generative adversarial networks.  arXiv preprint arXiv:1802.05957, 2018.
  * [24] L. G. Nyúl et al.  New variants of a method of mri scale standardization.  IEEE transactions on medical imaging, 19(2):143–150, 2000.