Further author information: (E-mail<EMAIL_ADDRESS>Telephone: +34
922433642

# Laboratory test of the VIS detector system of SOXS for the ESO-NTT telescope

Rosario Cosentino  INAF - Fundación Galileo Galilei, Breña Baja, Spain  INAF -
Osservatorio Astrofisico di Catania, Catania, Italy  Marcos Hernandez  INAF -
Fundación Galileo Galilei, Breña Baja, Spain  Hector Ventura  INAF - Fundación
Galileo Galilei, Breña Baja, Spain  Sergio Campana  INAF - Osservatorio
Astronomico di Brera, Merate, Italy  Riccardo Claudi  INAF - Osservatorio
Astronomico di Padova, Padua, Italy  Pietro Schipani  INAF - Osservatorio
Astronomico di Capodimonte, Naples, Italy  Matteo Aliverti  INAF -
Osservatorio Astronomico di Brera, Merate, Italy  Andrea Baruffolo  INAF -
Osservatorio Astronomico di Padova, Padua, Italy  Sagi Ben-Ami  Weizmann
Institute of Science, Rehovot, Israel  Harvard-Smithsonian Center for
Astrophysics, Cambridge, USA  Federico Biondi  Max-Planck-Institut für
Extraterrestrische Physik, Garching, Germany  Giulio Capasso  INAF -
Osservatorio Astronomico di Capodimonte, Naples, Italy  Francesco D’Alessio
INAF - Osservatorio Astronomico di Roma, Rome, Italy  Paolo D’Avanzo  INAF -
Osservatorio Astronomico di Brera, Merate, Italy  Ofir Hershko  Weizmann
Institute of Science, Rehovot, Israel  Hanindyo Kuncarayakti  Tuorla
Observatory, Department of Physics and Astronomy, University of Turku, Turku,
Finland  FINCA - Finnish Centre for Astronomy with ESO, Turku, Finland  Marco
Landoni  INAF - Osservatorio Astronomico di Brera, Merate, Italy  Matteo
Munari  INAF - Osservatorio Astrofisico di Catania, Catania, Italy  Giuliano
Pignata  Millennium Institute of Astrophysics (MAS), Santiago, Chile
Universidad Andres Bello, Santiago, Chile  Adam Rubin  European Southern
Observatory, Garching, Germany  Salvatore Scuderi  INAF - Osservatorio
Astrofisico di Catania, Catania, Italy  INAF - Istituto di Astrofisica
Spaziale e Fisica Cosmica, Milano, Italy  Fabrizio Vitali  INAF - Osservatorio
Astronomico di Roma, Rome, Italy  David Young  Queen’s University Belfast,
Belfast, UK  Jani Achrén  Incident Angle Oy, Turku, Finland  José Antonio
Araiza-Duran  Millennium Institute of Astrophysics (MAS), Santiago, Chile
Centro de Investigaciones en Optica A. C., León, Mexico  Iair Arcavi  Tel Aviv
University, Tel Aviv, Israel  Anna Brucalassi  Universidad Andres Bello,
Santiago, Chile  INAF-Osservatorio Astrofisico Arcetri, Firenze, Italy  Rachel
Bruch  Weizmann Institute of Science, Rehovot, Israel  Enrico Cappellaro  INAF
- Osservatorio Astronomico di Padova, Padua, Italy  Mirko Colapietro  INAF -
Osservatorio Astronomico di Capodimonte, Naples, Italy  Massimo Della Valle
INAF - Osservatorio Astronomico di Capodimonte, Naples, Italy  Marco De
Pascale  INAF - Osservatorio Astronomico di Padova, Padua, Italy  Rosario Di
Benedetto  INAF - Osservatorio Astrofisico di Catania, Catania, Italy  Sergio
D’Orsi  INAF - Osservatorio Astronomico di Capodimonte, Naples, Italy  Avishay
Gal-Yam  Harvard-Smithsonian Center for Astrophysics, Cambridge, USA  Matteo
Genoni  INAF - Osservatorio Astronomico di Brera, Merate, Italy  Jari
Kotilainen  Tuorla Observatory, Department of Physics and Astronomy,
University of Turku, Turku, Finland  FINCA - Finnish Centre for Astronomy with
ESO, Turku, Finland  Gianluca Li Causi  INAF - Istituto di Astrofisica e
Planetologia Spaziali, Rome , Italy  Laurent Marty  INAF - Osservatorio
Astronomico di Capodimonte, Naples, Italy  Seppo Mattila  Tuorla Observatory,
Department of Physics and Astronomy, University of Turku, Turku, Finland
Michael Rappaport  Harvard-Smithsonian Center for Astrophysics, Cambridge, USA
Kalyan Radhakrishnan  INAF - Osservatorio Astronomico di Padova, Padua, Italy
Davide Ricci  INAF - Osservatorio Astronomico di Padova, Padua, Italy  Marco
Riva  INAF - Osservatorio Astronomico di Brera, Merate, Italy  Bernardo
Salasnich  INAF - Osservatorio Astronomico di Padova, Padua, Italy  Alessandra
Slemer  INAF - Osservatorio Astronomico di Padova, Padua, Italy  Stephen
Smartt  Queen’s University Belfast, Belfast, UK  Ricardo Zanmar Sanchez  INAF
- Osservatorio Astrofisico di Catania, Catania, Italy  Maximilian Stritzinger
Aarhus University, Aarhus, Denmark

###### Abstract

SOXS is the new spectrograph for the ESO NTT telescope able to cover the
optical and NIR bands thanks to two different arms: the UV-VIS (350-850 nm),
and the NIR (800-2000 nm). In this article, we describe the final design of
the visible camera cryostats, the test facilities for the CCD
characterization, and the first results with the scientific detector. The UV-
VIS detector system is based on a e2v CCD 44-82, a custom detector head
coupled with the ESO Continuous Flowing Cryostat (CFC) cooling system and the
New General Detector Controller (NGC) developed by ESO. The laboratory
facility is based on an optical bench equipped with a Xenon lamp, filter
wheels to select the wavelength, an integrating sphere, and a calibrated diode
to measure the flux. This paper outlines the visible camera cryostat, the test
facilities for the CCD characterization and the first results with the
scientific detector in the laboratory and after the integration to the
instrument.

###### keywords:

spectrograph, UV-VIS, detector Control System, CCD

(Ref. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17).

##  1 The UV-VIS detector system

The detector head is a custom design, coupled with an ESO cooling system,
based on Continuous Flow Cryostats (CFC) and a Programmable Logic Controller
(PLC) for the temperature measurement and control (Figure 1).

---
Figure 1:  The design of the chamber and the chamber in the test bench.

The e2v detector is a back illuminated CCD with a 15 $\mu$m square pixel and
an image area of 30.7 x 61.4 mm. The high quantum efficiency (QE) in the
spectral response of the spectrograph (350-850 nm) and their geometric
characteristics (pixel size and dimensions) make this CCD the most suitable
for our instrument.

The detector selected for the UV-VIS arm is an e2v CCD44-82. This detector is
a high performance, back illuminated CCD with a 15.0 µm square pixel and an
image area of 30.7x61.4 mm and is characterized for the high Quantum
Efficiency (QE) for the instrument requirements. The detector wiring is based
on a custom flex PCB for the CCD biasing and clocking and a coaxial cable for
the video signals (Figure 2).

---
Figure 2:  The CCD assembled in the VIS chamber (left) and The CCD cold finger
with the flat PCB (right).

The CCD control electronics is a standard ESO CCD controller (NGC), operating
with the ESO control software, coupled with custom electronics and optimized
for SOXS spectroscopic operations. Several readout modes and speeds were
defined and tested to optimize the SOXS operations for different scientific
purposes.

##  2 The TNG facilities

The laboratory facility is based on an optical bench equipped with a Xenon
lamp, filter wheels to select the wavelength, an integrating sphere and a
calibrated diode to measure the flux. An ad-hoc mechanical adapter allows for
the coupling of the UV-VIV SOXS camera with the TNG facilities in order to
perform all the required tests (Figure 5).

The first test with the scientific CCD was done with the VIS chamber coupled
with the TNG characterization facilities, in order to tune up the CCD
parameters and to calculate the gain and the readout noise. In these test, we
measured the gain and readout noise at different readout modes and readout
speeds. The optical bench for the detector tests is made up by some
electronics apparatus, located in the optical laboratory, and the control
computers, located in the annexed control room. The source is a Xenon lamp and
the light goes through the shutter, the filter wheels and the integrating
sphere and come to the SOXS UV-VIS detector (Figure 3). The filter wheels make
available the wavelength from 350 to 1100 nm and the possibility to reduce the
flux with neutral filter from ND 0.1 to 4.

---
Figure 3:  The scheme of the laboratory facilities.
---
Figure 4:  Setup and test environment (architecture).
---
Figure 5:  The coupling of the chamber (left) and The test system (right).

##  3 The test results

The vacuum and cryogenic tests on the chamber show a good operation: the
degassing rate is negligible and the LN2 consumption is as expected. In these
tests, we tested the scientific CCD in the same cryogenic environment foreseen
for SOXS.

###  3.1 Test environment

Configuration:

  * -

NGC controller with video-bias and Clock cables

  * -

Preamplifier

  * -

SOXS camera and scientific CCD

  * -

CFC cooling system with the TeePee controller

  * -

Lakeshore temperature controller

  * -

Optical laboratory facilities

Tests:

  * -

Bias acquisition and video offset settings

  * -

Flat field acquisition at different signal level

  * -

Flat field at different wavelength (350-1100 nm)

  * -

Gain measurements

  * -

Readout noise measurement

  * -

Linearity measurements

  * -

Test of different readout mode

  * -

Test of different binning (1X2, 2X2, 1X4, 2X4)

###  3.2 Gain Measurements

For the gain measurements (conversion factor, CF) we used the method based on
the assumption that the CCD have a fixed electron noise and the photons
measured by the CCD follow the Poisson statistic. The CCD is uniformly
illuminated at different levels of light and we take several bias and pairs of
flat fields for each exposure time. The exposure time is increased and the
signal level is measured in ADUs until saturation is reached.

###  3.3 Readout noise

The readout noise for the different speeds and gains was calculated on the
bias images. These measurements will be repeated when the acquisition system
will be assembled with the spectrograph in the dummy derotator.

###  3.4 Linearity

The linearity of the CCD vs. the incoming signal was calculated for the
different speeds and gains and the results are shown in Figure 6 and in Figure
7. The two (red and green) lines of the figures, shown the linearity of the
two outputs of the CCD.

---
Figure 6:  CCD linearity: speed=3 $\mu$sec/pixel - Gain 2e/ADU
---
Figure 7:  CCD linearity: speed=6$\mu$sec/pixel (left) and 10$\mu$
sec/pixel(right) sec/pixel - Gain 2e/ADU.

###  3.5 Binning

Several binnings was tested (1X2, 2X2, 1X4, 2X4) and the results was as
expected (no changes in readout noise and gain).

###  3.6 Windowing of the CCD

The optical design foresee that the illuminated area is the central part of
the active area of the CCD (740X4096 pixels). This means that the readout
speed can be reduced by using the windowing of the CCD with a notable
increasing of the performances of the acquisition system. In Figure 8 are
shown the simulated image of the spectra in the CCD on the left and the flat
field obtained with the CCD chamber with the test bench used to test the
detector system.

---
Figure 8:  On the left is shown the theoretical image on the CCD, on the right
the flat field obtained in laboratory.

The measurement of the CCD performances shows a good linearity, low readout
noise and allow for different readout modes with various gain and readout
speed, in order to can select the modes that best fit with the observation
modes.

##  4 INTEGRATION WITH UV-VIS

The chamber was coupled with the UV-VIS arm of SOXS spectrograph at the end of
June 2022, the first tests were done and the data will be analyzed in the next
weeks. In the figure below are shown a picture of the UV-VIS spectrograph and
one of the spectra taken during the test campaign.

---
Figure 9:  The UV-VIS arm and an example of one of the UV-VIS spectra.

###  4.1 Conclusions

The tests on the final version of VIS detector system with the scientific CCD
show that the acquisition system works as expected. The test in the
characterization laboratory was done to optimize the performances of the
system in term of readout noise and data throughput, and provide the
measurement of gain and readout noise. The final test on the VIS detector
system coupled with the spectrograph was done in June 2022 and the results
will be analyzed in the next weeks to characterize the entire UV-VIS system.

###### Acknowledgements.

A special acknowledgement to the European Southern Observatory for the support
provided and for the availability to share its knowledge and to allow for the
use of the ESO laboratories in Garching.

## References

  * [1] Schipani, P. et al., “The new SOXS instrument for the ESO NTT,” in [Proc. SPIE 9908, 990841 ], (2016).
  * [2] Cosentino, R. et al., “The vis detector system of SOXS,” in [Proc. SPIE 107022J ], (2018).
  * [3] Aliverti, M. et al., “The mechanical design of SOXS for the NTT,” in [Proc. SPIE 10702 ], (2018).
  * [4] Capasso, G. et al., “SOXS control electronics design,” in [Proc. SPIE 10702 ], (2018).
  * [5] Zanmar Sánchez, R. et al., “Optical design of the SOXS spectrograph for ESO NTT,” in [Proc. SPIE 10702 ], (2018).
  * [6] Schipani, P. et al., “SOXS: a wide band spectrograph to follow up the transients,” in [Proc. SPIE 10702 ], (2018).
  * [7] Ricci, D. et al., “Development status of the SOXS instrument control software,” in [Proc. SPIE 11452-112 ], (2020).
  * [8] Rubin, A. et al., “MITS: the multi-imaging the transient spectrograph for SOXS,” in [Proc. SPIE 10702 ], (2018).
  * [9] Leander, M. et al., “State-of-the-art detector controller for eso instruments,” in [SDW 2013 Florence ], (2013).
  * [10] Aliverti, M. et al., “Manufacturing, integration, and mechanical verification of SOXS,” in [Proc. SPIE 11447-137 ], (2020).
  * [11] Claudi, R. et al., “Operational modes and efficiency of SOXS,” in [Proc. SPIE 11447-99 ], (2020).
  * [12] Cosentino, R. et al., “Development status of the uv-vis detector system of SOXS for the ESO-NTT telescope,” in [Proc. SPIE 114476C ], (2020).
  * [13] Scuderi, S. et al., “The vacuum and cryogenics system of the soxs spectrograph,” in [Proc. SPIE 12188-149 ], (2022).
  * [14] Schipani, P. et al., “Progress on the soxs transients chaser for the ESO-NTT,” in [Proc. SPIE SPIE 12184-24 ], (2022).
  * [15] Rubin, A. et al., “Integration and lab performance of the uv-vis arm of SOXS,” in [Proc. SPIE 12184-299 ], (2022).
  * [16] Claudi, R. et al., “SOXS AIT: a paradigm for system engineering of a medium class telescope instrument,” in [Proc. SPIE 10702 ], (2022).
  * [17] Aliverti, M. et al., “SOXS soxs mechanical integration and verification in italy,” in [Proc. SPIE 12184-304 ], (2022).


pquotem

> (P#1)

# “Against the Void”: An Interview and Survey Study on How Rust Developers Use
Unsafe Code

Ian McCormack1, Tomas Dougan2, Sam Estep1, Hanan Hibshi1, Jonathan Aldrich1,
and Joshua Sunshine1  1Carnegie Mellon University and 2Brown University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
{estep<EMAIL_ADDRESS>{jonathan.aldrich<EMAIL_ADDRESS>
###### Abstract

The Rust programming language is an increasingly popular choice for systems
programming, since it can statically guarantee memory safety without automatic
garbage collection. Rust provides its safety guarantees by restricting
aliasing and mutability, but many key design patterns, such as cyclic aliasing
and multi-language interoperation, must bypass these restrictions. Rust’s
unsafe keyword enables features that developers can use to implement these
patterns, and the Rust ecosystem includes useful tools for validating whether
unsafe code is used correctly. However, it is unclear if these tools are
adequate for all use cases. To understand developers’ needs, we conducted a
mixed-methods study consisting of semi-structured interviews followed by a
survey. We interviewed 19 Rust developers and surveyed 160 developers—all of
whom engaged with unsafe code. We found that 77% of survey respondents and a
majority of interview participants were motivated to use unsafe code because
they were unaware of a safe alternative. Developers typically followed best
practices such as minimizing and localizing their use of unsafe code, but only
23% were always certain that their encapsulations were sound. Limited tooling
support for inline assembly and foreign function calls prevented developers
from validating unsafe code, and differences between Rust and other languages
made foreign functions difficult to encapsulate. Verification tools were
underused, and developers rarely audited their dependencies. Our results
indicate a pressing need for production-ready tools that can validate the most
frequently used unsafe features.

##  I Introduction

The Rust programming language has been a consistently popular choice for
systems programming [1] due to its competitive performance [2] and static
safety guarantees. These guarantees are provided by Rust’s borrow checker,
which restricts aliasing and mutability to prevent types of errors that can
lead to security vulnerabilities, such as access out-of-bounds and double free
[3]. These restrictions are helpful, but they are also conservative; there are
safe design patterns that the borrow checker cannot validate. When necessary,
Rust developers can use the unsafe keyword to enable features that bypass
Rust’s restrictions.

Rust’s unsafe features are central to its design and at the foundation of its
ecosystem [4, 5]. However, if unsafe code is misused, it can undermine Rust’s
core promise of static safety. Using unsafe code correctly is not a trivial
task; Rust’s aliasing restrictions are challenging to reason about [6] and
they constrain the soundness of unsafe operations [7]. Dynamic analysis tools
like Miri [8] and Valgrind [9] can find problems automatically, but they are
underused and may miss bugs related to multi-language interoperation.
Understanding developers’ motivations for using unsafe code and their methods
for reasoning about its soundness is necessary to determine which
interventions will be effective. To that end, we address the following
research questions:

RQ1 \- What do Rust developers state as their motivations for using unsafe
code? We identify three distinct motivations that developers cite to justify
their use of unsafe code, which are not mutually exclusive.

RQ2 \- How do Rust developers reason about encapsulating unsafe code? We
outline how developers reason about safety when encapsulating unsafe code. We
describe the extent to which developers can ensure that unsafe code is minimal
and isolated. Then, we examine why developers choose to expose unsafe
interfaces and whether these interfaces are documented.

RQ3 \- How do Rust developers reason about memory safety across foreign
function boundaries? We describe how developers reason about interoperating
with foreign libraries, and we identify disparities between Rust’s memory
model and foreign memory models that may lead to undefined behavior.

RQ4 \- How do Rust developers validate unsafe code? We explore the role of
testing, auditing, and dynamic analysis tools in Rust developers’ workflows.
We focus on Miri [8], which is the only dynamic bug-finding tool capable of
validating unsafe code against Rust’s current operational semantics. We
determine how frequently Miri is used and identify key limitations that deter
developers from using it.

RQ5 \- How is unsafe code perceived by the members of the Rust community who
engage with it? We investigate how unsafe code is perceived by the Rust
community and we determine whether current documentation and community
guidelines meet developers’ needs.

To answer these questions, we used a mixed-methods approach consisting of
semi-structured interviews with 19 Rust developers followed by a survey that
had 160 valid responses. Our findings demonstrate that several key qualitative
results from Holtervenhoff et al. [10], who interviewed 26 participants,
generalize to our population of 179 developers. That is, developers prioritize
keeping unsafe code minimal, encapsulated, and well-documented, but they
rarely audit their dependencies and rely on ad-hoc reasoning to justify their
decisions. Additionally, we find new evidence that unsafe code is not
regularly tested and that validation tools, such as Miri and cargo-audit[11],
are underutilized.

Our findings indicate that when developers feel that there is no alternative
other than using unsafe code, Rust does not provide adequate resources to
ensure that they are not abandoning safety altogether. This correctness gap
can be addressed by improved support for validating certain unsafe features in
existing tools, such as Miri. Likewise, any future approaches to verification
should provide robust support for unsafe features while remaining lightweight,
intuitive, and tightly integrated with the Rust toolchain. Meanwhile, the Rust
community must proactively spread awareness that tool-supported auditing and
verification are necessary to make Rust’s promise of static safety a reality.

##  II Background

Rust’s safety guarantees build on several core principles. Values in Rust have
either move or copy semantics. Values with copy semantics do not have an owner
and can be copied freely, while values with move semantics have a unique
owner. Ownership is transferred through assignment, and a value will be
deallocated when its owner goes out of scope [12]. Ownership can be borrowed
temporarily by creating a reference, and each reference is associated with a
lifetime: the duration of the program for which it must be valid. A
reference’s lifetime cannot exceed the lifetime or scope of the value it
refers to, and a value can have either one mutable reference or many immutable
references in any scope [13].

These rules are conservative and are not universally compatible with certain
idioms of systems programming. Rust allows developers to circumvent its
restrictions using the unsafe keyword, which enables features that escape the
limitations of the borrow checker. Within an unsafe context, users can
dereference raw pointers, execute inline assembly instructions, modify static
mutable state, implement unsafe traits, and call unsafe functions [12]. There
are multiple dimensions to how these features can be used. In some cases,
unsafe is necessary to implement safe patterns that are incompatible with the
borrow checker’s restrictions, such as shared ownership. Foreign function
calls, inline assembly, and intrinsics allow developers to interact with
contexts external to Rust, which are not constrained by the borrow checker.
These features can also support methods for improving performance, such as
bypassing runtime checks or leveraging hardware acceleration.

The Rust community recommends that API developers avoid exposing unsafe
interfaces to consumers. Instead, developers should follow the “interior
unsafety” pattern [3] by hiding unsafe operations beneath a safe interface.
The interface should be sound for all possible inputs and program states.
Otherwise, developers should expose an unsafe interface, indicating that
consumers are responsible for meeting its safety requirements.

###  II-A Development Tools & Verification

Rust supports a diverse set of tools that developers can use to generate and
verify code. Automated reasoning is particularly useful in unsafe contexts,
where developers need to build interfaces and establish invariants that Rust’s
type system cannot verify.

####  II-A1 Linting & Static Analysis

Rust’s cargo build tool supports multiple first-party and third-party plugins.
By default, this includes rustfmt [14], a code formatting tool, and Clippy
[15], a linting tool. Other extensions can help enforce safety properties.
Both cargo-audit [11] and cargo-deny [16] detect security vulnerabilities in a
project’s dependencies. However, cargo-deny is more extensible. The cargo-
geiger [17] plugin provides statistics on unsafe code. Developers can also
track if code has been reviewed by a trusted party with cargo-vet [18].

##### Dynamic Analysis

Rust is compatible with several popular dynamic bug-finding tools, including
the LLVM project’s sanitizers [19], libFuzzer [20], and Valgrind [9]. The Rust
toolchain also includes Miri, an interpreter that can detect multiple types of
undefined behavior that occur due to unsafe code. Miri models Rust’s
operational semantics [7, 21] to detect when unsafe features violate the
borrow checker’s aliasing assumptions. Currently, Miri cannot execute foreign
functions or inline assembly, but the Krabcake project [22] aims to provide
this functionality by extending Valgrind with Miri’s borrow tracking features.

##### Verification

There are multiple approaches to formally verifying Rust code, and each has a
different level of compatibility with unsafe features. Verus [23], a deductive
verifier, and Kani [24], a model checker, both support a limited subset of
unsafe features, but neither can fully validate unsafe code against Rust’s
aliasing model. Other approaches do not have any support for unsafe features.
Prusti [25] and Creusot [26], both deductive verifiers, and Flux [27], a
refinement type checker, take approaches that rely on Rust’s aliasing
invariants to simplify formal specifications. RefinedRust [28] extends on
prior work on ownership verification for C [29], and it could provide more
comprehensive support for verifying unsafe features.

##### Foreign Bindings & Encapsulations

Multiple third-party tools can assist developers with interoperating between
Rust and other languages. There are typically two steps to this process;
creating an unsafe function declaration and encapsulating it in a safe
interface. The tool bindgen [30] automatically generates a Rust library
containing foreign function declarations, constants, and struct definitions
that match the interface of a given C or C++ library. Tools such as cbindgen
[31] operate in reverse, generating foreign interfaces for Rust libraries.
Other tools, such as CXX [32], provide support for both binding and
encapsulating foreign calls and can enable bidirectional interoperation.

###  II-B Community Guidelines

In addition to development tools, multiple community resources provide
documentation on how to use unsafe code correctly. Both The Rust Reference
[33] and The Rust Programming Language [12] have sections related to unsafe
features, while The Rustonomicon [34] is dedicated entirely to unsafe code.
The Rust Project’s Unsafe Code Guidelines Working Group created the Unsafe
Code Guidelines Reference [35] as an in-depth guide to the semantics of unsafe
features, but it is incomplete and mostly unmaintained. However, the “Issues”
section of its GitHub repository is still used to discuss Rust’s semantics.

##  III Related work

One approach to understanding the role of unsafe code is to evaluate it as it
is written. Evans et al. [5] and Astrauskas et al. [4] conducted surveys of
Rust crates 16 months apart and found that 29% and 23.6% of crates contained
unsafe code, respectively. Each study used different methods, but both
observed that most unsafe code blocks were relatively small and that unsafe
function calls were the most common unsafe operation. Each study also
categorized developers’ motivations for using unsafe code. Astrauskas et al.
[4] extracted motivations from patterns observed in code, and Evans et al. [5]
created a community survey that reached 50 Rust developers on Reddit. However,
in each study, initial motivations were defined without direct engagement with
developers and include topics which are more adequately described as use-cases
for unsafe code, such as calling foreign functions and implementing cyclic
aliasing patterns.

Other researchers have manually inspected bug reports and security
vulnerabilities to understand the risks of using unsafe code incorrectly. Both
Qin et al. [3] and Xu et al. [36] found that discrepancies between the
signature of a safe API and its interior unsafe implementation are a
significant source of vulnerabilities. Cui et al. [37] focus on unsafe APIs in
Rust’s standard library and define 19 distinct safety properties that occur as
preconditions or postconditions. These approaches illustrate that it is
difficult to correctly encapsulate unsafe code, but they, too, have limited
engagement with developers. Cui et al. [37] conducted a survey to evaluate
their classification, but they did not investigate developers’ experiences
with reasoning about safety properties.

Fulton et al. [38] used a similar mixed methods design to ours, which
consisted of interviews followed by a survey. They found that most
participants had used unsafe code and the most common use case was calling
foreign functions. Their approach was effective at evaluating the overall
strengths and drawbacks of Rust as a systems programming language, but they
only briefly discussed unsafe features.

Höltervennhoff et al. [10] are the only other group to use qualitative methods
to study Rust developers’ use of unsafe code. They conducted semi-structured
interviews with 26 developers, and similar to us, they examined developers’
motivations for using unsafe code and their practices for API design. We
compare our findings with theirs when discussing the results of our first and
second research questions in sections V-B and V-C. However, beyond these
findings, our goals and results differ. They primarily focus on evaluating
developers’ security practices when using unsafe code, while we were
interested in the challenges developers face when calling foreign functions,
the effectiveness of tooling, and the Rust community’s perception of unsafe
code.

##  IV Methods

We adapted a sequential exploratory mixed-methods design [39]. This consisted
of semi-structured interviews with 19 developers. We used our findings to
create a survey, which received 160 valid responses. We chose this method to
obtain a large sample size, which allowed us to assess how our qualitative
results generalized to a broader set of developers who use unsafe code.

###  IV-A Interviews

On May 3rd, 2023, we posted a call to participate in semi-structured
interviews on the Rust subreddit (/r/rust), the Rust Programming Language
forums, and the `#dark-arts` channel of the Rust Community Discord server. We
also contacted acquaintances with relevant experience. To be eligible to
participate, candidates needed to have at least one year of experience using
Rust and had to have “regularly written or edited Rust code within an unsafe
block or function.” Eligible candidates completed a series of multiple choice
and short answer questions about their use of unsafe features and development
tools, and we used their responses to guide each interview. We also collected
participants’ years of experience with Rust and asked them to describe their
affiliation. The first and third authors collaboratively coded each
affiliation as “Industry,” “Rust Team,” “Academia,” and “Open Source.”

The first author conducted all 19 interviews remotely over Zoom between May
and July 2023. We used snowball sampling [40] to expand the pool of
participants beyond the group that responded to our initial calls to
participate. If a participant shared new information that we had not yet
learned from prior interviews, we prompted them to invite any acquaintances
with relevant experience to complete our screening survey. We stopped
recruitment once we had reached saturation. Audio recordings were transcribed
using Whisper [41], and the first author reviewed each transcript to correct
errors and label whether responses came from the interviewer or the
interviewee.

###  IV-B Qualitative Analysis

We conducted an inductive, thematic analysis of our interview transcripts
following best practices by Miles, Huberman, and Saldaña [42]. This consisted
of an open coding phase followed by closed coding and auditing.

##### Open Coding

In the open coding phase, the first, second, and third authors independently
coded random samples of quotes. Each sample contained 19 quotes, with one
chosen from each interview. Throughout this process, each coder also reviewed
the full text of each interview transcript. We stopped when we had generated
202 unique codes from three random sets of quotes. Then, we collaboratively
conducted second-cycle coding, grouping subsets of codes under themes. To
refine these themes, the first and second authors conducted closed coding of 7
additional random samples of quotes, meeting after coding each sample as
before to resolve disagreements.

##### Closed Coding & Auditing

Then, the first and second authors divided the set of transcripts between each
other to code in full, independently, using ATLAS.ti Web [43]. The first
author coded 11 transcripts, while the second author coded the remaining 8
transcripts. Throughout this process, the first and second authors updated a
shared codebook with additional codes and themes to describe new observations.
After all transcripts had been coded once, the first author conducted an
additional coding pass for every transcript and resolved all of their
disagreements with the second author. The third author reviewed all coding
decisions relevant to RQ1 and resolved additional disagreements with the first
author. The final codebook includes 191 codes grouped under 21 themes.

###  IV-C Survey

We used the themes from our qualitative analysis to create a survey in
Qualtrics [44]. The first, second, and third authors each created sets of
potential questions, and we assembled these into a survey organized by
sections corresponding to themes and research questions. Two of our interview
participants piloted an initial draft, and we incorporated their feedback into
the final version.

We relaxed the eligibility criteria for the survey to include the population
of Rust developers studied by Holtervennhoff et al. [10]. We felt that this
was reasonable, since our interview population was a strict subset of theirs;
they interviewed developers who had committed unsafe code to a public GitHub
repository, but there was no indication if they had done so regularly.
Candidates were eligible to complete our survey if they had “written, edited,
read, audited, or engaged with unsafe Rust code in any way.”

We distributed our community survey on September 22nd, 2023 to the same
communities and acquaintances as we did for the screening survey. We also
published a link in the September 28th edition of the “This Week in Rust”
newsletter, and we advertised the survey during a presentation and poster
sessions at an industry partners conference hosted by our institution. At the
end of the survey, participants could provide a link to their profile on
either GitHub or the Rust Programming Language Forums to have a chance to
receive one of two $250 gift cards. Only participants with public account
activity prior to the start date of the survey were eligible to receive
compensation. We closed the survey on October 9th, 2023.

###  IV-D Ethics

All procedures in this study were reviewed and approved by our institution’s
IRB. Prior to each intervention, participants were read or presented a script
outlining the procedures of the study and they were asked to affirm consent.
Participants were reminded to avoid sharing personally identifying information
in their responses, and interview participants were asked to remain in a
private space. The first author reviewed the contents of the replication
package and redacted all direct (e.g. name, affiliation) and indirect (e.g.
position, project) personal identifiers.

###  IV-E Threats to Validity

We used a mixed methods design to improve construct validity and audited our
final coding decisions to ensure consistency. The internal validity of our
results depends on participants’ ability to accurately self-report their
behavior, and what developers perceive about their use of unsafe code may
differ from how unsafe is used in practice. External validity and
generalizability also remain threats to validity, which is common for research
studies that use surveys and interviews [45]. We configured Qualtrics to
reject multiple responses from the same individual and to detect fraudulent
responses using both RelevantID [46] and reCaptcha [47]. Recruitment via
snowball sampling and through community forums may bias our results toward
certain areas of the Rust community. Additionally, the gift cards that we
offered for compensation are most easily redeemable in North America and
Europe, which may have dissuaded certain individuals from participating. We
did not collect location data from participants, so we cannot determine if our
results are biased toward individuals from any particular area.

###  IV-F Replication

Our replication package is publicly available on GitHub
111github.com/icmccorm/against-the-void/. It contains all interview
transcripts, survey responses, codebooks, and coding decisions.

##  V Results

We find that developers use unsafe code for ergonomics, performance, and when
they perceive that there is no alternative. Despite attempts to keep unsafe
code minimal and encapsulated, developers feel uncertain about its
correctness. These results mirror findings from Holtervennhoff et al [10], and
we compare our results with theirs in Sections V-B and V-C. Foreign function
calls were used by most participants, and non-trivial differences between Rust
and other languages made these calls difficult to encapsulate. Developers
rarely audited their dependencies. Most had used Miri, but few used it to
validate their test cases, and its lack of support for foreign function calls
deterred some individuals from using it.

###  V-A Demographics

We invited all 42 eligible candidates to participate in interviews, and we
interviewed 19 who responded to our invitation. Of these participants, 15 had
experience in industry, 6 in open source development, 3 in academia, and 2
were members of the Rust team. They had 4.7 years of experience with Rust on
average, and the most experienced participant had been using Rust for 10
years.

We received 370 survey responses, of which 242 were complete, 203 met our
eligibility criteria, and 160 passed each of Qualtrics’ measures for fraud
detection. Hereafter, when we refer “respondents,” we include only those that
are eligible. The majority were young, educated, identified as male, and had
experience in either industry or open-source development. Nearly 66% had
completed at least a bachelor’s degree, and 30% had completed a graduate
degree. Most respondents (83%) were younger than 40 years old, and 56% were
between the ages of 18 and 29. We followed Spiel et al.’s [48] guidelines for
ethically surveying respondents’ gender; 80% identified as “Man”, 6%
identified as “Non-binary,” 3% identified as “Woman” and 9% did not disclose
their gender. On average, survey respondents had more than a decade of
experience in software engineering, more than 5 years of experience in both C
and C++, and just over 4 years of experience using Rust.

###  V-B Why unsafe?

Figure 1: How frequently participants measured the impact of their changes
when using unsafe to improve performance

Participants used unsafe code because they perceived that it was more
performant or ergonomic than safe alternatives, or that there were no safe
alternatives at all.

####  V-B1 Unsafe Performs Better

Some interview participants and 47% of survey respondents indicated that they
would use unsafe code when they perceived that it was faster or more space-
efficient than an equivalent safe API. In these situations, participants
usually identified a safe alternative, but they felt that it would be too
expensive to use; “so you could do it, right? But that would be some
performance overhead” (P12).

Most interview participants who attempted to use unsafe to improve performance
had implemented small-scale optimizations in existing applications. These
typically involved eliminating runtime checks, which were seen as unnecessary
due to local or global invariants. For example, one participant used an unsafe
API to consume a pointer to a string without checking if it contained valid
UTF-8 characters. This pointer was provided by a foreign call to Python, which
the participant reasoned would only ever provide text in a valid format.
Another participant chose to store a heap allocation as a raw pointer in a
static mutable variable instead of using one of Rust’s safe static
encapsulations, since they felt confident that the allocation would be valid
for the duration of the program.

However, a few interview participants used unsafe code to create entire
components that were purpose-built to achieve performance. In each situation,
performance was seen as a functional requirement for the application domain:
“I write a serialization framework…And obviously a goal for it is to be
extremely performant” (P3). A participant who took this approach found that it
significantly increased the amount of unsafe code in their application, but
they felt that this was a reasonable compromise to achieve greater
performance.

The 47% of survey respondents who reported using unsafe code to improve
performance did not have a strong tendency toward either large-scale or small-
scale use. We found that 26% only pursued “small-scale-optimizations in
existing applications,” 20% only built new, “large-scale components,” and 42%
used unsafe code for performance in either situation.

Participants did not consistently measure the performance impact of unsafe
code. In some situations, they relied on their intuition to determine which
design decisions would be best: “I’ll totally admit that…I think this is going
to be a hot thing, so I’m going to kind of prematurely optimize” (P11). Survey
respondents who used unsafe to increase performance were also inconsistent
about measuring its impact. However, 65% reported measuring performance at
least half the time or more, as shown in Figure 1. Similarly, Höltervennhoff
et al. [10] found that 6 participants would use unsafe code to improve
performance when the change was “noticeable.” One of our interview
participants performed sophisticated profiling on three versions of a
component, each of which had varying amounts of unsafe. The most unsafe-heavy
version performed the best, but they opted for one with a moderate amount of
unsafe code, balancing concerns for safety and performance.

####  V-B2 Unsafe is Easier or More Ergonomic

Participants also chose to use unsafe code when they felt it would be easier
than using a safe API. This was the least common motivation cited by interview
participants, and only 18% of survey respondents identified with this
motivation. Höltervennhoff et al.[10] indicate that some of their participants
would use unsafe code if it “saved them effort”, but it was unclear if this
motivation was typical. Our participants usually stated or implied a choice
between using safe or unsafe design patterns. For example, one participant
chose to use only Rust’s unsafe allocator API rather than a mix of unsafe and
safe allocation operations. They found that this approach was easier to reason
about, since “if there’s too much abstraction, you lose the information that
you need to make it actually sound” (P4).

The unsafe function transmute performs unrestricted type conversion, and it
was used by several participants to circumvent safe APIs that they found were
difficult to use. One participant used transmutation to simplify the
implementation of chess engine, which had three enumerations to represent the
position of a chess piece. They found that it was easier to use integer
conversion, bitwise operations, and transmutation to convert between
enumerations instead of safely matching on their values. Another participant
used transmutation when working with a Rust encapsulation of a C font library.
This library exposed C heap allocations as Rust object with short
lifetimes—even though these objects would remain valid on the heap for the
duration of the program. This participant found it was easier to transmute the
Rust objects to have the 'static lifetime, which is indefinite, instead of
adapting to the restrictions of the encapsulation. 14 I just…transmuted that
one to 'static… another use case for unsafe is working around a bad API when
you know that you can use it correctly.

####  V-B3 No Clear Alternative

In most situations, participants perceived that they had no clear alternative
to using unsafe code. The majority of interview participants and 77% of survey
respondents identified with this motivation. Höltervennhoff et al.[10]
observed this to a similar extent; 18 out of their 26 participants claimed to
use unsafe code by necessity. Our interview participants often reported being
constrained by a combination of Rust’s type system and the properties of their
applications. For example, in Rust, if a type implements the unsafe traits
Send and Sync, then users can freely use and share its values in multithreaded
contexts. One participant found that it was necessary to implement Send and
Sync in a single-threaded application to maintain compatibility with an
existing API. The implementation was sound by construction, but unsafe was
still necessary.

Participants who contributed to just-in-time (JIT) compilers and operating
systems found it necessary to access memory through raw pointers. This was
necessary for one participant to implement an unwinding table: “I have to jump
to that address and I really can’t do anything to verify it…” (P12). In other
situations, developers felt that there could be a safe alternative, but they
were unaware of how to implement it: “there’s probably a better way that I
just don’t know about yet or didn’t know about at the time and I haven’t
thought about it” (P18). However, 65% of survey respondents were certain at
least most of the time, if not always, that it would be impossible to
implement an equivalent safe pattern.

####  V-B4 Co-occurrence

These motivations were not mutually exclusive. One participant implemented a
zero-copy deserialization pattern using unsafe code, and their experience
relates to each of the reasons we identified. Performance was a domain-
specific constraint of the internationalization library that they contributing
to, which motivated them to use zero-copy deserialization. We reviewed the
library’s documentation, and it indicated that contributors had considered
using an existing crate, but its API was not ergonomic for their use case. The
participant decided to implement their own version, and they perceived that
unsafe code was inherently necessary.

RQ2 \- How do Rust developers reason about encapsulating unsafe code?
Participants were most often motivated to use unsafe code because they felt
that there was no safe alternative. In other situations, participants used
unsafe code because it performed better or was more ergonomic. For
performance, developers pursued both large-scale abstractions and small-scale
optimizations, but they did not consistently measure the impact of their
changes. Each of these motivations can influence a design decision
simultaneously.

###  V-C Design & Use of Unsafe APIs

####  V-C1 Invariants

Participants reasoned about the soundness and correctness of their unsafe code
in terms of invariants. When developing operating systems and embedded
applications, certain invariants were provided by hardware specifications. For
example, one participant who developed an embedded application would write a
“magic value” (P8) to RAM to indicate which operation had triggered a reboot.
The microcontroller that they used did not erase its memory on reboot, so they
could rely on the value being initialized. Some participants perceived
hardware-level correctness properties as “orthogonal to Rust” (P1), since they
are not related to the properties of the language. However, other participants
who developed embedded applications relied on Rust libraries that encode
hardware requirements into the type system so that if a device is
misconfigured, then their program will not compile.

Developers also leveraged Rust’s type system to uphold invariants that they
perceived as necessary for soundness. In operating systems, these typically
began as operation-specific refinement properties on primitive types, but they
quickly built up into system-level invariants; “it gets to a high-level really
quickly” (P3). Rust’s aliasing restrictions were a natural fit: 12 We only
allow… someone who has access to one of these mapped memory regions, to borrow
it… That’s a great example of taking something that’s unsafe inherently and
then kind of wrapping it up in a safe abstraction, using the power of Rust to
do the borrow checking for us. Another participant experimented with using
ghost permission to reason about pointer arithmetic; this method was
popularized for Rust by Yanovski et al.’s GhostCell type [49].

Other participants relied on ad-hoc reasoning to ensure that their programs
were correct and free of undefined behavior. Some appealed to their prior
experience using C++; “[in] C++, you do this all the time” (P15). One
participant was accustomed to reasoning about aliasing rules in compiler
development, so they felt confident that they were adhering to Rust’s
restrictions in unsafe contexts. Other participants validated their decisions
by auditing their code, but most had some level of implicit trust; “that’s
sort of another… I got to trust it type thing” (P5). This aligns with
Höltervennhoff et al. [10]—a majority of their participants reasoned about
unsafe code in terms of contracts or invariants, but some reported using it
carelessly, and most felt that it was difficult to write correctly due to its
context-sensitivity.

####  V-C2 Isolation

The majority of interview participants made a conscious effort to minimize and
isolate unsafe code, which made it easier to document and reason about. 10
That’s generally the most important thing, just being self-contained, being
well-isolated, being well-encapsulated…

Most survey respondents (61%) predicted that it would be at least somewhat
easy for another developer at their skill level to understand a random unsafe
block or function from their code. Interview participants who isolated their
unsafe code had more confidence that its requirements were met, especially if
they went beyond what Rust’s type system could verify. However, participants
who contributed JIT compilers and operating systems reported that unsafe code
was prevalent. Domain-specific, non-local reasoning was necessary to
understand any arbitrary unsafe code snippet: “you have to know the innards of
the JIT-compiled function…so it’s just not encapsulated” (P11). Though unsafe
code may have been isolated, it is unclear if participants used it minimally.
Among survey respondents, 63% only sometimes refactored their code to remove
unsafe, while 25% never did at all.

####  V-C3 Encapsulation

Most interview participants and 88% of survey respondents attempted to
encapsulate unsafe code beneath safe APIs. This was also common for
Höltervennhoff et al. [10], who found that 22 of their 26 participants
mentioned safe interfacing. Our interview participants would create
encapsulations where they perceived that the requirements for their unsafe
code were satisfied by either Rust’s type system or what they reasoned to be
local invariants. However 43% of survey respondents reported exposing unsafe
APIs, suggesting that it may be difficult to avoid. In contrast, only 6 of
Holtervenhoff et al. [10]’s participants indicated exposing an unsafe
interface.

Interview participants cited the same motivations for exposing unsafe APIs as
they did for using unsafe code in any capacity. One interview participant
exposed unsafe versions of safe API endpoints, which behaved the same as their
safe counterparts, but they did not include runtime checks. Another
participant exposed unsafe APIs to allow users to circumvent their safe
encapsulations in case they became difficult to use. One participant indicated
that it was a cultural value among Rust developers to make API surfaces as
detailed as possible: “Rust has a tendency to… make the API as complicated as
it needs to be to fully represent what is actually happening behind the
scenes” (P1). Other participants indicated that they would only expose an
unsafe API when it would be impossible to encapsulate without placing the
burden of correctness on the user. Höltervennhoff et al. [10] also found that
participants would expose unsafe APIs by necessity or to improve performance,
but it was unclear if ergonomics was also a motivation.

Our survey respondents also identified with these motivations for exposing
unsafe APIs. The majority (76%) exposed unsafe APIs because it would be
impossible to encapsulate them without preconditions, while 51% exposed unsafe
APIs for performance and 13% did so for ergonomics. Respondents typically
documented the safety requirements of their unsafe APIs—61% of respondents
only exposed unsafe APIs when they had requirements beyond Rust’s type system,
and 78% would always document these requirements.

####  V-C4 Uncertainty

Developers who created safe APIs for unsafe code were uncertain if their
encapsulations were sound in all possible situations.

9 No library author knows when they’re going to trigger [undefined behavior]…
we do our best to give you a sound interface, but god knows if there’s a hole
in it…

Only 23% of survey respondents were always certain that their safe
encapsulations met the requirements of the unsafe code within. Höltervennhoff
et al. [10] had similar findings; 16 of their 26 participants felt some degree
of uncertainty about whether their unsafe code was correct. Our interview
participants were most often uncertain when unsafe was pervasive or when using
Rust’s Foreign Function Interface (FFI). This is a significant contrast from
Holtervennhoff et al. [10], who found that it was more common for participants
perceive foreign functions as easy to encapsulate.

When describing their uncertainty about encapsulation, our participants
compared the “theory” of safety to the “practice” of how an API would be used.
When these concepts didn’t align, errors occurred. One participant observed
this mismatch with a Rust crate that encapsulated a foreign library. The
library used a memory-mapped file, and the API could unmap the file while the
user retained a reference to it, leading to a use-after-free error. Another
participant struggled with discrepancies between the interface and the
implementation of Rust’s Layout API, which describes the size and alignment of
an allocation. The design of Layout was changed to “tweak the safety
requirements” (P14), which regretfully caused previously correct code to
exhibit undefined behavior.

RQ3 \- How do Rust developers reason about memory safety across foreign
function boundaries? Participants derived requirements for unsafe code from
external specifications and the inherent guarantees of the borrow checker.
However, they predominantly relied on ad-hoc reasoning and tacit knowledge and
were uncertain if their safe encapsulations were sound in all situations.
Participants attempted to minimize and isolate unsafe code, but this was not
always possible in certain domain-specific contexts, such as JIT compilers and
operating systems. Additionally, when participants chose to expose unsafe
APIs, we found that they were motivated by performance, ergonomics, and
necessity to the same extent as for any arbitrary use of unsafe code.

###  V-D Foreign Bindings & Memory Models

Figure 2: How often survey respondents avoid passing abstract data types by
value across FFI boundaries (“ADTs by value”) and avoid converting foreign raw
pointers into safe references (“Pointers to refs.”).

The majority of interview participants and 70% of survey respondents used
Rust’s foreign function interface (FFI). Multiple prior studies of Rust
developers [10, 38, 4] have also found that FFI use is common. Most
participants found it difficult to reconcile the differences between Rust’s
memory model and foreign memory models. Participants reviewed the
documentation and code from foreign libraries to ensure they were used safely,
and they attempted to keep interoperation as straightforward as possible.

####  V-D1 Differing memory models

Interview participants encountered several key differences between Rust and
both C and C++ that made interoperation difficult. In C and C++, types can
implement methods with varying thread-safety properties, but Rust’s Send and
Sync traits impose requirements on every behavior of a type. This made it
difficult for one participant to encapsulate foreign types as wholly Send or
Sync in Rust. They also noted that foreign libraries may have aliasing
patterns that are fundamentally at odds with the borrow checker’s
restrictions. They observed that certain C APIs that receive source and
destination pointers assume that the source and destination can alias, but
aliasing and mutation are exclusive in Rust, so this pattern was difficult to
encapsulate. Participants also noted that C allows unrestricted integer-to-
pointer conversion, which is limited under Rust’s strict provenance model, and
that Rust’s Drop semantics are different from a C++ destructor.

To manage these differences, participants reviewed the documentation and
implementation of foreign libraries to determine their safety requirements.
The reverse of this situation occurred when participants exposed Rust
libraries to languages with fewer safety guarantees. Implicit properties from
Rust became documentation in other languages. 13 …in Rust, you can trivially
say, I’m going to return to you a thing, which just borrows my memory, and
then you just can’t access me while you’re using that …this is not very easily
mappable into a C API, or rather you could do it, but then you have to read
all this documentation… Not all foreign APIs had were difficult to
encapsulate, though. One participant observed that C APIs tend to expose pairs
of initialization and cleanup functions for each type, which can be linked
into the behavior of its Rust encapsulation. Others found that existing C APIs
were easy to encapsulate since “there [were] no complex lifetimes involved”
(P11).

####  V-D2 Simple FFI

Participants attempted to keep interoperation with foreign code as
straightforward as possible. When declaring bindings, one participant
preferred using primitive types and pointers over abstract data types like
Option and NonNull, even though these types can be implicitly cast into raw
pointers. They felt that these casts hid useful contextual information, so
they preferred to use raw pointers explicitly.

A few participants indicated that they would avoid passing structs by value in
certain situations. One of these participants found that certain Rust programs
would behave incorrectly when structs were moved across a foreign boundary
instead of copied, or if they had a destructor on the C++ side of the
boundary. They typically avoided passing structs by value in either situation,
and they recommended that “…if you’re doing FFI, the only things you should
pass by value are Copy types” (P2). In most cases, participants felt
comfortable passing structs by value if they were Copy and had equivalent
layouts on each side of the boundary. However, as shown in Figure 2, nearly
half (49%) of survey respondents avoided passing abstract data types at least
half of the time, if not more, suggesting that this practice is not
universally followed.

In addition to limiting the use of certain types at boundaries, interview
participants also attempted to minimize the interactions between each side of
the FFI boundary. They thought that a “very chatty API” (P7) would be
difficult to validate. One participant described the complexity of a foreign
API in terms of the heap object graph that is exposed to Rust. They perceived
that if the structure of the foreign heap is exposed in detail, then it will
make safe encapsulations difficult to use.

14 It’s hard to keep those objects around for very long unless you want all of
your structs to end up with a bunch of lifetimes… you don’t want that… it’ll
scare away new programmers for sure. Survey respondents found it difficult to
avoid converting foreign raw pointers into Rust’s reference types; only 23%
actively avoided this more than half of the time, as shown in Figure 2. This
does not necessarily indicate that foreign heap objects tend to span FFI
boundaries, but it does suggest that invalid pointer-to-reference casts may be
a significant, unchecked source of undefined behavior, since Rust-specific
development tools have limited support for foreign function calls.

RQ4 \- How do Rust developers validate unsafe code? Developers perceived that
certain API patterns were easy to encapsulate. However, unlike Holtervennhoff
et al. [10], interview participants typically found it difficult to use
foreign functions correctly because their aliasing and thread-safety
properties were opaque and conflicted with the restrictions of the borrow
checker and other Rust idioms. Participants minimized their interaction with
foreign code to mitigate these differences. Some interview participants and
survey respondents avoided passing structs by value, but survey respondents
were not likely to avoid converting foreign raw pointers into safe references.

###  V-E Validation

Figure 3: How often survey respondents wrote tests for unsafe code (“Write
tests”), how often they audited their dependencies’ use of unsafe code(“Audit
dependencies”), and how often participants executed their test cases in Miri
if they had used the Miri at least once (“Run in Miri”).

Participants used a wide variety of development tools to assist with
validating their design choices.

##### Dynamic Analysis

Most participants used dynamic bug-finding tools. Interview participants often
mentioned Miri, AddressSanitizer, and Valgrind. Survey respondents most
frequently used Miri (61%), Valgrind (44%), cargo-fuzz (26%), and
AddressSanitizer (26%). Each remaining tool was used by less than 15% of
survey respondents. Three interview participants and 31% of survey respondents
used fuzzing tools. One interview participant leveraged industry sponsorship
to fuzz a JIT compiler for hours-on-end. Another participant implemented a
B-Tree with optimizations for high-performance computing, and they used
libFuzzer to perform differential testing [50] against the BTree from Rust’s
standard library.

A majority (61%) of survey respondents wrote tests for their unsafe
applications at least most of the time. However, Figure 3 shows 56% of
participants who had used Miri only sometimes used it to validate their test
cases, if at all. This may be due to Miri’s lack of support for foreign
functions, which was noted by the majority of interview participants. This was
also a significant obstacle for survey respondents; 43% were deterred from
using Miri because it could not call foreign functions. Respondents also
avoided using Miri due to its performance (26%) and lack of support for inline
assembly (19%). One participant had frequently encountered aliasing violations
due to foreign function calls. They circumvented Miri’s lack of FFI support by
creating mocks of foreign functions, but this solution did not scale to
programs with “50,000 lines of code” (P4) This participant and a few others
were aware of the Krabcake project [22]. Each was already using Valgrind with
Rust codebases, and they perceived that Krabcake would be an ideal solution
for overcoming each of Miri’s limitations.

##### Debugging

Few interview participants reported using a debugger with Rust code. Only 10%
of survey respondents used a debugger daily, 29% used one monthly, and 44%
used one yearly at most, if at all. One interview participant had used both
MSVC’s debugger and LLDB, and while both worked, neither matched the quality
of the other parts of the Rust toolchain, “where you get all the bells and
whistles that you want, and things just work nicely together” (P3). Mozilla’s
RR debugger [51] was useful for one participant to resolve bugs found from
fuzzing, but it was generally more common for participants to report using
“printf-style” (P7) debugging.

##### Formal Methods

Few interview participants and only 6% of survey respondents used tools that
apply formal methods for static and dynamic verification. Interview
participants mentioned using Kani, Prusti, and Crucible. Kani was useful for
one participant, who needed to prove that a particular routine would never
execute more than twice for any input. Another participant who had “tried a
bunch of the tools” (P11) in this category found that Prusti was the most
complete, but they felt that it would be more effective if these tools were
“inherently part of Rust” (P11) so that they could “hoist a lot of currently
unsafe code into safe code” (P11). Of the survey respondents who had used
formal methods tools, 50% had used Kani. Both Prusti and Creusot were used by
30%, and Flux was used by 10%.

####  V-E1 Auditing

Only a few interview participants described auditing unsafe code. One
participant would typically only examine their dependencies if they were
curious about how they functioned. However, they did actively avoid
dependencies that used outdated versions of Rust or deprecated features.
Another participant was in the process of having a third-party institution
examine their codebase to determine if they could reduce the amount of unsafe
code. Survey respondents also rarely audited their dependencies’ use of unsafe
code; 88% audited their dependencies’ use of unsafe less than half the time,
as shown in Figure 3. Additionally, only 51% of respondents had used automated
auditing tools. Of this subset, 63% had used cargo-audit, 42% had used cargo-
update, 31% had used cargo-deny, 22% had used cargo-geiger, and only 7% had
used cargo-vet.

RQ5 \- How is unsafe code perceived by the members of the Rust community who
engage with it? Miri, Valgrind, cargo-fuzz, and AddressSanitizer were the most
popular dynamic bug-finding tools that developers used with Rust applications.
Though Miri was the most commonly used dynamic bug-finding tool, its
performance and lack of support for key features deterred developers from
using it. Holtervennhoff et al. [10] found that developers rarely audit their
code; our results illustrate that this is generally true. Developers rarely
audited their dependencies’ use of unsafe code, and less than half had used
auditing tools.

###  V-F Community & Culture

Safety is one of the Rust community’s core values. This informs positive
practices, like safe encapsulation of unsafe APIs, but can also be taken to
extremes, as newcomers report unexpectedly harsh feedback from more
experienced developers when they ask for advice on community forums. However,
community resources are essential to understand the nuances of Rust’s evolving
semantics.

####  V-F1 A Preference for Safety

Participants perceived that the Rust community has a strong preference for
safety that affects tool design and development practices. Participants
avoided exposing unsafe APIs in libraries partly because of the perception
that other Rust developers would otherwise avoid using their crate: “everyone
wants a safe interface… no one really wants to go use the unsafe heavy ones”
(P9). An overwhelming majority of survey respondents also preferred safe APIs;
83% would choose a safe API over an unsafe alternative most of the time, if
not always. One library author viewed safe encapsulation as an inevitable
requirement. Even though exposing an unsafe trait places the burden of
responsibility on users, they would still blame the library.

7 So in theory, the burden of proof is on them, but also if they implement
something improperly, then they’re going to create a ticket because they’ll be
like, oh, “I use your library and it’s a fault.”

####  V-F2 A Stigma Against Unsafe

Three participants observed a negative sentiment from other members of the
Rust community toward any risk of undefined behavior. This was typically
described in extreme terms: “you have even the slightest concession to
undefined behavior, and you’ve committed a crime against humanity” (P7). One
participant perceived that this type of “knee-jerk” (P7) reaction was common
toward newcomers asking for advice on community forums. They believed that
these individuals were likely unaware of safer alternatives or the risks of
their design choices. Fitting this pattern, another participant posted on a
forum asking for advice on a potentially unsound optimization, and they
received feedback that they felt was unnecessarily harsh.

Survey respondents were prompted to evaluate their perception of the Rust
community’s view toward unsafe code that they write, as well as unsafe code in
general. Most respondents were unsure about how their unsafe code was
perceived, but 64% felt that the Rust community views unsafe code at least
somewhat negatively overall.

####  V-F3 Shifting Ground

Interview participants had a collective sense of uncertainty about Rust’s
semantics. Participants perceived that Rust lacked a formal specification and
that guidelines for unsafe code were incomplete.

10 I think the biggest issue with Rust’s unsafe isn’t necessarily the tools,
but just the spec—or, rather, the absence of the spec. If you write unsafe
code today, you write it against the void.

A few interview participants encountered situations where the precise
definition of undefined behavior was still unclear. One participant needed to
create a mutable reference to uninitialized memory, but this is considered
undefined behavior by Rust’s current semantics [52]. A similar debate occurred
involving what one participant referred to as the “ref header problem” (P10),
which is also documented in an issue in the Unsafe Code Guidelines repository
[53]. They wished to be able to take a reference to the first field of a
product type and use it to create valid references to adjacent fields using
integer offsets. This was considered undefined behavior under Stacked Borrows,
but it is now valid under Tree Borrows semantics. Additionally, multiple
participants were uncertain about how to handle the case when a program panics
during a Drop. In contrast, the majority (70%) of survey respondents found
that the Rust community’s documentation and guidance was adequate for them to
know how to use unsafe code correctly most of the time, if not always. This
suggests that concerns about Rust’s evolving semantics, while important, are a
niche issue for most developers that engage with unsafe code.

RQ5 \- How is unsafe code perceived by the members of the Rust community who
engage with it? Interview participants perceived that Rust developers prefer
safe APIs, which incentivized them to avoid exposing unsafe endpoints. The
Rust community perceives unsafe code somewhat negatively, and when reaching
out to community members for feedback, newcomers are met with harsh reactions
to their use of unsafe code or their lack of caution toward undefined
behavior. Developers rely on community guidance to understand what comprises
undefined behavior, which has shifted as Rust has matured. However, Rust’s
documentation is typically adequate for them to know how to use unsafe
correctly.

##  VI Discussion

Our results indicate that Rust’s tooling and community practices do not yet
adequately support developers’ use cases for unsafe code. Safe encapsulation
is helpful, but it is not a complete solution—API designers still rely on ad-
hoc reasoning over formal assurances, and consumers rarely audit their
dependencies. This combination of uncertainty and implicit trust must be
addressed to make Rust’s promise of static safety a reality. We provide the
following recommendations:

##### For Rust Developers

Developers should reflect on their motivations and rationale before deciding
to use unsafe code. Is there empirical evidence that unsafe code increases
performance? Are the benefits from using unsafe to improve ergonomics worth
the additional overhead of manually reasoning about soundness? If unsafe code
is truly unavoidable, then developers should make use of Rust’s tight
integration with a variety of powerful development tools. Both Miri and cargo-
audit should be considered essential for effective Rust development, even if
an application only consumes safe APIs.

##### For Researchers

Developers cite layered motivations for using unsafe code, and the soundness
of any particular unsafe construct depends on both its semantics and its use
in practice. To accurately describe the role of unsafe in the Rust ecosystem,
researchers should separate developers’ cited motivations for using a given
feature from that feature’s functional characteristics. Any individual use of
a feature, such as marking a function as unsafe, may be motivated by a
combination of necessity, performance, or ease-of-use.

##### For the Rust Team

It is heartening that the majority of respondents used Miri, but it is also
also somewhat troubling that more than a third (39%) had never done so. The
Rust team should consider making Miri easier to access by moving it into
Rust’s stable toolchain or by having it installed by default with the nightly
toolchain. Additional emphasis on the benefits of Miri in introductory
resources would also be helpful, as projects that use only safe code may rely
on unsound encapsulations of unsafe features. Miri’s lack of support for
foreign function calls also represents a critical safety gap for Rust. The
Krabcake project is a promising solution, and the Rust team should invest
resources into making it production-ready. However, other architectural
approaches should also be considered, such LLVM’s Sanitizer API.

##### For Interoperation Tool Developers

Interoperation tools could be improved with automated reasoning for inferring
the aliasing and thread-safety properties of foreign functions. This
information could be used to generate safe encapsulations for generated
bindings. For example, contributors to Clang have proposed a form of lifetime
annotation inference for C++ [54]. If sound and feasible, these annotations
could be lifted from C++ function declarations to generate Rust bindings with
equivalent annotations. A similar form of verification for thread-safety
properties would also be helpful to determining if foreign types can implement
Send or Sync or to warn when foreign functions retain copies of reference-type
arguments.

##### For Educators

The Rustonomicon [34] is a helpful entry point for learning about Rust’s
unsafe features, but its one-size-fits-all approach will inevitably leave
certain categories of users wanting more targeted information. We agree with
Holtervennhoff et al. [10] that Rust developers need a dedicated resource that
succinctly summarizes ongoing debates about Rust’s semantics. Additionally, we
find that library developers may also benefit from targeted resources on how
to manage unsafe code in API design.

##  VII Conclusion

We used a mixed-methods approach to determine what Rust developers cite as
their motivations for using unsafe code and how they reason about its
soundness. We conducted semi-structured interviews with 19 Rust developers and
used our findings to create a survey that received 160 valid responses. We
found that developers typically use unsafe code when they perceive that there
is no safe alternative, and they attempt to minimize, localize, and
encapsulate it whenever possible. However, reliance on trust and ad-hoc
reasoning left developers uncertain if their designs were sound. To resolve
this safety gap, developers who engage with unsafe code need to make automated
validation an essential component of their workflow, and tool developers must
continue to add robust support for unsafe features.

## Acknowledgment

This material is based upon work supported by the Department of Defense and
the National Science Foundation under Grant Nos. CCF-1901033, DGE1745016, and
DGE2140739. Any opinions, findings, and conclusions or recommendations
expressed in this material are those of the authors and do not necessarily
reflect the views of the Department of Defense or the National Science
Foundation.

## References

  * [1] Stack Overflow. (2023) Stack overflow developer survey 2023. [Online]. Available: https://survey.stackoverflow.co/2023/
  * [2] R. Pereira, M. Couto, F. Ribeiro, R. Rua, J. Cunha, J. a. P. Fernandes, and J. a. Saraiva, “Energy efficiency across programming languages: How do energy, time, and memory relate?” in _Proceedings of the 10th ACM SIGPLAN International Conference on Software Language Engineering_ , ser. SLE 2017\. New York, NY, USA: Association for Computing Machinery, 2017, p. 256–267. [Online]. Available: https://doi.org/10.1145/3136014.3136031
  * [3] B. Qin, Y. Chen, Z. Yu, L. Song, and Y. Zhang, “Understanding memory and thread safety practices and issues in real-world rust programs,” in _Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation_ , ser. PLDI 2020. New York, NY, USA: Association for Computing Machinery, Jun. 2020, p. 763–779. [Online]. Available: https://doi.org/10.1145/3385412.3386036
  * [4] V. Astrauskas, C. Matheja, F. Poli, P. Müller, and A. J. Summers, “How do programmers use unsafe rust?” _Proc. ACM Program. Lang._ , vol. 4, no. OOPSLA, Nov. 2020. [Online]. Available: https://doi.org/10.1145/3428204
  * [5] A. N. Evans, B. Campbell, and M. L. Soffa, “Is rust used safely by software developers?” in _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ , ser. ICSE ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 246–257. [Online]. Available: https://doi.org/10.1145/3377811.3380413
  * [6] S. Zhu, Z. Zhang, B. Qin, A. Xiong, and L. Song, “Learning and programming challenges of rust: A mixed-methods study,” in _Proceedings of the 44th International Conference on Software Engineering_ , ser. ICSE ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 1269–1281. [Online]. Available: https://doi.org/10.1145/3510003.3510164
  * [7] R. Jung, H.-H. Dang, J. Kang, and D. Dreyer, “Stacked borrows: An aliasing model for rust,” _Proc. ACM Program. Lang._ , vol. 4, no. POPL, Dec. 2019\. [Online]. Available: https://doi.org/10.1145/3371109
  * [8] Rust Community, “Miri,” https://github.com/rust-lang/miri, Jan. 2024.
  * [9] N. Nethercote and J. Seward, “Valgrind: a framework for heavyweight dynamic binary instrumentation,” _SIGPLAN Not._ , vol. 42, no. 6, p. 89–100, Jun. 2007. [Online]. Available: https://doi.org/10.1145/1273442.1250746
  * [10] S. Höltervennhoff, P. Klostermeyer, N. Wöhler, Y. Acar, and S. Fahl, ““I wouldn’t want my unsafe code to run my pacemaker”: An interview study on the use, comprehension, and perceived risks of unsafe rust,” in _32nd USENIX Security Symposium (USENIX Security 23)_. Anaheim, CA: USENIX Association, Aug. 2023, pp. 2509–2525. [Online]. Available: https://www.usenix.org/conference/usenixsecurity23/presentation/holtervennhoff
  * [11] Rust Community, “cargo audit,” https://github.com/RustSec/rustsec/tree/main/cargo-audit, Feb. 2024.
  * [12] S. Klabnik and C. Nichols, “The rust programming language,” https://doc.rust-lang.org/book/, Nov. 2022.
  * [13] W. Crichton, “The usability of ownership,” _CoRR_ , vol. abs/2011.06171, 2020\. [Online]. Available: https://arxiv.org/abs/2011.06171
  * [14] Rust Community, “rustfmt,” https://rust-lang.github.io/rustfmt/, Mar. 2024\.
  * [15] ——, “Clippy documentation,” https://doc.rust-lang.org/clippy/#clippy, Mar. 2024.
  * [16] ——, “cargo-deny,” https://embarkstudios.github.io/cargo-deny/, Feb. 2024\.
  * [17] ——, “cargo-geiger,” https://github.com/geiger-rs/cargo-geiger, Feb. 2024\.
  * [18] ——, “Cargo vet,” https://mozilla.github.io/cargo-vet/index.html, Feb. 2024.
  * [19] LLVM Project, “Clang 18.0.0git documentation,” https://clang.llvm.org/docs/, Jan. 2024.
  * [20] ——, “libfuzzer – a library for coverage-guided fuzz testing.” https://github.com/llvm/llvm-project/blob/main/llvm/docs/LibFuzzer.rst, Oct. 2022.
  * [21] N. Villani, D. Dreyer, and R. Jung, “Tree borrows,” 2023. [Online]. Available: https://github.com/Vanille-N/tree-borrows/blob/master/full/main.pdf
  * [22] F. S. Klock and B. Garza, “Krabcake: A rust ub detector,” Rust Verification Workshop, Apr. 2023, accessed: 2023-10-20. [Online]. Available: https://pnkfx.org/presentations/krabcake-rust-verification-2023-april.pdf
  * [23] A. Lattuada, T. Hance, C. Cho, M. Brun, I. Subasinghe, Y. Zhou, J. Howell, B. Parno, and C. Hawblitzel, “Verus: Verifying rust programs using linear ghost types,” _Proc. ACM Program. Lang._ , vol. 7, no. OOPSLA1, Apr. 2023\. [Online]. Available: https://doi.org/10.1145/3586037
  * [24] A. VanHattum, D. Schwartz-Narbonne, N. Chong, and A. Sampson, “Verifying dynamic trait objects in rust,” in _Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice_ , ser. ICSE-SEIP ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 321–330. [Online]. Available: https://doi.org/10.1145/3510457.3513031
  * [25] V. Astrauskas, P. Müller, F. Poli, and A. J. Summers, “Leveraging Rust types for modular specification and verification,” in _Object-Oriented Programming Systems, Languages, and Applications (OOPSLA)_ , vol. 3, no. OOPSLA. ACM, 2019, pp. 147:1–147:30. [Online]. Available: http://doi.acm.org/10.1145/3360573
  * [26] X. Denis, J.-H. Jourdan, and C. Marché, “Creusot: A foundry for the deductive verification of rust programs,” in _Formal Methods and Software Engineering: 23rd International Conference on Formal Engineering Methods, ICFEM 2022, Madrid, Spain, October 24–27, 2022, Proceedings_. Berlin, Heidelberg: Springer-Verlag, 2022, p. 90–105. [Online]. Available: https://doi.org/10.1007/978-3-031-17244-1%5F6
  * [27] N. Lehmann, A. T. Geller, N. Vazou, and R. Jhala, “Flux: Liquid types for rust,” _Proc. ACM Program. Lang._ , vol. 7, no. PLDI, Jun. 2023. [Online]. Available: https://doi.org/10.1145/3591283
  * [28] L. Gäher, M. Sammler, R. Jung, R. Krebbers, and D. Dreyer, “Refinedrust: Towards high-assurance verification of unsafe rust programs,” Rust Verification Workshop, Apr. 2023, accessed: 2023-10-20. [Online]. Available: https://people.mpi-sws.org/~gaeher/slides/refinedrust%5Frw23.pdf
  * [29] M. Sammler, R. Lepigre, R. Krebbers, K. Memarian, D. Dreyer, and D. Garg, “Refinedc: Automating the foundational verification of c code with refined ownership types,” in _Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation_ , ser. PLDI 2021\. New York, NY, USA: Association for Computing Machinery, 2021, p. 158–174. [Online]. Available: https://doi.org/10.1145/3453483.3454036
  * [30] Rust Community, “The ‘bindgen‘ user guide,” https://rust-lang.github.io/rust-bindgen/, Oct. 2023.
  * [31] ——, “cbindgen user guide,” https://github.com/mozilla/cbindgen/blob/master/docs.md, Nov. 2023.
  * [32] ——, “Cxx — safe interop between rust and c++,” https://cxx.rs/, Jan. 2024.
  * [33] ——, “The rust reference,” https://doc.rust-lang.org/reference/, Nov. 2023.
  * [34] ——, “The rustonomicon,” https://doc.rust-lang.org/nomicon/, Oct. 2023\.
  * [35] ——, “Unsafe code guidelines reference,” https://rust-lang.github.io/unsafe-code-guidelines/, Oct. 2023.
  * [36] H. Xu, Z. Chen, M. Sun, Y. Zhou, and M. R. Lyu, “Memory-safety challenge considered solved? an in-depth study with all rust cves,” _ACM Trans. Softw. Eng. Methodol._ , vol. 31, no. 1, Sep. 2021. [Online]. Available: https://doi.org/10.1145/3466642
  * [37] M. Cui, S. Sun, H. Xu, and Y. Zhou, “Is unsafe an achilles’ heel? a comprehensive study of safety requirements in unsafe rust programming,” 2023\. [Online]. Available: https://doi.org/10.48550/arXiv.2011.06171
  * [38] K. R. Fulton, A. Chan, D. Votipka, M. Hicks, and M. L. Mazurek, “Benefits and drawbacks of adopting a secure programming language: Rust as a case study,” in _Proceedings of the Seventeenth USENIX Conference on Usable Privacy and Security_ , ser. SOUPS’21. USA: USENIX Association, 2021. [Online]. Available: https://www.usenix.org/conference/soups2021/presentation/fulton
  * [39] J. W. Creswell, V. L. P. Clark, M. L. Gutmann, and W. E. Hanson, “Advanced mixed methods research designs,” in _Handbook of Mixed Methods in Social and Behavioral Research_ , A. Tashakkori and C. Teddlie, Eds. Thousand Oaks, CA: Sage, 2003, ch. 8, pp. 209–240.
  * [40] M. Q. Patton, _Qualitative evaluation and research methods_. SAGE Publications, inc, 1990.
  * [41] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in _Proceedings of the 40th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, Eds., vol. 202. PMLR, 23–29 Jul 2023, pp. 28 492–28 518. [Online]. Available: https://proceedings.mlr.press/v202/radford23a.html
  * [42] M. B. Miles, A. M. Huberman, and J. Saldaña, _Qualitative data analysis: a methods sourcebook_ , 4th ed. Los Angeles London New Delhi Singapore Washington DC Melbourne: SAGE, 2020.
  * [43] A. S. S. D. GmbH. (2024) Atlas.ti web. [Online]. Available: https://atlasti.com/
  * [44] (2024) Qualtrics. [Online]. Available: https://www.qualtrics.com
  * [45] R. K. Yin, _Case study research: Design and methods_. SAGE Publications, 2009, vol. 5.
  * [46] “Relevantid®.” [Online]. Available: https://www.imperium.com/relevantid/
  * [47] W. Liu, “Introducing recaptcha v3: the new way to stop bots,” Oct. 2018. [Online]. Available: https://developers.google.com/search/blog/2018/10/introducing-recaptcha-v3-new-way-to
  * [48] K. Spiel, O. L. Haimson, and D. Lottridge, “How to do better with gender on surveys: a guide for hci researchers,” _Interactions_ , vol. 26, no. 4, p. 62–65, Jun. 2019. [Online]. Available: https://doi.org/10.1145/3338283
  * [49] J. Yanovski, H.-H. Dang, R. Jung, and D. Dreyer, “Ghostcell: Separating permissions from data in rust,” _Proc. ACM Program. Lang._ , vol. 5, no. ICFP, Aug. 2021. [Online]. Available: https://doi.org/10.1145/3473597
  * [50] W. M. McKeeman, “Differential testing for software,” _Digit. Tech. J._ , vol. 10, no. 1, pp. 100–107, 1998. [Online]. Available: https://www.hpl.hp.com/hpjournal/dtj/vol10num1/vol10num1art9.pdf
  * [51] R. O’Callahan, C. Jones, N. Froyd, K. Huey, A. Noll, and N. Partush, “Engineering record and replay for deployability,” in _Proceedings of the 2017 USENIX Conference on Usenix Annual Technical Conference_ , ser. USENIX ATC ’17. USA: USENIX Association, 2017, p. 377–389.
  * [52] C. Durham, “Document current justification for not requiring recursive reference validity (in particular, &mut uninit not being immediate ub),” https://github.com/rust-lang/unsafe-code-guidelines/issues/346, Jul. 2022.
  * [53] T. Chiovoloni, “Storing an object as &header, but reading the data past the end of the header,” https://github.com/rust-lang/unsafe-code-guidelines/issues/256, Nov. 2020\.
  * [54] M. Brænne, R. Dejanovska, G. Horváth, D. Gribenko, and L. Versari, “[rfc] lifetime annotations for c++,” Apr. 2022. [Online]. Available: https://discourse.llvm.org/t/rfc-lifetime-annotations-for-c/61377