#  BTSbot: A Multi-input Convolutional Neural Network to Automate and Expedite
Bright Transient Identification for the Zwicky Transient Facility

Nabeel Rehemtulla  Adam A. Miller  Michael W. Coughlin  Theophile Jegou du Laz
on behalf of ZTF

###### Abstract

The Bright Transient Survey (BTS) relies on visual inspection (“scanning”) to
select sources for accomplishing its mission of spectroscopically classifying
all bright extragalactic transients found by the Zwicky Transient Facility
(ZTF). We present BTSbot, a multi-input convolutional neural network, which
provides a bright transient score to individual ZTF detections using their
image data and 14 extracted features. BTSbot eliminates the need for scanning
by automatically identifying and requesting follow-up observations of new
bright ($m\,<18.5\,\mathrm{mag}$) transient candidates. BTSbot outperforms BTS
scanners in terms of completeness (99% vs. 95%) and identification speed (on
average, 7.4 hours quicker).

Machine Learning, ICML

##  1 Introduction

Large, wide-field surveys have recently revolutionized time-domain astronomy
by repeatedly imaging the entire night sky. These surveys produce staggering
amounts of data every night, an influx that demands the adoption of machine
learning (ML) techniques. ML models have been applied to a variety of tasks in
astronomy including real/bogus classification (e.g., Bloom et al., 2012; Brink
et al., 2013), photometric transient classification (e.g., Villar et al.,
2020), photometric redshift estimation (e.g., Carrasco Kind & Brunner, 2013),
and many others. For the most part, ML models in astronomy perform their task
using a small number of extracted numeric features. Limiting these models to
extracted features ignores potentially valuable information present in the
associated images. A comparatively small number of convolutional neural
networks (CNNs) have been built that make use of the information embedded in
astronomical images, and they have generally had great success (e.g., Dieleman
et al., 2015; Lanusse et al., 2018; Duev et al., 2019). CNNs are particularly
well suited to astronomy because they can capture properties, like galaxy
morphology, which often remain largely obscured to other image processing
techniques. Only a very small subset of these CNNs are multi-input, meaning
they take in images and input of another type, like extracted features
(Carrasco-Davis et al., 2021). Multi-input CNNs (MI-CNNs) have more and varied
information to draw from compared to analogous single-input models. Here, we
present a new MI-CNN for bright transient identification.

The Bright Transient Survey (BTS; Fremling et al., 2020; Perley et al., 2020)
aims to classify all extragalactic transients with $m<18.5\,\mathrm{mag}$ in
$g$ or $r$ band from the public alert stream of the Zwicky Transient Facility
(ZTF; Bellm et al., 2019). Every night, experts inspect candidate BTS sources,
a process called “scanning,” and select the bright extragalactic
transients111We use “supernova” interchangeably with “extragalactic
transient.” These are not equivalent, but it simplifies the prose. for
spectroscopic observation and classification. There are typically $\sim$50 BTS
candidates per night, of which $\sim$10 are real bright supernovae (SNe), with
the others being mostly dim SNe, active galactic nuclei (AGN), and cataclysmic
variables (CVs). Since its origin, BTS has maintained near-perfect
completeness of relevant sources and has rapidly and publicly released their
classifications: a monumental service to the community. The BTS sample enables
an enormous amount of science including some of the largest SN population
studies ever conducted. The MI-CNN introduced here, dubbed BTSbot, is built to
automate scanning for BTS by performing binary classification (bright SN / not
bright SN) on ZTF alert packets.

##  2 Model scope, architecture, and training

Figure 1: Diagram of BTSbot, a multi-input convolutional neural network,
showing the layers that the input images and extracted features pass through
before being rejoined and producing the score for bright extragalactic
transient binary classification.

Our MI-CNN, BTSbot, automates BTS scanning by assigning each individual ZTF
alert packet a bright transient score. Alerts from real bright SNe will have
high scores, allowing us to automatically select them from the pool of BTS
candidates for follow-up and classification. Further, our model identifies
sources and requests follow-up before a human can, helping to maintain the
survey’s near-perfect completeness by obtaining a spectrum before the
transient fades beyond detection.

Figure 1 shows how input is fed into BTSbot and how the information from the
extracted features and images is combined to produce the output. BTSbot
contains three main components: (1) The convolutional branch processes the
science, reference, and difference images as a three-channel image through a
VGG-like architecture (Simonyan & Zisserman, 2014) followed by flattening. (2)
The fully-connected branch processes the extracted features through two dense
layers. (3) The combined section concatenates the output of the two branches
and passes it through two more fully-connected layers; the second of which
produces the final output using a softmax activation function. The output is a
unit-interval score where higher scores represent increased confidence that
the source in the input alert packet is, or will become, a bright
extragalactic transient.

The choice of a MI-CNN is motivated by the fact that the images and the
extracted features provide different, valuable information for performing our
task. For example, the features distpsnr1 and sgscore1 respectively represent
the angular distance to and the star-galaxy score (Tachibana & Miller, 2018)
of the Pan-STARRS1 (PS1; Kaiser et al., 2002) cataloged source nearest to this
ZTF source. While the new SNe are not present in PS1, their host galaxies
often are. Thus, alerts from bright SNe tend to have moderate distpsnr1 and
small sgscore1 values, indicating a galaxy projected nearby to the source. In
contrast, AGN and CVs will typically be cataloged in PS1 and thus have
distpsnr1 very near to zero. The images also provide important information
following a similar heuristic. Bright SNe tend to be associated with prominent
off-center extended sources (their host galaxies); faint SNe tend to have less
prominent host galaxies because they tend to be farther away; AGN will appear
as exactly centered extended sources; CVs will often appear surrounded by many
bright point sources because they tend to occur near dense star fields. A MI-
CNN is able to pool information from all input types and consider them
together when making a prediction. Where a single-input CNN might fail due to
a lack of discriminating information, a MI-CNN leverages additional and
distinct information when making its prediction. For example, a very faint
alert that shows next to no features in its images can be identified as an AGN
by a MI-CNN because AGN often have a large number of previous detections at
their location (represented by ndethist), information a single-input CNN could
not have.

Given the scope and architecture of BTSbot we encounter a number of
challenges. First, we are requiring the model to learn multiple non-trivial
separations. BTSbot must learn to separate SNe from other sources without
using distance information because it is not known a priori. It must also
learn to identify bright SNe from a single alert irrespective of the SN’s
current phase. Early in its rise or late in its fade, a bright SN can appear
very similar to a near-peak dim SN. This is related to the second
complication, which is that BTSbot uses no time-series information. Although
light curves contain crucial information for evaluating a source as a bright
SN or not, we make the choice to omit time-series information, in part,
because it would introduce complexity and, likely, noise. There is no
established method for representing partial light curves of the wide variety
our model encounters in a way that is fit for input into a neural network.
There has been a great deal of work to accomplish this for SNe alone (Villar
et al., 2020), but these methods are not applicable to all the types of
sources that our model encounters. Further, omitting time-series information
offers BTSbot an advantage over light curve-based models. With its alert-based
architecture, BTSbot can identify a bright SN from its very first detection,
rather than preferring multiple detections to begin constraining a light
curve. We expect that a similar model with light curve information would
identify bright SNe as or less quickly than BTSbot does, potentially hurting
BTS’s completeness. Section 4.1 illustrates how prompt and automatic
identification of SNe with BTSbot can aid in uncovering poorly understood SN
physics. To our knowledge, no other CNNs exist that simultaneously filter out
non-SNe and predict a SN’s future behavior given a single snapshot in time.

Figure 2: Stacked bar chart showing the distribution of classification
outcomes as a function of the input alerts’ PSF magnitude. The highlighted
region to the left is most important because it marks the alerts which BTS
scanners use to trigger spectroscopic observations. Performance of alerts with
$m<18.5\,\mathrm{mag}$ is excellent and misclassifications are mostly in the
dimmest bin.

Figure 3: BTSbot performance when classifying sources as bright supernovae
given at least one alert with high score. Left panel: completeness (navy) and
purity (orange) as a function of peak magnitude and respective averages
(dashed lines). BTSbot produces a sample that is $99.1\%$ complete and
$90.7\%$ pure overall. The bright limit is noisy due to the small number of
sources in those bins. Right panel: distribution of delay between BTSbot and
BTS scanner identification times with median at -0.31 days, indicating that,
on average, BTSbot identifies sources about 7 hours quicker than BTS scanners.

###  2.1 Training

Solving this complex classification task requires a significant training set.
Having run since 2018, BTS has now amassed the largest set of public SNe
classifications ever. The size of this labeled data set enables the
construction of BTSbot. Bright SNe classified by BTS populate the positive
class, while AGN, CVs, and dim SNe rejected by BTS constitute the negative
class. In total, we have 561,000 alerts ($\sim$44% of which belong to the
positive class) from 14,348 sources ($\sim$33% of which belong to the positive
class). This difference in class distribution between alerts and sources stems
from some sources having many more alerts than others: AGN can have thousands,
bright SNe typically have dozens, and dim SNe can have as little as a few.
Training on every alert would result in some types of sources being over-
represented. To remedy this we define a hyperparameter called
$N_{\textrm{max}}$, the maximum number of alerts included per source. We find
$N_{\textrm{max}}=60$ to be optimal; it balances between thinning extra alerts
from sources like AGN and maximizing the training set size. We also weight
contributions to the loss function by the relative size of each class to
mitigate the effects of class imbalance.

BTSbot is trained with mostly standard hyperparameters. We adopt the Adam
optimizer (Kingma & Ba, 2014) and the binary cross-entropy loss function. We
perform a number of Bayesian hyperparameter searches with the Weights and
Biases platform (Biewald, 2020) to optimize our choice of batch size, Adam
parameters, $N_{\textrm{max}}$, which extracted features to include, and more.
We also employ data augmentation to the image cutouts like random rotations of
0∘, 90∘, 180∘, and 270∘ and random horizontal and vertical flipping. These
help prevent overfitting and ensure that BTSbot is invariant to these
transformations.

We employ a standard 81/9/10% train/validation/test split. We also prevent
sources from having alerts in multiple splits. This prevents different splits
from containing extremely similar data, which would introduce bias.

##  3 Model performance

The trained model is run on the full validation split (i.e. no
$N_{\textrm{max}}$ thinning) to create unbiased performance diagnostics.

Figure 2 illustrates classification outcomes as a function of PSF magnitude.
The highlight marks $m<18.5\,\mathrm{mag}$, the BTS magnitude threshold. The
alerts within the highlight are especially important to classify correctly
because they influence whether or not a spectrum of the source is to be
collected. In that region, misclassifications are very limited and mostly in
the dimmest bin. For $m>18.5\,\mathrm{mag}$, there are more misclassifications
and many more alerts overall.

Ultimately, alert-based metrics are not perfectly representative of the
model’s real-world performance; source-based classification is more relevant.
The chosen metrics must consider that the model has multiple chances to
correctly (or incorrectly) classify a source. To this end, we define a
“policy” that maps the real-time history of a source’s scores to a source-
based classification. For simplicity, we put aside policy optimization for now
and only consider one naive policy with the impression that we could improve
performance with a more sophisticated choice of policy.

Our naive policy is named gt1; it classifies a source as a bright SN once it
has one alert with score $\geq 0.5$. The left panel of Figure 3 shows the
purity and completeness of the sample produced when following this policy. We
observe that gt1 yields a sample with $99.1\%$ completeness and $90.7\%$
purity overall. BTS scanners produce a sample that is roughly $95\%$ in both
completeness and purity (Perley et al., 2020). In tests with other more
conservative policies like gt3 (defined analogously to gt1), we observe that
purity is increased at the cost of completeness. We favor gt1 because it
maximizes completeness, BTS’s highest priority. The right panel of Figure 3
compares the time between when BTSbot and the BTS scanners identified some
source. Here, negative numbers represent that BTSbot identified the bright SN
before the scanners and positive numbers represent the opposite. The median is
-0.31 days, indicating that BTSbot expedites the identification of bright SNe
by, on average, 7.44 hours. Expediting identification by several hours will
frequently mean observing the source an entire night earlier, thus yielding an
improvement of $\sim$24 hours in practice. With gt1, BTSbot outperforms human
scanners in completeness and speed, while making only a small compromise with
slightly lower purity.

##  4 Real-time, real-world usage

BTSbot has been integrated into Fritz, the first-party ZTF alert-broker and a
SkyPortal instance (Coughlin et al., 2023), and is currently posting scores to
all new ZTF alert packets. Alert packets are created and augmented with a
bright transient scores just minutes after the observations are taken with
ZTF. During the night, a tool we call autoscan checks for new BTS candidates
every half hour and immediately saves those that pass gt1. autoscan will soon
also simultaneously request follow-up for passing sources. With this, we can
monitor the model’s real-time, real-world performance and note any frequent
misclassifications. We have been awarded a large observing program for the
2023B semester with the SEDMv2 spectrograph on the Kitt Peak 2.1 m telescope.
With this time, we will build a spectroscopic sample that represents the
model’s unbiased performance. The resulting performance metrics will be
invaluable when considering adaptations of this MI-CNN, e.g., to specific SN
sub-types or exotic transients like kilonovae.

###  4.1 SN 2023ixf & Rapid follow-up with BTSbot

We take SN 2023ixf, the recent Type-II supernova in M101, as an example
illustrative of the additional discovery potential of an alert-based model
like BTSbot. SN 2023ixf was reported to TNS by Koichi Itagaki at 14:42 PDT on
May 19th (Itagaki, 2023). The earliest published spectrum was collected by
Daniel Perley less than an hour later with the SPRAT spectrograph (Perley et
al., 2023). About 14 hours before the first TNS report, SN 2023ixf was
detected by ZTF, and, just minutes later, this alert packet was assigned a
bright transient score of 0.840 by BTSbot. If it was in-place at the time,
autoscan would have identified this new source passing gt1 at about 01:00 PDT
and requested a spectrum from one of the numerous robotic spectrographs
associated with ZTF and BTS, e.g., SEDM. At this point, there is about half of
the night remaining at SEDM’s location, plenty of time for a spectrum to be
collected. Even if the spectrum is collected at the end of the night
($\sim$05:00 PDT), this represents a $\sim$10 hour speed-up over the otherwise
earliest spectrum. In this example, BTSbot and autoscan probe the early,
rapidly-evolving explosion physics mostly unavailable to traditional
triggering methods. This would not be possible with light curve-based models,
which typically require using the source’s evolution over multiple days to
identify it as a transient. This extremely rapid follow-up is enabled by the
alert-based architecture of BTSbot.

## Software and Data

Code relating to BTSbot and autoscan are made public at
https://github.com/nabeelre/BTSbot. In their development, we use Astropy
(Astropy Collaboration et al., 2013, 2018), cron (Reznick, 1993), Keras
(Chollet et al., 2015), Matplotlib (Hunter, 2007), NumPy (Harris et al.,
2020), pandas (pandas development team, 2020; Wes McKinney, 2010), penquins
(Duev et al., 2021), scikit-learn (Pedregosa et al., 2011), and Tensorflow
(Abadi et al., 2015).

## Acknowledgements

The material contained in this document is based upon work supported by a
National Aeronautics and Space Administration (NASA) grant or cooperative
agreement. Any opinions, findings, conclusions, or recommendations expressed
in this material are those of the author and do not necessarily reflect the
views of NASA. This work was supported through a NASA grant awarded to the
Illinois/NASA Space Grant Consortium.

M. W. Coughlin acknowledges support from the National Science Foundation with
grant numbers PHY-2010970 and OAC-2117997.

Based on observations obtained with the Samuel Oschin Telescope 48-inch and
the 60-inch Telescope at the Palomar Observatory as part of the Zwicky
Transient Facility project. Major funding has been provided by the U.S
National Science Foundation under Grant No. AST-1440341 and by the ZTF partner
institutions: the California Institute of Technology, the Oskar Klein Centre,
the Weizmann Institute of Science, the University of Maryland, the University
of Washington, Deutsches Elektronen-Synchrotron, the University of Wisconsin-
Milwaukee, and the TANGO Program of the University System of Taiwan.

## References

  * Abadi et al. (2015) Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., and Zheng, X.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.  URL https://www.tensorflow.org/.  Software available from tensorflow.org.
  * Astropy Collaboration et al. (2013) Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., Greenfield, P., Droettboom, M., Bray, E., Aldcroft, T., Davis, M., Ginsburg, A., Price-Whelan, A. M., Kerzendorf, W. E., Conley, A., Crighton, N., Barbary, K., Muna, D., Ferguson, H., Grollier, F., Parikh, M. M., Nair, P. H., Unther, H. M., Deil, C., Woillez, J., Conseil, S., Kramer, R., Turner, J. E. H., Singer, L., Fox, R., Weaver, B. A., Zabalza, V., Edwards, Z. I., Azalee Bostroem, K., Burke, D. J., Casey, A. R., Crawford, S. M., Dencheva, N., Ely, J., Jenness, T., Labrie, K., Lim, P. L., Pierfederici, F., Pontzen, A., Ptak, A., Refsdal, B., Servillat, M., and Streicher, O.  Astropy: A community Python package for astronomy.  _LABEL:@ jnlA&A_, 558:A33, October 2013.  doi: 10.1051/0004-6361/201322068.
  * Astropy Collaboration et al. (2018) Astropy Collaboration, Price-Whelan, A. M., Sipőcz, B. M., Günther, H. M., Lim, P. L., Crawford, S. M., Conseil, S., Shupe, D. L., Craig, M. W., Dencheva, N., Ginsburg, A., Vand erPlas, J. T., Bradley, L. D., Pérez-Suárez, D., de Val-Borro, M., Aldcroft, T. L., Cruz, K. L., Robitaille, T. P., Tollerud, E. J., Ardelean, C., Babej, T., Bach, Y. P., Bachetti, M., Bakanov, A. V., Bamford, S. P., Barentsen, G., Barmby, P., Baumbach, A., Berry, K. L., Biscani, F., Boquien, M., Bostroem, K. A., Bouma, L. G., Brammer, G. B., Bray, E. M., Breytenbach, H., Buddelmeijer, H., Burke, D. J., Calderone, G., Cano Rodríguez, J. L., Cara, M., Cardoso, J. V. M., Cheedella, S., Copin, Y., Corrales, L., Crichton, D., D’Avella, D., Deil, C., Depagne, É., Dietrich, J. P., Donath, A., Droettboom, M., Earl, N., Erben, T., Fabbro, S., Ferreira, L. A., Finethy, T., Fox, R. T., Garrison, L. H., Gibbons, S. L. J., Goldstein, D. A., Gommers, R., Greco, J. P., Greenfield, P., Groener, A. M., Grollier, F., Hagen, A., Hirst, P., Homeier, D., Horton, A. J., Hosseinzadeh, G., Hu, L., Hunkeler, J. S., Ivezić, Ž., Jain, A., Jenness, T., Kanarek, G., Kendrew, S., Kern, N. S., Kerzendorf, W. E., Khvalko, A., King, J., Kirkby, D., Kulkarni, A. M., Kumar, A., Lee, A., Lenz, D., Littlefair, S. P., Ma, Z., Macleod, D. M., Mastropietro, M., McCully, C., Montagnac, S., Morris, B. M., Mueller, M., Mumford, S. J., Muna, D., Murphy, N. A., Nelson, S., Nguyen, G. H., Ninan, J. P., Nöthe, M., Ogaz, S., Oh, S., Parejko, J. K., Parley, N., Pascual, S., Patil, R., Patil, A. A., Plunkett, A. L., Prochaska, J. X., Rastogi, T., Reddy Janga, V., Sabater, J., Sakurikar, P., Seifert, M., Sherbert, L. E., Sherwood-Taylor, H., Shih, A. Y., Sick, J., Silbiger, M. T., Singanamalla, S., Singer, L. P., Sladen, P. H., Sooley, K. A., Sornarajah, S., Streicher, O., Teuben, P., Thomas, S. W., Tremblay, G. R., Turner, J. E. H., Terrón, V., van Kerkwijk, M. H., de la Vega, A., Watkins, L. L., Weaver, B. A., Whitmore, J. B., Woillez, J., Zabalza, V., and Astropy Contributors.  The Astropy Project: Building an Open-science Project and Status of the v2.0 Core Package.  _AJ_ , 156(3):123, September 2018.  doi: 10.3847/1538-3881/aabc4f.
  * Bellm et al. (2019) Bellm, E. C., Kulkarni, S. R., Graham, M. J., Dekany, R., Smith, R. M., Riddle, R., Masci, F. J., Helou, G., Prince, T. A., Adams, S. M., Barbarino, C., Barlow, T., Bauer, J., Beck, R., Belicki, J., Biswas, R., Blagorodnova, N., Bodewits, D., Bolin, B., Brinnel, V., Brooke, T., Bue, B., Bulla, M., Burruss, R., Cenko, S. B., Chang, C.-K., Connolly, A., Coughlin, M., Cromer, J., Cunningham, V., De, K., Delacroix, A., Desai, V., Duev, D. A., Eadie, G., Farnham, T. L., Feeney, M., Feindt, U., Flynn, D., Franckowiak, A., Frederick, S., Fremling, C., Gal-Yam, A., Gezari, S., Giomi, M., Goldstein, D. A., Golkhou, V. Z., Goobar, A., Groom, S., Hacopians, E., Hale, D., Henning, J., Ho, A. Y. Q., Hover, D., Howell, J., Hung, T., Huppenkothen, D., Imel, D., Ip, W.-H., Ivezić, Ž., Jackson, E., Jones, L., Juric, M., Kasliwal, M. M., Kaspi, S., Kaye, S., Kelley, M. S. P., Kowalski, M., Kramer, E., Kupfer, T., Landry, W., Laher, R. R., Lee, C.-D., Lin, H. W., Lin, Z.-Y., Lunnan, R., Giomi, M., Mahabal, A., Mao, P., Miller, A. A., Monkewitz, S., Murphy, P., Ngeow, C.-C., Nordin, J., Nugent, P., Ofek, E., Patterson, M. T., Penprase, B., Porter, M., Rauch, L., Rebbapragada, U., Reiley, D., Rigault, M., Rodriguez, H., van Roestel, J., Rusholme, B., van Santen, J., Schulze, S., Shupe, D. L., Singer, L. P., Soumagnac, M. T., Stein, R., Surace, J., Sollerman, J., Szkody, P., Taddia, F., Terek, S., Van Sistine, A., van Velzen, S., Vestrand, W. T., Walters, R., Ward, C., Ye, Q.-Z., Yu, P.-C., Yan, L., and Zolkower, J.  The Zwicky Transient Facility: System Overview, Performance, and First Results.  _PASP_ , 131(995):018002, January 2019.  doi: 10.1088/1538-3873/aaecbe.
  * Biewald (2020) Biewald, L.  Experiment tracking with weights and biases, 2020.  URL https://www.wandb.com/.  Software available from wandb.com.
  * Bloom et al. (2012) Bloom, J. S., Richards, J. W., Nugent, P. E., Quimby, R. M., Kasliwal, M. M., Starr, D. L., Poznanski, D., Ofek, E. O., Cenko, S. B., Butler, N. R., Kulkarni, S. R., Gal-Yam, A., and Law, N.  Automating Discovery and Classification of Transients and Variable Stars in the Synoptic Survey Era.  _PASP_ , 124(921):1175, November 2012.  doi: 10.1086/668468.
  * Brink et al. (2013) Brink, H., Richards, J. W., Poznanski, D., Bloom, J. S., Rice, J., Negahban, S., and Wainwright, M.  Using machine learning for discovery in synoptic survey imaging data.  _MNRAS_ , 435(2):1047–1060, October 2013.  doi: 10.1093/mnras/stt1306.
  * Carrasco-Davis et al. (2021) Carrasco-Davis, R., Reyes, E., Valenzuela, C., Förster, F., Estévez, P. A., Pignata, G., Bauer, F. E., Reyes, I., Sánchez-Sáez, P., Cabrera-Vives, G., Eyheramendy, S., Catelan, M., Arredondo, J., Castillo-Navarrete, E., Rodríguez-Mancini, D., Ruz-Mieres, D., Moya, A., Sabatini-Gacitúa, L., Sepúlveda-Cobo, C., Mahabal, A. A., Silva-Farfán, J., Camacho-Iñiguez, E., and Galbany, L.  Alert Classification for the ALeRCE Broker System: The Real-time Stamp Classifier.  _AJ_ , 162(6):231, December 2021.  doi: 10.3847/1538-3881/ac0ef1.
  * Carrasco Kind & Brunner (2013) Carrasco Kind, M. and Brunner, R. J.  TPZ: photometric redshift PDFs and ancillary information by using prediction trees and random forests.  _MNRAS_ , 432(2):1483–1501, June 2013.  doi: 10.1093/mnras/stt574.
  * Chollet et al. (2015) Chollet, F. et al.  Keras.  https://keras.io, 2015.
  * Coughlin et al. (2023) Coughlin, M. W., Bloom, J. S., Nir, G., Antier, S., Jegou du Laz, T., van der Walt, S., Crellin-Quick, A., Culino, T., Duev, D. A., Goldstein, D. A., Healy, B. F., Karambelkar, V., Lilleboe, J., Shin, K. M., Singer, L. P., Ahumada, T., Anand, S., Bellm, E. C., Dekany, R., Graham, M. J., Kasliwal, M. M., Kostadinova, I., Weizmann Kiendrebeogo, R., Kulkarni, S. R., Jenkins, S., LeBaron, N., Mahabal, A. A., Neill, J. D., Parazin, B., Peloton, J., Perley, D. A., Riddle, R., Rusholme, B., van Santen, J., Sollerman, J., Stein, R., Turpin, D., Wold, A., Amat, C., Bonnefon, A., Bonnefoy, A., Flament, M., Kerkow, F., Kishore, S., Jani, S., Mahanty, S. K., Liu, C., Llinares, L., Makarison, J., Olliéric, A., Perez, I., Pont, L., and Sharma, V.  A data science platform to enable time-domain astronomy.  _arXiv e-prints_ , art. arXiv:2305.00108, April 2023.  doi: 10.48550/arXiv.2305.00108.
  * Dieleman et al. (2015) Dieleman, S., Willett, K. W., and Dambre, J.  Rotation-invariant convolutional neural networks for galaxy morphology prediction.  _MNRAS_ , 450(2):1441–1459, June 2015.  doi: 10.1093/mnras/stv632.
  * Duev et al. (2021) Duev, D., Shin, K. M., and Singer, L.  dmitryduev/penquins: a python client for dmitryduev/kowalski, November 2021.  URL https://doi.org/10.5281/zenodo.5651471.
  * Duev et al. (2019) Duev, D. A., Mahabal, A., Masci, F. J., Graham, M. J., Rusholme, B., Walters, R., Karmarkar, I., Frederick, S., Kasliwal, M. M., Rebbapragada, U., and Ward, C.  Real-bogus classification for the Zwicky Transient Facility using deep learning.  _MNRAS_ , 489(3):3582–3590, November 2019.  doi: 10.1093/mnras/stz2357.
  * Fremling et al. (2020) Fremling, C., Miller, A. A., Sharma, Y., Dugas, A., Perley, D. A., Taggart, K., Sollerman, J., Goobar, A., Graham, M. L., Neill, J. D., Nordin, J., Rigault, M., Walters, R., Andreoni, I., Bagdasaryan, A., Belicki, J., Cannella, C., Bellm, E. C., Cenko, S. B., De, K., Dekany, R., Frederick, S., Golkhou, V. Z., Graham, M. J., Helou, G., Ho, A. Y. Q., Kasliwal, M. M., Kupfer, T., Laher, R. R., Mahabal, A., Masci, F. J., Riddle, R., Rusholme, B., Schulze, S., Shupe, D. L., Smith, R. M., van Velzen, S., Yan, L., Yao, Y., Zhuang, Z., and Kulkarni, S. R.  The Zwicky Transient Facility Bright Transient Survey. I. Spectroscopic Classification and the Redshift Completeness of Local Galaxy Catalogs.  _ApJ_ , 895(1):32, May 2020.  doi: 10.3847/1538-4357/ab8943.
  * Harris et al. (2020) Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M. H., Brett, M., Haldane, A., Fernández del Río, J., Wiebe, M., Peterson, P., Gérard-Marchant, P., Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H., Gohlke, C., and Oliphant, T. E.  Array programming with NumPy.  _Nature_ , 585:357–362, 2020.  doi: 10.1038/s41586-020-2649-2.
  * Hunter (2007) Hunter, J. D.  Matplotlib: A 2D Graphics Environment.  _Computing in Science and Engineering_ , 9:90–95, May 2007\.  doi: 10.1109/MCSE.2007.55.
  * Itagaki (2023) Itagaki, K.  Transient Discovery Report for 2023-05-19.  _Transient Name Server Discovery Report_ , 2023-1158:1, May 2023.
  * Kaiser et al. (2002) Kaiser, N., Aussel, H., Burke, B. E., Boesgaard, H., Chambers, K., Chun, M. R., Heasley, J. N., Hodapp, K.-W., Hunt, B., Jedicke, R., Jewitt, D., Kudritzki, R., Luppino, G. A., Maberry, M., Magnier, E., Monet, D. G., Onaka, P. M., Pickles, A. J., Rhoads, P. H. H., Simon, T., Szalay, A., Szapudi, I., Tholen, D. J., Tonry, J. L., Waterson, M., and Wick, J.  Pan-STARRS: A Large Synoptic Survey Telescope Array.  In Tyson, J. A. and Wolff, S. (eds.), _Survey and Other Telescope Technologies and Discoveries_ , volume 4836 of _Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series_ , pp. 154–164, December 2002.  doi: 10.1117/12.457365.
  * Kingma & Ba (2014) Kingma, D. P. and Ba, J.  Adam: A Method for Stochastic Optimization.  _arXiv e-prints_ , art. arXiv:1412.6980, December 2014.  doi: 10.48550/arXiv.1412.6980.
  * Lanusse et al. (2018) Lanusse, F., Ma, Q., Li, N., Collett, T. E., Li, C.-L., Ravanbakhsh, S., Mandelbaum, R., and Póczos, B.  CMU DeepLens: deep learning for automatic image-based galaxy-galaxy strong lens finding.  _MNRAS_ , 473(3):3895–3906, January 2018.  doi: 10.1093/mnras/stx1665.
  * pandas development team (2020) pandas development team, T.  pandas-dev/pandas: Pandas, February 2020.  URL https://doi.org/10.5281/zenodo.3509134.
  * Pedregosa et al. (2011) Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011\.
  * Perley et al. (2020) Perley, D. A., Fremling, C., Sollerman, J., Miller, A. A., Dahiwale, A. S., Sharma, Y., Bellm, E. C., Biswas, R., Brink, T. G., Bruch, R. J., De, K., Dekany, R., Drake, A. J., Duev, D. A., Filippenko, A. V., Gal-Yam, A., Goobar, A., Graham, M. J., Graham, M. L., Ho, A. Y. Q., Irani, I., Kasliwal, M. M., Kim, Y.-L., Kulkarni, S. R., Mahabal, A., Masci, F. J., Modak, S., Neill, J. D., Nordin, J., Riddle, R. L., Soumagnac, M. T., Strotjohann, N. L., Schulze, S., Taggart, K., Tzanidakis, A., Walters, R. S., and Yan, L.  The Zwicky Transient Facility Bright Transient Survey. II. A Public Statistical Sample for Exploring Supernova Demographics.  _ApJ_ , 904(1):35, November 2020.  doi: 10.3847/1538-4357/abbd98.
  * Perley et al. (2023) Perley, D. A., Gal-Yam, A., Irani, I., and Zimmerman, E.  LT Classification of SN 2023ixf as a Type II Supernova in M101.  _Transient Name Server AstroNote_ , 119:1, May 2023.
  * Reznick (1993) Reznick, L.  Using cron and crontab.  _Sys Admin_ , 2(4):29–32, 1993.
  * Simonyan & Zisserman (2014) Simonyan, K. and Zisserman, A.  Very Deep Convolutional Networks for Large-Scale Image Recognition.  _arXiv e-prints_ , art. arXiv:1409.1556, September 2014.  doi: 10.48550/arXiv.1409.1556.
  * Tachibana & Miller (2018) Tachibana, Y. and Miller, A. A.  A Morphological Classification Model to Identify Unresolved PanSTARRS1 Sources: Application in the ZTF Real-time Pipeline.  _PASP_ , 130(994):128001, December 2018.  doi: 10.1088/1538-3873/aae3d9.
  * Villar et al. (2020) Villar, V. A., Hosseinzadeh, G., Berger, E., Ntampaka, M., Jones, D. O., Challis, P., Chornock, R., Drout, M. R., Foley, R. J., Kirshner, R. P., Lunnan, R., Margutti, R., Milisavljevic, D., Sanders, N., Pan, Y.-C., Rest, A., Scolnic, D. M., Magnier, E., Metcalfe, N., Wainscoat, R., and Waters, C.  SuperRAENN: A Semisupervised Supernova Photometric Classification Pipeline Trained on Pan-STARRS1 Medium-Deep Survey Supernovae.  _ApJ_ , 905(2):94, December 2020.  doi: 10.3847/1538-4357/abc6fd.
  * Wes McKinney (2010) Wes McKinney.  Data Structures for Statistical Computing in Python.  In Stéfan van der Walt and Jarrod Millman (eds.), _Proceedings of the 9th Python in Science Conference_ , pp. 56 – 61, 2010.  doi: 10.25080/Majora-92bf1922-00a.


Assessing language model deployment with risk cards.
arXiv preprint arXiv:2303.18190.

Dobbe, RIJ.

System Safety and Artificial Intelligence System
safety and artificial intelligence.
JB. Bullock . (), The Oxford
Handbook of AI Governance. The oxford handbook of ai governance.
Oxford University Press.

Etzioni, O.

How to know if artificial intelligence is about to destroy
civilization. How to know if artificial intelligence is about to destroy
MIT Technology Review.

EU High-Level Expert Group on AI

EU High-Level Expert Group on AI

EU High-Level Expert Group on AI.

Assessment List for Trustworthy Artificial Intelligence
(ALTAI) for self-assessment. Assessment list for trustworthy artificial
intelligence (ALTAI) for self-assessment.

European Foresight Platform

European Foresight Platform

European Foresight Platform.

Cross-Impact Analysis. Cross-impact analysis.

Fei, N.
, Lu, Z.
, Gao, Y.
, Yang, G.
, Huo, Y.
, Wen, J.
Wen, JR.

Towards artificial general intelligence via a multimodal
foundation model Towards artificial general intelligence via a multimodal
foundation model.
Nature Communications1313094.

Ganguli, D.
, Lovitt, L.
, Kernion, J.
, Askell, A.
, Bai, Y.
, Kadavath, S.
Clark, J.

Red Teaming Language Models to Reduce Harms: Methods,
Scaling, Behaviors, and Lessons Learned Red teaming language models to
reduce harms: Methods, scaling, behaviors, and lessons learned.
arXiv preprint arXiv:2209.07858.

Gebru, T.
, Morgenstern, J.
, Vecchione, B.
, Vaughan, JW.
, Wallach, H.
, Iii, HD.
Crawford, K.

Datasheets for Datasets Datasheets for
Communications of the ACM641286–92.

Goertzel, B.

Who coined the term “AGI”? Who coined the term

Good Judgment

Good Judgment

Good Judgment.

Delphineo. Delphineo.

Google and Alphabet Vulnerability Reward Program (VRP)
Rules. Google and Alphabet vulnerability reward program (VRP) rules.

Gordon, TJ.

Cross-Impact Method. Cross-impact method.

, Salvatier
, Dafoe
, Zhang

Grace, K.
, Salvatier, J.
, Dafoe, A.
, Zhang, B.
Evans, O.

When Will AI Exceed Human Performance? Evidence from
AI Experts When will AI exceed human performance? evidence from AI
arXiv preprint arXiv:1705.08807.

, Paradice

Gruetzemacher, R.
, Paradice, D.
Lee, KB.

Forecasting Transformative AI: An Expert Survey
Forecasting transformative AI: An expert survey.
arXiv preprint arXiv:1901.08579.

Gubrud, MA.

Nanotechnology and International Security. Nanotechnology and
international security.

Hazell, J.

Large Language Models Can Be Used To Effectively Scale
Spear Phishing Campaigns Large language models can be used to effectively
scale spear phishing campaigns.
arXiv preprint arXiv:2305.06972.

Hendrycks, D.
Mazeika, M.

X-Risk Analysis for AI Research X-risk analysis for
AI research.
arXiv preprint arXiv:2206.05862.

, Mazeika

Hendrycks, D.
, Mazeika, M.
Woodside, T.

An Overview of Catastrophic AI Risks An overview of
catastrophic AI risks.
arXiv preprint arXiv:2306.12001.

The HFACS Framework. The HFACS framework.

HM Government

HM Government

HM Government.

National AI Strategy. National AI strategy.

Hogarth, I.

We must slow down the race to God-like AI. We must slow
down the race to God-like AI.
Financial Times.

Hsiao, S.
Collins, E.

Try Bard and share your feedback. Try Bard and share your
Google Blog.

Hubinger, E.

How likely is deceptive alignment? How likely is deceptive
AI Alignment Forum.

31010:2019 Risk management — Risk assessment techniques.
31010:2019 Risk management — Risk assessment techniques.

Ishikawa, K.

Guide to quality control Guide to quality control.
Asian Productivity Organization.

31000:2018 Risk management — Guidelines. 31000:2018 Risk
management — Guidelines.

ISO & IEC

ISO & IEC

ISO & IEC.

Guide 51:2014 Safety aspects — Guidelines for their
inclusion in standards. Guide 51:2014 Safety aspects — Guidelines for
their inclusion in standards.

ISO & IEC

ISO & IEC

ISO & IEC.

23894:2023 Information technology — Artificial intelligence
— Guidance on risk management. 23894:2023 Information technology —
Artificial intelligence — Guidance on risk management.

Johnson, JL.

A ten-year Delphi forecast in the electronics
industry A ten-year Delphi forecast in the electronics industry.
Industrial Marketing Management5145–55.

Karnofsky, H.

Some Background on Our Views Regarding Advanced Artificial
Intelligence. Some background on our views regarding advanced artificial
Open Philanthropy Blog.

, Kohli
, Ibrahim
, Bloxwich

Kavukcuoglu, K.
, Kohli, P.
, Ibrahim, L.
, Bloxwich, D.
Brown, S.

How our principles helped define AlphaFold’s release. How
our principles helped define AlphaFold’s release.
Google DeepMind Blog.

Kenton, Z.
, Shah, R.
, Lindner, D.
, Varma, V.
, Krakovna, V.
, Phuong, M.
Catt, E.

Clarifying AI X-risk. Clarifying AI x-risk.
AI Alignment Forum.

Kenton, Z.
, Shah, R.
, Lindner, D.
, Varma, V.
, Krakovna, V.
, Phuong, M.
Catt, E.

Threat Model Literature Review. Threat model literature
AI Alignment Forum.

Khlaaf, H.

Toward Comprehensive Risk Assessments and Assurance of
AI-Based Systems. Toward comprehensive risk assessments and assurance of
AI-based systems.
Trail of Bits.

, Mishkin
, Achiam
, Krueger

Khlaaf, H.
, Mishkin, P.
, Achiam, J.
, Krueger, G.
Brundage, M.

A Hazard Analysis Framework for Code Synthesis Large
Language Models A hazard analysis framework for code synthesis large
language models.
arXiv preprint arXiv:2207.14157.

, Ventura

Kilian, KA.
, Ventura, CJ.
Bailey, MM.

Examining the differential risk from high-level
artificial intelligence and the question of control Examining the
differential risk from high-level artificial intelligence and the question of

, Lee
, Kim
, Park

Kim, J.
, Lee, J.
, Kim, G.
, Park, S.
Jang, D.

A Hybrid Method of Analyzing Patents for Sustainable
Technology Management in Humanoid Robot Industry A hybrid method of
analyzing patents for sustainable technology management in humanoid robot

Klein, E.

The Surprising Thing A.I. Engineers Will Tell You if You
Let Them. The surprising thing A.I. engineers will tell you if you let
The New York Times.

Kokotajlo, D.
Wei, D.

The Main Sources of AI Risk? The main sources of ai risk?
AI Alignment Forum.

Kosow, H.
Gaßner, R.

Methods of Future and Scenario Analysis: Overview, Assessment,
and Selection Criteria. Methods of future and scenario analysis: Overview,
assessment, and selection criteria.
Deutsches Institut für Entwicklungspolitik.

Krakovna, V.
Shah, R.

[Linkpost] Some high-level thoughts on the DeepMind alignment
team's strategy. [linkpost] some high-level thoughts on the DeepMind
alignment team's strategy.
AI Alignment Forum.

Leung, J.

Who will govern artificial intelligence? Learning from the
history of strategic politics in emerging technologies. Who will govern
artificial intelligence? learning from the history of strategic politics in
emerging technologies.
University of Oxford.

Leveson, NG.
Thomas, JP.

STPA handbook. STPA handbook.

Li, J.
Chignell, M.

FMEA-AI: AI fairness impact assessment using
failure mode and effects analysis FMEA-AI: AI fairness impact
assessment using failure mode and effects analysis.
AI and Ethics2837–850.

, Lauta

Liu, HY.
, Lauta, KC.
Maas, MM.

Governing Boring Apocalypses: A new typology of
existential vulnerabilities and exposures for existential risk research
Governing boring apocalypses: A new typology of existential vulnerabilities
and exposures for existential risk research.

Losi, S.

System Dynamics and AI Regulation. System dynamics and AI
Risk Musings.

, Stark
, Wortman Vaughan

Madaio, MA.
, Stark, L.
, Wortman Vaughan, J.
Wallach, H.

AI Fairness Checklist. AI fairness checklist.

Mahler, T.

Regulating Artificial General Intelligence (AGI)
Regulating artificial general intelligence (AGI).
B. Custers  E. Fosch-Villaronga (), Law
and Artificial Intelligence: Regulating AI and Applying AI in Legal
Practice Law and artificial intelligence: Regulating AI and applying AI
in legal practice ( 521–540).

Manheim, D.

Building a Culture of Safety for AI: Perspectives and
Challenges. Building a culture of safety for AI: Perspectives and

McChrystal, SA.
Butrico, A.

Risk: a user's guide Risk: a user's guide.

McConnell, P.
Davies, M.

Safety First – Scenario Analysis under Basel II. Safety
first – scenario analysis under Basel II.

Metz, C.

‘The godfather of A.I.’ leaves Google and warns of danger
ahead. ‘The godfather of A.I.’ leaves Google and warns of danger
The New York Times.

Michael, J.
, Holtzman, A.
, Parrish, A.
, Mueller, A.
, Wang, A.
, Chen, A.
Bowman, SR.

What Do NLP Researchers Believe? Results of the NLP
Community Metasurvey What do NLP researchers believe? results of the
NLP community metasurvey.
arXiv preprint arXiv:2208.12852.

Assessing harm: A guide for tech builders. Assessing harm: A
guide for tech builders.

Harms modeling. Harms modeling.

Responsible AI Impact Assessment Template. Responsible AI
impact assessment template.

, Ahmad
, Brundage
, Krueger

Mishkin, P.
, Ahmad, L.
, Brundage, M.
, Krueger, G.
Sastry, G.

DALL·E 2 Preview - Risks and Limitations. DALL·E 2
preview - risks and limitations.

Mitchell, M.
, Wu, S.
, Zaldivar, A.
, Barnes, P.
, Vasserman, L.
, Hutchinson, B.
Gebru, T.

Model Cards for Model Reporting Model cards for model
FAT* '19: Proceedings of the Conference on Fairness,
Accountability, and Transparency FAT* '19: Proceedings of the Conference
on Fairness, Accountability, and Transparency ( 220–229).

Morgan, MG.

Use (and abuse) of expert elicitation in support of
decision making for public policy Use (and abuse) of expert elicitation in
support of decision making for public policy.
Proceedings of the National Academy of

R. Müller

R. Müller

Müller, R.
Drax, C.

Fundamentals and Structure of Safety Management Systems
in Aviation Fundamentals and structure of safety management systems in
R. Müller, A. Wittmer  C. Drax (), Aviation Risk and Safety Management: Methods and Applications in Aviation
Organizations Aviation risk and safety management: Methods and applications
in aviation organizations ( 45–55).

VC. Müller

VC. Müller

Müller, VC.
Bostrom, N.

Future Progress in Artificial Intelligence: A Survey of
Expert Opinion Future progress in artificial intelligence: A survey of
expert opinion.
VC. Müller (), Fundamental Issues of
Artificial Intelligence Fundamental issues of artificial intelligence  ( 553–571).

Newman, J.

A Taxonomy of Trustworthiness for Artificial Intelligence. A
taxonomy of trustworthiness for artificial intelligence.
Center for Long-Term Cybersecurity.

, Chan

Ngo, R.
, Chan, L.
Mindermann, S.

The alignment problem from a deep learning perspective
The alignment problem from a deep learning perspective.
arXiv preprint arXiv:2209.00626.

Artificial Intelligence Risk Management Framework (AI RMF
1.0). Artificial Intelligence Risk Management Framework (AI RMF 1.0).

NITI Aayog

NITI Aayog

NITI Aayog.

National Artificial Intelligence Strategy. National
artificial intelligence strategy.

Advancing accountability in AI: Governing and managing risks
throughout the lifecycle for trustworthy AI. Advancing accountability in
AI: Governing and managing risks throughout the lifecycle for trustworthy
OECD Digital Economy Papers.

Announcing OpenAI’s Bug Bounty Program. Announcing
OpenAI’s bug bounty program.
OpenAI Blog.

GPT-4 Technical Report GPT-4 technical
arXiv preprint arXiv:2303.08774.

Ord, T.

The Precipice The precipice.
Hachette Books.

Ostrom, LT.
Wilhelmsen, CA.

Risk assessment: tools, techniques, and their applications
Risk assessment: tools, techniques, and their applications.
John Wiley & Sons, Inc.

Parente, R.
Anderson-Parente, J.

A case study of long-term Delphi accuracy A case
study of long-term Delphi accuracy.
Technological Forecasting and Social

Pei, J.
, Deng, L.
, Song, S.
, Zhao, M.
, Zhang, Y.
, Wu, S.
Shi, L.

Towards artificial general intelligence with hybrid
Tianjic chip architecture Towards artificial general intelligence with
hybrid Tianjic chip architecture.

, Li
, He
, Galley

Peng, B.
, Li, C.
, He, P.
, Galley, M.
Gao, J.

Instruction Tuning with GPT-4 Instruction tuning
with GPT-4.
arXiv preprint arXiv:/2304.03277.

Perez, E.
, Huang, S.
, Song, F.
, Cai, T.
, Ring, R.
, Aslanides, J.
Irving, G.

Red Teaming Language Models with Language Models Red
teaming language models with language models.
arXiv preprint arXiv:2202.03286.

Posner, RA.

Catastrophe: risk and response Catastrophe: risk and
Oxford University Press.

Potts, HW.
, Anderson, JE.
, Colligan, L.
, Leach, P.
, Davis, S.
Berman, J.

Assessing the validity of prospective hazard analysis
methods: a comparison of two techniques Assessing the validity of
prospective hazard analysis methods: a comparison of two techniques.
BMC Health Services Research1441.

Pritchard, CL.

Risk Management: Concepts and Guidance Risk management:
Concepts and guidance.
Auerbach Publications.

Procope, C.
, Cheema, A.
, Adkins, D.
, Alsallakh, B.
, Green, N.
, McReynolds, E.
Zvyagina, P.

System-Level Transparency of Machine Learning. System-level
transparency of machine learning.
Meta AI.

Raji, ID.
Buolamwini, J.

Actionable Auditing: Investigating the Impact of
Publicly Naming Biased Performance Results of Commercial AI Products
Actionable auditing: Investigating the impact of publicly naming biased
performance results of commercial AI products.
AIES 2019: Proceedings of the 2019 AAAI/ACM Conference
on AI, Ethics, and Society AIES 2019: Proceedings of the 2019 AAAI/ACM
Conference on AI, Ethics, and Society ( 429–435).

, Kumar
, Horowitz

Raji, ID.
, Kumar, IE.
, Horowitz, A.
Selbst, A.

The Fallacy of AI Functionality The fallacy of AI
FAccT '22: Proceedings of the 2022 ACM Conference on
Fairness, Accountability, and Transparency FAccT '22: Proceedings of the
2022 ACM Conference on Fairness, Accountability, and Transparency (  959–972).

Raji, ID.
, Smart, A.
, White, RN.
, Mitchell, M.
, Gebru, T.
, Hutchinson, B.
Barnes, P.

Closing the AI Accountability Gap: Defining an
End-to-End Framework for Internal Algorithmic Auditing Closing the AI
accountability gap: Defining an end-to-end framework for internal algorithmic
FAT* '20: Proceedings of the 2020 Conference on
Fairness, Accountability, and Transparency FAT* '20: Proceedings of the
2020 Conference on Fairness, Accountability, and Transparency (  33–44).

Delphi Method. Delphi Method.

, Vashistha
, Naik
, Aditya

Rao, A.
, Vashistha, S.
, Naik, A.
, Aditya, S.
Choudhury, M.

Tricking LLMs into Disobedience: Understanding,
Analyzing, and Preventing Jailbreaks Tricking LLMs into disobedience:
Understanding, analyzing, and preventing jailbreaks.
arXiv preprint arXiv:2305.14965.

Rausand, M.
Haugen, S.

Risk Assessment: Theory, Methods, and Applications Risk
assessment: Theory, methods, and applications.

Reason, J.

Human error: models and management Human error: models
and management.
British Medical Journal3207237768–770.

Rees, MJ.

Our final hour: A scientist's warning: how terror, error, and
environmental disaster threaten humankind's future in this century on earth
and beyond Our final hour: A scientist's warning: how terror, error, and
environmental disaster threaten humankind's future in this century on earth
and beyond.
Basic Books.

, Lövkvist-Andersen
, Olsson

Ritchey, T.
, Lövkvist-Andersen, AL.
, Olsson, R.
Stenström, M.

Modelling Society's Capacity to Manage Extraordinary
Events Developing a Generic Design Basis (GDB) Model for Extraordinary
Societal Events using Computer-Aided Morphological Analysis Modelling
society's capacity to manage extraordinary events developing a generic design
basis (GDB) model for extraordinary societal events using computer-aided
morphological analysis.
Society for Risk Analysis Conference. Society for
Risk Analysis Conference.

, Jaeger

Roli, A.
, Jaeger, J.
Kauffman, SA.

How Organisms Come to Know the World: Fundamental Limits
on Artificial General Intelligence How organisms come to know the world:
Fundamental limits on artificial general intelligence.
Frontiers in Ecology and Evolution9.

Rowe, G.
Wright, G.

The Delphi technique as a forecasting tool: issues and
analysis The Delphi technique as a forecasting tool: issues and
International Journal of Forecasting154353–375.

, Lawler

Rozo, M.
, Lawler, J.
Paragas, J.

Viral Agents of Human Disease: Biosafety Concerns
Viral agents of human disease: Biosafety concerns.
DP. Wooley  KB. Byers (), Biological safety: principles and practices Biological safety: principles
and practices ( 187–220).
ASM Press.

Russell, SJ.

Human compatible: Artificial intelligence and the problem of
control Human compatible: Artificial intelligence and the problem of

Salmon, PM.
, Baber, C.
, Burns, C.
, Carden, T.
, Cooke, N.
, Cummings, M.
Stanton, NA.

Managing the risks of artificial general intelligence: A
human factors and ergonomics perspective Managing the risks of artificial
general intelligence: A human factors and ergonomics perspective.
Human Factors and Ergonomics in Manufacturing &
Service Industries1–13.

Schuett, J.

AGI labs need an internal audit function AGI labs
need an internal audit function.
arXiv preprint arXiv:2305.17038.

Schuett, J.
, Dreksler, N.
, Anderljung, M.
, McCaffary, D.
, Heim, L.
, Bluemke, E.
Garfinkel, B.

Towards best practices in AGI safety and governance: A
survey of expert opinion Towards best practices in AGI safety and
governance: A survey of expert opinion.
arXiv preprint arXiv:2305.07153.

Schweizer, VJ.

Reflections on cross-impact balances, a systematic
method constructing global socio-technical scenarios for climate change
research Reflections on cross-impact balances, a systematic method
constructing global socio-technical scenarios for climate change
Climatic Change16241705–1722.

Searle, JR.

Minds, brains, and programs Minds, brains, and
Behavioral and Brain Sciences33417–424.

Shah, R.

AI Risk from Program Search. AI risk from program
AI Alignment Forum.

Shelby, R.
, Rismani, S.
, Henne, K.
, Moon, A.
, Rostamzadeh, N.
, Nicholas, P.
Virk, G.

Identifying Sociotechnical Harms of Algorithmic Systems:
Scoping a Taxonomy for Harm Reduction Identifying sociotechnical harms of
algorithmic systems: Scoping a taxonomy for harm reduction.
arXiv preprint arXiv:2210.05791.

Shevlane, T.
, Farquhar, S.
, Garfinkel, B.
, Phuong, M.
, Whittlestone, J.
, Leung, J.
Dafoe, A.

Model evaluation for extreme risks Model evaluation
for extreme risks.
arXiv preprint arXiv:2305.15324.

Shneiderman, B.

Bridging the Gap Between Ethics and Practice: Guidelines
for Reliable, Safe, and Trustworthy Human-centered AI Systems Bridging
the gap between ethics and practice: Guidelines for reliable, safe, and
trustworthy human-centered AI systems.
ACM Transactions on Interactive Intelligent

Soares, N.

A central AI alignment problem: capabilities generalization,
and the sharp left turn. A central AI alignment problem: capabilities
generalization, and the sharp left turn.
AI Alignment Forum.

, Weinstein-Raun

Stein-Perlman, Z.
, Weinstein-Raun, B.
Grace, K.

2022 Expert Survey on Progress in AI. 2022 Expert
survey on progress in AI.
AI Impacts.

, Sumwalt

Stolzer, AJ.
, Sumwalt, RL.
Goglia, JJ.

Safety management systems in aviation Safety management
systems in aviation.
CRC Press.

Suresh, H.
Guttag, JV.

A Framework for Understanding Sources of Harm throughout
the Machine Learning Life Cycle A framework for understanding sources of
harm throughout the machine learning life cycle.
arXiv preprint arXiv:1901.10002.

Taori, R.
, Gulrajani, I.
, Zhang, T.
, Dubois, Y.
, Li, X.
, Guestrin, C.
Hashimoto, TB.

Alpaca: A Strong, Replicable Instruction-Following
Model. Alpaca: A Strong, Replicable Instruction-Following
Stanford University.

The Vicuna Team

The Vicuna Team

The Vicuna Team.

Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%
ChatGPT Quality. Vicuna: An open-source chatbot impressing GPT-4 with
90% ChatGPT quality.
LMSYS Org Blog.

The White House

The White House

The White House.

National Artificial Intelligence Research and Development
Strategic Plan 2023 Update. National artificial intelligence research and
development strategic plan 2023 update.

UK Information Commissioner’s Office

UK Information Commissioner’s Office

UK Information Commissioner’s Office.

Data protection impact assessments. Data protection impact

UK Royal Academy of Engineering

UK Royal Academy of Engineering

UK Royal Academy of Engineering.

Building resilience: lessons from the Academy's review of the
National Security Risk Assessment methodology. Building resilience: lessons
from the academy's review of the national security risk assessment

, Lentzos
, Invernizzi

Urbina, F.
, Lentzos, F.
, Invernizzi, C.
Ekins, S.

Dual use of artificial-intelligence-powered drug
discovery Dual use of artificial-intelligence-powered drug
Nature Machine Intelligence43189–191.

US Environmental Protection Agency

US Environmental Protection Agency

US Environmental Protection Agency.

Guidelines for Ecological Risk Assessment. Guidelines for
ecological risk assessment.

van der Heijden

van der Heijden

van der Heijden, K.

Scenarios: The art of strategic conversation Scenarios: The
art of strategic conversation.

Weidinger, L.
, Uesato, J.
, Rauh, M.
, Griffin, C.
, Huang, PS.
, Mellor, J.
Gabriel, I.

Taxonomy of Risks posed by Language Models Taxonomy of
risks posed by language models.
FAccT '22: Proceedings of the 2022 ACM Conference on
Fairness, Accountability, and Transparency FAccT '22: Proceedings of the
2022 ACM Conference on Fairness, Accountability, and Transparency (  214–229).

Westerlund, M.

The Emergence of Deepfake Technology: A Review The
emergence of deepfake technology: A review.
Technology Innovation Management

Wiggers, K.

Anthropic raises $450M to build next-gen AI assistants.
Anthropic raises $450M to build next-gen AI assistants.

List of causal mapping software. List of causal mapping

Xia, B.
, Lu, Q.
, Perera, H.
, Zhu, L.
, Xing, Z.
, Liu, Y.
Whittle, J.

Towards Concrete and Connected AI Risk Assessment
(C2AIRA): A Systematic Mapping Study Towards concrete and connected AI
risk assessment (C2AIRA): A systematic mapping study.
arXiv preprint arXiv:2301.11616.

Yampolskiy, RV.

Taxonomy of Pathways to Dangerous AI Taxonomy of
pathways to dangerous AI.
arXiv preprint arXiv:1511.03246.

Yudkowsky, E.

Artificial intelligence as a positive and negative
factor in global risk Artificial intelligence as a positive and negative
factor in global risk.
N. Bostrom  MM. Ćirković (), Global Catastrophic Risks Global catastrophic risks ( 308–345).
Oxford University Press.

Yudkowsky, E.

Pausing AI Developments Isn't Enough. We Need to Shut it All
Down. Pausing AI developments isn't enough. we need to shut it all down.

, Murschitz
, Humenberger

Zendel, O.
, Murschitz, M.
, Humenberger, M.
Herzner, W.

CV-HAZOP: Introducing Test Data Validation for
Computer Vision CV-HAZOP: Introducing test data validation for computer
Proceedings of the 2015 IEEE International Conference
on Computer Vision (ICCV) Proceedings of the 2015 IEEE International
Conference on Computer Vision (ICCV) ( 2066–2074).

Zhang, B.
, Dreksler, N.
, Anderljung, M.
, Kahn, L.
, Giattino, C.
, Dafoe, A.
Horowitz, MC.

Forecasting AI Progress: Evidence from a Survey of
Machine Learning Researchers Forecasting AI progress: Evidence from a
survey of machine learning researchers.
arXiv preprint arXiv:2206.04132.

Zwetsloot, R.
Dafoe, A.

Thinking About Risks From AI: Accidents, Misuse and
Structure. Thinking about risks from AI: Accidents, misuse and

§ APPENDIX: OTHER POPULAR RISK ASSESSMENT TECHNIQUES

Technique     Explanation     Reason(s) for excluding

Brainstorming     Brainstorming is a group exercise to generate and explore ideas. It works best when facilitated to ensure stimulation, avoid common fallacies (digressions, group-think, some participants not speaking up), and make sure that ideas are captured. Brainstorming can be used on its own or within other techniques.     Included in many other techniques

Interviews     In structured interviews, interviewees are asked a set of prepared questions. In unstructured interviews, there is freedom to explore issues which arise. Semi-structured interviews are a mix of both pure forms. Interviews can be used on their own or within other techniques.     Similar to surveys, but often more time-consuming

Surveys     Surveys are questionnaires that provide a way to get responses from a large number of people. Surveys can be used on their own or within other techniques. Note that surveys and interviews are overlapping and not necessarily distinct.     Included in many other techniques which gather expert opinions

Nominal group technique     The Nominal group technique is a procedure to collect, explore, and decide on ideas. Participants answer questions independently and then each present their ideas. Next, participants discuss and aggregate these ideas, and finally vote on the ideas anonymously.     Somewhat similar to Delphi technique, but not fully anonymous and most useful for making decisions about risks (which already is part of risk treatment)

Cindynic approach     The Cindynic approach involves semi-structured interviews that are conducted with different stakeholders. Their answers are put together in a matrix. This helps to identify dissonances, inconsistencies, ambiguities, omissions, and ignorances between stakeholders.     Somewhat similar to Delphi technique, but much less established and does not comprise exchange of reasoning between stakeholders

Techniques for eliciting views from experts and stakeholders

Technique     Explanation     Reason(s) for excluding

Hazard identification (HAZID)	     HAZID is a high-level technique. It involves conducting a workshop to identify risks, for example, through brainstorming (see above).     Needs to be concretized through other techniques (e.g. brainstorming)

Preliminary hazard analysis (PHA)     PHA involves a preliminary collection of hazards through analyzing historical data, gathering expert forecasts, and brainstorming.     Relatively superficial; does not provide a very specific or structured approach

Failure modes and effects analysis (FMEA) and failure modes, effects and criticality analysis (FMECA)     A system or process is divided into elements, for each of which failure modes, their causes and effects (FMEA), and optionally their criticality (FMECA) are identified through brainstorming. If quantified, a risk priority number can be calculated. All information is captured in a worksheet.     Traditional hardware reliability technique; may be too simplistic for catastrophic risks from AI which involve complex interactions between various events (techniques like causal mapping or STPA may be more adequate); overlaps with fishbone method and bow tie analysis, which additionally provide visualizations

Hazard and operability (HAZOP)     Within a facilitated workshop, a system or process is divided into elements, for each of which possible deviations from design intent, their causes and effects are identified. The facilitator prompts the participants with a set of guide words like “no”, “less”, etc. All information is captured in a worksheet.     Traditional hardware reliability technique; may be too simplistic for catastrophic risks from AI which involve complex interactions between various events (techniques like causal mapping or STPA may be more adequate); overlaps with fishbone method and bow tie analysis, which provide more freedom (no predetermined set of guide words) and visualizations

What-if analysis     Within a facilitated workshop, a system or process is examined for potential risks and risk sources. The facilitator prompts the participants with “what if?” questions.     Implicitly performed in fishbone method, which provides more structure (through categories of causes) as well as a visualization

Structured what-if technique (SWIFT)     Within a facilitated workshop, a system or process is examined for potential risks. The facilitator prompts the participants with combinations of “what if?” and a set of guide words on timing, amount, etc. SWIFT is relatively little time consuming. It can be used on its own or within other techniques.     Similar to fishbone method, which provides more freedom (no predetermined set of guide words) as well as a visualization

Business impact analysis (BIA)     BIA is a method to identify critical business processes and functions. It aims to enable appropriate planning for disruptive events. BIA is undertaken using surveys, interviews, workshops or a combination of all of these.     Limited to financial risks (does not include risks to the organization’s environment for their own sake)

Process mapping     Process mapping involves drawing a graphical representation of the steps involved in industrial or other processes and identifying the associated risks.     Relatively superficial; does not provide a very specific approach for identifying risks once the process has been mapped

Job hazard analysis (JHA) / job safety analysis (JSA)     JHA/JSA are conducted before conducting a specific task. The task is broken down into its smallest sub-tasks, for each of which potential risks are brainstormed.     Best for routine safety-critical tasks or individual but not very complex safety-critical tasks (such as evals – the technique is usually not used for these kinds of tasks because it would be extremely comprehensive and might not add much as safety is already paramount when designing them)

Change analysis     Change analysis aims to identify risks by comparing a known system with an unknown system. It can be done when considering or observing changes.     Implicit within risk assessment (generally, risks should be assessed again whenever changes are considered or observed)

Gap analysis     Similar to change analysis, gap analysis compares two states of a system. However, it starts from an ideal system and aims to identify shortcomings in the current system.     Not very helpful if it is unclear what the ideal system would look like

Root cause analysis (RCA)     RCA is a high-level technique. It involves investigating a risk for its ultimate sources, especially after an incident has occurred. This is done, for example, through what-if analysis (see above).     Needs to be concretized through other techniques (e.g. what-if analysis)

Techniques for identifying risks

Technique     Explanation     Reason(s) for excluding

Bayesian networks     Bayesian networks are graphical representations of events and include their probability based on other events and the probability of those. There is readily available software to build Bayesian networks.     Similar to cross-impact analysis (cross-impact matrices and interdependency maps), which takes into account more complex interactions between events

Influence diagrams     Bayesian networks are called influence diagrams when they include actions and uncertainties. There is readily available software to build influence diagrams.     Similar to causal mapping and parts of cross-impact analysis (cross-impact matrices and interdependency maps), which take into account more complex interactions between events

Event tree analysis (ETA)     ETA involves drawing a graphical representation of the sequence of consequences resulting from a given initiating event (in binary terms of their failure / success). It is developed using forward reasoning to identify these consequences. ETA can be quantified to provide the probabilities of the identified possible outcomes.     Traditional hardware reliability technique; may be too simplistic for catastrophic risks from AI which involve complex interactions between non-binary events (techniques like causal mapping or STPA may be more adequate); overlaps with several other techniques (event trees can help to develop scenarios and constitute the right side of bow tie diagrams); see also Section <ref>

Fault tree analysis (FTA)     FTA involves drawing a graphical representation of the sequence of causes leading to a given undesired event, making use of Boolean logic (such as AND / OR). It is developed using backwards reasoning to identify these causes. FTA can be quantified to provide the probabilities of the identified possible failures.     Traditional hardware reliability technique; may be too simplistic for catastrophic risks from AI which involve complex interactions between non-binary events (techniques like causal mapping or STPA may be more adequate); overlaps with several other techniques (fault trees are implicit in fishbone diagrams and constitute the left side of bow tie diagrams); see also Section <ref>

Cause-consequence analysis (CCA)     CCA is a combination of ETA and FTA.     Same reasoning as for ETA and FTA

Master logic diagram (MLD)     MLD is similar to FTA (see above), but not limited to binary failure/success events. It can also be understood to be a summary FTA.     Same reasoning as for FTA (see above)

Human reliability analysis (HRA)     HRA is a family of techniques that aim to establish the likelihood of human error within a system. It involves identifying the steps and substeps of an activity, estimating the probability of human error and identifying the factors influencing this probability.     Best for routine safety-critical tasks, for which data on human reliability exists

Markov analysis     Markov analysis can be made of any system that has discrete, independent states. It uses the probabilities of the transitions between these states to estimate the long-run probability of the system being in a specified state, the expected time before its first failure and the expected time before its return to a specified state.     May be too simplistic for catastrophic risks from AI which involve complex interactions between various events (techniques like causal mapping or STPA may be more adequate)

Monte Carlo simulations     Monte Carlo simulations are a stochastic method that use random sample values to provide a probability distribution of in principle deterministic outcomes. They can be used on their own or within other techniques. Monte Carlo simulations tend to de-emphasize high consequence/low probability risks.     Neglects low-probability risks; included in cross-impact analysis

Value at risk (VaR)     VaR is a measure that indicates the amount of possible loss of financial assets under normal market conditions over a specific time period. It can be developed using Monte Carlo simulations, historical simulations, analytical methods or a combination of all of these. VaR calculations for the tails are often unstable.     Limited to financial risks (does not include risks to the organization’s environment for their own sake); neglects low-probability risks

Conditional value at risk (CVaR) or expected shortfall (ES)     In response to the issues of VaR calculations for the tails, CVaR/ES is the VaR that focuses on the outcomes generating the greatest loss.     Limited to financial risks (does not include risks to the organization’s environment for their own sake)

Techniques for analyzing causes, consequences, and likelihood of risks

Technique     Explanation     Reason(s) for excluding

Hazard analysis and critical control points (HACCP)     HACCP is a structure for identifying sources of risks and putting controls in place at all relevant parts of a process to protect against them. After identifying hazards, influencing factors, and possible preventive measures, the points in the process where monitoring and control are possible are determined. Then, a system of critical limits and corrective actions is established.     May be too simplistic for catastrophic risks from AI which involve complex interactions between various events (techniques like causal mapping or STPA may be more adequate); already constitutes transition into risk treatment

Layers of protection analysis (LOPA)     LOPA is a simplified version of ETA (see above), namely a single cause-consequence pair. It involves analyzing the reduction in risk that is achieved by controls by identifying independent protection layers and estimating their individual and the overall probability of failure.     Same reasoning as for ETA (see above)

Techniques for analyzing controls

Technique     Explanation     Reason(s) for excluding

Risk indices     Risk indices are indices that allow comparisons of different risks. Factors which are believed to influence the magnitude of risk are identified, scored and combined. In the simplest formulations, factors that increase the level of risk are multiplied together and divided by those that decrease the level of risk.     Similar to risk matrices, which are more common

As low as reasonably practicable (ALARP) and so far as is reasonably practicable (SFAIRP)     ALARP/SFAIRP involve establishing criteria and categories accounting for both the acceptability of risks and the practicability to reduce them. These help to decide whether it is reasonably practicable to further reduce risk.     Similar to risk matrices, which are more common

Frequency-number (F-N) diagrams     F-N diagrams are a special case of a risk matrix, namely one that focuses on fatalities. They are a graphical representation with x-axis as the cumulative number of fatalities and the y-axis as their frequency.     Similar to risk matrices, which are more comprehensive

S-curves     S-curves are a graphical display of the severity of consequences of a risk as a probability distribution function (PDF) or cumulative distribution function (CDF). They allow representing the significance of a risk where there is a distribution of consequences.     Neglects low-probability risks

Reliability centered maintenance (RCM)     RCM is a version of FMECA (see above) that focuses on situations where potential failures can be eliminated or reduced in frequency and/or consequence through maintenance of equipment. It enables decisions to be made based on the significance of risk. RCM spans all three risk assessment steps.     Only for hardware reliability

Techniques for evaluating risks


* [101] D. Yi, H. Fang, Y. Hua, J. Su, M. Quddus, and J. Han, “Improving synthetic to realistic semantic segmentation with parallel generative ensembles for autonomous urban driving,” IEEE Transactions on Cognitive and Developmental Systems, vol. 14, no. 4, pp. 1496–1506, 2021.
  * [102] Y. Hua, J. Sui, H. Fang, C. Hu, and D. Yi, “Domain-adapted driving scene understanding with uncertainty-aware and diversified generative adversarial networks,” CAAI Transactions on Intelligence Technology, 2023.
  * [103] B. Varga, D. Yang, and S. Hohmann, “Cooperative decision-making in shared spaces: Making urban traffic safer through human-machine cooperation,” arXiv preprint arXiv:2306.14617, 2023.
  * [104] M. Kloock and B. Alrifaee, “Coordinated cooperative distributed decision-making using synchronization of local plans,” IEEE Transactions on Intelligent Vehicles, vol. 8, no. 2, pp. 1292–1306, 2023.
  * [105] K. Zhang, L. Zhao, C. Dong, L. Wu, and L. Zheng, “Ai-tp: Attention-based interaction-aware trajectory prediction for autonomous driving,” IEEE Transactions on Intelligent Vehicles, vol. 8, no. 1, pp. 73–83, 2022.
  * [106] T.-y. Fu and W.-C. Lee, “Progrpgan: Progressive gan for route planning,” in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 393–403, 2021.
  * [107] J. Ocampo Jimenez and W. Suleiman, “Improving path planning performance through multimodal generative models with local critics,” arXiv e-prints, pp. arXiv–2306, 2023.
  * [108] H. He, C. Bai, K. Xu, Z. Yang, W. Zhang, D. Wang, B. Zhao, and X. Li, “Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning,” arXiv preprint arXiv:2305.18459, 2023.
  * [109] S. Huang, Z. Wang, P. Li, B. Jia, T. Liu, Y. Zhu, W. Liang, and S.-C. Zhu, “Diffusion-based generation, optimization, and planning in 3d scenes,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16750–16761, 2023.
  * [110] L. Ma, S. Qu, L. Song, Z. Zhang, and J. Ren, “A physics-informed generative car-following model for connected autonomous vehicles,” Entropy, vol. 25, no. 7, p. 1050, 2023.
  * [111] L. Ma and S. Qu, “Application of conditional generative adversarial network to multi-step car-following modeling,” Frontiers in Neurorobotics, vol. 17, p. 1148892, 2023.
  * [112] C. Cui, Y. Ma, X. Cao, W. Ye, and Z. Wang, “Drive as you speak: Enabling human-like interaction with large language models in autonomous vehicles,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 902–909, 2024.
  * [113] H. Sha, Y. Mu, Y. Jiang, L. Chen, C. Xu, P. Luo, S. E. Li, M. Tomizuka, W. Zhan, and M. Ding, “Languagempc: Large language models as decision makers for autonomous driving,” arXiv preprint arXiv:2310.03026, 2023.
  * [114] G. Zhou, Y. Hong, and Q. Wu, “Navgpt: Explicit reasoning in vision-and-language navigation with large language models,” arXiv preprint arXiv:2305.16986, 2023.
  * [115] L. Wen, D. Fu, X. Li, X. Cai, T. Ma, P. Cai, M. Dou, B. Shi, L. He, and Y. Qiao, “Dilu: A knowledge-driven approach to autonomous driving with large language models,” arXiv preprint arXiv:2309.16292, 2023.
  * [116] L. Feng, Q. Li, Z. Peng, S. Tan, and B. Zhou, “Trafficgen: Learning to generate diverse and realistic traffic scenarios,” in 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 3567–3575, IEEE, 2023.
  * [117] A. Li, S. Chen, L. Sun, N. Zheng, M. Tomizuka, and W. Zhan, “Scegene: Bio-inspired traffic scenario generation for autonomous driving testing,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 9, pp. 14859–14874, 2021.
  * [118] S. Tan, K. Wong, S. Wang, S. Manivasagam, M. Ren, and R. Urtasun, “Scenegen: Learning to generate realistic traffic scenes,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 892–901, 2021.
  * [119] Z. Guo, X. Gao, J. Zhou, X. Cai, and B. Shi, “Scenedm: Scene-level multi-agent trajectory generation with consistent diffusion models,” arXiv preprint arXiv:2311.15736, 2023.
  * [120] E. Pronovost, M. R. Ganesina, N. Hendy, Z. Wang, A. Morales, K. Wang, and N. Roy, “Scenario diffusion: Controllable driving scenario generation with diffusion,” arXiv preprint arXiv:2311.02738, 2023.
  * [121] A. Bandi, P. V. S. R. Adapa, and Y. E. V. P. K. Kuchi, “The power of generative ai: A review of requirements, models, input–output formats, evaluation metrics, and challenges,” Future Internet, vol. 15, no. 8, p. 260, 2023.
  * [122] S. Tan, B. Ivanovic, X. Weng, M. Pavone, and P. Kraehenbuehl, “Language conditioned traffic generation,” arXiv preprint arXiv:2307.07947, 2023.
  * [123] Y. Chen, F. Rong, S. Duggal, S. Wang, X. Yan, S. Manivasagam, S. Xue, E. Yumer, and R. Urtasun, “Geosim: Realistic video simulation via geometry-aware composition for self-driving,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7230–7240, 2021.
  * [124] X. Li, Y. Zhang, and X. Ye, “Drivingdiffusion: Layout-guided multi-view driving scene video generation with latent diffusion model,” arXiv preprint arXiv:2310.07771, 2023.