# Photo-$z$SNthesis: Converting Type Ia Supernova Lightcurves to Redshift
Estimates via Deep Learning

Helen Qu Department of Physics and Astronomy, University of Pennsylvania,
Philadelphia, PA 19104, USA  Masao Sako Department of Physics and Astronomy,
University of Pennsylvania, Philadelphia, PA 19104, USA Helen Qu
<EMAIL_ADDRESS>
###### Abstract

Upcoming photometric surveys will discover tens of thousands of Type Ia
supernovae (SNe Ia), vastly outpacing the capacity of our spectroscopic
resources. In order to maximize the science return of these observations in
the absence of spectroscopic information, we must accurately extract key
parameters, such as SN redshifts, with photometric information alone. We
present Photo-$z$SNthesis, a convolutional neural network-based method for
predicting full redshift probability distributions from multi-band supernova
lightcurves, tested on both simulated Sloan Digital Sky Survey (SDSS) and Vera
C. Rubin Legacy Survey of Space and Time (LSST) data as well as observed SDSS
SNe. We show major improvements over predictions from existing methods on both
simulations and real observations as well as minimal redshift-dependent bias,
which is a challenge due to selection effects, e.g. Malmquist bias. The PDFs
produced by this method are well-constrained and will maximize the
cosmological constraining power of photometric SNe Ia samples.

##  1 Introduction

The study of Type Ia supernovae (SNe Ia) has proven to be a crucial tool in
modern cosmology, providing insight into the expansion rate of the universe
and the properties of dark energy (Riess et al., 1998; Perlmutter et al.,
1999). Measuring the cosmological redshift of each SN, a proxy quantity for
recessional velocity, is essential for accurate estimation of the distance-
redshift relation and resulting cosmological analyses. However, traditional
methods of measuring redshifts are time-consuming and resource-intensive,
primarily relying on spectroscopic observations of the SNe themselves or their
host galaxies. Using host galaxy redshifts can also lead to cosmological
biases if the host is incorrectly identified (Qu et al., in prep).

Figure 1: True redshift distribution of the 73,620 simulated SNe Ia in the
LSST (PLAsTiCC) training dataset (left) and 4,057 simulated SNe Ia in the test
dataset (right).

Most SNe Ia cosmological analyses so far have relied on SN spectra for SN type
confirmation as well as redshift information, but only with samples of up to
$\sim 1,500$ SNe (e.g. Abbott et al., 2019; Brout et al., 2022). Spectroscopic
follow-up of all SNe Ia candidates or their host galaxies will become
infeasible with future sky surveys such as the Legacy Survey of Space and Time
at the Vera C. Rubin Observatory (LSST), which will discover tens of thousands
of SNe Ia over the course of their observational lifetimes (LSST Science
Collaboration et al., 2009). Recent improvements in photometric SN
classification (e.g. Möller & de Boissière, 2020; Qu et al., 2021) have
drastically reduced the likelihood of non-Ia contamination in photometric SNe
Ia samples and enabled cosmological analyses with photometrically classified
samples (Vincenzi et al., 2023). However, spectroscopic redshifts were still
available for the host galaxies of these photometrically confirmed SNe Ia and
were used as the SN redshifts. Accurate photometric redshift estimates for SNe
that are independent of host galaxy spectroscopic redshifts are thus the final
building block required to enable SNe Ia cosmology for the LSST era.

Cosmological inference frameworks that account for the inflated uncertainties
from photometric redshifts are currently being developed. Mitra et al. (2022);
Dai et al. (2018) show promising results with simulated LSST samples and SN
photometric redshifts fitted using host galaxy redshift priors. In particular,
Dai et al. (2018) recovers a fitted $\Omega_{m}$ value consistent with the
input cosmology when using SN photo-$z$s fitted with a host galaxy photo-$z$
prior. Mitra et al. (2022) shows a 2% effect on fitted cosmological parameters
of an assumed systematic uncertainty due to the use of SN photo-$z$s of 0.01.
Using observed data from the Dark Energy Survey, Chen et al. (2022) performed
a cosmological analysis using a subset of $\sim 100$ SNe Ia hosted by galaxies
in the redMaGiC catalog, which have both photometric and spectroscopic
redshifts. The difference in best-fit cosmological parameters between using
spectroscopic and photometric redshifts was found to be minimal, $\Delta w\sim
0.005$. redMaGiC galaxies were chosen for this analysis due to their
particularly well-constrained photometric redshift estimates, which is not
representative of the full population of SN host galaxies. However, even with
a sound cosmological inference framework, galaxy photometric redshifts are
often inaccurate or plagued with large uncertainties, and requiring host
galaxy information to produce SN photometric redshift estimates may introduce
additional biases.

These issues with traditional redshift determination along with the
development of a cosmological framework for photometric redshifts has led to a
growing interest in alternative techniques for predicting redshifts for Type
Ia supernovae. In this work, we intoduce a novel machine learning algorithm to
predict full redshift probability distributions for Type Ia supernovae based
solely on lightcurve data. Our estimator harnesses the constraining power on
redshift of the SN lightcurves and can additionally provide an independent
cross-check on host galaxy matches, minimizing mismatch rates. We present a
detailed analysis of our model, including its accuracy and limitations, and
discuss the potential implications of our findings for future cosmological
studies.

###  1.1 Photo-z Estimation

Most of the existing literature on photometric redshift estimation is on
galaxy photo-$z$s. These approaches generally use either (1) a training set of
galaxy photometric observables, such as colors and magnitudes, to determine a
mapping to spectroscopic redshifts (e.g. Brunner et al., 1997); or (2)
template fitting, in which observed properties are compared with redshifted
template spectra to determine the best fit redshift value (e.g., Benitez,
2000). There have also been successful machine learning-based galaxy photo-$z$
models developed, e.g. D´Isanto, A. & Polsterer, K. L. (2018); Pasquet et al.
(2018) using convolutional neural networks on galaxy images, and Buchs et al.
(2019) using a self-organizing map to relate color-magnitude space and
redshifts.

Figure 2: True redshift distribution of the 135,521 simulated SNe Ia in the
SDSS training dataset (left) and the 5,274 simulated SNe Ia and 489 observed
photometrically confirmed SNe Ia in the SDSS test sets (right).

Photometric redshift estimation for SNe uses many of the same techniques. Kim
& Miquel (2007) used the SALT2 lightcurve model (Guy et al., 2007) to
determine photometric redshifts and distances as well as uncertainties using
the Fisher information matrix. Palanque-Delabrouille et al. (2010) and Kessler
et al. (2010, LCFIT+Z) extended the SALT2 lightcurve fit to include redshift
as a fitted parameter, and incorporating the host galaxy redshift as a prior.
These approaches find the best fit redshift by comparing photometric
observables to those expected from the SALT2 model via $\chi^{2}$
minimization. An analytic approach was presented in Wang et al. (2015) which
assumed a functional form for the redshifts as a function of multi-band
photometric fluxes and fit for free parameters using a training set of SNe Ia
with spectroscopic redshifts. de Oliveira et al. (2022) applied machine
learning techniques to this problem, performing regression using features
obtained from principal component analysis. However, many of these results
suffer from redshift-dependent bias, in which photo-$z$s for high-redshift SNe
are systematically underestimated (see e.g. Figure 6 of Kessler et al. (2010),
Figure 7 of de Oliveira et al. (2022)). Most of these studies also estimate a
redshift value and uncertainty rather than the full probability density
function (PDF).

###  1.2 Overview

In this work we present Photo-$z$SNthesis, a convolutional neural network
model that uses multi-band photometry to predict a full redshift PDF. Our
approach uses the raw photometric data and does not require any manual feature
engineering to determine the most predictive features. We evaluated
Photo-$z$SNthesis using LSST and SDSS simulated SNe Ia as well as the SDSS
photometric SNe Ia sample, and show that our redshift predictions have low
scatter and suffer from minimal redshift-dependent bias on all tested SN
samples. Our results are a promising step towards precision photometric SNe Ia
cosmology.

We introduce the simulated and observed data samples used for evaluation in
§2, as well as the preprocessing step used to transform multi-band lightcurves
into convolutional neural network inputs. In §3, we describe the model
architecture and training strategy. We present results, comparisons with
LCFIT+Z, and further experiments in §4, and conclude in §5.

##  2 Data

###  2.1 Data Sources

We present results on simulated SNe Ia from two surveys: LSST (Ivezić et al.,
2019) and the SDSS-II SN survey (Frieman et al., 2008). We also demonstrate
that our model generalizes well to an observed photometric SNe Ia dataset by
showing photo-$z$ predictions on SDSS SNe classified as type Ia by SuperNNova
(Möller & de Boissière, 2020). We use the SuperNova ANAlysis software (SNANA,
Kessler et al., 2009) to produce all simulated SN lightcurves used in this
work.

####  2.1.1 Simulated LSST SNe Ia (PLAsTiCC)

LSST is a ground-based dark energy survey program that will discover millions
of SNe over the 10 year survey duration. The 8.4m Simonyi Survey optical
telescope at the Rubin Observatory uses a state-of-the-art 3200 megapixel
camera with a 9.6 $\text{deg}^{2}$ field of view that will provide deeper and
wider views of the universe with unprecedented quality. LSST will observe
nearly half the night sky each week to a depth of $24^{\text{th}}$ magnitude
in $ugrizY$ photometric bands spanning wavelengths from ultraviolet to near-
infrared.

We simulate LSST-like observations of SNe Ia in all six photometric bands
following the model, rates, and LSST observing conditions developed for the
PLAsTiCC dataset (The PLAsTiCC team et al., 2018; Kessler et al., 2019). We
use the SALT2 lightcurve model (Guy et al., 2007) with training parameters
derived from the Joint Lightcurve Analysis (Betoule et al., 2014) extended
into the ultraviolet and near-infrared by Hounsell et al. (2018) following the
procedure described in Pierel et al. (2018). While PLAsTiCC included two LSST
observing strategies, the Deep Drilling Fields (DDF) as well as the Wide-Fast-
Deep (WFD), we simulate only the DDF subsample. We coadd all observations
within the same night, following PLAsTiCC, resulting in observations that are
$\sim 2.5$x more frequent and $\sim 1.5$ mag deeper than the WFD sample.

As PLAsTiCC is the name of the dataset we emulated while LSST is the survey we
simulate, we will use both interchangeably to denote “simulated LSST SN
lightcurves following the PLAsTiCC dataset”.

####  2.1.2 Simulated SDSS SNe Ia

The SDSS-II Supernova Survey identified and measured light curves for
intermediate-redshift ($0.05<z<0.4$) SNe Ia using repeated five-band ($ugriz$)
imaging of Stripe 82, a stripe $2.5^{\circ}$ wide centered on the celestial
equator in the Southern Galactic Cap. The primary instrument for this survey
is the SDSS CCD camera mounted on a dedicated 2.5m telescope at Apache Point
Observatory, New Mexico. Over the three observing seasons between 2005 and
2007, SDSS discovered 10,258 transient and variable objects, with 536
spectroscopically confirmed SNe Ia and an additional 907 photometrically
classified SNe Ia candidates (Sako et al., 2018). Frieman et al. (2008)
provides an in-depth review of the SDSS-II SN survey.

We simulate SDSS SNe Ia in $ugriz$ photometric bands using the SALT3 model
described by Kenworthy et al. (2021) due to its improved wavelength range
coverage in the near infrared, allowing $z$ band observations to be simulated
for SNe at low redshifts. We additionally extend this model to $500$Å to
simulate $u$ band observations at high redshifts. We use simulated observing
conditions from Kessler et al. (2013) and host galaxy spectroscopic detection
efficiency from Kessler et al. (2009). We simulate a photometric SDSS SNe Ia
dataset using the host spectroscopic detection efficiency rather than a SN
spectroscopic detection efficiency to demonstrate the performance of
Photo-$z$SNthesis on the practical use case of a photometric sample.

####  2.1.3 Observed SDSS SNe Ia

In addition to simulated SNe, we test Photo-$z$SNthesis on SDSS lightcurves
from Sako et al. (2018) classified by SuperNNova (Möller & de Boissière, 2020)
as likely SNe Ia, defined as SNe with SNIa probability $P_{\text{Ia}}\geq
0.5$. “True” redshifts $z_{\text{true}}$ for the sample are from spectra of
the SNe themselves or their host galaxies. Details on the training data and
training procedure for the SuperNNova model used here can be found in Popovic
et al., in prep. 489 lightcurves remain after the selection cuts described in
Sections 2.1.4 and 2.2. The number of SDSS lightcurves remaining after each
set of cuts is shown in Table 1.

####  2.1.4 Lightcurve Selection

We apply basic selection cuts to the simulated LSST sample. We require

  * •

at least 5 observations of each SN

  * •

signal-to-noise ratio (SNR) $>3$ for at least one observation each in 2 of the
$griz$ filters

We apply selection cuts, following Popovic et al. (2020), to both the
simulated and observed SDSS data to remove poor quality lightcurves. We define
a rest-frame age, $T_{\text{rest}}=(t-t_{\text{peak}})/(1+z)$, where $t$ is
the observation date, $t_{\text{peak}}$ is the estimated epoch of SN peak
brightness from SNANA, and $z$ is the redshift of the event. We require

  * •

$0<T_{\text{rest}}<10$

  * •

SNR $>5$ for at least one observation each in 2 of the $griz$ filters

We additionally require SNe in the SDSS data to have a spectroscopic redshift
from either the SN spectrum or the SN host galaxy.

Cut | Number of SNe
---|---
Full sample | 10,258
Lightcurve selection (§2.1.4) | 2,044
Likely SNe Ia ($P_{\text{Ia}}\geq 0.5$) | 1,037
SALT fit cuts (§2.2) | 555
Successful LCFIT+Z fit | 489
Table 1: Number of SDSS lightcurves remaining after each selection cut. We
evaluate Photo-$z$SNthesis on the remaining 489 SNe. Figure 3: Examples of a
low (left), medium (middle), and high (right) redshift simulated PLAsTiCC SN.
(top panel) The multi-band lightcurves are shown in colored points, with
Gaussian process predictions and uncertainties for this lightcurve shown in
colored lines and shaded regions, respectively. (bottom panel) The flux matrix
created from the lightcurve in the top panel (see §2.2.1 for details). Large
flux values are colored in yellow while lower flux values are in dark blue.
The physical effects of redshifting, including longer wavelengths and longer
transient durations, are observed in the flux matrices in the downward
(towards higher wavelengths) shift of the yellow high flux region and the
increased width of the yellow region with increasing redshift.

###  2.2 Data Preprocessing

All SNe in all datasets are fit with a $\chi^{2}$-minimization program
included in SNANA to determine several restframe parameters under the
assumption that the event is a SN Ia: the time of SN peak brightness
$t_{\mathrm{peak}}$, a stretch-like parameter $x_{1}$, a color parameter $c$,
and the lightcurve normalization parameter $x_{0}$, as well as their
uncertainties and covariances (i.e., $\sigma_{x_{1}}$, etc.). These parameters
are used to calculate the distance modulus $\mu$, allowing the SNe to be
placed on the Hubble diagram. SDSS datasets are fit with the SALT3 model while
the LSST dataset is fit with the same extended SALT2 model used for the
simulations.

We additionally select only SNe in our datasets that are well-described by the
SALT model. These criteria were chosen following those used in past cosmology
analyses, e.g. Betoule et al. (2014):

  * •

$|c|<0.3$

  * •

$|x_{1}|<3$

  * •

$\sigma_{x_{1}}\leq 1$

  * •

$\sigma_{t_{\mathrm{peak}}}\leq 2$

Finally, we choose SNe within the redshift range $0.01\leq z\leq 0.4$ for SDSS
and $0.01\leq z\leq 1.2$ for LSST that have a successful LCFIT+Z (Kessler et
al., 2010) fit, in order to draw direct comparisons between the performance of
Photo-$z$SNthesis and LCFIT+Z. A more detailed description of LCFIT+Z can be
found in §4.1.2.

146,069 simulated SDSS SNe Ia and 81,734 simulated LSST SNe Ia remain after
all cuts, as well as 489 observed SDSS SNe photometrically classified as SNe
Ia. The number of observed SDSS lightcurves remaining after each cut is
described in Table 1. The redshift distributions of the PLAsTiCC training and
test datasets after selection cuts are shown in Figure 1, while the redshift
distributions of the SDSS simulated and observed datasets are shown in Figure
2.

####  2.2.1 Lightcurve Preprocessing

To preprocess each lightcurve, we first take observations within the range
$t_{\mathrm{peak}}-30\leq t\leq t_{\mathrm{peak}}+150$, where
$t_{\mathrm{peak}}$ is the epoch of SN peak brightness estimated by SNANA.
This is to ensure the lightcurves contain just the SN and no extraneous
information.

We use 2-dimensional Gaussian process (GP) regression to model each SN
lightcurve in time ($t$) and wavelength ($\lambda$) space, then use the
predictions of the fitted GP on a fixed $(\lambda,t)$ grid as the input to our
neural network model. Modelling lightcurves of astronomical transients using
2D Gaussian processes was originally introduced in Boone (2019); and Qu et al.
(2021, Q21) used the GP models to create “images” from lightcurve data, which
is the technique used for this work. Following Q21, we use the Matérn kernel
($\nu=3/2$) with a fixed $6000$Å length scale in wavelength space and fit for
the time length scale as well as the amplitude using maximum likelihood
estimation. This GP model is fit separately to each lightcurve and used to
produce a smooth 2D representation of the lightcurve by predicting flux values
at each point in a $(\lambda,t)$ grid. We choose 32 equally spaced points in
the range $3,000$Å $\leq\lambda\leq 10,100$Å ($\delta\lambda=221.875$Å) and
180 points in the range $t_{\mathrm{peak}}-30\leq t\leq t_{\mathrm{peak}}+150$
($\delta t=1$ day). This produces a $32\times 180$ matrix of predicted flux
values. We also produce a matrix of prediction uncertainties at each
$\lambda_{i},t_{j}$ of equal size.

We stack the flux and uncertainty matrices depthwise to produce a $32\times
180\times 2$ tensor and divide elementwise by the maximum flux value to
constrain all entries to [0,1].

We show low, medium, and high redshift examples of PLAsTiCC lightcurves, the
fitted GP models and their uncertainties, and the resulting flux matrices in
Figure 3. Redshifting increases the wavelength of light, which we see in the
figure as the yellow (high flux) region moving down (toward $\lambda=10,100$Å)
with increasing redshift. We also expect to observe a longer duration for
higher redshift transients, which is evident in the increase in width of the
yellow region. This data format is particularly well suited to the redshift
prediction task, as we are able to visibly see expected physical results of
redshifting in the flux matrices.

####  2.2.2 Redshift Preprocessing

The redshift range for each survey is discretized into $n_{z}$ discrete and
non-overlapping bins ($0.01\leq z\leq 0.4,n_{z}=50$ for SDSS and $0.01\leq
z\leq 1.2,n_{z}=150$ for LSST). We chose these $n_{z}$ values to preserve the
approximate width of each redshift bin across surveys ($\delta z\sim 0.0078$).
The bin corresponding to the true redshift of each SN is one-hot encoded and
passed into the model as the training label.

##  3 Model

Photo-$z$SNthesis is a convolutional neural network model (e.g. LeCun et al.,
1989; Zeiler & Fergus, 2014; Simonyan & Zisserman, 2014; Krizhevsky et al.,
2017) that takes in GP-interpolated lightcurves as well as the GP prediction
uncertainties, prepared as described in §2.2, and predicts a probability
distribution over fine-grained redshift bins. This approach allows us to
produce a discretized full PDF over redshift space for each individual SN
without any assumptions on the underlying distribution, and has been used in a
variety of contexts including image generation (van den Oord et al., 2016) and
prediction of precipitation probabilities (Sønderby et al., 2020). Treating
this as a categorical classification problem using the cross-entropy loss
function has also been shown to accurately approximate Bayesian posterior
probabilities (Richard & Lippmann, 1991).

###  3.1 Convolutional Neural Networks

The convolutional neural network (CNN) is a class of artificial neural network
with properties particularly suited to object and image recognition. It
requires fewer trainable parameters than the standard feedforward network due
to the convolution operation, learning a single weight matrix for small
neighborhoods of the input image. This property is not only parameter-
efficient but also imparts CNNs with translation-equivariance, i.e. the same
feature shifted by $n$ pixels will produce the same response shifted by $n$
pixels. These convolutional layers are paired with pooling layers, which
downsample the input to allow for the next set of convolutional layers to
learn hierarchically more complex features. CNNs are prized in the machine
learning community for being simple yet performant on image recognition
benchmark datasets such as ImageNet (Russakovsky et al., 2015).

###  3.2 Residual Learning

We implement residual learning (He et al., 2016) due to its state-of-the-art
performance on the ImageNet benchmark at the time of publication as well as
its widespread adoption over vanilla CNNs. Residual learning was presented as
a solution to the degradation problem in CNNs, in which performance degraded
past a certain threshold of network depth. Since the additional layers could
simply act as identity mappings and not affect the network performance, He et
al. (2016) decreased the difficulty by not only feeding inputs sequentially
through a series of layers, but also adding the inputs back in to the outputs
of those layers. This allows the layers to learn the zero mapping rather than
the identity mapping. Since these layers are tasked with learning the residual
with respect to the input, this is known as residual learning and the stack of
layers is known as a residual block. The residual connections in the
Photo-$z$SNthesis architecture are shown as lines curving around each residual
block in Figure 4.

###  3.3 Architecture

Figure 4: The model architecture developed for this work, described in the
text in §3.3. A layer shown in the figure as “$h\times d$ conv,
$n_{\mathrm{filters}}$” is a 2D convolutional layer with a kernel size of
$h\times d$ and $n_{\mathrm{filters}}$ filters. Similarly, “$h\times
d,n_{\mathrm{filters}}$ resid block” defines a residual block containing two
$h\times d$ conv, $n_{f}$ convolutional layers. “fc $n_{\mathrm{nodes}}$”
denotes a fully connected layer with $n_{\mathrm{nodes}}$ nodes. The pooling
layers apply max pooling, downsampling both the height and width by 2. The
output softmax layer has $N_{z}$ nodes, $N_{z}=50$ for SDSS and $N_{z}=150$
for LSST.

Our model (Figure 4) takes as input a $32\times 180\times 2$ tensor containing
the GP-interpolated lightcurve data on a $32\times 180$ grid of wavelength and
time values, and the GP uncertainties at each of those points. Table 2 shows
input and output dimensions of example layers in the model architecture. We
also provide the model with the one-hot encoded vector specifying the correct
redshift bin as the training label.

Layer | Input Shape | Output Shape
---|---|---
Zero Padding | $32\times 180\times 2$ | $32\times 182\times 2$
$32\times 3$ Convolutional | $32\times 182\times 2$ | $180\times 32\times 1$
Reshape | $180\times 32\times 1$ | $32\times 180\times 1$
Max Pooling | $32\times 180\times 1$ | $16\times 90\times 1$
Table 2: Description of example layers in the model architecture.

The input is zero-padded on both sides to $32\times 182\times 2$ to ensure
that feature maps output by the convolutional layers retain the original
shape, then passed through a convolutional layer with 32 filters and kernel
size $32\times 3$. Since the convolutional kernel determines the receptive
field of each unit in the layer, we choose a convolutional kernel that spans
the wavelength space to allow each unit to learn from all wavelengths
simultaneously while preserving the linearity of time. The output of this
convolutional layer is $1\times 180\times 32$, which we then reshape back to
$32\times 180\times 1$. This first feature map is now passed through a series
of residual blocks.

Each residual block contains two convolutional blocks, each consisting of a
ReLU nonlinearity, batch normalization, and the zero-padding, convolutional
layer, and reshaping layer identical to the ones described above. The input to
each residual block is then added to the output as described in §3.2. After
each series of three residual blocks, the output is passed through a $2\times
2$ max-pooling layer, downsampling the height and width of the output by a
factor of 2.

Finally, the output is flattened and passed through a fully connected layer
with 1,024 hidden nodes, which connects to the final softmax layer. The nodes
in this layer correspond to redshift bins, thus it has 50 nodes for processing
SDSS data and 150 for LSST data. The array of output probabilities is
interpreted as the probability density over redshifts for our input SN.

###  3.4 Calibration

We also performed temperature scaling (Guo et al., 2017) to ensure that the
probabilities output by Photo-$z$SNthesis are properly calibrated. In this
process, we learn a single “temperature” parameter used to scale the output
probabilities.

Before scaling, the output probabilities $\textbf{p}_{i}$ for input SN $i$ are
derived from the softmax function

$\textbf{p}_{i}^{(k)}=\sigma_{\text{SM}}(\textbf{q}_{i})^{(k)}=\frac{\text{exp}(\textbf{q}_{i}^{(k)})}{\sum_{j=1}^{K}\text{exp}(\textbf{q}_{i}^{(j)})}$
(1)

where $\textbf{q}_{i}$ is the vector of network logits corresponding to SN
$i$, i.e. the output of the final hidden layer of the network. The temperature
parameter $T$ is learned by minimizing the cross-entropy loss between the one-
hot encoded labels and the scaled probabilities, $\textbf{p}^{\prime}_{i}$,

$\textbf{p}^{\prime}_{i}=\sigma_{\text{SM}}(\textbf{q}_{i}/T)$ (2)

Calibration is typically evaluated using reliability diagrams (DeGroot &
Fienberg, 1983; Niculescu-Mizil & Caruana, 2005) and the expected calibration
error statistic (ECE, Naeini et al., 2015). Reliability diagrams show the
prediction accuracy as a function of confidence, which is defined as the
probability associated with the predicted class: $\text{max}(\textbf{p}_{i})$.
A reliability diagram for a perfectly calibrated classifier will show the
identity function, and any deviation from a perfect diagonal is a sign of
miscalibration. The reliability diagram before and after temperature scaling
for the PLAsTiCC model is shown in Figure 5. The scaled probabilities are much
closer to the diagonal, representing a significant improvement in calibration.

ECE is a weighted average of the difference between the accuracy and the
confidence in bins of confidence values. ECE is more precisely defined as

$\text{ECE}=\sum_{m=1}^{M}\frac{|B_{m}|}{N}|\text{acc}(B_{m})-\text{conf}(B_{m})|$
(3)

where $B_{m}$ is the $m^{\text{th}}$ confidence bin and $N$ is the total
number of samples in the dataset. Prior to temperature scaling, the ECE for
the PLAsTiCC model probabilities was 0.24, which improved to 0.08 after
scaling.

Figure 5: Reliability diagram showing PLAsTiCC model calibration before and
after temperature scaling.

###  3.5 Implementation Details

We split both the SDSS and LSST datasets into 90% training, 5% validation, and
5% test datasets and trained both models with batch sizes of 2048 for 750
epochs. We minimize the cross-entropy loss function with the Adam optimizer
(Kingma & Ba, 2014) with an initial learning rate of 1e-3 that is halved after
25 epochs of no improvement in the validation loss. The total number of
trainable parameters in the SDSS model is 451,654 and 554,154 for the LSST
model due to the difference in number of redshift bins and resulting
difference in output layer size. To prevent overfitting, we use a weight decay
of 1e-3 as well as dropout layers (Hinton et al., 2012). Calibration was
performed with the validation set of both datasets and with an initialization
of $T=1$. We minimize a cross-entropy loss function using Adam and find
$T_{\text{SDSS}}=0.82$ and $T_{\text{LSST}}=0.64$.

Figure 6: Examples of a high accuracy ($\Delta z<0.005$, top), medium accuracy
($0.01\leq\Delta z\leq 0.05$, middle), and outlier ($\Delta z>0.05$, bottom)
lightcurve and Gaussian process fit from the SDSS spectroscopic sample and
their predicted PDFs. The true (spectroscopic) redshift is shown in the black
dotted line, while the predicted redshift (mean(PDF)) is shown in orange. The
medium and poor accuracy lightcurves have larger errors and scatter than the
high accuracy lightcurve, and the low accuracy lightcurve appears to have much
fewer points.

##  4 Results and Discussion

Examples of lightcurves and their corresponding predicted redshift PDFs are
shown in Figure 6. We chose spectroscopically confirmed SDSS SNe Ia
lightcurves, and selected a high accuracy ($\Delta z<0.005,\Delta
z\equiv\frac{z_{\mathrm{pred}}-z_{\mathrm{true}}}{1+z_{\mathrm{true}}}$),
medium accuracy ($0.01\leq\Delta z\leq 0.05$), and an outlier ($\Delta
z>0.05$) example.

###  4.1 Evaluation Metrics and Basis for Comparison

####  4.1.1 Point Estimates and Metrics

Although full photometric redshift PDFs are preferred for further statistical
analyses (i.e. cosmological analyses), point estimates can also be computed
from each PDF. We require these point estimates to evaluate our model
performance against the true redshift values. We compare two possible methods
to condense a PDF into a point estimate:

$\text{mean(PDF)}=\frac{1}{2}\sum_{i}p(Z_{i})\frac{\lceil
Z_{i}\rceil^{2}-\lfloor Z_{i}\rfloor^{2}}{\lceil Z_{i}\rceil-\lfloor
Z_{i}\rfloor}$ (4)

where $Z$ represents the vector of redshift bins, $\lceil Z_{i}\rceil$ the
right edge of bin $i$, $\lfloor Z_{i}\rfloor$ the left edge of bin $i$, and
$p(Z_{i})$ the output probability assigned by the model to bin $i$; and

$\text{max(PDF)}=\tilde{Z}_{\text{argmax}_{i}(p(Z_{i}))}\\\ $ (5)

where $\tilde{Z}$ represents the array of midpoints of the redshift bins $Z$.
Taking the weighted mean is a common summary statistic for PDFs, while taking
the bin with maximum probability as the predicted output is typical for
classification tasks. As shown in Tables 3 and 4 and Figures 7 and 9, the two
methods give similar results and thus we use the mean point estimate for
Figures 8, 10, 11, 12, and 13.

We compute the following metrics for both of our point estimates, following
e.g. Pasquet et al. (2018):

  * •

the residuals $\Delta
z\equiv\frac{z_{\mathrm{pred}}-z_{\mathrm{true}}}{1+z_{\mathrm{true}}}$,

  * •

the bias $\langle\Delta z\rangle$,

  * •

the mean absolute deviation
$\sigma_{\mathrm{MAD}}=1.4826\times\text{median}(|\Delta
z-\text{median}(\Delta z)|)$,

  * •

the fraction of outliers $\eta$ with $|\Delta z|>0.05$.

Figure 7: Mean binned residuals $\Delta z$ as a function of true redshift
$z_{\mathrm{true}}$ for the PLAsTiCC simulated SNe Ia test set. Predictions
from our model have much lower biases as well as scatter compared to
predictions from LCFIT+Z. The max(PDF) and mean(PDF) point estimates for our
model also agree quite well, resulting in similar $\Delta z$ values. Dotted
lines are plotted at $\Delta z=\pm 0.02$ for reference. Figure 8: Predicted
vs. true redshifts for the PLAsTiCC simulated SNe Ia sample colored by the
fraction of each sample represented by each point. (left) Predictions from
Photo-$z$SNthesis described in this work, (right) predictions from LCFIT+Z.

####  4.1.2 Comparison with LCFIT+Z

LCFIT+Z (Kessler et al., 2010) is an extension of the lightcurve fitting code
described in §2.1 that treats redshift as an additional free parameter. It
determines the best fit lightcurve parameters by minimizing $\chi^{2}$ values
of the observed and model fluxes, which are computed as a function of the free
parameters (4 lightcurve parameters $t_{\mathrm{peak}}$, stretch $x_{1}$,
color $c$, flux normalization $x_{0}$; and redshift $z_{\mathrm{phot}}$).
LCFIT+Z is actively used for recent and ongoing experiments performing full
cosmological analyses with photometric redshifts (e.g. Dai et al., 2018; Mitra
et al., 2022). LCFIT+Z produces point estimates with uncertainties as opposed
to a full PDF, so we perform comparisons using our point estimates.

We compare our results with LCFIT+Z as opposed to other SN photometric
redshift estimators (e.g., Wang et al., 2015; de Oliveira et al., 2022) due to
its demonstrated performance, widespread use, and integration into SNANA. We
run LCFIT+Z on the same datasets used to evaluate our model, enabling a direct
comparison. Note that most documented uses of LCFIT+Z place a prior on the
redshift fit using the redshift of the SN host galaxy, resulting in much more
constrained fits. However, since (1) host spectroscopic redshifts are
unavailable in future survey environments, and (2) to draw direct comparisons
with Photo-$z$SNthesis, which does not require any host galaxy information, we
omit the host galaxy redshift prior when running LCFIT+Z. In order to compare
how these methods perform using all available information in the photometric
survey era, we test Photo-$z$SNthesis and LCFIT+Z with host galaxy photometric
redshift priors in Section 4.4.2.

###  4.2 LSST (PLAsTiCC) Results

We evaluate our model and our baseline for comparison, LCFIT+Z, on a test set
of 4,057 simulated PLAsTiCC-like lightcurves with true redshift distribution
shown in the right panel of Figure 1. The values of the evaluation metrics for
results from this work as well as LCFIT+Z are shown in Table 3. The two
methods for obtaining point estimates from Photo-$z$SNthesis PDFs, mean and
max, give similar results, though the mean point estimates have a degraded
$\sigma_{\mathrm{MAD}}$ value compared to the max point estimates but a
smaller outlier rate. However, the differences between mean and max point
estimates are orders of magnitude smaller than the improvement we see relative
to LCFIT+Z. The mean gives a $\sim 3.2\times$ larger result than max on
$\sigma_{\mathrm{MAD}}$, compared with a $\sim 180\times$ larger result from
LCFIT+Z.

Metric | this work | LCFIT+Z
---|---|---
| mean | max |
bias $\langle\Delta z\rangle$ | 0.00095 | 0.00075 | 0.058
$\sigma_{\mathrm{MAD}}$ | 0.0081 | 0.0025 | 0.0450
outlier rate $\eta$ | 3.87% | 4.24% | 32.3%
Table 3: Evaluation metrics computed for the PLAsTiCC test dataset for both
the mean(PDF) and max(PDF) point estimates for our model as well as LCFIT+Z.
The best result for each metric is shown in bold. Figure 9: Mean binned
residuals $\Delta z$ as a function of true redshift $z_{\mathrm{true}}$ for
the SDSS simulated and real SNe Ia samples. Predictions from this work have
much lower biases as well as scatter compared to predictions from LCFIT+Z. We
also show that two common methods of condensing redshift PDFs into point
estimates, max(PDF) and mean(PDF), resulting in similar errors for our model.
Dotted lines are plotted at $\Delta z=\pm 0.02$ for reference.

We show the mean binned residuals, $\Delta z$, as a function of true redshift
for our model and LCFIT+Z in Figure 7. We see that while the residuals of our
model and LCFIT+Z match quite well up to $z_{\mathrm{true}}\sim 0.4$, LCFIT+Z
has a tendency to dramatically overestimate at higher redshifts. We also see
that the spread within each residual bin in LCFIT+Z results is much larger, as
shown in the size of the error bars on each point. In contrast, our model
shows a relatively flat $\Delta z$ over the full redshift range with minimal
redshift-dependent bias, which has not been achieved by other SN photometric
redshift estimators.

Metric | SDSS simulated | SDSS real
---|---|---
| this work (mean) | this work (max) | LCFIT+Z | this work (mean) | this work (max) | LCFIT+Z
bias $\langle\Delta z\rangle$ | 7.8e-5 | 0.00073 | 0.023 | 0.0050 | 0.0052 | 0.027
$\sigma_{\mathrm{MAD}}$ | 0.011 | 0.010 | 0.028 | 0.018 | 0.020 | 0.047
outlier rate $\eta$ | 0.85% | 1.16% | 19.9% | 5.14% | 5.78% | 26.1%
Table 4: Evaluation metrics computed for the SDSS simulated and real test
datasets for both the mean(PDF) and max(PDF) point estimates for our model as
well as LCFIT+Z. The best result for each metric and dataset is shown in bold.

Figure 8 more clearly shows the spread of predicted redshifts, where
$z_{\mathrm{pred}}$ is calculated as the mean(PDF) for our model. The
predictions produced by our model (left panel) lie much closer to the
$z_{\mathrm{pred}}=z_{\mathrm{true}}$ line with approximately equal amounts of
scatter on either side, reinforcing the minimal redshift-dependent bias shown
in Figure 7. The dark red area along the $z_{\mathrm{pred}}=z_{\mathrm{true}}$
line also indicates that most of the sample is localized there. The LCFIT+Z
results (right panel) have much larger spread and little localization, as
evidenced by the lack of red in the plot.

###  4.3 SDSS Results

Figure 10: Predicted vs. true redshifts for the SDSS simulated and real SNe Ia
samples colored by the fraction of each sample represented by each point. (Top
row) Predictions from Photo-$z$SNthesis described in this work, (bottom row)
predictions from LCFIT+Z.  Figure 11: Predicted redshift error, $\Delta z$, as
a function of true redshift, $z_{\mathrm{true}}$, for the SDSS real SNe Ia
sample with and without bias correction computed from the simulated results.
The point estimates used in this figure are computed using the mean(PDF)
method. Dotted lines are plotted at $\Delta z=\pm 0.02$ for reference.

We evaluate our model and LCFIT+Z on a test set of 5,274 simulated SDSS
lightcurves and 489 real observed SDSS lightcurves with true redshift
distributions shown in the right panel of Figure 2. Table 4 shows the
evaluation metrics calculated with the mean and max point estimates on the
simulated and observed SDSS datasets, as well as the LCFIT+Z point estimates.
With the SDSS datasets, the similarity between the mean and max point
estimates is more pronounced. We also note that though our model’s performance
degrades slightly between the simulated test set and the observed test set,
all metrics still strongly favor our model as the better performer compared to
LCFIT+Z. However, the $\sigma_{\mathrm{MAD}}$ values differ less between our
model and LCFIT+Z compared to the PLAsTiCC test dataset, only offering a
$2\times$ improvement as opposed to a $55-180\times$ improvement for the
PLAsTiCC test dataset. This could be due to the narrower redshift range of
SDSS, as the LCFIT+Z performance on the PLAsTiCC dataset degrades
significantly after $z_{\mathrm{true}}\sim 0.4$.

The mean binned residuals for our model and LCFIT+Z evaluated on the simulated
and real SDSS datasets are shown in Figure 9. The residuals from both models
are much more constrained for SDSS than PLAsTiCC, as expected from the less
significant differences in evaluation metrics. LCFIT+Z still exhibits larger
mean $\Delta z$ values for both the simulated and real datasets as well as a
larger spread in each bin. Both models generalize relatively well from
simulations to real data, with our mean residuals staying within $|\Delta
z|<0.02$ and LCFIT within $|\Delta z|<0.05$; however, an overall increase in
$|\Delta z|$ values is noticeable between simulated and real data.

We show the comparison between predicted and true redshifts in Figure 10 for
the same datasets used in Figure 9. Our model (top row) clearly produces more
constrained predictions, as they lie much closer to the
$z_{\mathrm{true}}=z_{\mathrm{pred}}$ line with a high density of points
(shown in red) along the line. In contrast with the PLAsTiCC results, in which
LCFIT+Z performed relatively well at lower redshifts, the performance over the
full SDSS redshift range is poor.

####  4.3.1 Bias Correction

Simulations not only allow deep learning methods such as Photo-$z$SNthesis to
train on large datasets that would be infeasible with real data alone, but
also give valuable estimates of the biases produced from those models that can
be used to correct the results on real data. Here, we test this bias
correction method on the results of Photo-$z$SNthesis on the SDSS simulated
and real data. We compute the average $\Delta z$ values for the simulated SDSS
sample in 10 bins of $z_{\mathrm{true}}$ values and bin the real SDSS sample
in the same way. We then subtract the simulated $\Delta z$ value associated
with the bin of each real $z_{\mathrm{pred}}$ estimate to produce the
corrected curve in Figure 11. Specifically, the corrected value of the
$i^{\text{th}}$ SN in the real SDSS dataset belonging to bin $m$ is computed
as

$z_{\text{corrected},i}=z_{i}-\langle\Delta z_{\text{sim},m}\rangle$ (6)

where $z_{\text{corrected},i}$ is the corrected prediction, $z_{i}$ is the
uncorrected prediction, and $\langle\Delta z_{\text{sim},m}\rangle$ is the
average $\Delta z$ from the simulated SNe in bin $m$. The corrected $\Delta z$
values are computed as defined in §4.1.1,

$\Delta
z_{\text{corrected}}=\frac{z_{\text{corrected}}-z_{\text{true}}}{1+z_{\text{true}}}$
(7)

The corrected curve exhibits smaller biases than the uncorrected curve,
showing that this bias correction method is valid in cases where the
simulation is sufficiently representative of the real data.

###  4.4 Further Experiments

####  4.4.1 Using Photo-$z$SNthesis photo-zs for Cosmology

Figure 12: An illustration of our estimated cosmological biases arising from
photo-$z$ errors. (top) A Hubble diagram zoomed in to the neighborhood of
$z_{\text{model}}=1$ showing a fiducial cosmology (green:
$w=-1,\Omega_{m}=0.3$) and two biased cosmologies (pink: $\Delta w=-0.03$,
orange: $\Delta w=+0.03$). The dashed black line shows the distance modulus
value at $z_{\text{model}}=1$ for the fiducial cosmology,
$\mu_{\mathrm{model}}(z_{\text{model}}=1)$. The two dotted lines show the
redshifts that correspond to the value of
$\mu_{\mathrm{model}}(z_{\text{model}}=1)$ for the two biased cosmologies,
i.e. $\mu_{\text{model}}(z_{\text{model}}=1)=\mu_{\Delta
w=-0.03}(z=0.99528)=\mu_{\Delta w=+0.03}(z=1.00486)$. The values of these
redshifts are labeled on the redshift axis. We approximate the redshift error
required to create a bias of, e.g. $\Delta w=+0.03$, as the difference between
the biased and fiducial redshift values,
$dz_{\text{bias}}=1-1.00486=-0.00486$. (bottom) $\Delta z=dz/(1+z)$ values for
various choices of $\Delta w$, compared to the mean binned residuals $\Delta
z$ produced by the mean point estimates of Photo-$z$SNthesis PDFs (pink,
reproduced from Figure 7).

Though producing a full cosmological analysis using photo-$z$s predicted by
Photo-$z$SNthesis is outside the scope of this work, we provide some intuition
for the quality of our estimates in the context of cosmology. The redshift
error $dz_{\text{bias}}$ that results in a $w$ bias of $\Delta w$ at a
particular redshift, $z_{\text{model}}$, can be approximated by the redshift
difference

$dz_{\text{bias}}=z_{\text{model}}-z_{\Delta w}$ (8)

where $z_{\Delta w}$ is the redshift associated with the distance modulus
$\mu_{\text{model}}(z_{\text{model}})$ for the biased cosmology, i.e.

$\mu_{\text{model}}(z_{\text{model}})=\mu_{\Delta w}(z_{\Delta w})$ (9)

where $\mu_{\text{model}},\mu_{\Delta w}$ are the distance moduli associated
with the model and biased cosmologies, respectively.

A concrete example of this is illustrated in the top panel of Figure 12, which
shows a zoomed-in portion of the Hubble diagram in the neighborhood of our
chosen $z_{\text{model}}=1$ for our fiducial cosmology ($w=-1,\Omega_{m}=0.3$)
as well as examples of biases on $w$ ($\Delta w\pm 0.03$) plotted on either
side. Our choice of $z_{\text{model}}=1$ is motivated by the importance of
high redshift SNe on the constraining power on $w$, so we compare with our
PLAsTiCC results. We choose to focus on the $\Delta w=+0.03$ cosmology as our
redshifts are slightly underestimated at $z=1$. In this example, $z_{\Delta
w=+0.03}=1.00486$ at $z_{\text{model}}=1$ is shown as the dotted vertical line
on the right. We approximate $dz_{\text{bias}}$ for $\Delta w=+0.03$ at
$z_{\text{model}}=1$ to be

$dz_{\text{bias}}=z_{\text{model}}-z_{\Delta w=+0.03}=1-1.00486=-0.00486$ (10)

This corresponds to half the distance between the dotted vertical lines in the
figure. The mean redshift error for our results on the PLAsTiCC dataset at
$z_{\text{true}}=1$ are

$\Delta z=\frac{z_{\text{pred}}-z_{\text{true}}}{1+z_{\text{true}}}=-0.002$
(11)

translating to an expected redshift difference of

$z_{\text{pred}}-z_{\text{true}}=(1+z_{\text{true}})\cdot-0.002=-0.004$ (12)

This is below the expected $dz_{\text{bias}}$ associated with a $w$ shift of
$\Delta w=+0.03$.

The bottom panel of Figure 12 shows $\Delta z$ values for different choices of
cosmological biases (here we choose to vary $w$) across the full LSST redshift
range, along with the $\Delta z$ values produced by Photo-$z$SNthesis on the
PLAsTiCC dataset. Note that the $y$ axis of this plot is $\Delta z=dz/(1+z)$,
i.e. $dz_{\text{bias}}$ normalized by $(1+z_{\text{model}})$, for ease of
comparison with Photo-$z$SNthesis mean residual $\Delta z$ values. This
comparison over all redshifts shows that Photo-$z$SNthesis redshifts lack the
precision to constrain cosmology to $\Delta w=\pm 0.03$, since the pink line
representing Photo-$z$SNthesis redshift errors mostly does not lie in the area
within the $\Delta w=\pm 0.03$ teal lines. However, this high-level analysis
is an overestimate of the expected impact on cosmology, and a thorough
cosmological analysis with Photo-$z$SNthesis photo-$z$’s will be the subject
of a future work.

####  4.4.2 Comparison with LCFIT+Z with Host Galaxy Redshift Prior

In §4.2 and §4.3, we showed results for Photo-$z$SNthesis and LCFIT+Z with no
host galaxy information included. Photo-$z$SNthesis was formulated not to
require a host redshift prior in order to prevent biases due to incorrect host
redshifts. However, in a context in which host mismatches are rare and host
redshifts are reliable, we want to use all available information to produce
the most constrained SN photo-$z$ estimates.

We test both Photo-$z$SNthesis and LCFIT+Z with a prior on $z_{\text{pred}}$,
the predicted redshift, given by the host galaxy photometric redshift
estimate. We choose to use photometric redshifts as opposed to spectroscopic
to more closely emulate a future scenario in which most host galaxies will not
have spectroscopic information available.

We model this prior, $P(Z_{\text{host}})$, as a Gaussian centered on the
photo-$z$ of the host galaxy, $z_{\text{host}}$, and use the estimated
uncertainty, $\sigma_{z_{\text{host}}}$, as the standard deviation:

$Z_{\text{host}}\sim\mathcal{N}(z_{\text{host}},\sigma_{z_{\text{host}}})$
(13)

We treat Photo-$z$SNthesis PDFs, $P(Z_{\text{Photo-$z$SN}})$, as Bayesian
posteriors and apply the host prior using Bayes’ theorem:

$\displaystyle
P(Z_{\text{pred}})=\frac{P(Z_{\text{Photo-$z$SN}}|Z_{\text{host}})P(Z_{\text{host}})}{P(Z_{\text{Photo-$z$SN}})}$
(14)
$\displaystyle=\frac{P(Z_{\text{Photo-$z$SN}})P(Z_{\text{host}})}{\sum_{i}P(Z_{\text{Photo-$z$SN},i})P(Z_{\text{host},i})}$

simplified by the fact that $P(Z_{\text{host}})$ and
$P(Z_{\text{Photo-$z$SN}})$ are statistically independent.

LCFIT+Z uses a Markov chain Monte Carlo (MCMC) process to sample from the
posterior distribution over a 5-dimensional parameter space: 4 SALT fit
parameters color $c$, stretch-luminosity parameter $x_{1}$, time of peak
brightness $t_{0}$, flux normalization parameter $x_{0}$; and redshift
$z_{\text{phot}}$. The host prior constrains the search space, resulting in a
higher likelihood of convergence to a global minimum.

In Figure 13, we show $\Delta z$ values for the SDSS photometric SNe Ia
dataset for Photo-$z$SNthesis and LCFIT+Z with and without a host photo-$z$
prior. The $\Delta z$ results with no host prior are identical to those in the
right panel of Figure 9. The host prior noticeably improves LCFIT+Z results on
$z_{\text{true}}<0.2$ and $z_{\text{true}}>0.3$, as well as the scatter across
the full redshift range, shown in the smaller error bars. The
Photo-$z$SNthesis results with and without host prior are very similar,
however, as the host prior is often less constraining than the
Photo-$z$SNthesis PDFs themselves. We show the bias, $\sigma_{\text{MAD}}$,
and outlier rate for both models with and without the host prior in Table 5.
These results show that Photo-$z$SNthesis outperforms LCFIT+Z even in the
presence of a host galaxy redshift prior.

Metric | this work | LCFIT+Z
---|---|---
| no prior | with prior | no prior | with prior
bias $\langle\Delta z\rangle$ | 0.0050 | 0.0061 | 0.027 | 0.021
$\sigma_{\mathrm{MAD}}$ | 0.018 | 0.017 | 0.057 | 0.025
outlier rate $\eta$ | 5.14% | 4.93% | 26.1% | 17.6%
Table 5: Evaluation metrics computed for the SDSS observed photometric SNe Ia
sample both with and without a host galaxy photo-$z$ prior. The best results
for each metric and dataset are shown in bold. Figure 13: Mean binned
residuals, $\Delta
z\equiv\frac{z_{\mathrm{pred}}-z_{\mathrm{true}}}{1+z_{\mathrm{true}}}$, as a
function of true redshift, $z_{\mathrm{true}}$, for the SDSS observed
photometric SNe Ia sample. Errors from both Photo-$z$SNthesis and LCFIT+Z are
shown with (darker) and without (lighter) a prior on $z_{\text{pred}}$ from
the host galaxy photometric redshift. The lighter (no prior) curves are
identical to those in Figure 9, with the max(PDF) curves omitted for clarity.

####  4.4.3 Balancing Redshift Classes

Most machine learning-based classification models perform best when trained on
a dataset with balanced classes, i.e. a dataset with the same number of
examples for each class. In this case, this translates to a flat redshift
distribution. Due to the interplay between the SN volumetric rate, which
increases with redshift, and survey detection efficiency, which drops off at a
survey-dependent magnitude threshold, observed redshift distributions
typically exhibit a “humped” pattern with very few low- and high-redshift SNe
(see e.g. Figures 1 and 2). This effectively allows the model to learn a prior
on the distribution that would encourage predictions biased toward more
populated redshift bins, i.e. overestimating low-redshift SNe and
underestimating high-redshift SNe. We presented our nominal results with
Photo-$z$SNthesis trained on an unmodified redshift distribution, but provide
a discussion on the possibility of improved performance when trained on a flat
redshift distribution.

Demonstrating the performance of Photo-$z$SNthesis trained on a flat redshift
distribution requires either selecting a very small percentage of a large
suite of simulations to produce the same number of SNe in each redshift bin,
or a simulation with a manually tuned flat redshift distribution that ignores
physical constraints such as the survey detection efficiency. We demonstrate
the former and call the resulting dataset $D_{\text{flat}}$.
$N_{D_{\text{flat}}}$, the number of SNe in $D_{\text{flat}}$, is much lower
than that of our nominal training set since $D_{\text{flat}}$ is produced from
the same SDSS simulations described in §2.1.2. Since models trained with
datasets of different sizes can produce very different results due to effects
such as overfitting, we control for dataset size by producing a second dataset
of $N_{D_{\text{flat}}}$ SNe with an unmodified redshift distribution,
$D_{\text{observed}}$, which serves as a baseline for comparison.

Specifically, we select 200 simulated SDSS SNe Ia from each of the 50 redshift
bins to produce $D_{\text{flat}}$, resulting in a dataset of
$N_{D_{\text{flat}}}=10,000$ SNe. We select 10,000 SNe from the same SDSS
simulations at random to produce $D_{\text{observed}}$, which mirrors the
original unmodified redshift distribution. We also modified the final hidden
layer of the model to contain 256 nodes instead of 1,024 to accommodate for
the smaller dataset sizes. $\Delta z$ vs. $z_{\text{true}}$ for models trained
on both datasets are shown in Figure 14. The model trained on
$D_{\text{observed}}$ (i.e. with no redshift balancing) underpredicts more
often at higher redshifts, resulting in larger $\Delta z$ values for
$z_{\mathrm{true}}>0.25$ for both simulated and real data. This indicates that
the small redshift-dependent bias in our primary results (e.g. Figure 9) could
indeed be alleviated through balancing classes. However, creating a balanced
training set the size of the one used for Figure 9 would require a $10\times$
larger set of simulations, which would require hundreds of CPU hours. Thus, we
do not perform redshift balancing for our primary results.

Figure 14: Predicted redshift error, $\Delta
z\equiv\frac{z_{\mathrm{pred}}-z_{\mathrm{true}}}{1+z_{\mathrm{true}}}$ as a
function of true redshift, $z_{\mathrm{true}}$, for 1,000 of the SDSS
simulated and real SNe Ia samples with and without redshift balancing. The “no
balance” result is not the same as in Figure 9; instead, we created a separate
smaller unbalanced training set in order to match the size of the balanced
training set. See §4.4.3 for details.

##  5 Conclusions

In this work we presented Photo-$z$SNthesis, a convolutional neural network
model that predicts full redshift PDFs from multi-band photometric SNe Ia
lightcurves. We evaluated its performance on simulated SDSS and LSST
lightcurves, as well as a photometrically confirmed SDSS SNe Ia sample. We
compared our results against LCFIT+Z, the most frequently used photometric
redshift estimation method for SNe, and showed superior performance across all
evaluation metrics. Our model also exhibits minimal redshift-dependent bias,
which has plagued redshift estimators in the past, and generalizes well
between simulated and observed data.

Though Photo-$z$SNthesis does not require host galaxy information to produce
accurate SN photo-$z$ estimates, host galaxy redshifts can be incorporated as
a prior to further improve our predictions. We tested Photo-$z$SNthesis and
LCFIT+Z with a host galaxy photometric redshift prior to demonstrate the
performance of these models in the LSST era when host galaxy spectroscopic
redshifts will not be widely available. We show that while the host galaxy
prior improves LCFIT+Z estimates, Photo-$z$SNthesis still produces more
accurate photo-$z$ point estimates as well as constrained PDFs.

We also investigated the impact of the training set redshift distribution on
model predictions, and concluded that a flat redshift distribution, i.e.
balanced training data, may further reduce the underestimation of high-
redshift SNe. However, since balanced datasets are a small subset of the full
simulation, creating a large balanced dataset would require prohibitively
large simulations.

We envision Photo-$z$SNthesis to be useful for many tasks in supernova
science, including precise volumetric rate calculations, discovery of
incorrect host galaxy matches via redshift discrepancies, and photometric SNe
Ia cosmology. We briefly explored the cosmological constraints that can be
expected from Photo-$z$SNthesis photo-$z$s and concluded that the bias on $w$
may be on the order of $\Delta w\sim 0.1$.

In future work, we intend to develop a framework for incorporating redshift
PDFs and their associated uncertainties into the Hubble diagram and produce
more accurately modeled cosmological constraints from Photo-$z$SNthesis
redshift PDFs. We also plan to explore the use of Photo-$z$SNthesis redshift
predictions as a method of identifying host confusion and potential mismatches
for LSST. Though Photo-$z$SNthesis was developed for SNe Ia with cosmological
applications in mind, the data processing and model architecture can be
generalized for use with any astronomical time-domain events.

Accurate photometric redshift estimation for SNe Ia will become vital in the
imminent era of photometric SN Ia cosmology. We believe that the approach and
model presented here will allow us to maximize the constraining power of these
new datasets.

## acknowledgments

H.Q. and M.S. were supported by DOE grant DE-FOA-0002424 and NSF grant
AST-2108094. The authors would like to thank Sang Michael Xie for insightful
discussions on the network architecture and training process, Brodie Popovic
for providing SDSS simulations, Rebecca Chen for helpful discussions and
LCFIT+Z setup, Rick Kessler for assisting with LCFIT+Z, SNANA, and LSST
simulations, and Carles Sánchez and Jaemyoung Lee for helpful discussions.

This research used resources of the National Energy Research Scientific
Computing Center (NERSC), a U.S. Department of Energy Office of Science User
Facility located at Lawrence Berkeley National Laboratory, operated under
Contract No. DE-AC02-05CH11231. This work was completed in part with resources
provided by the University of Chicago’s Research Computing Center.

## References

  * Abbott et al. (2019) Abbott, T. M. C., Allam, S., Andersen, P., et al. 2019, The Astrophysical Journal, 872, L30, doi: 10.3847/2041-8213/ab04fa
  * Benitez (2000) Benitez, N. 2000, The Astrophysical Journal, 536, 571, doi: 10.1086/308947
  * Betoule et al. (2014) Betoule, M., Kessler, R., Guy, J., et al. 2014, A&A, 568, A22, doi: 10.1051/0004-6361/201423413
  * Boone (2019) Boone, K. 2019, The Astronomical Journal, 158, 257, doi: 10.3847/1538-3881/ab5182
  * Brout et al. (2022) Brout, D., Scolnic, D., Popovic, B., et al. 2022, The Astrophysical Journal, 938, 110, doi: 10.3847/1538-4357/ac8e04
  * Brunner et al. (1997) Brunner, R. J., Connolly, A. J., Szalay, A. S., & Bershady, M. A. 1997, The Astrophysical Journal, 482, L21, doi: 10.1086/310674
  * Buchs et al. (2019) Buchs, R., Davis, C., Gruen, D., et al. 2019, Monthly Notices of the Royal Astronomical Society, 489, 820, doi: 10.1093/mnras/stz2162
  * Chen et al. (2022) Chen, R., Scolnic, D., Rozo, E., et al. 2022, ApJ, 938, 62, doi: 10.3847/1538-4357/ac8b82
  * Dai et al. (2018) Dai, M., Kuhlmann, S., Wang, Y., & Kovacs, E. 2018, Monthly Notices of the Royal Astronomical Society, 477, 4142, doi: 10.1093/mnras/sty965
  * de Oliveira et al. (2022) de Oliveira, F. M. F., Vargas dos Santos, M., & Reis, R. R. R. 2022, Mon. Not. Roy. Astron. Soc., 518, 2385, doi: 10.1093/mnras/stac3202
  * DeGroot & Fienberg (1983) DeGroot, M. H., & Fienberg, S. E. 1983, Journal of the Royal Statistical Society. Series D (The Statistician), 32, 12.  http://www.jstor.org/stable/2987588
  * D´Isanto, A. & Polsterer, K. L. (2018) D´Isanto, A., & Polsterer, K. L. 2018, A&A, 609, A111, doi: 10.1051/0004-6361/201731326
  * Frieman et al. (2008) Frieman, J. A., Bassett, B., Becker, A., et al. 2008, AJ, 135, 338, doi: 10.1088/0004-6256/135/1/338
  * Guo et al. (2017) Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. 2017, On Calibration of Modern Neural Networks.  https://arxiv.org/abs/1706.04599
  * Guy et al. (2007) Guy, J., Astier, P., Baumont, S., et al. 2007, A&A, 466, 11, doi: 10.1051/0004-6361:20066930
  * He et al. (2016) He, K., Zhang, X., Ren, S., & Sun, J. 2016, in Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778
  * Hinton et al. (2012) Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. 2012, Improving neural networks by preventing co-adaptation of feature detectors, arXiv, doi: 10.48550/ARXIV.1207.0580
  * Hounsell et al. (2018) Hounsell, R., Scolnic, D., Foley, R. J., et al. 2018, ApJ, 867, 23, doi: 10.3847/1538-4357/aac08b
  * Ivezić et al. (2019) Ivezić, Ž., Kahn, S. M., Tyson, J. A., et al. 2019, ApJ, 873, 111, doi: 10.3847/1538-4357/ab042c
  * Kenworthy et al. (2021) Kenworthy, W. D., Jones, D. O., Dai, M., et al. 2021, The Astrophysical Journal, 923, 265, doi: 10.3847/1538-4357/ac30d8
  * Kessler et al. (2009) Kessler, R., Bernstein, J. P., Cinabro, D., et al. 2009, PASP, 121, 1028, doi: 10.1086/605984
  * Kessler et al. (2010) Kessler, R., Cinabro, D., Bassett, B., et al. 2010, The Astrophysical Journal, 717, 40, doi: 10.1088/0004-637x/717/1/40
  * Kessler et al. (2013) Kessler, R., Guy, J., Marriner, J., et al. 2013, ApJ, 764, 48, doi: 10.1088/0004-637X/764/1/48
  * Kessler et al. (2019) Kessler, R., Narayan, G., Avelino, A., et al. 2019, arXiv e-prints, arXiv:1903.11756.  https://arxiv.org/abs/1903.11756
  * Kim & Miquel (2007) Kim, A. G., & Miquel, R. 2007, Astroparticle Physics, 28, 448, doi: 10.1016/j.astropartphys.2007.08.009
  * Kingma & Ba (2014) Kingma, D. P., & Ba, J. 2014, arXiv e-prints, arXiv:1412.6980.  https://arxiv.org/abs/1412.6980
  * Krizhevsky et al. (2017) Krizhevsky, A., Sutskever, I., & Hinton, G. E. 2017, Communications of the ACM, 60, 84
  * LeCun et al. (1989) LeCun, Y., Boser, B., Denker, J., et al. 1989, in Advances in Neural Information Processing Systems, ed. D. Touretzky, Vol. 2 (Morgan-Kaufmann).  https://proceedings.neurips.cc/paper/1989/file/53c3bce66e43be4f209556518c2fcb54-Paper.pdf
  * LSST Science Collaboration et al. (2009) LSST Science Collaboration, Abell, P. A., Allison, J., et al. 2009, arXiv e-prints, arXiv:0912.0201.  https://arxiv.org/abs/0912.0201
  * Mitra et al. (2022) Mitra, A., Kessler, R., More, S., Hlozek, R., & Collaboration, T. L. D. E. S. 2022, Using Host Galaxy Photometric Redshifts to Improve Cosmological Constraints with Type Ia Supernova in the LSST Era, arXiv, doi: 10.48550/ARXIV.2210.07560
  * Möller & de Boissière (2020) Möller, A., & de Boissière, T. 2020, MNRAS, 491, 4277, doi: 10.1093/mnras/stz3312
  * Naeini et al. (2015) Naeini, M. P., Cooper, G. F., & Hauskrecht, M. 2015, in Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI’15 (AAAI Press), 2901–2907
  * Niculescu-Mizil & Caruana (2005) Niculescu-Mizil, A., & Caruana, R. 2005, in Proceedings of the 22nd International Conference on Machine Learning, ICML ’05 (New York, NY, USA: Association for Computing Machinery), 625–632, doi: 10.1145/1102351.1102430
  * Palanque-Delabrouille et al. (2010) Palanque-Delabrouille, N., Ruhlmann-Kleider, V., Pascal, S., et al. 2010, Astronomy and Astrophysics, 514, A63, doi: 10.1051/0004-6361/200913283
  * Pasquet et al. (2018) Pasquet, J., Bertin, E., Treyer, M., Arnouts, S., & Fouchez, D. 2018, Astronomy & Astrophysics, 621, A26, doi: 10.1051/0004-6361/201833617
  * Perlmutter et al. (1999) Perlmutter, S., Aldering, G., Goldhaber, G., et al. 1999, ApJ, 517, 565, doi: 10.1086/307221
  * Pierel et al. (2018) Pierel, J. D. R., Rodney, S., Avelino, A., et al. 2018, PASP, 130, 114504, doi: 10.1088/1538-3873/aadb7a
  * Popovic et al. (2020) Popovic, B., Scolnic, D., & Kessler, R. 2020, ApJ, 890, 172, doi: 10.3847/1538-4357/ab6deb
  * Qu et al. (2021) Qu, H., Sako, M., Möller, A., & Doux, C. 2021, AJ, 162, 67, doi: 10.3847/1538-3881/ac0824
  * Richard & Lippmann (1991) Richard, M. D., & Lippmann, R. P. 1991, Neural Computation, 3, 461, doi: 10.1162/neco.1991.3.4.461
  * Riess et al. (1998) Riess, A. G., Filippenko, A. V., Challis, P., et al. 1998, AJ, 116, 1009, doi: 10.1086/300499
  * Russakovsky et al. (2015) Russakovsky, O., Deng, J., Su, H., et al. 2015, International Journal of Computer Vision (IJCV), 115, 211, doi: 10.1007/s11263-015-0816-y
  * Sako et al. (2018) Sako, M., Bassett, B., Becker, A. C., et al. 2018, PASP, 130, 064002, doi: 10.1088/1538-3873/aab4e0
  * Simonyan & Zisserman (2014) Simonyan, K., & Zisserman, A. 2014, CoRR, abs/1409.1556
  * Sønderby et al. (2020) Sønderby, C. K., Espeholt, L., Heek, J., et al. 2020, MetNet: A Neural Weather Model for Precipitation Forecasting.  https://arxiv.org/abs/2003.12140
  * The PLAsTiCC team et al. (2018) The PLAsTiCC team, Allam, Tarek, J., Bahmanyar, A., et al. 2018, arXiv e-prints, arXiv:1810.00001.  https://arxiv.org/abs/1810.00001
  * van den Oord et al. (2016) van den Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. 2016, Pixel Recurrent Neural Networks.  https://arxiv.org/abs/1601.06759
  * Vincenzi et al. (2023) Vincenzi, M., Sullivan, M., Möller, A., et al. 2023, MNRAS, 518, 1106, doi: 10.1093/mnras/stac1404
  * Wang et al. (2015) Wang, Y., Gjergo, E., & Kuhlmann, S. 2015, MNRAS, 451, 1955, doi: 10.1093/mnras/stv1090
  * Zeiler & Fergus (2014) Zeiler, M. D., & Fergus, R. 2014, in European conference on computer vision, Springer, 818–833