11institutetext: School of Computer Science and Engineering, University of
Electronic Science and Technology of China, Chengdu, China
11email<EMAIL_ADDRESS>22institutetext: Department of Computer
Science, Hong Kong Baptist University, Hong Kong, China
22email<EMAIL_ADDRESS>
33institutetext: Department of Computer Science and Engineering, The Chinese
University of Hong Kong, Hong Kong, China
33email<EMAIL_ADDRESS>44institutetext: School of Computer Science and
Technology, Harbin Institute of Technology, Shenzhen, China
44email<EMAIL_ADDRESS>
# Generalized Category Discovery with Clustering Assignment Consistency

Xiangli Yang 11 0000-0003-0546-5440 Xinglin Pan 22 Irwin King 33
0000-0001-8106-6447 Zenglin Xu 44 0000-0001-5550-6461

###### Abstract

Generalized category discovery (GCD) is a recently proposed open-world task.
Given a set of images consisting of labeled and unlabeled instances, the goal
of GCD is to automatically cluster the unlabeled samples using information
transferred from the labeled dataset. The unlabeled dataset comprises both
known and novel classes. The main challenge is that unlabeled novel class
samples and unlabeled known class samples are mixed together in the unlabeled
dataset. To address the GCD without knowing the class number of unlabeled
dataset, we propose a co-training-based framework that encourages clustering
consistency. Specifically, we first introduce weak and strong augmentation
transformations to generate two sufficiently different views for the same
sample. Then, based on the co-training assumption, we propose a consistency
representation learning strategy, which encourages consistency between
feature-prototype similarity and clustering assignment. Finally, we use the
discriminative embeddings learned from the semi-supervised representation
learning process to construct an original sparse network and use a community
detection method to obtain the clustering results and the number of categories
simultaneously. Extensive experiments show that our method achieves state-of-
the-art performance on three generic benchmarks and three fine-grained visual
recognition datasets. Especially in the ImageNet-100 data set, our method
significantly exceeds the best baseline by 15.5% and 7.0% on the Novel and All
classes, respectively.

###### Keywords:

Generalized category discovery Open-world problem Novel category discovery.

##  1 Introduction

Figure 1: An illustration of the generalized category discovery task. Given a
set of labeled images of some known categories, and a set of unlabeled images
that contain both known and novel classes, the objective is to automatically
identity known and novel classes in the test dataset.

Deep learning has proven to be remarkably successful in image classification
tasks that heavily rely on vast amounts of high-quality labeled data [36, 29].
However, acquiring annotations for all data is often unfeasible, which
requires learning from unlabeled data. Semi-supervised learning [4, 53, 49]
and few-shot learning [22, 15] are two examples of such learning paradigms.
Most of these learning algorithms operate in closed set settings, where known
categories are predetermined.

In recent years, _Novel Category Discovery (NCD)_ has received significant
attention for the discovery of novel classes by transferring knowledge learned
from known classes. The implementation of NCD is based on the assumption that
all unlabeled samples pertain to novel categories whose number is
predetermined, which is excessively idealistic and impractical in the wild.
[46] relaxes these assumptions by recognizing instances of both known and
novel classes in the unlabeled dataset. This new setting is named _Generalized
Category Discovery_ (GCD). To illustrate the setting of GCD, we consider the
scenario in Figure 1, where the training set consists of labeled instances of
known classes (e.g., cane and cavallo) and unlabeled instances comprising
instances of known classes (i.e., cane and cavallo) and novel classes (i.e.,
gallina and elephant). An ideal model should not only be able to classify
known classes (i.e., cane, and cavallo), but also be able to discover novel
classes (i.e., gallina, and elephant).

Generalized Category Discovery (GCD) is an open-world problem that holds
promise for numerous real-world applications [46, 20, 2, 6]. However, it also
poses significant obstacles, namely: a) the lack of supervised prior knowledge
for novel classes and b) the absence of clear distinctions between samples
belonging to known and novel categories. To address the GCD problem, [46]
incorporates contrastive loss to learn discriminative representations of the
unlabeled dataset. Furthermore, semi-supervised k-means [3] is applied to
implement clustering. However, we find that using all unlabeled instances as
negative samples in contrastive learning will treat unlabeled instances of the
same category as negative pairs, which will cause category collision problem
[56]. Moreover, in real-world problems, prior information regarding the number
of clustering assignments is rarely available, which makes it inappropriate to
use semi-supervised k-means.

To overcome these limitations, we present a co-training consistency strategy
for clustering assignments to uncover latent representations among unlabeled
dataset. For the final clustering goal, we leverage community detection
techniques to assign labels to unlabeled instances and automatically determine
the optimal number of clustering categories based on the learned
representations. Our approach In particular, our approach consists of two
stages: semi-supervised representation learning and community detection. In
semi-supervised representation learning, we use supervised contrastive loss to
derive the labeled information fully. Additionally, since the contrastive
learning approach aligns with the co-training assumption, we introduce weak
and strong augmentations of the same sample to extract two distinct views. We
subsequently deploy the co-training framework to enforce the consistency of
feature-prototype similarity and clustering assignment between the two views.
In community detection, we construct the primary graph utilizing feature
embeddings learned in semi-supervised representation learning, then apply a
community detection method to obtain the outcomes.

Our contributions to this work are as follows,

  * •

We employ weak and strong data augmentations to generate two sufficiently
different views.

  * •

We introduce a co-training-based framework to effectively learn embeddings of
unlabeled data.

  * •

We propose a community detection approach to clustering assignments that
accurately detects novel categories of unlabeled instances while
simultaneously determining the number of clusters.

  * •

Through experimentation on six benchmarks, our approach demonstrates state-of-
the-art performance and confirms the efficacy of our method.

##  2 Related Work

Novel Category Discovery. The objective of Novel Category Discovery (NCD) is
to cluster unlabeled instances using transferable knowledge derived from the
labeled dataset of known categories, with all unlabeled instances regarded as
novel categories [45]. KCL [31] addresses the problem of transfer learning
across domains, especially cross-task transfer learning, which is similar with
NCD. Similar to KCL, MCL [32] constructs pairwise pseudo-labels on unlabeled
dataset using a similarity prediction network trained on the labeled dataset,
and determines whether two instances belong to the same category by using the
inner product. DTC [27] utilizes a cross-entropy classifier to initialize
representations on the labeled dataset. Then it maintains a list of class
prototypes that represent cluster centers, and assigns instances to the
closest prototype. In this way, a good clustering representation is learned.
RankStats or AutoNovel [25, 26] applies RotNet [23] to perform an
initialization of the encoder. Afterward, it generates pseudo-labels through
dual-ranking statistics and enforces consistency between the predictions of
the two branch networks. NCL [57] retrieves and aggregates pseudo positive
pairs by contrastive learning, and produces hard negative pairs by mixing
labeled and unlabeled instances in the feature space. Unlike the methods that
separately design loss functions for labeled and unlabeled samples, UNO [21]
introduces a unified objective function to discover new classes that
effectively supports the collaboration between supervised and unsupervised
learning. MEDI [12] proposes a new approach using MAML [22] to solve the NCD
problem and provided a solid theoretical analysis of the underlying
assumptions in the NCD field. ComEx [52] modifies UNO [21] by designing two
groups of experts to learn the entire dataset in a complementary way, thus
mitigating the limitations of previous NCD methods. Recently, GCD [46] extends
the NCD task to include the recognition of known classes in the unlabeled
dataset. It leverages a well-trained visual transformer model [17] to improve
visual representation. ORCA [6] enables the simultaneous clustering of various
novel classes in the unlabeled set, assuming a pre-defined number of novel
classes.

Self-supervised learning. Self-supervised learning [1] can simultaneously
utilize supervised and unsupervised learning during the fine-tuning stage of
training without the need for manual annotations [28]. Furthermore, self-
supervised learning can be divided into two types: auxiliary pretext learning
and contrastive learning. When manual annotations are not available, auxiliary
pretext tasks are used as supervised information to learn representations
[30]. Some commonly used pretext tasks include exemplar-based methods [18],
rotation [23], predicting missing pixels [41], grayscale images [54], and
patch context and jigsaw puzzles [16, 40]. However, designing a suitable
auxiliary pretext task requires domain-specific knowledge in order to better
serve downstream tasks. Contrastive learning minimizes the latent embedding
distance between positive pairs and maximizes the distance between negative
pairs [51]. SimCLR [10] learned useful representations based on contrastive
loss by maximizing the similarity between the original sample and augmented
views of it. MoCo [28] utilized a momentum contrast to calculate the
similarity. BYOL [24] proposed a new framework for learning feature
representations without the help of contrasting negative pairs. This was
achieved by using a Siamese architecture, where the query branch is equipped
with a predictor architecture in addition to the encoder and projector.
Following BYOL [24], In addition to removing the impulse key encoder, Simsiam
[11] also used the stop-gradient strategy in the approach while adopting it to
alleviate issues related to collapsing. Roh et al. [43] utilized consistency
losses for the intersection region of the ROIs from two views to improve the
encoder representation in subsequent detection tasks. Moreover, the KL loss is
a popular approach for consistency learning, as demonstrated in prior works
such as [50] and [39], with the incorporation of regularization to ensure that
embeddings derived from different data augmentation techniques are consistent.

##  3 Method

###  3.1 Preliminaries

GCD aims to automatically classify unlabeled instances containing both known
and unknown categories [46]. This is a more realistic open-world setting than
the common closed-set classification, which assumes that labeled and unlabeled
data belong to the same categories. Our settings follows [46]. Formally, let
the train dataset be $\mathcal{D}=\mathcal{D}_{L}\cup\mathcal{D}_{U}$, where
$\mathcal{D}_{L}=\\{(\bm{x_{i}},y_{i})|y_{i}\in\mathcal{Y}_{known}\\}$ and
$\mathcal{D}_{U}=\\{\bm{x}_{i}|y_{i}\in\mathcal{Y}_{U}\\}$ and
$\mathcal{Y}_{U}=\\{\mathcal{Y}_{known},\mathcal{Y}_{novel}\\}$. Here,
$\mathcal{Y}_{U},\mathcal{Y}_{known}$ and $\mathcal{Y}_{novel}$ denote the
label set of All, Known, and Novel classes, respectively. This formalization
enables us to easily distinguish between the NCD setting and GCD setting. In
NCD, it is assumed that $\mathcal{Y}_{known}\cap\mathcal{Y}_{U}=\emptyset$.

Figure 2: Diagram of Clustering Assignment Consistency. For original image
$\bm{x}$, we apply two transformations $\mathcal{T}_{w}$ and $\mathcal{T}_{s}$
to obtain two different views $\bm{x}_{w}$ and $\bm{x}_{s}$. Then they go
through a backbone network, resulting in two embedded representations
$\bm{z}_{w}$ and $\bm{z}_{s}$. In addition, $\bm{C}$ is the matrix whose
columns are the $K$ trainable prototypes vectors,
$\\{\bm{c}_{1},\bm{c}_{2},\cdots,\bm{c}_{K}\\}$. $\otimes$ denotes the dot
products of $\bm{z}_{i}$ and all prototypes in $\bm{C}$. Then we can compute
codes (clustering assignments) $\bm{q}_{w}$ and $\bm{q}_{s}$ by matching these
representations to the prototypes vectors.

To tackle the issue of GCD, we leverage co-trained clustering assignment
consistency for acquiring discriminative representations of the unlabeled
data. We then employ a customized community detection technique to
automatically obtain the clustering results and the number of categories.
Specifically, we use supervised contrastive learning to make maximum use of
the supervised information for labeled instances. As depicted in Figure 2, we
apply a co-training framework to unlabeled data to ensure the consistency
between the clustering assignment and the feature-prototype similarities of
the same image. Following the semi-supervised representation learning, we
utilize a modified community detection strategy to allocate category labels
for each unlabeled instance, regardless if it is a known one or a novel one,
and to determine the number of categories.

###  3.2 Semi-supervised representation learning

####  3.2.1 Supervised contrastive learning.

We apply supervised contrastive learning to minimize the similarity between
embeddings of the same class while maximize the similarity among embeddings
from different classes at the same time. Utilizing label information only in
the supervised contrastive learning instead of in a cross-entropy loss results
in equal treatment of labelled and unlabeled data. The supervised contrastive
component nudges the network towards a semantically meaningful representation
to reduce overfitting on the labeled classes. The supervised contrastive loss,
as in [33, 46], is defined as,

$\mathcal{L}_{sup}=-\frac{1}{|\mathcal{N}(i)|}\sum_{p\in\mathcal{N}(i)}\log\frac{\exp\left(sim(\bm{z}_{i},\bm{z}_{p})/\tau_{sup}\right)}{\sum_{n}\mathds{1}_{[n\neq
i]}\exp\left(sim(\bm{z}_{i},\bm{z}_{n})/\tau_{sup}\right)},$ (1)

where $\bm{z}_{i}=h(f(\bm{x}_{i}))$, $f(\cdot)$ is the feature extractor, and
$h(\cdot)$ is a multi-layer perceptron (MLP) projection head.
$sim(\bm{z}_{i},\bm{z}_{p})$ denote the cosine similarity between $\bm{z}_{i}$
and $\bm{z}_{p}$. $\tau$ is a temperature value. $\mathcal{N}(i)$ is the
indices of other images having the same label as $\bm{x}_{i}$.
$\mathds{1}_{[n\neq i]}$ denotes an indicator function evaluating to $1$ if
$n\neq i$.

####  3.2.2 Unsupervised contrastive learning.

We found that unsupervised contrastive learning in [46] treats different
unlabeled samples from the same semantic category as false negatives,
resulting in a class collision problem [56]. Inspired by this, we introduce a
soft approach to mitigate this problem. In contrast learning [10], two
different augmentations generate two different views, and this scenario
satisfies the co-training assumption, that is,

$f_{1}(\bm{v}_{1})=f_{2}(\bm{v}_{2}),\forall\bm{x}=(\bm{v}_{1},\bm{v}_{2})\quad\text{(Co-
Training Assumption)},$ (2)

where $f(\cdot),f_{1}(\cdot),f_{2}(\cdot)$ refer to different prediction
functions. $\bm{v}_{1}$ and $\bm{v}_{2}$ are two different views of $\bm{x}$.
In this paper, weak augmentation and strong augmentation are utilized to
generate two separate views. We rethink both the co-training assumption and
the swapped prediction problem discussed in [8]. We consider the probability
of feature similarity to the prototypes and promote the minimization of
divergence between the soft clustering assignment and the probability in the
weakly augmented case. Here, We ignore the case of strong augmentation since
the induced distortions could significantly alter the image structures and
make it difficult to preserve the identity of the original instances.
Additionally, we measure the fit between the feature embeddings and their soft
assignments.

Formally, let $\bm{x}_{w}$ and $\bm{x}_{s}$ denote the weakly augmented and
strongly augmented views of the same image $\bm{x}$, respectively.
$\\{\bm{c}_{1},\bm{c}_{2},\cdots,\bm{c}_{K}\\}$ is a set of $K$ learnable
prototypes. The feature is projected to the unit sphere, i.e.,
$\bm{z}_{w}=\frac{f(\bm{x}_{w})}{\|f(\bm{x}_{w})\|_{2}}$. We consider the
feature-prototype similarity probability and the clustering assignment as two
different predictions as in Eq. (2). Following [42], We use a measure of
similarity, the Jensen-Shannon divergence between feature-prototype similarity
probability $\bm{p}_{w}$ and clustering assignment $\bm{q}_{w}$, the
consistency loss is written as

$\mathcal{L}(\bm{z}_{w},\bm{q}_{w})=H\left(\frac{1}{2}(\bm{p}_{w}^{(k)}+\bm{q}_{w}^{(k)})\right)-\frac{1}{2}\left(H(\bm{p}_{w}^{(k)})+H(\bm{q}_{w}^{(k)})\right),$
(3)

where $H(\bm{p})$ is the entropy of $\bm{p}$, and

$\bm{p}_{w}^{(k)}=\frac{\exp(sim(\bm{z}_{w},\bm{c}_{k})/\tau_{u})}{\sum_{n}\mathds{1}_{[n\neq
i]}\exp\left(sim(\bm{z}_{w},\bm{c}_{n})/\tau_{u}\right)}.$ (4)

We encourage the consistency of clustering assignments between weakly and
strongly augmented views of the same image, and the loss function is written
as follows,

$\displaystyle\mathcal{L}(\bm{z}_{w},\bm{z}_{s})$ $\displaystyle=$
$\displaystyle\mathcal{L}(\bm{z}_{w},\bm{q}_{s})+\mathcal{L}(\bm{z}_{s},\bm{q}_{w})$
(5) $\displaystyle=$ $\displaystyle
H(\bm{p}_{w},\bm{q}_{s})+H(\bm{p}_{s},\bm{q}_{w})$ (6)

where $H(\cdot,\cdot)$ represents the cross entropy loss,

$\bm{p}_{s}^{(k)}=\frac{\exp(sim(\bm{z}_{s},\bm{c}_{k})/\tau_{u})}{\sum_{n}\mathds{1}_{[n\neq
i]}\exp\left(sim(\bm{z}_{s},\bm{c}_{n})/\tau_{u}\right)},$ (7)

and $sim(\cdot,\cdot)$ denote the cosine similarity as in Eq. (1). Similar to
[7, 8], we compute clustering assignment matrix
$\bm{Q}=[\bm{q}_{1},\bm{q}_{2},\cdots,\bm{q}_{B}]$ with $B$ feature embeddings
$\bm{Z}=[\bm{z}_{1},\bm{z}_{2},\cdots\bm{z}_{B}]$ and the prototypes
$\bm{C}=[\bm{c}_{1},\bm{c}_{2},\cdots,\bm{c}_{K}]$,

$\displaystyle\max_{\bm{Q}\in\mathcal{Q}}\textbf{Tr}(\bm{Q}^{T}\bm{C}^{T}\bm{Z})+\epsilon
H(\bm{Q}),$ (8)
$\displaystyle\mathcal{Q}=\left\\{\bm{Q}\in\mathbb{R}_{+}^{K\times
B}|\bm{Q}\bm{1}_{B}=\frac{1}{K}\bm{1}_{K},\bm{Q}^{T}\bm{1}_{K}=\frac{1}{B}\bm{1}_{B}\right\\},$
(9)

where $\epsilon$ is a weight coefficient. $\textbf{Tr}(\cdot)$ denotes the
trace function, and $\mathcal{Q}$ is the transportation polytope. The solution
to Eq. (8) is obtaining using the Sinkhorn-Knopp algorithm [13]. The pseudo-
code for computing $\bm{Q}$ and training loop is in Algorithm 1.

Thus, the overall loss to optimize the model can be formulated as follows,

$\mathcal{L}=\mathbb{E}_{(\bm{x},y)\in\mathcal{D}_{L}}\mathcal{L}_{sup}+\alpha\mathbb{E}_{\bm{x}\in\mathcal{D}}\mathcal{L}(\bm{z}_{w},\bm{q}_{w})+(1-\alpha)\mathbb{E}_{\bm{x}\in\mathcal{D}}\mathcal{L}(\bm{z}_{w},\bm{z}_{s}),$
(10)

where $\alpha$ is a weight coefficient.

Input: batch size $B$, prototypes $C$, extractor $f\left(\cdot\right)$, MLP
$h\left(\cdot\right)$, temperature $\tau$, dataset $\mathcal{D}_{U}$;

Function  _sinkhorn(_$scores$ , $\epsilon=0.05$, $n_{iters}=3$_)_:

$u^{(0)}=1,K=\exp(scores/\epsilon)$;

for  _$i=1$ to $n_{iters}$_ do

$v^{(i)}=1/K^{T}u^{(i-1)}$; $u^{(i)}=1/K^{T}v^{(i)}$;

end for

return $\text{diag}(u^{(n_{iters})})K\text{diag}(v^{(n_{iters})})$

End Function

for  _$x\in$ minibatch_ do

$x_{w}\leftarrow h(f(\text{WeakAugment}(x)))$;

$x_{s}\leftarrow h(f(\text{StrongAugment}(x)))$;

$score_{w}\leftarrow x_{w}\times C$;

$score_{s}\leftarrow x_{s}\times C$;

$p_{w}=\text{Softmax}(scores_{w}/\tau)$;

$p_{s}=\text{Softmax}(scores_{s}/\tau)$;

stop gradients to $score_{w}$, $score_{s}$;

$q_{w}=\text{sinkhorn}(scores_{w})$;

$q_{s}=\text{sinkhorn}(scores_{s})$;

compute $l$ by Eq. (10);

update $h$ and $C$ to minimize $l$;

$C\leftarrow\text{Normlization}(C)$;

end for

Algorithm 1 The torch style pseudo-code for unsupervised contrastive learning.

###  3.3 Label assignment with Louvain algorithm

Semi-supervised k-means [3] is applied in [46] for assigning cluster labels.
However, this approach necessitates prior knowledge of the number of clusters,
a requirement that is not realistic because such information is typically not
available beforehand. The learned embeddings can be used to assign cluster
labels to unlabeled instances and determine the number of clusters adaptively
through a community detection method, Louvain algorithm [5, 19]. The
representation embeddings can be represented as an embedding graph
$G=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}=\\{\bm{z}_{i}\\}_{i=1}^{N}$,
$N$ is the number of embeddings, and $\mathcal{E}$ consists of edges that
connect vertices in $\mathcal{V}$. We define a matrix $\bm{W}$ as the
adjacency matrix of $G$, and the entry $\bm{W}_{i,j}$ between vertices $i$ and
$j$ is given by,

${\bm{W}}_{ij}=\begin{cases}1,&\text{ if
}\bm{x}_{i},\bm{x}_{j}\in\mathcal{D}_{L},\text{and}~{}{y}_{i}={y}_{j}\\\
sim(\bm{z}_{i},\bm{z}_{j}),&\text{ if
}\bm{x}_{i}~{}\text{or}~{}\bm{x}_{j}\in\mathcal{D}_{U},\text{and}~{}\bm{z}_{j}\in\text{Neighbor}(\bm{z}_{i})\\\
0,&\text{ otherwise }\end{cases},$ (11)

where
$\text{Neighbor}(\bm{z}_{i})=\arg{topM}(\\{sim(\bm{z}_{i},\bm{z}_{j})|\bm{z}_{j}\in\mathcal{V}\\})$
denotes the neighbors of $\bm{z}_{i}$,i.e., the $M$ embeddings with the
greatest similarity to $\bm{z}_{i}$. In this way, we construct a graph $G$,
which represents the possible connection relationships among all instances. We
can now assign category labels for all instances in the training dataset,
either from the known classes or novel ones. Using the Louvain algorithm, we
automatically obtain the clustering assignments and the number of categories
on the constructed graph.

##  4 Experiment

###  4.1 Datasets

We evaluate our framework on three generic object recognition datasets, namely
CIFAR-10 [35], CIFAR-100 [35] and ImageNet-100 [14]. These standard image
recognition datasets establish the performance of different methods. We
further evaluate our method on Semantic Shift Benchmark (SSB) [47], includeing
CUB-200 [48], Stanford Cars [34] and Herbarium19 [44]. The dataset splits are
described in 1. We follow [46] sample a subset of half the classes as ‘’Old"
categories. $50\%$ of instances of each labeled class are drawn to form the
labeled set, and all the remaining data constitute the unlabeled set. For
evaluation, we measure the clustering accuracy by comparing the predicted
label assignment with the ground truth, following the protocol in [46]. The
accuracy scores for All, Known, and Novel categories are reported.

Table 1: Dataset splits in the experiments.

Dataset | CIFAR10 | CIFAR100 | ImageNet-100 | CUB-200 | Stanford Cars | Herbarium19
---|---|---|---|---|---|---
Labelled | Classes | 5 | 80 | 50 | 100 | 98 | 341
Images | 12.5k | 20k | 31.9k | 1498 | 2000 | 8.9k
Unlabelled | Classes | 10 | 100 | 100 | 200 | 196 | 683
Images | 37.5k | 30k | 95.3k | 4496 | 6144 | 25.4k

###  4.2 Implementation Details.

We follow the implementations and learn schedules in [46] as far as possible.
Specifically, we take the ViT-B-16 pre-trained by DINO [9] on ImageNet [36] as
our backbone model and we use the [CLS] token as the feature representation.
We train semi-supervised contrastive learning with $\tau_{sup}=0.07$,
$\tau_{u}=0.05$ and $\alpha=0.3$. For image augmentation, we use ResizedCrop,
ColorJitter, Grayscale, HorizontalFlip as the weak image augmentation. The
strong transformation strategy is composed of five randomly selected from
RandAugment. For model optimization, we use the AdamW provided by [37]. The
initial learning rate is 0.1. The batch size are chosen based on available GPU
memory. All the experiments are conducted on a single RTX-2080 and averaged
over 5 different seeds.

###  4.3 Compare with State-of-the-Arts

We summarize the baselines compared in our experiments, including $k$-means
[3], RankStats+ [25], UNO+ [21] and GCD [46].

Table 2: Results on three generic datasets. Accuracy scores are reported. †denotes adapted methods. | CIFAR-10 | CIFAR-100 | ImageNet-100
---|---|---|---
Method | All | Known | Novel | All | Known | Novel | All | Known | Novel
KMeans [3] | 83.6 | 85.7 | 82.5 | 52.0 | 52.2 | 50.8 | 72.7 | 75.5 | 71.3
RankStats† [25] | 46.8 | 19.2 | 60.5 | 58.2 | 77.6 | 19.3 | 37.1 | 61.6 | 24.8
UNO† [21] | 68.6 | 98.3 | 53.8 | 69.5 | 80.6 | 47.2 | 70.3 | 95.0 | 57.9
GCD [46] | 91.5 | 97.9 | 88.2 | 73.0 | 76.2 | 66.5 | 74.1 | 89.8 | 66.3
Ours | 92.3 | 91.4 | 94.4 | 78.5 | 81.4 | 75.6 | 81.1 | 80.3 | 81.8

Evaluation on generic datasets. The results on generic benchmarks are shown in
Table 2. As we can see that our method achieves state-of-the art performance
on All and Novel tested on all generic datasets, especially on ImageNet-100.
Our method also achieves comparable results with other methods on Known.
Specifically, for the All classes, our method beats the GCD method by 0.8%,
5.5%, and 7.0% on CIFAR-10, CIFAR-100, and ImageNet-100, respectively. For the
Novel class, it is 6.2% higher on CIFAR-10, 9.1% higher on CIFAR-100, and
15.5% higher on ImageNet-100. These results experimentally show that our
method learns a more compact representation on the unlabeled dataset. In
addition, UNO+ uses a linear classifier, which shows strong accuracy on the
Known classes, but leads to poor performance on the Novel classes.

Table 3: Results on three fine-grained datasets. Accuracy scores are reported. †denotes adapted methods. | CUB-200 | Stanford-Cars | Herbarium19
---|---|---|---
Method | All | Known | Novel | All | Known | Novel | All | Known | Novel
$k$-means [3] | 34.3 | 38.9 | 32.1 | 12.8 | 10.6 | 13.8 | 12.9 | 12.9 | 12.8
RankStats+ [25] | 33.3 | 51.6 | 24.2 | 28.3 | 61.8 | 12.1 | 27.9 | 55.8 | 12.8
UNO+ [21] | 35.1 | 49.0 | 28.1 | 35.5 | 70.5 | 18.6 | 28.3 | 53.7 | 14.7
GCD [46] | 51.3 | 56.6 | 48.7 | 39.0 | 57.6 | 29.9 | 35.4 | 51.0 | 27.0
Ours | 58.0 | 65.0 | 43.9 | 47.6 | 70.6 | 33.8 | 36.3 | 53.1 | 30.7

Evaluation on fine-grained datasets. We report the results on three fine-
grained datasets (as in [46] in Table 3. Our method shows optimal performance
on the All classes for the three datasets tested, and achieves comparable
results on the Known and Novel classes, demonstrating the effectiveness of our
method for fine-grained category discovery. Specifically, on the CUB-200,
Stanford-Scars, and Herbarium19 datasets, our method achieves 6.7%, 8.6%, and
0.9% improvement over the state-of-the-art method on the All classes,
respectively. For the Novel classes, our method outperforms GCD by 3.9% and
3.7% on Stanford-cars and Herbarium19, respectively. Meanwhile, we find that
due to the low variability between fine-grained datasets, which makes it more
difficult to discover novel classes, the precision on the Novel classes is
generally low in terms of results.

###  4.4 Ablation Study

Effectiveness of each component. We conduct extensive ablation experiments,
and perform four experiments on the CIFAR-100 and CUB-200 datasets. Table 4
shows the contribution of introducing different components on the objective
loss function, including $\mathcal{L}_{sup}$,
$\mathcal{L}(\bm{z}_{w},\bm{q}_{w})$ and $\mathcal{L}(\bm{z}_{w},\bm{z}_{s})$.
We can observe from the results that all components contribute significantly
to our proposed approach: according to the results of experiments (1) and (2),
we find that $\mathcal{L}(\bm{z}_{w},\bm{z}_{s})$ is the most important
component, which proves that the co-training consistency can close the intra-
class embeddings and push away the inter-class boundaries. Moreover, compared
to the original DINO features, the features of our approach show more
favorable clustering results on CUB-200 dataset. Comparing experiments (3) and
(4), we can find that supervised contrastive learning can further improve the
performance on known novel categories, demonstrating the importance of
supervised information.

Table 4: Ablation study on the components of the loss function. Index | Component | CIFAR100 | CUB-200
---|---|---|---
$\mathcal{L}_{sup}$ | $\mathcal{L}(\bm{z}_{w},\bm{q}_{w})$ | $\mathcal{L}(\bm{z}_{w},\bm{z}_{s})$ | All | Known | Novel | All | Known | Novel
(1) | ✗ | ✗ | ✗ | 34.9 | 36.1 | 33.6 | 15.0 | 13.5 | 21.6
(2) | ✗ | ✗ | ✓ | 69.7 | 71.4 | 68.0 | 56.7 | 62.5 | 40.8
(3) | ✗ | ✓ | ✓ | 75.7 | 79.2 | 72.5 | 57.5 | 63.9 | 42.8
(4) | ✓ | ✓ | ✓ | 78.5 | 81.4 | 75.5 | 58.0 | 65.0 | 43.9
Table 5: Ablation study on neighborhood size. M | CIFAR-100 | CUB-200
---|---|---
All | Known | Novel | All | Known | Novel
5 | 78.5 | 81.4 | 75.5 | 58.0 | 65.0 | 43.9
10 | 69.6 | 70.4 | 68.7 | 53.5 | 60.8 | 35.1
15 | 64.9 | 65.0 | 64.8 | 46.8 | 54.5 | 26.8
20 | 63.5 | 62.3 | 64.8 | 44.5 | 49.8 | 33.7
25 | 59.7 | 60.3 | 59.1 | 39.7 | 44.6 | 30.5
30 | 59.8 | 60.4 | 59.2 | 27.3 | 29.3 | 29.7

Effectiveness of the neighborhood size $M$. Table 5 illustrates the effect of
the neighborhood size for vertex $\bm{z}_{i}\in G(\mathcal{V},\mathcal{E})$ on
the final clustering results. We select $K=5,10,15,20,25,30$ for the ablation
experiments on both CIFAR-100 and CUB-200 datasets. We find that the final
performance varies greatly depending on the number of neighbors. When $K=5$,
it can reach the optimum in all All, Known and Novel classes. We conjecture
that too many links will negatively affect the community detection.

(a) DINO

(b) GCD

(c) Ours

Figure 3: T-SNE visualization of the embeddings on CIFAR-10. The embedding
clustering shows that our proposed method encourages the expansion of the
distance between different clusters.

Visualization. In order to explore the clustering features on different
methods more intuitively, we further visualized the features extracted by
DINO, GCD and our method using T-SNE [38] on CIFAR-10. As shown in Figure 3,
compared with DINO and GCD, our method obtains clearer boundaries between
different groups, and furthermore obtains more compact clusters.

##  5 Conclusion

In this paper, we propose a co-training strategy for GCD. To implement this
idea, we introduce a clustering assignment consistency framework that explores
discriminative representations alternately. Additionally, we propose a
community detection method to address the semi-supervised clustering problem
in GCD. Experimental results demonstrate that our method achieves state-of-
the-art performance in both generic and fine-grained tasks.

####  5.0.1 Acknowledgements

Please place your acknowledgments at the end of the paper, preceded by an
unnumbered run-in heading (i.e. 3rd-level heading).

## References

  * [1] Albelwi, S.: Survey on self-supervised learning: Auxiliary pretext tasks and contrastive learning methods in imaging. Entropy 24(4), 551 (2022)
  * [2] An, W., Tian, F., Zheng, Q., Ding, W., Wang, Q., Chen, P.: Generalized category discovery with decoupled prototypical network. CoRR abs/2211.15115 (2022)
  * [3] Arthur, D., Vassilvitskii, S.: k-means++: the advantages of careful seeding. In: SODA. pp. 1027–1035. SIAM (2007)
  * [4] Berthelot, D., Carlini, N., Goodfellow, I.J., Papernot, N., Oliver, A., Raffel, C.: Mixmatch: A holistic approach to semi-supervised learning. In: NeurIPS. pp. 5050–5060 (2019)
  * [5] Blondel, V.D., Guillaume, J.L., Lambiotte, R., Lefebvre, E.: Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment 2008, P10008 (2008)
  * [6] Cao, K., Brbic, M., Leskovec, J.: Open-world semi-supervised learning. In: ICLR. OpenReview.net (2022)
  * [7] Caron, M., Bojanowski, P., Joulin, A., Douze, M.: Deep clustering for unsupervised learning of visual features. In: ECCV (14). Lecture Notes in Computer Science, vol. 11218, pp. 139–156. Springer (2018)
  * [8] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. In: NeurIPS (2020)
  * [9] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: ICCV. pp. 9630–9640. IEEE (2021)
  * [10] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.E.: A simple framework for contrastive learning of visual representations. In: ICML. Proceedings of Machine Learning Research, vol. 119, pp. 1597–1607. PMLR (2020)
  * [11] Chen, X., He, K.: Exploring simple siamese representation learning. In: CVPR. pp. 15750–15758. Computer Vision Foundation / IEEE (2021)
  * [12] Chi, H., Liu, F., Yang, W., Lan, L., Liu, T., Han, B., Niu, G., Zhou, M., Sugiyama, M.: Meta discovery: Learning to discover novel classes given very limited data. In: ICLR. OpenReview.net (2022)
  * [13] Cuturi, M.: Sinkhorn distances: Lightspeed computation of optimal transport. In: NIPS. pp. 2292–2300 (2013)
  * [14] Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR. pp. 248–255. IEEE Computer Society (2009)
  * [15] Dhillon, G.S., Chaudhari, P., Ravichandran, A., Soatto, S.: A baseline for few-shot image classification. In: ICLR. OpenReview.net (2020)
  * [16] Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learning by context prediction. In: ICCV. pp. 1422–1430. IEEE Computer Society (2015)
  * [17] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR. OpenReview.net (2021)
  * [18] Dosovitskiy, A., Fischer, P., Springenberg, J.T., Riedmiller, M.A., Brox, T.: Discriminative unsupervised feature learning with exemplar convolutional neural networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(9), 1734–1747 (2016)
  * [19] Dugué, N., Perez, A.: Directed louvain : maximizing modularity in directed networks (2015)
  * [20] Fei, Y., Zhao, Z., Yang, S., Zhao, B.: Xcon: Learning with experts for fine-grained category discovery. In: BMVC. p. 96. BMVA Press (2022)
  * [21] Fini, E., Sangineto, E., Lathuilière, S., Zhong, Z., Nabi, M., Ricci, E.: A unified objective for novel class discovery. In: ICCV. pp. 9264–9272. IEEE (2021)
  * [22] Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: ICML. Proceedings of Machine Learning Research, vol. 70, pp. 1126–1135. PMLR (2017)
  * [23] Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. In: ICLR (Poster). OpenReview.net (2018)
  * [24] Grill, J., Strub, F., Altché, F., Tallec, C., Richemond, P.H., Buchatskaya, E., Doersch, C., Pires, B.Á., Guo, Z., Azar, M.G., Piot, B., Kavukcuoglu, K., Munos, R., Valko, M.: Bootstrap your own latent - A new approach to self-supervised learning. In: NeurIPS (2020)
  * [25] Han, K., Rebuffi, S., Ehrhardt, S., Vedaldi, A., Zisserman, A.: Automatically discovering and learning new visual categories with ranking statistics. In: ICLR. OpenReview.net (2020)
  * [26] Han, K., Rebuffi, S., Ehrhardt, S., Vedaldi, A., Zisserman, A.: Autonovel: Automatically discovering and learning novel visual categories. IEEE Trans. Pattern Anal. Mach. Intell. 44(10), 6767–6781 (2022)
  * [27] Han, K., Vedaldi, A., Zisserman, A.: Learning to discover novel visual categories via deep transfer clustering. In: ICCV. pp. 8400–8408. IEEE (2019)
  * [28] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.B.: Momentum contrast for unsupervised visual representation learning. In: CVPR. pp. 9726–9735. Computer Vision Foundation / IEEE (2020)
  * [29] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778. IEEE Computer Society (2016)
  * [30] Holmberg, O.G., Köhler, N.D., Martins, T., Siedlecki, J., Herold, T., Keidel, L., Asani, B., Schiefelbein, J., Priglinger, S., Kortuem, K.U., Theis, F.J.: Self-supervised retinal thickness prediction enables deep learning from unlabelled data to boost classification of diabetic retinopathy. Nat. Mach. Intell. 2(11), 719–726 (2020)
  * [31] Hsu, Y., Lv, Z., Kira, Z.: Learning to cluster in order to transfer across domains and tasks. In: ICLR (Poster). OpenReview.net (2018)
  * [32] Hsu, Y., Lv, Z., Schlosser, J., Odom, P., Kira, Z.: Multi-class classification without multi-class labels. In: ICLR (Poster). OpenReview.net (2019)
  * [33] Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D.: Supervised contrastive learning. In: NeurIPS (2020)
  * [34] Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3d object representations for fine-grained categorization. In: Proceedings of the IEEE international conference on computer vision workshops. pp. 554–561 (2013)
  * [35] Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
  * [36] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)
  * [37] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (Poster). OpenReview.net (2019)
  * [38] van der Maaten, L.: Accelerating t-sne using tree-based algorithms. J. Mach. Learn. Res. 15(1), 3221–3245 (2014)
  * [39] Mitrovic, J., McWilliams, B., Walker, J.C., Buesing, L.H., Blundell, C.: Representation learning via invariant causal mechanisms. In: ICLR. OpenReview.net (2021)
  * [40] Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving jigsaw puzzles. In: ECCV (6). Lecture Notes in Computer Science, vol. 9910, pp. 69–84. Springer (2016)
  * [41] Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: CVPR. pp. 2536–2544. IEEE Computer Society (2016)
  * [42] Qiao, S., Shen, W., Zhang, Z., Wang, B., Yuille, A.L.: Deep co-training for semi-supervised image recognition. In: ECCV (15). Lecture Notes in Computer Science, vol. 11219, pp. 142–159. Springer (2018)
  * [43] Roh, B., Shin, W., Kim, I., Kim, S.: Spatially consistent representation learning. In: CVPR. pp. 1144–1153. Computer Vision Foundation / IEEE (2021)
  * [44] Tan, K.C., Liu, Y., Ambrose, B., Tulig, M., Belongie, S.J.: The herbarium challenge 2019 dataset. CoRR abs/1906.05372 (2019)
  * [45] Troisemaine, C., Lemaire, V., Gosselin, S., Reiffers-Masson, A., Flocon-Cholet, J., Vaton, S.: Novel class discovery: an introduction and key concepts. CoRR abs/2302.12028 (2023)
  * [46] Vaze, S., Han, K., Vedaldi, A., Zisserman, A.: Generalized category discovery. In: CVPR. pp. 7482–7491. IEEE (2022)
  * [47] Vaze, S., Han, K., Vedaldi, A., Zisserman, A.: Open-set recognition: A good closed-set classifier is all you need. In: ICLR. OpenReview.net (2022)
  * [48] Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-ucsd birds-200-2011 dataset (2011)
  * [49] Wang, Y., Chen, H., Heng, Q., Hou, W., Fan, Y., Wu, Z., Savvides, M., Shinozaki, T., Raj, B., Schiele, B.: Freematch: Self-adaptive thresholding for semi-supervised learning. CoRR abs/2205.07246 (2022)
  * [50] Wei, C., Wang, H., Shen, W., Yuille, A.L.: CO2: consistent contrast for unsupervised visual representation learning. In: ICLR. OpenReview.net (2021)
  * [51] Yang, C., An, Z., Cai, L., Xu, Y.: Mutual contrastive learning for visual representation learning. In: AAAI. pp. 3045–3053. AAAI Press (2022)
  * [52] Yang, M., Zhu, Y., Yu, J., Wu, A., Deng, C.: Divide and conquer: Compositional experts for generalized novel class discovery. In: CVPR. pp. 14248–14257. IEEE (2022)
  * [53] Zhang, B., Wang, Y., Hou, W., Wu, H., Wang, J., Okumura, M., Shinozaki, T.: Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In: NeurIPS. pp. 18408–18419 (2021)
  * [54] Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: ECCV (3). Lecture Notes in Computer Science, vol. 9907, pp. 649–666. Springer (2016)
  * [55] Zhao, B., Wen, X., Han, K.: Learning semi-supervised gaussian mixture models for generalized category discovery. CoRR abs/2305.06144 (2023)
  * [56] Zheng, M., Wang, F., You, S., Qian, C., Zhang, C., Wang, X., Xu, C.: Weakly supervised contrastive learning. In: ICCV. pp. 10022–10031. IEEE (2021)
  * [57] Zhong, Z., Fini, E., Roy, S., Luo, Z., Ricci, E., Sebe, N.: Neighborhood contrastive learning for novel class discovery. In: CVPR. pp. 10867–10875. Computer Vision Foundation / IEEE (2021)


On the Effectiveness of Observations in the Mid-Infrared Wavelength Range on
the 2.5-Meter Telescope of the Caucasus Mountain Observatory of Moscow State
University with Commercial IR Cameras

S. G. Zheltoukhov1,2, A. M. Tatarnikov1,2a

1Sternberg Astronomical Institute, Moscow State University, Moscow 119234,
Russia.
2Department of Experimental Astronomy, Faculty of Physics, M.V.Lomonosov
Moscow State University, Moscow 119991, Russia.
E-mail<EMAIL_ADDRESS>
Received July 11, 2022; revised October 10, 2022; accepted October 12, 2022

The main factors that influence the success of observations in the infrared
range (central wavelengths of the photometric bands at 3.75 and 4.8 $\mu$m) on
the multipurpose optical telescope are considered. Estimates of the sky
background brightness are obtained for the Caucasus Mountain Observatory (CMO)
of Moscow State University: $1.3\cdot 10^{6}$ photons/(s pixel) in the 3.75
$\mu$m band and $3.4\cdot 10^{6}$ photons/(s pixel) in the 4.8 $\mu$m; and the
instrumental background for the 2.5-m CMO telescope at $0^{\circ}$С: $3.2\cdot
10^{6}$ photons/(s pixel) in the 3.75 $\mu$m band and $4.3\cdot 10^{6}$
photons/(s pixel) in the 4.8 $\mu$m band. It is shown that at this background
signal level with the currently available commercial cameras in the $3-5$
$\mu$m spectral range, the telescope-camera coupling capabilities for
observing faint objects will still be limited by the thermal background. For
different observational conditions, estimates of the limiting magnitudes of
objects available for observations in the 3.75 and 4.8 $\mu$m ranges are
obtained. For average observation conditions (instrument temperature of
$0^{\circ}$С and stellar image size of $1^{\prime\prime}$), the limit is $\sim
10.6^{m}$ and $\sim 8.4^{m}$, respectively.

Keywords: Infrared: general – Instrumentation: photometers – Instrumentation:
miscellaneous

PACS: 07.57.-c.

## Introduction

Infrared (IR) radiation is electromagnetic radiation over a wide range of
wavelengths — from 0.8 to 300-500 $\mu$m. The infrared range borders the
visual range on the short wavelength side and the microwave range on the long
wavelength side. For convenience, this range is divided into several sub-
ranges. One of the most commonly used splits is as follows: near-infrared
range with boundaries of 0.8 - 2.5 $\mu$m, mid-infrared range with boundaries
of 2.5 - 25 $\mu$m, far-infrared range with boundaries of 25 - 500 $\mu$m. In
the near-infrared range, it is possible to carry out astronomical observations
with ordinary non-adapted reflector telescopes, in the mid-infrared range to
reduce the instrumental background it is necessary to apply various design
solutions (a cooled Lyot stops, no telescope baffles, small secondary mirror
size, radiation flux modulation, etc.; the longer the wavelength, the more
adaptation of the telescope is required), in the far IR, observations are
possible only from a spacecraft or high-altitude airplane.

At present, there are not many specialized IR telescopes in operation in the
world, on which observations in the mid-IR range are possible. In Russia,
there is only one such telescope — AZT-33IR [1], on which observations are
currently carried out in the optical wavelength range of 0.3-1.1 $\mu$m. In
the IR in Russia, regular observations are carried out on only two non-adapted
telescopes — in the 1-5 $\mu$m range at the 1.25-m telescope ZTE of the
Crimean Astronomical Station of SAI (with a single-element photometer [2],
[3]) and in the 1-2.5 $\mu$m range at the 2.5-m telescope of the Caucasus
Mountain Observatory of SAI (CMO, [4]) with a camera-spectrograph ASTRONIRCAM
[5].

In recent years, due to the progress of IR detection technology, the cost of
IR detectors and IR cameras have tended to decrease. At the same time, the
characteristics are improving and the spectral range accessible to
commercially available serial products is increasing. The purpose of the
present work is to estimate the background level and determine the limiting
magnitudes of the current unadapted for IR observations 2.5-m telescope of CMO
when operated with a commercial IR detector at wavelengths from 3 to 5 $\mu$m
and to present the design of a IR camera prototype.

##  1\. Problem statement

CMO is located 20 km south of Kislovodsk (Russia) at an altitude of 2100 m.
Astroclimate monitoring performed by Kornilov et al. in 2007 -2013 [6] showed
that for the CMO, the median value of the FWHM of stars (Full Width at Half
Magnitude — width of the star’s image profile at the level of half of the
maximum value) in the visible wavelength range is 0.96 arcsec, the median
value of precipitated water column for clear nights is $PWV=7.8$ mm, and the
greatest number of clear nights falls in the fall-winter period. The
relatively low atmospheric water content (especially in late fall and winter)
and the high altitude of the observatory above sea level make observations in
the infrared possible. The main instrument of the observatory — 2.5-m
telescope, which has been operating in test mode since 2014 and will be
officially commissioned at the end of 2021.

The 2.5-m telescope of CMO is a universal multitasking telescope that allows
us to perform a wide range of studies. It is built according to the optical
scheme of the classical Ritchey-Chretien reflector (Table 1) with an
additional tertiary Nasmyth mirror M3. Movement and rotation of the M3 mirror
help to realize five different positions of the focal plane: the Cassegrain
focus (the M3 mirror is out of the beam) and the four Nasmyth focuses (the M3
mirror is introduced into the beam and tilted at an angle of $45^{\circ}$ to
the optical axis of the main mirror), realized by rotating the M3 mirror
around the optical axis of the telescope’s main mirror. It is planned that the
mid-infrared camera will be located at one of the Nasmyth focuses. From a
camera design point of view, all these focuses are equivalent.

At present, there are detectors and cameras of the mid-IR of several vendors
on the market costing 50k-70k EUR. This is an order of magnitude (or more)
lower than the cost of scientific detectors with export restrictions. They use
InSb or HgCdTe matrix as a sensor, and cooling to temperatures of $<100$K is
carried out by a Stirling machine. However, not all existing even commercial
cameras are available for ordering in Russia. Therefore, for further
consideration, we will take average parameters of available commercial ones.
As will be shown below, an important property of the detector is its
sensitivity in the wavelength range required for astronomical observations. We
will assume that the used detector operates in the entire wavelength range
(i.e., from 3 to 5 $\mu$m) with a sharp fall in sensitivity at the long-
wavelength boundary.

The parameters that determine the possibility of observing bright objects or
working at high background level are the minimum accumulation time and the
charge capacity of a pixel. Characteristic values of these parameters are
$t_{min}<1$ ms and $P_{e}\sim 8\cdot 10^{6}$ e-. The parameters determining
the angular field of view and angular resolution of the camera for
astronomical observations on a particular telescope are the size of the
detector sensitive area and the pixel size $a$. Let us set the first value to
the often encountered format of 640x512 pixels, and for the second — $a=15$
µm.

We set the basic parameters of the entire path (Earth’s atmosphere, telescope,
and detector) through which the radiation of observed astronomical objects
passes. Based on these parameters, we investigate the limitations imposed by
them on the accuracy of photometric observations and evaluate the limits of
combining a telescope not adapted to IR observations with a commercial mid-IR
camera.

Table 1: Main parameters of the 2.5-m telescope of CMO Parameter | Value
---|---
Optical scheme | Ritchey-Chretien
Main mirror diameter $D$ | 2.5 m
Relative aperture $A$ | 1/8
Linear central obscuration $d$ | 0.408
Mirror reflectivity | 0.95
Diameter of the star image, where 80% of the energy is collected | $0.3^{\prime\prime}$
Image Scale |  $10^{\prime\prime}/$мм
Maximum pointing speed | 3∘/с
Maximum pointing acceleration | 1∘/с2

##  2\. Estimation of sky background brightness

One of the main factors interfering with ground-based observations in the IR
range is the influence of the Earth’s atmosphere. The smoothed characteristic
curve of atmospheric transmission in the IR range is shown in Fig. 1. Up to a
wavelength of 20 $\mu$m, we can see the alternation of absorption and
transmission bands (called windows of atmospheric transparency). The main
absorbing agents are water vapor, carbon dioxide, and ozone. Their
concentration in the atmosphere is not constant, which causes the shape of
absorption bands near their boundaries to change.

Figure 1: Dependence of the transmission of the Earth’s atmosphere on
wavelength (for altitude 2000 m above the Earth’s surface and PWV=1 mm) and
the transmission curves of the $L^{\prime}$ (dashed-dotted line) and
$M^{\prime}$ (dashed line) filters

CMO is located at an altitude of 2100 m above sea level. For IR observations
in the conditions of the mountain observatory, a broadband photometric system
MKO-NIR (see [7] and[8]) with bands $J,H,K,L^{\prime},M^{\prime}$ was
developed in the early 2000s. The reaction curves of the system are chosen in
such a way that the strong absorption bands of water vapor and carbon dioxide,
as well as the variable boundaries of these absorption bands, are avoided as
far as possible. ASTRONIRCAM IR camera operates in the near-IR range (bands
$J,H,K$ of the MKO-NIR system) on the 2.5-m telescope of CMO. Therefore, the
MKO-NIR system was also chosen to realize the mid-IR photometric bands with
the new camera. The parameters of the $L^{\prime}$ and $M^{\prime}$ filters
are given in Table 2, and their response curves are shown in Fig. 1. It can be
seen that for the $M^{\prime}$ band, it was not possible to achieve complete
independence of the system response curve from the atmospheric transmission.

Table 2: Main parameters of the $L^{\prime}$ and $M^{\prime}$ filters Parameter | $L^{\prime}$ | $M^{\prime}$
---|---|---
Central wavelength, $\mu$m | 3.75 | 4.70
Bandwidth | 0.7 | 0.21
(at 50% of the peak), $\mu$m |  |
Peak transmission | 0.9 | 0.9
Transmission outside the workig range | $<0.005$ | $<0.005$

In the infrared range, the Earth’s atmosphere is a powerful source of
background radiation. The value of the radiation flux depends strongly on the
wavelength — if in the $K$ band (2.2 $\mu$m) the surface brightness of the sky
is $13.1^{m}$ per square arcsec (measured from observations at CMO), then in
the $L^{\prime}$ band it is $4^{m}$, and in the $M^{\prime}$ band it is
$1^{m}$ 111https://www.gemini.edu/observing/telescopes-and-sites/sites#Near-
IR-long memorandum [9] was used to calculate the data on this page. . The
corresponding values in terms of energy are $F_{K}=3\cdot 10^{-5}$ W/m2/sr,
$F_{L^{\prime}}=4\cdot 10^{-2}$ W/m2/sr, $F_{M^{\prime}}=8\cdot 10^{-2}$
W/m2/sr. The brightness in the $L^{\prime}$ and $M^{\prime}$ bands depend on
PWV —as the amount of precipitated water increases from 2 mm to 10 mm, the
brightness of the sky background in these bands changes by about 20%
11footnotemark: 1. In Fig. 2 shows a plot of the dependence of the number of
photons emitted by 1 square arcsec of sky on wavelength. By convolving the
above data with the transmission curves of the $L^{\prime}$ and $M^{\prime}$
filters, we can obtain the photon flux in them determined by the background:
for the $L^{\prime}$ band at the 2.5-meter telescope of CMO (see Table 1) the
flux is $1.3\cdot 10^{6}$ photons/s, for the $M^{\prime}$ flux is $3.4\cdot
10^{6}$ photons/s.

The accuracy of observations is affected not only by the high brightness of
the sky background but also by its rapid fluctuations that are typical for the
mid-infrared range. To account for fluctuations, it is necessary to
simultaneously (quasi-simultaneously) observe both the background and the
object, which is possible due to the high frame rate of the IR detectors used.

Figure 2: Sky background spectrum from Gemini website
(https://www.gemini.edu/observing/telescopes-and-sites/sites#Near-IR-long) for
airmass 1.5. Blue curve — sky background for PWV=10 mm. Orange curve — for
PWV=2.3 mm. The dotted lines show the transmission curves of the $L^{\prime}$
and $M^{\prime}$ filters

##  3\. Estimation of instrumental background

In the mid-IR and far-IR, the thermal radiation of the telescope and some
parts of the equipment strongly affects the obtained images. Starting from
about 2 $\mu$m wavelength, the power of thermal radiation of a telescope not
adapted to IR observations becomes comparable to the power of the background
radiation of the sky. Below are calculations of the photon flux falling on the
detector from the telescope and structural elements of the camera for
different temperatures.

###  3..1 Telescope Radiation

To determine the contribution of the telescope radiation to the instrumental
background, we measured the thermal radiation from the telescope elements
using a thermal imager. The operating wavelength range of the thermal imager
is 7-14 µm, which allows us to see the intrinsic thermal radiation, which is
most intense in this range. The thermal imager was placed in the same location
where the IR camera will be installed in the future. As a result, we obtained
a heat map for the internal parts of the telescope that will eventually fall
within the camera’s field of view. The corresponding image can be seen in Fig.
3.

Figure 3: Thermal map of the inner parts of the telescope obtained on December
24, 2021 at an air temperature outside the dome of $-15^{\circ}$C. An enlarged
fragment of the inner part of the image is shown on the right; the color scale
is different in the images. Point 1 ($-7.2^{\circ}$C) — the M3 mirror baffle,
point 2 ($-32.9^{\circ}$C) — the sky visible after reflection from the three
mirrors of telescope, point 3 ($+7.2^{\circ}$C) — the reflection of a warm
observer with a thermal imager in the telescope’s secondary mirror, point 4
($-8.1^{\circ}$C) — reflection of the inner part of the M3 mirror baffle in
the secondary mirror, point 5 ($-30.5^{\circ}$C) — temperature measured near
the image of the metal spider on which the secondary mirror is attached

To a first approximation, the emissivity coefficient of the non-mirror
elements of the telescope can be considered close to 1 both at the operating
wavelengths of the thermal imager and in the $L^{\prime}$ and $M^{\prime}$
bands. In this case, the temperature data obtained with the thermal imager and
the angular dimensions of the telescope elements visible from the detector are
unambiguously converted to the signal seen by the detector. Let us estimate
the number of photons $N_{ph}$ coming from an element with temperature $T$ and
angular area $\Omega$ per detector pixel with linear size $a$ in some
wavelength range:

$N_{ph}=\Omega
a^{2}\int^{\lambda_{2}}_{\lambda_{1}}\frac{\lambda}{hc}\frac{2hc^{2}}{\lambda^{5}}\frac{1}{\exp\left(\frac{hc}{\lambda
kT}\right)-1}d\lambda$ (1)

The total solid angle that a detector pixel must see during observations is
determined by the relative telescope aperture $A$, and for a 2.5-meter
telescope of CMO it is 41 square degrees. Of these, 35 square degrees — the
area of relatively cold sky visible in the telescope (or more precisely — its
reflections from the telescope mirrors) and at least 6 square degrees will be
occupied by the warm elements of the telescope. Since the telescope is not
optimized for IR observations, the detector’s field of view includes the
radiation from the primary mirror rim and the secondary mirror baffle, as well
as the spider on which the secondary mirror is attached (see Fig. 3).

From the data presented in Fig. 3, we can get an estimate of the emissivity of
the mirrors of a 2.5-meter telescope. To do this, we need to determine the
’’temperature’’ of the sky observed without the telescope and the sky seen
through the telescope. After disabling the internal calibrations of the
thermal imager, which take into account the influence of the distance to the
object and the radiation reflected from it, we can consider that the flux
received by the thermal imager is directly proportional to the integral of
Planck’s formula within the sensitivity of the thermal imager (from 7 to 14
microns) with the body temperature equal to that given by the thermal imager.
Let us denote this integral by $B_{f}(T)$. We can now make an equation for the
total emissivity of the mirrors, $\varepsilon$. Measurements showed that the
’’temperature’’ of the sky was $T_{s}=-39^{\circ}$C and the ’’temperature’’ of
the sky through the telescope — $T_{t}=-32.9^{\circ}$. At the same time,
sensors placed directly on the surface of the mirror showed that its
temperature $T_{m}=-5^{\circ}$C. Thus we obtain the equation 2, from which we
obtain the value $\varepsilon=0.14$, which is quite plausible for 3
consecutive mirrors.

The final results of the flux calculations from the telescope can be seen in
Table 3.

$B_{f}(T_{t})=(1-\varepsilon)B_{f}(T_{s})+\varepsilon B_{f}(T_{m})$ (2)

Table 3: Background created by the telescope Temperature of the telescope, ∘C | Signal in the $L^{\prime}$ band, ph/s/pix | Signal in the $M^{\prime}$ band, ph/s/pix
---|---|---
-20 | 470 000 | 1 000 000
-10 | 820 000 | 1 600 000
0 | 1 400 000 | 2 500 000
10 | 2 200 000 | 3 700 000
20 | 3 500 000 | 5 300 000

To verify these estimates, we measured the telescope’s contribution to the
background radiation using an independent method. For this purpose, we used
observations made with the ASTRONIRCAM near-infrared camera [5] has already
been installed on the telescope. At wavelength >2 $\mu$m, the contribution of
the telescope thermal emission becomes noticeable against the background of
other sources. Therefore, we derived the dependence of the background signal
recorded in the $K$ band (2.2 $\mu$m) on the telescope temperature (Fig. 4).
We assume that the brightness of the sky background depends weakly on the
temperature of the near-surface air layer (due to the high atmospheric
transparency outside the absorption bands). This makes it possible to separate
the contribution to the total background flux from the sky and from the
telescope. By approximating the obtained points with a model dependence, we
obtain that the signal from the sky in each element of the ASTRONIRCAM
detector is approximately 210 counts per second, and from the telescope at 0∘C
— 250 counts per second. The filters and structural elements of the
ASTRONIRCAM camera are cooled to cryogenic temperatures, so here we neglect
their radiation. Next, with the knowledge of the photoelectron-to-counts
conversion factor (gain=2.2) and the quantum efficiency of the ASTRONIRCAM
camera detector [5], we can determine the number of $N_{K}$ photons falling in
the $K$ band per unit time per unit area of the detector. Considering the
radiation to be blackbody, we can calculate the flux coming from the telescope
and in the filters $L^{\prime}$ and $M^{\prime}$ of interest. To do this,
multiply the obtained value $N_{K}$ by the ratio of integrals from the Planck
function for the corresponding spectral ranges and recalculate the flux by the
pixel size $a$ of the detector under study (see formula 3).

$N_{LM}=N_{K}\cdot
a^{2}\frac{\int^{\lambda_{2}}_{\lambda_{1}}\frac{\lambda}{hc}\frac{2hc^{2}}{\lambda^{5}}\frac{1}{\exp\left(\frac{hc}{\lambda
kT}\right)-1}d\lambda}{\int^{\lambda_{K2}}_{\lambda_{K1}}\frac{\lambda}{hc}\frac{2hc^{2}}{\lambda^{5}}\frac{1}{\exp\left(\frac{hc}{\lambda
kT}\right)-1}d\lambda}$ (3)

After calculating these values, we obtain data that coincide well with the
results presented in the table 3, which were calculated in a different way.
However, it should be taken into account that even small changes in the
initial data can change the calculated background level by a factor of almost
2, i.e., the obtained results can be considered as a characteristic estimate.
Nevertheless, the coincidence of the results obtained with a high probability
means a good plausibility of the models used and a correct estimate of the
radiative angular area of the telescope in the first calculation method.

Figure 4: Dependence of the total background value in the $K$ band on the
temperature of the telescope mirror. The line shows the model dependence used
in further calculations

###  3..2 Radiation from photometric filters

In addition to the radiation from various parts of the telescope, an important
source of the background signal is the structural elements of the infrared
camera itself. First of all, it is the entrance filter as an element occupying
the largest part of the field of view. A formula similar to the 1 was used to
calculate the intrinsic emission of the filters:

$E_{ph}=\Omega_{T}a^{2}\int^{\lambda_{2}}_{\lambda_{1}}\varepsilon(\lambda)\frac{\lambda}{hc}\frac{2hc^{2}}{\lambda^{5}}\frac{1}{\exp\left(\frac{hc}{\lambda
kT}\right)-1}d\lambda$ (4)

In this formula, $\Omega_{T}$ is the total solid angle of the beam coming from
the telescope, and $\varepsilon$ is the emissivity of the filter. If the
filter were completely transparent (or completely mirrored), its emissivity
$\varepsilon$ would be 0. A real filter will have a transmission curve
$T(\lambda)$ determined by the desired photometric system (see Fig. 1). In
addition, the filters will have a reflectance curve $R(\lambda)$ when viewed
from the detector side. Typical transmission and reflection curves of
interference filters, are shown in Fig. 5. Thus, the magnitude of the
emissivity can be estimated by the formula
$\varepsilon(\lambda)=1-T(\lambda)-R(\lambda)$. It usually ranges from a few
percent to 10-15 % in the range of our interest.

Figure 5: The shape of the transmission (solid blue curve) and reflection
curves from both sides (dashed line — looking at the filter from the detector
side, dotted line — from the telescope side) for a typical interferometric
filter

Then, it is necessary to take into account the temperature of the radiation
reflected from the filter. Due to the features of the device design described
further in section 4. in this calculation, we took it equal to the internal
temperature of the detector cryostat $-100^{\circ}$C. This reflected flux
turns out to be much smaller than the intrinsic radiation of the filters. The
obtained values of fluxes from filters strongly depend on the chosen value of
emissivity of the filter material. The results of calculations for the value
$\varepsilon=0.04$ can be seen in Table 4. The values for both filters are the
same because it is considered that $\varepsilon$ does not depend on the
spectral range.

Table 4: Background produced by photometric filters (for $\varepsilon=4\%$ over the entire sensitivity range of the detector) Filter temperature, ∘C | Background, ph/(s$\cdot$pix)
---|---
-20 | 730 000
-10 | 1 200 000
0 | 1 800 000
10 | 2 600 000
20 | 3 800 000

##  4\. Possible design of the IR camera

One of the main tasks to be solved when creating an infrared device is to
minimize the instrumental background. This requires minimizing the number of
warm optical elements in the instrument circuitry. In our case, the detector
is shipped with its own cryostat and cooling system based on a Stirling
machine. The low power of the cooler leads to the fact that only the detector
has direct contact with the site cooled by the machine. The field of view of
the detector is limited by a cold aperture located in one vacuumized volume
with the detector. The entrance window is a thin germanium plate mounted on
the end of a polished steel cone, which can be used to mount the camera. In
this case, the camera can be mounted on an additional cryostat with a
vacuumized volume in which to arrange the cooled filters, optics, and limiting
cold apertures. However, the use of an additional cryostat dramatically
increases the final cost of the project. Therefore, we propose a simplified
camera design for consideration.

In a design without a cryostat, it is possible to limit to one element located
in the light beam feeding the instrument, namely the photometric filter. The
relative aperture of the 2.5-m telescope is $A=1/8$, which gives small ray
convergence angles and allows the interference filters to operate in a
converging beam without a noticeable change in transmission bandwidth.
However, this angle is much smaller than the angular field of view of the
detector, which is often calculated based on $A=1/4$. Outside of their
transmission bandwidth, interference filters are mirrors with high reflection
coefficient. Characteristic transmission and reflection curves can be seen in
Fig. 5. Thus, to reduce the instrumental background, it is necessary to
minimize the temperature of the reflected radiation. By installing the filters
in close proximity to the entrance window of the detector, it is possible to
achieve that the light filters will reflect the cold interior of the detector.
To limit the magnitude of the signal through the filter due to the difference
in relative apertures of the telescope and detector, a mirrored aperture with
an appropriate inner size and an outer size larger than the filter should be
installed. The mirrored diaphragm should have a concave surface to exclude
reflection of warm parts of the camera in it. The general scheme of the
resulting assembly is shown on the figure 6.

Figure 6: Beam path, aperture and filter position in camera design

##  5\. Estimation of the limiting magnitude

On the basis of the obtained data on the instrumental background and sky
background in the analyzed ranges for the selected variant of the camera
design, it is possible to estimate the limiting stellar magnitudes. For this
purpose, let us write down the theoretical signal-to-noise ratio ($SNR$):

$SNR=\frac{F_{0}\cdot 2.512^{-m}\cdot t}{\sqrt{F_{0}\cdot 2.512^{-m}\cdot
t+2((a_{s}/a_{pix})^{2}\cdot F_{bg}\cdot
t+(a_{s}/a_{pix})^{2}\cdot\sigma_{d}^{2}/n)}},$ (5)

where $t$ is the signal accumulation time, $F_{0}$ is the number of
photoelectrons captured by the detector from a star with brightness $0^{m}$,
$m$ is the stellar magnitude of the object, $F_{bg}$ is the number of
photoelectrons detected by the detector from the background (the sum of the
instrumental background, sky background, and dark current) per 1 square arcsec
of the sky, $a_{s}$ and $a_{pix}$ are the dimensions of the star and pixel
image in arcsec, respectively, $n$ is the number of single frames involved in
obtaining the final image. The factor 2 before the second summand in the
denominator accounts for the fact that subtracting the background during
processing will double the corresponding noise. The calculations assume that
the quantum efficiency of the detector is 0.85, and the transmission of the
atmosphere, telescope, and filter is 0.5. Unfortunately, the manufacturer does
not specify the values of the readout noise and dark current of the detector,
but for such detectors the noise caused by these effects is usually much lower
than the Poisson background noise[10]. Under these conditions, the value of
the expected limiting stellar magnitude for a given background level, $SNR$,
and accumulation time can be easily obtained by solving the equation 5 with
respect to $m$.

Limitations on the minimum exposure time by the electronics of the detector
affect the ability to observe bright objects. A typical value of the minimum
exposure of commercial detectors can be considered to be 300-400 $\mu$s. With
a cell capacity of $8\cdot 10^{6}$ e- we can observe stars no brighter than
$-4.5^{m}$ in $L^{\prime}$ and $-7^{m}$ in $M^{\prime}$.

The maximum duration of a single exposure is limited by the maximum pixel
capacity ($8\cdot 10^{6}$ e-). For the characteristic background values found
above, the maximum exposure will be $t_{max}=1..3$ s. However, if the readout
noise is negligible, one long exposure is not fundamentally distinguishable
from the sum of many short exposures. This allows to bypass limitations on the
value of $t_{max}$ and, if necessary, to accumulate hundreds of frames to
obtain the necessary $SNR$. In addition, observations in the mode of stacking
of separate frames to obtain the final image allow to organize the modulation
of the light flux, when the center of the field of view moves by $5~{}-10$
arcsec between separate frames. The modulation frequency is quite high
($0.3..10$ Hz) and the implementation of this method requires the use of an
additional chopping mirror installed between the entrance window of the camera
and the elements of the telescope optical scheme. For further calculations, we
set the accumulation time $t$ equal to 1 s.

We will consider the value $SNR=3$ as the detection limit of a point object on
the frame. In Table 5 we present the results of calculations for different
cases. The first three rows of the table are obtained with fixed values of
$\varepsilon=0.04$ and star image size $FWHM=1^{\prime\prime}$. It can be seen
that the transition from a cold telescope and cold filters to heated filters
leads to a loss of about $1^{m}$. Observations made at low image quality lead
to the same loss. The use of filters with high emissivity also degrades the
camera limit.

In Fig. 7, the dependence of the integral accumulation time $t$ required to
reach a certain stellar magnitude $m$ of a point object for two typical values
of $SNR$ is given for the observing conditions specified in the second row of
Table 5 for two typical values of $SNR$: $SNR=3$ is the limit of detectability
and $SNR=100$ is the case of accurate photometry.

Table 5: The estimated limiting stellar magnitudes of point sources Conditions |  $L^{\prime}$ band |  $M^{\prime}$ band
---|---|---
$T_{tel}=-20^{\circ}$C, $T_{fltr}=-40^{\circ}$C, $\varepsilon_{fltr}=0.04$, $FWHM=1^{\prime\prime}$ | 11.2m | 8.8m
$T_{tel}=0^{\circ}$C, $T_{fltr}=0^{\circ}$C, $\varepsilon_{fltr}=0.04$, $FWHM=1^{\prime\prime}$ | 10.6m | 8.4m
$T_{tel}=+20^{\circ}$C, $T_{fltr}=+20^{\circ}$C, $\varepsilon_{fltr}=0.04$, $FWHM=1^{\prime\prime}$ | 10.2m | 8.1m
$T_{tel}=+20^{\circ}$C, $T_{fltr}=+20^{\circ}$C, $\varepsilon_{fltr}=0.2$, $FWHM=1^{\prime\prime}$ | 9.7m | 7.6m
$T_{tel}=+20^{\circ}$C, $T_{fltr}=+20^{\circ}$C, $\varepsilon_{fltr}=0.04$, $FWHM=2.5^{\prime\prime}$ | 9.2m | 7.1m

Figure 7: The dependence of exposure time on the brightness of the star at
telescope temperature and filters $0^{\circ}$C, $FWHM=1^{\prime\prime}$ and
$\varepsilon_{fltr}=0.04$. The red solid line — $L^{\prime}$ band for $SNR=3$.
Red dashed line — $L^{\prime}$ band for $SNR=100$. Blue solid line —
$M^{\prime}$ band for $SNR=3$. Blue dashed line — $M^{\prime}$ band for
$SNR=100$

## Conclusion

For the combination of a universal optical telescope and a commercial IR
camera operating in the short-wave end of the mid-IR range (3-5 $\mu$m)
considered in our work, for various observing conditions we obtained the
values of the contribution to the background signal from all components: the
Earth’s atmosphere, the telescope, and the structural elements of the camera
(see Tables 3, 4, and Fig. 2). A comparison of the values of these components
suggests that their contributions are roughly equal in the $L^{\prime}$ band,
while the $M^{\prime}$ band is dominated by atmospheric radiation at telescope
temperatures below $10^{\circ}$ C, and dominated by telescope radiation at
warmer telescopes. To minimize the contribution from the camera components
(while keeping the system cost to a minimum), we propose a layout of a filter
with a concave mirrored aperture (Fig. 6).

Calculations of the limiting stellar magnitude of objects accessible to
observations with the camera were carried out for different observing
conditions. Table 5 shows that the losses during the transition from variant
to variant are $0.5-1^{m}$. The greatest influence is the quality of the image
during observations. In the paper[10], an estimate of the limiting magnitude
in the $M^{\prime}$ band ($M^{\prime}=13^{m}$) was obtained for an adapted 10
m-diameter telescope installed in a high-altitude observatory for the same
$SNR$ values and accumulation time. A direct conversion to the 2.5-m CMO
telescope results in a limiting magnitude value of $10^{m}$, which is about
$1^{m}$ better than the estimate we obtained for the proposed camera design.
This difference is mainly due to the smaller contribution of the atmosphere at
high altitude and the lower temperature of the telescope.

Our estimates of the limiting magnitude show that more than 1 million objects
will be available for observations with the proposed infrared camera,
according to the WISE[11] catalog. These are stars of various types and
classes, galaxies of various types, planetary nebulae, and other objects. It
is easy to get estimates of the sizes of the Solar System bodies that can be
accessible to observations with the described equipment. For example, an
’’gray’’ asteroid with a diameter of 100 km in the main belt in opposition
with an albedo of 0.2 would have a brightness of $m_{L^{\prime}}=8.7^{m}$ and
$m_{M^{\prime}}=8.5^{m}$, which is close to our detection limit. A body with a
diameter of 30 m at the Moon’s distance with the same albedo would have
brightness $m_{L^{\prime}}=9.6^{m}$ and $m_{M^{\prime}}=7.7^{m}$. This large
change in the color index is due to the strongly increased intrinsic emission
of the body closer to the Sun in the longer wavelength. If we consider low
Earth orbit, the camera will have access to centimeter-sized objects.

The study was supported by the Scientific Educational School of Lomonosov
Moscow State University ‘‘Fundamental and Applied Space Research’’. The work
of S. Zheltoukhov was supported by the Foundation for the Advancement of
Theoretical Physics and Mathematics ‘‘BASIS’’. This publication makes use of
data products from the Wide-field Infrared Survey Explorer, which is a joint
project of the University of California, Los Angeles, and the Jet Propulsion
Laboratory/California Institute of Technology, funded by the National
Aeronautics and Space Administration.

## References

  * 1. Kamus S., Denisenko N.A., Lipin V.I. // J. Opt. Technol. 2002. 69. P. 674.
  * 2. Nadjip A.E., Shenavrin V.I., Tichonov V.G. // Trudy Gosudarstvennogo Astronomicheskogo Instituta. 1986. 58. P. 119.
  * 3. Shenavrin V.I., Taranova O.G., Nadzhip A.E. // Astronomy Reports. 2011. 55. P. 31.
  * 4. Shatsky N., Belinski A., Dodin A. et al. // Ground-Based Astronomy in Russia. Proc. 2020. 1. P. 127.
  * 5. Nadjip A.E., Tatarnikov, A.M., Toomey D. W. et al. // Astrophysical Bulletin 2017. 72. P. 349.
  * 6. Kornilov V., Safonov B., Kornilov M. et al. // Publications of the Astronomical Society of the Pacific. 2014. 126. P. 482.
  * 7. Simons D.A., Tokunaga A. // Publications of the Astronomical Society of the Pacific. 2002. 114. P. 169.
  * 8. Tokunaga A.T., Simons D.A., Vacca W.D. // Publications of the Astronomical Society of the Pacific. 2002. 114. P. 180.
  * 9. Lord S.D. // NASA Technical Memorandum. 1992.
  * 10. Zhao Zhi-Jun, Zhou Hai-Jing, Zhang Yu-Chen et al. // Research in Astronomy and Astrophysics. 2021. 21. P. 81.
  * 11. Wright E. L., Eisenhardt P.R.M., Mainzer A.K. et al. // Astronomical Journal. 2010. 140. P. 1868.


# The Python Sky Model 3 software

Andrea Zonca  Ben Thorne  Nicoletta Krachmalnicoff  Julian Borrill

††margin:  DOI: https://doi.org/DOI unavailablehttps://doi.orgDOI unavailable
Software • N/Ahttps://doi.orgReview • NO_REPOSITORYhttps://doi.orgRepository •
DOI unavailablehttps://doi.orgArchive Editor:
https://example.comhttps://doi.orgPending Editor
Reviewers: • https://github.com/Pending Reviewershttps://doi.org@Pending
Reviewers Submitted: N/A
Published: N/A License
Authors of papers retain copyright and release the work under a Creative
Commons Attribution 4.0 International License
(http://creativecommons.org/licenses/by/4.0/https://doi.orgCC BY 4.0).

## Statement of Need

The Cosmic Microwave Background (CMB) radiation, emitted just 370 thousand
years after the Big Bang, is a pristine probe of the Early Universe. After
being emitted at high temperatures, the CMB was redshifted by the subsequent
13.8 billion years of cosmic expansion, such that it is brightest at microwave
frequencies today. However, our own Milky Way galaxy also emits in the
microwave portion of the spectrum, obscuring our view of the CMB. Examples of
this emission are thermal radiation by interstellar dust grains, and
synchrotron emission by relativistic electrons spiraling in magnetic fields.
Cosmologists need to create synthetic maps of the CMB and of the galactic
emission based on available data and on physical models that extrapolate
observations to different frequencies. The resulting maps are useful to test
data reduction algorithms, to understand residual systematics, to forecast
maps produced by future instruments, to run Monte Carlo analysis for noise
estimation, and more.

## Summary

The Python Sky Model (PySM) is a Python package used by Cosmic Microwave
Background (CMB) experiments to simulate maps, in HEALPix (Górski et al.,
2005; Zonca et al., 2019) pixelization, of the various diffuse astrophysical
components of Galactic emission relevant at CMB frequencies (i.e. dust,
synchrotron, free-free and Anomalous Microwave Emission), as well as the CMB
itself. These maps may be integrated over a given instrument bandpass and
smoothed with a given instrument beam. The template emission maps used by PySM
are based on Planck (Planck Collaboration, 2018) and WMAP (Bennett et al.,
2013) data and are noise-dominated at small scales. Therefore, PySM simulation
templates are smoothed to retain the large-scale information, and then
supplemented with modulated Gaussian realizations at smaller scales. This
strategy allows one to simulate data at higher resolution than the input maps.

PySM 2 (Thorne et al., 2017), released in 2016, has become the de-facto
standard for simulating Galactic emission, for example it is used by CMB-S4,
Simons Observatory, LiteBird, PICO, CLASS, POLARBEAR and other CMB
experiments, as shown by the
https://scholar.google.com/scholar?start=0&hl=en&as_sdt=2005&sciodt=0,5&cites=16628417670342266167&scipsc=https://doi.org80+
citations of the PySM 2 publication. As the resolution of upcoming experiments
increases, the PySM 2 software has started to show some limitations:

  * •

Emission templates are provided at 7.9 arcminutes resolution (HEALPix
$N_{side}=512$), while the next generation of CMB experiments will require
sub-arcminute resolution.

  * •

The software is implemented in pure numpy, meaning that it has significant
memory overhead and is not multi-threaded, precluding simply replacing the
current templates with higher-resolution versions

  * •

Emission templates are included in the PySM 2 Python package, this is still
practical when each of the roughly 40 input maps is ~10 Megabytes, but will
not be if they are over 1 Gigabyte.

The solution to these issues was to reimplement PySM from scratch focusing on
these features:

  * •

Reimplement all the models with the numba (Lam et al., 2015) Just-In-Time
compiler for Python to reduce memory overhead and optimize performance: the
whole integration loop of a template map over the frequency response of an
instrument is performed in a single pass in automatically compiled and multi-
threaded Python code.

  * •

Use MPI through mpi4py to coordinate execution of PySM 3 across multiple
nodes, this allows to support template maps at a resolution up to 0.4
arcminutes (HEALPix $N_{side}=8192$).

  * •

Rely on libsharp (Reinecke & Seljebotn, 2013), a distributed implementation of
spherical harmonic transforms, to smooth the maps with the instrument beam
when maps are distributed over multiple nodes with MPI.

  * •

Employ the data utilities infrastructure provided by astropy (Astropy
Collaboration et al., 2018, 2013) to download the input templates and cache
them when requested.

At this stage we strive to maintain full compatibility with PySM 2, therefore
we implement the exact same astrophysical emission models with the same naming
scheme. In the extensive test suite we compare the output of each PySM 3 model
with the results obtained by PySM 2.

## Performance

As an example of the performance improvements achieved with PySM 3 over PySM
2, we run the following configuration:

  * •

An instrument with 3 channels, with different beams, and a top-hat bandpass
defined numerically at 10 frequency samples.

  * •

A sky model with the simplest models of dust, synchrotron, free-free and AME
[a1,d1,s1,f1 in PySM terms].

  * •

Execute on a 12-core Intel processor with 12 GB of RAM.

The following tables shows the walltime and peak memory usage of this
simulation executed at the native PySM 2 resolution of $N_{side}=512$ and at
two higher resolutions:

Output $N_{side}$ | PySM 3 | PySM 2
---|---|---
512 | 1m 0.7 GB | 1m40s 1.45 GB
1024 | 3m30s 2.3 GB | 7m20s 5.5 GB
2048 | 16m10s 8.5 GB | Out of memory

At the moment it is not very useful to run at resolutions higher than
$N_{side}=512$ because there is no actual template signal at smaller scales.
However, it demonstrates the performance improvements that will make working
with higher resolution templates possible.

## Future work

PySM 3 opens the way to implement a new category of models at much higher
resolution. However, instead of just upgrading the current models to smaller
scales we want to also update them with the latest knowledge of Galactic
emission and gather feedback from each of the numerous CMB experiments. For
this reason we are collaborating with the Panexperiment Galactic Science group
to lead the development of the new class of models to be included in PySM 3.

## How to cite

If you are using PySM 3 for your work, please cite this paper for the software
itself; for the actual emission modeling please also cite the original PySM 2
paper (Thorne et al., 2017). There will be a future paper on the generation of
new PySM 3 astrophysical models.

## Acknowledgments

  * •

This work was supported in part by NASA grant 80NSSC18K1487.

  * •

The software was tested, in part, on facilities run by the Scientific
Computing Core of the Flatiron Institute.

  * •

This research used resources of the National Energy Research Scientific
Computing Center (NERSC), a U.S. Department of Energy Office of Science User
Facility located at Lawrence Berkeley National Laboratory, operated under
Contract No. DE-AC02-05CH11231.

## References

Astropy Collaboration, Price-Whelan, A. M., Sipőcz, B. M., Günther, H. M.,
Lim, P. L., Crawford, S. M., Conseil, S., Shupe, D. L., Craig, M. W.,
Dencheva, N., Ginsburg, A., Vand erPlas, J. T., Bradley, L. D., Pérez-Suárez,
D., de Val-Borro, M., Aldcroft, T. L., Cruz, K. L., Robitaille, T. P.,
Tollerud, E. J., … Astropy Contributors. (2018). The Astropy Project: Building
an Open-science Project and Status of the v2.0 Core Package. _156_(3), 123.
https://doi.org/10.3847/1538-3881/aabc4f

Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., Greenfield, P.,
Droettboom, M., Bray, E., Aldcroft, T., Davis, M., Ginsburg, A., Price-Whelan,
A. M., Kerzendorf, W. E., Conley, A., Crighton, N., Barbary, K., Muna, D.,
Ferguson, H., Grollier, F., Parikh, M. M., Nair, P. H., … Streicher, O.
(2013). Astropy: A community Python package for astronomy. _558_ , A33.
https://doi.org/10.1051/0004-6361/201322068

Bennett, C. L., Larson, D., Weiland, J. L., Jarosik, N., Hinshaw, G., Odegard,
N., Smith, K. M., Hill, R. S., Gold, B., Halpern, M., Komatsu, E., Nolta, M.
R., Page, L., Spergel, D. N., Wollack, E., Dunkley, J., Kogut, A., Limon, M.,
Meyer, S. S., … Wright, E. L. (2013). Nine-year Wilkinson Microwave Anisotropy
Probe (WMAP) Observations: Final Maps and Results. _208_ , 20.
https://doi.org/10.1088/0067-0049/208/2/20

Górski, K. M., Hivon, E., Banday, A. J., Wandelt, B. D., Hansen, F. K.,
Reinecke, M., & Bartelmann, M. (2005). HEALPix: A Framework for High-
Resolution Discretization and Fast Analysis of Data Distributed on the Sphere.
_622_ , 759–771. https://doi.org/10.1086/427976

Lam, S. K., Pitrou, A., & Seibert, S. (2015). Numba: A LLVM-based python JIT
compiler. _Proceedings of the Second Workshop on the LLVM Compiler
Infrastructure in HPC_. https://doi.org/10.1145/2833157.2833162

Planck Collaboration. (2018). _Planck 2018 results. I. Overview and the
cosmological legacy of Planck_. http://arxiv.org/abs/1807.06205

Reinecke, M., & Seljebotn, D. S. (2013). Libsharp – spherical harmonic
transforms revisited. _Astronomy & Astrophysics_, _554_ , A112.
https://doi.org/10.1051/0004-6361/201321494

Thorne, B., Dunkley, J., Alonso, D., & Næss, S. (2017). The python sky model:
Software for simulating the galactic microwave sky. _Monthly Notices of the
Royal Astronomical Society_ , _469_(3), 2821–2833.
https://doi.org/10.1093/mnras/stx949

Zonca, A., Singer, L. P., Lenz, D., Reinecke, M., Rosset, C., Hivon, E., &
Gorski, K. M. (2019). Healpy: Equal area pixelization and spherical harmonics
transforms for data on the sphere in python. _Journal of Open Source Software_
, _4_(35), 1298. https://doi.org/10.21105/joss.01298