personalized model for each user. For fairness, we simply leverage the
specific user embedding for the personalized model, and for the non-
personalized model we simply derive an aggregate global embedding of a typical
user, and leverage this global non-personalized model to rank the
visualizations. More formally, the non-personalized ML-based approach uses a
global user embedding derived as,

(47)
$\displaystyle\boldsymbol{\mathrm{u}}_{g}=\frac{1}{n}\sum_{i=1}^{n}\boldsymbol{\mathrm{U}}_{i}$

where $\boldsymbol{\mathrm{u}}_{g}$ is called the global user embedding and
represents the centroid of the user embeddings from PVisRec. Everything else
remains the same as the personalized visualization recommendation approach.
More formally, given a user $i$ along with an arbitrary visualization
$\mathcal{V}=(\boldsymbol{\mathrm{X}}^{(k)},C_{t})$ generated from some
dataset $\boldsymbol{\mathrm{X}}$, we derive a score for the visualization
$\mathcal{V}$ using the global user embedding $\boldsymbol{\mathrm{u}}_{g}$
from Eq. 47 as follows:

(48)
$\displaystyle\phi_{g}(V)=\boldsymbol{\mathrm{u}}_{g}\boldsymbol{\mathrm{Z}}_{t,:}^{\top}\prod_{\boldsymbol{\mathrm{x}}_{j}\in\boldsymbol{\mathrm{X}}^{(k)}}\boldsymbol{\mathrm{u}}_{g}\boldsymbol{\mathrm{V}}_{j,:}^{\top}$

where $\boldsymbol{\mathrm{X}}^{(k)}$ is a subset of attributes used in the
visualization $\mathcal{V}$ from the dataset $\boldsymbol{\mathrm{X}}$ (hence,
$|\boldsymbol{\mathrm{X}}^{(k)}|\leq|\boldsymbol{\mathrm{X}}|$) and
$\mathcal{C}_{t}\in\mathbcal{C}$ is the visual-configuration of $\mathcal{V}$.
Hence, instead of leveraging user $i$’s personalized visualization
recommendation model to obtain a user personalized score for visualization
$\mathcal{V}$ (that is
$\phi(V)=\boldsymbol{\mathrm{U}}_{i,:}\boldsymbol{\mathrm{Z}}_{t,:}^{\top}\prod_{\boldsymbol{\mathrm{x}}_{j}\in\boldsymbol{\mathrm{X}}^{(k)}}\boldsymbol{\mathrm{U}}_{i,:}\boldsymbol{\mathrm{V}}_{j,:}^{\top}$),
we replace $\boldsymbol{\mathrm{U}}_{i,:}$ with the global user embedding
$\boldsymbol{\mathrm{u}}_{g}$ representing a “typical” user. Results are
provided in Table 7. For both models in Table 7, we use the same experimental
setup from Section 6.1. This PVisRec model is used for learning
$\boldsymbol{\mathrm{U}}$, then Eq. 47 is used for the non-personalized model.
Notably, the non-personalized approach that uses the same global user model
for all users performs significantly worse (as shown in Table 7) compared to
the user-level personalized approach that leverages the appropriate learned
model to personalize the ranking of visualizations with respect to the user at
hand. This demonstrates the significance of learning individual models for
each user that are personalized based on the users attribute/data preferences
along with their visual design choice preferences (RQ2).

Table 8. Space vs. Accuracy Trade-off Results using Meta-Feature Embeddings (MFE). Results for the space-efficient variants of our personalized visualization recommendation methods that use meta-feature embeddings. In particular, we set $d=10$ and vary the dimensions of the meta-feature embeddings from $\\{1,2,4,8,16\\}$. See text for discussion.  |  | HR@K | NDCG@K
---|---|---|---
Model | MFE dim. | @1 | @2 | @3 | @4 | @5 | @1 | @2 | @3 | @4 | @5
PVisRec ($\boldsymbol{\mathrm{A}}$,$\boldsymbol{\mathrm{C}}$,$\boldsymbol{\mathrm{M}}$ only) | 1 | 0.284 | 0.413 | 0.480 | 0.512 | 0.529 | 0.282 | 0.364 | 0.398 | 0.412 | 0.418
| 2 | 0.245 | 0.348 | 0.395 | 0.417 | 0.429 | 0.244 | 0.308 | 0.333 | 0.342 | 0.346
| 4 | 0.265 | 0.388 | 0.444 | 0.468 | 0.481 | 0.263 | 0.341 | 0.369 | 0.380 | 0.385
| 8 | 0.304 | 0.419 | 0.462 | 0.492 | 0.506 | 0.302 | 0.376 | 0.397 | 0.410 | 0.416
| 16 | 0.294 | 0.404 | 0.452 | 0.471 | 0.483 | 0.292 | 0.362 | 0.386 | 0.395 | 0.399
PVisRec | 1 | 0.467 | 0.589 | 0.641 | 0.667 | 0.681 | 0.464 | 0.542 | 0.569 | 0.580 | 0.585
| 2 | 0.542 | 0.685 | 0.744 | 0.771 | 0.792 | 0.539 | 0.630 | 0.660 | 0.672 | 0.680
| 4 | 0.544 | 0.713 | 0.779 | 0.815 | 0.829 | 0.541 | 0.649 | 0.682 | 0.698 | 0.704
| 8 | 0.608 | 0.806 | 0.874 | 0.906 | 0.925 | 0.604 | 0.731 | 0.765 | 0.779 | 0.787
| 16 | 0.616 | 0.794 | 0.865 | 0.896 | 0.916 | 0.613 | 0.726 | 0.762 | 0.776 | 0.784

###  6.3. Improving Space-Efficiency via Meta-Feature Embeddings

In this section, we investigate using a low-rank meta-feature embedding matrix
to significantly improve the space-efficiency of our proposed approach. In
particular, we replace the original meta-feature matrix
$\boldsymbol{\mathrm{M}}$ with a low-rank approximation that captures the most
important and meaningful meta-feature signals in the data. In addition to
significantly reducing the space requirements of PVisRec, we also investigate
the performance when the low-rank meta-feature embeddings are used, and the
space and accuracy trade-off as the number of meta-feature embedding
dimensions varies from $\\{1,2,4,8,16\\}$. We set $d=10$ and vary the
dimensions of the dimensionality of the meta-feature embeddings (MFE) from
$\\{1,2,4,8,16\\}$ across the different proposed approaches. We provide the
results in Table 8 for the space-efficient variants of our personalized
visualization recommendation methods that use meta-feature embeddings.
Overall, we find that in nearly all cases, we find similar HR@K and NDCG@K
compared to the original variants, while obtaining a significantly more
compact model with orders of magnitude less space. For instance, when MFE dim.
is 16, PvisRec has a HIT@1 of 0.616 compared to 0.630 using the original
1006-dimensional meta-feature matrix, which uses roughly 63x more space
compared to the 16-dimensional MFE variant. As an aside, since PVisRec
($\boldsymbol{\mathrm{A}}$, $\boldsymbol{\mathrm{C}}$,
$\boldsymbol{\mathrm{D}}$) does not use $\boldsymbol{\mathrm{M}}$, it does not
have a meta-feature embedding (MFE) variant. This implies that we can indeed
significantly reduce the space requirements of our approaches by trading off
only a tiny amount of accuracy (RQ3).

###  6.4. Neural Personalized Visualization Recommendation

In this section, we study the performance of the proposed Neural Personalized
Visualization Recommendation models (RQ4). For these experiments, we use
$d=10$ and $\alpha=0.5$ for Neural PVisRec-CMF. All models use three layers
and ReLU for the activation function. See Section 4 for further details. The
results are provided in Table 9. Both neural personalized visualization
recommendation models outperform the simpler and faster graph-based PVisRec
approach (across both rank-based evaluation metrics and across all top-$K$
personalized visualization recommendations). This is expected since the neural
visualization models all leverage the graph-based PVisRec model in some
fashion. Neural PVisRec uses the learned low-dimensional embeddings of the
users, visual-configurations, attributes, and meta-features of the attributes
as input into the first layer whereas Neural PVisRec-CMF also uses the learned
low-dimensional embeddings, but also uses the predicted visualization scores
from the PVisRec model for each user and combines these with the predicted
scores from the neural component. Notably, both neural personalized
visualization recommendation models outperform the simpler and faster graph-
based approach. In Table 9, Neural PVisRec-CMF outperforms the simpler Neural
PVisRec network. This holds for HR@$K$ and NDCG@$K$, and across all top-$K$
personalized visualization recommendations where $K\in\\{1,...,5\\}$.

Table 9. Results for the _Neural_ Personalized Visualization Recommendation Models. | HR@K | NDCG@K
---|---|---
Model | @1 | @2 | @3 | @4 | @5 | @1 | @2 | @3 | @4 | @5
Neural PVisRec | 0.656 | 0.825 | 0.889 | 0.923 | 0.946 | 0.652 | 0.761 | 0.793 | 0.808 | 0.817
Neural PVisRec-CMF | 0.762 | 0.879 | 0.922 | 0.944 | 0.961 | 0.729 | 0.822 | 0.845 | 0.855 | 0.861

####  6.4.1. Nonlinear activation function.

Neural PVisRec is flexible and can leverage any nonlinear activation functions
for the fully-connected layers of our multilayer neural network architecture
for personalized visualization recommendation. In Table 10, we compare three
non-linear activation functions $\sigma$ for learning a personalized
visualization recommendation model including hyperbolic tangent (tanh)
$\sigma(\boldsymbol{\mathrm{x}})=\tanh(\boldsymbol{\mathrm{x}})$, sigmoid
$\sigma(\boldsymbol{\mathrm{x}})=1/(1+\exp[-\boldsymbol{\mathrm{x}}])$, and
ReLU $\sigma(\boldsymbol{\mathrm{x}})=\max(0,\boldsymbol{\mathrm{x}})$. The
results in Table 10 show that ReLU performs best by a large margin followed by
sigmoid and then tanh. ReLU likely performs well due to its ability to avoid
saturation, handle sparse data and be less likely to overfit.

Table 10. Ablation study results of Neural PVisRec with different nonlinear activation functions. We report HR@1 for brevity. All results use $d=10$ and $\alpha=0.5$.  |  nonlinear activation $\sigma$ |
---|---|---
Model | tanh | sigmoid | ReLU |
Neural PVisRec | 0.615 | 0.624 | 0.656 |
Neural PVisRec-CMF | 0.613 | 0.640 | 0.762 |
Table 11. Comparing performance of Neural PVisRec with different number of hidden layers.  | HR@K | NDCG@K
---|---|---
# Hidden Layers | @1 | @2 | @3 | @4 | @5 | @1 | @2 | @3 | @4 | @5
1 | 0.579 | 0.773 | 0.844 | 0.880 | 0.896 | 0.578 | 0.701 | 0.737 | 0.752 | 0.758
2 | 0.618 | 0.801 | 0.865 | 0.892 | 0.907 | 0.618 | 0.733 | 0.765 | 0.777 | 0.783
3 | 0.656 | 0.825 | 0.889 | 0.923 | 0.946 | 0.652 | 0.761 | 0.793 | 0.808 | 0.817
4 | 0.646 | 0.754 | 0.813 | 0.842 | 0.869 | 0.499 | 0.639 | 0.680 | 0.694 | 0.705

####  6.4.2. Hidden layers.

To understand the impact of the number of layers on the performance of the
neural personalized visualization recommendation models, we vary the number of
hidden layers from $L\in\\{1,2,3,4\\}$. In Table 11, the performance increases
as additional hidden layers are included, and begins to decrease at $L=4$. The
best performance is achieved with three hidden layers. This result indicates
the benefit of deep learning for personalized visualization recommendation.

####  6.4.3. Layer size.

Recall that our network structure followed a tower pattern where the layer
size of each successive layer is halved. In this experiment, we investigate
larger layer sizes while fixing the final output embedding size to be 8 and
using 4 hidden layers. In Table 12, we observe a significant improvement in
the visualization ranking when using larger layer sizes.

Table 12. Varying layer sizes in the deep personalized visualization recommendation model (Neural PVisRec). | HR@K
---|---
Layer Sizes | @1 | @2 | @3 | @4 | @5
8-16-32-64 | 0.701 | 0.790 | 0.832 | 0.865 | 0.883
8-32-128-512 | 0.734 | 0.797 | 0.846 | 0.874 | 0.886
8-48-288-1728 | 0.752 | 0.822 | 0.869 | 0.895 | 0.913

####  6.4.4. Runtime performance.

Neural PVisRec is also fast, taking on average 10.85 seconds to train using
the large personalized visualization corpus from Section 5. The other neural
visualization recommender is nearly as fast, as it contains only an additional
step that is linear in the output embedding size. For these experiments, we
used a 2017 MacBook Pro with 16GB memory and 3.1GHz Intel Core i7 processor.

##  7\. Related Work

###  7.1. Visualization Recommendation

Rule-based visualization recommendation systems such as Voyager (Vartak et
al., 2017; Wongsuphasawat et al., 2017, 2015), VizDeck (Perry et al., 2013),
and DIVE (Hu et al., 2018) use a large set of rules defined manually by domain
experts to recommend appropriate visualizations that satisfy the rules (Lee,
2020; Mackinlay, 1986; Roth et al., 1994; Casner, 1991; Mackinlay et al.,
2007a; Derthick et al., 1997; Stolte et al., 2002; Feiner, 1985; Seo and
Shneiderman, 2005). Such rule-based systems do not leverage any training data
for learning or user personalization. There have been a few “hybrid”
approaches that combine some form of learning with manually defined rules for
visualization recommendation (Moritz et al., 2018), _e.g._ , Draco learns
weights for rules (constraints) (Moritz et al., 2018). Recently, there has
been work that focused on the end-to-end ML-based visualization recommendation
problem (Qian et al., 2020; Dibia and Demiralp, 2019). However, this work
learns a global visualization recommendation model that is agnostic of the
user, and thus not able to be used for the personalized visualization
recommendation problem studied in our work.

All of the existing rule-based (Vartak et al., 2017; Wongsuphasawat et al.,
2017, 2015; Perry et al., 2013; Hu et al., 2018), hybrid (Moritz et al.,
2018), and pure ML-based visualization recommendation (Qian et al., 2020)
approaches are unable to recommend personalized visualizations for specific
users. These approaches do not model users, but focus entirely on learning or
manually defining visualization rules that capture the notion of an effective
visualization (Mackinlay et al., 2007b; Wills and Wilkinson, 2010; Key et al.,
2012; Elzen and Wijk, 2013; Wilkinson and Wills, 2008; Dang and Wilkinson,
2014; Vartak et al., 2015; Demiralp et al., 2017; Cui et al., 2019; Lin et
al., 2020; Lee et al., 2019a; Siddiqui et al., 2016). Therefore, no matter the
user, the model always gives the same recommendations. The closest existing
work is VizRec (Mutlu et al., 2016). However, VizRec is only applicable when
there is a single dataset shared by all users (and therefore a single small
set of visualizations that the users have explicitly liked and tagged). This
problem is unrealistic with many impractical assumptions that are not aligned
with practice. Nevertheless, the problem solved by that prior work is a simple
special case of the personalized visualization recommendation problem
introduced in our paper.

###  7.2. Simpler Design and Data Tasks

Besides visualization recommendation, there are methods that solve simpler
sub-tasks such as improving expressiveness, improving perceptual
effectiveness, matching task types, etc. These simpler sub-tasks can generally
be divided two categories (Lee, 2020; Wongsuphasawat et al., 2016): whether
the solution focuses on recommending data (what data to visualize), such as
Discovery-driven Data Cubes (Sarawagi et al., 1998), Scagnostics (Wilkinson et
al., 2005), AutoVis (Wills and Wilkinson, 2010), and MuVE (Ehsan et al.,
2016)) or recommending encoding (how to design and visually encode the data),
such as APT (Mackinlay, 1986), ShowMe (Mackinlay et al., 2007a), and
Draco–learn (Moritz et al., 2018)). While some of those are ML-based, none are
able to recommend entire visualizations (nor are they personalized), which is
the focus of this work. For example, VizML (Hu et al., 2019) predicts the type
of a chart (e.g., bar, scatter, etc.) instead of complete visualization. Draco
(Moritz et al., 2018) infers weights for a set of manually defined rules.
VisPilot (Lee et al., 2019b) recommended different drill-down data subsets
from datasets. As an aside, not only does these works not solve the
visualization recommendation problem, they are also not personalized for
individual users. Instead of solving simple sub-tasks such as predicting the
chart type of a visualization, we focus on the _end-to-end personalized
visualization recommendation problem (Sec. 2)_: given a dataset of interest to
user $i$, the goal is to automatically recommend the top-k most effective
visualizations personalized for that individual user. This paper fills the gap
by proposing the first _personalized_ visualization recommendation approach
that is completely automatic, data-driven, and most importantly recommends
personalized visualizations based on a users previous feedback, behavior, and
interactions with the system.

###  7.3. Traditional Recommender Systems

In traditional item-based recommender systems (Adomavicius and Tuzhilin, 2005;
Ricci et al., 2011; Zhao et al., 2020; Noel et al., 2012; Zhang et al., 2017),
there is a _single shared set of items_ (_i.e._ , movies (Bennett et al.,
2007; Covington et al., 2016; Harper and Konstan, 2015), products (Linden et
al., 2003), hashtags (Sigurbjörnsson and Van Zwol, 2008; Wang et al., 2020),
documents (Xu et al., 2020; Kanakia et al., 2019), news (Ge et al., 2020),
books (Liu et al., 2014), and location (Ye et al., 2011; Zhou et al., 2019;
Bennett et al., 2011)). However, in the personalized visualization
recommendation problem studied in this work, since visualizations are dataset
dependent, there is not a shared set of visualizations to recommend users.
Therefore, given $N$ datasets, there are $N$ completely disjoint sets of
visualizations that can be recommended. Every dataset consists of its own
completely separate set of relevant visualizations that are exclusive to the
dataset. Therefore, in contrast to the goal of traditional item recommender
systems, the goal of personalized visualization recommendation is to learn a
personalized vis. rec. model for each individual user, which is capable of
scoring and ultimately recommending personalized visualizations to that user
from any unseen dataset in the future. Some recent works have adapted various
deep learning approaches for collaborative filtering (Sedhain et al., 2015; Li
et al., 2020; He et al., 2017; Chen et al., 2020; Guan et al., 2019). However,
none of these works have focused on the problem of _personalized visualization
recommendation_ studied in this work. The personalized visualization
recommender problem has a few similarities with cross-domain recommendation
(Tang et al., 2012; Gao et al., 2013; Man et al., 2017; Shapira et al., 2013;
Hu et al., 2013). In cross-domain item recommendation, there is only a few
datasets as opposed to tens of thousands of different datasets in our problem
(Zhao et al., 2020). More importantly, in cross-domain item recommendation,
the different datasets are assumed to share at least one mode between each
other, whereas in personalized visualization recommendation, each new dataset
gives rise to a completely different set of visualizations to recommend.

##  8\. Conclusion

In this work, we introduced the problem of user-specific personalized
visualization recommendation and proposed an approach for solving it. The
approach learns individual personalized visualization recommendation models
for each user. In particular, the personalized vis. rec. models for each user
are learned by taking into account the user feedback including both implicit
and explicit feedback regarding the visual and data preferences of the users,
as well as users whom have also explored similar datasets and visualizations.
We overcome the issues with data sparsity and limited user feedback by
leveraging the data and visualization preferences of users whom are similar,
despite that the visualizations from such users are from completely different
datasets. The models are able to learn better visualization recommendation
models for each individual user by leveraging the data and visualization
preferences of users whom are similar. In addition, we proposed a deep neural
network architecture for _neural_ personalized visualization recommendation
that can learn complex non-linear relationships between the users, their
attributes of interest, and visualization preferences. This paper is a first
step in the direction of learning personalized visualization recommendation
models for individual users based on their data and visualization feedback,
and the data and visual preferences of users with similar data and visual
preferences. Future work should investigate and develop better machine
learning models and learning techniques to further improve the personalized
visualization recommendation models and the visualization recommendations for
individual users.

## References

  * (1)
  * Adomavicius and Tuzhilin (2005) Gediminas Adomavicius and Alexander Tuzhilin. 2005.  Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions.  TKDE 17, 6 (2005), 734–749.
  * Balasubramaniam et al. (2020) Thirunavukarasu Balasubramaniam, Richi Nayak, Chau Yuen, and Yu-Chu Tian. 2020.  Column-wise element selection for computationally efficient nonnegative coupled matrix tensor factorization.  IEEE Transactions on Knowledge and Data Engineering (2020).
  * Bennett et al. (2007) James Bennett, Stan Lanning, et al. 2007\.  The netflix prize. In KDD Cup. 35.
  * Bennett et al. (2011) Paul N Bennett, Filip Radlinski, Ryen W White, and Emine Yilmaz. 2011.  Inferring and using location metadata to personalize web search. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval. 135–144.
  * Bouchard et al. (2013) Guillaume Bouchard, Dawei Yin, and Shengbo Guo. 2013.  Convex collective matrix factorization. In AISTATS. PMLR, 144–152.
  * Casner (1991) Stephen M Casner. 1991\.  Task-Analytic Approach to the Automated Design of Graphic Presentations.  ACM Transactions on Graphics (ToG) 10, 2 (1991), 111–151.
  * Chen et al. (2020) Chong Chen, Min Zhang, Yongfeng Zhang, Yiqun Liu, and Shaoping Ma. 2020.  Efficient neural matrix factorization without sampling for recommendation.  ACM Transactions on Information Systems (TOIS) 38, 2 (2020), 1–28.
  * Choi et al. (2019) Dongjin Choi, Jun-Gi Jang, and U Kang. 2019.  S3 CMTF: Fast, accurate, and scalable method for incomplete coupled matrix-tensor factorization.  PloS one 14, 6 (2019), e0217316.
  * Covington et al. (2016) Paul Covington, Jay Adams, and Emre Sargin. 2016\.  Deep neural networks for youtube recommendations. In RecSys. 191–198.
  * Cui et al. (2019) Zhe Cui, Sriram Karthik Badam, M Adil Yalçin, and Niklas Elmqvist. 2019.  Datasite: Proactive Visual Data Exploration With Computation of Insight-Based Recommendations.  Information Visualization 18, 2 (2019), 251–267.
  * Dang and Wilkinson (2014) Tuan Nhon Dang and Leland Wilkinson. 2014.  ScagExplorer: Exploring Scatterplots by Their Scagnostics. In 2014 IEEE Pacific visualization symposium. IEEE, 73–80.
  * Demiralp et al. (2017) Çağatay Demiralp, Peter J Haas, Srinivasan Parthasarathy, and Tejaswini Pedapati. 2017.  Foresight: Recommending Visual Insights. In Proceedings of the VLDB Endowment International Conference on Very Large Data Bases, Vol. 10.
  * Derthick et al. (1997) Mark Derthick, John Kolojejchick, and Steven F Roth. 1997\.  An interactive visualization environment for data exploration. In KDD. 2–9.
  * Dibia and Demiralp (2019) Victor Dibia and Çağatay Demiralp. 2019\.  Data2vis: Automatic generation of data visualizations using sequence-to-sequence recurrent neural networks.  IEEE computer graphics and applications 39, 5 (2019), 33–46.
  * Ehsan et al. (2016) Humaira Ehsan, Mohamed Sharaf, and Panos Chrysanthis. 2016\.  Muve: Efficient multi-objective view recommendation for visual data exploration. In ICDE.
  * Elzen and Wijk (2013) Stef van den Elzen and Jarke J. van Wijk. 2013.  Small Multiples, Large Singles: A New Approach for Visual Data Exploration. In Computer Graphics Forum, Vol. 32. 191–200.
  * Feiner (1985) Steven Feiner. 1985\.  APEX: An Experiment in the Automated Creation of Pictorial Explanations.  IEEE Computer Graphics and Applications 5, 11 (1985), 29–37.
  * Gao et al. (2013) Sheng Gao, Hao Luo, Da Chen, Shantao Li, Patrick Gallinari, and Jun Guo. 2013\.  Cross-domain recommendation via cluster-level latent factor model. In Joint European conference on machine learning and knowledge discovery in databases. Springer, 161–176.
  * Ge et al. (2020) Suyu Ge, Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2020.  Graph Enhanced Representation Learning for News Recommendation. In WWW.
  * Guan et al. (2019) Xinyu Guan, Zhiyong Cheng, Xiangnan He, Yongfeng Zhang, Zhibo Zhu, Qinke Peng, and Tat-Seng Chua. 2019.  Attentive aspect modeling for review-aware recommendation.  ACM Transactions on Information Systems (TOIS) 37, 3 (2019), 1–27.
  * Harper and Konstan (2015) F Maxwell Harper and Joseph A Konstan. 2015.  The movielens datasets: History and context.  TIIS 5, 4 (2015), 1–19.
  * He et al. (2017) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017\.  Neural collaborative filtering. In WWW. 173–182.
  * He et al. (2016) Xiangnan He, Hanwang Zhang, Min-Yen Kan, and Tat-Seng Chua. 2016.  Fast matrix factorization for online recommendation with implicit feedback. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 549–558.
  * Hu et al. (2019) Kevin Hu, Michiel A. Bakker, Stephen Li, Tim Kraska, and César Hidalgo. 2019.  VizML: A Machine Learning Approach to Visualization Recommendation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12.  https://doi.org/10.1145/3290605.3300358
  * Hu et al. (2018) Kevin Hu, Diana Orghian, and César Hidalgo. 2018.  Dive: A mixed-initiative system supporting integrated data exploration workflows. In Workshop on Human-In-the-Loop Data Anal. 1–7.
  * Hu et al. (2013) Liang Hu, Jian Cao, Guandong Xu, Longbing Cao, Zhiping Gu, and Can Zhu. 2013\.  Personalized recommendation via cross-domain triadic factorization. In Proceedings of the 22nd International Conference on World Wide Web. 595–606.
  * Kanakia et al. (2019) Anshul Kanakia, Zhihong Shen, Darrin Eide, and Kuansan Wang. 2019\.  A scalable hybrid research paper recommender system for microsoft academic. In WWW.
  * Key et al. (2012) Alicia Key, Bill Howe, Daniel Perry, and Cecilia Aragon. 2012\.  VizDeck: Self-Organizing Dashboards for Visual Analytics. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data. 681–684.
  * Kim et al. (2014) Jingu Kim, Yunlong He, and Haesun Park. 2014.  Algorithms for nonnegative matrix and tensor factorizations: A unified view based on block coordinate descent framework.  Journal of Global Optimization 58, 2 (2014), 285–319.
  * Lee (2020) Doris Jung-Lin Lee. 2020\.  Insight Machines: The Past, Present, and Future of Visualization Recommendation.  (August 2020).
  * Lee et al. (2019a) Doris Jung-Lin Lee, Himel Dev, Huizi Hu, Hazem Elmeleegy, and Aditya Parameswaran. 2019a.  Avoiding Drill-Down Fallacies With VisPilot: Assisted Exploration of Data Subsets. In Proceedings of the 24th International Conference on Intelligent User Interfaces. 186–196.
  * Lee et al. (2019b) Doris Jung-Lin Lee, Himel Dev, Huizi Hu, Hazem Elmeleegy, and Aditya Parameswaran. 2019b.  Avoiding drill-down fallacies with VisPilot: assisted exploration of data subsets. In IUI. 186–196.
  * Li et al. (2020) Xiangsheng Li, Maarten de Rijke, Yiqun Liu, Jiaxin Mao, Weizhi Ma, Min Zhang, and Shaoping Ma. 2020.  Learning Better Representations for Neural Information Retrieval with Graph Information. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 795–804.
  * Lin et al. (2020) Halden Lin, Dominik Moritz, and Jeffrey Heer. 2020\.  Dziban: Balancing Agency & Automation in Visualization Design via Anchored Recommendations. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–12.
  * Linden et al. (2003) Greg Linden, Brent Smith, and Jeremy York. 2003.  Amazon.com recommendations: Item-to-item collaborative filtering.  Internet Computing 7, 1 (2003), 76–80.
  * Liu et al. (2014) Yidan Liu, Min Xie, and Laks VS Lakshmanan. 2014.  Recommending user generated item lists. In RecSys. 185–192.
  * Mackinlay (1986) Jock Mackinlay. 1986\.  Automating the design of graphical presentations of relational information.  ACM Trans. Graph. 5, 2 (1986), 110–141.
  * Mackinlay et al. (2007a) Jock Mackinlay, Pat Hanrahan, and Chris Stolte. 2007a.  Show Me: Automatic presentation for visual analysis.  TVCG 13, 6 (2007), 1137–1144.
  * Mackinlay et al. (2007b) Jock Mackinlay, Pat Hanrahan, and Chris Stolte. 2007b.  Show Me: Automatic Presentation for Visual Analysis.  IEEE transactions on visualization and computer graphics 13, 6 (2007), 1137–1144.
  * Man et al. (2017) Tong Man, Huawei Shen, Xiaolong Jin, and Xueqi Cheng. 2017\.  Cross-Domain Recommendation: An Embedding and Mapping Approach.. In IJCAI. 2464–2470.
  * Moritz et al. (2018) Dominik Moritz, Chenglong Wang, Greg L Nelson, Halden Lin, Adam M Smith, Bill Howe, and Jeffrey Heer. 2018.  Formalizing visualization design knowledge as constraints: Actionable and extensible models in draco.  IEEE transactions on visualization and computer graphics 25, 1 (2018), 438–448.
  * Mutlu et al. (2016) Belgin Mutlu, Eduardo Veas, and Christoph Trattner. 2016\.  Vizrec: Recommending personalized visualizations.  ACM Transactions on Interactive Intelligent Systems (TIIS) 6, 4 (2016), 1–39.
  * Noel et al. (2012) Joseph Noel, Scott Sanner, Khoi-Nguyen Tran, Peter Christen, Lexing Xie, Edwin V Bonilla, Ehsan Abbasnejad, and Nicolás Della Penna. 2012.  New objective functions for social collaborative filtering. In Proceedings of the 21st International Conference on World Wide Web. 859–868.
  * Oh et al. (2015) Jinoh Oh, Wook-Shin Han, Hwanjo Yu, and Xiaoqian Jiang. 2015\.  Fast and robust parallel SGD matrix factorization. In SIGKDD. ACM, 865–874.
  * Perry et al. (2013) Daniel B Perry, Bill Howe, Alicia MF Key, and Cecilia Aragon. 2013.  VizDeck: Streamlining exploratory visual analytics of scientific data.  (2013).
  * Qian et al. (2020) Xin Qian, Ryan A. Rossi, Fan Du, Sungchul Kim, Eunyee Koh, Sana Malik, Tak Yeon Lee, and Joel Chan. 2020\.  ML-based Visualization Recommendation: Learning to Recommend Visualizations from Data.  arXiv:2009.12316 (2020).
  * Ricci et al. (2011) Francesco Ricci, Lior Rokach, and Bracha Shapira. 2011\.  Introduction to recommender systems handbook.  In Rec. Sys. handbook. 1–35.
  * Rossi and Zhou (2016) Ryan A. Rossi and Rong Zhou. 2016.  Parallel Collective Factorization for Modeling Large Heterogeneous Networks. In Social Network Analysis and Mining. 30.
  * Roth et al. (1994) Steven F Roth, John Kolojejchick, Joe Mattis, and Jade Goldstein. 1994.  Interactive graphic design using automatic presentation knowledge. In CHI. 112–117.
  * Sarawagi et al. (1998) Sunita Sarawagi, Rakesh Agrawal, and Nimrod Megiddo. 1998\.  Discovery-driven exploration of OLAP data cubes. In Extending Database Tech. 168–182.
  * Satyanarayan et al. (2016) Arvind Satyanarayan, Dominik Moritz, Kanit Wongsuphasawat, and Jeffrey Heer. 2016.  Vega-lite: A grammar of interactive graphics.  IEEE Transactions on Visualization and Computer Graphics 23, 1 (2016), 341–350.
  * Schenker et al. (2021) Carla Schenker, Jeremy E Cohen, and Evrim Acar. 2021.  An optimization framework for regularized linearly coupled matrix-tensor factorization. In 28th European Signal Processing Conference (EUSIPCO). 985–989.
  * Sedhain et al. (2015) Suvash Sedhain, Aditya Krishna Menon, Scott Sanner, and Lexing Xie. 2015.  Autorec: Autoencoders meet collaborative filtering. In Proceedings of the 24th International Conference on World Wide Web. 111–112.
  * Seo and Shneiderman (2005) Jinwook Seo and Ben Shneiderman. 2005.  A Rank-by-Feature Framework for Interactive Exploration of Multidimensional Data.  Information visualization 4, 2 (2005), 96–113.
  * Shapira et al. (2013) Bracha Shapira, Lior Rokach, and Shirley Freilikhman. 2013\.  Facebook single and cross domain data for recommendation systems.  User Modeling and User-Adapted Interaction 23, 2-3 (2013), 211–247.
  * Siddiqui et al. (2016) Tarique Siddiqui, Albert Kim, John Lee, Karrie Karahalios, and Aditya Parameswaran. 2016.  Effortless Data Exploration With zenvisage: An Expressive and Interactive Visual Analytics System.  arXiv preprint arXiv:1604.03583 (2016).
  * Sigurbjörnsson and Van Zwol (2008) Börkur Sigurbjörnsson and Roelof Van Zwol. 2008.  Flickr tag recommendation based on collective knowledge. In WWW. 327–336.
  * Singh and Gordon (2008) Ajit P Singh and Geoffrey J Gordon. 2008.  Relational learning via collective matrix factorization. In KDD. 650–658.
  * Stolte et al. (2002) Chris Stolte, Diane Tang, and Pat Hanrahan. 2002.  Polaris: A system for query, analysis, and visualization of multidimensional relational databases.  TVCG 8, 1 (2002), 52–65.
  * Tang et al. (2012) Jie Tang, Sen Wu, Jimeng Sun, and Hang Su. 2012\.  Cross-domain collaboration recommendation. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining. 1285–1293.
  * Vartak et al. (2017) Manasi Vartak, Silu Huang, Tarique Siddiqui, Samuel Madden, and Aditya Parameswaran. 2017\.  Towards visualization recommendation systems.  ACM SIGMOD 45, 4 (2017), 34–39.
  * Vartak et al. (2015) Manasi Vartak, Sajjadur Rahman, Samuel Madden, Aditya Parameswaran, and Neoklis Polyzotis. 2015\.  Seedb: Efficient data-driven visualization recommendations to support visual analytics. In Proceedings of the VLDB Endowment International Conference on Very Large Data Bases, Vol. 8. NIH Public Access, 2182.
  * Wang et al. (2020) Xueting Wang, Yiwei Zhang, and Toshihiko Yamasaki. 2020\.  Earn More Social Attention: User Popularity Based Tag Recommendation System. In WWW.
  * Wilkinson et al. (2005) Leland Wilkinson, Anushka Anand, and Robert Grossman. 2005\.  Graph-theoretic scagnostics. In IEEE Symposium on Information Visualization. 157–164.
  * Wilkinson and Wills (2008) Leland Wilkinson and Graham Wills. 2008.  Scagnostics Distributions.  Journal of Computational and Graphical Statistics 17, 2 (2008), 473–491.
  * Wills and Wilkinson (2010) Graham Wills and Leland Wilkinson. 2010.  Autovis: automatic visualization.  Information Visualization 9, 1 (2010), 47–69.
  * Wongsuphasawat et al. (2015) Kanit Wongsuphasawat, Dominik Moritz, Anushka Anand, Jock Mackinlay, Bill Howe, and Jeffrey Heer. 2015.  Voyager: Exploratory analysis via faceted browsing of visualization recommendations.  IEEE transactions on visualization and computer graphics 22, 1 (2015), 649–658.
  * Wongsuphasawat et al. (2016) Kanit Wongsuphasawat, Dominik Moritz, Anushka Anand, Jock Mackinlay, Bill Howe, and Jeffrey Heer. 2016.  Towards a general-purpose query language for visualization recommendation. In Workshop on Human-In-the-Loop Data Anal.
  * Wongsuphasawat et al. (2017) Kanit Wongsuphasawat, Zening Qu, Dominik Moritz, Riley Chang, Felix Ouk, Anushka Anand, Jock Mackinlay, Bill Howe, and Jeffrey Heer. 2017\.  Voyager 2: Augmenting visual analysis with partial view specifications. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 2648–2659.
  * Xu et al. (2020) Xuhai Xu, Ahmed Hassan Awadallah, Susan T. Dumais, Farheen Omar, Bogdan Popp, Robert Rounthwaite, and Farnaz Jahanbakhsh. 2020.  Understanding User Behavior For Document Recommendation. In WWW. 3012–3018.
  * Ye et al. (2011) Mao Ye, Peifeng Yin, Wang-Chien Lee, and Dik-Lun Lee. 2011\.  Exploiting Geo. Influence for Collaborative Point-of-Interest Recommendation. In SIGIR.
  * Yun et al. (2014) Hyokun Yun, Hsiang-Fu Yu, Cho-Jui Hsieh, SVN Vishwanathan, and Inderjit Dhillon. 2014.  NOMAD: Non-locking, stOchastic Multi-machine algorithm for Asynchronous and Decentralized matrix completion.  VLDB 7, 11 (2014), 975–986.
  * Zhang et al. (2017) Yongfeng Zhang, Qingyao Ai, Xu Chen, and W Bruce Croft. 2017\.  Joint representation learning for top-n recommendation with heterogeneous information sources. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 1449–1458.
  * Zhao et al. (2020) Cheng Zhao, Chenliang Li, Rong Xiao, Hongbo Deng, and Aixin Sun. 2020.  CATN: Cross-domain recommendation for cold-start users via aspect transfer network. In SIGIR. 229–238.
  * Zhou et al. (2019) Fan Zhou, Ruiyang Yin, Kunpeng Zhang, Goce Trajcevski, Ting Zhong, and Jin Wu. 2019\.  Adversarial point-of-interest recommendation. In WWW. 3462–34618.


# A Conditional Denoising Diffusion Probabilistic Model
for Radio Interferometric Image Reconstruction

Ruoqi Wang  Zhuoyang Chen  Qiong Luo Qiong Luo is the corresponding author.
Her other affiliation is the Hong Kong University of Science and Technology.
Feng Wang  The Hong Kong University of Science and Technology (Guangzhou)
{rwang280<EMAIL_ADDRESS><EMAIL_ADDRESS>Guangzhou University
<EMAIL_ADDRESS>
###### Abstract

In radio astronomy, signals from radio telescopes are transformed into images
of observed celestial objects, or sources. However, these images, called dirty
images, contain real sources as well as artifacts due to signal sparsity and
other factors. Therefore, radio interferometric image reconstruction is
performed on dirty images, aiming to produce clean images in which artifacts
are reduced and real sources are recovered. So far, existing methods have
limited success on recovering faint sources, preserving detailed structures,
and eliminating artifacts. In this paper, we present VIC-DDPM, a Visibility
and Image Conditioned Denoising Diffusion Probabilistic Model. Our main idea
is to use both the original visibility data in the spectral domain and dirty
images in the spatial domain to guide the image generation process with DDPM.
This way, we can leverage DDPM to generate fine details and eliminate noise,
while utilizing visibility data to separate signals from noise and retaining
spatial information in dirty images. We have conducted experiments in
comparison with both traditional methods and recent deep learning based
approaches. Our results show that our method significantly improves the
resulting images by reducing artifacts, preserving fine details, and
recovering dim sources. This advancement further facilitates radio
astronomical data analysis tasks on celestial phenomena. Our code is available
at https://github.com/RapidsAtHKUST/VIC-DDPM.

##  1 Introduction

Radio interferometers, or radio telescopes, receive radio waves from
astronomical sources and produce interferometric measurement data, called
_visibility_. These visibility data are in the spectral domain, and can be
sampled and converted through inverse Fourier transforms into images for
subsequent analysis. However, due to data sparsity and other factors, the
result images are often dominated by artifacts [23]. As such, these _dirty
images_ must be reconstructed before they can be used in scientific analysis.
Figure 1 illustrates the workflow of imaging and reconstruction. In this
paper, we propose an image reconstruction method, aiming to effectively
recover real sources and eliminate artifacts in dirty images.

Reconstruction of interferometric images is a challenging problem. Its
effectiveness heavily depends on suitable priors to select from a vast array
of potentially valid images in an infinite solution space [33]. For example,
the classic CLEAN method [14] uses human knowledge as priors and assumes that
the sky is composed of point-like sources [7]. As real celestial objects can
be either point or extended sources, this assumption limits the quality of
reconstructed images [33]. Therefore, recent deep-learning based approaches
[18, 19, 33, 23] learn data-driven priors from training datasets. These
methods successfully predict general structures of main objects in observation
areas, but they still struggle in recovering faint sources, preserving
details, and eliminating artifacts.

Figure 1: Imaging and Reconstruction

To perform high-quality radio interferometry reconstruction, we propose VIC-
DDPM, a Visibility and Image Conditioned Denoising Diffusion Probabilistic
Model (DDPM). We adopt a DDPM [13, 26] because it can generate images with
fine details. We use both the original visibility data in the spectral domain
and dirty images in the spatial domain to guide the clean image generation
with DDPM. This way, our model can leverage the complementary strengths of
both domains: Visibility data offers frequency information where noise can be
effectively separated from signals [3], aiding in identifying artifacts; dirty
images provide spatial information, helping recover dim sources and object
structures. Consequently, our model removes artifacts as well as restores fine
details, including details in main structures and surrounding dim sources. To
the best of our knowledge, VIC-DDPM is the first work that adapts diffusion
models to radio interferometric image reconstruction.

In short, our main contributions are as follows:

  * •

We present a DDPM based generative model with two conditioning mechanisms, for
dirty image and visibility data, respectively, in radio interferometric image
reconstruction.

  * •

We show that data-driven priors in interferometric datasets can be effectively
learned by incorporating both visibility and image data.

We have conducted experiments to evaluate VIC-DDPM in comparison with both
popular traditional methods and a few recent state-of-the-art deep-learning
based models. Our results show that our method significantly improves the
output images. In particular, VIC-DDPM removes artifacts more thoroughly,
reconstructs structural details better, and recovers surrounding dim sources.
This advancement facilitates a more comprehensive and accurate analysis of
radio astronomical data, enabling researchers to better understand complex
celestial phenomena.

The remainder of this paper is organized as follows. In section 2, we describe
the background of radio interferometric imaging and image reconstruction,
related reconstruction algorithms, and basic denoising diffusion probabilistic
models. In section 3, we present the details of VIC-DDPM, including the two
conditioning factors – dirty image and visibility. In section 4, we describe
experimental settings and present the results with analysis.

##  2 Background and Related Work

###  2.1 Interferometric Imaging

In radio astronomy, observational data are gathered from radio telescopes, or
_interferometers_ , which detect radio signals from remote celestial objects.
These interferometers utilize aperture synthesis [4] to process their captured
signals, generating _visibility_ data through Fourier transforms of the sky’s
brightness distribution [16]. Then, the _imaging_ process constructs images of
the observed sky from the visibility data for subsequent analysis [31] .

Specifically, as the complex value of the visibility of an astronomical object
is equivalent to the Fourier transform of the object’s intensity distribution
function, we can apply an inverse Fourier transform to convert the $(u,v)$
coordinates in the Fourier domain into $(l,m)$ in the image domain [33]:

$\displaystyle I(l,m)$ $\displaystyle=$ $\displaystyle\int_{u}\int_{v}e^{2\pi
i(ul+vm)}V(u,v)dudv$ (1)

where $V(u,v)$ denotes the complex visibility in the Fourier space, and
$I(l,m)$ denotes the intensity distribution in the image.

###  2.2 Interferometric Image Reconstruction

Due to the limited number of radio telescopes and their non-uniform
distribution on the ground, visibility data is often sparsely and irregularly
sampled, leading to an incomplete representation of the true sky. When the
inverse Fourier transform is applied to this sparse data, the resulting image,
called a _dirty image_ , is dominated by artifacts [23]. For better
applicability in scientific analysis, dirty images must go through further
reconstruction to enhance the clarity of sources and reduce artifacts.

Interferometric image reconstruction is an ill-posed problem because there are
infinite solution images that fit the observed visibility [28]. Traditional
reconstruction methods [1, 14] utilize human prior knowledge to reduce the
solution space. In particular, the maximum entropy method (MEM) [1]
mathematically determines the solution that best agrees with the available
sparse measurement, whereas the CLEAN algorithm [14] iteratively operates on
the dirty image to distinguish real structures from sidelobe disturbances. For
the past decades, CLEAN [14] has served as the main-stream radio
interferometric reconstruction method; however, its resulting images have
limitations [33] due to its assumption of all point-like sources in the sky
[7].

Recently, a flurry of deep-learning based data processing methods have been
proposed in radio astronomy and computational imaging. Morningstar et al. [18,
19] used neural networks in radio interferometry and can generate reproducible
results [23]. Sun et al. presented a variational depth probability imaging
method for quantifying reconstruction uncertainty, which can efficiently
sample posteriori distributions [28]. Schmidt et al. [23] used radionets with
a convolutional neural network structure to generate reproducible clean radio
images on simulated data. Wu et al. [33] proposed to perform sparse-to-dense
inpainting in the spectral domain with coordinate-based neural fields,
outperforming other established interferometry techniques.

We experimented several of these state-of-the art deep learning based methods
[33, 23, 22] on galaxy images each of which contains a number of astronomical
sources, and found that these methods reconstructed the main structure of the
most obvious object in the image. However, some details and surrounding dim
sources were missing, and considerable amounts of noise remained in the
reconstructed image. Therefore, we turn to diffusion models, which could
potentially address these limitations, since they are able to generate diverse
high-resolution images with high semantic fidelity of conditioning [29]. To
the best of our knowledge, no prior studies have used diffusion models to
reconstruct dirty images in radio interferometry.

Figure 2: Overview of the Proposed VIC-DDPM. The diffusion process starts from
$\mathbf{x_{0}}$ and Gaussian noise is gradually added to $\mathbf{x_{t-1}}$
to obtain $\mathbf{x_{t}}$ until $t=T$. The reverse process starts from random
noise and generates $\mathbf{x_{t-1}}$ from $\mathbf{x_{t}}$. Visibility data
and dirty images are added to the reverse process to condition the clean image
generation. Details are described in Section 3.1.

###  2.3 DDPM

Denoising Diffusion Probabilistic Models (DDPMs) are a class of deep
generative models that learn to generate realistic samples from a given data
distribution. The core idea behind DDPMs, introduced by Sohl-Dickstein et al.
[26] and further developed by Ho et al. [13], is to reverse a denoising
process that transforms the data distribution into Gaussian noise. The
diffusion process can be thought of as a series of noise-corrupted versions of
the original data, where each step progressively adds more noise to the data
until the result becomes Gaussian noise [26]. The model then learns to denoise
the data by predicting the preceding, less noisy version of the data at each
step, given the current noisy version. To generate new samples, the model
starts from Gaussian noise and simulates the reverse diffusion process by
iteratively denoising the data at each step [13]. As the model has learned to
denoise data at every step of the process, the generated samples become
progressively more realistic and structured.

Diffusion models show great generative capabilities to produce fine levels of
details [8]. Current diffusion architectures are well designed to work with
image data in the spatial domain. When their underlying neural backbone is
implemented as a U-Net, the generative capacity of these models shifts from
the natural fitting of inductive biases to image-like data [10, 13, 22, 27,
21]. Latent diffusion models further introduce cross-attention layers into the
U-Net model architecture, making diffusion models more powerful and flexible
with general conditioning input [29]. In our work, we propose to apply cross-
attention with visibility data and dirty image for radio interferometric image
reconstruction.

##  3 Our Method

In this section, we present our Visibility and Dirty Image Conditional DDPM
(VIC-DDPM) for radio interferometric image reconstruction. We first give an
overview of our method and formulate the VIC-DDPM. Then we present the details
of visibility and dirty image conditioning, respectively.

###  3.1 Formulation of VIC-DDPM

Suppose $\mathbf{x}\in\mathbb{R}^{n}$ is an image of real sky’s brightness
distribution and $V\in\mathbb{R}^{m}$ is the sampled visibility. They follow
the forward model:

$\displaystyle V=\mathbf{P}_{\Omega}F(\mathbf{x})+\epsilon^{\prime}$ (2)

where $F$ refers to the Fourier transformation,
$\mathbf{P}_{\Omega}\in\mathbb{R}^{m\times n}$ is an under-sampling matrix
with $\Omega$ denoting the sample pattern produced by the sampling function of
the telescope, and $\epsilon^{\prime}$ denotes noise.

The task of radio interferometric image reconstruction can be simplified to
estimating the posterior distribution $q\left(\mathbf{x}\mid V\right)$. Since
$\mathbf{x^{d}}=F^{-1}\left(V\right)$, the task can then be converted to
estimate $q\left(\mathbf{x}\mid\mathbf{x^{d}},V\right)$, leveraging the
complementary advantages of visibility observations and dirty images. Then we
define our diffusion model in the following form:

$\displaystyle p_{\theta}\left(\mathbf{x}_{0}\mid\mathbf{x^{d}},V\right):=\int
p_{\theta}\left(\mathbf{x}_{0:T}\mid\mathbf{x^{d}},V\right)\mathrm{d}\mathbf{x}_{1:T}.$
(3)

The reverse process
$p_{\theta}\left(\mathbf{x}_{0:T}\mid\mathbf{x^{d}},V\right)$ is defined as:

$\displaystyle
p_{\theta}\left(\mathbf{x}_{0:T}\mid\mathbf{x^{d}},V\right)=p\left(\mathbf{x}_{T}\right)\prod_{t=1}^{T}p_{\theta}\left(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},\mathbf{x^{d}},V\right),$
(4) $\displaystyle
p_{\theta}\left(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},\mathbf{x^{d}},V\right)=\mathcal{N}\left(\mathbf{x}_{t-1};\boldsymbol{\mu}_{\theta}\left(\mathbf{x}_{t},t,\mathbf{x^{d}},V\right),\sigma_{t}^{2}\mathbf{I}\right).$
(5)

Similar to the original DDPM [13], we fix the diffusion process to a Markov
chain, defined as:

$\displaystyle
q\left(\mathbf{x}_{1:T}\mid\mathbf{x}_{0},\mathbf{x^{d}},V\right):=\prod_{t=1}^{T}q\left(\mathbf{x}_{t}\mid\mathbf{x}_{t-1},\mathbf{x^{d}},V\right),$
(6) $\displaystyle
q\left(\mathbf{x}_{t}\mid\mathbf{x}_{t-1},\mathbf{x^{d}},V\right):=\mathcal{N}\left(\mathbf{x}_{t};\alpha_{t}\mathbf{x}_{t-1},\beta_{t}^{2}\mathbf{I}\right).$
(7)

The diffusion process gradually adds Gaussian noise to data based on scheduled
$\alpha_{1},...,\alpha_{t}$ and $\beta_{1},...,\beta_{t}$.

Different from the original DDPM, we follow most recent work [10, 34] to set
the $\alpha_{t}$ and $\beta_{t}$ schedule.

Let
$\bar{\alpha}_{t}=\prod_{i=i}^{t}\alpha_{i},\bar{\beta}_{t}^{2}=\sum_{i=1}^{t}\frac{\bar{\alpha}_{t}^{2}}{\bar{\alpha}_{i}^{2}}\beta_{i}^{2},\bar{\alpha}_{0}=1,\bar{\beta}_{0}=0$:

$\displaystyle
q\left(\mathbf{x}_{t}\mid\mathbf{x}_{0},\mathbf{x^{d}},V\right)$
$\displaystyle=\mathcal{N}\left(\mathbf{x}_{t};\bar{\alpha}_{t}\mathbf{x}_{0},\bar{\beta}_{t}^{2}\mathbf{I}\right),$
(8) $\displaystyle
q\left(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},\mathbf{x}_{0},\mathbf{x^{d}},V\right)$
$\displaystyle=\mathcal{N}\left(\mathbf{x}_{t-1};\tilde{\boldsymbol{\mu}}_{t}\left(\mathbf{x}_{t},\mathbf{x}_{0}\right),\bar{\beta}_{t}^{2}\mathbf{I}\right),$
(9)

where

$\displaystyle\tilde{\boldsymbol{\mu}}_{t}\left(\mathbf{x}_{t},\mathbf{x}_{0}\right)=\frac{\alpha_{t}\bar{\beta}_{t-1}^{2}}{\bar{\beta}_{t}^{2}}\mathbf{x}_{t}+\frac{\bar{\alpha}_{t-1}\beta_{t}^{2}}{\bar{\beta}_{t}^{2}}\mathbf{x}_{0},\text{and
}\tilde{\beta}_{t}=\frac{\beta_{t}\bar{\beta}_{t-1}}{\bar{\beta}_{t}}.$ (10)

With our schedule setting of $\alpha_{t}$, we have $\bar{\alpha}_{T}\approx
0$.

Then, we perform the training of
$p_{\theta}\left(\mathbf{x}_{0}\mid\mathbf{x}_{t-1},\mathbf{x^{d}},V\right)$
by optimizing the variational bound on the negative log likelihood, similar to
DDPM [13]:

$\displaystyle\mathbb{E}\left[-\log
p_{\theta}\left(\mathbf{x}_{0}\mid\mathbf{x^{d}},V\right)\right]$ (11)
$\displaystyle\leq\mathbb{E}_{q}\left[-\log\frac{p_{\theta}\left(\mathbf{x}_{0:T}\mid\mathbf{x^{d}},V\right)}{q\left(\mathbf{x}_{1:T}\mid\mathbf{x}_{0},\mathbf{x^{d}},V\right)}\right]$
$\displaystyle=\mathbb{E}_{q}\Bigg{[}-\log
p\left(\mathbf{x}_{T}\mid\mathbf{x^{d}},V\right)$
$\displaystyle\qquad\quad-\sum_{t\geq
1}\log\frac{p_{\theta}\left(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},\mathbf{x^{d}},V\right)}{q\left(\mathbf{x}_{t}\mid\mathbf{x}_{t-1},\mathbf{x^{d}},V\right)}\Bigg{]}$
$\displaystyle=:L.$

We choose the parameterization:

$\displaystyle\boldsymbol{\mu}_{\theta}\left(\mathbf{x}_{t},t,\mathbf{x^{d}},V\right)$
$\displaystyle=\frac{1}{\alpha_{t}}\left(\mathbf{x}_{t}-\frac{\beta_{t}^{2}}{\bar{\beta_{t}}}\boldsymbol{\epsilon}_{\theta}\left(\mathbf{x}_{t},t,\mathbf{x^{d}},V\right)\right)$
(12)

where $\boldsymbol{\epsilon}_{\theta}$ is a function approximator for
predicting $\boldsymbol{\epsilon}$ from $\mathbf{x}_{t}$. If $t=0$, we set
$\mathbf{z}_{t}=0$; otherwise, we sample $\mathbf{x}\sim
p_{\theta}\left(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},\mathbf{x^{d}},V\right)$ by
computing
$\mathbf{x}_{t-1}=\frac{1}{\alpha_{t}}\left(\mathbf{x}_{t}-\frac{\beta_{t}^{2}}{\bar{\beta_{t}}}\boldsymbol{\epsilon}_{\theta}\left(\mathbf{x}_{t},t,\mathbf{x^{d}},V\right)\right)+\sigma_{t}\mathbf{z}_{t}$,
where $\mathbf{z}_{t}\sim\mathcal{N}(0,\mathbf{I})$.

Then, based on (Eq. 9), and with the parameterization (Eq. 12), we simplify
$L$ to:

$\displaystyle
L_{simple}(\theta)=\mathbb{E}_{\mathbf{x}_{0},t,\boldsymbol{\epsilon}}\left\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_{\theta}\left(\bar{\alpha}_{t}\mathbf{x}_{0}+\bar{\beta}_{t}\boldsymbol{\epsilon},t,\mathbf{x^{d}},V\right)\right\|_{2}^{2}$
(13)

where $\boldsymbol{\epsilon}\sim\mathcal{N}\left(\mathbf{0},\mathbf{I}\right)$
and $t\sim\text{Uniform}({0,...,T})$.

In summary, Figure 2 illustrates the diffusion process and reverse process in
VIC-DDPM. Algorithm 1 presents the diffusion process using the loss function
(Eq. 13). Algorithm 2 shows the sampling procedure.

Algorithm 1 Training

1:repeat

2: $\\{\mathbf{x}\\}\sim q\left(\mathbf{x}\right)$, obtain $V$,
$\mathbf{x^{d}}=F^{-1}\left(V\right)$

3: $\boldsymbol{\epsilon}\sim\mathcal{N}(0,\mathbf{I})$

4: $t\sim\text{Uniform}({0,...,T})$

5:
$\mathbf{x}_{t}=\bar{\alpha}_{t}\mathbf{x}_{0}+\bar{\beta_{t}}\boldsymbol{\epsilon}$

6: Compute the loss
$L(\theta)=\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_{\theta}\left(\mathbf{x}_{t},t,\mathbf{x^{d}},V\right)\|_{2}^{2}$

7: Update $\theta$ with gradient descent on $L(\theta)$

8:until convergence

Algorithm 2 Sampling

1:Given $V$, $\mathbf{x^{d}}=F^{-1}\left(V\right)$

2:for $t=T,...,1$, do

3: $\mathbf{z}_{t}\sim\mathcal{N}(0,\mathbf{I})$ if $t>1$, else
$\mathbf{z}_{t}=0$

4:
$\mathbf{x}_{t-1}=\frac{1}{\alpha_{t}}\left(\mathbf{x}_{t}-\frac{\beta_{t}^{2}}{\bar{\beta_{t}}}\boldsymbol{\epsilon}_{\theta}\left(\mathbf{x}_{t},t,\mathbf{x^{d}},V\right)\right)+\sigma_{t}\mathbf{z}_{t}$

5:return $\mathbf{x}_{0}$

###  3.2 Details of Conditioning

We use the U-Net [22] architecture with ResNet [11] blocks and attention
layers [32] as our backbone neural network to represent
$\boldsymbol{\epsilon}_{\theta}\left(\mathbf{x}_{t},t,\mathbf{x^{d}},V\right)$.
Figure 3 gives an overview of the network.

Figure 3: The U-Net architecture in our method to represent
$\boldsymbol{\epsilon}_{\theta}\left(\mathbf{x}_{t},t,\mathbf{x^{d}},V\right)$.

####  3.2.1 Dirty Image Conditioning

As shown on the left in Figure 3, we concatenate the dirty image $x_{d}$ with
the input $x_{t}$ of the U-Net, to leverage the information provided by the
dirty image. The concatenation is done by each channel of the image. This way,
we enable the U-Net to extract features from both the input data and the dirty
image, thereby incorporating prior information from the dirty image into the
reconstruction process.

####  3.2.2 Sparse Visibility Encoding

Visibility data is sparse, so we apply encoding to increase the information
density and then incorporate the encoded visibility $\hat{V}$ into our
network, as shown in the top left of Figure 3.

Figure 4 illustrates our encoding method. As shown on the left in Figure 4,
the sparsely sampled visibility data is in the form of
$\left\\{u_{s},v_{s},{V}\left(u_{s},v_{s}\right)\right\\}$, where
$(u_{s},v_{s})$ are the coordinates at which a measurement is sampled and
${V}\left(u_{s},v_{s}\right)$ is the complex value of the sample.

Figure 4: Sparse Visibility Encoding.

To integrate each visibility value and its corresponding position, we first
encode each sample individually using positional embedding
($\text{PE}(u_{s},v_{s})$ in Figure 4). Specifically, we first encode the
positional information of a sample point using a NeRF-style [17] embedding
method (axis-aligned, power-of-two frequency sinusoidal encoding) [33]. It
facilitates the following Multi-Layer-Perception (MLP) to learn high frequency
information [30, 33]. After that, the positional embedding
$\text{PE}(u_{s},v_{s})$ and the complex value of visibility
${V}\left(u_{s},v_{s}\right)$ are concatenated to form visibility tokens
$V^{\prime}$.

Denote the sparse visibility tokens as
$V^{\prime}=\left[v^{\prime}_{1};v^{\prime}_{2};v^{\prime}_{3};\ldots
v^{\prime}_{n}\right]$, and $V^{\prime}\in\mathbb{R}^{n\times d}$, where $n$
is the number of sampled measurement points and $d$ is the number of
dimensions of visibility tokens. Furthermore, the sampled points are divided
into $m$ groups by their rotation angles. Then, we apply a linear mapping
layer $M$ and a mean pooling for encoding, as shown on the right of Figure 4.
The encoding result is
$\hat{V}=\left[\hat{v}_{1};\hat{v}_{2};\hat{v}_{3};\ldots\hat{v}_{m}\right]$,
where $\hat{V}\in\mathbb{R}^{m\times d}$:

$\displaystyle\hat{v}_{i}=\frac{m}{n}\sum_{j\in Group_{i}}M(v_{j}),\text{
}i=1\text{ to }m.$ (14)

This encoded visibility is then used as the input of the cross-attention
module in the U-Net.

####  3.2.3 Cross-Attention Fusion

Next, we employ a cross-attention mechanism [32, 21] to fuse the encoded
visibility information into the U-Net. Cross-attention can effectively learn
relationships between different data domains. Specifically, we use the
concatenation of both image features and visibility features to form Key and
Value:

$\displaystyle{CrossAttention}=\operatorname{softmax}\left(\frac{\textit{Query}\cdot\textit{Key}^{T}}{\sqrt{d}}\right)\cdot\textit{Value},$
(15)

where

$\displaystyle\textit{Query}=W_{\textit{Query}}\cdot\varphi\left(\mathbf{x}_{t},t,\mathbf{x^{d}}\right),$
(16)
$\displaystyle\textit{Key}=W_{\textit{Key}}\cdot\operatorname{Concat}(\varphi\left(\mathbf{x}_{t},t,\mathbf{x^{d}}\right),\text{E}(V)),$
(17)
$\displaystyle\textit{Value}=W_{\textit{Value}}\cdot\operatorname{Concat}(\varphi\left(\mathbf{x}_{t},t,\mathbf{x^{d}}\right),\text{E}(V)).$
(18)

Here, $\varphi\left(\mathbf{x}_{t},t,\mathbf{x^{d}}\right)$ denotes the
intermediate latent representation of the U-Net that implements the predictor
$\epsilon_{\theta}$. $W_{\textit{query}},W_{\textit{Key}},W_{\textit{Value}}$
are learnable weights. $\text{E}(V)$ refers to the encoded visibility tokens.

The aim of this cross-attention mechanism is to weigh the contributions of
features from both the dirty image and the sparse visibility, allowing the
U-Net to selectively attend to the most relevant information when generating
the reconstructed image.

In summary, we design the
$\boldsymbol{\epsilon}_{\theta}\left(\mathbf{x}_{t},t,\mathbf{x^{d}},V\right)$
as follows:
$\boldsymbol{\epsilon}_{\theta}\left(\mathbf{x}_{t},t,\mathbf{x^{d}},V\right)=f_{out}(\text{Cross-
Attn}(f_{in}(g(\mathbf{x}_{t},\mathbf{x^{d}}),t),\text{E}(V)),t)$

where $f_{in}$ refers to the input blocks , and $f_{out}$ the output blocks
after the cross-attention module in the U-Net structure, as illustrated in
Figure 3. $\text{E}(V)$ refers to the encoded visibility tokens, and
$g(\cdot,\cdot)$ the concatenation of images by channel.

##  4 Experiments

In this section, we perform a comprehensive evaluation on our method in
comparison with several classic and recent state-of-the-art methods to
demonstrate the overall improvement achieved by our approach. We also conduct
ablation experiments to study the effects of individual techniques in our
method. Additionally, we vary the number of sample noise used to obtain a
result clean image, the number of diffusion steps to perform towards each
noise image, and the total number of training images to examine our model
performance.

###  4.1 Experimental Setup

Platform. We conduct all experiments on a server with two AMD EPYC 7302 CPUs,
128GB main memory, and eight Nvidia RTX 3090 GPUs each with 24GB device
memory. The server is equipped with an NVME 2TB SSD and four 1TB SATA hard
disks. The operating system is Ubuntu 20.04. Our model is implemented in
PyTorch 1.8.1 [20].

Datasets. We use the Galaxy10 DECals dataset [12] in our experiments, as in
the most recent work on astronomical interferometric image reconstruction
[33]. This dataset contains a total of 17,736 galaxy images, derived from the
DESI Legacy Imaging Surveys [9], which combine data from the Beijing-Arizona
Sky Survey (BASS) [35], the DECam Legacy Survey (DECaLS) [2], and the Mayall
z-band Legacy Survey [24]. In our experiments, all images are of a size 128
$\times$ 128 in pixels. With these images as the ground truth, we then use the
eht-imaging toolkit [5, 6] to generate visibility data in the form of
$\left\\{u_{s},v_{s},{V}\left(u_{s},v_{s}\right)\right\\}$, with the
observation parameters set to match an 8-telescope Event Horizon Telescope
(EHT) array [33]. The number of sampled visibility points on each image is
1660. Also following Wu et al. [33], we use the discrete Fourier transform
(DFT) method to generate dirty images from the visibility data. Finally, we
randomly split the 17,736 images into a training set of 17,224 images and a
testing set of 512 images following Wu et al. [33].

Implementation Details. Our experimental model architectures are based on
U-Net [22] and augmented with timestep embedding and channel-wise self-
attention block following DDPM [13] and Guided-Diffusion [10]. In visibility
encoding, the number of elements in a positional embedding vector is set to 22
and the number of points in each point group is 20. In addition, we add
spatial-wise attention layers and cross-attention layers in U-Net. Finally,
the training batch size in VIC-DDPM is set to 32 images, and the total number
of training steps is 25k.

Methods under Comparison. We compare VIC-DDPM with three other methods for
radio interferometric image reconstruction, including the classic method CLEAN
[14] and recent deep learning-based approaches radionets [23] and Transformer-
Conditioned Neural Fields [33]. We use the original code of each of these
methods and follow the parameter setting in the original code for the best
performance. In addition, we test our U-Net in both visibility domain only,
U-Net(Vis), and image domain only, U-Net(Img), as supplementary baselines.

Evaluation Metrics. We use two common metrics to evaluate image quality: Peak
Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM).
They quantify the overall image quality of reconstructed images and the
perceptual similarity to the ground truth images, respectively. We compute
these two metrics using the scikit-image package [25], which follow the
formulas presented by Hore et al. [15].

###  4.2 Overall Comparison

Figure 5: Visual Results of Overall Comparison.

First, we present in Figure 5 five representative reconstructed images on the
Galaxy10(DECals) dataset from all methods under comparison as well as the
dirty images and the ground truth images. Comparing the dirty images in Figure
5 (a) and the ground truth in Figure 5 (h), we see that dirty images contain
many artifacts and that actual objects are often distorted.

As shown in Figure 5 (b), the CLEAN [14] method reduces most artifacts, but
misses the original structures of the sources. Due to the assumptions in the
CLEAN [14] algorithm, the emission distribution cannot be smoothly
reconstructed. In comparison, the U-Net approach in the visibility domain
(Figure 5 (c)) leaves a lot of artifacts. The spatial domain U-Net (Figure 5
(d)) achieves relatively better reconstruction, but does not fix the
distortion and overlooks some faint sources or generates non-existent faint
sources. Radionets [23] (Figure 5 (e)) performs well in denoising and
recovering from distortion; however, it has problems distinguishing separate
sources that are close to one another. The Transformer-Conditioned Neural
Fields method (denoted TransNF) [33] (Figure 5 (f)) faithfully recovers the
main objects in the observation area except when the edges of the objects are
blurry or when the surrounding objects are small and dim.

The results of VIC-DDPM (Figure 5 (g)) show that our method effectively
removes artifacts and reconstructs details of the true objects, including
relatively small or faint sources and detailed structures. These results
demonstrate the effectiveness of our DDPM conditioned with both dirty images
and visibility data.

Next, we compute the PSNR and SSIM scores, with mean and standard deviation,
for all 512 reconstructed images produced by our method and the methods under
comparison. The results are shown in Table 1. The greater the PSNR and SSIM
scores, the better.

The experimental results demonstrate that our method consistently achieves
better performance in both PSNR and SSIM, indicating the effectiveness of our
approach in providing accurate reconstructions.

Table 1: Performance of Different Methods. Models | Domain | Metrics
---|---|---
Img | Vis | PSNR$\uparrow$ | SSIM$\uparrow$
Dirty Image |  |  | 10.406 $\pm$ 1.101 | 0.6321 $\pm$ 0.0590
CLEAN | ✓ |  | 17.366 $\pm$ 2.406 | 0.7130 $\pm$ 0.0405
U-Net (Img) | ✓ |  | 21.414 $\pm$ 1.654 | 0.8028 $\pm$ 0.0194
U-Net (Vis) |  | ✓ | 13.015 $\pm$ 1.103 | 0.7079 $\pm$ 0.0230
Radionets |  | ✓ | 21.433 $\pm$ 2.105 | 0.7940 $\pm$ 0.0237
Transformer_NF |  | ✓ | 20.707 $\pm$ 2.970 | 0.8084 $\pm$ 0.0404
VIC-DDPM | ✓ | ✓ | 22.615 $\pm$ 1.531 | 0.8242 $\pm$ 0.0224

###  4.3 Ablation Experiments

In the ablation experiments, we test the following variants of our method by
selectively removing one or more components to investigate the individual
contributions of the components to the performance: (1) without any
conditioning; (2) without the visibility data conditioning; and (3) without
the dirty image conditioning. We train these variants by add no conditioning,
replacing the visibility data with zeros, and by replacing the dirty image
with zeros, respectively. We compare these three variants with the original
dirty images and the reconstructed images from the full VIC-DDPM.

Figure 6: Ablation Results.

Figure 7: Visual Examples of Ablation Results. Columns from let to right are
(a) original dirty image, (b) unconditional DDPM, (c) our model trained
without dirty image conditioning, (d) our model trained without visibility
conditioning, (e) full model of VIC-DDPM, and (f) real sky, respectively.

We evaluate the performance of each ablation model using the same dataset and
evaluation metrics (PSNR and SSIM) as in the previous experiment. The results
in Figure 6 show that our full VIC-DDPM model consistently achieves the best
performance, demonstrating the effectiveness of combining dirty image
conditioning and visibility conditioning. The dirty image conditioning seems
more effective than the visibility data conditioning, suggesting that spatial
information plays a more important role than signal-to-noise separation in
DDPM. Some visual examples of ablation results are shown in Figure 7. These
results confirm that the full model (column e) is best, and that dirty image
conditioning (column d) is more effective than visibility conditioning (column
c). Without any conditioning, column (b) is the least effective.

###  4.4 Effect of Parameter Setting

We also conduct experiments to investigate the effect of two important
parameters in our model – number of random noise sample images to test for
each dirty image and number of diffusion steps. We report the performance on
PSNR and SSIM in Figure 8. Figure 8(a) shows that as the number of random
noise sample images used increases for each test dirty image, the mean quality
of the result images also increases; however, when the number of sample images
further increases, the mean quality remains stable or slightly decreases. In
comparison, Figure 8(b) shows that the result quality is almost constant
regardless of number of diffusion steps. These results indicate that in our
experimental setting, selecting 5 samples for each test dirty image and
running each test for 1000 steps would be a good choice for best result
quality.

Figure 8: Effect of Parameters: (a) number of random noise images to test for
each dirty image; (b) number of diffusion steps.

###  4.5 Effect of Training Data Size

Table 2: Performance (PSNR/SSIM) with training dataset size varied Data | Methods
---|---
RadioNet | Transformer_NF | VIC-DDPM
0.5k | 19.401 $\pm$ 1.639 | 18.235 $\pm$ 2.802 | 17.149 $\pm$ 6.856
0.7586 $\pm$ 0.0210 | 0.7610 $\pm$ 0.0634 | 0.7417 $\pm$ 0.1220
1k | 20.267 $\pm$ 1.822 | 18.935 $\pm$ 2.900 | 22.420 $\pm$ 3.765
0.770 $\pm$ 0.0222 | 0.7781 $\pm$ 0.0675 | 0.8236 $\pm$ 0.0354
2k | 20.610 $\pm$ 1.926 | 18.565 $\pm$ 2.360 | 22.170 $\pm$ 1.986
0.7765 $\pm$ 0.0241 | 0.7735 $\pm$ 0.0578 | 0.8137 $\pm$ 0.0211
4k | 20.657 $\pm$ 1.996 | 20.069 $\pm$ 2.611 | 22.403 $\pm$ 2.172
0.7787 $\pm$ 0.0269 | 0.8028 $\pm$ 0.0518 | 0.8210 $\pm$ 0.0328
17k | 21.433 $\pm$ 2.105 | 20.707 $\pm$ 2.970 | 22.615 $\pm$ 1.531
0.7940 $\pm$ 0.023 | 0.8084 $\pm$ 0.040 | 0.8242 $\pm$ 0.022

Model performance may vary with the training dataset size. We therefore use
different sizes of training datasets to evaluate our model performance in
PSNR/SSIM in comparison with two latest deep-learning methods – Radionets [23]
and Transformer-Conditioned Neural Fields (denoted Transformer_NF) [33]. To
create training sets of different sizes, we randomly choose 500, 1000, 2000,
and 4000 of the 17k training images in the Galaxy10 DECals dataset [12]. As
shown in Table 2, when the dataset size increases from 0.5k to 17k, the
PSNR/SSIM performance increases by 10.5%/4.7%, 13.6%/6.2% and 31.9%/11.1% on
three models, respectively. Compared with the other two methods, the
performance of VIC-DDPM significantly improves when the dataset size increases
from 500 to 1000. With a training set of 1000 images, VIC-DDPM performance is
already considerably higher than the other two methods, and further increase
of training dataset size shows little effect. In contrast, both radionets and
Transformer_NF continue to improve as the training dataset size increases, but
the performance improvement is not as significant as VIC-DDPM. As a result,
VIC-DDPM outperforms the other two on data sizes from 1K to 17K. This result
suggests that VIC-DDPM requires a much smaller training set than the other two
to obtain high-quality reconstruction. We plan to find larger training
datasets to give more room to the other two methods for performance
improvement.

To further investigate the performance differences, we examine the visual
effects of reconstructed images and provide a few examples with different
training dataset sizes in Figure 9. Subfigures on each row are from a single
method, and those on each column are on the same training dataset size. We
find that, even though our model is slightly lower than the other two methods
on the numeric performance metrics when the training set size is 500 in Table
2, our method’s reconstruction of the structures, in the bottom left of each
example (a) through (d) is noticeably superior to the other two methods when
visually inspected by the human eye, as shown in Figure 9. These visual
results suggest that the lower metric values of our method at 500 training
images are possibly because the background noise has not been effectively
reduced by then.

As observed in Figure 9 (a) , (c) and (d), VIC-DDPM is capable of
reconstructing the basic contours of astronomical sources after learning only
500 samples. In contrast, other methods require learning from 1000 to 4000
samples to reconstruct the basic contours. On reconstruction of images
containing multiple astronomical sources, as shown in Figure 9 (a) and (b),
our method can discern the positions and structures of multiple astronomical
sources after learning from 500 training samples, while other methods cannot.

Figure 9: Visual results on four example images (a)-(d) from three models
including (1) radionets, (2) Transformer-Conditioned Neural Fields, and (3)
VIC-DDPM trained with datasets of different sizes.

##  5 Conclusion and Future Work

In conclusion, we have presented VIC-DDPM, a Visibility and Image Conditioned
Denoising Diffusion Probabilistic Model, for radio interferometric image
reconstruction. Inheriting the capability of fine detail generation from DDPM
as well as incorporating the complementary strengths of data in spectral and
spatial domains, VIC-DDPM improves the accuracy of reconstructed images with
effective removal of artifacts, and recovery of structural details and dim
sources. Experimental results demonstrate the superiority of VIC-DDPM over
both traditional and deep-learning based approaches on the Galaxy10 DECals
dataset.

In the future, we will apply our methods on both real and synthetic datasets
based on different kinds of radio telescope arrays and generalize our methods
to images in other domains.

This work was partially supported by Grant 2020SKA0110300 from the National
SKA Program of China.

## References

  * [1] JG Ables, ‘Maximum entropy spectral analysis’, Astronomy and Astrophysics Supplement, Vol. 15, p. 383, 15, 383, (1974).
  * [2] Robert D Blum, Kaylan Burleigh, Arjun Dey, David J Schlegel, Aaron M Meisner, Michael Levi, Adam D Myers, Dustin Lang, John Moustakas, Anna Patej, et al., ‘The decam legacy survey’, in American Astronomical Society Meeting Abstracts# 228, volume 228, pp. 317–01, (2016).
  * [3] Alexandre Boulch and Renaud Marlet, ‘Deep learning for robust normal estimation in unstructured point clouds’, in Computer Graphics Forum, volume 35, pp. 281–290. Wiley Online Library, (2016).
  * [4] Bernard F Burke, Francis Graham-Smith, and Peter N Wilkinson, An introduction to radio astronomy, Cambridge University Press, 2019.
  * [5] Andrew A Chael, Katherine L Bouman, Michael D Johnson, Ramesh Narayan, Sheperd S Doeleman, John FC Wardle, Lindy L Blackburn, Kazunori Akiyama, Maciek Wielgus, Chi-kwan Chan, et al., ‘ehtim: Imaging, analysis, and simulation software for radio interferometry’, Astrophysics Source Code Library, ascl–1904, (2019).
  * [6] Andrew A Chael, Michael D Johnson, Katherine L Bouman, Lindy L Blackburn, Kazunori Akiyama, and Ramesh Narayan, ‘Interferometric imaging directly with closure phases and closure amplitudes’, The Astrophysical Journal, 857(1), 23, (2018).
  * [7] Liam Connor, Katherine L Bouman, Vikram Ravi, and Gregg Hallinan, ‘Deep radio-interferometric imaging with polish: Dsa-2000 and weak lensing’, Monthly Notices of the Royal Astronomical Society, 514(2), 2614–2626, (2022).
  * [8] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah, ‘Diffusion models in vision: A survey’, IEEE Transactions on Pattern Analysis and Machine Intelligence, (2023).
  * [9] Arjun Dey, David J Schlegel, Dustin Lang, Robert Blum, Kaylan Burleigh, Xiaohui Fan, Joseph R Findlay, Doug Finkbeiner, David Herrera, Stéphanie Juneau, et al., ‘Overview of the desi legacy imaging surveys’, The Astronomical Journal, 157(5), 168, (2019).
  * [10] Prafulla Dhariwal and Alexander Nichol, ‘Diffusion models beat gans on image synthesis’, Advances in Neural Information Processing Systems, 34, 8780–8794, (2021).
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, ‘Deep residual learning for image recognition’, in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, (2016).
  * [12] Leung Henry.  Galaxy10 decals dataset.  https://github.com/henrysky/Galaxy10, 2021.
  * [13] Jonathan Ho, Ajay Jain, and Pieter Abbeel, ‘Denoising diffusion probabilistic models’, Advances in Neural Information Processing Systems, 33, 6840–6851, (2020).
  * [14] JA Högbom, ‘Aperture synthesis with a non-regular distribution of interferometer baselines’, Astronomy and Astrophysics Supplement Series, 15, 417, (1974).
  * [15] Alain Hore and Djemel Ziou, ‘Image quality metrics: Psnr vs. ssim’, in 2010 20th international conference on pattern recognition, pp. 2366–2369. IEEE, (2010).
  * [16] Honghao Liu, Qiong Luo, and Feng Wang, ‘Efficient radio interferometric imaging on the gpu’, in 2022 IEEE 18th International Conference on e-Science (e-Science), pp. 95–104. IEEE, (2022).
  * [17] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng, ‘Nerf: Representing scenes as neural radiance fields for view synthesis’, Communications of the ACM, 65(1), 99–106, (2021).
  * [18] Warren R Morningstar, Yashar D Hezaveh, Laurence Perreault Levasseur, Roger D Blandford, Philip J Marshall, Patrick Putzky, and Risa H Wechsler, ‘Analyzing interferometric observations of strong gravitational lenses with recurrent and convolutional neural networks’, arXiv preprint arXiv:1808.00011, (2018).
  * [19] Warren R Morningstar, Laurence Perreault Levasseur, Yashar D Hezaveh, Roger Blandford, Phil Marshall, Patrick Putzky, Thomas D Rueter, Risa Wechsler, and Max Welling, ‘Data-driven reconstruction of gravitationally lensed galaxies using recurrent inference machines’, The Astrophysical Journal, 883(1), 14, (2019).
  * [20] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al., ‘Pytorch: An imperative style, high-performance deep learning library’, Advances in neural information processing systems, 32, (2019).
  * [21] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer, ‘High-resolution image synthesis with latent diffusion models’, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695, (2022).
  * [22] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, ‘U-net: Convolutional networks for biomedical image segmentation’, in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234–241. Springer, (2015).
  * [23] Kevin Schmidt, Felix Geyer, Stefan Fröse, Paul-Simon Blomenkamp, Marcus Brüggen, Francesco de Gasperin, Dominik Elsässer, and Wolfgang Rhode, ‘Deep learning-based imaging in radio interferometry’, arXiv preprint arXiv:2203.11757, (2022).
  * [24] David R Silva, Robert D Blum, Lori Allen, Arjun Dey, David J Schlegel, Dustin Lang, John Moustakas, Aaron M Meisner, Francisco Valdes, Anna Patej, et al., ‘The mayall z-band legacy survey’, in American Astronomical Society Meeting Abstracts# 228, volume 228, pp. 317–02, (2016).
  * [25] Himanshu Singh and Himanshu Singh, ‘Basics of python and scikit image’, Practical Machine Learning and Image Processing: For Facial Recognition, Object Detection, and Pattern Recognition Using Python, 29–61, (2019).
  * [26] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli, ‘Deep unsupervised learning using nonequilibrium thermodynamics’, in International Conference on Machine Learning, pp. 2256–2265. PMLR, (2015).
  * [27] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole, ‘Score-based generative modeling through stochastic differential equations’, arXiv preprint arXiv:2011.13456, (2020).
  * [28] He Sun and Katherine L Bouman, ‘Deep probabilistic imaging: Uncertainty quantification and multi-modal solution characterization for computational imaging’, in Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 2628–2637, (2021).
  * [29] Yu Takagi and Shinji Nishimoto, ‘High-resolution image reconstruction with latent diffusion models from human brain activity’, Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (In Press), (2023).
  * [30] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng, ‘Fourier features let networks learn high frequency functions in low dimensional domains’, Advances in Neural Information Processing Systems, 33, 7537–7547, (2020).
  * [31] A Richard Thompson, James M Moran, and George W Swenson, Interferometry and synthesis in radio astronomy, Springer Nature, 2017.
  * [32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin, ‘Attention is all you need’, Advances in neural information processing systems, 30, (2017).
  * [33] Benjamin Wu, Chao Liu, Benjamin Eckart, and Jan Kautz, ‘Neural interferometry: Image reconstruction from astronomical interferometers using transformer-conditioned neural fields’, in Proceedings of the AAAI Conference on Artificial Intelligence, (2022).
  * [34] Yutong Xie and Quanzheng Li, ‘Measurement-conditioned denoising diffusion probabilistic model for under-sampled medical image reconstruction’, in Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part VI, pp. 655–664. Springer, (2022).
  * [35] Hu Zou, Xu Zhou, Xiaohui Fan, Tianmeng Zhang, Zhimin Zhou, Jundan Nie, Xiyan Peng, Ian McGreer, Linhua Jiang, Arjun Dey, et al., ‘Project overview of the beijing–arizona sky survey’, Publications of the Astronomical Society of the Pacific, 129(976), 064101, (2017).