# SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation

Yen-Chi Cheng1 Hsin-Ying Lee2 Sergey Tulyakov2 Alexander Schwing1∗ Liangyan
Gui1∗
1University of Illinois Urbana-Champaign 2Snap Research
<EMAIL_ADDRESS><EMAIL_ADDRESS>
https://yccyenchicheng.github.io/SDFusion/

###### Abstract

In this work, we present a novel framework built to simplify 3D asset
generation for amateur users. To enable interactive generation, our method
supports a variety of input modalities that can be easily provided by a human,
including images, text, partially observed shapes and combinations of these,
further allowing to adjust the strength of each input. At the core of our
approach is an encoder-decoder, compressing 3D shapes into a compact latent
representation, upon which a diffusion model is learned. To enable a variety
of multi-modal inputs, we employ task-specific encoders with dropout followed
by a cross-attention mechanism. Due to its flexibility, our model naturally
supports a variety of tasks, outperforming prior works on shape completion,
image-based 3D reconstruction, and text-to-3D. Most interestingly, our model
can combine all these tasks into one swiss-army-knife tool, enabling the user
to perform shape generation using incomplete shapes, images, and textual
descriptions at the same time, providing the relative weights for each input
and facilitating interactivity. Despite our approach being shape-only, we
further show an efficient method to texture the generated shape using large-
scale text-to-image models.

Figure 1:  Applications of SDFusion. The proposed diffusion-based model
enables various applications. (left) SDFusion can generate shapes conditioned
on different input modalities, including partial shapes, images, and text.
SDFusion can even jointly handle multiple conditioning modalities while
controlling the strength for each of them. (right) We leverage pretrained 2D
models to texture 3D shapes generated by SDFusion.

##  1 Introduction

Generating 3D assets is a cornerstone of immersive augmented/virtual reality
experiences. Without realistic and diverse objects, virtual worlds will look
void and engagement will remain low. Despite this need, manually creating and
editing 3D assets is a notoriously difficult task, requiring creativity, 3D
design skills, and access to sophisticated software with a very steep learning
curve. This makes 3D asset creation inaccessible for inexperienced users. Yet,
in many cases, such as interior design, users more often than not have a
reasonably good understanding of what they want to create. In those cases, an
image or a rough sketch is sometimes accompanied by text indicating details of
the asset, which are hard to express graphically for an amateur.

Due to this need, it is not surprising that democratizing the 3D content
creation process has become an active research area. Conventional 3D
generative models require direct 3D supervision in the form of point clouds
[2, 21], signed distance functions (SDFs) [9, 25], voxels [42, 47], _etc_.
Recently, first efforts have been made to explore the learning of 3D geometry
from multi-view supervision with known camera poses by incorporating inductive
biases via neural rendering techniques [6, 37, 5, 14, 52]. While compelling
results have been demonstrated, training is often very time-consuming and
ignores available 3D data that can be used to obtain good shape priors. We
foresee an ideal collaborative paradigm for generative methods where models
trained on 3D data provide detailed and accurate geometry, while models
trained on 2D data provide diverse appearances. A first proof of concept is
shown in Figure 1.

In our pursuit of flexible and high-quality 3D shape generation, we introduce
SDFusion, a diffusion-based generative model with a signed distance function
(SDF) under the hood, acting as our 3D representation. Compared to other 3D
representations, SDFs are known to represent well high-resolution shapes with
arbitrary topology[18, 9, 23, 30]. However, 3D representations are infamous
for demanding high computational resources, limiting most existing 3D
generative models to voxel grids of $32^{3}$ resolution and point clouds of
$2K$ points. To side-step this issue, we first utilize an auto-encoder to
compress 3D shapes into a more compact low-dimensional representation. Because
of this, SDFusion can easily scale up to a $128^{3}$ resolution. To learn the
probability distribution over the introduced latent space, we leverage
diffusion models, which have recently been used with great success in various
2D generation tasks [22, 40, 4, 26, 19, 35]. Furthermore, we adopt task-
specific encoders and a cross-attention [34] mechanism to support multiple
conditioning inputs, and apply classifier-free guidance [17] to enable
flexible conditioning usage. Because of these strategies, SDFusion can not
only use a variety of conditions from multiple modalities, but also adjust
their importance weight, as shown in Figure 1. Compared to a recently proposed
autoregressive model [25] that also adopts an encoded latent space, SDFusion
achieves superior sample quality, while offering more flexibility to handle
multiple conditions and, at the same time, features reduced memory usage. With
SDFusion, we study the interplay between models trained on 2D and 3D data.
Given 3D shapes generated by SDFusion, we take advantage of an off-the-shelf
2D diffusion model [34], neural rendering [24], and score distillation
sampling [31] to texture the shapes given text descriptions as conditional
variables.

We conduct extensive experiments on the ShapeNet [7], BuildingNet [38], and
Pix3D [43] datasets. We show that SDFusion quantitatively and qualitatively
outperforms prior work in shape completion, 3D reconstruction from images, and
text-to-shape tasks. We further demonstrate the capability of jointly
controlling the generative model via multiple conditioning modalities, the
flexibility of adjusting relative weight among modalities, and the ability to
texture 3D shapes given textual descriptions, as shown in Figure 1.

We summarize the main contributions as follows:

  * •

We propose SDFusion, a diffusion-based 3D generative model which uses a signed
distance function as its 3D representation and a latent space for diffusion.

  * •

SDFusion enables conditional generation with multiple modalities, and provides
flexible usage by adjusting the weight among modalities.

  * •

We demonstrate a pipeline to synthesize textured 3D objects benefiting from an
interplay between 2D and 3D generative models.

Figure 2:  SDFusion Overview. (left) To enable high-resolution generation, we
first encode 3D shapes into a latent space, where a diffusion model is
trained. Furthermore, to enable flexible conditional generation, we adopt
class-specific encoders along with classifier-free guidance to enable multi-
modality conditioning. (right) At inference time, we can control the
importance of each conditioning modality.

##  2 Related Work

#### 3D Generative Models.

Different from 2D images, it is less clear how to effectively represent 3D
data. Indeed, various representations with different pros and cons have been
explored, particularly when considering 3D generative models. For instance, 3D
generative models have been explored for point clouds [2, 21], voxel grids
[42, 47, 20], meshes [51], signed distance functions (SDFs) [9, 25, 11],
_etc_. In this work, we aim to generate an SDF. Compared to other
representations, SDFs exhibit a reasonable trade-off regarding expressivity,
memory efficiency, and direct applicability to downstream tasks. Moreover,
conditioning 3D generation of SDFs on different modalities further enables
many applications, including shape completion, 3D reconstruction from images,
3D generation from text, _etc_. The proposed framework can handle these tasks
in a single model which makes it different from prior work.

Recently, thanks to the advancement of neural rendering [24], a new stream of
research has emerged to learn 3D generation and manipulation from only 2D
supervision [6, 37, 5, 28, 41, 49, 1, 39]. We believe the interplay between
two streams of work is promising in the foreseeable future.

#### Diffusion Models.

Diffusion models have recently emerged as a popular family of generative
models with competitive sample quality. In particular, diffusion models have
shown impressive quality, diversity, and expressiveness in various tasks such
as image synthesis [15, 16, 27, 13], super-resolution [36], image editing [22,
40, 10], text-to-image synthesis [26, 4, 19, 35, 33], _etc_. In contrast to
the flourishing research on diffusion models for 2D data, diffusion models
have not yet been fully explored for 3D data. Notable exceptions include
attempts to apply diffusion models to point clouds [53, 21].

Differently, in this work, we apply diffusion models on SDF representations.
As reasonable resolutions of SDFs are demanding to model, we study the use of
a latent diffusion technique [34] and the classifier-free conditional
generation mechanism [17], both of which have been shown to yield promising
results when being used in 2D diffusion models.

##  3 Approach

We aim at synthesizing 3D shapes using diffusion models. Towards this goal, we
model the distribution over 3D shapes $\mathbf{X}$, a volumetric Truncated
Signed Distance Field (T-SDF). However, applying diffusion models directly on
reasonably high-resolution 3D shapes is computationally very demanding.
Therefore, we first compress the 3D shape into a discretized and compact
latent space (Section 3.1). This allows us to apply diffusion models in a
lower-dimensional space (Section 3.2). The proposed framework can further
incorporate various user conditions such as partial shapes, images, and text
(Section 3.3). Finally, we showcase an interplay between the proposed
framework and diffusion models trained on 2D data to texture 3D shapes
(Section 3.4).

###  3.1 3D Shape Compression of SDF

A 3D shape representation is high-dimensional and thus difficult to model. To
make learning of high-resolution 3D shape distributions via diffusion models
feasible, we compress the 3D shape ${\bf X}$ into a lower-dimensional yet
compact latent space. For this, we leverage a 3D-variant of the Vector
Quantised-Variational AutoEncoder (VQ-VAE) [29]. Specifically, the employed 3D
VQ-VAE contains an encoder $E_{\varphi}$ to encode the 3D shape into the
latent space, and a decoder $D_{\tau}$ to decode the latent vectors back to 3D
space. Given an input shape represented via the T-SDF ${\bf
X}\in\mathbb{R}^{D\times D\times D}$, we have

$\displaystyle{\bf z}=E_{\varphi}({\bf X}),\quad\text{and}\quad{\bf
X}^{\prime}=D_{\tau}(\text{VQ}({\bf z})),$ (1)

where ${\bf z}\in\mathbb{R}^{d\times d\times d}$ is the latent vector, latent
dimension $d$ is smaller than 3D shape dimension $D$, and VQ is the
quantization step which maps the latent variable ${\bf z}$ to the nearest
element in the codebook $\mathcal{Z}$. The encoder $E_{\varphi}$, decoder
$D_{\tau}$, and codebook $\mathcal{Z}$ are jointly optimized. We pre-train the
VQ-VAE with reconstruction loss, commitment loss, and VQ objective, similar to
[29] using ShapeNet or BuildingNet data.

###  3.2 Latent Diffusion Model for SDF

Using the trained encoder $E_{\varphi}$, we can now encode any given SDF into
a compact and low-dimensional latent variable ${\bf z}=E_{\varphi}({\bf X})$.
We can then train a diffusion model on this latent representation.
Fundamentally, a diffusion model learns to sample from a target distribution
by reversing a progressive noise diffusion process. Given a sample ${\bf z}$,
we obtain ${\bf z}_{t},t\in\\{1,\ldots,T\\}$ by gradually adding Gaussian
noise with a variance schedule. Then we use a time-conditional 3D UNet
$\epsilon_{\theta}$ as our denoising model. To train the denoising 3D UNet, we
adopt the simplified objective proposed by Ho  _et al_. [15]:

$\displaystyle L_{\mathrm{simple}}(\theta)\coloneqq\mathbb{E}_{{\bf
z},\epsilon\sim N(0,1),t}\left[\left\lVert\epsilon-\epsilon_{\theta}({\bf
z}_{t},t)\right\rVert^{2}\right].$ (2)

At inference time, we sample $\widehat{{\bf z}}$ by gradually denoising a
noise variable sampled from the standard normal distribution $N(0,1)$, and
leverage the trained decoder $D_{\tau}$ to map the denoised code
$\widehat{{\bf z}}$ back to a 3D T-SDF shape representation $\widehat{{\bf
X}}=D_{\tau}(\widehat{\bf z})$, as shown in Figure 2.

###  3.3 Learning the Conditional Distribution

Being able to randomly sample shapes provides limited ability for interaction.
Therefore, learning of a conditional distribution is essential for user
applications. Importantly, multiple forms of conditional inputs are desirable
such that the model can account for various kinds of scenarios. Thanks to the
flexible conditional mechanism provided by a latent diffusion model [34], we
can incorporate multiple conditional input modalities at once with task-
specific encoders $E_{\phi}$ and a cross-attention module. To further allow
for more flexibility in controlling the distribution, we adopt classifier-free
guidance for conditional generation. The objective function reads as follows:

$\displaystyle
L(\theta,\\{\phi_{i}\\})\coloneqq\mathop{\mathbb{missing}}{E}_{{\bf z},{\bf
c},\epsilon,t}\left[\left\lVert\epsilon-\epsilon_{\theta}({\bf
z}_{t},t,F\\{D\circ E_{\phi_{i}}({\bf c}_{i})\\})\right\rVert^{2}\right],$ (3)

where $E_{\phi_{i}}({\bf c}_{i})$ is the task-specific encoder for the
$i^{\text{th}}$ modality, $D$ is a dropout operation enabling classifier-free
guidance, and $F$ is a feature aggregation function. In this work, $F$ refers
to a simple concatenation.

At inference time, given conditions from multiple modalities, we perform
classifier-free guidance as follows

$\displaystyle\epsilon_{\theta}({\bf z}_{t},t,F\\{E_{\phi_{i}}~{}\forall
i\\})=\epsilon_{\theta}({\bf z}_{t},t,\boldsymbol{\emptyset})+$ (4)
$\displaystyle\sum_{i}s_{i}\left(\epsilon_{\theta}({\bf
z}_{t},t,F\\{E_{\phi_{i}}({\bf c}_{i}),E_{\phi_{j}}({\bf c}_{j}):{\bf
c}_{j}=\boldsymbol{\emptyset}~{}\forall j\neq i\\})\right.$
$\displaystyle\left.-\epsilon_{\theta}({\bf
z}_{t},t,\boldsymbol{\emptyset})\right),$

where $s_{i}$ denotes the weight of conditions from the $i^{\text{th}}$
modality and $\boldsymbol{\emptyset}$ denotes a condition filled with zeros.
Intuitively, modalities with larger weights play more important roles in
guiding the conditional generation.

In this work, we study SDFusion combined with three conditional modalities
applied separately or jointly. For shape completion, given a partial
observation of a shape, we perform blended diffusion similar to [4]. For
single-view 3D reconstruction, we adopt CLIP [32] as the image encoder. For
text-guided 3D generation, we adopt BERT [12] as the text encoder. The encoded
features are then used to modulate the diffusion process with cross-attention.

Figure 3:  3D Shape Texturing. We demonstrate an application where models
trained on 2D and 3D data are combined. The shapes generated by SDFusion are
converted to a density tensor, then the color information is learned via
neural rendering. The gradients are provided by an off-the-shelf 2D diffusion
model [34].  Figure 4:  Shape Completion. (Top) We compare SDFusion with
AutoSDF [25] and MPC [45] on ShapeNet and BuildingNet data. SDFusion generates
shapes of better quality and diversity, while being consistent with the input
partial shapes. We convert the generated SDFs from SDFusion and AutoSDF to
point clouds to compare with MPC. (Bottom) We present more results on diverse
shape completion using various object categories.

###  3.4 3D Shape Texturing with a 2D Model

While sampled 3D data often exhibits compellingly detailed geometry, textures
of 3D data are generally more difficult to collect and even more often of
limited quality. Here, we explore how to make the best use of 2D data and
models, so as to aid 3D asset generation. Thanks to the recent success of
neural rendering [24] and 2D text-to-image models trained on extremely large-
scale data [34], using a 2D model to perform 3D synthesis is made possible
with a score distillation sampling technique [31].

Here, we illustrate the procedure of texturing a 3D shape given a guiding
input sentence $S$. Starting from a generated T-SDF $\mathbf{X}$, we first
convert it to a density tensor $\sigma$ by using VolSDF [50]. Our goal is to
learn a 5D radiance field to obtain color
$F_{\theta}:(\mathbf{x},\mathbf{d})\rightarrow\mathbf{c}$, similar to the
conventional NeRF [24] setting. However, different from the conventional NeRF
setting, the density tensor is fixed. With a sampled camera pose, we can
render an image $I$ by alpha-compositing the densities and colors along the
rays for each pixel. Then, we distill the knowledge from a pre-trained stable
diffusion model [34], denoted as a 2D UNet $\tilde{\epsilon}_{\phi}({\bf
z}_{t},t,S)$ that predicts the noise at timestep $t$. The most straightforward
way to update $F_{\theta}$ is to obtain and backpropagate gradients all the
way from the noise prediction loss term of the stable diffusion loss function
to the input. However, the UNet Jacobian term is in practice expensive to
compute. Therefore, it was proposed in [31] to bypass the UNet and treat
$\tilde{\epsilon}_{\phi}$ as a frozen critic providing scores by computing

$\mathbb{E}_{t,\epsilon}\left[\frac{\partial
I(F_{\theta}(\cdot))}{\partial\theta}w(t)(\tilde{\epsilon}_{\phi}({\bf
z}_{t},t,S)-\epsilon)\right],$ (5)

where $w$ is a time-dependent weight function defined in the stable diffusion
model. The mechanism is called Score Distillation Sampling. Please refer to
[31] for details. The score then provides an update direction to $F_{\theta}$.
We illustrate the process in Figure 3.

##  4 Experiments

In this section, we conduct extensive qualitative and quantitative experiments
to demonstrate the efficacy and generalizability of SDFusion. We evaluate
methods on three tasks: shape completion, single-view 3D reconstruction, and
text-guided generation. We then demonstrate two additional use cases: multi-
conditional generation and 3D shape texturing.

###  4.1 Shape Completion

Table 1:  Quantitative comparison of Shape Completion. We evaluate methods on fidelity (UHD) and diversity (TMD) using the ShapeNet and BuildingNet data. SDFusion outperforms other methods in both metrics, especially diversity.  | ShapeNet | BuildingNet
---|---|---
Method | UHD $\shortdownarrow$ | TMD $\shortuparrow$ | UHD $\shortdownarrow$ | TMD $\shortuparrow$
MPC [45] | 0.0627 | 0.0303 | 0.1350 | 0.0467
AutoSDF [25] | 0.0567 | 0.0341 | 0.1208 | 0.0649
Ours | 0.0557 | 0.0885 | 0.1116 | 0.0745
Table 2:  Quantitative evaluation of single-view reconstruction. We evaulate methods on the Pix3D dataset using Chamfer Distance and F-Score. We outperform other methods in both metrics.  Method | CD $\shortdownarrow$ | F-Score $\shortuparrow$
---|---|---
Pix2Vox [46] | 3.001 | 0.385
ResNet2TSDF | 4.582 | 0.351
ResNet2Voxel | 4.670 | 0.357
AutoSDF [25] | 2.267 | 0.415
Ours | 1.852 | 0.432
Figure 5:  Single-view 3D Reconstruction. (Top) We qualitatively compare all
methods on the Pix3D dataset. SDFusion generates shapes with the best visual
quality. (Bottom) Here we present more reconstruction results from SDFusion.

We evaluate the shape completion task on the ShapeNet [7] and the BuildingNet
[38] datasets. ShapeNet is a large-scale 3D CAD model dataset with 16 common
object classes. We use the train/test splits provided by Xu _et al_. [48].
BuildingNet is a new large-scale 3D building model dataset. Compared to
objects in ShapeNet, building models provide more geometric details and thus
require higher resolution for representing the data. Hence, we use an SDF of
$64^{3}$ resolution for ShapeNet, and a $128^{3}$ resolution for BuildingNet.
For both datasets, we compare the completion quantitatively by providing the
bottom half of the ground truth shape as input, and evaluating the completed
shapes generated by different methods.

Figure 6:  Text-guided 3D shape generation. (Top) We compare SDFusion with
AutoSDF on the Text2Shape dataset. SDFusion generates shapes with higher
quality while conforming to the description. (Bottom) We present more diverse
and interesting text-guided generation results from SDFusion.

We compare SDFusion to the state-of-the-art point cloud completion method MPC
[45] and the autoregressive SDF generation method AutoSDF [25]. We adopt
metrics from MPC [45]. For each partial shape, we generate $k$ complete
shapes. To evaluate completion fidelity, we measure the Unidirectional
Hausdorff Distance (UHD) between partial shapes and generated shapes. To
evaluate completion diversity, we measure the Total Mutual Difference (TMD) by
computing the average Chamfer distance among $k$ generated shapes. We use
$k=10$ in the experiments.

As shown in Table 1, the proposed SDFusion performs favorably compared to all
methods in the completion fidelity metric, and outperforms the baselines
substantially in completion diversity. The advantages of fidelity and
diversity are also apparent in Figure 4. Especially on the BuildingNet
dataset, SDFusion shows its advantages in modeling high-resolution and diverse
data. AutoSDF and MPC struggle to model the distribution correctly.

Table 3:  Quantitative evaluation of text-guided generation. We compare the proposed SDFusion method, AutoSDF, and real data using a pretrained neural evaluator on the ShapeGlot dataset. $P$ denotes the neural evaluator preference rate for the target $P(\text{Tr})$ or the distractor $P(\text{Dis})$. If the preference is too close ($\leq 0.2$), we count the comparison as confused (conf.). Rows sum to $100\%$.  Target (Tr) | Distractor (Dis) | $P(\text{Tr})$ | $P(\text{Dis})$ | $P(\text{conf.})$
---|---|---|---|---
Ours |  AutoSDF [25] | 49% | 36% | 15%
Ours | GT | 33% | 45% | 22%
AutoSDF [25] | GT | 30% | 49% | 21%
Figure 7:  Conditional generation from multiple modalities. Here, we present
generated samples from joint conditions of (left) partial shape and text and
(right) partial shapes and images. The generated results are diverse and
consistent with the provided conditions.

###  4.2 Single-view 3D Reconstruction

Next, we assess 3D shape reconstruction from a single image on the real-world
benchmark Pix3D [43] dataset. We use the provided train/test splits on the
chair category. In the absence of official splits for other categories, we
randomly split the dataset into disjoint train/test splits.

We compare with the ResNet2TSDF and ResNet2Voxel baselines. Both encode images
and directly output 3D shapes in the form of a T-SDF and a voxel grid. We also
compare to two state-of-the-art methods for 3D reconstruction, _i.e_., Pix2Vox
[46] and AutoSDF [25] . We evaluate all methods after aligning resolutions to
$32^{3}$ voxels. We use Chamfer Distance (CD), and F-score$@1\%$ [44] as
evaluation metrics.

Quantitatively, as shown in Table 2, SDFusion outperforms other methods on
both metrics. Qualitatively, SDFusion generates 3D shapes that are of higher
visual quality and are more visually consistent with the objects shown in the
images, regardless of camera poses, as shown in Figure 5.

Figure 8:  Multiple conditioning variables with weight control. Given a
partial shape, an image, and a sentence as conditional input, we show that the
model is sensitive to weights that control the importance of different
modalities.  Figure 9:  3D Shape Texturing. We texture the generated 3D shapes
with a 2D diffusion model trained on large-scale data. This permits to
generate textures from diverse textual inputs, including style and material
descriptions. The pipeline can also generate diverse results given the same
input description.

###  4.3 Text-guided Generation

Next, we evaluate 3D shape generation conditioned on text input. For a
qualitative comparison, we use the Text2shape dataset [8] that provides
descriptions for the ‘chair’ and ‘table’ categories in ShapeNet. For a
quantitative evaluation, we adopt the ShapeGlot [3] dataset which provides
text utterances describing the difference between a target shape and two
distractors based on the ShapeNet dataset. We compare SDFusion with AutoSDF,
which recently demonstrated state-of-the-art results on the text-guided 3D
shape generation task.

We follow the evaluation pipeline proposed by ShapeGlot [3]. We train a neural
evaluator to distinguish the target shape from a distractor given the
description. Given two shapes from different methods, the neural evaluator
provides a confidence score for each of them based on the binary
classification logits. For the absolute difference between two confidence
scores $\leq 0.2$, we count the comparison as confused (conf.).

As shown in Table 3, SDFusion quantitatively outperforms AutoSDF by a large
margin with a low confusion rate. SDFusion also performs better than AutoSDF
when compared with ground truth data. Qualitatively, we show in Figure 5 that
SDFusion not only generates shapes with better quality, but that the generated
shapes are also more diverse. Notably, SDFusion reacts to very specific
descriptions like “L-shaped table” and “table with two surfaces.” It generates
objects of high diversity while remaining faithful to the provided
description.

###  4.4 Multi-conditional Generation

In addition to the conditional generation tasks which take a single
conditioning variable as input, we further demonstrate the efficacy of
SDFusion in handling multiple modalities. First, SDFusion can jointly consider
multiple conditioning modalities. On the left of Figure 7, we present the
diverse generation conditional on both partial shapes and text. On the right
of Figure 7, we show that given partial shapes and images, SDFusion can
complete the different parts based on images. When there is ambiguity in
images (_e.g_., rear-view of chairs), SDFusion can produce diverse
predictions. Second, as shown in Figure 8, SDFusion can not only be jointly
conditioned on multiple inputs, but a weight can be used to control the
importance of the conditioning modalities, enabling more flexible user
control. For example, for the left sample in Figure 8, the larger the weight
for the input image, the more similar the results are to the shapes in the
image. Similarly, the larger the weight for input text, the more “egg-shaped”
the results. We envision such a fine-grained form of control to be
particularly useful for interactive user applications.

###  4.5 3D Shape Texturing

Finally, we showcase an application that uses SDFusion to generate 3D shapes
of detailed geometry, and uses a pretrained text-to-image 2D diffusion model
[34] to provide textures. As shown in Figure 9, the diffusion model pre-
trained on large-scale 2D data can provide semantically meaningful and diverse
guidance to texture the 3D shapes. The model shows superior expressiveness to
interpret abstract concepts (_e.g_., Chinese- and Halloween-style) and
materials (_e.g_., gingerbread, cream cheese). Given a single description, the
texturing pipeline can also generate diverse results, as shown in the
rightmost part of Figure 9.

##  5 Conclusion

In this work, we present SDFusion, an attempt to adopt diffusion models to
signed distance functions for 3D shape generation. To alleviate the
computationally demanding nature of 3D representations, we first encode 3D
shapes into an expressive low-dimensional latent space, which we use to train
the diffusion model. To enable flexible conditional usage, we adopt class-
specific encoders along with a cross-attention mechanism for handling
conditions from multiple modalities, and leverage classifier-free guidance to
facilitate weight control among modalities. Foreseeing the potential of a
collaborative symbiosis between models trained on 2D and 3D data, we further
demonstrate an application that takes advantage of a pretrained 2D text-to-
image model to texture a generated 3D shape.

Although the results look promising and exciting, there are quite a few future
directions for improvement. First, SDFusion is trained on high-quality signed
distance function representations. To make the model more general and to
enable the use of more diverse data, a model that operates on various 3D
representations simultaneously is desirable. Another future direction is
related to the diversity of the data: we currently apply SDFusion on object-
centric data. It is interesting to apply the model to more challenging
scenarios (_e.g_., entire 3D scenes). Finally, we believe there is room to
further explore how to combine models trained on 2D and 3D data.

Acknowledgements: Work supported in part by NSF under Grants 2008387, 2045586,
2106825, MRI 1725729, and NIFA award 2020-67021-32799. Thanks to NVIDIA for
providing a GPU for debugging.

## References

  * [1] Rameen Abdal, Hsin-Ying Lee, Peihao Zhu, Menglei Chai, Aliaksandr Siarohin, Peter Wonka, and Sergey Tulyakov.  3davatargan: Bridging domains for personalized editable avatars.  In CVPR, 2023.
  * [2] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas.  Learning representations and generative models for 3d point clouds.  In ICML, 2018.
  * [3] Panos Achlioptas, Judy Fan, Robert Hawkins, Noah Goodman, and Leonidas J Guibas.  Shapeglot: Learning language for shape differentiation.  In CVPR, 2019.
  * [4] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In CVPR, 2022.
  * [5] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In CVPR, 2022.
  * [6] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In CVPR, 2021.
  * [7] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  * [8] Kevin Chen, Christopher B Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese.  Text2shape: Generating shapes from natural language by learning joint embeddings.  In ACCV, 2018.
  * [9] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In CVPR, 2019.
  * [10] Shin-I Cheng, Yu-Jie Chen, Wei-Chen Chiu, Hsin-Ying Lee, and Hung-Yu Tseng.  Adaptively-realistic image generation from stroke and sketch with diffusion model.  In ACCV, 2022.
  * [11] Zezhou Cheng, Menglei Chai, Jian Ren, Hsin-Ying Lee, Kyle Olszewski, Zeng Huang, Subhransu Maji, and Sergey Tulyakov.  Cross-modal 3d shape generation and manipulation.  In ECCV, 2022.
  * [12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In NAACL, 2019.
  * [13] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  NeurIPS, 2021.
  * [14] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt.  Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis.  In ICLR, 2022.
  * [15] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  NeurIPS, 2020.
  * [16] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.  Cascaded diffusion models for high fidelity image generation.  arXiv preprint arXiv:2106.15282, 2021.
  * [17] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  arXiv preprint arXiv:2207.12598, 2022.
  * [18] Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker.  Sdfdiff: Differentiable rendering of signed distance fields for 3d shape optimization.  In CVPR, 2020.
  * [19] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  arXiv preprint arXiv:2210.09276, 2022.
  * [20] Chieh Hubert Lin, Hsin-Ying Lee, Willi Menapace, Menglei Chai, Aliaksandr Siarohin, Ming-Hsuan Yang, and Sergey Tulyakov.  Infinicity: Infinite-scale city synthesis.  arXiv preprint arXiv:2301.09637, 2023.
  * [21] Shitong Luo and Wei Hu.  Diffusion probabilistic models for 3d point cloud generation.  In CVPR, 2021.
  * [22] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Image synthesis and editing with stochastic differential equations.  In ICLR, 2022.
  * [23] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In CVPR, 2019.
  * [24] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [25] Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani.  AutoSDF: Shape priors for 3d completion, reconstruction and generation.  In CVPR, 2022.
  * [26] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [27] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In ICML, 2021.
  * [28] Michael Niemeyer and Andreas Geiger.  Giraffe: Representing scenes as compositional generative neural feature fields.  In CVPR, 2021.
  * [29] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.  Neural discrete representation learning.  In NeurIPS, 2017.
  * [30] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In CVPR, 2019.
  * [31] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  arXiv preprint arXiv:2209.14988, 2022.
  * [32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [33] Tanzila Rahman, Hsin-Ying Lee, Jian Ren, Sergey Tulyakov, Shweta Mahajan, and Leonid Sigal.  Make-a-story: Visual memory conditioned consistent story generation.  In CVPR, 2022.
  * [34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * [35] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [36] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  IEEE TPAMI, 2022.
  * [37] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  Graf: Generative radiance fields for 3d-aware image synthesis.  In NeurIPS, 2020.
  * [38] Pratheba Selvaraju, Mohamed Nabail, Marios Loizou, Maria Maslioukova, Melinos Averkiou, Andreas Andreou, Siddhartha Chaudhuri, and Evangelos Kalogerakis.  Buildingnet: Learning to label 3d buildings.  In ICCV, 2021.
  * [39] Aliaksandr Siarohin, Willi Menapace, Ivan Skorokhodov, Kyle Olszewski, Jian Ren, Hsin-Ying Lee, Menglei Chai, and Sergey Tulyakov.  Unsupervised volumetric animation.  In CVPR, 2023.
  * [40] Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon.  D2c: Diffusion-decoding models for few-shot conditional generation.  NeurIPS, 2021.
  * [41] Ivan Skorokhodov, Aliaksandr Siarohin, Yinghao Xu, Jian Ren, Hsin-Ying Lee, Peter Wonka, and Sergey Tulyakov.  3d generation on imagenet.  In ICLR, 2023.
  * [42] Edward J Smith and David Meger.  Improved adversarial systems for 3d object generation and reconstruction.  In CoRL, 2017.
  * [43] Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B Tenenbaum, and William T Freeman.  Pix3d: Dataset and methods for single-image 3d shape modeling.  In CVPR, 2018.
  * [44] Maxim Tatarchenko*, Stephan R. Richter*, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox.  What do single-view 3d reconstruction networks learn?  In CVPR, 2019.
  * [45] Rundi Wu, Xuelin Chen, Yixin Zhuang, and Baoquan Chen.  Multimodal shape completion via conditional generative adversarial networks.  In ECCV, 2020.
  * [46] Haozhe Xie, Hongxun Yao, Xiaoshuai Sun, Shangchen Zhou, and Shengping Zhang.  Pix2vox: Context-aware 3d reconstruction from single and multi-view images.  In CVPR, pages 2690–2698, 2019.
  * [47] Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, and Ying Nian Wu.  Learning descriptor networks for 3d shape synthesis and analysis.  In CVPR, 2018.
  * [48] Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann.  Disn: Deep implicit surface network for high-quality single-view 3d reconstruction.  In NeurIPS, 2019.
  * [49] Yinghao Xu, Menglei Chai, Zifan Shi, Sida Peng, Ivan Skorokhodov, Aliaksandr Siarohin, Ceyuan Yang, Yujun Shen, Hsin-Ying Lee, Bolei Zhou, et al.  Discoscene: Spatially disentangled generative radiance fields for controllable 3d-aware scene synthesis.  In CVPR, 2023.
  * [50] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman.  Volume rendering of neural implicit surfaces.  NeurIPS, 2021.
  * [51] Song-Hai Zhang, Yuan-Chen Guo, and Qing-Wen Gu.  Sketch2model: View-aware 3d modeling from single free-hand sketches.  In CVPR, 2021.
  * [52] X. Zhao, F. Ma, D. Güera, Z. Ren, A. G. Schwing, and A. Colburn.  Generative Multiplane Images: Making a 2D GAN 3D-Aware.  In ECCV, 2022.
  * [53] Linqi Zhou, Yilun Du, and Jiajun Wu.  3d shape generation and completion through point-voxel diffusion.  In ICCV, 2021.


# A "network of networks" (from history to algebra)

Daniel Parrochia

(University of Lyon (France))

Abstract.

Recall first the algebraic treatment of flows or tensions in a transportation
network $N$, i.e. a connected antisymmetric 1-graph $G(X,U)$. Assume that,
unusually, we take the values of flows (resp. tensions) in $\mathbb{C}$. So
the algebraic lattices $\Gamma$ of flow (resp. tension) values associated to
$G(X,U)$ are lattices of $\mathbb{C}$. These lattices are congruent modulo the
action of the special linear group SL($2,\mathbb{C}$). Then, it is well known
one can define a lattice function $G_{k}(\Gamma)$, as a modular function of
weight $2k$, on the set $\mathcal{R}$ of all lattices of $\mathbb{C}$. Let now
$N_{1},N_{2},...,N_{p}$ be connected antisymmetric 1-graphs and $C_{n}$, the
set of hermitian symmetric matrices $n\times n$. Let also
$\mathcal{R^{\prime}}$ be the set of all the lattices of $C_{n}$. The previous
structure can be transposed to any $n\times n$ symmetric hermitian matrices of
flow (or tension) values of the $G_{i}$. In this case, the Siegel space
$S_{n}=C_{n}$ replaces the Poincaré half-plane, and the symplectic group
Sp$(2n,\mathbb{R})$ takes the place of the special linear group
SL($2,\mathbb{C}$). We get now the new lattice function as a function of all
the lattices of $S_{n}$, i.e. a model of the "network of networks"
$\mathcal{R^{\prime}}$. In the end, we study the tree of minimal length of
$\mathcal{R^{\prime}}$.

Key words. network, flows, tensions, lattices, lattice functions, modular
functions, network algebra.

##  1 Introduction

Consider a connected 1-graph $G$ whose arcs are denoted by 1, 2, …, $m$ and
let some quantities $b_{i},c_{i}$ be such that

$-\infty\leq b_{i}\leq c_{i}\leq+\infty$

with the conditions:

1) $b_{i}=0\qquad(i=1,2,...,m)$;

2) $c_{i}\geq 0\qquad$ for all $i$, and $c_{i}=+\infty$;

3) Arc $i$ = 1 is the arc $(b,a)$ which connects a point $b$ named the output
with a point $a$ named the input, these two points verifying:

$\omega^{-}(a)=(1,0,0,...,0),$ $\omega^{+}(a)=(1,0,0,...,0);$

4) $G$ is an antisymmetric 1-graph.

The arc 1 = ($b,a$), that will not be drawn, is named the return arc and is
just introduced to maintain the Kirchoff law at the vertices $a$ and $b$.

######  Definition 1.1.

A graph $G$, with a capacity $c_{i}$ associated to any arc $i$, and which
satisfies all these conditions, is called a transportation network (see
[3])111Historically, before Ford and Fulkerson, it seems that interest for
combinatorial optimization may be found in an article of A. N. Tolstoĭ from
1930, in which the transportation problem is studied, as well as an, until
recently secret, RAND report of T. E. Harris and F. S. Ross from 1955, that
Ford and Fulkerson mention as motivation to study the maximum flow problem.
These papers have in common that they both apply their methods to the Soviet
railway network. As Schrijver recalled, the transportation problem was
formulated by ([8], and a cycle criterion for optimality was considered by
[9], [10], [12], [13], [19], [20] [5], [6], [14], [4] and [11]. On all that,
see [21]. and it will be denoted by:

$N=(X,U,c(u)).$

In the following, as we will not pay attention to capacities, the previous
network will be reduced to a connected 1-graph $G$.

##  2 Flows and tensions in networks

######  Definition 2.1.

A flow in a connected graph $G$ is usually defined as a vector
$\phi=(\phi_{1},\phi_{2},...,\phi_{m})\in\mathbb{Z}^{m}$ such that:

(1) $\phi_{i}\in\mathbb{Z}$ for $i=1,2,...,m$. (The integer $\phi_{i}$ is
called an arc flow and may be regarded as the number of vehicules (signals,
etc.) travelling through arc $i$ along its direction if $\phi_{i}\geq 0$ or
against its direction if $\phi_{i}<0$.)

(2) For each vertex $x$, the sum of the arc flows entering $x$ equals the sum
of the arc flows leaving $x$ (Kirchoff law), i.e.,

$\sum_{i\in\omega^{-}(x)}\phi_{i}=\sum_{j\in\omega^{+}(x)}\phi_{j}\qquad(x\in
X).$

According to Berge (see [1], 85), it is possible to develop an algebraic study
of flows in such a graph.

Firstable, as $\mathbb{Z}^{m}$ is a module on $\mathbb{Z}$ (not a vector
space, because $\mathbb{Z}$ is not a field), the set $\Phi$ of all flows in
the graph $G$ constitutes a submodule of $\mathbb{Z}^{m}$, i.e we have:

$\phi^{1},\phi^{2}\in\Phi\Rightarrow\phi^{1}+\phi^{2}\in\Phi,$
$s\in\mathbb{Z},\phi\in\Phi\Rightarrow s\phi\in\Phi.$

Berge proves the following theorem:

######  Theorem 2.1.

Let $G=(X,U)$ a connected graph; $H=(X,V)$ an arbitrary tree of $G$; 1, 2, …,
$k$, the arcs of $U-V$; $\mu^{1},\mu^{2},...,\mu^{k}$ the cycles associated
with $H$. A flow $\phi$ is uniquely defined by its values
$\phi_{1},\phi_{2},...,\phi_{k}\in U-V$ by:

$\phi=\phi_{1}\vec{\mu}^{1}+\phi_{2}\vec{\mu}^{2}+...+\phi_{k}\vec{\mu}^{k},$

where the $\phi_{i}$ are scalars and the $\vec{\mu}^{i}$ are vectors
associated with independent elementary cycles.

This means that a flow is uniquely defined by its components on a cotree of
$G$.

Let now come to tensions.

######  Definition 2.2.

A tension (or potential difference) in a connected graph $G$ is defined to be
a vector $\theta=(\theta_{1},\theta_{2},...,\theta_{m})\in\mathbb{Z}^{m}$ such
that, for each elementary cycle $\mu$,

$\sum_{i\in\mu^{+}}\theta_{i}=\sum_{i\in\mu^{-}}\theta_{i}.$

For every arc $i$, we have: $\theta_{i}=t$ (terminal end of arc $i$) - $t$
(initial end of arc $i$).

Let $\Theta$ denote the set of all tensions. Note that $\Theta$ is also a
submodule of $\mathbb{Z}^{m}$, i.e.,

$\theta^{1},\theta^{2}\in\Theta\Rightarrow\theta^{1}+\theta^{2}\in\Theta,$
$s\in\mathbb{Z},\theta\in\Theta\Rightarrow s\theta\in\Theta.$

Here again, Berge proves the following theorem:

######  Theorem 2.2.

Let $G=(X,U)$ a connected graph; $H=(X,V)$ an arbitrary tree of $G$; 1, 2, …,
$k$, the arcs of this tree;
$\vec{\omega}^{1},\vec{\omega}^{2},...,\vec{\omega}^{\ell}$ the cocycles
associated with $H$. A tension $\theta$ is uniquely defined by its values
$\theta_{1},\theta_{2},...,\theta_{\ell}$ on the arcs of the tree by:

$\theta=\theta_{1}\vec{\omega}^{1}+\theta_{2}\vec{\omega}^{2}+...+\theta_{\ell}\vec{\omega}^{\ell},$

where the $\theta_{i}$ are scalars and the $\vec{\omega}^{i}$ are vectors
associated with independent elementary cocycles.

This means that a tension is uniquely defined by its components on a tree of
$G$.

We can easily see that $\Theta$ and $\Phi$ are two orthogonal submodules of
$\mathbb{Z}^{m}$, which means that, for every elementary cycle $\mu$, we have:

$\langle\phi,\theta\rangle=\sum_{i=1}^{m}\phi_{i}\theta_{i}=0.$

##  3 Algebraic lattices

We propose to extend the previous model. Let us consider now the set of all
possible values of tensions or flows in some network $N$. We will prove that
this set can be associated to a metanetwork $G_{k}(\Gamma)$ which satisfies
good properties. Recall first the following definition:

######  Definition 3.1.

A lattice $\Gamma$, in an $\mathbb{R}$-vector space $V$ of finite dimension,
is a subgroup of $V$ verifying one of the following equivalent conditions
enumerated by Serre (see [22], 133):

1) $\Gamma$ is discrete and $V/\Gamma$ is compact;

2) $\Gamma$ is discrete and generates the $\mathbb{R}$-vector space $V$;

3) There exists an $\mathbb{R}$-basis $\\{e_{1},...e_{n}\\}$ of $V$, which is
a $\mathbb{Z}$-basis of $\Gamma$ and
$\Gamma=\mathbb{Z}e_{1}\oplus...\oplus\mathbb{Z}e_{n}$.

Now, let us choose values of flows (or tensions) in an $\mathbb{R}$-vector
space $V=\mathbb{R}^{n}$.

######  Theorem 3.1.

The set of all possible flow (resp. tension) values of the network $N$ is a
lattice in $\mathbb{R}^{n}$.

###### Proof.

Let $\epsilon=(\epsilon_{1},\epsilon_{2},...,\epsilon_{n})$, a flow (resp. a
tension) in some arc(s) of $G$. By definition, $\epsilon$ belongs to
$\mathbb{R}^{n}$, viewed as a vector space on $\mathbb{R}$. Moreover,
according to the definition of flows (Def. 2.1) and of tensions (Def. 2.2),
the set $\Gamma$, of all flow (resp. tension) values in the graph $G$, is the
subgroup of all linear combinations with integer coefficients of the basis
vectors of $\mathbb{R}^{n}$ (cycles, resp. cocyles). So it is such that:

$\Gamma=\mathbb{Z}\epsilon_{1}\oplus...\oplus\mathbb{Z}\epsilon_{n},$

for any basis of $\mathbb{R}^{n}$. In other words, it forms a lattice in
$\mathbb{R}^{n}$. ∎

##  4 The lattices of $\mathbb{C}$

Assume now that the flow (resp. tension) values of $G$ are in $\mathbb{C}$,
and consider only two-valued flows (resp. tensions).

Let us call $\mathcal{R}$ the set of lattices of $\mathbb{C}$, considered as
an $\mathbb{R}$-vector space, and let us now choose a pair of flow (resp.
tension) values ($\alpha_{1},\alpha_{2})\in\mathbb{C}^{*}$ so that
Im($\alpha_{1}/\alpha_{2})>0$. $M$ will be the set of these pairs.

To such a pair ($\alpha_{1},\alpha_{2})$, we associate the lattice:

$\Gamma(\alpha_{1},\alpha_{2})=\mathbb{Z}\alpha_{1}\oplus\mathbb{Z}\alpha_{2}.$

with basis $\\{\alpha_{1},\alpha_{2}\\}$.

Thus we get a map $M\rightarrow\mathcal{R}$, which is clearly surjective.

Now let:

$g=\begin{pmatrix}a&b\\\ c&d\end{pmatrix}\in\textnormal{SL}(2,\mathbb{Z})$

the special linear group of square matrices $2\times 2$ with relative
coefficients, and let $(\alpha_{1},\alpha_{2})\in M.$ One proves (see [22],
134) the following theorem:

######  Theorem 4.1.

For two elements of $M$ to define the same lattice, it is necessary and
sufficient that they are congruent modulo SL($2,\mathbb{Z}$).

###### Proof.

(Serre) The condition is sufficient. Let us put:

$\alpha^{\prime}_{1}=a\alpha_{1}+b\alpha_{2}\ \textnormal{and}\
\alpha^{\prime}_{2}=c\alpha_{1}+d\alpha_{2}.$

Il is clear that $\\{\alpha^{\prime}_{1},\alpha^{\prime}_{2}\\}$ is a basis of
$\Gamma(\alpha_{1},\alpha_{2})$. Moreover, if the set
$z=\alpha_{1}/\alpha_{2}$ and
$z^{\prime}=\\{\alpha^{\prime}_{1}/\alpha^{\prime}_{2}\\}$, we have:

$z^{\prime}=\frac{az+b}{cz+d}=gz.$

This shows that Im$(z^{\prime})>0$, hence that
$(\alpha^{\prime}_{1},\alpha^{\prime}_{2})$ belongs to $M$.

Conversely, if $(\alpha_{1},\alpha_{2})$ and
$(\alpha^{\prime}_{1},\alpha^{\prime}_{2})$ are two elements of $M$ which
define the same lattice, there exists an integer matrix

$g=\begin{pmatrix}a&b\\\ c&d\end{pmatrix}$

of determinant $\pm 1$ which transforms the first basis into the second. If
det($g$) was $<0$, the sign of Im$(\alpha^{\prime}_{1}/\alpha^{\prime}_{2})$
would be the opposite of Im$(\alpha_{1}/\alpha_{2})$ as one sees by an
immediate computation. The two signs being the same, we have necessarily
det($g$) = 1, which proves the theorem. ∎

Hence we can identify the set $\mathcal{R}$ of all the lattices of
$\mathbb{C}$ (which are, for us, sets of flow (or tension) values associated
to connected 1-graphs (or networks) with the quotient of $M$ by the action of
SL($2,\mathbb{Z}$).

##  5 Modular functions

Let now $F$ be a function on $\mathcal{R}$, with complex values, and let
$k\in\~{A}\"{a}\mathbb{Z}$. We say (with Serre) that $F$ is of weight $2k$ if:

$F(\lambda\Gamma)=\lambda^{-2k}F(\Gamma),$ (1)

for all lattices $\Gamma$ and all $\lambda\in\mathbb{C}^{*}$.

Let $F$ be such a function. If $(\alpha_{1},\alpha_{2})\in M$, we denote by
$F(\alpha_{1},\alpha_{2})$ the value of $F$ on the lattice
$\Gamma(\alpha_{1},\alpha_{2})$. The formula (1) translates to:

$F(\lambda\alpha_{1},\lambda\alpha_{2})=\lambda^{-2k}F(\alpha_{1},\alpha_{2}).$
(2)

Writing that $F$ is invariant by SL(2, $\mathbb{Z})$, we can see that it
satisfies the identity:

$F(z)=(cz+d)^{-2k}F(\frac{az+b}{cz+d}),$ (3)

for all:

$\begin{pmatrix}a&b\\\ c&d\end{pmatrix}\in\textnormal{SL(}2,\mathbb{Z}).$

Conversely, if $F$ verifies (3), $F$ is a function on $\mathcal{R}$ which is
of weight $2k$. We can thus identify modular functions of weight 2k with some
lattice functions of weight 2k.

Then we know that some lattice functions, that are modular functions, can be
identified with Eisenstein series, which are themselves convergent. Serre
(1973) proves the following lemma:

######  Lemma 5.1.

Let $\Gamma$ be a lattice in $\mathbb{C}$. The series:

$\sum^{\prime}_{\gamma\in\Gamma}1/|\gamma|^{\sigma}$

is convergent for $\sigma>2$.

(The symbol $\sum^{\prime}$ signifies that the summation runs over the nonzero
elements of $\Gamma$.)

Now let $k$ be an integer >1\. If $\Gamma$ is a lattice of $\mathbb{C}$, put:

$G_{k}(\Gamma)=\sum_{\gamma\in\Gamma}^{\prime}1/\gamma^{2k}.$

This series converges absolutely thanks to the preceding lemma. It proves the
existence of a lattice function on the set $\mathcal{R}$ of lattices of
$\mathbb{C}$.

In other words, all the lattices of $\mathbb{C}$, which represent sets of flow
(or tension) values in connected 1-graphs (or networks) are themselves
connected by this lattice function.

Let now $k$ be an integer >1\. Like all the Eisentein series of the type
$G_{k}(z)$:

1) $G_{k}(\Gamma)$, which is a modular form of weight $2k$, is holomorphic
everywhere (including at the infinite);

2) $G_{k}(\infty)=2\zeta(2k)$;

3) $G_{k}$ has a limit for Im$(z)\rightarrow\infty,\ z$ being the value for
which $\Gamma$ vanishes at one and only one point.

##  6 Siegel space

We can still extend the previous construction.

Let $N_{1},N_{2},...,N_{m}$ be some finite connected 1-graphs and consider,
for each of them, their associated matrices of flow (or tension) values. Let
$Z_{1},Z_{2},...,Z_{m}$ be such matrices with complex coefficients.

Let $L$ be the set of all $n\times n$ complex symmetric matrices and $C_{n}$
the set of matrices $Z$ of $L$ such that the hermitian matrix $I-Z\bar{Z}$ is
strictly positive.

Let now $S_{n}$ (the Siegel space) be the set of matrices $Z$ of $L$ whose
imaginary part Im$\ Z=(1/2i)(Z-\bar{Z})$ is strictly positive. It is well
known that the so-called "Cayley transformations" apply $C_{n}$ to $S_{n}$ and
vice versa (see [2], 437-438).

Hence, the real symplectic group Sp$(2n,\mathbb{R}$) plays the same role, with
respect to the Siegel space $S_{n}$, than the group Sp(2, $\mathbb{R})$ =
SL(2, $\mathbb{R}$) with respect to the upper half-plane of the complex plane.
When the group SL(2, $\mathbb{R}$) operates in $\mathbb{C}$ by the Poincaré
Fuchsian transformations, the group Sp$(2n,\mathbb{R}$) now operates in the
Siegel space $S_{n}$ by the transformations:

$g^{\prime}=\begin{pmatrix}A&B\\\
C&D,\end{pmatrix}\in\textnormal{Sp}(2n,\mathbb{R}).$ (4)

So we have:

$g^{\prime}Z=(AZ+B)(CZ+D)^{-1}.$

Now let us call $\mathcal{R}^{\prime}$ the set of all the matrix lattices of
$C_{n}$, and let $M^{\prime}$ be the set of pairs ($A_{1},A_{2})\in C_{n}$,
such that Im$(A_{1},A_{2}^{-})>0$, which supposes that $A_{2}$ is inversible.

To such a pair ($A_{1},A_{2})$, we associate now the lattice:

$\Gamma^{\prime}(A_{1},A_{2})=\mathbb{Z}A_{1}\oplus\mathbb{Z}A_{2}.$

with basis $\\{A_{1},A_{2}\\}$. Thus, we get a map $M^{\prime}\rightarrow
R^{\prime}$, which is clearly surjective.

One gets the following theorem:

######  Theorem 6.1.

So that two elements of $M^{\prime}$ define the same lattice, it is necessary
and sufficient that they are congruent modulo Sp($2n,\mathbb{R}$).

###### Proof.

The condition is sufficient. Let $A_{1},A_{2}\in M^{\prime}$. Then, put :

$A^{\prime}_{1}=aA_{1}+bA_{2}\ \textnormal{and}\
A^{\prime}_{2}=cA_{1}+dA_{2}.$

Il is clear that $\\{A^{\prime}_{1},A^{\prime}_{2}\\}$ is a basis of
$\Gamma(A_{1},A_{2})$. Moreover, if $Z=A_{1}A_{2}^{-}$ and
$Z^{\prime}=A^{\prime}_{1}A^{\prime-}_{2}$,

$Z^{\prime}=(AZ+B)(CZ+D)^{-}=g^{\prime}Z.$

This shows that Im$(Z^{\prime})>0$, hence that
$(A^{\prime}_{1},A^{\prime}_{2})$ belongs to $M^{\prime}$.

Conversely, if $(A_{1},A_{2})$ and $(A^{\prime}_{1},A^{\prime}_{2})$ are two
elements of $M^{\prime}$ which define the same lattice, there exists an
integer matrix

$g^{\prime}=\begin{pmatrix}A&B\\\ C&D\end{pmatrix}$

of determinant >0 which transforms the first basis into the second. If
det($g^{\prime}$) was <0, the sign of Im$(A^{\prime}_{1}A^{\prime-}_{2})$
would be the opposite of Im$(A_{1}A^{-}_{2})$ as one sees by an immediate
computation. The two signs being the same, we have necessarily
det($g^{\prime}$) >0, which proves the theorem.

Thus, we can identify the set $M^{\prime}$ of all the lattice matrices of
$C_{n}$ with the quotient of $S_{n}$ by the action of Sp(2$n,\mathbb{R}$). ∎

For the same reasons, we can also define, as previously, a lattice function of
weight $2k$.

Let $F^{\prime}$ be such a function. If $(A_{1},A_{2})\in M^{\prime}$, we
denote by $F^{\prime}(A_{1},A_{2})$ the value of $F^{\prime}$ on the lattice
$\Gamma^{\prime}(A_{1},A_{2})$. The formula (2) translates to:

$F^{\prime}(\lambda A_{1},\lambda
A_{2})=\lambda^{-2k}F^{\prime}(A_{1},A_{2}).$ (5)

Writing now that $F^{\prime}$ is invariant by Sp($2n,\mathbb{R})$, we can see
that this function satisfies the identity:

$F^{\prime}(Z)=(XZ+D)^{-2k}f(\frac{AZ+B}{CZ+D}),$ (6)

for all:

$\begin{pmatrix}A&B\\\ C&D\end{pmatrix}\in\textnormal{Sp(}2n,\mathbb{R}).$

As previously, this function can be identified with an Eisenstein series
$G^{\prime}_{k}(\Gamma^{\prime})$ on the set $\mathcal{R}^{\prime}$ of the
matrix lattices of $C_{n}$, which is absolutely convergent.

In other words, all the $n\times n$ matrix lattices of $C_{n}$, which
represent sets of subsets of flows (or tensions) in connected 1-graphs (or
networks), are linked by this lattice function.

If we associate networks with subsets of flow (or tension) values, this proves
the existence of a "network of networks".

##  7 The tree of minimal length

Let $G_{k}(\Gamma)$ be the graph associated with the set of all subsets of
flows and $A_{0}$, the minimal tree of $G_{k}(\Gamma)$. If $U$ is the set of
arcs of $G_{k}(\Gamma),\ U-A_{0}=A^{\prime}_{0}$ is the maximal cotree of
$G_{k}(\Gamma)$222On trees and co-trees, see ([7], 103-128.. Now, it is easy
to see that :

(1) The smallest arc of all cocycles (tensions) is in $A_{0}$;

(2) The greatest arc of all cycles (flows) is in $U-A_{0}=A^{\prime}_{0}$.

We finally obtain a set of arcs without a maximal cycle and we can always find
an optimal flow in the graph because any flow does not circulate in all the
arcs of the whole graph but only in those of a tree whose capacities, which do
not admit a higher bound, are infinite.

Let us now precise the form of the tree of minimal length $A_{0}$. Let $n$ be
the number of vertices of $A_{0},\ A$ the set of its arcs, $a$ an arc of $A,\
d$ the distance between two vertices $s$ and $s^{\prime}$. We have:

(1) Card($A)=n(n-1)/2$;

(2) $a=\\{s,s^{\prime}\\}:d(a)=d(s,s^{\prime})$.

Now let $P$ be a polygon, i.e. a set of edges which is a subset of $A_{0}$.
The support of $P$ will be the union of the edges of $A_{0}$, that is, the set
of vertices of $G_{k}(\Gamma)$ which are ends of at least one edge of $P$. One
can speak of polygon $P$ on $G_{k}(\Gamma)$ (resp. in $G_{k}(\Gamma)$)
according to whether the support of $P$ is $G_{k}(\Gamma)$ or a subset of
$G_{k}(\Gamma)$ distinct from itself.

$A_{0}$, which is the set of all possible edges on $G_{k}(\Gamma)$, is a
complete polygon of $G_{k}(\Gamma)$.

A graph being the conjunction of a polygon and its support, a chain $C$ will
be a polygon in $G_{k}(\Gamma)$ whose vertices that form its support can be
ordered in a sequence ($s_{0},s_{1},...s_{p}$). We have:

(1) For every $i\in\ ]P[,\\{s_{i-1},s_{i}\\}\in C$;

(2) For every $i,j\in[P],i\neq j\Rightarrow s_{i}\neq s_{j}.$

A cycle is a chain where condition (2) holds for all the points of its support
except $s_{0}$ and $s_{p}$ which are merged (the ends of $C$).

To exhibit $G_{k}(\Gamma)$, we need the following complementary
considerations:

A) A tree is a connected polygon that does not contain a loop.

B) The length of a polygon is the sum of the lengths of all its edges.

C) The width of a polygon is the length of its longest edge.

Suppose that the polygon reduced to the edge $(s,s^{\prime})$ represents the
chain of $A$ with minimum width joining $s$ to $s^{\prime}$, then
$\\{s,s^{\prime}\\}$ is an element of the tree of minimal length $T$ on
$G_{k}(\Gamma)$ and there exists at least one such edge on $A$, the edge of
minimal length.

Conversely, if $\\{r,s\\}$ is an element of the tree $T$ on $G_{k}(\Gamma)$,
then $\\{r,s\\}$ is the chain of $A$ having the smallest width and joining $r$
to $s$.

In this context, $G_{k}(\Gamma)$ can be identified with a classification of
classifications. This would amount to doing a factor analysis on all parts of
the representative tree. Such a classification would correspond to all the
axes of a factor analysis, with an original calculation on the first axis.

##  8 Construction of $G_{k}(\Gamma)$

In order to construct $G_{k}(\Gamma)$, we must first look at the lattice
$\Gamma=\mathbb{Z}\alpha_{1}\oplus\mathbb{Z}\alpha_{2}$, which makes possible
to distinguish a lattice function and a modular function. It must be assumed
that the minimal bases of this lattice suppose a matroid $M$. If $B$ is the
set of these bases, then $C$, the set of cycles of $M$ (resp. $D$ the set of
cocycles of $M$) is the set of subsets which are not included in any basis
(resp. which have a non-empty intersection with any basis) and minimal for
inclusion with this property.

Let now $B\in\mathcal{B},\ b\in B,\ c\in X-B$. Let $D(b,B)$ be the unique
cocycle satisfying $B\cap D(b,B)=\\{b\\}$ and $C(c,B)$, the unique cycle
satisfying $C(c,B)-B=\\{c\\}$. We then have:

$B\in C(c,B)\iff c\in D(b,B)\iff B-{b}\cup{c}\in\mathcal{B}.$

$C$ and $D$ are the sets of cycles and minimal cocycles for the inclusion of
the graph.

The minimum tree of a graph is the set of minimum edges of a cocycle, its
complement being the set of the maximum edges of a cycle.

If we consider the lattice $(F,\cup,\cap$), a sublattice of $M$, the algebraic
properties of $F$ (distributivity) are stronger than those of $M$ (semi-
modularity). It is thus possible to construct $G_{k}(\Gamma)$, the super-
lattice, by defining it as the set of distributive sub-lattices of any
geometric lattice, that is to say, a sub-lattice of the semi-geometric lattice
associated with $M$.

##  9 Possible applications

Let’s finish with some more epistemological considerations: after all,
mathematical physics and philosophy are not so far apart (see [18]).

The space associated with this "network of networks", that is, the zeros and
poles of the modular function of all networks, has been studied in hard proof
theorems, because one does not define a structure of complex analytic variety
on the single compactified network. (A natural way of proceeding would be to
define a compactified isomorphism on the Riemann sphere
$S=\mathbb{C}\cup\\{\infty\\}.)$

Whatever the difficulties of study, it is proved that this network function
exists, and we have thus proved also that the set of all sets of possible
flows exists as a modular function of all networks in the algebraic sense of
the term.

Let us now consider some possible applications of the previous formalism.

1\. Since the old work of [Von Neuman 1946], quantum mechanics represents all
the physical states of the universe by a vector space of infinite dimension
called "Hilbert space". However, the separability property and the convergence
condition make it possible to reduce to closed subspaces. In this case, the
complex vectors form a finite dimensional subspace and their mathematics is
identical to that of flows or tensions on a graph, except that their
coefficients can take on complex values. This situation makes it possible, as
we have seen, to apply known theorems of arithmetic to them.

2\. Because of the flow-tension duality, the network function defines as well
the set of all the sets of possible tensions, and hence it specifies the
shortest path in the total set of all possible paths, as well as the most
rational scheduling of tasks in the set of all possible actions. Here we have
a theorem of the existence of an optimal behavior, whatever the field we
consider.

3\. Moreover, the problem of the shortest path in a graph is related to the
question of the tree of minimum length, which itself formalizes the notion of
classification. A "network of networks" with a maximum voltage would thus make
it possible both: to confirm the existence of a tree of minimum length of the
network of all networks, and hence, of a classification of classifications
(see [17]).

4\. In general, the variable "weights" can receive different meanings
(reliability, economy, etc.) on a tree, other than the length of the arcs. So
the network of networks $G_{k}(\Gamma)$ can still make it possible to
calculate the maximum reliability path, or the most economical route, etc., in
the set of all possible paths.

5\. I will say a final word about the aim of this construction : though the
world may be multiple and chaotic, circulations and actions can be ordered in
relation to the same structure, which is expressed - in the linear case -
through the form of this remarkable holomorphic function which has been here
constructed. Doing that, we tried in fact to formalize the intuition of a
"network of networks", as it is expressed in the conclusion of our book on
networks (see [15], 265-286). This is also the achievement of what we called
elsewhere a "rationalité réticulaire" (reticular rationality)(see [16]).

## References

  * [1] Berge, C., Graphes et hypergraphes, Dunod, Paris, 1970.
  * [2] Deheuvels, R., Formes quadratiques et groupes classiques, P.U.F., Paris, 1981.
  * [3] Ford, L. R., Fulkerson, D. R., Flows in Networks, Rand Corporation, Santa Monica, 1962.
  * [4] Fulkerson, D. R., "An out-of-kilter method for minimal-cost flow problems", Journal of the Society for Industrial and Applied Mathematics 9, 18-27, 1961.
  * [5] Gallai, T., "Gráfokkal kapcsolatos maximum-minimum tételek, I rész",[Hungarian: Maximum-minimum theorems for networks (part I)] A Magyar Tudományos Akadémia Matematikai és Fizikai Tudományok Osztályának Közleményei, 7, S. 305-338, 1957.
  * [6] Gallai, T., "Maximum-minimum Sätze Über Graphen", Acta Mathematica Academiae Scientiarum Hungaricae 9, 395-434, 1958.
  * [7] Gondran, M., Minoux, M., Graphes et Algorithmes, Eyrolles, Paris, 1979.
  * [8] Hitchcock, F. L., "The distribution of a product from several sources to numerous localities", Journal of Mathematics and Physics 20, 224-230, 1941.
  * [9] Kantorovich, L. V., "O peremeshchenii mass" [Russian], Doklady Akademii Nauk SSSR 37, 7-8, 227-230, 1942. [English translation: "On the translocation of masses", Comptes Rendus (Doklady) de l’Académie des Sciences de l’U.R.S.S, 37, 199-201,1942, [reprinted: Management Science 5, 1-4, 1958].
  * [10] Kantorovich L. V., Gavurin, M. K., "Primenenie matematicheskikh metodov v voprosakh analiza gruzopotokov" [Russian; "The application of mathematical methods to freight flow analysis"], in Problemy povysheniya effectivnosti raboty transporta [Russian; Collection of Problems of Raising the Efficiency of Transport Performance], Akademiia Nauk SSSR, Moscow-Leningrad, 1949, pp. 110-138.
  * [11] Klein, M., "A primal method for minimal cost flows with applications to the assignment and transportation problems", Management Science 14, 205-220, 1967.
  * [12] Koopmans, "Optimum utilization of the transportation system", in: The Econometric Society Meeting Washington, D.C., September 6-18, 1947; D.H. Leavens, (ed.), Proceedings of the International Statistical Conferences \- Volume V, 1948, pp. 136-146; reprinted in Econometrica 17 (Supplement) 136-146, 1949; reprinted in Scientific Papers of Tjalling C. Koopmans, Springer, Berlin, 184-193.
  * [13] Koopmans, Tj.C., Reiter, S., "A model of transportation", Activity Analysis of Production and Allocation - Proceedings of a Conference, Tj.C. Koopmans (ed.), 222-259, Wiley, New York, 1951.
  * [14] Lur’e, A. L., "Methods of establishing the shortest running distances for freights on setting up transportation systems" [in Russian], in Primenenie matematiki èkonomicheskikh issle-dovaniyakh [Russian; V.S. Nemchinov (ed.), Application of Mathematics in Economical Studies, Izdatel’stvo Sotsial’no-Èkonomicheskĭ Literatury, 249-382, Moscow, 1959. English translation in: V.S Nemchinov (ed), The Use of Mathematics in Economics, 322-355, Oliver and Boyd, Edinburgh, 1964.
  * [15] Parrochia, D., Philosophie des réseaux, P.U.F., Paris, 1993.
  * [16] Parrochia, D., "La rationalité réticulaire", in D. Parrochia (ed.), Penser les réaux, 7-23, Champ Vallon, Seyssel, 2001.
  * [17] Parrochia D., Neuville, P., Towards a general theory of classifications, Basel, Birkhäuser, 2013.
  * [18] Parrochia, D., Mathematics and Philosophy, Wiley-Iste, London, 2018.
  * [19] Robinson, J., "On the Hamiltonian Game (A Traveling Salesman Problem)", Research Memorandum RM-303, The RAND Corporation, Santa Monica, California, 1949.
  * [20] Robinson, J., "A Note on the Hitchcock-Koopmans Problem", Research Memorandum RM-407, The RAND Corporation, Santa Monica, California, 1950.
  * [21] Schrijver, A., "On the history of the transportation and maximum flow problems", Math. Program., 91, 437-445, 2002.
  * [22] Serre, J.-P., Course in Arithmetics, Springer-Verlag, Berlin, 1973.