[1,2]Jiajie Fan

[1] BMW Group, Bremer Straße 6, Munich, 80788, Bavaria, Germany

2] LIACS, Leiden University, Niels Bohrweg 1, Leiden, 2333, The Netherlands

# Adversarial Latent Autoencoder with Self-Attention for Structural Image
Synthesis

<EMAIL_ADDRESS>Laure Vuaille<EMAIL_ADDRESS>Hao Wang
<EMAIL_ADDRESS>Thomas Bäck<EMAIL_ADDRESS>*  [

###### Abstract

Generative Engineering Design approaches driven by Deep Generative Models
(DGM) have been proposed to facilitate industrial engineering processes. In
such processes, designs often come in the form of images, such as blueprints,
engineering drawings, and CAD models depending on the level of detail. DGMs
have been successfully employed for synthesis of natural images, e.g.,
displaying animals, human faces and landscapes. However, industrial design
images are fundamentally different from natural scenes in that they contain
rich structural patterns and long-range dependencies, which are challenging
for convolution-based DGMs to generate. Moreover, DGM-driven generation
process is typically triggered based on random noisy inputs, which outputs
unpredictable samples and thus cannot perform an efficient industrial design
exploration. We tackle these challenges by proposing a novel model Self-
Attention Adversarial Latent Autoencoder (SA-ALAE), which allows generating
feasible design images of complex engineering parts. With SA-ALAE, users can
not only explore novel variants of an existing design, but also control the
generation process by operating in latent space. The potential of SA-ALAE is
shown by generating engineering blueprints in a real automotive design task.

###### keywords:

Deep generative model, Generative engineering design, Self-attention, Latent
space manipulation, Structure generation

##  1 Introduction

Generative Engineering Design (GED) [30, 25] is a novel trend of Generative
Design for a more function-oriented design exploration making use of existing
designs and algorithms. GED derives novel designs depending on factors such as
the initial structure, desired performance and specific project requirements.
As an automated exploration of industrial designs, GED shows a great potential
to assist engineers in developing complex structures and speed up industrial
development processes.

Deep Generative Models (DGMs) [16, 2, 35] seem to be a natural choice to
facilitate GED, given its compelling performance in synthesizing natural
images, such as landscapes, human faces, and animals. However, the
implementation of DGMs as such is unlikely to bring much benefit to industrial
engineering processes. One reason is that most DGMs fail to provide a
controllable generation process. Recent top performing DGMs tend to use
Generative Adversarial Networks (GANs) [11, 24] as a framework, which derives
a single-generator sampling procedure and typically generates a novel image
from a random noisy input. Novel models [20, 15, 6] have been proposed to
condition the models on additional input variables, hereby enabling fine-
grained control over the generation process. These input variables are
predefined attributes, such as gender, hair style or face shape in the context
of human face generation. However, it is difficult to preset input
conditioning variables in industrial use cases because of the lack of
universal conventions used to define design parameters and large consistent
labelled data. Furthermore, the relationship between the parameters and the
engineering target often requires complex numerical methods to be resolved. It
is thus difficult to define relevant class labels to condition the generation
process. Alternatively, Adversarial Latent Autoencoder (ALAE) [23] utilizes
GAN’s adversarial training strategy but provides an encoder-decoder network,
which allows for a mapping between image space and latent space. This
architecture produces diverse images by interpreting random samples in latent
space. Additionally, with sufficient analysis in the learned latent space,
ALAE can be used to control the generation process by conditioning and
modifying attributes in the latent space.

Figure 1: Example of an automotive A-Pillar with a free motion head-form
[22](left) and an engineering blueprint (right) obtained by extracting a
cross-section from the 3D-object.

Another challenge lies in the fundamental difference between design images
from industrial engineering processes and natural images. Natural images
typically consist of textures and continuously changing color gradients, while
industrial design images are defined by structural patterns, such as distinct
edges, sparse shapes and their associated long-range dependencies. Most DGMs
rely heavily on the convolutional operator, which excels in synthesizing
realistic textures in local neighborhoods but fails to model large structural
patterns [19, 36]. With convincing results, recent research [34, 36]
demonstrates the potential of self-attention mechanism to enable DGMs to
capture and model large-scale features and long-range dependencies.

To successfully adapt DGMs for engineering design applications, we propose a
novel model called Self-Attention Adversarial Latent Autoencoder (SA-ALAE). We
show the potential of SA-ALAE by applying it in the context of automotive
design exploration, where structural design is represented in engineering
blueprints. To measure the quality of the synthesized blueprints, we reproduce
the data-driven criteria comparison proposed by [13] based on the blueprints,
and identify Fréchet inception distance (FID) as the most stable method among
the criteria compared. An example of the engineering blueprint is shown in
Figure 1. We further show with one experiment that SA-ALAE enables a
controlled generation process by semantically modifying the encoded latent
variables. The result of this experiment demonstrates the potential of SA-ALAE
to understand the structural information in industrial design images, and
explore novel design alternatives in desired ways. The contribution of our
work is as follows:

  1. 1.

We propose a novel DGM based on ALAE and the Self-Attention mechanism to
enable the generation of complex structural patterns.

  2. 2.

We address the need for a more reliable quantitative method than FID for
evaluating the quality of synthesized engineering blueprints.

  3. 3.

We propose a method to control the generative process of SA-ALAE by
conditioning and modifying variables in latent space.

##  2 Related work

Within the domain of Deep Generative Models (DGMs), Generative Adversarial
Networks (GANs) [11, 24] have achieved state-of-the-art performance in many
image synthesis tasks [2, 35]. Novel models, e.g., Conditional Generative
Adversarial Networks (CGANs) [20] and InfoGAN [6], allow users to control the
generation process by adding conditional variables such as gender and face
shape in the context of human-face generation. The input variables are
predefined and require labeled data, which is often obtained by manual labour.
In case of engineering blueprint, such classes are not available because of
the complexity of the shapes represented and also due to the lack of
parameters available or conventions used to define them. However, inspired by
the use of latent space to control the generation process [24, 29, 1],
progress has been made in enabling GANs to map images to latent variables. The
most successful trend is to combine GANs with encoders, which has already
yielded convincing models such as BiGAN [7], Adversarially Learned Inference
(ALI) [8] and Adversarial Latent Autoencoder (ALAE) [23]. These models allow
an analysis of the learned latent space after training. After identification
and understanding of the latent variables, semantic modification in the latent
space can influence the generation process in desired ways. Among BiGAN, ALI
and ALAE, the latter achieves the best image quality when tasked to
reconstruct MNIST [17] digits and also the highest accuracy when applying
MNIST classification to its generated outputs [23]. After training, ALAE
provides an encoder-decoder inference, which allows ALAE to conduce a mapping
between image and latent space. Diverse designs can be produced by encoding-
decoding the initial design with random noise added in the latent space.

Additionally, with sufficient analysis in the learned latent space, the ALAE
can be used to condition the generated outputs by modifying only specific
latent variables [24, 29, 1].

The architecture of ALAE is shown in Figure 2(a). The ALAE implements a mapper
$M$ that generates latent variables $\omega\subset\mathbb{R}^{d}$
($d$-dimensional) for random noise $z\sim\mathcal{N}(0,\mathcal{I})$. The
space of the latent variables is indicated as $\mathcal{W}$. The latent
variables $\omega$ serves as input to both discriminator
($D\colon\mathcal{W}\rightarrow\mathbb{R}$) and generator
($G\colon\mathcal{W}\rightarrow\mathcal{X}$), where $\mathcal{X}$ stands for
the space of data points. Each generated data point $G(\omega,\eta)$ is
dependent not only on the latent variables $\omega$ but also on an optional
noise $\eta$. ALAE adds an encoder
($E\colon\mathcal{X}\rightarrow\mathcal{W}$) to interpret the generated data
$G(\omega,\eta)$ or real data $x$ into latent variables $\omega$, which is
validated by the discriminator $D$ to a score $D(\omega)$. ALAE can be seen as
a GAN in which the pair ($M,G$) plays the role of the generator and the pair
($E,D$) that of the discriminator. ALAE utilizes an adversarial training in
the data space with the optimization $\min_{M,G}\max_{E,D}V(G\circ M,D\circ
E)$, where $\circ$ is the chaining operator. Note that $G$ and $E$ together
construct an autoencoder architecture ($G,E$) in the latent space
$\mathcal{W}$, which is trained by optimizing
$\min_{E,G}\Delta(\omega\parallel E(G(\omega)))$. For the inference, the pair
of networks $(E,G)$ builds up an autoencoder network in the data space, which
requires an source image $x$ and a noise $\eta\sim\mathcal{N}(0,\mathcal{I})$
as inputs and generates novel images. Additionally, feeding sampled noise $z$
to the pair of networks $(M,G)$ is able to generate random samples.

Since ALAE implements an adversarial training strategy like GANs, ALAE also
suffers from training instability [11, 9, 21]. Many methods focus on solving
this issue, e.g., zero-centered gradient penalty [26], Spectral Normalization
[21], Batch Normalization [28, 10], etc. Taking Residual Networks (ResNet) as
backbone enables the training of deep architectures without suffering from
vanishing gradients and degradation in performance [12].

In Generative Engineering Design (GED) [5, 4, 3, 18], DGM has been shown its
potential in synthesizing airfoil shapes from the UIUC database [33], in which
an airfoil shape is a single 2D curve. However, real-world engineering design
images are often much more complex, as they contain multiple clear edges,
sparse shapes and long-range geometric constraints. Generating such complex
design images is still a challenge for convolution-only neural networks
because the convolutional operator can only process information within a local
neighbourhood [19, 36]. Self-Attention Generative Adversarial Network (SAGAN)
proposed by [36] introduces the self-attention mechanism into GANs. The
implemented mechanism, Scaled Dot-Product Attention [34], constructs an
attention map by computing the attention weights between each feature vector
and the other feature vectors. This attention map presents the relationships
between input features. With this attention map the model gains a global
understanding of the input features and is able to efficiently capture long-
range dependencies.

There is no universal criterion for evaluating the quality of images generated
by DGMs [27, 31, 32]. The most popular strategy is to measure the statistical
distance between the distribution of generated images and source images, for
which the Fréchet Inception Distance (FID), the Earth-Mover Distance (EMD),
the Kullback-Leibler Divergence (KLD), and the Jensen-Shannon Divergence (JSD)
are the most popular criteria. The performance of the criterion heavily
depends on the dataset used [31]. The approach proposed by [13] enables to
identify a suitable criterion for a given dataset. In this approach, the
distance between a source dataset and a perturbed dataset is measured
repeatedly with increased level of perturbation. The suitable criterion should
accurately capture the increase of perturbation.

##  3 Methods

In this work, we aim at developing an appropriate DGM for enabling an
efficient industrial design exploration. For this purpose, we select
Adversarial Latent Autoencoder (ALAE) [23] as a baseline framework for its
flexible sampling process. We enhance the model robustness and develop an
improved version by introducing modern techniques, e.g., Batch Normalization,
Spectral Normalization (SN) [21], ResNet [12], etc. We further implement the
self-attention mechanism [34, 36] to enable the generation of structural
patterns.

(a) Training

(b) Sampling

Figure 2: SA-ALAE architecture. ALAE and SA-ALAE share an identical training
strategy. The sampling procedure of SA-ALAE utilizes the off-the-shelf mapper
$M$ to sample an additional noisy latent variables $\omega_{noise}$, hereby
introducing randomness in the sampling.

###  3.1 Improved ALAE

The vanilla ALAE faces a major issue in its training for the following
reasons. First, the ALAE implements the adversarial training strategy in image
space, which exposes it to the same training instability commonly encountered
by GANs [11, 26, 9]. The possible cause for the unstable adversarial training
is that the gradients computed on the discriminator function vary too much.
Besides, generating large and detailed engineering design images necessitates
a deep model, which results in gradients that can easily vanish or explode.
Moreover, contrary to expectations, increasing the depth of a neural network
may result in a degradation of performance [12].

We implement the following techniques to improve the model robustness and
propose a novel version of the ALAE called ALAEimproved. Firstly, to ensure
that generator $G$ receives effective gradients for the optimization, we need
to prevent the computed gradients from vanishing or exploding. For this
purpose, we apply Spectral Normalization (SN) [21] in encoder $E$ and
discriminator $D$. Spectral Normalization is a powerful method that constrains
the Lipschitz continuity of the functions optimized by $E$ and $D$ through
layer-wise controlling the spectral norm, hereby constraining the size of the
gradients. Spectral Normalization replaces every given weight matrix $W$ by
$W_{sn}$ with the formula:

$W_{\text{sn}}=\frac{W}{\sigma(W)},$ (1)

where $\sigma(W)$ is the greatest singular value of $W$. Moreover, for
stabilizing the adversarial training, we implement Batch Normalization in
generator $G$, allowing for a larger range of learning rates and to learn the
data distribution more efficiently. Lastly, to address the degradation caused
by increasing the depth of the neural network, we modify generator $G$ and
encoder $E$ of ALAE to become Residual Networks (ResNet). ResNet features in
the so-called residual blocks of convolutional operators contain a “skip
connection” that can bypass the error information in back-propagation, hence
alleviating the vanishing gradient problem in deeper architectures.

###  3.2 Self-Attention Adversarial Latent Autoencoder (SA-ALAE)

Convolution-based DGMs excel in generating real-world images, i.e., sky,
mountain, and lakes in landscape images. But DGMs have a hard time modeling
structural objects in engineering design images. One of the main causes is
that the objects in structural images can be sparsely located but are still
strongly geometrically connected, e.g., in bicycle design, both wheels should
be connected by an axle and lie on the ground. In this simple example, the
pixels describing each wheel are depending on one another but are far away in
the image. However, the convolution mechanism, as a local operator on images,
cannot capture long-range dependencies, suggesting that using convolutional
layers alone is unsuitable in generating engineering design images. Self-
Attention Generative Adversarial Network (SAGAN)[36] leverages the self-
attention mechanism to gain a global understanding of the input features,
which enables the model to efficiently capture and model long-range
dependencies. Following the success of SAGAN, we introduce the same mechanism
respectively into generator $G$ and encoder $E$ in the ALAE framework and
propose a new model, Self-Attention Adversarial Latent Autoencoder (SA-ALAE).

In industrial design exploration, the generation process is tasked to propose
variants of an existing design. To achieve the stochastic generation of a
novel image from a source image in the vanilla ALAE, the latent variables
$\omega$ encoded from the original design are perturbed with sampled noise
$\eta\sim\mathcal{N}(0,\mathcal{I})$. We note that this approach compromises
the quality of the encoded latent variables, which leads to blurring of the
generated images. To avoid this, SA-ALAE leverages the trained mapper $M$ to
sample random latent variables as noisy perturbation, which follows the
learned distribution of the latent space $\mathcal{W}$. The novel sampling
procedure of SA-ALAE is displayed in Figure 2(b). Adding randomly sampled
latent variables as an independent noise protects the quality of perturbed
latent variables so that the stochastic encoder-decoder inference does not
harm the quality of the generated image. The calculation of the latent
variables, used to explore the variants of existing designs, is based on the
following formula:

$\omega=(1-\mu)\cdot E(x)+\mu\cdot M(z),$ (2)

where $\mu\in[0,1]$ is a tunable parameter, $x$ refers to an original design
and $z\sim\mathcal{N}(0,\mathcal{I})$ is sampled noise.

(a) Example of swirl effect on an engineering blueprint.

(b) Distances measured between source dataset and datasets with increased
swirl perturbation.

Figure 3: Comparison of criteria for evaluating image quality based on swirl
perturbation. Three datasets are created by perturbing the source dataset with
increasing levels of swirl. FID increases with the level of swirl
perturbation, whereas other criteria fail to illustrate the increasing
divergences brought by increasing swirl levels.

(c) Examples of blueprints from the sub-dataset APH of $Dataset_{1}$.

(d) Influence of sample size on FID. FIDs are measured among various cross-
sections from the sub-dataset APH of $Dataset_{1}$.

Figure 4: Example of vehicle A-Pillar horizontal structures and the FIDs
measured in terms of sample size. The FIDs are measured between the
engineering blueprints of the vehicle A-Pillar HS-0 structure and engineering
blueprints of the vehicle A-Pillars HS-1,2,3 structures.

##  4 Experiments

To evaluate the performance of SA-ALAE for industrial design exploration, we
trained the model to generate blueprints of vehicle parts from a real
industrial design task. This dataset consists of 157 234 blueprints of various
automotive structures, e.g., A-Pillars, B-Pillars, upper roof, and side rails.
Extracting cross-sections from the vehicle structures at various angles and
locations produces different engineering blueprints, e.g., horizontal
sagittal, horizontal superior, vertical sagittal, etc. In particular, for a
given vehicle structure, the horizontal-sagittal (HS-0) engineering blueprint
and the horizontal-superior-1 (HS-1) engineering blueprint are produced by two
parallel cross-sections close to one another and thus display a high
similarity in structures. The FID measured between them are used as baseline
for evaluating model performance in our work. The horizontal-superior-2 (HS-2)
and horizontal-superior-3 (HS-3) engineering blueprints are also parallel
cross-sections albeit taken further away in space. The sub-dataset APH
consists of 12 876 blueprints of the A-Pillar structures, including HS-0,
HS-1, HS-2 and HS-3 engineering blueprints. It is used as a simpler dataset to
test the generative power of SA-ALAE. The blueprints are grayscale images with
a resolution of $256\times 256$. Two different versions of this dataset are
used in this study: $Dataset_{1}$, as some examples displayed in Figure 5(a)
and some samples from its sub-dataset APH shown in Figure 6(a); and
$Dataset_{2}$, as some samples of its sub-dataset APH shown in Figure 7(a).
$Dataset_{1}$ is an earlier database that contains irrelevant information such
as a head-form model used in a specific engineering process and wire frames to
fill specific areas. This additional but irrelevant information complicates
the training and requires preprocessing. $Dataset_{2}$ is a cleaner dataset,
in which the head-form has been removed, images are centered on the structure
and specific areas are filled with continuous grayscale color.

For the training, the initial learning rates for the optimizations
$lr_{(D,E)}$, $lr_{(M,G)}$, $lr_{(G,E)}$ are respectively $1e-4$, $2e-4$,
$1e-4$. The batch size is $128$ and the length of latent variables is set to
$64$. We apply a self-attention layer after each upsampling and downsampling
module, where the size of the output features is $16\times 16$.

From each target dataset used, we randomly select 1 000 images as test data
and 100 images as validation data, while the remaining images serve as
training data. In accordance with the results in Section 4.1, we select
Fréchet Inception Distance (FID) as the criterion to evaluate the quality of
generated images during training. A higher FID indicates a larger difference
between the two image datasets. In the evaluation, the model samples images
for random noisy inputs, and the FID is measured between the sampled images
and the corresponding source data. For computational efficiency, the FID is
measured between 100 source images and 100 generated images after every epoch
during the training to evaluate the model performance. If the measured FID
does not improve within 20 epochs, the training procedure is terminated. The
FID baseline is determined by the FID measured between neighboring cross-
sections, i.e., HS-0 and HS-1 engineering blueprints taken from the sub-
dataset APH of the same dataset, as one example from $Dataset_{1}$ shown in
Figure 3(c). The measured FID baselines are $55.40$ for $Dataset_{1}$ and
$35.58$ for $Dataset_{2}$.

In this work, we implement various strategies to train the models:

  * •

ALAE [23]: vanilla ALAE model trained with $Dataset_{1}$;

  * •

ALAEimproved: improved version of ALAE, trained with $Dataset_{1}$

  * •

SA-ALAEwhole: introducing self-attention layers into ALAEimproved model,
trained with $Dataset_{1}$;

  * •

SA-ALAEaph: initial SA-ALAE model trained with the sub-dataset APH of
$Dataset_{1}$;

  * •

SA-ALAEretrain: fine-tuning the trained SA-ALAEwhole model on the sub-dataset
APH of $Dataset_{1}$;

  * •

SA-ALAEiaph: initial SA-ALAE model trained with the sub-dataset APH of
$Dataset_{2}$.

###  4.1 Evaluation methods

A quantitative assessment is needed to evaluate model performance. Typically,
a statistical distance between generated image set and source image set is
implemented. The adequacy of the commonly used evaluation criteria, e.g.,
Fréchet Inception Distance (FID), Earth-Mover Distance (EMD), Kullback-Leibler
Divergence (KLD), and Jensen-Shannon Divergence (JSD), depends on the source
dataset. Therefore, we reproduce the comparison proposed by [13] to identify
the most appropriate criterion when applied to our engineering design images.

For the comparison, we create new image sets by adding various disturbances,
i.e., Gaussian noise, blur, white blocks, and swirl effect, of increasing
intensity to the source images. A good criterion should detect the increasing
intensity of added disturbances. Adding swirl effect transforms parts of the
image as a spiral, as an example of swirl effect shown in Figure 4. The larger
the intensity level is, the more the swirl effect perturbs the source images.
The relationship between the measured distance and the intensity level of
swirl effect is shown in Figure 3(b). FID shows the most stable performance
for all types of disturbances added.

As a rule of thumb, FID is calculated with 50 000 generated samples to
evaluate the performance of a trained model [13]. However, generating images
of this high number incurs significant computational costs. As shown in Figure
3(d), the measured FID values are very sensitive to the sample size until 1
000, where we observe a plateau. Based on this observation, it is safe to take
a smaller set of 1 000 images for empirical assessment of our dataset.

We also evaluate the model performance with qualitative analysis, i.e., we
assess the visual quality of the generated images in terms of blurriness,
detailed structure and readability.

###  4.2 Quantitative Analysis

The quantitative performance measured with FID of ALAE, ALAEimproved, and SA-
ALAEwhole are listed in Table 1. The FIDs measured of SA-ALAEaph, SA-
ALAEretain and SA-ALAEiaph are shown in Table 2. ALAEimproved significantly
reduces the FID by $52\%$ compared to the vanilla ALAE model, while SA-
ALAEwhole shows a further improvement over ALAEimproved, with a $34\%$
decrease in FID. However, SA-ALAEwhole performs significantly worse than the
baseline with $62\%$ larger in terms of FID. Training an initial SA-ALAE on
the sub-dataset APH from scratch improves the performance by $23\%$ but is
still outperformed by $25\%$ by the baseline. Notably, fine-tuning SA-
ALAEwhole on the sub-dataset APH leads to an improvement of the performance by
$12\%$ compared to the baseline. When using $Dataset_{2}$ as the source, SA-
ALAEiaph trained on the sub-dataset APH from scratch is able to overperform
the baseline by $15\%$ in terms of FID.

###  4.3 Qualitative Analysis

For the models trained on $Dataset_{1}$, some generated engineering blueprints
are shown in Figure 5. The synthesized images from SA-ALAEretain and SA-
ALAEaph are shown in Figure 6. The blueprints generated by SA-ALAEiaph, a
further implementation of an initial SA-ALAE model trained on $Dataset_{2}$,
are shown in Figure 7.

Samples generated by vanilla ALAE are not always corresponding to the input
blueprints in terms of structure, and the images exhibit a substantial level
of blurriness such that the edges and shapes can be completely unrecognizable.
After enhancing the training stability, ALAEimproved significantly reduces the
blurriness of the generated structures, but the geometrical objects remain
unrecognizable. SA-ALAEwhole provides more detailed blueprints and empirically
validates the benefits of the self-attention mechanism. Fine-tuning SA-
ALAEwhole on the sub-dataset APH, SA-ALAEretrain is able to generate high-
quality images, which however are not yet sufficiently recognizable to
engineers. SA-ALAEaph, trained on the sub-dataset APH from scratch shows a
more stable performance and its generated images are of a higher visual
quality than those from SA-ALAEretrain. SA-ALAEiaph, trained on $Dataset_{2}$,
excels in generating high-quality blueprints that not only correspond to the
input designs but also contain recognizable structural details.

Table 1: FID scores. Performance of ALAE, ALAEimproved and SA-ALAE trained
with $Dataset_{1}$.

Objective |  FID
---|---
ALAE [23] |  $284.01$
ALAEimproved |  $135.52$
SA-ALAEwhole |  90.09

00footnotetext: Bold numbers indicate the best results in this experiment

Table 2: FID scores. Performance of SA-ALAE trained with various image sets.

Objective |  FID
---|---
Using $Dataset_{1}$ as source |
Baseline |  $55.40$
SA-ALAEwhole |  $90.09$
SA-ALAEaph |  $69.24$
SA-ALAEretrain |  48.78
Using $Dataset_{2}$ as source |
Baseline |  $35.58$
SA-ALAEiaph |  30.10

00footnotetext: Bold numbers indicate the best results in this experiment

(a) Input blueprints

(b) Blueprints generated by ALAE

(c) Blueprints generated by ALAEimproved

(d) Blueprints generated by SA-ALAEwhole

Figure 5: Examples of source blueprints from $Dataset_{1}$ and blueprints
generated by ALAE and SA-ALAEwhole

(a) Input blueprints

(b) Blueprints generated by SA-ALAEretrain

(c) Blueprints generated by SA-ALAEaph

Figure 6: Examples of source blueprints from sub-dataset APH of $Dataset_{1}$
and blueprints generated by SA-ALAE

(a) Input blueprints

(b) Blueprints generated by SA-ALAEiaph

Figure 7: Examples of source blueprints from sub-dataset APH from $Dataset_{2}$ and blueprints generated by SA-ALAEiaph | source 1
---
$\alpha=0$
$\alpha=0.2$ | $\alpha=0.4$ | $\alpha=0.6$ | $\alpha=0.8$ |  | source 2
---
$\alpha=1.0$
|  |  |  |  |
Figure 8: Example of interpolation in latent space. The interpolation is
conditioned on $\alpha$, which is a variable ranging from 0 to 1.

###  4.4 Interpolation in Latent Space

After training, SA-ALAE can generate novel design alternatives based on a
given blueprint. However, it would be useful to guide the generation process
and influence the generated outputs. We hypothesize that this is made possible
by the learned semantic relationship between the latent space and the patterns
represented in the images. We identify two source images in which a similar
object is rotated in the second image compared to the first. We show that
various linear combinations of the latent variables used as input for the
generator $G$ of SA-ALAE produce novel structures with various degrees of
rotation, as shown in Figure 8. The linear interpolation [14, 15] between the
latent variables encoded from two source images is obtained by the formula:

$\omega_{interpolated}=(1-\alpha)\cdot\omega_{1}+\alpha\cdot\omega_{2},$ (3)

in which $\alpha$ is a variable, ranging from 0 to 1, and $\omega_{1}$,
$\omega_{2}$ are the encoded latent variables.

###  4.5 Discussion

ALAE provides a flexible generation process and is shown in the literature
[23] to have a GAN-like generative power on natural images. However, instead
of recognizable structures, the vanilla ALAE on the vehicle A-Pillar
blueprints yields only vague structures. After enhancing the model with
ResNet, weight normalization, and self-attention mechanism, our SA-ALAE
achieves a better FID score, and more importantly, it is able to achieve a
higher image quality under expert visual inspections of the generated images.
When trained with blueprints from one single structure, e.g., vehicle
A-Pillars, our SA-ALAE generates blueprints that are recognizable to
engineers. However, when trained on blueprints with higher variability, i.e.,
structures from various vehicle components instead of A-Pillar alone, SA-ALAE
demonstrates an uneven performance. An interpolation in the learned latent
space shows that SA-ALAE is able to guide and control the generation process
by modifying latent variables. Yet, since the components comprising a
structural design are highly complex, making modifications to latent variables
requires substantial effort in analyzing the learned latent space. Lastly, our
selected evaluation criterion FID shows a reliable performance in
distinguishing poor-quality blueprints. However, we notice that images
generated by SA-ALAEaph demonstrate a higher visual quality than those from
SA-ALAEretrain, but ALAEretrain achieves a better FID score. This observation
indicates that the FID may not serve as a reliable evaluation criterion when
the quality of generated images reaches a certain threshold.

##  5 Conclusion

In this paper, we focus on a Deep Generative Models (DGMs)-driven industrial
application, which leverages existing designs and algorithms to enable
automated exploration of engineering designs. We select Adversarial Latent
Autoencoder (ALAE) as the base model and extend the capability of the
Adversarial Latent Autoencoder (ALAE) by modifying the ALAE model with the
ResNet architecture and implementing several weight regularization methods for
stabilizing the training. In order to enable the generation of feasible
engineering design images, we introduce the self-attention mechanism into the
improved ALAE model, resulting in a novel DGM, Self-Attention Adversarial
Latent Autoencoder (SA-ALAE). In contrast to the previous work in DGM-driven
shape synthesis, SA-ALAE performs an image-to-image generation and is able to
generate high-quality engineering blueprints of complex structures. An
interpolation experiment conducted in the latent space demonstrates that the
latent variables accurately represent structural information. Furthermore,
modifying the encoded latent variables leads to corresponding semantic
modifications in the generated structures. Despite the advances, several
issues hinder an efficient application of SA-ALAE in Generative Engineering
Design: (1) the current capability of SA-ALAE is insufficient for exploring
designs for multiple vehicle components with a single model; (2) structural
manipulation by modifying the latent space necessitates significant effort to
analyze the learned latent variables and the structural attributes they
represent.

## Declarations

  * •

Authors’ contributions JF conceived and designed the study, conducted the
research, analyzed the data, and wrote the manuscript. LV, HW, and TB
supervised the research, provided guidance and expertise, and critically
reviewed and edited the manuscript. All authors read and approved the final
manuscript.

  * •

Funding This work is supported by BMW Group.

  * •

Availability of data and materials Not Applicable.

  * •

Code availability Not Applicable.

  * •

Conflict of interest The authors have no relevant financial or non-financial
interests to disclose.

  * •

Ethics approval Not Applicable.

  * •

Consent to participate Not Applicable.

  * •

Consent for publication Not Applicable.

## References

  *   * Bau . [2018] GANDissectionBau, D., Zhu, JY., Strobelt, H., Zhou, B., Tenenbaum, J.B., Freeman, W.T. Torralba, A.  2018\.  GAN Dissection: Visualizing and Understanding Generative Adversarial Networks. Gan dissection: Visualizing and understanding generative adversarial networks.  arXiv.  https://arxiv.org/abs/1811.10597
  * Brock . [2018] LargeScaleGANBrock, A., Donahue, J. Simonyan, K.  2018\.  Large Scale GAN Training for High Fidelity Natural Image Synthesis. Large scale gan training for high fidelity natural image synthesis.  arXiv.  https://arxiv.org/abs/1809.11096
  * W. Chen Ahmed [2020] padganChen, W. Ahmed, F.  202011\.  PaDGAN: Learning to Generate High-Quality Novel Designs Padgan: Learning to generate high-quality novel designs.  Journal of Mechanical Design143031703,  https://doi.org/10.1115/1.4048626
  * W. Chen . [2019] chen2019aerodynamicChen, W., Chiu, K. Fuge, M.  2019\.  Aerodynamic design optimization and shape exploration using generative adversarial networks Aerodynamic design optimization and shape exploration using generative adversarial networks.  AIAA Scitech 2019 Forum Aiaa scitech 2019 forum ( 2351).
  * W. Chen Fuge [2018] BézierGANChen, W. Fuge, M.  2018\.  BézierGAN: Automatic Generation of Smooth Curves from Interpretable Low-Dimensional Parameters. Béziergan: Automatic generation of smooth curves from interpretable low-dimensional parameters.  arXiv.  https://arxiv.org/abs/1808.08871
  * X. Chen . [2016] chen2016infoganChen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I. Abbeel, P.  2016\.  InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets Infogan: Interpretable representation learning by information maximizing generative adversarial nets.  arXiv preprint arXiv:1606.03657,
  * Donahue . [2017] donahue2017adversarialDonahue, J., Krähenbühl, P. Darrell, T.  2017\.  Adversarial Feature Learning. Adversarial feature learning.
  * Dumoulin . [2016] ALIDumoulin, V., Belghazi, I., Poole, B., Mastropietro, O., Lamb, A., Arjovsky, M. Courville, A.  2016\.  Adversarially Learned Inference. Adversarially learned inference.  arXiv.  https://arxiv.org/abs/1606.00704
  * I. Goodfellow [2017] tutorialGANGoodfellow, I.  2017\.  NIPS 2016 Tutorial: Generative Adversarial Networks. Nips 2016 tutorial: Generative adversarial networks.  arXiv.  https://arxiv.org/abs/1701.00160
  * I.J. Goodfellow . [2016] GoodBengCour16Goodfellow, I.J., Bengio, Y. Courville, A.  2016\.  Deep Learning Deep learning.  Cambridge, MA, USAMIT Press.  http://www.deeplearningbook.org
  * I.J. Goodfellow . [2014] GANGoodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S.Bengio, Y.  2014\.  Generative Adversarial Networks. Generative adversarial networks.  arXiv.  https://arxiv.org/abs/1406.2661
  * He . [2016] resNetHe, K., Zhang, X., Ren, S. Sun, J.  2016\.  Deep Residual Learning for Image Recognition Deep residual learning for image recognition.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016 ieee conference on computer vision and pattern recognition (cvpr) ( 770-778).
  * Heusel . [2018] TTURHeusel, M., Ramsauer, H., Unterthiner, T., Nessler, B. Hochreiter, S.  2018\.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in Neural Information Processing Systems 30 (NIPS 2017),  arXiv:1706.08500 [cs.LG]
  * Karras . [2018] karras2018progressiveKarras, T., Aila, T., Laine, S. Lehtinen, J.  2018\.  Progressive Growing of GANs for Improved Quality, Stability, and Variation. Progressive growing of gans for improved quality, stability, and variation.
  * Karras, Laine Aila [2019] karras2019styleKarras, T., Laine, S. Aila, T.  2019\.  A Style-Based Generator Architecture for Generative Adversarial Networks A style-based generator architecture for generative adversarial networks.  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Proceedings of the ieee conference on computer vision and pattern recognition.
  * Karras, Laine, Aittala . [2019] istyleGANKarras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J. Aila, T.  2019\.  Analyzing and Improving the Image Quality of StyleGAN. Analyzing and improving the image quality of stylegan.  arXiv.  https://arxiv.org/abs/1912.04958
  * Lecun . [1998] MNISTLecun, Y., Bottou, L., Bengio, Y. Haffner, P.  199811\.  Gradient-based learning applied to document recognition Gradient-based learning applied to document recognition.  Proceedings of the IEEE86112278–2324,  https://doi.org/10.1109/5.726791 Conference Name: Proceedings of the IEEE
  * Li . [2021] Li_2021Li, R., Zhang, Y. Chen, H.  2021oct.  Learning the Aerodynamic Design of Supercritical Airfoils Through Deep Reinforcement Learning Learning the aerodynamic design of supercritical airfoils through deep reinforcement learning.  AIAA Journal59103988–4001,  https://doi.org/10.2514/1.j060189 https://doi.org/10.25142F1.j060189
  * Luo . [2016] luo2016understandingLuo, W., Li, Y., Urtasun, R. Zemel, R.  2016\.  Understanding the effective receptive field in deep convolutional neural networks Understanding the effective receptive field in deep convolutional neural networks.  Advances in neural information processing systems Advances in neural information processing systems ( 4898–4906).
  * Mirza Osindero [2014] mirza2014cganMirza, M. Osindero, S.  2014\.  Conditional generative adversarial nets Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784,
  * Miyato . [2018] spectralNormalizationMiyato, T., Kataoka, T., Koyama, M. Yoshida, Y.  2018\.  Spectral Normalization for Generative Adversarial Networks. Spectral normalization for generative adversarial networks.  arXiv.  https://arxiv.org/abs/1802.05957
  * National Highway Traffic Safety Administration [1998] FMVSS201UNational Highway Traffic Safety Administration  1998\.  Federal Motor Vehicle Safety Standard No. 201U. Federal Motor Vehicle Safety Standard No. 201U.  TP201U-01
  * Pidhorskyi . [2020] ALAEPidhorskyi, S., Adjeroh, D.A. Doretto, G.  2020\.  Adversarial latent autoencoders Adversarial latent autoencoders.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Proceedings of the ieee/cvf conference on computer vision and pattern recognition ( 14104–14113).
  * Radford . [2015] dcganRadford, A., Metz, L. Chintala, S.  2015\.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. Unsupervised representation learning with deep convolutional generative adversarial networks.  arXiv.  https://arxiv.org/abs/1511.06434
  * Regenwetter . [2021] generativeDesignRegenwetterRegenwetter, L., Nobari, A.H. Ahmed, F.  2021\.  Deep Generative Models in Engineering Design: A Review. Deep generative models in engineering design: A review.  arXiv.  https://arxiv.org/abs/2110.10863
  * Roth . [2017] stabilizingGANRoth, K., Lucchi, A., Nowozin, S. Hofmann, T.  2017\.  Stabilizing Training of Generative Adversarial Networks through Regularization. Stabilizing training of generative adversarial networks through regularization.  arXiv.  https://arxiv.org/abs/1705.09367
  * Salimans . [2016] salimans2016inceptionSalimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A. Chen, X.  2016\.  Improved Techniques for Training GANs Improved techniques for training gans.  Advances in Neural Information Processing Systems. Advances in neural information processing systems.
  * Santurkar . [2018] howDoesBNworkSanturkar, S., Tsipras, D., Ilyas, A. Madry, A.  2018\.  How Does Batch Normalization Help Optimization? How does batch normalization help optimization?  arXiv.  https://arxiv.org/abs/1805.11604
  * Shen . [2020] shen2020interpretingShen, Y., Gu, J., Tang, X. Zhou, B.  2020\.  Interpreting the Latent Space of GANs for Semantic Face Editing. Interpreting the latent space of gans for semantic face editing.
  * Skarka [2007] Skarka2007ApplicationOMSkarka, W.  2007\.  Application of MOKA methodology in generative model creation using CATIA Application of moka methodology in generative model creation using catia.  Eng. Appl. Artif. Intell.20677-690,
  * Theis . [2015] DGMevaluationTheis, L., Oord, A.v.d. Bethge, M.  2015\.  A note on the evaluation of generative models. A note on the evaluation of generative models.  arXiv.  https://arxiv.org/abs/1511.01844
  * Torfi . [2020] torfi2020evaluationTorfi, A., Beyki, M. Fox, E.A.  2020\.  On the Evaluation of Generative Adversarial Networks By Discriminative Models. On the evaluation of generative adversarial networks by discriminative models.
  * University of Illinois at Urbana-Champaign [2022] UIUCAirfoilDBUniversity of Illinois at Urbana-Champaign  2022\.  UIUC Airfoil Database. UIUC Airfoil Database.  http://m-selig.ae.illinois.edu/ads/coord_database.html.  Accessed on 18.10.2022
  * Vaswani . [2017] AttentionIsAllYouNeedVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N.Polosukhin, I.  2017\.  Attention Is All You Need. Attention is all you need.  arXiv.  https://arxiv.org/abs/1706.03762
  * Wu . [2019] LoGANWu, Y., Donahue, J., Balduzzi, D., Simonyan, K. Lillicrap, T.  2019\.  LOGAN: Latent Optimisation for Generative Adversarial Networks. Logan: Latent optimisation for generative adversarial networks.  arXiv.  https://arxiv.org/abs/1912.00953
  * Zhang . [2018] Self-AttentionGANZhang, H., Goodfellow, I., Metaxas, D. Odena, A.  2018\.  Self-Attention Generative Adversarial Networks. Self-attention generative adversarial networks.  arXiv.  https://arxiv.org/abs/1805.08318