11institutetext: University of Coimbra, CISUC/LASI – Centre for Informatics
and Systems of the University of Coimbra, Department of Informatics
Engineering, Portugal
11email<EMAIL_ADDRESS>
22institutetext: Department of Computer Engineering, Automatics, and Robotics
and CITIC, University of Granada, Spain
22email<EMAIL_ADDRESS>
# Evaluation Metrics for Automated Typographic Poster Generation††thanks: This
paper is based upon work from a scholarship supported by SPECIES, by the
Foundation for Science and Technology, I.P./MCTES through national funds
(PIDDAC), within the scope of CISUC R&D Unit - UIDB/00326/2020 or project code
UIDP/00326/2020, and by Ministerio español de Economía y Competitividad
(Spanish Ministry of Competitivity and Economy) under project
PID2020-115570GB-C22 (DemocratAI::UGR).

Sérgio M. Rebelo 11 0000-0002-7276-8727 J. J. Merelo 22 0000-0002-1385-9741
João Bicker 11 0000-0002-0670-1217 Penousal Machado 11 0000-0002-6308-6484

###### Abstract

Computational Design approaches facilitate the generation of typographic
design, but evaluating these designs remains a challenging task. In this
paper, we propose a set of heuristic metrics for typographic design
evaluation, focusing on their legibility, which assesses the text visibility,
aesthetics, which evaluates the visual quality of the design, and semantic
features, which estimate how effectively the design conveys the content
semantics. We experiment with a constrained evolutionary approach for
generating typographic posters, incorporating the proposed evaluation metrics
with varied setups, and treating the legibility metrics as constraints. We
also integrate emotion recognition to identify text semantics automatically
and analyse the performance of the approach and the visual characteristics
outputs.

###### Keywords:

Computational Creativity Design Measures Evolutionary Design Graphic Design
LayoutPoster Design

##  1 Introduction

Computational Design has revealed a significant potential to transform
practices in the field of Graphic Design (GD), including Typography and Layout
domains [7, 18]. This potential involves automating typesetting tasks (e.g.
applying text styles, creating tables, etc.) and facilitating design
exploration. Nonetheless, assessing the outcomes of these computational
processes remains a challenging task, given that their evaluation relies on
subjective design factors.

In this paper, we present a set of ten heuristic metrics for evaluating
typographic designs. These metrics consist of the application of design rules
that enable automated assessment of various characteristics of typographic
designs, especially posters. Our motivation behind developing these metrics is
to streamline and facilitate computational typesetting processes, ultimately
leading to faster and more efficient GD practices. The proposed set of metrics
encompasses the evaluation of designs in terms of their legibility, aesthetic
features, and coverage of content semantics. Legibility metrics determine
whether all text content is adequately displayed and readable within the
design, including the evaluation of text legibility and grid appropriateness.
Aesthetic metrics assess the visual quality of the designs, examining aspects
like alignment, balance, justification, regularity, typeface pairing, and
negative space fraction. Semantic metrics focus on assessing how the composed
text effectively conveys the semantic meaning of the content in terms of
layout and typography.

We study the practical application of these metrics in guiding the creation of
typographic poster designs by developing an Evolutionary Computation (EC)
approach. In this context, we considered that the primary goal of posters is
to fully display their content. This way, we explored a constrained
evolutionary methodology where legibility metrics are treated as constraints
that the generated outputs must satisfy, while other metrics define the
objective value of designs. This approach is inspired by the workflow of
traditional typography design processes, as conducted in nineteenth-century
print houses. Back then, typographers employed an algorithmic method for
typesetting content to fill all the space in a matrix. They use condensed
typefaces for lengthy sentences and extended typefaces for shorter ones, while
also emphasising the most significant parts of the content typographically
[12].

Furthermore, we designed a user interface to support the developed approach,
enabling users to input text content and specify the desired visual features
of the outputs. To facilitate the generative process, we develop procedures to
automatise the text division and to recognise the more semantically
significant parts of the content using emotional recognition. The development
of this approach follows an agile science methodology, structured around
potential user cases and scenarios for the application of the proposed metrics
in poster design [13]. The code repository for this project is accessible at
github.com/sergiomrebelo/evo-poster. Supplementary materials are available at
cdv.dei.uc.pt/projects/evoposter (websites visited: 8 November 2023).

We conducted experiments to examine the influence of the proposed evaluation
metrics on the evolutionary generation of typographic poster designs. These
experiments involved the legibility-constrained evolution of typographic
posters using input texts of varying lengths and emotional content. The
experiments consisted of three stages, each focusing on the evolution of
posters based on either semantic metrics, aesthetic metrics, or a combination
of both. The results demonstrate that the proposed metrics effectively guide
an evolutionary process, producing finished and legible designs from a variety
of text inputs while considering both aesthetics and semantics.

The primary contribution of this paper is the set of metrics for evaluating
typographic designs. Other prominent contributions include (i) a functional
constrained evolution approach for creating typographic posters; (ii) a multi-
purpose, domain-driven, and easily understandable representation of poster
designs; (iii) an investigation into weighted objective function strategies
for evaluated designs based on aesthetics and semantics; (iv) an exploration
of integrating computational design into a poster design field.

The remainder of the paper is organised as follows. The next section provides
a review of the related work, specifically focusing on design metrics. The
third section comprehensively describes the metrics. The fourth section
explains our experimental approach, the setup, and the results. The last
section summarises our contributions and outlines potential directions for
future research.

##  2 Related Work

Heuristic metrics and measures for visual assessment are frequently used in
the computational generation of visuals, especially in the context of EC.
These metrics have been explored to overcome the limitations of cooperative
evolutionary approaches, like Interactive Evolutionary Computation strategies,
allowing the incorporation of subjective human-related data into the
evolutionary design processes. However, they can lead to user fatigue and
inconsistent evaluations [11].

Typography and Layout metrics are employed for either generative or
optimisation objectives. Geigel and Loui [5] employed a set of design
measures, including page balance, spacing, and emphasis, to automatically
evolve page layouts for photography albums. Harrington et al.[6] proposed a
non-linear layout evaluation measure, combining a set of heuristic metrics for
document design, such as alignment, regularity, separation, balance, white-
space fraction, white-space flow, proportion, uniformity, and page security.
Building upon these measures, Purvis et al [15] and Rebelo et al. [16]
developed generative evolutionary approaches, adapting the proposed measures
for the characteristic of the outputs. Lok et al. [8] introduced a technique
for computing visual balance using lightness weight maps from images. Balinsky
et al. [1] presented measures for determining page alignment and regularity,
based on the extraction and quantification of “alignment lines.” Bylinskii et
al. [3] and Xie et al [21] developed Machine Learning (ML) models which
unveiled some potential for auto-completion and layout retrieval. More
recently, Lopes et al. [9] developed a pixel-based approach to assess the
balance of design.

The reviewed related work primarily focuses on visual attributes like balance,
regularity, and alignment while overlooking typographic attributes.
Nevertheless, these typographic attributes are an essential consideration for
graphic designers during their creative processes. Factors like typography
pairing and content emphasis significantly contribute to the distinctiveness
and effectiveness of the conveyed message. Recent ML approaches show a
promising manner to evaluate the designs. However, their effectiveness in
poster design scenarios is constrained by the limited availability of poster
layout data.

##  3 Metrics

We introduced a set of ten metrics to evaluate typographic designs,
specifically posters that feature short text messages. These metrics aim to
address the lack of consideration of typography and representation of content
semantics in current work on computational assessment of visual designs.

The computation of these metrics assumes that designs consist of text boxes,
each representing a single line of text. These text boxes are characterised by
attributes such as size, alignment, and font. They are organised sequentially
within a one-column grid. In addition, typefaces used must provide data
related to the category, e.g. serif, mono-space, sans-serif, etc. Certain
metrics require quantifying the emotional charge of each text line. An optimal
poster layout is also considered, which describes the expected distribution of
each text box in terms of percentages of the poster’s height based on the
emotional analysis.

The metrics are evaluated on a scale ranging from 0 (poor score) to 1 (perfect
score), and they can be divided into three evaluation objectives: legibility;
aesthetics; and semantics. They have been implemented as a standalone module,
and the source code is available in the project repository.

Legibility metrics evaluate whether the text content on a poster is fully
visible in the design. This objective includes two distinct metrics: (i) the
text legibility and (ii) the grid appropriateness.

The text legibility metric assesses whether all text content is fully visible
within the text boxes that compose the poster. It is calculated by the
arithmetic mean of the legibility score of all text boxes. The legibility
score of each text box is the variance between the width of the text box’s
content, as rendered, and the available width of the design (i.e. the width of
the poster without the horizontal margins). This score is then normalised,
gradually prejudicing the cases when the text exceeds the available width. A
score of 0 indicates that the text width doubles the width of the container,
while a value of 1 indicates that the text fits within the space.

The grid appropriateness metric assesses whether the grid used in the design
is suitable. It does so by comparing the size of the grid (both width and
height, including the margins) with the size of the poster. This metric can
only have two values. A value of 1 indicates that the grid entirely fits
within the design dimensions, while a value of 0 suggests that it does not.

Aesthetic metrics are used to evaluate the visual and typographic features of
a design. These metrics include (iii) alignment, (iv) balance, (v)
justification, (vi) regularity, (vii) typeface pairing, and (viii) negative
space fraction. We defined these metrics based on works of Harrington et al.
[6] and typographic principles outlined by Lupton [10] and Bringhurst [2].

The alignment metric assesses the consistency of the horizontal alignment of
text. The estimation involves two main steps. First, it computes the variance
in text width between neighbouring text boxes. Then, it calculates the
arithmetic mean of these variances $(d)$, using the non-linear $A/(A+d)$ where
$A$ is a constant that controls how fast values fall away from 1 as the
distance between entries increases. Second, the metric checks if the line
alignment of the text box is uniform. This is determined by the division of 1
(the expected line alignments on the design) by the count of different line
alignments identified on the posters. The overall alignment score is a
weighted average, with width variance contributing 80% and text alignment
contributing 20%. This score is subsequently normalised within a range from 0
(high width variance and different line alignments) to 1 (low width variance
and consistent use of a single line alignment).

The estimation of balance metric initially involves the calculation of the
visual weights and the balance centres of each text box which compose it. The
visual weight of a text box $(vw)$ is defined by multiplying its area by its
optical density. The optical density $(oD)$ is calculated by considering the
relative Luminance, which is computed on the average pixel values for the red
$(r)$, green $(g)$, and blue $(b)$ colour channels within the text box, using
the formula:

$oD=\log_{10}\left(\frac{1}{n}\sum_{n}^{t=1}0.2126\times r_{t}+0.7152\times
g_{t}+0.0722\times b_{t}\right)$ (1)

The balance centre of a text box $(x_{t},y_{t})$ is determined based on its
line alignment $(x_{t})$ and the overall vertical alignment of the poster. For
instance, if the text box is left-aligned and vertically aligned to the top,
its balance centre $(x_{t},y_{t})$ is the upper-left corner. When the vertical
alignment is set to centre, the vertical balance centre position is visually
adjusted moving one-twelfth towards the top. Next, it calculates the centre of
the visual weight of the entire design $(w_{x},w_{y})$, by considering all
text boxes $(t)$ within it as:

$w_{x}=\left(\frac{\sum_{t=1}^{n}x_{t}\times
vw_{t}}{\sum_{t=1}^{n}vw_{t}}\right)\>\text{and}\>w_{y}=\left(\frac{\sum_{t=1}^{n}y_{t}\times
vw_{t}}{\sum_{t=1}^{n}vw_{t}}\right)$ (2)

Subsequently, it estimates the expected balance centre of the design
$(c_{x},c_{y})$ employing the same method as used for text boxes, considering
the alignment of the first text box $(c_{x})$ and the same global vertical
alignment $(c_{y})$, albeit taking into consideration the full poster size.
Finally, the overall balance $(B)$ score is determined considering the
calculated current and expected balance centres and poster sizes as follows:

$B=1-\left[\left(\left(\frac{w_{x}-c_{x}}{width}\right)^{2}+\left(\frac{w_{y}-c_{y}}{height}\right)^{2}\right)/2\right]^{\frac{1}{2}}$
(3)

The justification metric evaluates whether the text fully occupies the
available space. This metric is inspired by the traditional aesthetics of
nineteenth-century letterpress posters where text content was traditionally
justified within the available space, ensuring that text occupies all the
available areas when possible. The calculation of this metric is similar to
the text legibility. The overall justification score is determined as the
arithmetic mean of the justification scores of all text boxes. The
justification score of a text box is calculated by considering the variance
between the text width of each text box and the available space. The variance
value is normalised penalising both designs with text overflow and with
excessive white space. The penalty is lower when text fully fits within the
poster, being divided by a factor. A lower variance results in a higher
justification score. A justification score of 1 indicates zero width variance,
while if the variance doubles the design width the justification is set to 0.

The regularity metric evaluates how regular is text box heights in the design.
It measures the distances $(d)$ between the vertical positions of the top
edges in neighbouring text boxes using a non-linear function $A/(A+d)$, where
$A$ is a constant that determines the rate at which values decrease as the
distance between entries increases, similar to alignment. The overall
regularity score is the arithmetic mean of the value of all neighbouring pairs
is normalised within a range between 0 (low regularity) and 1 (high
regularity).

The typeface pairing metric evaluates the compatibility of the typefaces used
on the poster, considering their categories on typographic classification. It
begins to create a unique list of categories from the typefaces used in the
design. The overall typographic pairing score is determined by the count of
categories found in the design, normalised between 1 (if there is only one
category) and 0 (if all the used typefaces are from different categories).

The negative space fraction metric assesses the appropriateness of the
percentage of background colour in the design. It calculates the current
percentage of the poster occupied by background colours and computes the
deviation from an optimal value chosen by the user. The differences are then
normalised to a range between 0 (twice the optimal background percentage) and
1 (optimal background percentage).

Semantic metrics assess whether the placement of the typography in the designs
conveys the semantic meaning of the content. This analysis encompasses the
evaluation of the semantic significance of the (ix) layout, and (x)
typography.

The semantic significance of the layout metric evaluates whether text lines
which contain more emotional charge are appropriately highlighted in the
layout. The greater the emotional charge on a line, the more height this line
of text should be typeset on the design. This calculation is done at the line
(or text box) level. So, for each text box, it calculates the distance between
its height and the height in the optimal layout, represented as a percentage
of the poster’s total height. This metric can operate in two modes, “Fixed”
and “Relative”. In the “Fixed” mode, it considers that content must fulfil the
total poster available height. In “Relative” mode, it only considers the
current height of the composition, ignoring the empty space. By default, it
operates in “Fixed” mode. The overall score for the semantic significance of
the layout is the arithmetic mean of the normalised distance of all text
boxes, ranging from 0 (significant differences between the actual and optimal
layout) to 1 (perfect match between the optimal and current layout).

The semantic significance of typography metric evaluates if the most emotional
parts of the content are typographically emphasised. This involves calculating
variations in weight, stretch, and type design across fonts used in the text
boxes within the design. The score for the variable typographic features
(weight and stretch) is computed as the mean of the distances between the
current and expected values. These distances are normalised between 0 (maximum
distance) and 1 (no distance). To determine the expected values for each text
box, the range of values used in the design is established. Then, for each
text box, an expected distance is assigned based on the recognised emotional
levels in the content. The text box with the highest emotional context
recognised is assigned the maximum distance value within the range, while the
one with lesser emotional charge receives no distance. Ultimately, for each
text box, the distance between the current value and the value of a text box
with no expected difference is calculated, and the current distance is
determined by comparing the results with the assigned expected distance. The
type design score is calculated by counting the number of typefaces that
deviate from the expected ones, normalised based on the total number of text
boxes. A score of 0 indicates that the output aligns with type design
expectations, whereas a score of 1 signifies that it does not meet any
expectations. To calculate the expected typefaces, typefaces are initially
assigned to emotional levels based on their first use within each emotional
level. Each typeface is uniquely associated with one emotional level. If a
typeface is already assigned to another level, the typeface for this level is
set as undefined. The overall score of the semantic significance of typography
is the highest score among the three types of variations. To prevent excessive
emphasis on the same content, the score is penalised when multiple features
receive evaluations exceeding a high threshold. In such cases, the score is
divided by the number of features that exceeded this threshold.

##  4 Experiments

We conduct evolutionary experiments using the proposed metrics and a Genetic
Algorithm to automatically evolve typographic posters using different texts.
We experiment with a constrained evolution approach employing a stochastic
ranking method [19] to fitness assignment the designs. Legibility metrics
serve as constraints that the generated outputs must adhere to. Aesthetic and
semantic metrics determine the design value of the outputs, through a weighted
multi-criteria objective evaluation function. In addition, we implement a
constraint penalty approach to facilitate the sorting and visualisation of
population and elitism. The penalty is calculated as the inverted weighted
arithmetic mean of the two legibility metrics.

We adopted a Domain-Driven Design [4] approach to define the representation of
posters. Each genotype is encoded in the JSON format, with each
representational attribute identified by a key term commonly used in the GD
domain. The genotype comprises two parts: (i) text boxes data and
characteristics, namely content, typeface, weight, stretch, size, and
alignment; and (ii) poster characteristics, specifying the size (width and
height), margins (in percentages of size), and vertical alignment. The grid of
the poster is inferred based on the text box size and the poster
characteristics. One descriptive example of a genotype is provided in
supplementary materials.

The generative process starts with the random initialisation of a poster
population. For each line of text, a text box is assigned, and its
characteristics are randomly determined from available choices. This process
involves randomly selecting a typeface, along with the weight and stretch
based on its available options. The font size is defined within a predefined
range, while alignment is randomly set to left, centre, or right. The vertical
alignment of the poster is also randomly defined as top, middle, or bottom.

Tournament selection is used to choose individuals for breeding based on their
fitness, and variation operators (mutation and crossover) are applied to the
selected individuals, with a certain probability, to generate new offspring.
We use a uniform crossover method [20], which randomly determines which parent
will pass down genetic material to the descendant. It flips the vertical
alignment gene and all text boxes on the new individual to determine which
parent will pass down its characteristics to the offspring. The text boxes are
passed to the descendants with all of their characteristics.

The mutation variation operator determines if an attribute of the individual
in new offspring will undergo change, based on a certain probability. The
mutation method employed varies depending on the selected attribute. When
poster vertical alignment or the alignment, weight, and stretch attributes of
a text box are selected for mutation, the current value is modified by one of
the available options. If the typeface attribute of the text box is selected,
it replaces the current typeface with one of the available options, and it
adjusts the weight and stretch attributes if needed, selecting values close to
the current ones. In the case of the font size of the text box, it adjusts the
size based on random values within a specific range, typically between -5 and
5.

We employ an elitist process, where the best individuals from the current
generation are combined with the new offspring. To select these individuals,
it sorts the population using a penalty constraint approach.

Before the evolution, the system emotionally analyses input content to
determine the parts that must be emphasised on the layout. The analysis starts
with the preprocessing of the text by (i) handling contracted word forms, (ii)
replacing abbreviations and slang expressions with formal equivalents, (iii)
substituting emojis with their actual meanings, (iv) changing negations to
antonyms, (v) eliminating stop words and URLs, and (vi) tokenising the text.
Subsequently, it conducts a lexicon-based analysis using a word-emotion
association lexicon [14]. Based on the result, emotional scores are assigned
to individual words. The emotional scores for each line are calculated by
summing the scores of words on it.

We designed a user interface that allows users to input the text and configure
the desired visual features of the outputs and the evolutionary approach.
Furthermore, new typefaces, options, and default evolution parameters through
a dedicated configuration file. Typefaces are incorporated into the system in
the OpenType Variable Font format. Users can input content either divided into
lines or choose to let the system automatically divide the text. Text division
is performed using a Sentence Boundary Detection algorithm [17] to split the
text into sentences and then divide the longer sentences based on a random
factor within a predetermined line size range. Screenshots of the developed
interface are available in the supplementary material.

Table 1: Metrics weights in the respective evaluation objective. Aesthetic metrics | Semantic metrics
---|---
Alignment | 10% | Layout | 50%
Regularity | 10% | Typography | 50%
Balance | 20% |  |
Negative Space Fraction | 20% |  |
Justification | 30% |  |
Typeface Pairing | 10% |  |
Table 2: Experimental Parameters. Type | Parameter | Value
---|---|---
Evolutionary | Generations | 400
Population size | 30
Elite size | 1
Crossover probability | 90%
Mutation probability | 10%
| Tournament size | 10
Posters’ features | Poster size (px) | $141\times 100$
Default margins (ltrb) | 5% 5% 5% 5%
Min colour contrast | 2.5
Aesthetic metrics | Alignment/regularity A constant | 10
Justification factor | 3
| Optimal negative space factor | 50%
Semantic metrics | Significance of layout mode | Fixed
Significance of typography threshold | 0.2

###  4.1 Experimental Setup

The conducted experiments are divided into three different stages: (S1)
evolving only considering semantic objective; (S2) evolving only considering
aesthetic objective; and (S3) evolving considering both objectives, with equal
weight on objective evaluation. The score of each evaluation objective is
determined as the weighted arithmetic mean of the score of metrics that
compose it. The weight of each metric on the objective function is defined
empirically, with the goal of creating poster designs inspired by the
aesthetics of nineteenth-century letterpress posters, which was one of the
main motivations for the development of this work. Table 1 displays the weight
assigned to each metric in both the semantic and aesthetic groups.

The conducted experiments occur on the client side of a Chromium web browser.
We loaded 8 OpenType Variable Font, all of which have weight and stretch axes
available. The experimental parameters for these experiments were empirically
defined and are summarised in Table 2. In the three evaluation stages, we
generated typographic posters for 20 different texts in three languages
(English, Portuguese, and French). These texts expressed various purposes,
emotional contexts, and lengths. On average, each input text contained 39.55
characters, divided into 7.15 words, with a maximum of 83 characters and a
minimum of 17. The average text line length was 4.15 lines. The selected
texts, apart from being neutral, were classified by representing emotions such
as anticipation, disgust, joy, sadness, surprise, and trust. Further
description of the text and the typographic available options are provided in
the supplementary material.

###  4.2 Experimental Results

The experimental results demonstrate that the proposed metrics and approach
enable the evolution of legible designs for various types of text inputs.
Analysing the progression of constraint penalty in the population (see Fig.
1), we observed that the proposed approach leads to a faster reduction in the
number of individuals that do not comply with the legibility constraints,
either in the best individuals or the population average. Individuals
evaluated with maximum fitness are present in the population but do not are
legible. A detailed analysis of grid appropriateness metric reveals that the
proposed approach can consistently generate and maintain individuals that use
valid grids. On the other hand, the evolution of text legibility is inversely
proportional to the constraint penalty value. The results also reveal that the
fittest individuals at the end of evolution are always individuals with all
text legible.

Figure 1: Progression of constraint penalty and legibility metrics across
generations. Visualised results present the average of three stages, totalling
1,800 runs. Black lines depict penalty constraint values, while pink and blue
lines show the scores of text legibility and grid appropriateness metrics,
respectively. Fittest individuals are depicted with solid lines, and the
average population with shaded lines.

The evolution based on semantic metrics (S1) reveals that the proposed
semantic metrics can guide the evolution of posters towards designs where the
more emotional parts of the content are highlighted. The experimental results
(see Fig. 2) unveil that the best individual in the population achieves high
fitness values as well as high scores in both metrics. However, the metrics
values stabilise after a dozen generations, primarily due to the legibility
constraints. These constraints restrict the semantic evolution of certain
individuals, as they compromise legibility. In this sense, some individuals
with high semantic scores are also not fully legible designs. It is also noted
that the evolution of the semantic significance of layout is slightly faster
than the typography.

Figure 2: Evolutionary progress in the S1 experiment. The results are averages
from 600 runs, considering varied text inputs. The chart on the left
illustrates best individual fitness (solid lines), constraint penalty (dashed
lines) and maximum fitness (dotted). On the right, the chart shows score
metrics related to the significance of semantics in layout (green lines) and
typography (orange lines). Solid lines represent the average of the fittest
individuals, while shaded lines represent the average of the entire
population.

The generated outputs exhibit certain parts of the content that are emphasised
in terms of layout, typography, or both (see Fig. 3(a)). This emphasis is
primarily directed toward the more emotional sections of the content. However,
when the emphasis affects both layout and typography, it can sometimes become
excessive, leading to a visually heavy impact, particularly in designs with
fewer emotional variations. Additionally, certain design issues were observed
in the outputs, including non-uniform alignment, unbalanced layout, or
excessive use of different typefaces.

(a) S1 Stage

(b) S2 Stage

(c) S3 Stage

Figure 3: Examples of designs evolved during the three experimental stages.
More results can be found in the supplementary material folder.

The outcomes of the aesthetics evolution (S2) indicate that it is possible to
evolve designs complying with the proposed aesthetic metrics. Nonetheless,
these results also reveal the challenge of evolving designs that fulfil all
the metrics simultaneously. The performance of metrics depends on the text
content, and the necessity for legibility limits the range of aesthetic design
possibilities.

Experimental results (see Fig. 4) reveal that global fitness increases faster
in the initial generations but slows down as the constraint penalty decreases.
We noted that some individuals in the population achieve higher than the best
individuals in the population, but they are not legible posters. When
examining the score of each metric individually, regularity, alignment, and
typographic pairing progressively increase over the generation both in the
fittest individuals and population average. This suggests that enhancing these
metrics does not compromise the legibility of the posters. On the other hand,
balance, justification, and negative space fraction maintain relatively stable
scores over the generations, with slight decreases in scores on the best
individuals, while legibility and other aesthetic metrics increase. The
justification score of the best individual was even, on average, below the
population average. Further experimentation is necessary to understand how
other metrics and experimental parameters impact their evaluation. However,
high scores on these metrics result in the creation of invalid individuals,
even in more advanced generations, who are subsequently removed from the
population. Due to this, legibility metrics had worse scores than in the S1
stage.

Figure 4: Evolutionary progress in the S2 experiment. The results are averages
from 600 runs, considering multiple text inputs. The chart on the left
illustrates best individual fitness (solid lines), constraint penalty (dashed
lines) and maximum fitness (dotted). On the right, the chart shows score
metrics related to aesthetics, including alignment (navy), regularity (green),
justification (violet), typography pairing (pink), balance (turquoise), and
negative space fraction (orange). Solid lines represent the average of the
fittest individuals, while shaded lines represent the average of the entire
population.

The visual analysis of the outputs (see Fig. 3(b)) unveils that, even though
they are created with the same input texts, the generated posters are more
balanced and regular in typographical terms compared with S1 outputs.
Nonetheless, we noticed that in some outputs less meaningful parts of the
content are typographically highlighted (e.g. conjunctions), making the
content more difficult to read.

The results of the evolution combining both aesthetic and semantic metrics
(S3) indicate that optimising all the metrics simultaneously is a challenge,
being the design possibilities influenced by the text content and the
recognised emotional charge. The experimental results (see Fig. 5) exhibit an
evolutionary pattern akin to the two other experiments: faster evolution in
early generations followed by gradual stabilisation as the constraint penalty
values decrease. However, we noted that there are individuals with higher
fitness values in the population, yet they are not legible.

Figure 5: Progression of evolution in the S3 experiment. The results are
averages from 600 runs, considering various text inputs. The top-left chart
presents the progression of the best individual fitness (solid lines),
constraint penalty (dashed lines) and maximum fitness (dotted). The top-right
and bottom charts illustrated the scores of metrics related to semantics and
aesthetics, respectively. Semantic metrics include the significance of
semantics in layout (green) and typography (orange). Aesthetic metrics include
alignment (navy), regularity (green), justification (violet), typography
pairing (pink), balance (turquoise), and negative space fraction (orange).
Solid lines represent the average of the fittest individuals, while shaded
lines represent the average of the entire population.

The analysis of the evolution of semantic metrics separately reveals that
their combined evaluation with aesthetic metrics presents challenges to
evolving semantic metrics, being constrained by the other aesthetic metrics.
In terms of the significance of layout, this experiment indicates that it is
restricted by aesthetic metrics, showing a relatively flat trajectory, with
the fittest individuals and the population average achieving similar values.
The evolution of the significance of typography, although exhibiting some
improvement in the early generations, it faster stabilises and remains below
the scores achieved in the S1 experiment. The results of aesthetic metrics
resemble the results of S2. Alignment, regularity, and typographic pairing
progressively increase over the generations, while balance, justification, and
negative space fraction maintain relatively stable scores, seemingly
constrained in their progress by legibility metrics. We observed a slight
evolution of justification scores, which appear to be related to the inclusion
of semantic metrics and the exploration of layouts that are not highly valued
by other aesthetic metrics but, from a semantic perspective, are considered
superior and enable improved content justification.

The empirical visual analysis of the outputs (see Fig. 3(c)) reveals an
increase in aesthetic features value present, compared to S1. Simultaneously,
it is possible to observe the emphasis of parts with more emotional weight,
whether through layout or font selection, in comparison to S2. However, this
emphasis in certain designs should be more pronounced in some posters.

##  5 Conclusions and Future Work

Computational approaches in GD unveil potential, but evaluating outcomes
remains challenging due to reliance on subjective factors. We introduced ten
heuristic metrics for autonomously evaluating typographic designs, addressing
both visual and semantic aspects. These metrics are categorised into three
evaluation objectives: Legibility metrics assess text display effectiveness,
including text legibility and grid appropriateness. Aesthetic metrics evaluate
visual quality, considering alignment, balance, justification, regularity,
typeface pairing, and negative space fraction. Semantic metrics focus on
typographic content representation for layout and typography. We applied these
metrics in a constrained evolution approach to generate typographic poster
designs for various texts, with legibility metrics as constraints and other
metrics in a weighted multi-criteria objective evaluation function. Our
preliminary experiments covered three stages, evaluating posters using
semantic, aesthetic, and both metrics.

The main motivation behind developing these metrics is to streamline
computational typesetting processes. Despite their preliminary nature,
evaluation experiments reveal their effectiveness in guiding typographic
poster design generation for diverse text contents, highlighting their
potential for automation in GD practices. However, evolving multiple metrics
simultaneously, in this constrained setting, restricts metrics’ progression,
necessitating further research to understand the individual metrics’
progression and their interaction with constraints. Future work also includes
less constrained evolutionary experiments with different parameters. Moreover,
we intend to conduct further evaluation studies with human participants,
aiming to assess the visual impact of the proposed metrics in output designs.

## References

  * [1] Balinsky, H.Y., Wiley, A.J., Roberts, M.C.: Aesthetic measure of alignment and regularity. In: Borghoff, U.M., Chidlovskii, B. (eds.) DocEng ’09: Proceedings of the 9th ACM symposium on Document Engineering. pp. 56–65. ACM (9 2009). https://doi.org/10.1145/1600193.1600207
  * [2] Bringhurst, R.: The Elements of Typographic Style. Hartley & Marks Publishers, 2nd edn. (1997)
  * [3] Bylinskii, Z., Kim, N.W., O’Donovan, P., Alsheikh, S., Madan, S., Pfister, H., Durand, F., Russell, B., Hertzmann, A.: Learning visual importance for graphic designs and data visualizations. In: Gajos, K., Mankoff, J., Harrison, C. (eds.) Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. pp. 57–69. ACM (10 2017). https://doi.org/10.1145/3126594.3126653
  * [4] Evans, E.: Domain-Driven Design Reference: Definitions and Pattern Summaries. self-publishing (2015), https://www.domainlanguage.com/ddd/reference, (visited: 18 January 2024)
  * [5] Geigel, J., Loui, A.C.P.: Automatic page layout using genetic algorithms for electronic albuming. In: Beretta, G.B., Schettini, R. (eds.) Internet Imaging II. vol. 4311, p. 79 – 90. SPIE (2000). https://doi.org/10.1117/12.411879
  * [6] Harrington, S.J., Naveda, J.F., Jones, R.P., Roetling, P., Thakkar, N.: Aesthetic measures for automated document layout. In: Munson, E.V., Vion-Dury, J.Y. (eds.) DocEng ’04: Proceedings of the 2004 ACM Symposium on Document Engineering. pp. 109–111. ACM (10 2004). https://doi.org/10.1145/1030397.1030419
  * [7] Levin, G., Brain, T.: Code as Creative Medium: A Handbook for Computational Art and Design. The MIT Press (2021)
  * [8] Lok, S., Feiner, S., Ngai, G.: Evaluation of visual balance for automated layout. In: Vanderdonckt, J., Nunes, N.J., Rich, C. (eds.) Proceedings of the 9th international conference on Intelligent user interfaces. pp. 101–108. ACM (1 2004). https://doi.org/10.1145/964442.964462
  * [9] Lopes, D., Correia, J., Machado, P.: Towards the automatic evaluation of visual balance for graphic design posters. In: Pease, A., Cunha, J.M., Ackerman, M., Brown, D.G. (eds.) Proceedings of the 14th International Conference on Computational Creativity, ICCC 2023, Waterloo, June 19-23, 2023. pp. 192–199. Association for Computational Creativity (6 2023)
  * [10] Lupton, E.: Thinking with Type: A Critical Guide for Designers, Writers, Editors, & Students. Princeton Architectural Press, 2nd edn. (2014)
  * [11] Machado, P., Romero, J., Manaris, B.: Experiments in Computational Aesthetics, pp. 381–415. Springer Berlin Heidelberg (2008). https://doi.org/$10.1007/978-3-540-72877-1_{1}8$
  * [12] Meggs, P.B., Purvis, A.W.: Meggs’ History of Graphic Design. John Wiley & Sons, Inc., 6th edn. (2016)
  * [13] Merelo, J.J.: Agile (data) science: A (draft) manifesto. arXiv preprint arXiv:2104.12545 (7 2022). https://doi.org/10.48550/arXiv.2104.12545, (visited: 18 January 2024)
  * [14] Mohammad, S., Turney, P.: Emotions evoked by common words and phrases: Using mechanical turk to create an emotion lexicon. In: Inkpen, D., Strapparava, C. (eds.) Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text. pp. 26–34. ACL (6 2010)
  * [15] Purvis, L., Harrington, S., O’Sullivan, B., Freuder, E.C.: Creating personalized documents: an optimization approach. In: Proceedings of the 2003 ACM Symposium on Document Engineering. pp. 68–77. ACM (8 2003). https://doi.org/10.1145/958220.958234
  * [16] Rebelo, S.M., Martins, T., Bicker, J., Machado, P.: Exploring automatic fitness evaluation for evolutionary typesetting. In: Sas, C., Maiden, N.A.M., Bailey, B.P., Latulipe, C., Do, E.Y.L. (eds.) C&C ’21: Creativity and Cognition Virtual Event Italy June 22 - 23, 2021. ACM (6 2021). https://doi.org/10.1145/3450741.3465247, (Article no. 12)
  * [17] Reynar, J.C., Ratnaparkhi, A.: A maximum entropy approach to identifying sentence boundaries. In: Grishman, R. (ed.) Proceedings of the fifth conference on Applied natural language processing. pp. 16–19. ACL (3 1997). https://doi.org/10.3115/974557.974561
  * [18] Richardson, A.: Data-driven Graphic Design: Creative Coding for Visual Communication. Bloomsbury Publishing Plc (2016)
  * [19] Runarsson, T.P., Yao, X.: Stochastic ranking for constrained evolutionary optimization. IEEE Transactions on Evolutionary Computation 4, 284–294 (9 2000). https://doi.org/10.1109/4235.873238
  * [20] Syswerda, G.: Uniform crossover in genetic algorithms. In: Schaffer, J.D. (ed.) Proceedings of the 3rd International Conference on Genetic Algorithms. pp. 2–9. Morgan Kaufmann Publishers Inc. (6 1989)
  * [21] Xie, Y., Huang, D., Wang, J., Lin, C.Y.: Canvasemb: Learning layout representation with large-scale pre-training for graphic design. In: MM’21: Proceedings of the 29th ACM International Conference on Multimedia. pp. 4100–4108. ACM (10 2021). https://doi.org/10.1145/3474085.3475541

  *[GD]: Graphic Design
  *[EC]: Evolutionary Computation
  *[ML]: Machine Learning
  *[JSON]: JavaScript Object Notation


# Soft Partitioning of Latent Space for Semantic Channel Equalization

Tomás Hüttebräucker, Mohamed Sana, Emilio Calvanese Strinati  CEA-Leti,
Université Grenoble Alpes, F-38000 Grenoble, France
Email : {tomas.huttebraucker; mohamed.sana<EMAIL_ADDRESS>
###### Abstract

Semantic channel equalization has emerged as a solution to address language
mismatch in multi-user semantic communications. This approach aims to align
the latent spaces of an encoder and a decoder which were not jointly trained
and it relies on a partition of the semantic (latent) space into atoms based
on the the semantic meaning. In this work we explore the role of the semantic
space partition in scenarios where the task structure involves a one-to-many
mapping between the semantic space and the action space. In such scenarios,
partitioning based on hard inference results results in loss of information
which degrades the equalization performance. We propose a soft criterion to
derive the atoms of the partition which leverages the soft decoder’s output
and offers a more comprehensive understanding of the semantic space’s
structure. Through empirical validation, we demonstrate that soft partitioning
yields a more descriptive and regular partition of the space, consequently
enhancing the performance of the equalization algorithm.

##  I Introduction

Semantic communication, as introduced by Weaver [1] in its prelude to
Shannon’s seminal paper [2], is a paradigm where communication serves as a
means to solve a given task rather than an end in itself. Semantic
communication systems can significantly reduce overall network rate
requirements by extracting and transmitting only task-relevant information
from the data. Recently, they have been identified as a key enabler of future
communication systems [3] [4] [5]. The main driver behind the recent
popularity of semantic communications is the success of Artificial
Intelligence (AI), particularly Machine Learning (ML), for automatic task
solving. ML can be utilized to learn a language (communication protocol) that
enables effective communication and collaboration between connected agents .
These communication protocols are well-suited for future technologies like
smart cities and autonomous vehicles, where multiple intelligent agents
communicate and collaborate to solve downstream tasks. Consequently, the
design and development of such protocols are of paramount importance.

The development of ML semantic protocols is an active area of research.
Studies have shown that the bandwidth or energy consumption of wireless
networks can be greatly reduced while still successfully completing downstream
tasks without any performance loss [6] [7]. However, most literature assumes
that the language between transmitter and receiver is shared as a result of a
joint learning procedure, which might not hold in practical scenarios. In
future networks, the set of agents participating in communication and
collaboration to solve a given task will likely dynamically change depending
on geographical factors and resource availability. In such cases, constantly
re-learning a shared language proves to be a resource-intensive endeavor,
which is infeasible in networks characterized by constrained energy and
bandwidth resources. Conversely, when the language between agents is not
shared, semantic mismatch arises, and task performance significantly drops
[8]. To solve this, Semantic Channel Equalization (SCE) was introduced [9].
This framework models language as a partition of the communication space into
multiple atoms, with each atom associated with a distinct semantic meaning. By
doing so, SCE facilitates effective semantic translation among different
languages with a low complexity algorithm. The efficacy of this approach in
various domains, including image classification [9] and reinforcement learning
scenarios [10] was extensively shown. While previous studies showcase the
empirical effectiveness of the SCE framework, they do not delve into the
crucial role of the semantics captured by the languages.

In this work, we aim to elucidate how different partitions of the latent space
capture diverse semantic meanings and how these variations impact the
equalization algorithm. Our primary contribution lies in introducing a novel
methodology for partitioning the latent space, which captures richer semantic
meaning and consequently enhances the performance of SCE.

##  II System Model

Figure 1: Proposed communication scenario shared with [10]. A distributed
control problem is explored, where the language of the encoder does not match
the language of the decoder. Using a codebook of transformations and a
selection policy, semantic channel equalization is performed.

Consider the distributed inference problem illustrated in Fig. 1. Here, an
encoder $\lambda:\mathcal{O}\to\mathcal{X}$ transforms observations
$o\in\mathcal{O}$ from the environment into a semantic representation
$x\in\mathcal{X}$. This semantic representation $x$ encapsulates the relevant
information present in $o$, which the decoder
$\gamma:\mathcal{X}\to\mathcal{A}$ interprets to select an action
$a\in\mathcal{A}$ (we will assume $\mathcal{A}$ to be discrete). In this
study, we focus on the scenario where the communication between the encoder
and the decoder is through a noisy channel over multiple time steps. It is
worth noting that this general formulation includes image classification,
which can be seen as a specialized case with only one time-step. Following the
terminology introduced in SCE [9], we refer to $\lambda$ as the language
generator and to $\gamma$ as the language interpreter. Furthermore, we denote
$\mathcal{O}$, $\mathcal{X}$, and $\mathcal{A}$ as the observation space,
semantic space, and action space, respectively and we denote $\mu_{o}$ as the
probability distribution of observations. We assume that the language
(communication strategy) between $\lambda$ and $\gamma$ is a result of a joint
learning process. Through this process, the agents learn $Q(a,o)$, an
approximation of the true action-value function $Q^{*}(a,o)$, which is an
indicator of how “good” it is to play action $a$ when $o$ is observed. Based
on the learned action-value function $Q$, the agents make decisions following
a greedy policy:
$\gamma(\lambda(o))=\operatorname*{arg\,max}_{a\in\mathcal{A}}Q(a,o)$.

Figure 2: Two possible partitions for the latent space of a language
generator. The generator was trained with the task of MNIST classification.
Different ways to partition the semantic space depend on the criteria used
(digit classification or digit parity classification)

###  II-A Semantic space partitioning and language effectiveness

A language generator $\lambda$ defines a method for encoding information into
the semantic space. When the generator operates effectively, it ensures that
the task-relevant data features of each observation $o$ are appropriately
encoded in the semantic representation $x=\lambda(o)$. The resulting semantic
space encapsulates all pertinent information in a structured manner, where
observations with shared semantic characteristics are encoded similarly. This
structure can be exploited to partition the space into multiple subspaces
termed atoms. Each atom corresponds to a distinct semantic meaning, with all
semantic representations within an atom reflecting observations that share
that particular meaning. We denote the set of chosen atoms as a partition of
the semantic space, represented as $P=\\{P_{0},P_{1},\dots,P_{N}\\}$, where
$P_{i}$ denotes the $i$-th atom of the partition.

Different semantic space partitions can capture different levels of semantics.
For example, in an image classification task, each atom of the semantic space
can be associated with a label of the images, which is a high level
description of semantics. However, if the encoder is descriptive enough,
lower-level descriptions are also possible, and features such as the colors or
shapes can be captured with the appropriate partitioning. As an example, see
Fig. 2 where two different partitions of the semantic space are shown for a
generator trained to solve the MNIST classification task. Points in the latent
(semantic) space are partitioned according to different criteria. One
partition captures the digit information present in the semantic symbols while
other possible partition captures the parity of this symbols.

###  II-B Language mismatch in multi-user communication

When the language generator and the language interpreter are not trained
jointly, it is unlikely that they employ the same language and a semantic
mismatch arises even if the training procedures follow the same architectures,
data and objective functions [11]. In SCE [9], the semantic mismatch arising
between a language generator and a language interpreter was modeled as a
misalignment of the atoms of their corresponding partitions. More precisely,
when the source generator (transmitter) sends a message
$x_{s}=\lambda_{s}(o)$, it will not be interpreted correctly at the target
interpreter (receiver) if it does not fall in the corresponding target atom of
$x_{t}=\lambda_{t}(o)$. We denote the source
$(\lambda_{s},\gamma_{s},\mathcal{X}_{s},\mathcal{A}_{s},\mathcal{O})$ and
target $(\lambda_{t},\gamma_{t},\mathcal{X}_{t},\mathcal{A}_{t},\mathcal{O})$
languages which were trained independently on the same observation space and
we will explore the case where $\lambda_{s}$ and $\gamma_{t}$ communicate.

Figure 3: Different ways to partition the semantic space capture different
semantics. When using the hard decision outcome to define the partition, the
structure of the task and the relationship between actions is lost. When using
the soft decision values, all the task-relevant information is exploited for
the partition.

###  II-C Compensating for language mismatch via Semantic Channel Equalization

To deal with the language mismatch, the SCE algorithm leverages a codebook
$\mathcal{T}$ of linear transformations between individual atoms and a
selection policy to operate the codebook. For a source
$P^{s}=\\{P_{0}^{s},P_{1}^{s},\dots,P_{J_{s}}^{s}\\}$ and target
$P^{t}=\\{P_{0}^{t},P_{1}^{t},\dots,P_{J_{t}}^{t}\\}$ partition of the
semantic space, each transformation
$T:\mathcal{X}_{s}\to\mathcal{X}_{t}\in\mathcal{T}$ is learned using optimal
transport and aims to maximize the transported volume for a given pair of
source and target atoms:

$\begin{split}\rho_{P_{i}^{s}\xrightarrow{}P_{j}^{t}}(T)=\frac{\mu_{T\lambda_{s}}\left(T\left(P_{i}^{s}\right)\cap
P_{i}^{t}\right)}{\mu_{T\lambda_{s}}\left(P_{i}^{s}\right)}.\end{split}$ (1)

Here $\mu_{T\lambda_{s}}$ is the post-transformation distribution on the
semantic space, which depends on the observation distribution $\mu_{o}$ and,
if $\lambda$ and $T$ are injective, can be written as
$\mu_{T\lambda_{s}}=\mu_{o}\circ\lambda_{s}^{-1}\circ T^{-1}$. The codebook
$\mathcal{T}$ has as many transformations as the total pairs of source and
target atoms, i.e. $|\mathcal{T}|=J_{s}\cdot J_{t}$.

The operation policy selects a transformation $T$ from the codebook of
transformation following

$\pi_{\text{sem}}=\operatorname*{arg\,max}_{T\in\mathcal{T}}\Bigg{[}\sum_{i=1}^{J_{s}}\mu_{\lambda_{s}}\left(P_{i}^{s}|o\right)\sum_{j\in\kappa(i)}\rho_{P_{i}^{s}\xrightarrow{}P_{j}^{t}}(T)\Bigg{]}$
(2)

where $\mu_{\lambda_{s}}$ is the source language distribution (which can be
written as $\mu_{\lambda_{s}}=\mu_{o}\circ\lambda_{s}^{-1}$ if $\lambda_{s}$
is injective) on the semantic space and $\kappa(i)$ is a (problem dependent)
mapping function between source and target atoms. The policy
$\pi_{\text{sem}}$ aims to perfectly align source and target atoms according
to their semantic meaning without regards to the downstream task performance.
On [10], a new equalization policy which aims to maximize downstream task
performance was proposed as

$\pi_{\text{eff}}=\operatorname*{arg\,max}_{T\in\mathcal{T}}\Bigg{[}\sum_{i=1}^{J_{s}}\mu_{\lambda_{s}}\left(P_{i}^{s}|o\right)\sum_{j=1}^{J_{t}}\rho_{P_{i}^{s}\xrightarrow{}P_{j}^{t}}(T)Q_{t}(a_{j},o)\Bigg{]}.$
(3)

Where $Q_{t}(a,o)$ is the target language’s estimation of the true action-
value function. Here, it is implicitly assumed that each target atom should
correspond to a unique action, this is the assumption we challenge in this
work. The policy $\pi_{\text{eff}}$ aims to maximize performance rather than
perfect semantic alignment, which is not required to complete the task. We
call $\pi_{\text{sem}}$ and $\pi_{\text{eff}}$ the semantic and effectiveness
equalization policies respectively.

The effectiveness of SCE heavily relies on how the semantic space is
partitioned. Essentially, this partitioning serves as a means of compressing
the information intended for transmission. It groups distinct observations
sharing the same semantic meaning into atoms. As SCE aligns these atoms, only
the information captured by the partition is transmitted. Therefore, the
selection of the language partition is a critical aspect of SCE as it
determines the relevant information to be conveyed. If the chosen partition
isn’t suitable for the downstream task, the equalization process will likely
fail. For instance, consider Fig. 2, showcasing two partitions of the semantic
space generated by a MNIST classification model. If the objective is to
classify digits, only the left partition captures the necessary semantics and
aids the receiver effectively. Conversely, if the task involves classifying
data parity, both partitions will convey the required semantics. However,
opting for a more detailed partition comes with a trade-off—a more intricate
equalization algorithm due to the increased number of atoms. This study aims
to identify the optimal approach to partitioning the semantic space,
considering the underlying task structure.

##  III The Impact of Semantic Space Partitioning on Equalization Performance

###  III-A Hard partitioning

To partition the semantic space, previous work considers hard partitioning [9,
10]. This approach defines an atom $P_{i}$ of a partition as the set of
semantic symbols (i.e., states being mapped) in the semantic space that result
in the same action $a_{i}\in\mathcal{A}$:

$\displaystyle P_{i}=$
$\displaystyle\big{\\{}x\in\mathcal{X}\leavevmode\nobreak\
|\leavevmode\nobreak\ x=\lambda(o);$ (4) $\displaystyle\leavevmode\nobreak\
\gamma(x)=a_{i}=\operatorname*{arg\,max}_{a\in\mathcal{A}}Q(a,o),\leavevmode\nobreak\
\forall o\in\mathcal{O}\big{\\}}$

This approach is based on one assumption: there exists a one-to-one
relationship between the atoms of the partition and the actions. Yet, in many
control tasks, different possible actions may exist for a given observed
state, thus leading to action ambiguities. Action ambiguities can be
detrimental to equalization performance when ignored. Indeed, when action
ambiguity is present, hard partitioning assigns the semantic symbol $x$ to the
output of the decision $\gamma(x)$, ignoring its true semantic meaning. We
show later that this approach leads to irregular atom shapes, which are hard
to equalize. As an alternative, we propose a soft-value based partitioning,
which leverages the action-values $Q(\cdot,o)\in\mathbb{R}^{|\mathcal{A}|}$.

###  III-B Proposed solution via soft-values based partitionning

On Fig. 3 the main idea behind soft values semantic space partitioning is
shown. Using the information present in the soft decision process of the
interpreter allows for a more descriptive partition of the semantic space.
Following the figure, if hard decisions were to be used for partitioning, the
state will fall withing the semantic atom corresponding to the the estimated
optimal action. However, there are two actions that lead to optimal behaviour.
The learned $Q$ values scores capture this by assigning high value to these
actions, where the difference between them is only due to noise in the
learning process. Using the information provided by $Q$, the partition of the
semantic space could be able to differentiate between states in which only one
action is optimal and states where multiple actions are. Not only this is a
better description of the semantic space, but also boosts the performance of
the equalization algorithm, since, as we will show, the resulting atoms will
be more regular.

We propose then to build the partition on the action-value space to build the
atoms. For each observation $o\in\mathcal{O}$, we can represent the function
$Q(\cdot,o)$ as a vector in the space $\mathbb{R}^{|\mathcal{A}|}$. Using
standard clustering techniques, it is possible to divide the action-value
space and translate this into the semantic space. More precisely, using a
clustering algorithm
$C:\mathbb{R}^{|\mathcal{A}|}\to\left\\{0,1,\ldots,n_{c}-1\right\\}$ which
maps each point in the action value space $\mathbb{R}^{|\mathcal{A}|}$ into an
index value indicating belonging to one of $n_{c}$ atoms, we can define a
partition of the semantic space

$P=\\{P_{0},\ldots,P_{n_{c}-1}\\}$ (5)

where each atom is constructed following

$P_{i}=\left\\{\lambda(o)|o\in\mathcal{O}\text{ and
}C\left(Q(\cdot,o)\right)=i\right\\}.$ (6)

The choice of the clustering algorithm is not simple and it is most likely
problem dependent. In our work we choose to use the well known k-means
algorithm [12]. This algorithm requires to define the number of atoms $n_{c}$
beforehand so we test multiple atom numbers in our experiments. While we are
aware of the limitations of the algorithm with respect to convergence, the
need to define the number of clusters beforehand and also the clustering
criteria, we chose k-means both for its simplicity and popularity. Finding the
optimal clustering criteria is beyond the scope of this paper, our objective
is rather to show the influence of different latent space partition on SCE.

Figure 4: Projection of the action-value space of dimension $n_{a}=4$ into the
first two data maximum variance directions for the source language. Each point
corresponds to an observation. Colors are shown according to the action that
maximized the value for each observation. Figure 5: Different partitions of
the semantic space using the k-means algorithm with varying number of atoms.
The actions (hard partitioning) are visualized as different shapes for each
partition. The color of the points corresponds to a given atom and the actions
associated with each atom are shown next to it in the color legend.

##  IV Numerical results

We evaluate the proposed system using a language generated through
Reinforcement Learning (RL) techniques, aiming to address the environment
depicted in Fig. 1. In this case, the observation space $\mathcal{O}$
represents the state of a grid world featuring an agent and a treasure. The
encoder $\lambda$ maps each observation into $\mathcal{X}=\mathbb{R}^{2}$.
During training, the average output power of the encoder is standardized to 1.
This is achieved through the implementation of a rolling mean normalization
technique, where a normalization constant $\tau$ is computed as
$\tau_{i}=\eta\cdot\tau_{i-1}+(1-\eta)||x^{*}_{i}||^{2}$. Here,
$x^{*}_{i}\in\mathcal{X}$ denotes the non-normalized semantic symbol
($x_{i}=x_{i}^{*}/\tau_{i}$) chosen at training step $i$, with $\eta=0.1$
representing the momentum value. During testing, the value of $\tau$ is fixed
as the final value obtained during training. Subsequently, the decoder
$\gamma$ processes the noisy version of the transmitted symbol and selects an
action from $\mathcal{A}=\\{\text{right, down, left, up}\\}$, which the agent
executes. An episode concludes either when the agent reaches the treasure or
when the maximum number of steps (150) is attained. We operate under the
assumption that the encoder decoder pairs $(\lambda_{s},\gamma_{s})$ and
$(\lambda_{t},\gamma_{t})$ are provided and have each undergone a joint
training in a centralized manner utilizing RL techniques, all while adhering
to the same reward signal and a Signal to Noise Ratio (SNR) of 5 dB. For
training purposes, we employ Deep Q-Learning (DQN) [13], which is well-suited
for discrete action spaces. Additionally, we set
$\mathcal{X}_{s}=\mathcal{X}_{t}=\mathcal{X}$ and
$\mathcal{A}_{s}=\mathcal{A}_{t}=\mathcal{A}$.

###  IV-A Semantic space partitioning

We first explore the possible semantic space partitions for the learned
languages. We only show the results for the source language for lack of space.
We first show that the information included in the action-value space is more
descriptive of the task than the hard-decision clustering. On Fig. 4 the
projection of the four dimensional action-value space for the source language
is shown. It is clear that, even if the total amount of actions is four, the
action-value space hints that different partitions are possible. It is easy to
see at least eight clusters, four corresponding to unique actions (originated
from states where only one action is optimal) and four others corresponding to
pairs of actions (originated from states where more than one action is
optimal, as such shown on Fig. 3).

On Fig. 5 we show the resulting partitions on the semantic space when using
k-means on the action-value space for multiple choices of number of atoms
$n_{c}$. In general, we can observe that the resulting atoms are more regular
in shape compared to the hard partitioning (which is indicated by the shape of
the plotted points). Different partitions capture different semantic
descriptions of the task. In particular, for $n_{c}=8$ the eight atoms of the
semantic space correspond to the observed clusters in the action-value space
on Fig. 4. However, for four and six atoms, the semantic meaning for some
individual actions is lost, as we will show next, this will be detrimental for
the performance of the equalization.

###  IV-B Performance of proposed solution

Leveraging the partitions of the semantic space shown on Fig. 5, we implement
both equalization policies $\pi_{\text{sem}}$ and $\pi_{\text{eff}}$
introduced by the SCE framework as described on Eq. 2 and Eq. 3 respectively.
When using k-means based soft partitioning, we replace $\mu_{\lambda_{s}}$ by
the normalized inverse of the distances to the k-means centers and $\kappa(i)$
as the index of the target atom whose center in the action-value space lies
closest to the one from $P_{i}^{s}$. To compute the action value of each
target atom (equivalent to $Q_{t}$ in Eq. 3) we use the average Q-value for
the actions in it, i.e.

$Q_{t}(P_{i}^{t},o)=\frac{1}{|P_{i}^{t}|}\sum_{x\in
P_{i}^{t}}Q_{t}(\gamma_{t}(x_{j}),o).$ (7)

Figure 6: Performance of the policy $\pi_{\text{sem}}$ as a function of SNR
for hard partitioning and soft partitioning with different numbers of atoms
$n_{c}$.

The results for $\pi_{\text{sem}}$ are shown on Fig. 6 and for
$\pi_{\text{eff}}$ on Fig. 7. The performance of both policies depends on the
partition of the semantic space and we note that soft partitioning is not
always beneficial. For example, when using soft partitioning with four and six
atoms, the performance of the equalization is worse than when using the four
action hard partitioning. To understand why, it suffices to look at the
resulting partitions on Fig. 5. For the case of four and six atoms, all unique
actions do not have a corresponding atom, as shown by the atom labels. For
four atoms, no unique action is captured, and for six atoms, unique actions
“left” and “right” are not captured. This degrades the performance, as the
equalization algorithm can not transmit the message of taking only a singular
action, which in the particular task is necessary to reach the goal. However,
when the soft partition allows to capture all the relevant information of the
task, the performance is greatly improved. From our results, we can conclude
that the best number of atoms for our problem is eight, since it is the
smallest number of atoms for which we obtain the best performance. This
observation reinforces our intuition on the optimal number of atoms, in which
each atom should be associated to all singular actions and all possible action
ambiguities, which, for this particular example, gives us a total of eight
atoms.

###  IV-C On the descriptiveness of soft partitioning for multi-task
equalization

Soft partitioning offers a more nuanced depiction of the semantic space,
enabling a richer understanding of semantics that can be beneficial for multi-
task equalization. For instance, consider a scenario where the decoder is
expanded to accommodate eight actions, including diagonal moves, instead of
the original four horizontal and vertical ones. In this case, hard
partitioning may hinder the encoder’s ability to transmit semantic meanings
associated with diagonal actions. On the other hand, soft partitioning the
semantic space into eight atoms (as illustrated in Fig. 5) could empower the
decoder to interpret atoms containing information about double actions as
diagonal moves. This straightforward example underscores the potential of soft
clustering for multi-task equalization. Further exploration of this aspect is
left for future research.

Figure 7: Performance of the policy $\pi_{\text{eff}}$ as a function of SNR
for hard partitioning and soft partitioning with different numbers of atoms
$n_{c}$.

##  V Conclusions

Motivated by the recent advancement on Semantic Channel Equalization, in this
work we address the role of the semantic space partitioning on the
equalization performance. We first justify why partitioning the semantic space
according to output actions (hard partitioning) is sub-optimal in cases where
there are action ambiguities, i.e. multiple optimal actions. To address this
problem, we propose to use soft partitioning, which leverages the estimates
action-values to define the semantic atoms. We show that, using soft
partitioning, the resulting partition of the space is more regular, which
improves the equalization performance. Moreover, we show that richer semantics
of the problem can be captured by soft partitioning, which is a promising
result that can be applied to multi tasking equalization.

## Acknowledgements

The present work was supported by the EU Horizon 2020 Marie Skłodowska-Curie
ITN Greenedge (GA No. 953775), by “6G-GOALS”, an EU-funded project, and by the
French project funded by the program “PEPR Networks of the Future” of France
2030.

## References

  * [1] W. Weaver, “Recent contributions to the mathematical theory of communication,” ETC: a review of general semantics, pp. 261–281, 1953.
  * [2] C. E. Shannon, W. Weaver, and N. Wiener, “The mathematical theory of communication,” Physics Today, vol. 3, no. 9, p. 31, 1950.
  * [3] E. Calvanese Strinati and S. Barbarossa, “6G networks: Beyond shannon towards semantic and goal-oriented communications,” Computer Networks, vol. 190, p. 107930, 2021.
  * [4] M. Kountouris and N. Pappas, “Semantics-empowered communication for networked intelligent systems,” IEEE Communications Magazine, vol. 59, no. 6, pp. 96–102, 2021.
  * [5] E. C. Strinati, P. Di Lorenzo, V. Sciancalepore, A. Aijaz, M. Kountouris, D. Gündüz, P. Popovski, M. Sana, P. A. Stavrou, B. Soret, et al., “Goal-Oriented and Semantic Communication in 6G AI-Native Networks: The 6G-GOALS Approach,” arXiv preprint arXiv:2402.07573, 2024.
  * [6] E. Bourtsoulatze, D. B. Kurka, and D. Gündüz, “Deep joint source-channel coding for wireless image transmission,” IEEE Transactions on Cognitive Communications and Networking, vol. 5, no. 3, pp. 567–579, 2019.
  * [7] T.-Y. Tung, S. Kobus, J. P. Roig, and D. Gündüz, “Effective communications: A joint learning and communication framework for multi-agent reinforcement learning over noisy channels,” IEEE Journal on Selected Areas in Communications, vol. 39, no. 8, pp. 2590–2603, 2021.
  * [8] M. Sana and E. C. Strinati, “Learning semantics: An opportunity for effective 6G communications,” in 2022 IEEE 19th Annual Consumer Communications & Networking Conference (CCNC), pp. 631–636, IEEE, 2022.
  * [9] M. Sana and E. Calvanese Strinati, “Semantic Channel Equalizer: Modelling Language Mismatch in Multi-User Semantic Communications,” in GLOBECOM 2023 - 2023 IEEE Global Communications Conference, pp. 2221–2226, 2023.
  * [10] T. Huttebraucker, M. Sana, and E. Calvanese Strinati, “Pragmatic Goal-Oriented Communications Under Semantic-Effectiveness Channel Errors,” in 2024 IEEE 21st Consumer Communications & Networking Conference (CCNC), pp. 683–689, 2024.
  * [11] L. Moschella, V. Maiorca, M. Fumero, A. Norelli, F. Locatello, and E. Rodolà, “Relative representations enable zero-shot latent space communication,” in The Eleventh International Conference on Learning Representations, 2022.
  * [12] M. Ahmed, R. Seraj, and S. M. S. Islam, “The k-means Algorithm: A Comprehensive Survey and Performance Evaluation,” Electronics, vol. 9, no. 8, 2020.
  * [13] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533, 2015.

  *[AI]: Artificial Intelligence
  *[ML]: Machine Learning
  *[SCE]: Semantic Channel Equalization
  *[RL]: Reinforcement Learning
  *[SNR]: Signal to Noise Ratio