# UrbanWorld: An Urban World Model for 3D City Generation

Yu Shang1, Jiansheng Chen2, Hangyu Fan3, Jingtao Ding1, Jie Feng1, Yong Li1

1Tsinghua University 2University of Science and Technology Beijing 3Tsingroc
Inc.
Corresponding author, correspondence to<EMAIL_ADDRESS>
###### Abstract

Cities, as the most fundamental environment of human life, encompass diverse
physical elements such as buildings, roads and vegetation with complex
interconnection. Crafting realistic, interactive 3D urban environments plays a
crucial role in constructing AI agents capable of perceiving, decision-making,
and acting like humans in real-world environments. However, creating high-
fidelity 3D urban environments usually entails extensive manual labor from
designers, involving intricate detailing and accurate representation of
complex urban features. Therefore, how to accomplish this in an automatical
way remains a longstanding challenge. Toward this problem, we propose
UrbanWorld, the first generative urban world model that can automatically
create a customized, realistic and interactive 3D urban world with flexible
control conditions. UrbanWorld incorporates four key stages in the automatical
crafting pipeline: 3D layout generation from openly accessible OSM data, urban
scene planning and designing with a powerful urban multimodal large language
model (Urban MLLM), controllable urban asset rendering with advanced 3D
diffusion techniques, and finally the MLLM-assisted scene refinement. The
crafted high-fidelity 3D urban environments enable realistic feedback and
interactions for general AI and machine perceptual systems in simulations. We
are working on contributing UrbanWorld as an open-source and versatile
platform for evaluating and improving AI abilities in perception, decision-
making, and interaction in realistic urban environments.

##  1 Introduction

Cities are the most complex human-centric environments, characterized by their
intricate structures, diverse elements, and dynamic interactions. Creating
near-realistic 3D urban world environments is a fundamental and pivotal
technique for broad research and real applications across various domains such
as AI agents (Yang et al., 2024), urban planning (Schrotter & Hürzeler, 2020),
urban simulation (Xu et al., 2023) and metaverse (Allam et al., 2022).
Traditionally, achieving this involves expensive labor costs for human
designers on detailed asset modeling, texture mapping, and scene composition.
With the advancement of generative AI, there have emerged more automatic
approaches for 3D scene generation based on volumetric rendering(Lin et al.,
2023; Xie et al., 2024) and diffusion models(Deng et al., 2023; Lu et al.,
2024). These approaches have revolutionized the paradigm of 3D scene
generation, alleviating the high costs of manual design. However, the crafted
3D scenes are only visually appealing videos, which are significantly
different from the real embodied physical world. Regarding this issue, a
recent series of methods known as world models have emerged, preliminarily
focusing on autonomous driving scenes (Hu et al., 2023; Wang et al., 2023).
These models are shown to possess the capability of understanding the scene
dynamics and predicting future states, uplifting the interactivity of 3D scene
generation. However, there is still a large gulf between the created urban
environments and the real urban world in which humans live. To sum up, there
is still a long way from the actual “urban world models”, which we define as
models able to create urban environments that are (1) realistic and
interactive (2) customizable and controllable (3) capable of supporting
embodied agent learning.

Urban world models are of great significance in developing embodied
intelligence and Artificial General Intelligence (AGI). Firstly, it is
promising to bridge the gap between virtual environments and the real world,
enabling embodied agents to interact with and learn from richly detailed,
realistic urban environments. Secondly, by crafting synthetic 3D urban
environments, researchers can gain complete control over data generation, with
full access to all generative parameters. Machine perceptual systems can thus
be trained on tasks that are not well suited to conduct in the real world or
require various environments. Finally, a sophisticated urban world model can
simulate a wide variety of environments, from bustling city centers to quiet
residential neighborhoods, with realistic visual appearances of physical
infrastructures such as buildings, roads, and natural spaces. This is crucial
to avoid overfitting and creating agents with high generalization in diverse
and dynamic environments. Some preliminary explorations have been witnessed in
some commercial platforms like Omniverse111https://www.nvidia.com/en-
us/omniverse/, an open platform for creating and simulating detailed 3D world
environments. However, there are no specialized urban world models for
crafting interactive 3D urban environments, hindering the development of
agents in the realistic urban environments.

Toward this issue, we propose UrbanWorld, a generative urban world model that
can automatically create realistic, controllable and embodied 3D urban
environments just from a text and
OpenStreetMap222https://www.openstreetmap.org/ (OSM) prompt. In detail, there
are four key modules in the framework of UrbanWorld. Firstly, UrbanWorld
automatically generates 3D urban layouts and conducts detailed asset
processing based on open-accessible OSM data via
Blender333https://www.blender.org/. Then, UrbanWorld adopts a fine-tuned
urban-specific multimodal large language model (called Urban MLLM) to
effectively plan and design urban scenes following user instructions,
generating detailed textual descriptions of urban elements. Next, UrbanWorld
integrates a 3D asset renderer based on texture diffusion and refinement,
flexibly controlled by textual and visual conditions. Finally, to further
optimize the visual appearance, UrbanWorld utilizes Urban MLLM to scrutinize
the crafted 3D urban scenes, generating detailed suggestions for refinement
and activate an additional iteration of rendering. This framework fully
exploits the controllable generation capabilities of diffusion models and the
reasoning and planning abilities of MLLMs, contributing to the high-fidelity
urban environment rendering and superior generation flexibility.

The created urban environment features detailed and realistic visual
representations of urban infrastructures, including buildings, roads, and
natural areas, greatly benefiting the deployment of AI systems. Notably, it
can support near-realistic physical interactions between urban environments
and agents. It is tailored to enhance a range of embodied abilities including
sensory grounding, perception and decision-making. For example, embodied
agents can be deployed and trained for object recognition, route planning and
automatic navigation via interaction with the realistic urban environment.
Besides, UrbanWorld can serve as an open platform to support the creation and
manipulation of more advanced 3D urban environments, facilitating the
advancement of broad AI communities.

The contributions of this work can be summarized as follows:

  * •

We present UrbanWorld, the first urban world model for automatically creating
realistic, customized and interactive embodied 3D urban environments with
flexible controls.

  * •

UrbanWorld demonstrates its superior generative ability to craft high-fidelity
3D urban environments, greatly enhancing the authenticity of interactions in
the environment.

  * •

We provide UrbanWorld as an open-source tool and contribute a 3D asset dataset
with various urban environments, which facilitates the advancement of broad
AI-related research, such as embodied intelligence and AI agents, laying the
groundwork for advancing AGI.

##  2 Related Works

Table 1: Comparison between existing works for 3D urban scene generation and
UrbanWorld from four aspects: text-controllable, image-controllable, creative
(i.e., whether new assets can be created) and embodied/interactive (whether
the created urban environment is physically interactive).

Method | Text-controllable | Image-controllable | Creative | Embodied/Interactive
---|---|---|---|---
SceneDreamer (Chen et al., 2023) | $\times$ | $\checkmark$ | $\checkmark$ | $\times$
PersistentNature (Chai et al., 2023) | $\times$ | $\checkmark$ | $\checkmark$ | $\times$
Infinicity Lin et al. (2023) | $\times$ | $\checkmark$ | $\checkmark$ | $\times$
CityDreamer (Xie et al., 2024) | $\times$ | $\times$ | $\checkmark$ | $\times$
CityGen (Deng et al., 2023) | $\times$ | $\times$ | $\checkmark$ | $\times$
SceneCraft (Hu et al., 2024) | $\checkmark$ | $\times$ | $\times$ | $\checkmark$
CityCraft Deng et al. (2024) | $\checkmark$ | $\times$ | $\times$ | $\checkmark$
UrbanWorld | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$

###  2.1 3D Urban Scene Generation

3D urban scene generation aims to create realistic 3D urban environments with
sophisticated urban planning and visual element design, usually requiring high
human efforts such as complex asset modeling, texture mapping, and scene
composition. With the advancement of deep learning techniques, recently there
are three lines of work trying to achieve this in an automated way, including
NeRF-based methods (Lin et al., 2023; Xie et al., 2024; Chen et al., 2023),
diffusion-based methods (Deng et al., 2023; Inoue et al., 2023; Wu et al.,
2024) and professional software script-based methods (Zhou et al., 2024; Hu et
al., 2024). An overview of the method comparison is shown in Table 1. NeRF-
based methods implicitly represent the urban scene and perform the volumetric
rendering for the neural fields. For example, CityDreamer (Xie et al., 2024)
first separates the scene into buildings and backgrounds then introduces
different types of neural fields for asset rendering. These methods can
produce a high-quality visual appearance while potentially losing geometric
fidelity. Diffusion-based methods utilize diffusion models to generate city
layouts or urban scenes. CityGen (Deng et al., 2023) provides an end-to-end
pipeline to create diverse 3D city layouts with Stable Diffusion. These
methods are creative in generating scene images or videos, but hard to obtain
embodied 3D environments, limiting the practical usages. Recently, some
professional software script-based methods have been proposed, trying to
develop an automatic agentic workflow using LLMs to control the professional
software for scene creation. A representative example is Scenecraft (Hu et
al., 2024), which establishes an LLM-based agent to translate textual
descriptions into Python scripts for scene creation in Blender. A more recent
work CityCraft (Deng et al., 2024) adopts LLMs in designing and organizing 3D
urban environments from off-the-shelf asset libraries. Such approaches are
effective but only create urban environments by retrieving and organizing
existing assets, unable to flexibly create new assets when necessary.
Differently, our approach adopts and integrates both diffusion-based and
professional software script-based methods, providing an effective,
controllable and creative way for 3D urban world generation.

###  2.2 3D World Simulator

A persistent objective in AI research has been to develop machine agents
capable of engaging with various environments in 3D space like humans. Toward
this goal, researchers have been devoted to building various interactive world
simulators in the format of videos (Bruce et al., 2024) or embodied
environment (Shen et al., 2022). Existing world simulation environments and
platforms are mostly for indoor scenes (Puig et al., 2018; Xia et al., 2018;
Kolve et al., 2017; Xia et al., 2020). Differently, Threedworld (Gan et al.,
2020) pays attention to creating outdoor environments by retrieving and
compositing objects from an existing asset library. When it comes to urban
scenes which is the most important open environment, existing works mostly
focus on the generative world model for autonomous driving capable of learning
scene dynamics and understanding the geometry of the physical world (Hu et
al., 2023; Wang et al., 2023; 2024). However, these models can only generate
new scenes in the format of videos, hard to provide an embodied and
interactive urban environment for real use. UGI (Xu et al., 2023) takes a
forward step toward urban world simulation, proposing to develop an embodied
urban environment for agent development. Although it conceptualized some
relevant ideas, there is a lack of specific frameworks for implementation. To
address this challenge, we propose UrbanWorld, which is expected to facilitate
the construction of diverse embodied urban environments with controllable and
refined visual appearance, supporting agent development or simulation in
various urban scenes.

##  3 Methodology

Figure 1: Illustration of the whole framework of UrbanWorld, including four
key components: (A) OSM-guided urban layout generation; (B) MLLM-empowered
urban scene design; (C) Diffusion-based urban asset texture rendering; (D)
MLLM-assisted scene refinement.

There are three main challenges to solve for building an “urban world model”:
efficient embodied environment construction, professional urban scene planning
and high-quality texture generation. Towards these objectives, UrbanWorld
introduces four key components: (1) OSM-guided urban layout generation, an
automatic 2D-to-3D transformation module based on globally open-accessible OSM
data, which can address the first challenge. (2) MLLM-empowered urban scene
design, which exploits the superior urban scene understanding ability of the
trained urban MLLM to draft reasonable urban scenes like human designers
towards the second challenge. (3) Controllable diffusion-based urban asset
texture renderer, a flexible urban asset rendering based on 3D diffusion
following customized prompts. (4) MLLM-assisted urban scene refinement, a
final reflection module to further improve the scene design, inspired by the
iterative revision in the standard operation process of human designers. The
last two components contribute to high-fidelity textures of 3D assets,
effectively tackling the third challenge. The overview of UrbanWorld is
illustrated in Figure 1.

###  3.1 OSM-guided Urban Layout Generation

Considering the easy accessibility and global coverage of OSM data, UrbanWorld
is mainly developed based on OSM to generate 3D urban layouts. OSM data
contains a wealth of information, mainly including the geographic locations
and attributes of roads, buildings, vegetation, forests, water, and other
infrastructure elements. The contained urban assets such as buildings, forest,
vegetation, water and roads are then separated as independent objects for
subsequent element-wise rendering. In this step, UrbanWorld also records the
object center location for further reorganization of assets, making it match
the real urban layout.

###  3.2 MLLM-empowered Urban Scene Design

Aiming to effectively craft customized urban environments, UrbanWorld
integrates an advanced urban MLLM trained on extensive urban street-view
imagery data. Specifically, we first collected urban street-view images
globally and labeled the corresponding textual descriptions with GPT-4 and
then conducted a manual check and filtered the low-quality data. Then we fine-
tune an open-source MLLM, LLaVA-1.5 (Liu et al., 2024) on around 100K image-
text pairs from the collected dataset. We have validated that the obtained
urban MLLM possesses outstanding performance on urban scene understanding
tasks such as image captioning and scene classification thus can benefit the
urban scene analysis and design. In UrbanWorld, the urban MLLM is introduced
to act as a human-like designer, which automatically drafts high-quality and
detailed urban scene descriptions, ensuring the urban scenes are visually
coherent. Specifically, taking a simple textual instruction (e.g., a teaching
area in the university) from users and the selected OSM layout image as input,
UrbanWorld calls the urban MLLM with carefully designed prompts and returns
diverse detailed descriptions about visual appearance and materials for each
asset. The produced asset descriptions will be used as the condition
controlling the later rendering process.

###  3.3 Controllable diffusion-based urban asset texture rendering

Figure 2: Illustration of the urban asset rendering method in UrbanWorld,
mainly including two stages: Depth-aware UV texture generation with flexible
control under textual and visual prompts and UV position-aware texture
refinement.

Rendering a large-scale urban scene is challenging due to the existence of
complex elements and relations, whereby the scene-level rendering will
inevitably result in mismatching and low-resolution texture. Therefore, we
follow the element-wise rendering principle to ensure the rendering quality.
Simultaneously to speed up the rendering process, we merge some urban element
types and finally define four main categories: buildings, roads and paths,
forest and vegetation, and water. We implement the rendering with a
controllable diffusion-based method consisting of two stages: UV texture
generation and texture refinement detailed as shown in Figure 2.

Following the previous procedures, we have obtained the untextured 3D mesh of
the whole urban scene $S$ and each asset $S_{i}$. For each type of element, we
now have corresponding textual descriptions $t_{i}$ from the MLLM. here the
model also supports taking reference image $r_{i}$ as prompts to control the
generation. The UV map of the i-th asset $S_{i}$ is denoted as $U_{i}$.

We first set a series of camera views $v_{i}$ = $\\{v_{k}\\}_{k=1}^{N}$ to
capture the multi-view appearance of the object. Next, we utilize the depth-
aware ControlNet (Zhang et al., 2023) to control a 2D diffusion model $F$ to
generate an image $I_{i}$ showing the visual appearance on different views,
together with the condition $c_{i}\in\\{t_{i},r_{i},(t_{i},r_{i})\\}$:

$I_{i}=F(c_{i};d_{i};z),$ (1)

where $z$ is the latent embedding for the diffusion process, the depth map
from different views $d_{i}$ is obtained from the rendering process
$d_{i}=P(S_{i},v_{i})$. Then we conduct a reverse process $P^{-1}$ of
rendering to back-project $I_{i}$ into the UV texture space from each view:

$U_{i}=P^{-1}(v_{i};I_{i};S_{i}),$ (2)

Then we crop $U_{i}$ into texture maps from each view
$\\{U_{i}^{k}\\}_{k=1}^{N}$, each containing unique textual information from
different views. Subsequently, we merge these texture maps into a single
texture map $U_{i}$:

$U_{i}=\sum_{k=1}^{n}M_{i}^{k}\odot U_{i}^{k},$ (3)

where $M_{i}^{k}$ denotes the corresponding mask in the UV space from the view
$v_{i}$.

Up to now, we have obtained the preliminary texture map for the asset $S_{i}$,
while practically we have found that there are still some untextured areas on
the object due to the discrete sampling of views, especially for buildings
with many faces. Inspired by the inpainting capability of diffusion models, we
introduce an additional UV texture inpainting process to get complete and
natural textures. However, such inpainting can not be directly achieved with
general diffusion-based inpainting considering there are strict position
mapping requirements for UV texture maps. Therefore, we follow the UV
inpainting process of general 3D objects (Zeng et al., 2024), and introduce a
position map-guided building UV inpainting process.

Figure 3: Illustration of the evolution of created urban environments,
including the untextured urban scene, initial textured urban scene and refined
urban scene.

To be specific, we add a UV position map encoder $E_{V}$ to encode the
position map $V_{i}\in\mathbb{R}^{H\times W\times 3}$, indicating the
adjacency relation of the UV texture fragments, where $E_{V}$ shares the same
architecture of the image encoder in the diffusion model. Then we curate a set
of paired UV position maps and UV texture maps for urban assets with complex
surfaces and train the position encoder following the pipeline of ControlNet
(Zhang et al., 2023). With the control of UV position maps, it’s expected to
achieve accurate and reasonable inpainting for UV texture maps. The UV
inpainting process is formulated as follows:

$U_{i}^{refine}=F(c_{i};U_{i};E_{V}(V_{i})).$ (4)

With the above texture generation and completion steps, UrbanWorld can produce
coherent and high-fidelity textures for various urban elements.

For a better visual aesthetic of the rendering, we conduct image upscaling
with ControlNet-tile to further enhance the structure sharpness and realism of
the texture map, producing more detailed and realistic appearances for urban
assets.

###  3.4 MLLM-assisted urban scene refinement

After urban asset rendering, UrbanWorld automatically reorganizes the assets
guided by the location information extracted from the real OSM data,
effectively recovering the original urban layout. Inspired by the standard
operation process of human designing, where experts will take an overview of
the scene and make minor adjustments. To mimic such effort, UrbanWorld
activates urban MLLM again to scrutinize the crafted 3D urban scenes and
texture details. We prompt the MLLM to identify inconsistencies between the
scene imagery and previous design prompts and examine whether the texture is
as realistic as in the real world. Finally, the urban MLLM will provide
sophisticated suggestions for further refinement, including elements to be
modified and refined design prompts. Then the rendering module will be
activated and the involved elements will be rendered under the refined text
prompts and updated in the scene. With such a refinement process, UrbanWorld
can further align the generated urban environment to the real cities. We
provide a visualization example showing the evolution of the created urban
environments in Figure 3. It can be seen that UrbanWorld works in an iterative
refinement manner to create high-fidelity urban environments, where the low-
quality textures will be automatically identified and refined with the
powerful urban MLLM.

##  4 Experiments

Figure 4: Qualitative comparisons of generated 3D urban environments from
Infinicity, CityGen (results from the original paper since the code is not
open-source), CityDreamer and UrbanWorld. By comparison, our method can craft
more diverse 3D urban scenes with high-fidelity textures following user
instructions.

In this section, we first introduce the experimental setup and implementation
details (see Section 4.1), and then we present the generation results of
UrbanWorld for qualitative estimation (see Section 4.2). Finally, we provide
some quantitative evaluations of the created urban environments to demonstrate
the superiority of UrbanWorld (see Section 4.3).

###  4.1 Implementation Details

UrbanWorld incorporates three key techniques: Blender as the professional 3D
modeling software, diffusion-based rendering and MLLM-empowered scene design
and refinement. Specifically, we use Blender-3.2.2 for Linux systems and the
compatible Blosm addon to handle the OSM transformation. In terms of
diffusion-based rendering, we utilize Stable Diffusion-1.5 (Rombach et al.,
2022) as the fundamental diffusion backbone, together with ControlNet-Depth
(Zhang et al., 2023) in the generation of multi-view images. We also introduce
IP-Adapter (Ye et al., 2023) to support taking reference images as the
additional generation condition. We use ControlNet-inpainting (Zhang et al.,
2023) as the diffusion controller in the UV texture refinement and ContrlNet-
tile (Zhang et al., 2023) in the realness enhancement stage. For the hyper-
parameter settings in the diffusion-based rendering, we set the number of
camera views $N=4$, which can basically satisfy the rendering needs of most
urban assets. The number of inference steps in all diffusion processes is set
as 30 by default. The UV maps of 3D assets are unwrapped in the “smart
projection” mode operated in Blender. As for the urban MLLM, we fine-tune
LLaVA-1.5 on a curated dataset with 100K streetview image-text data pairs. All
experiments are conducted on a single NVIDIA Tesla A100 GPU.

###  4.2 Qualitative Results

We present some generation results of UrbanWorld in Figure 4, including
various representative urban scenes, including educational places, commercial
blocks and residential areas. For intuitive comparison, we also provide some
generation samples from Infinicity (Lin et al., 2023), CityGen (Deng et al.,
2023) and CityDreamer (Xie et al., 2024). The results of Infinicity and
CityGen are taken from the original paper because the codes are not open-
source. It can be seen that scenes from Infinicity are short of clear textures
and well-maintained building structures. Scenes from CityGen feature
homogeneous styles without clear characteristics of urban functions.
Similarly, the visual appearance of urban elements (especially buildings) in
the environments from CityDreamer lacks diversity and is hard to distinguish.
Besides, there are also clear geometric distortions of the building boundaries
in CityDreamer. These issues will pose great challenges for the real
interactions between subjects and urban environments. For example, embodied
agents are hard to be trained to conduct urban navigation because the
surrounding elements are too similar to recognize. By comparison, the urban
elements created by UrbanWorld possess distinct functional characteristics,
benefiting from the high controllability with text and reference image
prompts. Besides, the overall scene is more authentic and visually harmonious,
demonstrating the effectiveness of intelligent scene planning and design of
the MLLM.

###  4.3 Quantitative Evaluations

Table 2: Quantitative evaluation of existing works for 3D urban scene
generation and UrbanWorld on depth error, homogeneity index and realistic
score.

Method | Depth Error (↓) | Homogeneity Index (↓) | Realistic Score (↑)
---|---|---|---
SceneDreamer (Chen et al., 2023) | 0.152 | 0.746 | 5.6
CityDreamer (Xie et al., 2024) | 0.147 | 0.745 | 6.0
UrbanWorld | 0.089 | 0.683 | 6.8

To better demonstrate the superior generation performance of UrbanWorld, in
this section, we provide quantitative results from the following three
aspects:

Depth error. Depth error (DE) is utilized to evaluate the 3D geometry
accuracy, following the implementation of EG3D (Chan et al., 2022) and
CityDreamer (Xie et al., 2024). Specifically, we use the pre-trained depth
estimation model (Ranftl et al., 2020) to obtain the “ground truth” of depth
maps via density accumulation. DE is then calculated as the L2 distance
between the normalized predicted depth maps and the “ground truth”. The final
result is averaged on the result from 100 captured frames of generated urban
scenes.

Homogeneity index. Realistic cities are featured by complex elements with
diverse visual appearances, indicating various functional uses of different
urban areas. In order to capture this key character, we propose to evaluate
the homogeneity of generated scenes, mainly measuring the variance of
different urban scenes. To be specific, we first extract the visual feature of
each generated scene image with ResNet (He et al., 2016). The homogeneity
index is then calculated as the averaged cosine similarity of each pair of
scenes in the feature space. The smaller value of the homogeneity index means
a higher diversity of generated urban environments.

Realistic score. Another important aspect to evaluate is the realness of
generated urban environments, which is challenging even for human designers.
Here we introduce an off-the-shelf MLLM (GPT-4o) as the evaluator to score the
realistic level and quality of generated urban elements (ranging from 1 to
10). A higher realistic score indicates better realness and fidelity of
generated textures of urban assets.

We compare our model with some existing methods for urban scene generation,
mainly including SceneDreamer (Deng et al., 2023) and Citydreamer Xie et al.
(2024), representing the most advanced performance of automatic 3D urban
environment generation. We don’t provide results from other methods such as
CityGen (Deng et al., 2023), CityCraft (Deng et al., 2024) and SceneCraft (Hu
et al., 2024) because the source codes are not open when this work was done.

From the results presented in Table 2, it can be seen that UrbanWorld
outperforms on each quantitative metric compared with baselines.
Statistically, UrbanWorld achieves 39.5% on depth error compared with the most
competitive baseline, demonstrating the geometry-preserving ability of
UrbanWorld. By comparison, rendering methods such as SceneDreamer and
CityDreamer can produce visually appealing urban scenes, but commonly lose
geometry consistency. In terms of the homogeneity index, UrbanWorld has 8.3%
improvement compared with baselines. Consistent with the observation on the
qualitative results in Section 4.2, the generated scenes from existing methods
exhibit high homogeneity, limited to the style of training data. By
comparison, UrbanWorld can produce more diverse urban environments according
to user instruction, effectively achieving customized creation. This makes it
real for crafting any type of 3D urban environment adapting to the needs of
different scenes. Lastly, the created urban environments from UrbanWorld show
outstanding realness and high texture fidelity, increasing 11.8% realistic
score compared with baselines. The overall impression of scenes crafted by
SceneDreamer and CityDreamer is harmonious, however, the texture quality with
a closer look is unsatisfying. Differently, UrbanWorld conducts element-wise
texture rendering based on the specific 3D mesh, ensuring the geometric
matching and fidelity of generated textures.

###  4.4 Ablation study

Table 3: Study of the effectiveness of three key designs in UrbanWorld: MLLM-
empowered urban scene design, texture enhancement and MLLM-assisted scene
refinement.

Method | Depth Error (↓) | Homogeneity Index (↓) | Realistic Score (↑)
---|---|---|---
UrbanWorld | 0.089 | 0.683 | 6.8
w/o scene design | 0.104 | 0.701 | 6.1
w/o texture enhancement | 0.125 | 0.687 | 6.5
w/o scene refinement | 0.096 | 0.690 | 6.4

To further validate the effectiveness of key designs in UrbanWorld, we conduct
ablation studies and show the results in Table 3. We explore the influence of
three designs on the performance, including MLLM-empowered urban scene design,
texture enhancement and MLLM-assisted scene refinement. The results indicate
that all these techniques contribute to the final generation performance.
Specifically, the scene design from the MLLM contributes most to the realness
of generated urban environments, benefiting from the powerful knowledge
understanding and reasoning ability of MLLMs. The texture completion and
enhancement operation has the most notable effect on the estimated depth error
because better texture fidelity can help with geometric perception. Besides,
the final scene refinement process leads to a gain in all evaluation metrics,
further promoting the whole performance.

##  5 Conclusion

We propose UrbanWorld, the first generative urban world model to create
realistic, customized and interactive 3D urban environments with flexible
control conditions in a fully automatic manner. Integrating the powerful urban
scene understanding ability of urban MLLM and the controllable generation
ability of diffusion models, UrbanWorld can effectively craft high-fidelity
and diverse urban environments outperforming existing urban scene generation
methods. More importantly, the created urban environments can provide
realistic and high-fidelity information feedback, ensuring embodied agents
interact with and learn from the richly detailed, realistic urban world. We
are contributing UrbanWorld as an open-source tool to benefit broad research
communities, including but not limited to AI agents and embodied intelligence.
We believe this work can pave a new way to efficiently establish the 3D
virtual urban environment, accelerating the development of AGI.

For future work, we will continue to enhance the current version of UrbanWorld
and there are mainly three directions. Firstly, we will supplement more
elements into urban environment creation, such as persons and vehicles,
further improving the richness and realness of crafted environments. Secondly,
we will conduct experiments on interactive tasks such as visual recognition
and navigation for embodied agents to validate the real usability of created
3D environments. Lastly, we are organizing the codes and the generated 3D
urban environment assets for release. We expect to make UrbanWorld an open-
source toolkit for convenient use in various real applications.

## References

  * Allam et al. (2022) Zaheer Allam, Ayyoob Sharifi, Simon Elias Bibri, David Sydney Jones, and John Krogstie.  The metaverse as a virtual form of smart cities: Opportunities and challenges for environmental, economic, and social sustainability in urban futures.  _Smart Cities_ , 5(3):771–801, 2022.
  * Bruce et al. (2024) Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al.  Genie: Generative interactive environments.  In _Forty-first International Conference on Machine Learning_ , 2024.
  * Chai et al. (2023) Lucy Chai, Richard Tucker, Zhengqi Li, Phillip Isola, and Noah Snavely.  Persistent nature: A generative model of unbounded 3d worlds.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 20863–20874, 2023.
  * Chan et al. (2022) Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 16123–16133, 2022.
  * Chen et al. (2023) Zhaoxi Chen, Guangcong Wang, and Ziwei Liu.  Scenedreamer: Unbounded 3d scene generation from 2d image collections.  _IEEE transactions on pattern analysis and machine intelligence_ , 2023.
  * Deng et al. (2023) Jie Deng, Wenhao Chai, Jianshu Guo, Qixuan Huang, Wenhao Hu, Jenq-Neng Hwang, and Gaoang Wang.  Citygen: Infinite and controllable 3d city layout generation.  _arXiv preprint arXiv:2312.01508_ , 2023.
  * Deng et al. (2024) Jie Deng, Wenhao Chai, Junsheng Huang, Zhonghan Zhao, Qixuan Huang, Mingyan Gao, Jianshu Guo, Shengyu Hao, Wenhao Hu, Jenq-Neng Hwang, et al.  Citycraft: A real crafter for 3d city generation.  _arXiv preprint arXiv:2406.04983_ , 2024.
  * Gan et al. (2020) Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, et al.  Threedworld: A platform for interactive multi-modal physical simulation.  _arXiv preprint arXiv:2007.04954_ , 2020.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Hu et al. (2023) Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado.  Gaia-1: A generative world model for autonomous driving.  _arXiv preprint arXiv:2309.17080_ , 2023.
  * Hu et al. (2024) Ziniu Hu, Ahmet Iscen, Aashi Jain, Thomas Kipf, Yisong Yue, David A Ross, Cordelia Schmid, and Alireza Fathi.  Scenecraft: An llm agent for synthesizing 3d scenes as blender code.  In _Forty-first International Conference on Machine Learning_ , 2024.
  * Inoue et al. (2023) Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi.  Layoutdm: Discrete diffusion model for controllable layout generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10167–10176, 2023.
  * Kolve et al. (2017) Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al.  Ai2-thor: An interactive 3d environment for visual ai.  _arXiv preprint arXiv:1712.05474_ , 2017.
  * Lin et al. (2023) Chieh Hubert Lin, Hsin-Ying Lee, Willi Menapace, Menglei Chai, Aliaksandr Siarohin, Ming-Hsuan Yang, and Sergey Tulyakov.  Infinicity: Infinite-scale city synthesis.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 22808–22818, 2023.
  * Liu et al. (2024) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 26296–26306, 2024.
  * Lu et al. (2024) Fan Lu, Kwan-Yee Lin, Yan Xu, Hongsheng Li, Guang Chen, and Changjun Jiang.  Urban architect: Steerable 3d urban scene generation with layout prior.  _arXiv preprint arXiv:2404.06780_ , 2024.
  * Puig et al. (2018) Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba.  Virtualhome: Simulating household activities via programs.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 8494–8502, 2018.
  * Ranftl et al. (2020) René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  _IEEE transactions on pattern analysis and machine intelligence_ , 44(3):1623–1637, 2020.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 10684–10695, 2022.
  * Schrotter & Hürzeler (2020) Gerhard Schrotter and Christian Hürzeler.  The digital twin of the city of zurich for urban planning.  _PFG–Journal of Photogrammetry, Remote Sensing and Geoinformation Science_ , 88(1):99–112, 2020.
  * Shen et al. (2022) Yuan Shen, Wei-Chiu Ma, and Shenlong Wang.  Sgam: Building a virtual 3d world through simultaneous generation and mapping.  _Advances in Neural Information Processing Systems_ , 35:22090–22102, 2022.
  * Wang et al. (2023) Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, and Jiwen Lu.  Drivedreamer: Towards real-world-driven world models for autonomous driving.  _arXiv preprint arXiv:2309.09777_ , 2023.
  * Wang et al. (2024) Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang.  Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 14749–14759, 2024.
  * Wu et al. (2024) Zhennan Wu, Yang Li, Han Yan, Taizhang Shang, Weixuan Sun, Senbo Wang, Ruikai Cui, Weizhe Liu, Hiroyuki Sato, Hongdong Li, et al.  Blockfusion: Expandable 3d scene generation using latent tri-plane extrapolation.  _arXiv preprint arXiv:2401.17053_ , 2024.
  * Xia et al. (2018) Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese.  Gibson env: Real-world perception for embodied agents.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 9068–9079, 2018.
  * Xia et al. (2020) Fei Xia, William B Shen, Chengshu Li, Priya Kasimbeg, Micael Edmond Tchapmi, Alexander Toshev, Roberto Martín-Martín, and Silvio Savarese.  Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environments.  _IEEE Robotics and Automation Letters_ , 5(2):713–720, 2020.
  * Xie et al. (2024) Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, and Ziwei Liu.  Citydreamer: Compositional generative model of unbounded 3d cities.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 9666–9675, 2024.
  * Xu et al. (2023) Fengli Xu, Jun Zhang, Chen Gao, Jie Feng, and Yong Li.  Urban generative intelligence (ugi): A foundational platform for agents in embodied city environment.  _arXiv preprint arXiv:2312.11813_ , 2023.
  * Yang et al. (2024) Jihan Yang, Runyu Ding, Ellis Brown, Xiaojuan Qi, and Saining Xie.  V-irl: Grounding virtual intelligence in real life.  _arXiv preprint arXiv:2402.03310_ , 2024.
  * Ye et al. (2023) Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang.  Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.  _arXiv preprint arXiv:2308.06721_ , 2023.
  * Zeng et al. (2024) Xianfang Zeng, Xin Chen, Zhongqi Qi, Wen Liu, Zibo Zhao, Zhibin Wang, Bin Fu, Yong Liu, and Gang Yu.  Paint3d: Paint anything 3d with lighting-less texture diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 4252–4262, 2024.
  * Zhang et al. (2023) Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 3836–3847, 2023.
  * Zhou et al. (2024) Mengqi Zhou, Jun Hou, Chuanchen Luo, Yuxi Wang, Zhaoxiang Zhang, and Junran Peng.  Scenex: Procedural controllable large-scale scene generation via large-language models.  _arXiv preprint arXiv:2403.15698_ , 2024.


11institutetext: Philipp M. Scholl, Benjamin Völker and Bernd Becker
22institutetext: University of Freiburg, 22email:
<EMAIL_ADDRESS>33institutetext: Kristof Van
Laerhoven 44institutetext: University of Siegen 44email<EMAIL_ADDRESS>
# A Multi-Media Exchange Format for Time-Series Dataset Curation

Philipp M. Scholl, Benjamin Völker, Bernd Becker, Kristof Van Laerhoven

###### Abstract

Exchanging data as character-separated values (CSV) is slow, cumbersome and
error-prone. Especially for time-series data, which is common in Activity
Recognition, synchronizing several independently recorded sensors is
challenging. Adding second level evidence, like video recordings from multiple
angles and time-coded annotations, further complicates the matter of curating
such data. A possible alternative is to make use of standardized multi-media
formats. Sensor data can be encoded in audio format, and time-coded
information, like annotations, as subtitles. Video data can be added easily.
All this media can be merged into a single container file, which makes the
issue of synchronization explicit. The incurred performance overhead by this
encoding is shown to be negligible and compression can be applied to optimize
storage and transmission overhead.

###### Keywords:

Data Curation; Activity Recognition; Multi-Media Format; Data Storage; Comma-
Separated-Values

##  1 Introduction

At the heart of each Activity Recognition task is a dataset. This dataset
might be formed from multiple media streams, like video, audio, motion and
other sensor data. Recorded at different rates, sparsely or uniformly sampled
and with different numerical ranges, these streams are challenging to process
and store. Commonly, datasets are published in multiple character-separated
values (CSV) files, either with a constant rate or time-coded. For small,
independent time-series this is a worthwhile approach, mostly due to its
simplicity and universality. However, when observing with multiple independent
sensors, synchronization quickly becomes a challenge. Different rate
recordings have to be resampled, time-coded files have to be merged. Storing
such data in several (time-coded) CSV files hides this issue, until the
dataset is going to be used. Furthermore parsing CSV files incurs a large
performance and storage overhead, compared to a binary format.

Examples of Activity Recognition Datasets HASC Challenge Kawaguchi2011 : >100
subjects, time-coded CSV files. Box/Place-Lab Intille2006 : A sensor-rich
home, in which subjects are monitored for long terms. Data is available in
time-coded CSV files. Opportunity Roggen2010 : 12 subjects were recorded with
72 on- and off-body sensors in an Activities of Daily Living (ADL) setting.
Multiple video cameras were used for post-hoc annotations. Data is published
in synchronized CSV files. Kasteren’s Home Kasteren2010 : 12 sensors in 3
houses. Data is stored in matlab files. Borazio’s Sleep Borazio2014 : 1
sensor, 42 subjects. Data is stored in numpy’s native format. Freiburg
Longitudinal kristof : 1 sensor, 1 subject, 4 weeks of continuous recording.
Data is stored in numpy’s native format.

One alternative is storing such datasets in databases, like SQL, NOSQL or
HDF5-formats. This eases the management of large datasets, but shows the same
issues as a CSV format, namely that there is no direct support for time-series
or video data. An alternative approach is to store time-series in existing
multi-media formats. Encoding all multi-media data in one file allows to merge
streams, to synchronize them and to store (meta-)data in a standardized
format. In the next section, we will first look at the formats commonly used
to exchange data in Activity Recognition, afterwards we detail a multi-media
format and evaluate the incurred performance and storage overhead of each
format.

##  2 Related Work

In the classic activity recognition pipeline Bulling2014 , the first step is
to record and store sensor data. The observed activities, executed by humans,
animals or other actors, are recorded with different sensors. Each sensor
generates a data _stream_ , whether this is a scene camera for annotation
purposes, a body-worn motion capturing system or binary sensors like switches.
Sampled at different rates, with differing resolutions, ranges, units and
formats these streams offer a large variety of recording parameters. These
parameters are usually documented in an additional file that resides next to
the actual data The actual data is commonly stored in a CSV file, in a binary
format for Matlab or NumPy, or in format specific to some Machine Learning
framework like ARFF Hall2009  or libSVM Chang2011 .

Synchronizing such multi-modal data, i.e. converting this data to the same
rate and making sure that recorded events happened at the same time presents a
major challenge Possible approaches range from offline recording with post-hoc
synchronization on a global clock, to live streaming with a minimum delay
assumption - all but the last one require some form of clock synchronization
and careful preparation. Storing events with timestamps on a global clock is
then one possible way to allow for post-recording synchronization, i.e. each
event is stored as a tuple of <timestamp, event data>.

The subsequent step of merging such time-coded streams often requires to adapt
their respective rates. Imagine, for example, a concurrent recording of GPS at
$3$\mathrm{H}\mathrm{z}$$ and acceleration at $100$\mathrm{H}\mathrm{z}$$. To
merge both streams: will GPS be upsampled or acceleration downsampled, or both
resampled to a common rate? Which strategy is used for this interpolation, is
data simply repeated or can we assume some kind of dependency between samples?
How is jitter and missing data handled? These question need to be answered
whenever _time-coded_ sensor data is used. A file format which makes the
choice of possible solutions explicit is the goal of this paper.

##  3 Multi-Media Container Approach

Sensor data commonly used in Activity Recognition is not different from low-
rate audio or video data. Common parameters are shared, and one-dimensional
sensor data can be encoded with a lossless audio codec for compression. Rate,
sample format and number of channels need to be specified for an audio track.
The number of channels is equivalent to the number of axis an inertial sensor
provides, as well as its sample rate. The sample format, i.e. how many bits
are used to encode one measurement, is also required for such a sensor. Other
typical parameters, like the range settings or conversion factor to SI units
(if not encoded as such), can be stored as additional meta-data, as those are
usually not required for an audio track.

Lossless compression, like FLAC flac  or WavPack wavpack , can be applied to
such encoded data streams. This allows to trade additional processing for
efficient storage. In the evaluation section several lossless schemes are
compared. These include the general LZMA2 and ZIP compressors, and the FLAC
flac  and WavPack wavpack  audio compressors. All but the first two can be
easily included in multi-media container formats. To use audio streams, data
needs to be sampled at a constant rate, i.e. the time between two consecutive
samples is constant and only jitter smaller than this span is allowed. Put
differently, the time between two consecutive data samples $t_{i}$ and
$t_{i+1}$ at frame $i$ must always be less than or equal to the sampling rate
$r$: $\forall i\in N:t_{i+1}-t_{i}\leq\frac{1}{r}$. Compared to time-coded
storage, the recording system has be designed to satisfy this constraint.
Problems with a falsely assumed constant rate recording setup will therefore
surface faster. Especially in distributed recording settings, where the just
mentioned constraint is checked only against local clocks which might drift
away from a global clock, is a common challenge.

Sparsely sampled events can be encoded as subtitles. Here, each sample is
recorded independently of its preceding event, i.e. the above mentioned
constraint does not hold. Each event needs to be stored with a time-code and
the actual event data. Depending on the chosen format, this can also include a
position in the frame of an adjacent video stream or other information. For
example, this can be used to annotate objects in a video stream. A popular
format is the Substation Alpha Subtitle (SSAsubstation ) encoding, which
includes the just mentioned features. Since data is encoded as strings, it is
suitable for encoding ground truth labels. To a limited extent, since no
compression is available, it can be used for sensor events as well. For
example, low rate binary sensors, like RFID readers could be encoded as
subtitles.

Encoded sensor and subtitle data can then be combined with audio and video
streams in a multi-media container format. One such standard is the Matroska
matroska  format, that is also available in a downgraded version called WebM
webm  for webbrowsers. Once the data streams are combined into one such file,
this data can be ”played” back in a synchronous manner. This means that
streams recorded at different rates, and in different formats, need to be
converted to a common rate and possibly common format. Meta-data that contains
additional information like recording settings, descriptions and identifiers
can be stored in addition to the parameters already contained in the stream
encoding. For this task off-the-shelf software, like FFmpeg ffmpeg  can be
used, which also provides functionality like compression, resampling, format
conversion and filtering. Annotation tasks can be executed with standard
subtitle editing software, discouraging the creation of yet another annotation
tool. Furthermore, video streaming servers can be used for transporting live
sensor data recordings to remote places.

The use of such a standard format for curating datasets allows for re-using
existing software, however not without limitations. Asynchronous, also called
sparsely sampled, data recorded at high rates is not supported. This mainly
stems from the simplifying assumption that streams are recorded with a
constant rate. We presume that satisfying this constraint while recording to
be easier than handling asynchronicity later on. For example, breaks, shifts
or jitter due to firmware bugs can be detected earlier. In general this is a
hard limitation, however different data types can also be encoded in multiple
streams. Also, the en- and decoding overhead might be a limitation, which we
will look at in the next section.

(a) storage efficiency

(b) runtime overhead

Figure 1: Comparison of runtime overhead and storage efficiency, which is to
be traded off for each format. Each is given relative to a deflated CSV file.

Compared to the de-facto standard of using CSV files, encoding sensor data as
audio, annotations as subtitles and audio- and video-data in standard formats
provides several improvements. Important parameters like sampling rate, format
and number of axes are included in the file. Adding additional information as
meta-data leads to a compleltly _self-descriptive_ format. _Synchronous_
playback of multiple streams, which requires re-sampling, is supported by off-
the-shelf software. Related problems, like un-synchronized streams can be
caught earlier, since this step is explicit. The container format is
_flexible_ enough to support different number formats, i.e. values can be
encoded as floats or integers of varying bit-size. Optional compression leads
to _compact_ storage, which allows for efficient storage and transmission.
Additionally, when thinking about large datasets, such a container format
requires _divisible_ storage. This functionality (seeking without reading the
whole dataset into memory111which would be required for time-coded storage) is
provided. Such divisible storage also allows for streaming applications,
which, for multi-media format, also provides network protocols to cover
transmission via un-reliable links.

##  4 Evaluation

Compressing sensor data as an audio stream incurs an en- and decoding
overhead, and provides optimized storage. In this section both are quantified.
By a repetitive measurement of the relative wall clock time for decompression,
the processing overhead is measured. This runtime overhead is reported as the
fraction of reading time relative to reading and _converting_ the CSV file
into main memory. The compression factor is determined by comparing the number
of bytes required to store the compressed file to the original, deflated CSV
file. Binary and text-based storage is compared. The Zip and LZMA2 algorithms
are used for general byte-wise compression, and the lossless FLAC and WavPack
compressor for audio-based compression. Additionally storing in the sqlite3,
and MongoDB database, as well as the HDF5 format is compared. The approach of
compressing binary files with a general compressor is used by Numpy, MongoDB
and HDF5 for example.

The test were run on the Kitchen CMU de2009guide , Opportunity Roggen2010 ,
HASC Challenge Kawaguchi2011  and on twenty days of the Freiburg Longitudinal
Wrist kristof  datasets. A machine with an i7-4600U CPU running at
$2.1$\mathrm{GHz}$$ with $8$\mathrm{GB}$$ of memory was used for all tests. 1b
and 1a show the results of these tests, _csv/gz_ refers to a zip-compressed
CSV file, _csv/bz2_ to an LZMA2 compressed file222the XZ utils package was
used, _f32_ refers to a $32-$\mathrm{bit}$$-float binary format, _wv32_ to
WavPack compression of $32-$\mathrm{bit}$$-floats, and _flac_ to the _FLAC_
compressor which only supports $24$\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$$
values. _nosql_ refers to storing the data in a MongoDB (_NoSQL_) database,
_sql_ to sqlite3 storage, and _hdf5/gz_ to encoding the sensor data in a zip-
compressed HDF5 container. For _MongoDB_ storage, each data stream is stored
together with its metadata into a so called _collection_ inside the database.
Since MongoDB’s _BSON_ format only allows for certain data types, we choose
the $64$\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$$ _Double_ format to store
the individual event data with the default compression parameters. Equivalent
to the _MongoDB_ structure, each datastream is stored as a separate table in
the _sqlite_ database and the event data is stored in
$64$\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$$ _REAL_ format. The stream’s
metadata is stored in a separate table. De- and encoding was performed using
the corresponding python interfaces _pymongo_ and _sqlite3_. The _HDF5_ hdf5
data was generated and de-/encoded using the _h5py_ python interface and
stored with zip-compression.

###  4.1 Processing Overhead

It is interesting to check how much overhead is incurred for decompression by
each storage paradigm, as this gives an insight if data needs to be stored in
an intermediate format while evaluating a recognition pipeline. If the
overhead is comparatively low, no intermediate storage format is required and
data can always be loaded from such a file. However, should decoding require
more processing time than the actual processing, an intermediate file needs to
be used.

Naturally such a format would be binary, at best a memory image which can be
mapped into main memory as a $32$\mathrm{bit}$$-float. The baseline is,
therefore, the time required to convert a CSV from disk into a binary format
in memory. The fraction of time required to do the same for each storage
scheme is reported in 1b. Each test is repeated six times, and the first run
is discarded, i.e. data is always read from the disk cache.

Just parsing a CSV file incurs an up to hundred-fold overhead compared to
reading a binary file (_f32_ in 1b). Compressing CSV data333note that the 1b
represent the factor between the compression and simply _reading_ an
uncompressed CSV file can increase the runtime by $1.4-3.0$ times. So, looking
only at runtime performance a CSV or compressed CSV file should hardly be used
for large datasets. When comparing compression schemes, it can be seen that a
$32$\mathrm{bit}$$ WavPack compression provides the second lowest runtime
overhead, only the hdf5-scheme is roughly twice as fast. Storing data into a
_MongoDB_ database comes with the cost that interfacing is done over a TCP
socket in plaintext JSON format, which incurs an up to $4$-times overhead
compared to CSV, and at least $100$-fold compared to raw binary storage. A
trade-off between storage efficiency and performance has to be found.

###  4.2 Storage Efficiency

General compression and audio compression algorithms were tested. Raw binary,
WavPack wavpack  and FLAC flac  compression were taken from the FFmpeg ffmpeg
suite with default parameters. 1a shows the amount of compression that was
achieved for each dataset per algorithm compared to uncompressed CSV files.

The used datasets show different characteristics found in other datasets as
well. For example the Longitudinal kristof  dataset can be massively
compressed with general algorithms, almost down to $2\%$ of its original size.
This is mainly owed to the fact that the contained acceleration data was
recorded with a resolution of only 8-bits, and that a run-length compression
was already applied during recording. This run-length compression is deflated
for CSV storage first, adding a lot of redundancy. Generally, text formats
provide a larger storage efficiency only when less characters are required per
sample than their binary counterparts.

This effect is partially visible for the kitchen dataset de2009guide . The
relative storage requirements for binary storage (f32 in 1a) is a lot larger
than for other datasets. Here, the text representation of values is smaller
since they range from $0-1$, i.e. the text values are almost always of the
same length. Other datasets provide a range of values with larger pre-
decimals, hence a longer text representation. The maximum dynamic range that
can be stored with a text-based format more efficiently is therefore limited
to the (decimal) encoding, (less than $10000$ for five digits), while a
comparable binary encoding can range up to $2^{5*8}$.

When optimizing data for space efficiency, the encoding of each value is the
most critical factor. Limiting the number of
$\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$ per value, in essence assuming a
limited dynamic range of the encoded signal, has the strongest influence on
the storage efficiency. The 24 bit flac encoding shows the best overall
efficiency due to this. If a dynamic range of more than $2^{24}$ is required,
the wavpack encoding should be considered. However, when encoding values in
text format with a limited dynamic range (equivalent to four characters), a
text compression algorithm is not worse than encoding data in binary format.
For the general case and when binary storage can be used, the WavPack
compression provides the same storage efficiency as the more general LZMA2
compressor and provides comparable compression factors to the hdf5 format.

NoSql, and SQL storage do hardly provide a benefit over CSV concerning storage
efficiency in our tests. For SQL, this is probably an implementation artifact,
since sqlite3 stores date internally as a string representation. NoSql,
represented via MongoDB here, stores data in a compressed binary JSON format.
This provides a benefit only for one dataset, which contains a lot of
redundancy.

##  5 Conclusion

Curated time-series data provides the basis for comparing Activity Recognition
and other Machine Learning approaches in an objective and repeatable manner.
This data usually includes low-rate sensor data, e.g. motion sensors, time-
coded annotations, and second-level evidence like video and audio recordings.
The state of the art for exchanging this data seems to be a mix of time-coded
CSV format, and dedicated audio- and video-codecs. Synchronizing the stored
data-streams is usually not done by the dataset provider and the dataset
consumer is left with this challenge that usually requires information of the
recording setup. This is especially problematic when video or audio data is
recorded in addition to sensor data. The HDF5 which can provide a smaller
decoding overhead, provides no support for decoding audio- and video-data
directly.

The CSV format incurs a large overhead both in runtime and storage. A possible
alternative, with lower overhead, is presented here. Motion and other sensor
data, as well as extracted features, can be stored in lossless audio formats.
Ground truth labels and other time-coded information can be stored in subtitle
format. These streams can then be merged in a common multi-media container
(e.g. Matroska), with additional video streams. One recording session is
stored in a single file, that can be _self-descriptive_ , _synchronized_ ,
with a fitting _storage-runtime trade-off_ for multi-media data and supports
_live-streaming_.

## References

  * (1) Borazio, M., Berlin, E., Kücükyildiz, N., Scholl, P.M., Van Laerhoven, K.: Towards Benchmarked Sleep Detection with Inertial Wrist-worn Sensing Units. In: Healthc. Informatics (2014)
  * (2) Bryant, D.: The WavPack Codec, https://www.wavpack.org/
  * (3) Bulling, A., Blanke, U., Schiele, B.: A tutorial on human activity recognition using body-worn inertial sensors. ACM Comput. Surv. 46(3), 1–33 (2014), http://dl.acm.org/citation.cfm?doid=2578702.2499621
  * (4) Chang, C.C., Lin, C.J.: LIBSVM: A Library for Support Vector Machines 2(3), 1–27 (2011)
  * (5) group, F.: FFmpeg, https://www.ffmpeg.org/
  * (6) Group, T.H.: High-performance data management and storage suite, https://www.hdfgroup.org/solutions/hdf5/
  * (7) Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The WEKA data mining software. SIGKDD Explor. Newsl. 11(1), 10 (2009), http://portal.acm.org/citation.cfm?doid=1656274.1656278
  * (8) Intille, S.S., Larson, K., Tapia, E.M., Beaudin, J.S., Kaushik, P., Nawyn, J., Rockinson, R.: Using a Live-In Laboratory for Ubiquitous Computing Research. Pervasive Comput. pp. 349–365 (2006)
  * (9) Kasteren, T.L.M.V., Englebienne, G., Kr, B.J.A.: Human Activity Recognition from Wireless Sensor Network Data : Benchmark and Software. In: Act. Recognit. Pervasive Intell. Environ. (2010)
  * (10) Kawaguchi, N., Ogawa, N., Iwasaki, Y.: Hasc challenge: gathering large scale human activity corpus for the real-world activity understandings. Proc. 2nd Augment. Hum. Int. Conf. p. 27 (2011), http://dl.acm.org/citation.cfm?id=1959853
  * (11) van Laerhoven, K.: Longitudinal Wrist Motion Dataset, https://earth.informatik.uni-freiburg.de/application/files/hhg{_}logs/0089/index.html
  * (12) Lamparter, D.: Advanced Sub Station Alpha, http://fileformats.wikia.com/wiki/SubStation{_}Alpha
  * (13) Matroska, N.P.O.: The Matroska File Format (2016), https://www.matroska.org/
  * (14) Roggen, D., Calatroni, A., Rossi, M., Holleczek, T., Kilian, F., Tr, G., Lukowicz, P., Bannach, D., Pirkl, G., Ferscha, A., Doppler, J., Holzmann, C., Kurz, M., Holl, G., Creatura, M., Mill, R.: Collecting complex activity datasets in highly rich networked sensor environments. In: Int. Conf. Networked Sens. Syst. (2010)
  * (15) De la Torre, F., Hodgins, J., Montano, J., Valcarcel, S., Forcada, R., Macey, J.: Guide to the carnegie mellon university multimodal activity (cmu-mmac) database. Robotics Institute, Carnegie Mellon University (2009)
  * (16) WebM, C.: The WebM File Format, http://www.webmproject.org/
  * (17) Xiph.org, F.: The Free Lossless Audio Codec (FLAC), https://www.xiph.org/flac/


# MaskedSpeech: Context-aware Speech Synthesis with Masking Strategy

###### Abstract

Humans often speak in a continuous manner which leads to coherent and
consistent prosody properties across neighboring utterances. However, most
state-of-the-art speech synthesis systems only consider the information within
each sentence and ignore the contextual semantic and acoustic features. This
makes it inadequate to generate high-quality paragraph-level speech which
requires high expressiveness and naturalness. To synthesize natural and
expressive speech for a paragraph, a context-aware speech synthesis system
named MaskedSpeech is proposed in this paper, which considers both contextual
semantic and acoustic features. Inspired by the masking strategy in the speech
editing research, the acoustic features of the current sentence are masked out
and concatenated with those of contextual speech, and further used as
additional model input. The phoneme encoder takes the concatenated phoneme
sequence from neighboring sentences as input and learns fine-grained semantic
information from contextual text. Furthermore, cross-utterance coarse-grained
semantic features are employed to improve the prosody generation. The model is
trained to reconstruct the masked acoustic features with the augmentation of
both the contextual semantic and acoustic features. Experimental results
demonstrate that the proposed MaskedSpeech outperformed the baseline system
significantly in terms of naturalness and expressiveness.

Index Terms—  TTS, Speech synthesis, prosody, speech editing, context, masked
speech

##  1 Introduction

With the development of deep learning technology, current text-to-speech (TTS)
systems [1, 2, 3, 4] could generate high-quality speech. However, most speech
synthesis systems only consider the information in the current sentence to be
synthesized and ignore the contextual text and speech. With the rising
requirements on the naturalness and expressiveness of TTS systems, such as
synthesizing audiobooks, these speech synthesis systems lack the ability to
generate paragraph-level speech with adequate naturalness and expressiveness,
as human speech is context coherent in a paragraph.

Thus, recent works [5, 6, 7, 8, 9, 10] incorporated semantic features of the
current sentence or contextual sentences to improve the prosody of synthesized
speech, especially for paragraph speech generation. Xiao et al. [9] extracted
semantic information from the current sentence by a pretrained BERT [11] model
and showed that the incorporation of semantic features could improve the
prosody of synthesized speech. Then works [5, 6, 7, 10] further employed cross
utterances semantic features in the speech synthesis systems, and all of them
showed significant rhythm-enhancing effectiveness. Although contextual
semantic information could improve the prosodic performance to a certain
extent, these works do not take into account the variation of contextual
acoustic features, which is also very important for generating expressive and
natural speech for a paragraph.

Some works [12, 13, 6] used acoustics features to improve the prosody
modeling, but most of them extracted sentence-level prosody representation and
lost the detailed information of contextual acoustic features. In addition,
for the reference-based models [12, 13], it’s non-trivial to select proper
reference audio. [14] and [15] introduced an acoustic encoder to learn fine-
grained prosody features instead of utterance-level features and claimed that
their models could improve prosody generation and relive the one-to-many
mapping issue. However, the prosody features need to be predicted by the text
when reference audio is unavailable during inference, this led to another one-
to-many mapping issue, as the same text can potentially be used to predict
acoustic features from different speech.

To incorporate both contextual semantic and acoustic features into the TTS
model and improve prosody generation for a paragraph, we proposed MaskedSpeech
in this paper. Inspired by the speech editing works [16, 17, 18], acoustic
features of the current sentence are masked out and concatenated with those of
neighboring sentences, and further used as augmented input for the proposed
network. Then the network could learn fine-grained prosody features from the
contextual speech, and no prosody prediction from text is required.
Neighboring texts are also concatenated with current text and sent into the
phoneme encoder of the proposed model, which helps the model learn fine-
grained semantic features from the contextual text. Furthermore, a cross-
utterance (CU) encoder is used to extract coarse-grained sentence-level
semantic representation. The proposed MaskedSpeech is trained by
reconstructing the masked acoustic features, by utilizing the contextual text,
contextual speech, and sentence-level semantic representation, which enables
MaskedSpeech to improve prosody modeling and alleviate the one-to-many mapping
issue naturally. Experimental results indicate that the proposed MaskedSpeech
can significantly improve the naturalness and expressiveness of synthesized
speech over the baseline Fastspeech2 model.

##  2 Related Work

This section reviews the acoustic masking strategies used in speech
representation learning and speech editing tasks, which is the inspiration of
the proposed model. Unsupervised speech representation learning [19, 20, 21]
has demonstrated its superiority in the field of speech recognition and
related downstream tasks. To learn meaningful speech representation, these
technologies usually mask several frames of acoustic features randomly, and
contrastive learning is used to learn discrete quantized speech units as
latent representations.

Although the success of speech representation learning in speech understanding
tasks, it’s not suitable for the speech synthesis task due to its discrete
nature. MAM [22] and FAT-MLM [23] masked several acoustic segments randomly
and used the masked acoustic features as network input, then Transformer-based
[24] network is employed to reconstruct the masked acoustic features. But the
quality of the reconstructed acoustic features could not meet the requirement
of speech synthesis. Bai et al. [16] proposed a network that uses masked
acoustic features as input and the network is trained to predict high-quality
acoustic features, then their proposed network could be further applied to
speech synthesis and speech edit tasks. CampNet [17] introduced a speech edit
model, the acoustic features are randomly masked out by a prior ratio and then
sent to the decoder for masked speech reconstruction. Each decoder layer used
an attention mechanism to learn contextual features from unmasked part.

All the mentioned works for speech reconstruction try to predict high-quality
and contextual consistent acoustic features by learning the context
relationship of masked and unmasked speech. Inspired by these works, we
incorporate the masking strategy into cross-utterance speech synthesis task to
improve the prosody modeling, with the assumption that both the textual and
acoustic context could facilitate the prosody improvement of speech synthesis.
Instead of masking on word-level speech for speech editing tasks, the acoustic
features of the current sentence are concatenated with those of neighboring
sentences, and then acoustic features from the current sentence are masked
out. The proposed model is trained to reconstruct the masked acoustic
features. In other words, we proposed a novel context-aware speech synthesis
method that reconstructs speech for the current sentence with improved prosody
by utilizing both the semantic and acoustic features of contextual sentences.

##  3 Proposed Method

The proposed MaskedSpeech utilizes FastSpeech2 [25] as the network backbone.
The model learns to reconstruct the masked acoustic features with both
contextual semantic and acoustic features. MelGAN [26] is used as the neural
vocoder in the experiments to convert acoustic features to speech.

Fig. 1:  The architecture of our proposed MaskedSpeech model.

###  3.1 The MaskedSpeech Model

Fig. 1 illustrates the network architecture of the proposed MaskedSpeech
model. The backbone network consists of a phoneme encoder, a variance adaptor,
a decoder, and a PostNet module. Besides, to improve prosody for the speech in
a paragraph by contextual features, a masked mel-encoder is designed to
extract local prosody features from the concatenated mel-spectrograms, and
then the decoder takes the output of the masked mel-encoder as augmented input
and learns global prosody dependency from the contextual speech. Furthermore,
a CU text encoder is employed to obtain CU semantic features from neighboring
sentences.

####  3.1.1 Learning from contextual text

The prosody of the current sentence could be impacted by the relative sentence
position, discourse relations (DRs) in neighboring sentences, the emotion of
contextual sentences, etc. To improve the prosody modeling with contextual
sentences, the concatenated phoneme sequence of neighboring sentences are sent
into the phoneme encoder. Further, a CU encoder is used to incorporate
sentence-level semantic representation.

The phoneme sequences of the previous sentence, current sentence, and
following sentence are concatenated as the input of the phoneme encoder,
rather than only using the phonemes of the current sentence in general
methods. Thus, the phoneme encoder could learn fine-grained phoneme-level
semantic features implicitly from the contextual sentences. Following our
previous work [5], Paired BERT embedding (PBE) is used to extract sentence-
level coarse-grained semantic features between sentence pairs in the CU
encoder. As shown in Fig. 1, the neighboring sentences
$\\{u_{N-L},...,u_{N},u_{N+1},...,u_{N+L-1}\\}$ can be used to derive 2$L$
sentence pairs, where $u_{N}$ means the current sentence and $L$ is the number
of sentences taken into account in the past or following context. Then PBE is
extracted by a pre-trained BERT model from the $[CLS]$ token. A multi-head
attention module is used to incorporate PBEs into the proposed model, where
PBEs are considered as keys and values, and the output of the phoneme encoder
is considered as the query. The output of the CU encoder is concatenated with
the output of the phoneme encoder, then linear projected.

####  3.1.2 Learning from contextual speech

Human speech is continuous in a paragraph coherently, the prosody and emotion
of the current sentence could be affected by the contextual speech. Contextual
speech usually contains very rich, expressive, and detailed prosody
information, which could be utilized to improve the prosodic performance for
the current sentence in speech synthesis systems, especially for the speech
generation in a paragraph.

A masked mel-encoder is incorporated into the FastSpeech2 network in the
proposed MaskedSpeech. The acoustic features from contextual sentences, i.e.
the mel-spectrograms of the previous sentence and following sentence, and the
current sentence, are concatenated and used as the input for the masked mel-
encoder. The acoustic features of the current sentence are masked with a
special MASK token. The masked mel-encoder consists of two 1-dimensional
convolution (Conv1d) layers, it could learn to extract local prosody features
from the acoustic features in the neighboring sentences. ReLU activation
function is used in each Conv1d layer, followed by layer normalization [27]
and dropout [28].

The output of the masked mel-encoder is concatenated with the output of the
variance adaptor and then projected to the dimension of the decoder. The
decoder consists of several self-attention-based Conformer [29] blocks, so the
decoder could learn both local and global prosody dependency implicitly from
the contextual acoustic features. The predicted acoustic features of
contextual sentences are replaced with ground truth features, and only
acoustic features for the current sentence are kept. Then a PostNet module
with 5 Conv1d layers is used to refine the predicted acoustic features. As the
PostNet consists of Conv1d layers and learns local features, it could improve
the boundary prosody transition for the current sentence.

By conditioning on contextual semantic and acoustic features, the proposed
MaskedSpeech could alleviate the one-to-many mapping problem and generate
speech with high naturalness and expressiveness for a paragraph.

###  3.2 Training and Inference Strategies

During training, the model is trained to reconstruct the masked acoustic
features with the augmentation of both the semantic and acoustic features from
contextual sentences. We only compute training objectives for the current
sentence and ignore the loss of neighboring sentences, the losses include
phoneme-level duration prediction loss, pitch and energy prediction losses,
and mel-spectrogram reconstruction loss.

At the inference stage, the speech of the previous or following sentence is
not always available. In general, we usually read a paragraph in order.
Therefore, there usually exists intact paragraph-level sentences and speech of
previous sentences, and do not always have the following speech. Thus, at
inference stage, it is assumed that there is only consecutive text and
previous speech as default. The speech from the previous sentence is taken as
the contextual speech for the synthesis procedure of the current sentence, and
the input form of the phoneme encoder and CU text encoder is the same as that
at the training stage. In addition, a preliminary experiment showed that there
is no noticeable deviation when not using the acoustic features of the
following speech.

Actually, the inputs of MaskedSpeech are very flexible. There could be no
acoustic features from neighboring sentences, and then MaskedSpeech
degenerates into a PBE-based TTS model. When inputting a single sentence and
masking speech for a few words, the model degenerates into a speech editing
model which can achieve inserting, replacing, and deleting part of an audio
file like we edit text.

Table 1: The naturalness MOS results of different models with 95% confidence intervals. Model | MOS
---|---
Ground truth |  4.41$\pm$0.07
Fastspeech2 [25] |  4.09 $\pm$0.06
MaskedSpeech | 4.19$\pm$0.06

##  4 Experiments

Table 2:  Objective evaluation results of different models. models | MSD |  F0 RMSE (Hz) |  VUV (%) | F0 CORR |  F0 SD (Hz)
---|---|---|---|---|---
Fastspeech2[25] | 4.502 | 31.938 | 9.402 | 0.726 | 39.712
PBE [5] | 4.466 | 31.147 | 8.687 | 0.740 | 40.245
MaskedSpeech | 4.591 | 30.464 | 10.267 | 0.753 | 41.462
Recordings | - | - | - | - | 45.250

###  4.1 Experimental Setup

A Chinese Mandarin audiobook corpus with 29 hours of recordings was used to
train the proposed model in our experiments. The corpus was recorded by a
professional male speaker. The text and recordings in this corpus are
consecutive.200 sentences were selected for testing, and 200 sentences for
validation, the remaining sentences are used for training. The test set
contains several paragraphs of consecutive sentences. The audio files were
down-sampled to 16kHz sample-rate. 80-dimensional mel-spectrograms were used
as the acoustic features in the experiment and were extracted with a frame
shift of 12 ms and frame length of 48 ms. The phoneme boundaries and duration
were extracted by Montreal forced alignment (MFA) [30] tool. Pitch was
extracted by the PyWORLD111https://github.com/JeremyCCHsu/Python-Wrapper-for-
World-Vocoder toolkit.

The pre-trained “bert-base-chinese” model [31] released by Huggingface
222https://github.com/huggingface/transformers was used to extract cross
utterance sentence embedding in the CU text encoder. The BERT model consists
of 12 Transformer layers, each layer contains 768 hidden units and 12-heads
are used for each attention layer. It is worth mentioning that the parameters
of the BERT model are fixed during training. The multi-head attention in the
CU text encoder has 4 heads and 512 hidden units. In our experiments, PBEs are
generated by 2 preceding sentences, 2 following sentences, and the current
sentence. The masked mel-encoder is constructed by 2 Conv1d layers, with a
filter size of 256 and kernel size of 3 in each layer.

The proposed model is trained by Adam [32] optimizer, with a batch size of 16
and noam [24] learning rate schedule. The learning rate is warmed up to a
maximum value of $10^{-3}$ in the first 4000 steps and then exponentially
decayed. Adam optimizer is configured with $\beta_{1}$=0.9, $\beta_{2}$=0.98
and $\epsilon$=$10^{-9}$. Mean squared error (MSE) is used as the training
objectives for pitch, energy, and duration prediction, and mean absolute error
(MAE) is used for the mel-spectrogram reconstruction. The model is trained by
a total of 200,000 steps, with L2 parameter regularization of weight 1e-5. A
Fastspeech2 model was trained and used as the baseline model. 333The audio
samples can be found at https://zyj008.github.io/MaskedSpeech/.

###  4.2 Subjective Evaluation

To evaluate the quality of the proposed MaskedSpeech model, the Mean Opinion
Score (MOS) evaluation is conducted to evaluate the naturalness and
expressiveness of the synthesized speech. 20 sentences were randomly selected
from the test set and synthesized by different models, each audio is rated by
10 native Chinese listeners. The listeners are asked to rate the speech’s
naturalness and expressiveness on a scale from 1 to 5 with intervals of 0.5.
Score 1 means the speech is unnatural completely and score 5 means the speech
is completely natural. Tab. 1 shows the MOS results. Our proposed MaskedSpeech
model achieved a higher MOS score than the baseline Fastspeech2 model in terms
of naturalness and expressiveness. This indicates that with the help of both
contextual semantic and acoustic features, the proposed model could improve
the prosody generation.

Table 3: Subjective preference scores (%) among three models, where N/P denotes “No Preference” and $p$ means the $p$-value of $t$-test between two systems. Fastspeech2 | PBE | MaskedSpeech | N/P | p
---|---|---|---|---
14.5 | - | 50.0 | 35.5 | <0.001
- | 22.5 | 48.5 | 29.0 | <0.001

###  4.3 Ablation Study on Contextual Features

In this section, an ablation study is conducted to evaluate the effectiveness
of both contextual semantic and acoustic features. With the incorporation of
contextual text and speech, it’s expected to help the synthesized audio to
have a suitable and consecutive speaking style and consistent prosody contour
under the condition of neighboring sentences. To evaluate the effectiveness of
these contextual features, two AB preference tests were conducted for the
ablation study. First, we remove both the CU text encoder and masked mel-
encoder from MaskedSpeech, which is a pure Fastspeech2 model. Then, we only
remove the masked mel-encoder, which is a PBE-based model. The MaskedSpeech
was compared with Fastspeech2 and the PBE-based model separately. To evaluate
the style consecutiveness and prosody consistency of synthesized speech with
contextual sentences, the synthesized audio is concatenated with its previous
and following recordings, then the judges are asked to select which one is
more natural and expressive according to the ground truth neighboring
recordings. Tab. 3 shows the results of AB preference tests. MaskedSpeech
outperforms the Fastspeech2 model significantly, which proves the
effectiveness of using contextual semantic and acoustic features. Furthermore,
MaskedSpeech outperforms the PBE-based model significantly, which proves the
fine-grained prosody representations learned from the masked acoustic features
can further improve the prosody modeling.

In addition, an objective evaluation was conducted on these three models. The
mel-spectrogram distortion (MSD), the root of mean square errors of F0 (F0
RMSE), the error rate of voicing/unvoicing flags (VUV), the Pearson
correlation factor of F0 (F0 CORR), and F0 standard deviation (F0 SD) were
adopted as the objective evaluation metrics. The predicted mel-spectrograms
were aligned towards their corresponding natural ones by dynamic time warping
(DTW). The F0s were aligned towards natural ones following the DTW paths from
mel-spectrogram alignment. Tab. 2 shows the objective results. Compared to the
F0 SD of 45.25Hz for natural recordings, the F0 SD metric from MaskedSpeech is
the closest, which is 41.462Hz. While the MSD of MaskedSpeech is the highest
among the three models, which may be caused by the higher variation of F0. In
addition, MaskedSpeech achieves the best F0 RMSE and F0 CORR among the three
models. The objective results further proved the advantages of MaskedSpeech on
prosody modeling.

Fig. 2:  Comparisons between the mel-spectrograms of three models and natural
recordings.

###  4.4 A Case Study

Fig. 2 shows the mel-spectrograms of the synthesized speech of three different
models and the natural recording. Each sub-figure shows the concatenated mel-
spectrograms of previous, current, and following sentences with distinct
silence as the boundary, only the central speech is the synthetic audio. Fig.
2 (d) shows that the speaking speed of the nature recording is relatively
faster and the fundamental frequency is lower. MaskedSpeech perceives this
property according to the contextual semantic and acoustic features and
generates speech with a relatively faster speaking rate and lower pitch.

##  5 Conclusion

A context-aware speech synthesis system is proposed in this paper, with
improved prosody by utilizing the contextual semantic and acoustic features
from neighboring sentences. The proposed model takes the concatenated and
masked mel-spectrograms as augmented input and learns to reconstruct the
masked mel-spectrograms. The phoneme encoder takes account of the phonemes
from context sentences and those of the current sentence. A CU text encoder is
used to extract cross-utterance semantic representation. Experimental results
showed the effectiveness of the proposed MaskedSpeech model.

## References

  * [1] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan _et al._ , “Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions,” in _Proc. ICASSP_ , 2018, pp. 4779–4783.
  * [2] Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T. Liu, “Fastspeech: Fast, robust and controllable text to speech,” in _Proc. NeurIPS_ , 2019\.
  * [3] C. Yu, H. Lu, N. Hu, M. Yu, C. Weng, K. Xu, P. Liu, D. Tuo, S. Kang, G. Lei, D. Su, and D. Yu, “DurIAN: Duration informed attention network for speech synthesis,” in _Proc. Interspeech_ , 2020.
  * [4] W. Song, X. Yuan, Z. Zhang, C. Zhang, Y. Wu, X. He, and B. Zhou, “Dian: duration informed auto-regressive network for voice cloning,” in _Proc. ICASSP_. IEEE, 2021, pp. 8598–8602.
  * [5] G. Xu, W. Song, Z. Zhang, C. Zhang, X. He, and B. Zhou, “Improving prosody modelling with cross-utterance bert embeddings for end-to-end speech synthesis,” in _Proc. ICASSP_. IEEE, 2021, pp. 6079–6083.
  * [6] J. Cong, S. Yang, N. Hu, G. Li, L. Xie, and D. Su, “Controllable Context-Aware Conversational Speech Synthesis,” in _Proc. Interspeech_ , 2021, pp. 4658–4662.
  * [7] Y.-J. Zhang and Z.-H. Ling, “Learning deep and wide contextual representations using bert for statistical parametric speech synthesis,” in _Proc. ICDSP_ , 2021, pp. 146–150.
  * [8] P. Oplustil-Gallegos, J. O’Mahony, and S. King, “Comparing acoustic and textual representations of previous linguistic context for improving text-to-speech,” in _Proc. SSW 11_ , 2021, pp. 205–210.
  * [9] Y. Xiao, L. He, H. Ming, and F. K. Soong, “Improving prosody with linguistic and bert derived features in multi-speaker based mandarin chinese neural tts,” in _Proc. ICASSP_. IEEE, 2020, pp. 6704–6708.
  * [10] L. Xue, F. K. Soong, S. Zhang, and L. Xie, “Paratts: Learning linguistic and prosodic cross-sentence information in paragraph-based tts,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 30, pp. 2854–2864, 2022.
  * [11] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proc. NAACL-HLT_ , Jun. 2019, pp. 4171–4186.
  * [12] Y. Wang, D. Stanton, Y. Zhang, R. Skerry-Ryan, E. Battenberg, J. Shor, Y. Xiao, F. Ren, Y. Jia, and R. A. Saurous, “Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis,” in _Proc. ICML_ , 2018, pp. 5167–5176.
  * [13] Y.-J. Zhang, S. Pan, L. He, and Z.-H. Ling, “Learning latent representations for style control and transfer in end-to-end speech synthesis,” in _Proc. ICASSP_. IEEE, 2019, pp. 6945–6949.
  * [14] Y. Liu, Z. Xu, G. Wang, K. Chen, B. Li, X. Tan, J. Li, L. He, and S. Zhao, “Delightfultts: The microsoft speech synthesis system for blizzard challenge 2021,” _arXiv preprint arXiv:2110.12612_ , 2021.
  * [15] M. Chen, X. Tan, B. Li, Y. Liu, T. Qin, S. Zhao, and T.-Y. Liu, “Adaspeech: Adaptive text to speech for custom voice,” _arXiv preprint arXiv:2103.00993_ , 2021.
  * [16] H. Bai, R. Zheng, J. Chen, M. Ma, X. Li, and L. Huang, “A3T: Alignment-aware acoustic and text pretraining for speech synthesis and editing,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 1399–1411.
  * [17] T. Wang, J. Yi, L. Deng, R. Fu, J. Tao, and Z. Wen, “Context-aware mask prediction network for end-to-end text-based speech editing,” in _Proc. ICASSP_. IEEE, 2022, pp. 6082–6086.
  * [18] D. Tan, L. Deng, Y. T. Yeung, X. Jiang, X. Chen, and T. Lee, “Editspeech: A text based speech editing system using partial inference and bidirectional fusion,” in _Proc. ASRU_. IEEE, 2021, pp. 626–633.
  * [19] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” _Proc. NeurIPS_ , vol. 33, 2020.
  * [20] A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, “Unsupervised Cross-Lingual Representation Learning for Speech Recognition,” in _Proc. Interspeech_ , 2021, pp. 2426–2430.
  * [21] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [22] J. Chen, M. Ma, R. Zheng, and L. Huang, “Mam: Masked acoustic modeling for end-to-end speech-to-text translation,” _arXiv preprint arXiv:2010.11445_ , 2020.
  * [23] R. Zheng, J. Chen, M. Ma, and L. Huang, “Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 12 736–12 746.
  * [24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [25] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Fastspeech 2: Fast and high-quality end-to-end text to speech,” 2021.
  * [26] K. Kumar, R. Kumar, T. de Boissiere, L. Gestin, W. Z. Teoh, J. Sotelo, A. de Brébisson, Y. Bengio, and A. C. Courville, “Melgan: Generative adversarial networks for conditional waveform synthesis,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [27] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _arXiv preprint arXiv:1607.06450_ , 2016.
  * [28] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” _The journal of machine learning research_ , vol. 15, no. 1, pp. 1929–1958, 2014.
  * [29] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu _et al._ , “Conformer: Convolution-augmented transformer for speech recognition,” _arXiv preprint arXiv:2005.08100_ , 2020.
  * [30] M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi,” in _Interspeech_ , 2017, pp. 498–502.
  * [31] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush, “Transformers: State-of-the-art natural language processing,” in _Proc. EMNLP_ , 2020, pp. 38–45.
  * [32] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _Proc. ICLR_ , San Diego, 2015.