11institutetext:  RheinMain University of Applied Sciences, Wiesbaden, Germany
11email: {marcel.lamott111Corresponding author, adrian.ulges,
<EMAIL_ADDRESS>22institutetext: Insiders Technologies GmbH,
Kaiserslautern, Germany
22email: {y.weweler<EMAIL_ADDRESS>33institutetext:
National University of Sciences and Technology, Islamabad, Pakistan
33email<EMAIL_ADDRESS>
# LAPDoc: Layout-Aware Prompting for Documents††thanks: Currently under review
at ICDAR2024.

Marcel Lamott 11 Yves-Noel Weweler 22 Adrian Ulges 11 Faisal Shafait 33 Dirk
Krechel 11 0000-0003-0984-5918 Darko Obradovic 22

###### Abstract

Recent advances in training large language models (LLMs) using massive amounts
of solely textual data lead to strong generalization across many domains and
tasks, including document-specific tasks. Opposed to that there is a trend to
train multi-modal transformer architectures tailored for document
understanding that are designed specifically to fuse textual inputs with the
corresponding document layout. This involves a separate fine-tuning step for
which additional training data is required. At present, no document
transformers with comparable generalization to LLMs are available That raises
the question which type of model is to be preferred for document understanding
tasks. In this paper we investigate the possibility to use purely text-based
LLMs for document-specific tasks by using layout enrichment. We explore drop-
in modifications and rule-based methods to enrich purely textual LLM prompts
with layout information. In our experiments we investigate the effects on the
commercial ChatGPT model and the open-source LLM Solar. We demonstrate that
using our approach both LLMs show improved performance on various standard
document benchmarks. In addition, we study the impact of noisy OCR and layout
errors, as well as the limitations of LLMs when it comes to utilizing document
layout. Our results indicate that layout enrichment can improve the
performance of purely text-based LLMs for document understanding by up to 15%
compared to just using plain document text. In conclusion, this approach
should be considered for the best model choice between text-based LLM or
multi-modal document transformers.

###### Keywords:

Document Understanding Large Language Models Layout Understanding Prompt
Enrichment

##  1 Introduction

In today’s business environment, companies face the problem of an ever growing
amount of digital documents that need to be processed. The possibilities of
smart devices to capture documents has lead to new digital business models
that make heavy use of camera captures, while promising high automation. This
induces a dramatic growth in digitized documents of varying quality that need
to be processed. In addition to document types such as invoices, forms,
complaints, receipts and notices, other, less standardized types of documents
are increasingly important, such as contract documents, business reports or
legal texts.

Understanding documents completely necessitates understanding of textual and
visual modalities as well as the comprehension of the spatial relations
between the document’s content elements, which guide the reading process and
are essential for interpretation [1, p. 1]. Recently, automated document image
understanding has taken strides forward: (i) Larger-scale benchmarks that
align with real applications [1, 2, 3, 4] allow for real world evaluation and
training. (ii) Self-supervised pre-training tasks – with which large amounts
of data can be leveraged without the need for hand-crafted annotations – have
led to multi-modal neural models. These models can either take document images
and text (e.g., extracted by OCR) as input [5, 6, 7, 8], or can operate end-
to-end from a purely visual input, essentially also learning OCR in the
process [9, 10].

One of the most prominent recent developments in the field of AI has been the
rise of large language models (LLMs) such as OpenAI’s ChatGPT [11]. These
models have been found to excel at various natural language understanding
tasks, and have been instruction-tuned to serve as open-domain problem
solvers. Key to their success is their scale – with large-scale training data
and billions of parameters – which leads to impressive capabilities [12]. In
contrast to the aforementioned multi-modal document comprehension models,
traditional LLMs process only text222Though there is a recent trend towards
multi-modal inputs, we will focus on large language models in the strict sense
here: The model’s input and output are text sequences.. By that the modality
of spatial layout, which seems vital for the processing of documents [6, 8],
is partially lost due to its reduction to a one-dimensional text sequence.

In this study we focus on an LLM-centric document comprehension pipeline that
fuses the text with document layout. First, a document’s content is extracted
with OCR, resulting in a set of words equipped with box geometries. Second,
this information is packaged into a purely textual representation that encodes
both the document’s text and its spatial structure. We will refer to this step
as "verbalization" in the following. Third, the resulting verbalized document
is combined with the task description, resulting in a prompt for a pre-trained
generative LLM, which solves the document comprehension task at hand without
further fine-tuning.

This pipeline offers two benefits: First, it exploits the superior knowledge
capacity and reasoning capabilities of LLMs – which at the present time have
been trained at larger scale and offer larger parametric capacity compared to
current multi-modal document-specific models. Second – which is particularly
relevant for practical applications – the pipeline offers the benefit of
simplicity, since it involves no model fine-tuning, thereby allowing us to
keep a single generalist model.

Specifically, we focus on the key step of document verbalization, which raises
several interesting questions: How well do LLMs perform at document
comprehension tasks that involve challenging layout reasoning, even with
no/little information on document geometry? How are LLMs influenced by the way
we feed them document representations and particularly, can we alter the
document representations in a way that allows a LLM to exploit document
geometry to achieve the same performance as a multi-modal model?

We investigate these questions with experiments on several document
understanding datasets including tasks from the DUE benchmark, SROIE, WebSRC,
and proprietary KIE datasets (from real-world industry scenarios). We examine
two LLMs, namely ChatGPT3.5333gpt-3.5-turbo-1106 and the open-source LLM
Solar[13].

Overall, we make the following contributions:

  1. 1.

A novel rule-based approach that enriches the prompts of existing text-centric
LLMs with spatial structure information from documents. The approach works
across various kinds of documents and tasks and can be applied to various
layouts without the need for fine-tuning.

  2. 2.

A set of comprehensive experiments using both research and real-world document
datasets as well as commercial and open-source models. We cover various
document-specific tasks, different reading orders, and effects of noise being
added to the OCR data.

  3. 3.

Besides quantitative results, we also explore LLMs’ limitations when it comes
to interpreting document layout in-depth on particularly challenging cases,
for which we have annotated a subset of SROIE. 444Our annotated SROIE-
Challenge dataset is available for future research, see Section 4.1..

  4. 4.

We also discuss efficiency issues, i.e. the extra tokens required for
different approaches of encoding spatial layout information into the prompts.

##  2 Related Work

LLMs: Language models built upon the attention-based transformer architecture
[14] are probably among the currently most intensely studied models in AI. Due
to the high growth they experienced, often involving several billion
parameters, the capacity and reasoning capabilities of these models have
rapidly progressed [12]. In addition to commercial providers such as OpenAI
[15], a variety of open-source models such as Llama 2 [16] or Solar[13] are
currently evolving. Two fundamental types of models are distinguished: (1)
Encoders, which generate representations of input texts and use them to can
make decisions about texts. They are equipped with additional head layers that
are be fine-tuned to the specific problem. (2) Decoders that generate text and
can be instructed using prompts without additional training. Recently, the
latter paradigm has emerged as the dominant approach, as the resulting LLMs
can serve as generalist agents for ad-hoc problem solving, without fine-tuning
to specific tasks. Instruction tuning is used as an additional training step
to facilitate this: It aims to bridge the gap between the LLM’s goal of next-
word prediction and the user’s goal of having the LLM follow human
instructions. [17]. Accordingly, we focus on instruction-tuned decoder models
in this work.

Multi-Modal Models: Many multi-modal models outsource OCR into preprocessing
and operate on a combined input of document image and recognized text+geometry
[5, 7, 18]: For example, the LayoutLM series, including the most recent
version LayoutLMv3 [19, 20, 21], utilizes a BERT-type transformer encoder
[22], which feeds on a concatenation of word embeddings and visual patch
embeddings, and is trained with several masked language modeling (MLM) and
word/patch alignment tasks. The model is applied to downstream tasks via fine-
tuning specialied head models. Similarly, DocFormer [23] applies an early
fusion of image and text signals and a pre-training with global text-image
alignment. UDOP [24] follows a generative approach and reconstructs text
layout by an encoder-decoder model.

Other models operate end-to-end, feeding only on the document image and
addressing text understanding in the process: Donut [9] uses an encoder-
decoder architecture, which is pretrained to recognize the document images’
text on large-scale real-world (IIT-CDIP) and synthetic documents. Similarly,
Dessurt [25] integrates OCR as part of its model. Many of the aforementioned
papers include ablation studies demonstrating that models benefit from
including geometry information in the input – when trained accordingly. In
this work, we extend this question to instruction-tuned LLMs.

The work most similar to ours is LATIN-Prompt by Wang et al. [26], who have
recently proposed a combination of a layout-aware document representation and
a task-aware prompting, and have also investigated fine-tuning in the process.
We extend on this work by (1) investigating multiple verbalization strategies,
(2) thoroughly treating the prompt templates as a free, dataset-agnostic
parameter to be optimized carefully and independently from the verbalization,
and (3) explore the limitations of LLMs’ layout reasoning capabilities in more
detail by inspecting challenge cases and evaluating the effect of layout and
OCR inaccuracies.

##  3 Approach

Figure 1 shows an overview of our approach: Given a document, we extract its
text and corresponding word geometries using off the shelf OCR solutions. The
document is converted into a purely textual representation, using a step we
refer to as _verbalization_. We propose different verbalization strategies to
add geometric and layout information to the textual document representation
(see Section 3.1). To study the robustness of the verbalization with respect
to inaccuracies of OCR geometries we degrade the OCR before verbalization by
either applying noise to word positions or emulating layout analysis errors.
The verbalized document is then inserted into a prompt template together with
task-specific directives, e.g. questions to be answered (Section 3.3). The
prepared prompt is then fed into an LLM and the response is parsed from the
output.

Figure 1: Overview of our approach: Document OCR is converted into a text
representation using different verbalization strategies (blue). Before
verbalization, we optionally degrade the OCR by applying noise to the spatial
position of OCR geometries (red). The resulting document text representation
is then inserted into a task specific prompt (yellow) and fed into a LLM
(green). Finally, the answers are extracted from the LLM output.

###  3.1 Verbalizers

We refer to verbalizers as strategies that create a textual document
representation from an ordered collection of bounding boxes and the text
associated with these boxes. This representation can serve as input to a text-
based LLM. Further, each verbalizer offers a textual description of its output
format to guide the LLM in interpreting the verbalizer’s outputs. We outline
multiple different verbalization strategies in the following. For each, we
include the verbalization of an example word box with the text "TAX INVOICE"
and coordinates $(x_{left},y_{top},x_{right},y_{bottom})=(100,50,321,100)$ and
center point $(x,y)=(211,75)$:

  1. 1.

PlainText Serves as a baseline by only adopting the text $t$ without extra
layout information. The text lines retrieved from the OCR are concatenated
using newlines to form the document representation. When no line candidates
are available, we concatenate words with spaces.

  2. 2.

BoundingBox Uses both the bounding box coordinates and the text. The box
geometries are encoded together with the text of each box using a custom
format. Coordinates are rounded to whole numbers and are encoded as "left",
"top", "right" and "bottom". Example:
left:$100$ top:$50$ right:$321$ bottom:$100$ text:’TAX INVOICE’

  3. 3.

BoundingBoxMarkup Formats the bounding box coordinates in a XML style markup
format followed by the text. Coordinates are rounded to whole numbers and are
encoded as "left", "top", "right" and "bottom". Example: <box left=$100$
top=$50$ right=$321$ bottom=$100$/>TAX INVOICE

  4. 4.

CenterPoint Formats both the bounding box center point coordinates in a XML
style markup format followed by the text. Coordinates are rounded to whole
numbers. Example:
<box x=$211$ y=$75$/>TAX INVOICE

  5. 5.

SpatialFormat Uses the geometries to restore the original document layout via
insertion of spaces and newlines. To this end, the characters are placed on a
grid such that their spatial location is similar to that on the document.
Figure 2 shows a example output of the SpatialFormat verbalizer. At most 4
consecutive newlines are inserted.

  6. 6.

SpatialFormatY Similar to SpatialFormat, but it only encodes spatial
information on the vertical dimension, i.e. only newlines are inserted and no
spaces are used for horizontal alignment. At most 4 consecutive newlines are
inserted.

  7. 7.

PlainHTML Serves as a control run for the WebSRC dataset, where a structured
HTML representation of the document is available. Example:
…
TAX INVOICE
…

When verbalizing with SpatialFormat and SpatialFormatY, each page is
verbalized individually. The resulting page verbalizations are then
concatenated with an empty newline.

Figure 2: Verbalization strategies on a random sample from the SROIE datset:
original (left), SpatialFormat (middle) and PlainText (right).

###  3.2 Noise Models

The OCR geometries generated by common OCR systems are subject to fluctuations
and are rarely perfectly aligned with each other. To study the robustness of
the verbalization strategies with respect to inaccuracies in the order and
spatial relationship of layout elements, we optionally apply noise models to
the OCR output before feeding it into the verbalizers. Each noising model
takes an ordered list of bounding boxes as input, where the initial order
corresponds to the reading order of the underlying OCR engine.

  1. 1.

NONE: Identity function. The coordinates and text of the OCR are not modified
(serves as a control run).

  2. 2.

TRANSLATE: Degenerates each bounding box $b_{i}$ according to the formula
$(x_{0},y_{0},x_{2},y_{2})\rightarrow(x_{0}{+}\Delta_{i}^{x},y_{0}{+}\Delta_{i}^{y},x_{2}{+}\Delta_{i}^{x},y_{2}{+}\Delta_{i}^{y})$,
where $\Delta_{i}^{x},\Delta_{i}^{y}\in[-20,20]$ are uniformly sampled random
numbers per box. Note that 20px is approximately the average character width
in our data, such that two boxes can move up to 40px (or two letters) relative
to each other.

  3. 3.

SHUFFLE: Shuffles the list of bounding boxes randomly.

  4. 4.

NEAREST_NEIGHBOR: Reorders a list of bounding boxes by selecting for each
bounding box a successor box which is closer than min_char_height and
min_char_width pixels. When there are no or multiple candidates, the successor
box is selected under consideration of the original order of the boxes. The
procedure emulates the _natural_ reading order mode of Microsoft OCR555This
behaviour of _natural_ reading order mode of Microsoft OCR has been shown
empirically through our experiments. and tends to read tables column wise
instead of row wise, as the spacing between consecutive rows is usually
smaller than between consecutive columns.

###  3.3 Prompts

To prompt the LLM, we insert the verbalized document into a task-specific
prompt template. As this prompt template influences quality just like the
verbalization (order, wording and phrasing appear to matter), we aim to
rigorously separate the effects of prompting from the effects of
verbalization. To determine a suitable prompt structure, we subdivide each
prompt into common building blocks and determine an optimal composition.
Following known guidelines for prompt creation [27], we generate 10 different
prompt structures and evaluate them on a small subset of our data. These
structures differ in their ordering of the individual building blocks and are
evaluated using a QA task on the SROIE Challenge dataset (see Section 4.1).

Our prompts are divided into four components: DOCUMENT corresponds to the
verbalized document representation. TASK encodes the task to solve. FORMAT
describes the format used for verbalization. Finally, OUTPUT describes the
expected output format using an example. We identified two patterns A, B to
work best: DOCUMENT TASK OUTPUT (pattern A) and DOCUMENT TASK FORMAT OUTPUT
(pattern B). Based on these two structures, we create prompt templates for the
tasks KIE, QA and NLI. For efficency reasons, we group multiple questions
(QA), statements (NLI) or keys which are to be retrieved (KIE) into a single
prompt. For QA and NLI samples that contain multiple questions to be answered,
we enumerate those starting with 0. Figure 3 shows an example for QA prompt B
with multiple questions.666Please refer to Appendix 0.A for a comprehensive
overview of the prompt templates.

Figure 3: Example for QA prompt B with three questions. Pattern B structure
is: DOCUMENT (black), TASK (blue), FORMAT (orange) and OUTPUT (green).

###  3.4 Answer Extraction

Due to the probabilistic nature of LLMs as text generators, their outputs are
not guaranteed to conform with the requested format. To ensure good readout of
the answers, we process the output as follows:  (1) We request a single JSON
object which assigns the answers to the respective enumeration numbers (QA,
NLI) or keys (KIE).  (2) Given a single valid response object we parse the
answers for the questions.  (3) Given multiple valid response objects we
choose the object with the most answers for the questions asked.  (4) We use
the enumeration number (QA, NLI) or the key (KIE) to extract a specific answer
from the selected object.  (5) If no valid JSON object is returned, we do not
generate any answer.  See Figure 3 for an example of the output format
specification.

While the generation of JSON works reliably in most cases, LLMs will
occasionally generate output that does not parse to valid JSON objects. Edge
cases that we observed during development involve JSON objects which contain
the correct value but a hallucinated key, e.g. price_of_green_tea instead of
answer. Another common mistakes are nested objects, e.g. {"price":
{"green_tea": ...} }. In these cases no answer is extracted.

##  4 Experiments

In the following experiments, we investigate whether suitable verbalization
strategies can support LLMs with better layout reasoning and provide exemplary
comparisons of open-source and commercial solutions. In most experiments, we
measure the awareness of the LLM towards layout aspects via accuracy on
document understanding tasks (which include research benchmarks and industry
datasets, see Section 4.1). To investigate layout awareness in depth, we also
take a qualitative look at a subset of manually annotated challenge cases (see
Section 4.3.4).

###  4.1 Datasets

DUE Benchmark We evaluate our approach using the DUE benchmark[1],
specifically on the datasets _DocVQA_ , _InfographicsVQA_ , _TabFact_ and
_WikiTableQuestions_ with the tasks VQA (DocVQA & InfographicsVQA), TableNLI
(TabFact) and TableQA (WikiTableQuestions). We did not analyze the other
datasets DeepForm, Kleister Charity and PWC, which are also part of the due
benchmark, as these documents have a very high number of pages777A limitation
when working with long documents is the context length of LLMs. While
solutions to this exist, such verbose documents are not part of our scope..

WebSRC WebSRC[28] is a collection of 360K question-answer pairs, which are
collected from 60 different websites spanning 11 different domains. Besides
the QA pairs the dataset also consists of web page segments, where each
consists of a simplified version of the source HTML, a screenshot and a JSON
file which contains additional spatial and layout information. Due to
difficulties in retrieving text level bounding boxes from the JSON and HTML
data, we manually perform OCR on the screenshots and use this data for further
evaluation.888This OCR data has been contributed to the authors of WebSRC and
is also made publicly available at https://github.com/46692/WebSRC˙OCR Only
this dataset uses the PlainHTML verbalizer with the given HTML.

SROIE and SROIE Challenge SROIE [29] is a collection of 973 scanned receipts
and the corresponding OCR results.999We use the revised version of the dataset
from https://www.kaggle.com/datasets/urbikn/sroie-datasetv2 The task of the
dataset is KIE with 4 keys to be extracted for each sample: company, date,
address and total.

The original SROIE asks for the same 4 keys to be extracted for each sample.
We argue that these keys in particular require no comprehensive understanding
of the document’s layout. For example, the company name is almost always the
first thing written on the receipt, where date and address follow shortly
after. To investigate LLM’s understanding of layout more closely, we created a
challenge set that queries the value of a specific table cell (_"How many of
the item ’Green Tea’ were purchased?"_) or directly reference the document’s
layout (_"Which entity is written above the card expiry date?"_). To do so, we
manually annotated 101 samples from the train split of the SROIE dataset to
create a challenging QA dataset101010Made publicly available at
https://github.com/46692/SROIEChallenge. We categorize the questions into
_quantity_ , _currency_ and _string_ , where the latter refers to any other
question that corresponds to neither of the first two types.

Proprietary KIE Datasets We further evaluate KIE performance on two
proprietary KIE datasets from Insiders Technologies, which both contain
particularly diverse and challenging examples from real world business
correspondence: ITForms is a collection of 100 multipage form documents in
German language, with 9 keys each. It includes, among others, forms for
applying for insurance benefits, registering vehicles and bank forms, e.g.
opening a depot. ITInvoices is a collection of 104 invoice documents in German
language, which are predominantly single page and contain 21 keys each. It
includes both business invoices as well as scanned receipts.

###  4.2 Setup

LLMs We evaluate our approach with two LLMs: ChatGPT and Solar[13]. For the
evaluation of ChatGPT we use gpt-3.5-turbo-1106111111The model was used in the
period of November 2023 to February 2024. in JSON mode[30], with a temperature
of $0$, and enter each prompt in the role of user. We further evaluate the 8
bit quantized version121212https://huggingface.co/upstage/SOLAR-0-70b-8bit of
the recent open-source LLM Solar 70b on SROIE and SROIE Challenge.131313The
evaluation of other datasets had to be omitted due to time constraints. For
Solar each prompt is also entered in the role of user.141414As stated on the
Hugging Face model card, Solar expects the role being given in the prompt.
Therefore we used the following wrapper ### User:PROMPT\n\n\n### Assistant:
where PROMPT is replaced with the prompt prepared by our pipeline and \n
symbolizes an empty line. Unless stated otherwise, experiments use the ChatGPT
model and prompt template A, i.e. the template without verbalizer format
description.

OCR Each dataset in the DUE benchmark comes with a selection of pre-applied
OCR engines, where we use microsoft_cv and tesseract as a fallback in case the
former is not available, which is only the case for TabFact. The OCR results
in these datasets contain information about the page and line index, which is
used to join all word bounding boxes on the same page with the same line index
together. Microsoft Computer Vision OCR is used for WebSRC, ITForms and
ITInvoices. We contribute the OCR for WebSRC train and test splits. For SROIE
and SROIE Challenge we use the OCR results delivered with the dataset.

Metrics For evaluation of the DUE datasets we use the official evaluation
framework151515https://github.com/due-benchmark/evaluator with the metrics
given in [1]: ANLS for DocVQA and InfographicsVQA and accuracy for TabFact and
WikiTableQuestions. WebSRC is evaluated according to the procedure in the
GitHub repository161616https://github.com/X-LANCE/WebSRC-Baseline and the
scores are given as EM and F1. For SROIE, SROIE Challenge, ITForms and
ITInvoices we create a type aware accuracy measure: Each response is assigned
one of four types based on the expected response, which describe how it is
compared to the ground truth (the procedures described are applied to both the
GT and the response extracted from the LLM output): For string values a case-
insensitive comparison is made. date values are parsed via the Python
dateparser library171717https://github.com/scrapinghub/dateparser and then
compared for equality. currency values are sanitized via a
RegEx181818\d+(?:(\\.|,)\d1,2)?, replacement of commas with dots and then
compared for equality. quantity values are sanitized via a RegEx191919(?:[
a-zA-Z]*)(\d+)(?:[ a-zA-Z]*) and then compared for equality. The proposed
accuracy measure defaults to EM for string and also for currency and quantity
after units are neglected, i.e. no rounding is performed for the latter two.
In case a value cannot be parsed to its specified type, an empty answer is
returned.

###  4.3 Results

See Section 0.B in the appendix for a comparison of the token overhead added
by each verbalization strategy. See Section 0.C in the appendix for an
analysis of the effects that the verbalizer format description has on the
different prompt templates $A$ and $B$.

####  4.3.1 Dataset Results

We report the results on the various datasets with the metrics laid out in
section 4.2: type aware accuracy for SROIE, SROIE Challenge, ITForms,
ITInvoices; ANLS for DocVQA, InfographicsVQA; accuracy for TabFact,
WikiTableQuestions; F1 and EM for WebSRC. The results in tables 1 and 2 show,
the our approach can compete with state-of-the-art models. Specifically, the
results of the DUE benchmark in table 1 demonstrate that the introduction of
layout information to the prompt proves beneficial. Our approach achieves
state-of-the-art performance on InfographicsVQA and
WikiTableQuestions.202020State as of 10th February 2024 according to
https://duebenchmark.com Throughout the benchmark, SpatialFormat proves to be
the best verbalization strategy on average. With a peak gain of of 15% (from
47.7% to 54.9%) on InfoVQA.

Table 2 shows that we achieve competitive results for some of the other
datasets. Specifically, our approach achieves the 3rd best F1 score on WebSRC
with the SpatialFormat verbalization.212121State as of 10th February 2024
according to [31] The PlainHTML baselines further shows promising results for
HTML formatted document representations, achieving best performance out of all
verbalizations. However, this verbalization strategy is not viable for real
world documents, as these would have to exist as HTML documents in the first
place or would introduce a separate layout processing model into the pipeline,
eliminating the need for our approach. Results on SROIE show that StructTexT
significantly outperforms our approach, demonstrating the superiority of
multi-modal models on the dataset. Comparison with the other datasets is
difficult: For our custom SROIE Challenge subset no comparisons exist of
course. While ITForms and ITInvoices are proprietary datasets that do not
allow direct comparison with other approaches, they give us an insight into
how the models operate on real world business documents. These data sets are
characterized by the fact that not all keys have to generate a value.
Information is often missing on real documents and not every key can be
assigned to a value. The model must therefore have sufficient ability to
reject a value, i.e. it should only output a value if it can be found on the
document. Our results show that the LLM-based approaches perform significantly
worse on these datasets. We observed that in most cases the LLMs produce
outputs and rarely provide an empty response, which lowers their overall score
in the evaluation. We believe that this problem can be reduced by clearer
instructions in the prompt.

A trend that can again be observed throughout the datasets is the good
performance of SpatialFormat and SpatialFormatY among the verbalization
strategies.

Table 1: Comparison with other models published on the DUE-Benchmark. Underlines denote the best verbalization strategy in the dataset. It shows that our approach achieves competitive results and even state-of-the-art results on the two datasets InfographicsVQA and WikiTableQuestions, with an improvement of 15% compared to the baseline for the former. On WebSRC, we rank third in terms of F1 score. Further, it is shown that SpatialFormat is the best verbalization strategy among the ones tested. *LATIN Prompt templates appear to be optimized for the specific dataset and partially include excerpts of examples. Model | Modality | Question Answering |  | Table QA/NLI | Avg.
---|---|---|---|---|---
DocVQA | InfoVQA |  | WTQ | TabFact
$\text{BERT}_{\textsc{LARGE}}$ [22] | T | 67.5 | - |  | - | - | -
Donut [9] | V | 72.1 | - |  | - | - | -
$\text{T5}_{\textsc{LARGE}}$+2D+U [32] | T+L | 81.0 | 46.1 |  | 43.3 | 78.6 | 62.3
$\text{LayoutLMv2}_{\textsc{LARGE}}$ \+ QG [20] | T+L+V | 86.7 | - |  | - | - | -
$\text{LayoutLMv3}_{\textsc{LARGE}}$ [21] | T+L+V | 83.4 | 45.1 |  | 45.7 | 78.1 | 63.1
UDOP [6] | T+L+V | 84.7 | 47.4 |  | 47.2 | 78.9 | 64.6
LATIN-Prompt (Claude) [26] | T+L | 82.6 |  _54.5_ * |  | - | - | -
Ours | PlainText | T | 76.3 | 47.7 |  | 45.1 | 68.4 | 59.4
Ours | SpatialFormat | T+L | 79.8 | 54.9 |  | 47.7 | 70.1 | 63.1
Ours | SpatialFormatY | T+L | 76.3 | 49.6 |  | 45.5 | 70.3 | 60.4
Ours | BoundingBox | T+L | 74.8 | 46.4 |  | 35.0 | 68.5 | 56.2
Ours | BoundingBoxMarkup | T+L | 74.6 | 45.8 |  | 36.2 | 68.6 | 56.3
Ours | CenterPoint | T+L | 75.1 | 47.4 |  | 38.2 | 67.8 | 57.1
Table 2: Evaluation results for SROIE, ITForms, ITInvoices, WebSRC and
SROIEChallenge. Underlines denote the best verbalization strategy for the
dataset. WebSRC results of other models are taken from the official
leaderboard and show that the performance our approach is close to that of the
third-placed model. Proprietary KIE Model refers to an internal model of
Insiders Technologies, which is a multi-modal LLM free approach.: ITForms and
ITInvoices contain samples for which not all keys have a value on the
documents. While this works to some extent, it is not properly supported using
our current prompt. *For WebSRC left score is EM and right score is F1.

Model | Modality | KIE |  | Question Answering
---|---|---|---|---
SROIE | ITForms | ITInvoices |  | WebSRC* | SROIEChallenge
SageGPT-small-v0.2 [31] | ? | - | - | - |  | 89.1 / 92.2 | -
DocPrompt (ErnieLayout-Large) [33] | T+L+V | - | - | - |  | 77.4 / 85.0 | -
TIE (MarkupLM-Large) [34] | T+L | - | - | - |  | 76.3 / 80.5 | -
StructTexT [35] | T+L+V | 98.7 | - | - | - | - |
Proprietary KIE Model | T+L | 91.7 | 86.2 | 90.1 |  | - | -
PlainText | T | 79.9 | 68.4 | 54.5 |  | 72.9 / 80.5 | 81.2
SpatialFormat | T+L | 77.0 | 73.9 | 54.2 |  | 74.2 / 80.7 | 86.1
SpatialFormatY | T+L | 79.0 | 69.0 | 54.6 |  | 72.4 / 80.3 | 81.2
BoundingBox | T+L | 75.4 | 64.2 | 54.1 |  | 68.3 / 76.6 | 72.3
BoundingBoxMarkup | T+L | 74.3 | 65.6 | 53.9 |  | 68.1 / 75.9 | 71.3
CenterPoint | T+L | 73.3 | 65.1 | 51.8 |  | 68.9 / 76.9 | 74.3
PlainHTML | T+L | - | - | - |  | 80.0 / 84.1 | -

####  4.3.2 Comparison of ChatGPT to Solar

We compare the performance of our approach when applied to two different LLMs,
specifically ChatGPT 3.5 and Solar70B8Bit. The results in table 3 show that
open-source LLMs provide a viable alternative to commercial solutions for
document comprehension using our approach: The results of both LLMs are on par
on SROIE, with Solar performing slightly better. On SROIE Challenge, ChatGPT
has a lead of 4.8 pp. on average. Further, it is shown that Solar is
apparently able to make better usage of the layout information delivered by
BoundingBox, BoundingBoxMarkup and CenterPoint verbalizers compared to
ChatGPT.

While it is unclear whether SROIE is part of ChatGPT’s training data, we
checked the training data of Solar222222As given under
https://huggingface.co/upstage/SOLAR-0-70b-8bit and could find no SROIE data.
However, we can assure that neither of both models has seen the questions of
SROIE Challenge during training.

Table 3: Evaluation results for the comparison of Solar and ChatGPT 3.5. Underlines denote the best verbalization strategy for the LLM in the dataset. It shows that open-source LLMs provide a viable alternative to commercial solutions for document comprehension using our approach: Solar performs slightly better than ChatGPT on SROIE, while ChatGPT has an advantage of 4.8 pp. on our custom SROIE Challenge set. Verbalizer | Modality | SROIE |  | SROIE Challenge
---|---|---|---|---
ChatGPT | Solar |  | ChatGPT | Solar
| PlainText | T | 79.9 | 76.6 |  | 81.2 | 72.3
| SpatialFormat | T+L | 77.0 | 76.7 |  | 86.1 | 81.2
| SpatialFormatY | T+L | 79.0 | 77.3 |  | 81.2 | 79.2
| BoundingBox | T+L | 75.4 | 76.2 |  | 72.3 | 66.3
| BoundingBoxMarkup | T+L | 74.3 | 77.7 |  | 71.3 | 66.3
| CenterPoint | T+L | 73.3 | 76.9 |  | 74.3 | 72.3
Avg. |  | 76.5 | 76.9 |  | 77.7 | 72.9

####  4.3.3 Noise Model Analysis

We evaluate the robustness of our verbalization strategies against noise and
layout misinterpretations introduced to the document data. We simulate this by
applying the noise models TRANSLATE, SHUFFLE and NEAREST_NEIGHBOR (see Section
3.2). For each noise model, the average of the scores achieved with each
verbalization strategy across all datasets is determined. 232323In line with
the DUE benchmark, we resort to an arithmetic mean of different metrics. [1]
For WebSRC, where two metrics are reported, we use the F1 score. The results
presented in figure 4 show that SpatialFormat and SpatialFormatY are the least
affected by the noise models. Further, it shows that PlainText is very
susceptible to wrong layout interpretation by the OCR system.

Figure 4: Comparison of the noise models for each verbalizer. Values shown are
scores averaged over all datasets. It shows, that PlainText’s performance
diminishes when the layout is misinterpreted. Further is shown that
SpatialFormat and SpatialFormatY verbalizers are the least prone to noise
introduced to the OCR data. Note that they are not affected by changes to the
bounding box ordering, as they operate only using the bounding box
coordinates.

####  4.3.4 Qualitative Analysis: SROIE Challenge

We explore the impact of document layout on LLMs in-depth on our SROIE
Challenge dataset, which features demanding questions on specific table cells
and the relative position of items. See Section 0.D in the appendix for
examples of these challenge cases. We found the LLM to work surprisingly well,
answering 59 of 101 questions with all verbalizers and 82 with the PlainText
verbalizer. The failures on the remaining 19 samples were related to layout
misinterpretations that follow directly from the limited plain-text
verbalization:  (i) column-wise order of OCR output instead of a row-wise
(ii) delayed table cells, which were placed after all other cells at the end
of a table.  (iii) overly complex samples (e.g. tables spanning over 12 rows
and 6 columns)  (iv) empty cells, which lead the LLM to wrong conclusions
based on the ordering of the bounding boxes, and  (v) neighboring cells merged
to a single bounding box.  Especially combinations of these factors provided
challenging cases. Overall, the challenge cases were more reliably solved by
the SpatialFormat strategy (87 out of 101 correct). This indicates – on a
limited number of manually inspected samples – a better resiliency of the
SpatialFormat verbalization with respect to OCR layout misinterpretations.

##  5 Conclusion

We have investigated techniques for adding layout information to prompts for
instruction-tuned LLMs to enhance document understanding performance. This
approach only requires pre-processing of document text and the prompt without
the need for extra fine-tuning.We achieve higher scores compared to layout-
unaware document representations on 7 out of 9 datasets across different
document tasks , reaching state-of-the-art results on two datasets and often
times yielding results competitive with those of specially trained multi-modal
models. We have shown that our approach works for both commercial as well as
open-source LLMs. A potential threat to validity is that that standard
datasets may have been part of the training data for the proprietary LLM
ChatGPT, while this has been shown not to be the case for Solar. Our results
indicate, however, that the improvements of our approach on the non-public
datasets (ITForms, ITInvoices) and the specifically annotated (SROIE-
Challenge), are in line with the findings on public datasets. The proposed
method is particularly suited for structured documents that make heavy use of
spatial alignments and blanks. In these cases, our approach should be
considered as the best model choice between text-based LLMs and multi-modal
document transformers. It comes with little to no overhead and requires no
extra training while also not adding significantly more tokens to the input.
For future research, an interesting study focus are recent instruction-tuned
LLMs with additional visual input such as GPT-4 [36]. As these models are
associated with higher costs, we have focused on text-only representations in
this work. However, exploring the benefits and trade-offs of including visual
input is worthwhile. We also recon that more work is needed when scaling our
solution to multi-page reasoning problems, especially when the number of pages
becomes larger.

##  Appendix 0.A Prompt Templates

For each task QA, NLI and KIE 2 prompt templates A and B are created. This
section gives an overview of the prompt templates used.

⬇ $$$ <<<CONTENT>>> $$$ From the above document, which is enclosed by "$$$", answer the following questions: <<<QUESTION>>> <<<FORMAT>>> The questions are numbered, e.g. "(0)". Write the answers into a JSON dictionary and use the question numbers as keys and as datatype string. Here is an example of the expected JSON format: { "0": <ANSWER_TO_QUESTION_0>, "1": <ANSWER_TO_QUESTION_1>, … } Listing 1: QA prompt template B. In version A of the template <FORMAT> is removed. |  ⬇ $$$ <<<CONTENT>>> $$$ From the above document, which is enclosed by "$$$", validate the following statements: <<<QUESTION>>> <<<FORMAT>>> The statements are numbered, e.g. "(0)". Write the answers into a JSON dictionary and use the statement numbers as keys and as datatype string. Answer with the string value "1" in case of a true statement and with the string value "0" in case of a false statement. Here is an example of the expected JSON format: { "0": <ANSWER_FOR_STATEMENT_0> "1": <ANSWER_FOR_STATEMENT_1>, … } Listing 2: NLI prompt template B. In version A of the template <FORMAT> is removed.
---|---
⬇ $$$ <<<CONTENT>>> $$$ From the above document, which is enclosed by "$$$", extract the values to the following keys: <<<QUESTION>>> <<<FORMAT>>> Write the answers into a JSON dictionary with one entry for each requested key. Here is an example of the expected JSON format: { KEY0: <VALUE_FOR_KEY_0>, KEY1: <VALUE_FOR_KEY_1>, … } Listing 3: KIE prompt template B. In version A of the template <FORMAT> is removed. |

##  Appendix 0.B Verbalizer Token Count Analysis

We quantify the additional tokens added to the prompt by each verbalization
strategy. To this end, we compare the number of tokens required by the
PlainText verbalizer to the number of tokens required by the other
verbalizers. Tokens are counted with the
tiktoken242424https://github.com/openai/tiktoken BPE tokenizer, which is
provided by OpenAI for use with their models. The analysis plotted in figure 5
shows that SpatialFormat and SpatialFormatY verbalization strategies introduce
the least token overhead compared to the PlainText baseline.
Counterintuitively, it further shows that SpatialFormatY results in less
tokens than the baseline.

Figure 5: Relative token overhead introduced by each verbalization strategy
compared to PlainText baseline. Values are given in percentage of number of
tokens required by PlainText verbalizer and averaged over all documents in all
datasets. It shows that SpatialFormat and SpatialFormatY introduce the least
token overhead compared to the other verbalizers. Furthermore, it shows that
SpatialFormatY requires in even less tokens than the PlainText baseline.

##  Appendix 0.C Format Description Analysis

We evaluate the benefit added by including a verbalizer-specific format
description (see Sections 3.1 and 3.3. Figure 6 shows that the format
description did not lead to a significant performance gain, and that some
verbalization strategies actually declined in performance. Curiously, the
results vary from dataset to dataset, e.g. SpatialFormat benefits on SROIE
from the format description, but declines in performance on all other
datasets. The results are inconclusive and the performance changes never
exceed 2 pp, which we see as insignificant. In addition to that, only one
format description for each verbalizer was evaluated, while a vast amount of
possible format descriptions exist for each.

Figure 6: Comparison of the impact of verbalizer format description in the
prompt. Values shown are scores averaged over the datasets SROIE, SROIE
Challenge, DocVQA, InfographicsVQA, TabFact and WikiTableQuestions. It shows
that the addition of the format description lead to no significant changes,
proving beneficial for some and detrimental for other verbalization
strategies. Further, the results were inconclusive and differed for each
dataset: on some, the format description benefits some verbalizers, on other
datasets it does not.

##  Appendix 0.D SROIE Challenge Examples

Figure 7 shows three difficult cases from the SROIE Challenge dataset, for
three of the categories mentioned in Section 4.3.4: (iii) overly complex
samples, (iv) empty cells, which lead the LLM to wrong conclusions based on
the ordering of the bounding boxes, and (v) neighboring cells merged to a
single bounding box.

(a) X51006619772

(b) X51005806679

(c) X51006332575

Figure 7: Overview of three difficult cases in the SROIE Challenge dataset:
(a) shows a receipt with empty cells, which leads the LLM to wrong conclusions
based on the ordering of the bounding boxes when using the PlainText baseline.
The receipt shown in (b) has a rich layout, with a table consisting of 6
columns and 18 rows. Due to layout misinterpretation of the OCR system, the
texts QTY and U/PRICE were merged into a single bounding box in (c).

## References

  * [1] Łukasz Borchmann et al.  “DUE: End-to-End Document Understanding Benchmark”  In _NeurIPS Datasets and Benchmarks_ , 2021  URL: https://api.semanticscholar.org/CorpusID:244906279
  * [2] Minghao Li et al.  “Tablebank: Table benchmark for image-based table detection and recognition”  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , 2020, pp. 1918–1925
  * [3] Minghao Li et al.  “DocBank: A benchmark dataset for document layout analysis”  In _arXiv preprint arXiv:2006.01038_ , 2020
  * [4] Yiheng Xu et al.  “XFUND: A Benchmark Dataset for Multilingual Visually Rich Form Understanding”  In _Findings of the Association for Computational Linguistics: ACL 2022_ Dublin, Ireland: Association for Computational Linguistics, 2022, pp. 3214–3224  DOI: 10.18653/v1/2022.findings-acl.253
  * [5] Haoyu Cao et al.  “GMN: Generative Multi-modal Network for Practical Document Information Extraction”  In _arXiv preprint arXiv:2207.04713_ , 2022
  * [6] Zineng Tang et al.  “Unifying vision, text, and layout for universal document processing”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 19254–19264
  * [7] Chuwei Luo, Changxu Cheng, Qi Zheng and Cong Yao  “GeoLayoutLM: Geometric Pre-training for Visual Information Extraction”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 7092–7101
  * [8] Dongsheng Wang et al.  “DocLLM: A layout-aware generative language model for multimodal document understanding”  In _arXiv preprint arXiv:2401.00908_ , 2023
  * [9] Geewook Kim et al.  “OCR-Free Document Understanding Transformer”  In _Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVIII_ Tel Aviv, Israel: Springer-Verlag, 2022, pp. 498–517  DOI: 10.1007/978-3-031-19815-1_29
  * [10] Tengchao Lv et al.  “Kosmos-2.5: A multimodal literate model”  In _arXiv preprint arXiv:2309.11419_ , 2023
  * [11] Yi-Hsueh Liu et al.  “Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models”  In _ArXiv_ abs/2304.01852, 2023  URL: https://api.semanticscholar.org/CorpusID:263893278
  * [12] Jason Wei et al.  “Emergent Abilities of Large Language Models”  In _Trans. Mach. Learn. Res._ 2022, 2022  URL: https://openreview.net/forum?id=yzkSU5zdwD
  * [13] Dahyun Kim et al.  “SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling”, 2023  arXiv:2312.15166 [cs.CL]
  * [14] Ashish Vaswani et al.  “Attention Is All You Need”  In _CoRR_ abs/1706.03762, 2017  arXiv: http://arxiv.org/abs/1706.03762
  * [15] OpenAI  “GPT-4 Technical Report”  In _ArXiv_ abs/2303.08774, 2023  URL: https://arxiv.org/abs/2303.08774
  * [16] Hugo Touvron et al.  “Llama 2: Open Foundation and Fine-Tuned Chat Models”  In _ArXiv_ abs/2307.09288, 2023  DOI: 10.48550/arXiv.2307.09288
  * [17] Shengyu Zhang et al.  “Instruction Tuning for Large Language Models: A Survey”, 2023  arXiv:2308.10792 [cs.CL]
  * [18] Hao Feng et al.  “Unidoc: A universal large multimodal model for simultaneous text detection, recognition, spotting and understanding”  In _arXiv preprint arXiv:2308.11592_ , 2023
  * [19] Yiheng Xu et al.  “LayoutLM: Pre-training of Text and Layout for Document Image Understanding”  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD ’20  Virtual Event, CA, USA: Association for Computing Machinery, 2020, pp. 1192–1200  DOI: 10.1145/3394486.3403172
  * [20] Yang Xu et al.  “LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding”  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ Online: Association for Computational Linguistics, 2021, pp. 2579–2591  DOI: 10.18653/v1/2021.acl-long.201
  * [21] Yupan Huang et al.  “LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking”  In _Proceedings of the 30th ACM International Conference on Multimedia_ , MM ’22  <conf-loc>, <city>Lisboa</city>, <country>Portugal</country>, </conf-loc>: Association for Computing Machinery, 2022, pp. 4083–4091  DOI: 10.1145/3503161.3548112
  * [22] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova  “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ Minneapolis, Minnesota: Association for Computational Linguistics, 2019, pp. 4171–4186  DOI: 10.18653/v1/N19-1423
  * [23] Srikar Appalaraju et al.  “DocFormer: End-to-End Transformer for Document Understanding”  In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021, pp. 973–983  URL: https://api.semanticscholar.org/CorpusID:235592814
  * [24] Zineng Tang et al.  “Unifying Vision, Text, and Layout for Universal Document Processing”  In _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022, pp. 19254–19264  URL: https://api.semanticscholar.org/CorpusID:254275326
  * [25] Brian Davis et al.  “End-to-End Document Recognition and Understanding with Dessurt”  In _Computer Vision – ECCV 2022 Workshops: Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part IV_ Tel Aviv, Israel: Springer-Verlag, 2023, pp. 280–296  DOI: 10.1007/978-3-031-25069-9_19
  * [26] Wenjin Wang, Yunhao Li, Yixin Ou and Yin Zhang  “Layout and Task Aware Instruction Prompt for Zero-shot Document Image Question Answering”, 2023  arXiv:2306.00526 [cs.CL]
  * [27] “OpenAI Docs Prompt Engineering”, 2024  URL: https://platform.openai.com/docs/guides/prompt-engineering/six-strategies-for-getting-better-results
  * [28] Xingyu Chen et al.  “WebSRC: A Dataset for Web-Based Structural Reading Comprehension”  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ OnlinePunta Cana, Dominican Republic: Association for Computational Linguistics, 2021, pp. 4173–4185  URL: https://aclanthology.org/2021.emnlp-main.343
  * [29] Zheng Huang et al.  “ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction”  In _2019 International Conference on Document Analysis and Recognition (ICDAR)_ , 2019, pp. 1516–1520  DOI: 10.1109/ICDAR.2019.00244
  * [30] “OpenAI Docs JSON Mode”, 2024  URL: https://platform.openai.com/docs/guides/text-generation/json-mode
  * [31] In _WebSRC - A Dataset For Web-Based Structual Reading Comprehension_ URL: https://x-lance.github.io/WebSRC/
  * [32] Rafał Powalski et al.  “Going full-tilt boogie on document understanding with text-image-layout transformer”  In _Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5–10, 2021, Proceedings, Part II 16_ , 2021, pp. 732–747  Springer
  * [33] Sijin Wu, Dan Zhang, Teng Hu and Shikun Feng  “DocPrompt: Large-scale continue pretrain for zero-shot and few-shot document question answering”  In _arXiv preprint arXiv:2308.10959_ , 2023
  * [34] Junlong Li, Yiheng Xu, Lei Cui and Furu Wei  “Markuplm: Pre-training of text and markup language for visually-rich document understanding”  In _arXiv preprint arXiv:2110.08518_ , 2021
  * [35] Yulin Li et al.  “Structext: Structured text understanding with multi-modal transformers”  In _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021, pp. 1912–1920
  * [36] In _OpenAI - ChatGPT can now see, hear, spealk_ URL: https://openai.com/blog/chatgpt-can-now-see-hear-and-speak


11institutetext: Heinz Nixdorf Chair for Distributed Information Systems,
Friedrich Schiller University Jena, Germany 22institutetext: Competence Center
for Digital Research, Michael Stifel Center Jena, Friedrich Schiller
University Jena, Germany 33institutetext: Institute of Data Science, German
Aerospace Center (DLR), Jena, Germany 33email:
<EMAIL_ADDRESS>
33email<EMAIL_ADDRESS>
# Need for Design Patterns: Interoperability Issues and Modelling Challenges
for Observational Data

Trupti Padiya 1133 0000-0003-4514-4473 Frank Löffler 1122 0000-0001-6643-6323
Friederike Klan 2233 0000-0002-1856-7334

###### Abstract

Interoperability issues concerning observational data have gained attention in
recent times. Automated data integration is important when it comes to the
scientific analysis of observational data from different sources. However, it
is hampered by various data interoperability issues. We focus exclusively on
semantic interoperability issues for observational characteristics.

We propose a use-case driven approach to identify general classes of
interoperability issues. In this paper, this is exemplarily done for the use-
case of citizen science fireball observations. We derive key concepts for the
identified interoperability issues that are generalizable to observational
data in other fields of science. These key concepts contain several modeling
challenges and we broadly describe each modeling challenge associated with its
interoperability issue. We believe, addressing these challenges with a set of
ontology design patterns will be an effective means for unified semantic
modeling, paving the way for a unified approach for resolving interoperability
issues in observational data. We demonstrate this with one design pattern,
highlighting the importance and need for ontology design patterns for
observational data, and leave the remaining patterns to future work. Our paper
thus describes interoperability issues along with modeling challenges as a
starting point for developing a set of extensible and reusable design
patterns.

###### Keywords:

Citizen Science Fireball observations Observational data Ontology design
patterns Semantic interoperability.

111This work was carried out when the first author was affiliated with
Friedrich Schiller University Jena, Germany, and Institute of Data Science,
DLR Jena, Germany. At the time of submission on arxiv, the first author is
affiliated with TIB Leibniz Information Centre for Science and Technology,
Hanover, Germany.

##  1 Introduction

Interoperability issues with regard to observational data have recently gained
attention [1]. Observational data can have different formats, structures, and
diversity in semantic representation of observational characteristics,
resulting in interoperability issues at all these levels. In this paper, we
focus exclusively on issues regarding data interoperability in the context of
semantic representations of observational characteristics. Automated data
integration is vital for observational data. However, this is hard to achieve.
One of the major reasons is that use-cases and data interoperability are not
taken into consideration during the development of observational data models.
As a result, a variety of observational data models came gradually into
existence and this diversity in data modeling approaches is hampering
automated data integration [2].

In order to tackle this fundamental problem, we propose a bottom-up approach
which is a use-case driven approach. Based on real-world requirements and
discussions with domain experts, we identify semantic interoperability issues
exemplarily for the use-case of citizen science fireball observations. We
derive key concepts for the identified interoperability issues that are
generalizable to observational data in other fields of science. The ability to
derive key concepts for the identified issues being applicable to general
observational data demonstrates the potential of our approach. These derived
key concepts contain several modeling challenges which we broadly describe. We
believe, these challenges can be resolved in a unified and reusable manner by
developing corresponding ontology design patterns. Moreover, existing design
patterns can foster reuse of best practice solutions for modeling, resulting
in more robust and unified modeling solutions. Here, we limit ourselves to a
demonstration of one such design pattern. We call for the community to follow
this approach and contribute ontology design patterns that can address
interoperability issues for observational data, especially for citizen
science.

##  2 Use-Case: Citizen Science Fireball Observations

Citizen science fireball observations consist of vital information about
fireball-observations and are used as a basis for scientific study and
analysis of fireballs and fireball events, e.g. determining the trajectory and
mass of the observed objects. Typically, a fireball observation can be
described as follows: A fireball event occurs in the sky, and an observer
witnesses the event and records various relevant properties, e.g., duration of
event and color of the fireball. These observational data are then reported
through citizen science reporting platforms, usually via web-app or web-form.
There are several citizen science apps to report fireball observations like
Skywatcher from Nachtlicht-BüHNE [3], Fireballs in the sky [4] and many
others. Web-forms are also provided from different fireball networks across
the world, like AMS/IMO [5], DFN [6], Allsky7 [7], ESA [8], and many others.
Our use-case is based on real-world datasets from AMS/IMO, Nachtlicht-BüHNE,
data from camera images, and data derived by scientists. We also consider
concrete data needs from scientists for identifying and modeling
interoperability issues inherent to fireball observational data.

Table 1: Example data from different citizen science fireball observational
datasets pertaining to same semantics

Observational properties | Dataset1 | Dataset2 | Dataset3
---|---|---|---
location | Jena, Germany | 07745, Jena | 50.9271°N, 11.5892°E
level of experience | intermediate | 7 | good
viewing-direction | 57° | north | east-northeast
color | red | yellow, blue, white | #FF0000,#E6E6FA
fireball-train-duration | 0.033 minute | unknown | 1 second
moving-direction | 112.5° | east to southeast | top-left to bottom-right

Scientific analysis of these data calls for data integration from multiple
fireball observation datasets. However, there are several interoperability
challenges. e.g., observable properties of the fireball are represented
differently in different datasets, both at schema level and instance level,
despite having the same semantics. Table 1 depicts data at instance level. It
is important to map these semantically similar data in order to get an
accurately integrated dataset.

##  3 Semantic Interoperability Issues and Modelling Challenges

We describe some of the semantic interoperability issues along with modeling
challenges based on our use-case. We derive key concepts that represent the
generalizable interoperability issue, which eminently apply to observational
data in other fields of science. Derived key concepts are written as a header
for every challenge. Furthermore, we demonstrate one design model to resolve
one of the listed challenges.

Expertise-Level:  Table 1 shows various ways of representing the level of
experience in our use-case. It is represented using an interval scale with
varying ranges like $[1-5]$ or $[1-3]$ or an ordinal scale referring to non-
mathematical ideas like (beginner, intermediate, expert) or (poor, fair,
average, good, excellent). There is a possibility of mapping interval scale to
ordinal scales whereas doing the reverse is problematic. Moreover, it is
important to map varying ranges that belong to a similar scale of measure.
Semantically mapping different representations of expertise-level is a general
interoperability issue that needs to be addressed.

Location: Table 1 also depicts some of the ways a location of observer is
represented in our use-case. In itself, it is a general interoperability
issue, as location is represented differently by location name, a postal code,
nearby location, or geo-coordinates as latitude and longitude. Furthermore,
latitude/longitude can also be represented as ecliptic, galactic, and
supergalactic latitude/longitude. Sometimes, latitude and longitude are
represented with a notion of declination and right ascension respectively.
Additionally, different units and formats exist to represent the coordinates.

Visual characteristics:  In our use-case, the property color is represented in
different ways as shown in Table 1. Color is represented by dictionary color
names or color codes like RGB, HSV, and others. It is possible to map
dictionary color names to color codes. However, not all color codes have
names. Moreover, there are color names that are difficult to represent with a
color code, e.g., light blue-green. In addition, a fireball or other feature
might show temporal variation of color. E.g., a sequence of colors appear with
a timeline and it is important to maintain this information while integrating
data. Same challenges are applicable to a large fraction of observable
objects, e.g., sky, animals, plants. Here different semantic representations
of visual characteristics is a generalizable interoperability issue that
covers not only color but also shape, form, texture, etc.

Relative and cardinal directions: The moving direction of a fireball can have
multiple representations as shown in Table 1. It may be represented using
angular degrees, or body relative directions like left, right, up (top), down
(bottom), forward, backward, and their permutations like left to right, or
top-left to top-right, and so on with varying granularity information. Another
representation is using cardinal directions like east to west, or northeast to
northwest, or north-northeast to north-northwest, and so on with different
granularity information. The generalizable interoperability issue is mapping
angles, relative directions, and cardinal directions with additional
granularity information.

Angles and directions: In our use-case, viewing-direction refers to the
direction that an observer is facing while observing an event. It is
represented either using a cardinal direction or a viewing angle. Moreover,
the granularity level of the cardinal direction also varies in different
datasets. E.g., some datasets might have 8 compass sectors: east (E), west(W),
north(N), south(S), north-east(NE), north-west(NW), south-east(SE), south-
west(SW). Whereas some might have 16 compass sectors: north northeast (NNE),
east northeast(ENE), and so on in addition to 8 compass sectors. One such
example of variation in granularity and representation in the actual dataset
is shown in Table 1.

A solution to the general interoperability issue of mapping different concepts
like angles and directions with appropriate granularity information is
depicted in Fig. 1. We use SWRL rules to infer angles from directions and
vice-versa. An example of SWRL rule for NNE is: $AnglesDirections(?v)\wedge
hasValue(?v,?x)\wedge swrlb:greaterThan(?x,12)\wedge
swrlb:lessThan(?x,33)\implies NNE(?v)$.
Similarly, for every direction, SWRL rules are created using the ranges as
given in Fig. 1. This modeling solution can be reused, e.g., in wind
observations, and/or can be extended for other domains based on requirements.

Figure 1: Graphical representation of modeling: Angles and Directions

The modeling solution in Fig. 1 is inspired by the view-inheritance design
pattern [9], which demonstrates that extensible design patterns foster re-
usability, provide clear modeling structure, decrease complexity, increase
reliability, and become robust with progression, which gradually leads towards
unified modeling solutions. We believe, addressing modeling challenges in this
manner will be an effective way to unified semantic modeling of
interoperability aspects and thus enabling automatic data integration. It is
important to note that modeling observational data in a unified way is a
prerequisite for resolving interoperability issues in a unified manner using
design patterns. Moreover, there is a strong requisite of design patterns for
observational data and their characteristics.

##  4 Conclusions

In this paper, we describe some of the important semantic interoperability
challenges for observational data. We argue that addressing these challenges
in terms of a set of extensible and reusable ontology design patterns will be
an effective way to unified semantic modeling of interoperability aspects
leading towards automatic semantic data integration. We demonstrated a
potential way to address semantic interoperability issues in a unified manner.
It will accommodate for other scientific domains that leverage observational
data to reuse these design patterns and extend them based on their needs. It
is a call for the community to address modeling challenges by contributing
ontology design patterns.

## References

  * [1] Magagna, B., et al.: The i-adopt interoperability framework: a proposal for fairer observable property descriptions . https://doi.org/10.5194/egusphere-egu21-13155, https://doi.org/10.5194/egusphere-egu21-13155,
  * [2] Schindler, S., Klan, F.: Automated data integration-how to enable a not (yet) redeemed promise? (2019), https://www.w3.org/Data/events/data-ws-2019/assets/position/Schindler-Klan.pdf
  * [3] Skywatcher app, https://nachtlicht-buehne.de/feuerkugeln/
  * [4] Fireballs in the sky app, http://fireballsinthesky.com.au/
  * [5] International meteor organization, https://fireballs.imo.net/members/imo/
  * [6] Desert fireball network, https://dfn.gfo.rocks/
  * [7] Allsky7 fireball network, https://www.allsky7.net/
  * [8] European space agency, https://neo.ssa.esa.int/search-for-fireballs
  * [9] Odp submission: View inheritance, http://ontologydesignpatterns.org/wiki/Submissions:View_Inheritance


# 142 GHz Multipath Propagation Measurements and Path Loss Channel Modeling in
Factory Buildings

Shihao Ju and Theodore S. Rappaport This research is supported by the NYU
WIRELESS Industrial Affiliates Program and National Science Foundation (NSF)
Research Grants: 1909206 and 2037845. NYU WIRELESS, Tandon School of
Engineering, New York University, Brooklyn, NY, 11201
{shao<EMAIL_ADDRESS>
###### Abstract

This paper presents sub-Terahertz (THz) radio propagation measurements at 142
GHz conducted in four factories with various layouts and facilities to explore
sub-THz wireless channels for smart factories in 6G and beyond. Here we study
spatial and temporal channel responses at 82 transmitter-receiver (TX-RX)
locations across four factories in the New York City area and over distances
from 5 m to 85 m in both line-of-sight (LOS) and non-LOS (NLOS) environments.
The measurements were performed with a sliding-correlation-based channel
sounder with 1 GHz RF bandwidth with steerable directional horn antennas with
27 dBi gain and 8° half-power beamwidth at both TX and RX, using both vertical
and horizontal antenna polarizations, yielding over 75,000 directional power
delay profiles. Channel measurements of two RX heights at 1.5 m (high)
emulating handheld devices and at 0.5 m (low) emulating automated guided
vehicles (AGVs) were conducted for automated industrial scenarios with various
clutter densities. Results yield the first path loss models for indoor factory
(InF) environments at 142 GHz and show the low RX height experiences a mean
path loss increase of 10.7 dB and 6.0 dB when compared with the high RX height
at LOS and NLOS locations, respectively. Furthermore, flat and rotatable metal
plates were leveraged as passive reflecting surfaces (PRSs) in channel
enhancement measurements to explore the potential power gain on sub-THz
propagation channels, demonstrating a range from 0.5 to 22 dB improvement with
a mean of 6.5 dB in omnidirectional channel gain as compared to when no PRSs
are present.

###### Index Terms:

Channel Measurement; Channel Modeling; Indoor Factory; Sub-Terahertz; 140 GHz;
Reconfigurable intelligent surface (RIS); 5G; 6G

##  I Introduction

Terahertz (THz) communications show great promise for sixth-generation (6G)
wireless systems due to the vast available bandwidth [1, 2]. Federal
Communications Commission (FCC) released the first report and order, ET Docket
18-21, in 2019 [3], which offered experimental licenses for the 95 GHz to 3
THz range and opened up 21.2 GHz of the spectrum between 116 GHz and 246 GHz
for unlicensed use. Extensive propagation measurements and accurate channel
modeling are paramount to building a high-performance THz wireless system with
communication and sensing functions. Significant efforts on channel
measurements and modeling above 100 GHz have been made over the past few years
[1, 4, 5, 6], but there is virtually no understanding of real-world factory
channels [7]. Most measurement campaigns focused on indoor office and outdoor
urban environments between 100 and 300 GHz, which have shown that, despite the
fewer observed multipath components than mmWave frequencies, the non-line-of-
sight (NLOS) propagation created by specular reflection and diffuse scattering
is remarkably substantial in sub-THz wireless system design and deployment [4,
5, 8].

Advanced industrial Internet of things (IIOT) with increasing automated guided
vehicles (AGVs) and autonomous mobile robots (AMRs) has rapidly transformed
modern factories, creating an unprecedented demand for fast and reliable in-
factory communication networks and centimeter-level positioning and sensing
[9]. Therefore, the ultrawide transmission bandwidth at sub-THz frequencies,
together with the steerable pencil beams generated by large antenna arrays
with a massive number of antenna elements, may satisfy the stringent
requirements for communication and sensing in smart factories.

This work presents a recent 142 GHz radio propagation measurement campaign in
four factory buildings in Brooklyn, NY. Extensive channel measurements were
conducted at 16 transmitter (TX) locations and 60 receiver (RX) locations
across four factories, resulting in 30 line-of-sight (LOS) location pairs, 48
NLOS location pairs, and four outage location pairs over a T-R separation
distance range from 5 m to 85 m. The TX antenna was vertically polarized, and
the RX antenna was systematically switched between vertical (V) and horizontal
(H) polarizations, providing co- and cross-polarized channel characterization.
Directional and omnidirectional path loss models for co- and cross-
polarization antenna configurations are derived in Section III with a
statistical cross-polarization discrimination (XPD) characterization.

The measurement setup mimicked an indoor downlink transmission from an access
point to user equipment with TXs at 2.5 m and RXs at 1.5 m heights above the
floor in the four factories, labeled A, B, C, and D. At Factory C and D,
additional channel measurements were conducted with the RX height of 0.5 m to
investigate the channel properties of near-the-ground mobile receivers (e.g.,
AGVs) commonly used in factories. The measurement results in Section IV show
that the low-height RXs experience larger path loss than the high-height RXs
due to increasing blockage probability and complex scattering environment
caused by surrounding clutter.

The larger path loss at the low RX height motivates us to propose a channel
enhancement scheme using passive reflecting surfaces (PRS). There have been
some early works employing metal surfaces to enhance mmWave and sub-THz
propagation channel by fixing a metal plate at the corner of an L-shaped
corridor [10] or attaching aluminum foil to walls [11] and pillars [12].
Section V demonstrates a new measurement setup using flat metal plates to
improve propagation channels. A 1 m x 1 m flat smooth aluminum plate was
placed near the walls of factories to increase the received power for a
specified RX location by either boosting an existing weak multipath component
(MPC) or creating an additional MPC. The PRS was set next to walls and pillars
and manually rotated in the horizontal plane to the best reflecting
orientation, which mimicked the flexible beamforming operation of future
reconfigurable intelligent surfaces (RIS) at sub-THz frequencies [13]. The
PRS-aided channel measurements were conducted at all twelve locations at
Factory D with both RX heights of 1.5 m and 0.5 m for comparative analysis.

##  II 142 GHz Propagation Measurements in Four Factories

###  II-A Measurement Equipment

The 142 GHz wideband channel sounder adopts a sliding-correlation-based
transceiver architecture [14]. A pseudorandom noise (PN) sequence of length
2047 at 500 MHz was generated at baseband, then upconverted to a center
frequency of 142 GHz, and transmitted through a directional and steerable horn
antenna with 8° half power beamwidth (HPBW) and 27 dBi gain from the TX. The
signal was captured by the RX via an identical steerable horn antenna and
downconverted to baseband signals. The baseband signal was sliding correlated
with a local copy of the transmitted PN sequence at a slower rate [15],
resulting power delay profile (PDP), which was sampled by a high-speed
oscilloscope. TX and RX horn antennas were mechanically steered by two gimbals
in the azimuth and elevation planes. The RX horn antenna was switched between
vertical and horizontal polarization while the TX antenna remained vertically
polarized. Both co- and cross-polarization measurements followed the identical
measurement procedure as described below. The 142 GHz channel sounder had a
null-to-null RF bandwidth of 1 GHz and can measure the directional propagation
path loss up to 152 dB [14].

###  II-B Measurement Environments and Locations

The 142 GHz propagation measurements were conducted in four factory buildings
in Brooklyn, NY. The selected factory buildings constitute a wide range of
factory size, layout, and manufacturing facility. TX locations were picked to
be typical of where WiFi access points are deployed to cover entire factory
floors, and RX locations were selected along common trajectories of human
mobile workers and AGVs in factories.

#### Site A: Factory Building

The building ($\sim$104 m L$\times$ 39 m W $\times$ 25 m H) was partitioned
into manufacturing labs and conference rooms covered by large glass windows
with metal frames. A mezzanine floor was built above the rooms along external
facades with a height of 3.7 m above the ground, leaving a 7 m wide open
corridor at the center of the building. Six TX locations and 17 RX locations
were selected to cover the entire open area in Factory A. Fig. 1 shows the 27
TX-RX location pairs consisting of 10 LOS locations, 15 NLOS locations, and
two outage locations. RX12 and RX16 were in outage. The TX was set to 3 m
above the ground, except that TX4 was set at 3 m high on the mezzanine floor,
having a height of 6.7 m above the ground. The RX was set to 1.5 m above the
ground. The T-R separation distance ranged from 6.5 m to 86.5 m.

Figure 1: TX and RX locations in Factory A. Six TX locations are denoted as
stars, and 17 RX locations are denoted as circles, resulting in 27 TX-RX
location pairs for channel measurements.

#### Site B: Electronics Manufacturing Facility

The single-story structure covered various production phases, such as
manufacturing, testing, and assembly. We selected three production areas
including automated manufacturing, testing and assembly for channel
measurements. Factory B had heavy clutter containing massive metal objects
such as robotics, partitions, machineries, and workstations as shown in Fig.
2. We selected four TX locations and 20 RX locations, covering the three areas
of interest. The 20 TX-RX location pairs consisted of three LOS locations and
17 NLOS locations. The TX height was set to 2.5 m above the ground near the
ceiling, and the RX height was set to 1.5 m above the ground. The T-R
separation distance ranged from 5.6 m to 31.0 m.

Figure 2: Photos taken from Factory B. The TX was set at 2.5 m, and the RX was
set at 1.5 m.

#### Site C: Warehouse Facility ($\sim$72 m L$\times$ 30 m W $\times$ 4 m H)

Site C was a mid-size warehouse. As shown in Fig. 3, large metal multi-layer
shelves with a height of 3 m and a spacing distance of 1.5 m were placed at
two sides of the building, creating a 2 m wide walkway in the center. These
large metal shelves were filled with cardboard boxes containing commodities.
We selected three TX locations and 11 RX locations in Factory C, covering the
stock and shipping areas of the warehouse facility. The 11 TX-RX location
pairs consisted of three LOS locations, six NLOS locations, and two outage
locations. RX1 and RX8 were in outage. The TX was 2.5 m above the ground, and
the RX was 1.5 m above the ground. The T-R separation distance ranged from 7.7
m to 37.6 m. In addition, the low RX height at 0.5 m above the ground was
adopted at three LOS locations to study the effect of antenna height and
ground reflection.

Figure 3: Photos taken from Factory C. The TX was set at 2.5 m, and the RX was
set at 1.5 m and 0.5 m.

#### Site D: Manufacturing Space ($\sim$35 m L$\times$ 25 m W $\times$ 5 m H)

This space was a single large room for lightweight manufacturing and
production, with various machineries such as laser cutters, CNC machines, and
3D printers. Large glass window panels coated with metallic frames covered the
exterior walls on the north, south, and east of this room, whereas thick
drywall constituted the wall on the west, as shown in Fig. 4. Plastic tables
and wooden chairs were placed in a few office regions. Black-filled circles in
Fig. 4 represent concrete pillars which created strong reflections during
channel measurements.

Figure 4: TX and RX locations in Factory Site D. Three TX locations are
denoted as stars, and nine RX locations are denoted as circles, resulting in
12 TX-RX location pairs for channel measurements. The short blue and red bars
denote the PRS positions for high and low RXs, respectively.

We selected three TX locations and nine RX locations in Factory D, covering
the entire manufacturing space. The TX was set to 2.5 m above the ground, and
the RX was set to 1.5 m and 0.5 m above the ground at each RX location for
studying the impact of antenna heights. The 12 TX-RX location pairs consisted
of three LOS locations and nine NLOS locations for high RX. In contrast, only
TX1 and RX3 were in LOS for low RX. The T-R separation distance ranged from
7.0 m to 35.1 m. In addition, the PRS measurements were conducted at twelve
locations at both RX heights.

###  II-C Measurement Procedure

The TX and RX were placed at the designated locations with vertical-to-
vertical (V-V) antenna polarization. The TX and RX antenna pointing angles in
the azimuth and elevation planes receiving the strongest power
($P_{\textup{max}}$) were identified as “best”, which was the boresight
direction for LOS locations or antennas pointing to a strong reflection for
NLOS locations. The TX antenna was rotated from the best pointing angle in
step increments of the antenna half power beamwidth (HPBW) (8°) in the azimuth
plane, resulting in 45($=360\degree/8\degree$) TX pointing angles. With the TX
pointed at the $i^{\textup{th}}$ pointing angle, the RX antenna performed a
360° continuous sweep in the azimuth plane, taking about six seconds, and
recorded the peak received power ($P^{i}_{\textup{peak}}$). We selected the TX
angles based on a power threshold, which was set to the smaller value between
30 dB below the highest peak power from 45 TX angles, and 10 dB above the
average noise floor. Those TX angles in which the peak received power exceeded
the specified threshold were considered “good” angles, where detectable
multipath existed. For each of the “good” TX pointing angles, the RX was
rotated at step increments of 8° in the complete azimuth plane. One
directional PDP was recorded at each RX pointing step. The RX was then up- and
down-tilted by 8°, and the exact stepped azimuthal sweep to acquire signal
energy was repeated [16]. For example, if a TX has ten “good” pointing angles,
there are at most $10\times 45\times 3=1350$ possible directional PDPs
measured at this location. In practice, we measured 1000 PDPs on average at
each particular TX-RX location. This measurement procedure is exhaustive,
taking typically two hours for each TX-RX location, but provides complete
angular statistics for both angle of departure (AOD) and angle of arrival
(AOA). Identical measurements were performed for both V-V and V-H antenna
polarization configurations at each TX-RX location pair.

##  III 142 GHz Path Loss Models for Co- and Cross-Polarization Antennas

We adopt the well-known close-in free space reference distance (CI) path model
for omnidirectional and directional path loss measurement data because the CI
path loss model has been proven to be superior for modeling path loss over
many environments and frequencies [17]. The CI path loss model is
parameterized solely by the path loss exponent (PLE) denoted as $n$.
$\mathrm{PL}^{\textup{CI}}$ represents the path loss in dB scale, which is a
function of distance and frequency [18, 4, 5]:

$\displaystyle\mathrm{PL}(f,d)[\mathrm{dB}]=$
$\displaystyle\textup{FSPL}(f,d_{0})+10n\log_{10}\left(\frac{d}{d_{0}}\right)+\chi_{\sigma}^{\mathrm{CI}}\text{,}$
(1)

for $d\geq d_{0}$, where $d_{0}=1m$. $d$ is the 3-D T-R separation distance
and $d_{0}$ is the reference distance. The free space path loss at a reference
distance of $d_{0}=1$ m at the carrier frequency $f$ is given by
$\mathrm{FSPL}(f,1~{}\textup{m})[\mathrm{dB}]=20\log_{10}\left(4\pi f\times
10^{9}/c\right)=32.4[\mathrm{dB}]+20\log_{10}(f)$.
$\chi^{\mathrm{CI}}_{\sigma}$ denotes shadow fading (SF), commonly modeled as
a normal random variable with zero mean and $\sigma$ standard deviation in dB.
The CI path loss model uses the FSPL at the reference distance as a fixed
intercept and finds a straight line fit to the measured path loss data that
achieves the minimum mean square error [18, 4, 5].

Figure 5: Directional path loss data and best fit CI path loss models
($d_{0}=1$ m) for V-V polarizations for 142 GHz indoor factory channels from
four factories.

Fig. 5 shows the directional path loss scatter plots and best fit CI (1)
models for the 142 GHz LOS and NLOS scenarios for the co-polarized (V-V) and
cross-polarized (V-H) antenna configurations from all four factory sites. Each
scatter point denotes a unique pointing direction pair of TX and RX horn
antennas with 8° HPBW. The LOS direction denotes the unique TX-RX angles when
the TX and RX point to each other. The NLOS direction denotes all the
remaining TX-RX angles in which the receive power exceeds the threshold. The
Best NLOS direction is picked from the NLOS directions for each TX-RX location
pair, which contains the most received power. Note that the directions
adjacent to the LOS direction are excluded from the selection of NLOS Best
directions to avoid the effect of antenna pattern.

For V-V antenna polarization, the LOS PLE is 2.06, close to the free space PLE
of 2. The NLOS PLE is reduced from 5.01 to 3.27 by pointing antennas to the
strongest NLOS direction instead of an arbitrary direction, suggesting that
beam steering is critical for THz communication systems. However, Site A and
Site D produce smaller NLOS and NLOS-Best PLEs than Site B and Site C,
possibly because Site A and Site D are more open-planned with fewer partitions
and less clutter density. For the V-H polarized measurements, the LOS, NLOS-
Best, and NLOS PLEs are 4.27, 5.16, and 6.17, respectively, which are much
greater than the PLEs for V-V antenna polarization due to the polarization
mismatch between antennas and channels. Using the PLEs of co-polarized path
loss data, the CIX model [7] provides a much better fit than the CI model by
reducing the standard deviation from 7.1 dB to 2.9 dB for LOS directions and
from 9.4 dB to 6.4 dB for NLOS-Best directions. The XPD for LOS directions is
27.1 dB, close to the antenna XPD measured in free space of 27.5 dB [19]. The
NLOS-Best directions produce an XPD of 22.2 dB, and the NLOS directions
produce an XPD of 13.3 dB. The NLOS-Best directions show an XPD reduction of 5
dB compared to the LOS direction, indicating that reflection and scattering
have a prominent de-polarization effect.

Figure 6: Omnidirectional CI and CIX path loss model parameters with $d_{0}=1$
m for 142 GHz indoor factory channels for V-V and V-H antenna polarization
orientations in four factories.

Fig. 6 displays the omnidirectional path loss scatter plot and best fit CI and
CIX models at 142 GHz in LOS and NLOS environments for co-polarization (V-V)
and cross-polarization (V-H) antenna configurations from all four factory
sites. Factory sites are denoted as different markers. For V-V polarization,
the LOS PLE is 1.80, which is similar to the value in office environments
(1.75) [4, 20] and better than the outdoor environments (1.91) [21]. The NLOS
PLE is 3.07, which is worse than 2.7 in office environment [4, 20] and 2.9 in
outdoor environment [22, 5], indicating that the factories may be harsh for
signal propagation in NLOS scenarios due to 1) dense clutter such as
workstations and shelves, 2) large space leading to long propagation distances
and a high chance of blockage, 3) smooth surfaces of metallic objects
producing only specular reflections with limited scattering. Previous works on
channel measurements in factories at mmWave frequencies showed similar PLEs
[23, 24, 25].

##  IV High and low RX heights

According to the terminologies from 3GPP TR 38.901 [26], a device is clutter-
elevated if the antenna is higher than the average environmental clutter
height. A device is clutter-embedded if the device antenna is lower than the
average environmental clutter height. We conducted comparative propagation
measurements at 1.5 m and 0.5 m RX heights emulating high (clutter-elevated)
UEs and low (clutter-embedded) UEs at factory site D, where the average
clutter height is 1 m. The TX was fixed at 2.5 m as indoor access points. As
shown in Fig. 4, three TX locations and nine RX locations were selected in
Site D, resulting in 12 TX-RX location pairs. Both high and low UEs were
measured at 12 identical locations. For the high UE measurements, there were
five LOS location pairs and seven NLOS location pairs. There were one LOS
location pair and eleven NLOS location pairs for the low UE measurements.

Figure 7: Location-wise comparison of omnidirectional path loss for high and
low RXs in factory site D.

Fig. 7 compares omnidirectional path loss between the high and low RX heights
at each measurement location at Factory Site D. The omnidirectional path loss
increases at every LOS and NLOS location because the minimum path loss
variation is -1.7 dB (i.e., the high-UE path loss is smaller than the low-UE
path loss). The maximum path loss increases are 20.8 dB and 13.8 dB in the LOS
and NLOS scenarios. The mean path loss increases are 10.7 dB and 6.0 dB in the
LOS and NLOS scenarios. The larger path loss increase in the LOS scenario is
expected due to the potential blockage of the LOS path. For example, the
largest path loss increase was recorded at TX2 and RX6, where a 1 m tall
machine blocked the boresight path between TX2 and RX6. The larger standard
deviation of 6.3 dB in the LOS scenario also suggests the risk of the
diminishing LOS path. The derived NLOS PLEs are 2.50 and 2.88 for the high and
low RXs, respectively, indicating that the low RX often experiences additional
path loss due to the blockage and shadowing effect of clutter in the
proximity.

##  V Channel Enhancement Using Passive Reflecting Surfaces

We conducted regular channel measurements at each location without the PRS by
rotating the TX and RX antennas in the azimuth and elevation planes to record
omnidirectional channel responses. The PRS position was selected based on the
regular measurement results to improve a single-bounce reflection path. Next,
a visual ray tracing was performed to identify the reflecting object. Only the
reflecting objects such as walls, pillars, corners, and large fixed furniture
were considered candidate PRS locations because hard or flexible RISs would be
installed on fixed large surfaces in future indoor scenarios. The metal plate
was placed next to the reflecting object at a height between the TX and RX,
depending on the relative positions. We manually and carefully rotated the
metal plate in the horizontal plane to find the best orientation that provided
the largest channel gain, which emulated the configurability of RISs steering
the beam to the best direction based on channel estimation. With the TX
antenna fixed, the RX antenna was swept over 360° in the azimuth plane in step
increments of the antenna half-power beamwidth (HPBW) of 8°. Identical azimuth
sweeps were performed at three elevation planes, which were boresight, 8°
above and below the boresight. The adjacent azimuth TX pointing directions
with 8° spacing that showed sufficient reflecting power were measured at
several locations.

(a) Directional path loss reduction with PRS aid at high (H) and low (L) RX
heights in factory site D.

(b) Omnidirectional path loss reduction with PRS aid at high (H) and low (L)
RX heights in factory site D.

Figure 8: Directional and omnidirectional path loss comparison between the
natural and the PRS-aided channels measured at high (H) and low (L) RX heights
in factory site D.

The directional path loss reduction in the reflecting direction enhanced by
the PRS is shown in Fig. 8(a) location by location, where, for each TX-RX
location pair, the TX-RX antenna pointing combination that presents the
maximum directional power gain is plotted. Note that the directional path loss
power is calculated by summing all the samples recorded in a directional PDP,
which may contain multiple MPCs. The PRS provides a mean power increase of 25
dB with a maximum of 37.8 dB and a minimum of 6.6 dB. The TX2 and low RX6 show
the maximum gain of 37.8 dB because the distance between the TX2, PRS2-6L, and
RX6 is the shortest among all the measurement locations. In addition, the
reflection from the partition wall where PRS2-6L is located is partially
blocked by the machinery near the RX6 without the aid of a PRS. However, the
PRS emulates the anomalous reflector that does not obey the specular
reflection from the original furniture surface and creates a strong single-
bounce reflection path. The TX2 and low RX8 present the minimum directional
path power gain of 6.6 dB because the signal is transmitted over the largest
T-R separation distance of 38 m measured in site D, where many machines and
tables are located between the TX and RX causing partial blockage. Thus, the
link created by the PRS does not have a clear first-order reflection path,
indicating that the choice of location of PRS2-8L might not be optimal. Fig.
8(b) shows the influence of the PRS on omnidirectional path loss. The PRS
increases the omnidirectional received power by a maximum of 22 dB and a
minimum of 0.5 dB with a mean of 6.5 dB. This increase depends on the PRS and
other existing MPCs. A slight increase indicates that other MPCs dominate the
total received power, which is typical for the LOS scenario. However, a
significant increase often occurs at NLOS and near-outage locations, where the
PRS-enhanced path becomes the strongest path and dominates the channel.

##  VI Conclusions

This paper presents the first extensive sub-THz channel measurement campaign
at 142 GHz for the indoor factory scenarios. The directional and
omnidirectional path loss are modeled based on 82 TX-RX locations, including
30 LOS locations, 48 NLOS locations, and four outage locations. The
omnidirectional PLE for NLOS locations is 3.1, which is higher than the values
obtained from indoor office (2.7) and outdoor urban environments (2.9),
suggesting that factories may induce additional attenuation due to heavy
clutter for NLOS environments. Furthermore, the channel measurements conducted
at a low RX height indicate that near-the-ground UEs such as AGVs may
experience larger path loss (i.e., 10.7 dB and 6.0 dB increase for LOS and
NLOS locations) compared to a high RX height due to a higher probability of
blockage and scattering loss caused by surrounding obstructions. Moreover, the
channel enhancement measurements demonstrate that a single large metal plate
can provide an average gain of 25 dB in the desired pointing directions by
accurately orientating the plate. In general, the PRS increases the
omnidirectional received power by a maximum of 22 dB and a minimum of 0.5 dB
with a mean of 6.5 dB, suggesting that future implementations and deployments
of RISs can play an essential part in sub-THz wireless systems. This work may
motivate more effort on sub-THz applications in factories and provide insights
on factory channel model standardization above 100 GHz.

## References

  * [1] T. S. Rappaport, Y. Xing, O. Kanhere, S. Ju, A. Madanayake, S. Mandal, A. Alkhateeb, and G. C. Trichopoulos, “Wireless communications and applications above 100 GHz: Opportunities and challenges for 6G and beyond,” _IEEE Access_ , vol. 7, pp. 78 729–78 757, 2019.
  * [2] T. S. Rappaport _et al._ , “Millimeter wave mobile communications for 5G cellular: It will work!” _IEEE Access_ , vol. 1, pp. 335–349, May 2013.
  * [3] FCC, “Spectrum horizons,” _First Report and Order – ET Docket 18-21_ , Washington D. C., 2019.
  * [4] S. Ju, Y. Xing, O. Kanhere, and T. S. Rappaport, “Millimeter wave and sub-Terahertz spatial statistical channel model for an indoor office building,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 6, pp. 1561–1575, 2021.
  * [5] Y. Xing and T. S. Rappaport, “Millimeter wave and Terahertz urban microcell propagation measurements and models,” _IEEE Communications Letters_ , vol. 25, no. 12, pp. 3755–3759, 2021.
  * [6] Y. Chen, Y. Li, C. Han, Z. Yu, and G. Wang, “Channel measurement and ray-tracing-statistical hybrid modeling for low-Terahertz indoor communications,” _IEEE Transactions on Wireless Communications_ , vol. 20, no. 12, pp. 8163–8176, 2021.
  * [7] S. Ju, Y. Xing, O. Kanhere, and T. S. Rappaport, “Sub-Terahertz channel measurements and characterization in a factory building,” in _2022 IEEE International Conference on Communications (ICC)_ , May 2022, pp. 1–6.
  * [8] S. Ju, S. H. A. Shah, M. A. Javed, J. Li, G. Palteru, J. Robin, Y. Xing, O. Kanhere, and T. S. Rappaport, “Scattering mechanisms and modeling for Terahertz wireless communications,” in _ICC 2019 - 2019 IEEE International Conference on Communications (ICC)_ , 2019, pp. 1–7.
  * [9] B. Chen, J. Wan, L. Shu, P. Li, M. Mukherjee, and B. Yin, “Smart factory of Industry 4.0: Key technologies, application case, and challenges,” _IEEE Access_ , vol. 6, pp. 6505–6519, 2018.
  * [10] W. Khawaja, O. Ozdemir, Y. Yapici, F. Erden, and I. Guvenc, “Coverage enhancement for nlos mmWave links using passive reflectors,” _IEEE Open Journal of the Communications Society_ , vol. 1, pp. 263–281, 2020.
  * [11] Y. Li, Y. Wang, Y. Chen, Z. Yu, and C. Han, “Channel measurement and characterization with modified SAGE algorithm in an indoor corridor at 300 GHz,” _arXiv_ , 2022.
  * [12] N. A. Abbasi _et al._ , “Ultra-wideband double directional channel measurements for THz communications in urban environments,” in _ICC 2021 - IEEE International Conference on Communications_ , 2021, pp. 1–6.
  * [13] S. W. Ellingson, “Path loss in reconfigurable intelligent surface-enabled channels,” in _2021 IEEE 32nd Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC)_ , 2021, pp. 829–835.
  * [14] Y. Xing and T. S. Rappaport, “Propagation measurement system and approach at 140 GHz-moving to 6G and above 100 GHz,” in _2018 IEEE Global Communications Conference (GLOBECOM)_ , 2018, pp. 1–6.
  * [15] T. S. Rappaport, R. Heath Jr, R. Daniels, and J. Murdock, _Millimeter wave wireless communications_ , 1st ed. Pearson Education, 2015.
  * [16] S. Sun _et al._ , “Synthesizing omnidirectional antenna patterns, received power and path loss from directional antennas for 5G millimeter-wave communications,” in _2015 IEEE Global Communications Conference (GLOBECOM)_ , 2015, pp. 1–7.
  * [17] ——, “Investigation of prediction accuracy, sensitivity, and parameter stability of large-scale propagation path loss models for 5G wireless communications,” _IEEE Transactions on Vehicular Technology_ , vol. 65, no. 5, pp. 2843–2860, 2016.
  * [18] T. S. Rappaport, G. R. MacCartney, M. K. Samimi, and S. Sun, “Wideband millimeter-wave propagation measurements and channel models for future wireless communication system design,” _IEEE Transactions on Communications_ , vol. 63, no. 9, pp. 3029–3056, 2015.
  * [19] D. Shakya, T. Wu, M. E. Knox, and T. S. Rappaport, “A wideband sliding correlation channel sounder in 65 nm CMOS: Evaluation board performance,” _IEEE Transactions on Circuits and Systems II: Express Briefs_ , vol. 68, no. 9, pp. 3043–3047, 2021.
  * [20] Y. Xing, T. S. Rappaport, and A. Ghosh, “Millimeter wave and sub-THz indoor radio propagation channel measurements, models, and comparisons in an office environment,” _IEEE Communications Letters_ , vol. 25, no. 10, pp. 3151–3155, 2021.
  * [21] Y. Xing and T. S. Rappaport, “Propagation measurements and path loss models for sub-THz in urban microcells,” in _ICC 2021 - IEEE International Conference on Communications_ , 2021, pp. 1–6.
  * [22] S. Ju and T. S. Rappaport, “Sub-Terahertz spatial statistical MIMO channel model for urban microcells at 142 GHz,” in _2021 IEEE Global Communications Conference (GLOBECOM)_ , 2021, pp. 1–6.
  * [23] M. Schmieder, F. Undi, M. Peter, E. Koenig, and W. Keusgen, “Directional wideband channel measurements at 28 GHz in an industrial environment,” in _2019 IEEE Global Communications Conference (GLOBECOM)_ , 2019, pp. 1–6.
  * [24] A. Al-Saman, M. Mohamed, M. Cheffena, and A. Moldsvor, “Wideband channel characterization for 6G networks in industrial environments,” _Sensors_ , vol. 21, no. 6, 2021.
  * [25] D. Chizhik _et al._ , “Path loss and directional gain measurements at 28 GHz for factory automation,” in _2019 IEEE International Symposium on Antennas and Propagation and USNC-URSI Radio Science Meeting_ , 2019, pp. 2143–2144.
  * [26] 3GPP, “Technical specification group radio access network; study on channel model for frequencies from 0.5 to 100 GHz (Release 16),” TR 38.901 V16.0.0, October 2019.