#  Jina Embeddings 2: $8192$-Token General-Purpose Text Embeddings for Long
Documents

Michael Günther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy
Abel,
Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua,
Bo Wang, Maximilian Werk, Nan Wang Han Xiao
Jina AI GmbH, Ohlauer Str. 43, 10999 Berlin, Germany
{michael.guenther, jackmin.ong, isabelle.mohr alaeddine.abdessalem,
tanguy.abel, kalim.akram, susana.guzman, georgios.mastrapas, saba.sturua,
bo.wang, maximilian.werk, nan.wang<EMAIL_ADDRESS>
(2023/10/31)

###### Abstract

Text embedding models have emerged as powerful tools for transforming
sentences into fixed-sized feature vectors that encapsulate semantic
information. While these models are essential for tasks like information
retrieval, semantic clustering, and text re-ranking, most existing open-source
models, especially those built on architectures like BERT, struggle to
represent lengthy documents and often resort to truncation. One common
approach to mitigate this challenge involves splitting documents into smaller
paragraphs for embedding. However, this strategy results in a much larger set
of vectors, consequently leading to increased memory consumption and
computationally intensive vector searches with elevated latency.

To address these challenges, we introduce Jina Embeddings v2, an open-source
text embedding model111Base model (0.27G): https://huggingface.co/jinaai/jina-
embeddings-v2-base-en;
Small model (0.07G): https://huggingface.co/jinaai/jina-embeddings-v2-small-
en;
The Large model will be released soon. capable of accommodating up to $8192$
tokens. This model is designed to transcend the conventional $512$-token limit
and adeptly process long documents. Jina Embeddings v2 not only achieves
state-of-the-art performance on a range of embedding-related tasks in the MTEB
benchmark but also matches the performance of OpenAI’s proprietary text-
embedding-ada-002 model. Additionally, our experiments indicate that an
extended context can enhance performance in tasks such as NarrativeQA.

##  1 Introduction

Using neural networks to encode text and images into embedding representations
has become a standard practice for analyzing and processing vast amounts of
unstructured data. In natural language processing, sentence embedding models
Reimers and Gurevych (2019) transform the semantics of phrases, sentences, and
paragraphs into points within a continuous vector space. These transformed
data points can subsequently be used for a myriad of downstream applications,
such as information retrieval, as well as clustering and classification tasks.

Despite the numerous applications of embedding models, a prevailing challenge
faced by many models is the limitation on the maximum sequence lengths of text
that can be encoded into a single embedding. To circumvent this, practitioners
often segment documents into smaller chunks prior to encoding. This tactic,
unfortunately, results in fragmented semantic meanings, causing the embeddings
to misrepresent the entirety of paragraphs. Furthermore, this method yields a
plethora of vectors, culminating in heightened memory usage, increased
computational demands during vector searches, and extended latencies. The
dilemma is exacerbated when embedding vectors are stored in database systems
that construct memory-intensive index structures.

The root of these text length restrictions can be traced back to the BERT
architecture, which underpins most of the current open-source models. The
authors of Press et al. (2022) demonstrated that these models struggle to
accurately represent long documents. They introduced an alternative positional
embedding method named ALiBi, enabling efficient training of models to encode
long text sequences. Regrettably, up until this point, the approach was
exclusively employed for generative language models, neglecting its potential
for open-source encoder language models aimed at crafting document embeddings.
This research bridges that gap by incorporating ALiBi bidirectionally into the
BERT framework, rendering it apt for encoding tasks. As a result, it empowers
users to utilize it for downstream operations on texts spanning up to $8192$
tokens. Moreover, we fine-tuned this enhanced BERT model, harnessing hundreds
of millions of text samples to encode texts into singular embedding
representations. Our model’s resultant embeddings outshine those of the Jina
Embeddings v1 model suite Günther et al. (2023) in the MTEB benchmark and
rival the prowess of state-of-the-art models like E5 Wang et al. (2022). We
also found that large context lengths can amplify the efficacy of numerous
downstream tasks tied to embeddings. Given that the majority of available
embedding evaluation datasets comprise mainly brief text passages, we have
curated datasets encompassing long text values to better evaluate embeddings.
These datasets, alongside our models, are made accessible via our Hugging Face
repository222https://huggingface.co/jinaai.

This paper is structured as follows: We begin with an overview of related work
in Section 2. This is followed by an outline of the training process in
Section 3, a description of the backbone model and its pre-training in Section
4, and a detailed walkthrough of the fine-tuning process for embeddings
generation in Section 5. We culminate with an exhaustive evaluation in Section
6 and conclusions in Section 7.

##  2 Related Work

Embedding training has undergone significant evolution, transitioning from
foundational techniques such as Latent Semantic Indexing (LSA) Deerwester et
al. (1990) and Latent Dirichlet Allocation (LDA) Blei et al. (2001) to the
sophisticated prowess of pre-trained models like Sentence-BERT Reimers and
Gurevych (2019). A notable shift in recent advancements is the emphasis on
unsupervised contrastive learning, as showcased by works like Gao et al.
(2022); Wang et al. (2022). Pioneering models like Condenser Gao and Callan
(2021) and RetroMAE Xiao et al. (2022) have brought forth specialized
architectures and pre-training methods explicitly designed for dense encoding
and retrieval.

The E5 Wang et al. (2022), Jina Embeddings v1 Günther et al. (2023), and GTE
Li et al. (2023) collections of embedding models represent another leap
forward. These models propose a holistic framework tailored for effective
training across a myriad of tasks. This framework adopts a multi-stage
contrastive training approach. An initial phase focuses on unsupervised
training using a vast collection of weak pairs sourced from public data,
enhancing the model’s domain generalization. Following this, a supervised
fine-tuning stage employs a curated set of annotated text triples,
representing diverse tasks. Together, these sequential stages yield state-of-
the-art outcomes on the MTEB benchmark.

Yet, despite such advancements, a glaring limitation persists: the $512$-token
constraint on input sequences, stemming from foundational models like BERT.
This cap is insufficient for encoding lengthy documents, often exceeding a
page. ALiBi Press et al. (2022) emerges as a promising solution, presenting a
technique that sidesteps conventional positional embeddings and facilitates
training on sequences exceeding $2048$ tokens. Notably, its typical
application is centered around generative models, which inherently adopt a
unidirectional bias, rendering it less suitable for embedding tasks.

Effective evaluation remains paramount for embedding models, ensuring they
meet the diverse demands of real-world applications. The BEIR benchmark Thakur
et al. (2021) stands out, offering evaluations across a set of retrieval tasks
and settings. Similarly, the MTEB benchmark Muennighoff et al. (2023)
highlights the extensive applicability of text embeddings, spanning a variety
of tasks and languages. However, a notable gap in both benchmarks is their
limited focus on encoding long documents — a critical aspect for comprehensive
embedding evaluation.

##  3 Training Process Overview

The training process for Jina Embeddings v2 is divided into three stages:

  1. I

Pre-training the Backbone: For the backbone pre-training, we design a modified
BERT model capable of encoding documents with up to $8192$ tokens. This model
is trained on a full-text corpus using a masked language modeling objective.

  2. II

First Fine-tuning with Text Pairs: To encode a text passage into a single
vector representation, the model is fine-tuned in an unsupervised manner on
text pairs.

  3. III

Second Fine-tuning with Hard Negatives: The model is further fine-tuned using
text pairs complemented with hard negatives. This stage is crucial for
enabling the model to better distinguish between relevant passages and
related, but irrelevant text passages.

While both stages II and III are geared towards training the models for
embedding tasks, the latter is especially critical for improving the model’s
performance in retrieval and classification tasks (refer to Section 6.2).

##  4 Backbone Pre-training

For the backbone training, we introduce a novel transformer model. Although
its architecture is akin to the BERT model proposed by Devlin et al. (2019),
we implement several modifications to enhance its ability to encode extended
text sequences and to generally bolster its language modeling capabilities.
For the training process, we largely adopt the approach described in Liu et
al. (2019a), incorporating additional performance optimizations.

###  4.1 Model Architecture

Jina BERT
---
Model |  Layers |  Hidden |  Params
jina-bert-small |  4 |  512 |  33M
jina-bert-base |  12 |  768 |  137M
Table 1: Architecture specifications for the Jina BERT models of varying
sizes. The number of attention heads is selected to ensure a consistent head
dimension of $64$.

##### Attention with Linear Biases:

(a) Encoder ALiBi

(b) Causal ALiBi

Figure 1: With ALiBi attention, a linear bias is incorporated into each
attention score preceding the softmax operation. Each attention head employs a
distinct constant scalar, $m$, which diversifies its computation. Our model
adopts the encoder variant where all tokens mutually attend during
calculation, contrasting the causal variant originally designed for language
modeling. In the latter, a causal mask confines tokens to attend solely to
preceding tokens in the sequence.

For the self-attention mechanism within the attention blocks, we adopt the
Attention with Linear Biases (ALiBi) approach Press et al. (2022). ALiBi
forgoes the use of positional embeddings. Instead, it encodes positional
information directly within the self-attention layer by introducing a constant
bias term to the attention score matrix of each layer, ensuring that proximate
tokens demonstrate stronger mutual attention. While the original
implementation was designed for causal language modeling and featured biases
solely in the causal direction, such an approach is not compatible with the
bidirectional self-attention inherent in our encoder model. For our purposes,
we employ the symmetric encoder variant where attention biases are mirrored to
ensure consistency in both
directions333https://github.com/ofirpress/attention_with_linear_biases/issues/5.
Figure 1 depicts the computation of attention scores within the multi-head
attention heads. Each head’s scaling value, $m_{i}$, out of the total $n$
heads, is derived using Equation (4.1).

$\displaystyle m_{i}=\begin{cases}b^{2i}&i<a\\\ b^{1+2(i-a)}&i\geq a\\\
\end{cases}$ $\displaystyle
a=2^{\left\lfloor\log_{2}n\right\rfloor}\;\;b=2^{\frac{-8}{2^{\lceil\log_{2}n\rceil}}}$
(1)

##### Gated Linear Units:

For the feedforward sublayers within the attention blocks, we adopt Gated
Linear Units (GLU), originally introduced in Dauphin et al. (2016). They’ve
demonstrated performance enhancements when incorporated into transformers
Shazeer (2020). For the small and base models, we employ the GEGLU variant,
which leverages the GELU activation function for the GLU. Conversely, for the
large model, we utilize the ReGLU variant with the ReLU activation function.
This choice was driven by our observation that training the large model with
GEGLU, despite its promising initial MLM accuracy, was unstable.

##### Layer Normalization:

Regarding Layer Normalization ba2016layernorm, we align with the post-layer
normalization approach from Vaswani et al. (2017) in our attention blocks.
Preliminary tests with pre-layer normalization, as mentioned in Shoeybi et al.
(2019) and Nguyen and Salazar (2019), didn’t enhance training stability or
performance. Consequently, we opted not to integrate it into our model.

###  4.2 Full-Text Datasets for Pre-training

For the pre-training phase, we leverage the English “Colossal, Cleaned, Common
Crawl (C4)” dataset 444https://huggingface.co/datasets/c4, encompassing
approximately 365 million text documents harvested from the web, summing to
around 170 billion tokens. As delineated in Raffel et al. (2020), the C4
dataset is a refined iteration of Common Crawl, utilizing heuristics for
cleanup and language recognition, retaining solely English content. As a
result, our models are monolingual and tailored exclusively for English texts.
The purification process also encompasses the removal of webpages hosting
inappropriate content. We reserve $1\%$ of the dataset for evaluating
validation loss and the accuracy of the masked language modeling (MLM) task.

###  4.3 Training Algorithm

Our model’s pre-training revolves around the masked language modeling
objective, excluding the next sentence prediction (NSP) task due to its
perceived limited contribution to downstream task performance Liu et al.
(2019a). We mask $30\%$ of the input tokens randomly, employing whole word
masking Devlin et al. (2019), and condition the models to infer these masked
tokens. Of these masked tokens, 80% are substituted with the [MASK] token,
$10\%$ with a random token, and the remaining $10\%$ stay unaltered.

The masked tokens are predicted by a decoder
$D:\mathbb{R}^{d}\to\mathbb{R}^{|V|}$, which takes the output token embedding
$\bm{e_{i}}\in\mathbb{R}^{d}$ of a masked token and predicts a probability for
each token in the vocabulary. The loss $L_{\mathit{MLM}}$ is computed by
evaluating the cross entropy between the predicted probabilities and the
actual masked tokens, as described in Equation (2). Here,
$I:\\{1,\ldots,n\\}\to|V|$ denotes the function that maps each of the $n$
masked tokens to its respective index in the vocabulary:

$\displaystyle\mathcal{L}_{\mathit{MLM}}(t)$
$\displaystyle:=\sum\limits_{k=1}^{n}\ln D(\bm{e_{i}})_{I(k)}$ (2)

Given our model’s reliance on ALiBi attention Press et al. (2022), training
position embeddings becomes unnecessary. This allows us to pre-train more
efficiently on shorter sequences and adapt to longer sequences in subsequent
tasks. Throughout our pre-training, we operate on sequences capped at $512$
tokens in length. Diverging from the methods in Devlin et al. (2019) and Liu
et al. (2019a), our sequences originate from individual documents without any
multi-document packing. Furthermore, we refrain from sampling multiple
sequences from a singular document. For each document, we exclusively consider
its initial 512 tokens, truncating any excess. Given our consistent global
batch size of 4096, each batch, due to its varying sequence length, contains a
unique number of masked tokens when calculating loss.

##### Optimizer:

Mirroring the optimization strategy of RoBERTa Liu et al. (2019a), we employ
the AdamW algorithm Loshchilov and Hutter (2017), characterized by parameters
$\beta_{1}=0.9$, $\beta_{2}=0.98$, $\epsilon=1e-6$, a weight decay of $0.01$,
dropout set at $0.1$, and attention dropout also at $0.1$. Our learning rate
schedule is linear, starting at 0 and peaking at a rate of $\eta$ post
$10,000$ steps. Here, the values of $\eta$ are designated as
$1\mathrm{e}{-3}$, $6\mathrm{e}{-4}$, and $4\mathrm{e}{-4}$ for the small,
base, and large models respectively. A linear decay to zero ensues after
reaching the $100,000$ steps threshold.

##### Mixed precision training:

We resort to FP16 dynamic mixed precision Micikevicius et al. (2018) for pre-
training our models, facilitated by the DeepSpeed software package Rasley et
al. (2020). Our preliminary tests using BF16 revealed unsatisfactory
performance metrics, both in MLM accuracy and the downstream GLUE tasks.

##  5 Fine-Tuning for Embeddings

After pre-training the Jina BERT models, we further fin-tune each of the
models to encode a text sequence into a single vector representation. The core
idea behind our embedding approach is inspired by the Sentence-BERT technique
Reimers and Gurevych (2019). To enable a model to perform a text operation, we
augment it with a mean pooling layer. This mean pooling step averages the
token embeddings to merge their information into a single representation,
without introducing additional trainable parameters. The training process for
this enhanced model consists of an unsupervised phase followed by a supervised
one.

###  5.1 First Unsupervised Fine-tuning

During the unsupervised fine-tuning phase, we train the models on a corpus of
text pairs $(q,p)\in D_{\mathit{pairs}}$, comprising a query string $q$ and a
target string $p$.

##### Construction of the Text-Pair Datasets:

We utilize roughly 40 diverse data sources, akin to the data preparation
outlined in the report we previously published about our inaugural embedding
model suite Günther et al. (2023). We observed that the inclusion of title-
abstract pairs from documents significantly enhances performance on clustering
tasks. As detailed in Günther et al. (2023), we implement consistency
filtering (Dai et al., 2023; Wang et al., 2022) to elevate the quality of the
text pair corpus. For batch creation, we adhere to our earlier strategy: for
every new batch, we randomly choose a data source and extract as many pairs as
needed to fill that batch. All pairs within the data sources are pre-shuffled.
Depending on the quality and quantity of the data sources, we assign different
sampling rates for the pairs.

##### Training:

The goal of unsupervised fine-tuning is to encode text values that constitute
a pair into analogous embedding representations, while encoding texts that
aren’t paired into distinct embeddings. To achieve this contrastive goal, we
employ the InfoNCE (van den Oord et al., 2018) loss function, similar to our
earlier embedding models Günther et al. (2023). This loss function calculates
the loss value for a pair $(q,p)\sim B$ within a batch $B\in D^{k}$ of $k$
text pairs as follows:

$\displaystyle\mathcal{L}_{\mathrm{NCE}}^{\mathrm{pairs}}(B):=\mathbb{E}_{(q,p)\sim
B}\left[-\ln\frac{e^{s(q,p)/\tau}}{\sum\limits_{i=1}^{k}e^{s(q,p_{i})/\tau}}\right]$
(3)

The function evaluates the cosine similarity $s(p,q)$ between a given query
$q$ and its corresponding target $p$, relative to the similarity of all other
targets in the batch. Given the typically symmetric nature of similarity
measures, we compute the loss in both directions:

$\displaystyle\mathcal{L}^{\mathrm{pairs}}(B)$
$\displaystyle:=\mathcal{L}^{\mathrm{pairs}}_{\mathrm{NCE}}(B)+\mathcal{L}^{\mathrm{pairs}}_{\overline{\mathrm{NCE}}}(B),\text{
with}$
$\displaystyle\mathcal{L}_{\overline{\mathrm{NCE}}}^{\mathrm{pairs}}(B)$
$\displaystyle:=\mathbb{E}_{(q,p)\sim
B}\left[-\ln\frac{e^{s(p,q)/\tau}}{\sum\limits_{i=1}^{k}e^{s(p,q_{i})/\tau}}\right]$
(4)

The constant temperature parameter $\tau$ influences how the loss function
weighs minor differences in the similarity scores Wang and Liu (2021).
Empirical testing suggests that $\tau=0.05$ is effective.

###  5.2 Second Supervised Fine-tuning

The goal of the supervised fine-tuning stage is to improve the models’ ranking
capabilities. This improvement is achieved by training with datasets that
include additional negative examples.

##### Dataset with annotated negatives:

We have prepared retrieval datasets, such as MSMarco Bajaj et al. (2016) and
Natural Questions (NQ) Kwiatkowski et al. (2019), in addition to multiple non-
retrieval datasets like the Natural Language Inference (NLI) dataset Bowman et
al. (2015). These datasets encompass a collection of queries with annotated
relevant passages and several negative examples, consistent with earlier work
Wang et al. (2022). Each training batch $B$, structured as
$(q,p,n_{1},\ldots,n_{15})$, includes one positive and 15 negative instances.
For retrieval datasets, hard negatives are discerned by identifying passages
deemed similar by retrieval models. This approach instructs the model to
prioritize relevant documents over those that are merely semantically related.
For non-retrieval datasets, negatives are selected randomly, since drawing a
clear line between positives and hard negatives isn’t feasible. This is
because, unlike relevancy, it’s challenging to make a binary determination
regarding the similarity or dissimilarity of two textual values. Consequently,
opting for hard negatives in such datasets seemed to diminish the models’
quality. Nonetheless, it remains crucial to integrate these datasets into the
stage III training to ensure continued performance on non-retrieval tasks. To
ensure that hard negative passages are indeed less relevant than the annotated
relevant ones, we employ a cross-encoder model to validate that their
relevance score is indeed lower.

##### Training:

Our training employs a modified variant of the InfoNCE loss function, denoted
as $\mathcal{L}_{\mathrm{NCE}^{+}}$ and described by Equation (5). Similar to
the preceding loss function, this one is bidirectional and incorporates the
additional negatives when pairing queries with passages:

$\displaystyle\mathcal{L}_{\mathrm{NCE}^{+}}(B):=$
$\displaystyle\;\;\;\;\;\mathbb{E}_{r\sim
B}\Bigg{[}-\ln\frac{e^{s(q,p)/\tau}}{\sum\limits_{i=1}^{k}\Big{[}e^{s(q,p_{i})/\tau}+\sum\limits_{j=1}^{15}e^{s(q,n_{j,i})/\tau}\Big{]}}\Bigg{]}$
$\displaystyle\,+\mathbb{E}_{r\sim
B}\Bigg{[}-\ln\frac{e^{s(p,q)/\tau}}{\sum\limits_{i=1}^{k}e^{s(p,q_{i})/\tau}}\Bigg{]}$
$\displaystyle\text{with}\;r=(q,p,n_{1},\ldots,n_{15}).$ (5)

###  5.3 Memory Optimizations

When training embedding models, having a large batch size is crucial. This is
because the InfoNCE loss functions $\mathcal{L}^{pairs}$ and
${L}_{\mathrm{NCE}^{+}}$ compute the loss values based on the entirety of the
batch. The batch size determines the number of text values each individual
text value is compared against. As a result, the computed loss value might not
be as expressive with smaller batches. Li et al. (2023) provided an in-depth
analysis, highlighting the positive impact of larger batch sizes on the
performance of the resultant embedding model. To accommodate larger batch
sizes, it becomes essential to minimize the memory overhead during training.
We achieved this by training our models in mixed precision Micikevicius et al.
(2018) and leveraging the deepspeed Rasley et al. (2020) framework for further
optimization. Activation checkpointing Chen et al. (2016) was also employed to
curtail memory usage. Specifically, we inserted a checkpoint after each BERT
layer within our model.

##  6 Evaluation

To evaluate the efficacy of our approach, we initiate with a comprehensive
analysis of our pre-trained backbone models, as outlined in Section 6.1. This
is followed by an in-depth assessment of our embedding models in Section 6.2.
Furthermore, we have conducted experiments to delve into the effects of
encoding extended sequence lengths on the performance of the embeddings,
presented in Section 6.2.2.

###  6.1 Evaluation of Jina BERT

GLUE
---
Model | Params | MNLI | QQP | QNLI | SST-2 | CoLa | STS-B | MRPC | RTE | WNLI | Average
BERT Base | 110M | 84.6/83.4 | 71.2 | 90.5 | 93.5 | 52.1 | 85.8 | 88.9 | 66.4 | - | -
BERT Large | 340M | 86.7/85.9 | 72.1 | 92.7 | 94.9 | 60.5 | 86.5 | 89.3 | 70.1 | - | -
RoBERTa | 355M | 90.8/90.2 | 90.2 | 98.9 | 96.7 | 67.8 | 92.2 | 92.3 | 88.2 | 89.0 | 88.5
Jina BERT Small | 33M | 80.1/78.9 | 78.9 | 86.0 | 89.6 | 28.8 | 84.8 | 84.1 | 68.8 | 55.5 | 72.9
Jina BERT Base | 137M | 85.7/85.4 | 80.7 | 92.2 | 94.5 | 51.4 | 89.5 | 88.4 | 78.7 | 65.1 | 80.7
Table 2: Evaluation of the Jina BERT models on the GLUE benchmark Figure 2:
Variation of model MLM accuracy in function of the sequence length

Following previous work Liu et al. (2019b), we evaluate our pretrained models
on the GLUE benchmark Wang et al. (2018). General Language Understanding
Evaluation (GLUE) is a collection of nine datasets for evaluating natural
language understanding systems. Six tasks are framed as either single-sentence
classification or sentence-pair classification tasks. The GLUE organizers
provide training, development, and test data splits, as well as a submission
server and leaderboard.555https://gluebenchmark.com The test split does not
contain labels, and the submission server allows participants to evaluate and
compare their systems against the private labels of the test split.

For the Jina BERT training described in Section 4, we fine-tune the pre-
trained models on the corresponding single-task training data using several
hyperparameter settings and, for each task, pick the best fine-tuning
hyperparameters on the development set.

Following the methodology of Phang et al. (2018), for RTE, STS, and MRPC, we
fine-tune starting from the MNLI single-task model, rather than the baseline
pretrained Jina BERT models. As in the BERT paper Devlin et al. (2019), our
fine-tuning procedure relies on representing the input sequence and using the
final hidden vector $C\in\mathbb{R}^{H}$ corresponding to the first input
token ([CLS]) as the aggregate representation.

We train for 10 epochs with batch sizes $\\{16,32\\}$ and learning rates
$\\{1\mathrm{e}{-5},2\mathrm{e}{-5},3\mathrm{e}{-5}\\}$. For each task, the
best fine-tuned model on the development set is used for the test set.

In Table 2, we report the results of the best-performing models on the test
sets after submission to the GLUE benchmark server.

Furthermore, we evaluate Jina BERT models on documents of long text sequences
by computing the accuracy of the MLM task with varying sequence lengths. The
accuracy of masked language modeling is computed on $50,000$ samples from the
C4 validation set where, for each chosen sequence length, each sample document
is tokenized and truncated to fit the sequence length. We compare Jina BERT to
RoBERTa and BERT models in Figure 2. It essentially shows that, even though
Jina BERT models were trained on a $512$ sequence length, the MLM accuracy
does not drop when we extrapolate to an $8192$ sequence length. For other BERT
and RoBERTa models, since they use absolute positional embeddings that are
trained on a $512$ sequence length, it’s not possible to compute the MLM
accuracy beyond $512$. The figure demonstrates ALiBi’s effectiveness in
maintaining MLM performance during inference for long documents.

###  6.2 Evaluation of Jina Embeddings v2

To comprehensively evaluate our embedding models, we employ the Massive Text
Embedding Benchmark (MTEB) Muennighoff et al. (2023). Our choice of MTEB is
motivated by its unparalleled breadth, distinguishing it among embedding
benchmarks. Rather than focusing on a single task and dataset, MTEB covers an
expansive set of 8 tasks, encompassing a rich collection of 58 datasets across
112 languages. This expansive benchmark allows us to scrutinize our model’s
adaptability across diverse applications and languages and benchmark it
against other top-performing models.

However, a limitation of the MTEB benchmark is its omission of very long
texts, which are essential for evaluating our model’s prowess in handling
$8192$ sequence lengths. Consequently, we introduce new retrieval and
clustering tasks featuring extended documents, and we detail the performance
of our model against its peers in Section 6.2.2.

Clustering: The goal here is to aptly group a collection of sentences or
paragraphs. Within the MTEB benchmark suite, a mini-batch $k$-means model is
employed, operating with a batch size of 32. Here, $k$ represents the number
of unique labels in the dataset. Model performance is evaluated using the
$\mathcal{V}$ measure, a metric insensitive to cluster label permutations,
guaranteeing that assessments are independent of label configurations.

We incorporate two new clustering tasks featuring extended documents within
the MTEB clustering task subset. The inaugural task, named PatentClustering,
draws from the BigPatent666https://huggingface.co/datasets/big_patent dataset
Sharma et al. (2019), challenging the k-means model to organize patents by
their respective categories. Patent documents average $6,376$ tokens, spanning
a range from a brief $569$ tokens to an extensive $218,434$ tokens. Our second
task, titled WikiCitiesClustering, sources from the English subset of the
refined Wikipedia dump Foundation (2022), available as a dataset on Hugging
Face777https://huggingface.co/datasets/wikipedia. For this task, we curate a
roster of nations from Wikidata and extract Wikipedia articles of their cities
from the refined dataset. The objective is to group cities by their parent
country. On average, articles consist of $2,031$ tokens, with the length
varying between a succinct 21 tokens to a comprehensive $20,179$ tokens.

Retrieval: This task entails a dataset comprising a corpus, a set of queries,
and associated mappings connecting each query to pertinent corpus documents.
The mission is to discern relevant documents for a specific query. Both
queries and corpus documents undergo encoding, after which their similarity
scores are derived using cosine similarity. Subsequently, metrics like
nDCG$@10$ (which serves as the primary metric), MRR$@k$, MAP$@k$,
precision$@k$, and recall$@k$ are computed for diverse $k$ values. This task
is inspired by datasets and evaluation methods presented by BEIR Thakur et al.
(2021).

To expand the scope of the MTEB, we introduce a new retrieval task named
NarrativeQA, derived from the
narrativeqa888https://huggingface.co/datasets/narrativeqa dataset. This
dataset boasts realistic QA instances, curated from literature (encompassing
both fiction and non-fiction) and film scripts. The corpus averages $74,843$
tokens per document, with the lengthiest document tallying up to $454,746$
tokens, and the most concise one comprising $4,550$ tokens.

####  6.2.1 Results on MTEB

MTEB
---
Model | Params | CF | CL | PC | RR | RT | STS | SM | Average
text-embedding-ada-002 | unknown | 70.93 | 45.9 | 84.89 | 56.32 | 49.25 | 80.97 | 30.8 | 60.99
e5-base-v2 | 110M | 73.84 | 43.8 | 85.73 | 55.91 | 50.29 | 81.05 | 30.28 | 61.5
all-MiniLM-L6-v2 | 23M | 63.05 | 42.35 | 82.37 | 58.04 | 41.95 | 78.9 | 30.81 | 56.26
all-mpnet-base-v2 | 110M | 65.07 | 43.69 | 83.04 | 59.36 | 43.81 | 80.28 | 27.49 | 57.78
jina-small-v2 | 33M | 68.82 | 40.08 | 84.44 | 55.09 | 45.14 | 80 | 30.56 | 58
jina-base-v2 | 137M | 73.45 | 41.73 | 85.38 | 56.98 | 47.87 | 80.7 | 31.6 | 60.38

CF: Classification Accuracy [%] CL: Clustering $\mathcal{V}$ measure[%] PC:
Pair Classification Average Precision [%]
RR: Reranking MAP [%] RT: Retrieval nDCG@10 STS: Sentence Similarity Spearman
Correlation [%]
SM: Summarization Spearman Correlation [%]

Table 3: Evaluation of the Jina Embeddings v2 models on the MTEB benchmark

Figure 3: Evaluation with Different Maximum Sequence Lengths999For e5-base-v2,
we abstained from employing specific prefixes like “query: ”, which might
result in varied evaluation outcomes. text-embedding-ada-002 caps its encoding
at 8191 tokens, not $8192$.

The evaluation of embedding models within the MTEB benchmark, as illustrated
in Table 3, reveals significant contrasts between Jina’s text embedding
models, namely jina-small-v2 and jina-base-v2, and other contemporary models.
These differences are especially pronounced in tasks showing marked
performance disparities, such as Classification (CF) and Retrieval (RT).

In Classification (CF), the jina-base-v2 model, equipped with 137 million
parameters, emerges as a leading performer. It records superior scores,
outpacing most competing models, underscoring its efficacy in text
classification. Conversely, the jina-small-v2 model, equipped with a modest 33
million parameters, trails behind some other models in this task. This
underscores the pivotal role model size plays in certain downstream tasks,
with more extensive architectures yielding potential benefits.

For the Retrieval (RT) task, jina-small-v2 showcases formidable performance,
signaling its adeptness for information retrieval. It ranks amidst top-tier
models, indicating its prowess in retrieval-centric tasks. Similarly, jina-
base-v2 excels, registering a slightly superior score, reaffirming its
formidable retrieval aptitude. Both models underscore their credibility in
tasks necessitating adept information retrieval. Given that models all-
MiniLM-L6-v2 and all-mpnet-base-v2 omit the second-stage finetuning which
jina-small-v2 and jina-base-v2 undergo, it’s foreseeable that our models would
excel in these tasks.

In conclusion, both the base and small text embedding models display
commendable performance within the MTEB benchmark. Their standout performance,
relative to other models in tasks like Classification and Retrieval, suggests
model size’s influential role in specific text processing endeavors. Both
models reaffirm their potency in retrieval, marking them as pivotal tools for
a plethora of natural language processing tasks.

####  6.2.2 Impact of Maximum Sequence Length

As delineated in Section 6.1, the pre-training generalizes across extended
sequence lengths. Consequently, the MLM accuracy for long sequences, spanning
up to $8192$ tokens, mirrors that of shorter sequences, despite the exclusive
training on abbreviated text sequences. During finetuning, our models train
solely on texts not exceeding $512$ tokens, yet they cater to texts reaching
$8192$ tokens for the MTEB evaluation detailed in Section 6.2.

To discern how sequence length impacts the accuracy of downstream tasks, we
executed long document clustering and retrieval tasks, modulating the
tokenizer’s maximum sequence length. This allows us to gauge the models’
performance on variable sequence lengths through truncation. Since a majority
of the extant tasks in the MTEB feature documents under $512$ tokens, we
resort to our three novel datasets elucidated in Section 6.2, accessible on
Hugging Face101010Our datasets are available at https://huggingface.co/jinaai.
Furthermore, we employ the SciFact dataset Wadden et al. (2020), given its
substantial count of texts exceeding $512$ tokens.

Figure 9 depicts the nDCG$@10$ retrieval and the $\mathcal{V}$ measure scores
for the jina-base-v2 alongside four other renowned embedding models. Given
that only jina-base-v2 and OpenAI’s text-embedding-ada-002 support an 8k
sequence length, results reported for an 8191 sequence length for other models
are truncated to their intrinsic maximum, typically $512$. Generally, Figure 9
suggests that elongated sequence lengths contribute to enhanced outcomes. This
assertion is particularly true for the NarrativeQA task, where extending the
sequence length substantially bolsters performance. Due to the inherent nature
of the dataset, models limited to the text’s commencement frequently
underperform.

On the BigPatent clustering task, larger sequence lengths also result in
better performance. However, on the WikiCities clustering task, longer
sequence lengths seem to slightly diminish the models’ performance in most
instances. This suggests that an increase in sequence length doesn’t always
yield better outcomes. One explanation for this observation is that the
initial paragraph of a Wikipedia article about a city typically mentions the
country the city is in. Information towards the middle and end of the articles
is often less pertinent for identifying the country and might alter the
attributes that influence the clustering of the city embeddings.

##  7 Conclusion

We have introduced Jina Embeddings v2, a novel embedding model based on a
modified BERT architecture. This model eschews positional embeddings and
instead employs bi-directional ALiBi slopes to capture positional information.
By training a series of embedding models with this innovative architecture on
the Web document corpus C4 and subsequently fine-tuning them, we have enabled
the encoding of the semantics of both short and long textual values into
meaningful vector representations. This effort has produced a new suite of
open-source embedding models capable of encoding texts containing up to $8192$
tokens. These embeddings signify a 16x increase in the maximum sequence length
compared to leading open-source embedding models. Additionally, our model
suite exhibits competitive performance on the MTEB benchmark. We also
demonstrate how utilizing extended sequence lengths can offer our models an
advantage over those without such capabilities.

## References

  * Reimers and Gurevych [2019] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, 2019.
  * Press et al. [2022] Ofir Press, Noah A. Smith, and Mike Lewis.  Train short, test long: Attention with linear biases enables input length extrapolation, 2022.
  * Günther et al. [2023] Michael Günther, Louis Milliken, Jonathan Geuter, Georgios Mastrapas, Bo Wang, and Han Xiao.  Jina embeddings: A novel set of high-performance sentence embedding models.  _arXiv preprint arXiv:2307.11224_ , 2023.
  * Wang et al. [2022] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei.  Text embeddings by weakly-supervised contrastive pre-training.  _arXiv preprint arXiv:2212.03533_ , 2022.
  * Deerwester et al. [1990] Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman.  Indexing by latent semantic analysis.  _Journal of the American Society for Information Science_ , 41(6):391–407, 1990.
  * Blei et al. [2001] David Blei, Andrew Ng, and Michael Jordan.  Latent dirichlet allocation.  In T. Dietterich, S. Becker, and Z. Ghahramani, editors, _Advances in Neural Information Processing Systems_ , volume 14. MIT Press, 2001.  URL https://proceedings.neurips.cc/paper_files/paper/2001/file/296472c9542ad4d4788d543508116cbc-Paper.pdf.
  * Gao et al. [2022] Tianyu Gao, Xingcheng Yao, and Danqi Chen.  Simcse: Simple contrastive learning of sentence embeddings, 2022.
  * Gao and Callan [2021] Luyu Gao and Jamie Callan.  Condenser: a pre-training architecture for dense retrieval, 2021.
  * Xiao et al. [2022] Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao.  Retromae: Pre-training retrieval-oriented language models via masked auto-encoder, 2022.
  * Li et al. [2023] Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang.  Towards general text embeddings with multi-stage contrastive learning, 2023.
  * Thakur et al. [2021] Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych.  Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models, 2021.
  * Muennighoff et al. [2023] Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers.  Mteb: Massive text embedding benchmark, 2023.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * Liu et al. [2019a] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach, 2019a.
  * Dauphin et al. [2016] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier.  Language modeling with gated convolutional networks.  _CoRR_ , abs/1612.08083, 2016.  URL http://arxiv.org/abs/1612.08083.
  * Shazeer [2020] Noam Shazeer.  GLU variants improve transformer.  _CoRR_ , abs/2002.05202, 2020.  URL https://arxiv.org/abs/2002.05202.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _CoRR_ , abs/1706.03762, 2017.  URL http://arxiv.org/abs/1706.03762.
  * Shoeybi et al. [2019] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.  Megatron-lm: Training multi-billion parameter language models using model parallelism.  _CoRR_ , abs/1909.08053, 2019.  URL http://arxiv.org/abs/1909.08053.
  * Nguyen and Salazar [2019] Toan Q. Nguyen and Julian Salazar.  Transformers without tears: Improving the normalization of self-attention.  _CoRR_ , abs/1910.05895, 2019.  URL http://arxiv.org/abs/1910.05895.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Fixing weight decay regularization in adam.  _CoRR_ , abs/1711.05101, 2017.  URL http://arxiv.org/abs/1711.05101.
  * Micikevicius et al. [2018] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.  Mixed precision training.  In _International Conference on Learning Representations_ , 2018.
  * Rasley et al. [2020] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 3505–3506, 2020.
  * Dai et al. [2023] Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming-Wei Chang.  Promptagator: Few-shot dense retrieval from 8 examples.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=gmL46YMpu2J.
  * van den Oord et al. [2018] Aäron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _CoRR_ , abs/1807.03748, 2018.  URL http://arxiv.org/abs/1807.03748.
  * Wang and Liu [2021] Feng Wang and Huaping Liu.  Understanding the behaviour of contrastive loss.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2495–2504. IEEE, 2021.
  * Bajaj et al. [2016] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al.  Ms marco: A human generated machine reading comprehension dataset.  _arXiv preprint arXiv:1611.09268_ , 2016.
  * Kwiatkowski et al. [2019] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.  Natural questions: a benchmark for question answering research.  _Transactions of the Association of Computational Linguistics_ , 2019\.
  * Bowman et al. [2015] Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, 2015.
  * Chen et al. [2016] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin.  Training deep nets with sublinear memory cost.  _arXiv preprint arXiv:1604.06174_ , 2016.
  * Liu et al. [2019b] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692, 2019b.  URL http://arxiv.org/abs/1907.11692.
  * Wang et al. [2018] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  _CoRR_ , abs/1804.07461, 2018.  URL http://arxiv.org/abs/1804.07461.
  * Phang et al. [2018] Jason Phang, Thibault Févry, and Samuel R. Bowman.  Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks.  _CoRR_ , abs/1811.01088, 2018.  URL http://arxiv.org/abs/1811.01088.
  * Sharma et al. [2019] Eva Sharma, Chen Li, and Lu Wang.  BIGPATENT: A large-scale dataset for abstractive and coherent summarization.  _CoRR_ , abs/1906.03741, 2019.  URL http://arxiv.org/abs/1906.03741.
  * Foundation [2022] Wikimedia Foundation.  Wikimedia downloads, 2022.  URL https://dumps.wikimedia.org.
  * Wadden et al. [2020] David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi.  Fact or fiction: Verifying scientific claims.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7534–7550, 2020.

##  Appendix A Appendix

| Accuracy [%]
---|---
Task |  jina-small-v2 |  jina-base-v2
AmazonCounterfactualClassification |  71.36 |  74.73
AmazonPolarityClassification |  82.9 |  88.54
AmazonReviewsClassification |  40.89 |  45.26
Banking77Classification |  78.25 |  84.01
EmotionClassification |  44.01 |  48.77
ImdbClassification |  73.64 |  79.44
MassiveIntentClassification |  67.61 |  71.93
MassiveScenarioClassification |  69.75 |  74.49
MTOPDomainClassification |  93.96 |  95.68
MTOPIntentClassification |  72.5 |  83.15
ToxicConversationsClassification |  71.54 |  73.35
TweetSentimentExtractionClassification |  59.4 |  62.06
Avg |  68.82 |  73.45
Table 4: Detailed Performance on the MTEB Classification Tasks |  $\mathcal{V}$ measure
---|---
Task |  jina-small-v2 |  jina-base-v2
ArxivClusteringP2P |  44.02 |  45.39
ArxivClusteringS2S |  35.16 |  36.68
BiorxivClusteringP2P |  35.57 |  37.05
BiorxivClusteringS2S |  29.07 |  30.16
MedrxivClusteringP2P |  31.86 |  32.41
MedrxivClusteringS2S |  27.51 |  28.09
RedditClustering |  49.28 |  53.05
RedditClusteringP2P |  57.09 |  60.31
StackExchangeClustering |  55.35 |  58.52
StackExchangeClusteringP2P |  34.42 |  34.96
TwentyNewsgroupsClustering |  41.57 |  42.47
Avg |  40.08 |  41.73
Table 5: Detailed Performance on the MTEB Clustering Tasks | Spearman correlation based on $\cos$ similarity
---|---
Task |  jina-small-v2 |  jina-base-v2
SummEval |  30.56 |  31.6
Table 6: Detailed Performance on the MTEB Summarization Tasks |  $\cos$-sim-ap
---|---
Task |  jina-small-v2 |  jina-base-v2
SprintDuplicateQuestions |  95.12 |  95.30
TwitterSemEval2015 |  72.15 |  74.74
TwitterURLCorpus |  86.05 |  86.09
Avg |  84.44 |  85.38
Table 7: Detailed Performance on the MTEB Pair Classification Tasks | mAP@10
---|---
Task |  jina-small-v2 |  jina-base-v2
AskUbuntuDupQuestions |  59.62 |  62.25
MindSmallReranking |  30.99 |  30.54
SciDocsRR |  79.76 |  83.10
StackOverflowDupQuestions |  49.99 |  52.05
Avg |  55.09 |  56.98
Table 8: Detailed Performance on the MTEB ReRanking Tasks | nDCG@10
---|---
Task |  jina-small-v2 |  jina-base-v2
ArguAna |  46.73 |  44.18
ClimateFEVER |  20.05 |  23.53
CQADupstackRetrieval |  38.03 |  39.34
DBPedia |  32.65 |  35.05
FEVER |  68.02 |  72.33
FiQA2018 |  33.43 |  41.58
HotpotQA |  56.48 |  61.38
MSMARCO |  37.28 |  40.92
NFCorpus |  30.40 |  32.45
NQ |  51.59 |  60.44
QuoraRetrieval |  87.19 |  88.20
SCIDOCS |  18.61 |  19.86
SciFact |  63.89 |  66.68
Touche2020 |  23.52 |  26.24
TRECCOVID |  65.18 |  65.91
Avg |  45.14 |  47.87
Table 9: Detailed Performance on the MTEB Retrieval Tasks | Spearman correlation based on cosine similarity
---|---
Task |  jina-small-v2 |  jina-base-v2
BIOSSES |  80.52 |  81.23
SICK-R |  76.72 |  79.65
STS12 |  73.66 |  74.27
STS13 |  83.30 |  84.18
STS14 |  79.17 |  78.81
STS15 |  87.30 |  87.55
STS16 |  83.61 |  85.35
STS17(en-en) |  88.23 |  88.88
STS22(en) |  63.46 |  62.20
STSBenchmark |  84.04 |  84.84
Avg |  80.00 |  80.70
Table 10: Detailed Performance on the MTEB STS Tasks


# Joint Optimization of Active and Passive Beamforming in Multi-IRS Aided
mmWave Communications
††thanks: This work was supported in part by the NSFC under Grant 62001071,
62101460 and 62061007, the Macao Young Scholars Program under Grant AM2021018,
the China Postdoctoral Science Foundation under Grant 2020M683291, and the
Science and Technology Research Program of Chongqing Municipal Education
Commission under Grant KJQN202200617. The work of S. Ma was supported in part
by the Science and Technology Development Fund, Macau SAR under Grants
0051/2022/A1, 0036/2019/A1 and SKL-IOTSC (UM)-2021-2023, and in part by the
Research Committee of University of Macau under Grant MYRG2020-00095-FST. The
work of Y. Xu was supported in part by the NSFC under Grant 62271094 and the
Scientific and Technological Research Key Program of Chongqing Municipal
Education Commission under Grant KJZD-K202200601. (Corresponding author: Qing
Xue.)

Renlong Wei  School of Communication and Information Engineering, Chongqing
University of Posts and Telecommunications, Chongqing 400065, China Qing Xue
School of Communication and Information Engineering, Chongqing University of
Posts and Telecommunications, Chongqing 400065, China State Key Laboratory of
Internet of Things for Smart City, University of Macau, Macao SAR, China
Shaodan Ma  State Key Laboratory of Internet of Things for Smart City,
University of Macau, Macao SAR, China Department of Electrical and Computer
Engineering, University of Macau, Macao SAR, China Yongjun Xu  School of
Communication and Information Engineering, Chongqing University of Posts and
Telecommunications, Chongqing 400065, China Li Yan  Key Lab of Information
Coding & Transmission, Southwest Jiaotong University, Chengdu 610031, China
E-mails<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Xuming Fang  Key Lab of Information Coding &
Transmission, Southwest Jiaotong University, Chengdu 610031, China
E-mails<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Intelligent reflecting surface (IRS) has been considered as a promising
technology to alleviate the blockage effect and enhance coverage in millimeter
wave (mmWave) communication. To explore the impact of IRS on the performance
of mmWave communication, we investigate a multi-IRS assisted mmWave
communication network and formulate a sum rate maximization problem by jointly
optimizing the active and passive beamforming and the set of IRSs for
assistance. The optimization problem is intractable due to the lack of
convexity of the objective function and the binary nature of the IRS selection
variables. To tackle the complex non-convex problem, an alternating iterative
approach is proposed. In particular, utilizing the fractional programming
method to optimize the active and passive beamforming and the optimization of
IRS selection is solved by enumerating. Simulation results demonstrate the
performance gain of our proposed approach.

###### Index Terms:

Intelligent reflecting surface, millimeter wave communication, beamforming,
fractional programming.

##  I Introduction

As one of the key technologies in 5G and beyond systems, millimeter wave
(mmWave) has huge spectrum resource and can support extremely high data rates
[1] However, mmWave signals suffer high propagation path loss and are
susceptible to blockage due to the short wavelength. To compensate the high
path loss, massive antenna array technology is widely adopted, but it fails to
mitigate the blockage problem [2]. Using ultra-dense network is an effective
way to alleviate blockages, but it involves more serious interference and
power consumption than traditional networking [3], [4].

Recently, intelligent reflecting surface (IRS), which is composed of many low-
cost reflecting elements, has attracted much attention. By altering the phase
shifts or/and amplitudes, each element of IRS can reflect incident signal to
receiver independently [5], [6]. Deploying IRS in wireless communication
system can intelligently reconfigure wireless environment and increase the
received signal strength by controlling the direction of the reflected signal.
This provides an alternative approach to tackle the problems in mmWave system
with low-cost and energy-efﬁcient.

Many studies on IRS aided wireless communications [7, 8, 9, 10, 11, 12] have
been conducted since IRS has numerous potential benefits. In the IRS-aided
wireless systems, the effective beamforming is the key to achieve the goal of
system performance optimization. However, compared with traditional
beamforming, the beamforming problem in IRS-aided system is more difficult to
solve because it involves two parts: the active beamforming at base station
(BS) and the passive beamforming at IRS. The performance of the two parts is
closely related to each other. Many efforts have been dedicated to the
research of the beamforming in IRS assisted system. For example, the authors
of [8] investigated the transmit power minimization by jointly optimizing the
active and passive beamforming. In [11], the weighted sum-rate maximization
problem is studied and the active and passive beamforming are solved by
quadratic transform method and quadratically constrained quadratic program
(QCQP) respectively. However, only one IRS is considered in most of these
works. Due to the severe path loss and the unique channel characteristics of
mmWave communications, it is very necessary to deploy multiple IRSs discretely
to ensure the performance of mmWave systems. Meanwhile, to improve the system
coverage, multiple IRSs are usually deployed far away from each other. In this
case, the assistance effect of lRSs deployed in different locations usually
varies significantly. Hence, considering the optimal IRS selection for
different user is necessary [13]. In this work, when deal with the beamforming
optimization problem, we jointly consider the optimal IRS selection in the
multi-IRS aided mmWave system. After introducing multi-IRS and the IRS
selection in the mmWave system, the joint beamforming becomes more complex and
challenging, so an effective joint optimization method needs to be redesigned.

Motivated by the above, to guarantee the robustness of the multi-user downlink
transmissions, we consider a multi-IRS assisted mmWave communication system
and mainly focus on the transmit beamforming at mmWave BS (mBS) and passive
beamforming at the IRS and jointly study the IRS selection problem. In order
to maximize the system throughput, we formulate a sum rate optimization
problem. The formulated problem is non-convex and complex due to the binary
selection variables and the high coupling of all optimization variables. To
tackle this problem, we propose an effective alternating optimization method.
Specifically, fractional programming is employed to optimize the active and
passive beamforming, and enumeration algorithm is used for the assisted IRS
selection.

The remainder of this paper is organized as follows. System model of multi-IRS
assisted mmWave system and the corresponding optimization problem are
described in Section II. Section III presents the method to solve the
formulated optimization problem. Simulation results are provided in Section IV
to verify the performance of the proposed method. Finally, Section V concludes
the paper.

##  II System Model and Problem Formulation

###  II-A System Model

A multi-IRS aided multi-user downlink mmWave multiple-input single-output
(MISO) system is considered, as shown in Fig. 1, where $N$ IRSs are deployed
around the mBS to assist in communication from mBS to $K$ single-antenna
users. The mBS is equipped with an uniform linear array (ULA), consisting of
$M$ antennas, while each IRS is modeled as an uniform planar array (UPA) with
$L={L_{v}}\\!\times\\!{L_{h}}$ reflecting elements, where ${L_{v}}$ and
${L_{h}}$ denote the number of reflecting elements in the vertical and
horizontal directions, respectively. The index sets of IRSs, users and IRS
elements are defined as ${\cal
N}\buildrel\Delta\over{=}\left\\{{1,2,...,N}\right\\}$, ${\cal
K}\buildrel\Delta\over{=}\left\\{{1,2,...,K}\right\\}$ and ${\cal
L}\buildrel\Delta\over{=}\left\\{{1,2,...,L}\right\\}$, respectively. Let
$\boldsymbol{\Theta}_{n}=\operatorname{diag}\left(\theta_{n,1},\theta_{n,2},...,\theta_{n,L}\right)\in\mathbb{C}^{L\times
L}$ denote the reflection coefficient matrix of IRS $n$, where
${\theta_{n,l}}=\sqrt{{\eta_{n,l}}}{e^{j{\varphi_{n,l}}}}$,
$\sqrt{{\eta_{n,l}}}\in\left[{0,1}\right]$ and
${\varphi_{n,l}}\in\left[{0,2\pi}\right]$ denote the amplitude and phase shift
associated with $l$-th element of IRS $n$, respectively. Since IRS is designed
to maximize the reflected signal gain, we assume that
$\sqrt{{\eta_{n,l}}}=1,n\in{\cal N},\forall l\in{\cal L}$, as in [8], [11],
[14]. Due to the high path loss of mmWave, we ignore the signals that
reflected by IRSs twice and more [7]. For simplicity, we assume that the
channel state information (CSI) is fully known at mBS [11], [14]. It should be
emphasized that the availability of perfect CSI in the IRS aided systems is a
challenging task. But it is out of the scope of this paper. And the uplink is
our future work.

Let ${\boldsymbol{h}_{k}^{H}}\in{\mathbb{C}^{1\times M}}$, $k\in{\cal K}$
denote the mmWave channel from mBS to user $k$,
${\boldsymbol{H}_{n}}\in{\mathbb{C}^{L\times M}}$, $n\in{\cal N}$ denote the
mmWave channel from mBS to IRS $n$, and
${\boldsymbol{h}_{n,k}^{H}}\in{\mathbb{C}^{1\times L}}$, $n\in{\cal N}$,
$k\in{\cal K}$ denote the channel from IRS $n$ to user $k$, respectively.
According to the Saleh-Valenzuela channel model [15], [16],
${\boldsymbol{h}_{k}}$ is expressed as

$\boldsymbol{h}_{k}^{H}=\sqrt{\frac{M}{N_{B-U}}}\sum_{l=1}^{N_{B-U}}\rho_{l}\boldsymbol{a}_{t}\left(\phi_{l}^{\mathrm{AOD}}\right),$
(1)

where $N_{B-U}$ is the number of paths between the mBS and user $k$,
$\rho_{l}$ is the complex gain of the $l$-th path, $\phi_{l}^{\mathrm{AOD}}$
is the associated angle of departure, and $\boldsymbol{a}_{t}$ denotes the
normalized array response vector at transmitter.

$\boldsymbol{a}_{t}\left({\phi_{l}^{\mathrm{AOD}}}\right)=\frac{1}{{\sqrt{M}}}\left[{1,{e^{j\pi\sin\phi_{l}^{\mathrm{AOD}}}},...,{e^{j\pi\left({M-1}\right)\sin\phi_{l}^{\mathrm{AOD}}}}}\right].$
(2)

Figure 1: A multi-IRS aided mmWave MISO system.

Meanwhile, the channel between mBS and IRS $n$ is given as

$\boldsymbol{H}_{n}=\sqrt{\frac{ML}{N_{B-I}}}\left(\sum_{l=1}^{N_{B-I}}\zeta_{l}\boldsymbol{a}_{r}\left(\vartheta_{al}^{\mathrm{AOA}},\vartheta_{el}^{\mathrm{AOA}}\right)\boldsymbol{a}_{t}^{H}\left(\phi_{l}^{\mathrm{AOD}}\right)\right),$
(3)

where $N_{B-I}$ is the number of paths between the mBS and IRS $n$, $l=1$
denotes the line-of-sight (LOS) path, $l>1$ denotes the non-line-of-sight
(NLOS) path, $\zeta_{l}$ is the complex gain associated with the $l$-th path,
$\vartheta_{al}^{\mathrm{AOA}}$($\vartheta_{el}^{\mathrm{AOA}}$) represents
the azimuth (elevation) angle of arrival of the $l$-th path,
$\boldsymbol{a}_{r}$($\boldsymbol{a}_{t}$) is the normalized array response
vector at the receiver (transmitter) as

$\begin{array}[]{l}{{\bf{a}}_{r}}\left({\vartheta_{al}^{{\rm{AOA}}},\vartheta_{el}^{{\rm{AOA}}}}\right)=\frac{1}{{\sqrt{L}}}\left[{1,{e^{j\pi\left({\sin\vartheta_{al}^{{\rm{AOA}}}\sin\vartheta_{el}^{{\rm{AOA}}}+\cos\vartheta_{el}^{{\rm{AOA}}}}\right)}},}\right.\\\
\ldots,{\left.{{e^{j\pi\left({\left({{L_{v}}-1}\right)\sin\vartheta_{al}^{{\rm{AOA}}}\sin\vartheta_{el}^{{\rm{AOA}}}+\left({{L_{h}}-1}\right)\cos\vartheta_{el}^{{\rm{AOA}}}}\right)}}}\right]^{T}}.\end{array}$
(4)

Moreover, the channel from IRS $n$ to user $k$ can be expressed as

$\boldsymbol{h}_{n,k}^{H}=\sqrt{\frac{L}{{{N_{I-k}}}}}\left({\sum\limits_{l=1}^{{N_{I-k}}}{{\varrho_{l}}}{{\bf{a}}_{t}}\left({\vartheta_{al}^{\mathrm{AOD}},\vartheta_{el}^{\mathrm{AOD}}}\right)}\right),$
(5)

where $N_{I-k}$ is the number of path from IRS $n$ to user $k$, $\varrho_{l}$
is the complex gain associated with the $l$-th path, and
$\vartheta_{al}^{\mathrm{AOD}}$($\vartheta_{el}^{\mathrm{AOD}}$) is the
azimuth (elevation) angle of departure associated with the $l$-th path.

Let $\boldsymbol{x}=\sum\limits_{k=1}^{K}{{\boldsymbol{p}_{k}}{s_{k}}}$ denote
the transmitted signal at mBS, where $s_{k}$ is the transmit symbol for user
$k$ and $\boldsymbol{p}_{k}\in{\mathbb{C}^{M\times 1}}$ is the corresponding
beamforming vector. In general, the received signal of user $k$ can be
expressed as

$y_{k}=\left(\boldsymbol{h}_{k}^{H}+\sum_{n\in\mathcal{N}}\boldsymbol{h}_{n,k}^{H}\boldsymbol{\Theta}_{n}^{H}\boldsymbol{H}_{n}\right)\sum_{i\in\mathcal{K}}\boldsymbol{p}_{i}s_{i}+n_{k},$
(6)

where $n_{k}$ is the complex additive white Gaussian noise, ${n_{k}}\sim{\cal
C}{\cal N}\left({0,{\sigma^{2}}}\right)$.

In this paper, we assume that the direct links from the mBS to users are
severely obstacled by buildings or other things. Meanwhile, to simplify the
design of the reflection matrix at IRS, we assume that each IRS only serves
maximal one user at a moment, and it serves multiple users in a time-division
manner. For the IRS selection, we define the binary variables
${a_{n,k}}\\!\in\\!\left\\{{0,1}\right\\},n\in{\cal N},k\in{\cal K}$, where
${a_{n,k}}=1$ indicates that user $k$ is served by IRS $n$, otherwise we have
${a_{n,k}}=0$. As such, for IRS $n$, we have $\sum\limits_{k\in{\cal
K}}{{a_{n,k}}=1},\forall n\in{\cal N}$, and the IRS selection matrix
$\boldsymbol{A}\in\mathbb{C}^{N\times K}$ can be written as

$\boldsymbol{A}=\left[{\begin{array}[]{*{20}{c}}{{a_{1,1}}}&\cdots&{{a_{1,K}}}\\\
\vdots&\ddots&\vdots\\\ {{a_{N,1}}}&\cdots&{{a_{N,K}}}\end{array}}\right].$
(7)

###  II-B Problem Formulation

The signal-to-interference-plus-noise ratio (SINR) of user $k$ is given by

$\operatorname{SINR}_{k}=\frac{\left|\sum_{n\in\mathcal{N}}a_{n,k}\left(\boldsymbol{h}_{n,k}^{H}\boldsymbol{\Theta}_{n}^{H}\boldsymbol{H}_{n}\right)\boldsymbol{p}_{k}\right|^{2}}{\sum_{\begin{subarray}{c}i\in\mathcal{K}\\\
i\neq
k\end{subarray}}\left|\left(\sum_{n\in\mathcal{N}}a_{n,k}\left(\boldsymbol{h}_{n,k}^{H}\boldsymbol{\Theta}_{n}^{H}\boldsymbol{H}_{n}\right)\right)\boldsymbol{p}_{i}\right|^{2}+\sigma^{2}}.$
(8)

Thereby, the achievable rate (bps/Hz) for user $k$ can be expressed as

$R_{k}=\log_{2}\left(1+\operatorname{SINR}_{k}\right).$ (9)

In this paper, we aim to maxmize the sum rate of all users by jointly
optimizing active and passive beamforming, subject to the constraint of SINR
at each user and the constraint of IRS selection. Then the problem can be
formulated as

$\displaystyle(\mathrm{P1})\max_{\boldsymbol{P},\boldsymbol{\Theta}_{n},\boldsymbol{A}}$
$\displaystyle~{}f_{1}\left(\boldsymbol{P},\boldsymbol{\Theta}_{n},\boldsymbol{A}\right)=\sum_{k\in\mathcal{K}}R_{k},$
(10) $\displaystyle{\rm s.t.}~{}$
$\displaystyle\sum_{k\in\mathcal{K}}\left\|\boldsymbol{p}_{k}\right\|^{2}\leq
P_{\max},$ (10a) $\displaystyle\left|\theta_{n,l}\right|^{2}=1,\forall
n\in{\cal N},\forall l\in{\cal L},$ (10b)
$\displaystyle\operatorname{SINR}_{k}\geq\operatorname{SINR}_{\min},\forall
k\in{\cal K},$ (10c) $\displaystyle\sum_{k\in\mathcal{K}}a_{n,k}=1,\forall
n\in{\cal N},$ (10d) $\displaystyle a_{n,k}\in\\{0,1\\},\forall n\in{\cal
N},\forall k\in{\cal K},$ (10e)

where
$\boldsymbol{P}={\left[{{\boldsymbol{p}_{1}}^{T},{\boldsymbol{p}_{2}}^{T},\cdots,{\boldsymbol{p}_{K}}^{T}}\right]^{T}}$,
constraint (10a) limits the total transmit power of the mBS, constraint (10b)
is the reflection coefficient constraint for passive beamforming, and
constraint (10c) denotes the minimum SINR for reliable communincation. Problem
$(\mathrm{P1})$ is a complex non-convex problem due to the non-convex
objective function (10) and the non-convex constraints (10b)-(10e). In next
section, we will explore how to effectively solve this problem.

##  III Problem Solution

In this section, we propose an alternative optimization method to solve
peoblem $(\mathrm{P1})$. Firstly, an equivalent problem of $(\mathrm{P1})$ is
established by employing the Lagrangian dual transform proposed in [17]. Then,
we decouple it into three sub-optimization problems, where the active and
passive beamfroming are optimized by fractional programming and the IRS
selection is optimized via enumeration algorithm.

###  III-A Equivalent Transformation of (P1)

We utilize the Lagrangian dual transform [17] to obtain an equivalent problem
of $(\mathrm{P1})$. By introducing an auxiliary variable
$\boldsymbol{\alpha}=\left[\alpha_{1},\alpha_{2},\cdots,\alpha_{K}\right]^{T}$,
the objective function (10) can be equivalently expressed as

$\displaystyle
f_{1a}\left(\boldsymbol{P},\boldsymbol{\Theta}_{n},\boldsymbol{A},\boldsymbol{\alpha}\right)$
(11)
$\displaystyle=\sum_{k\in\mathcal{K}}\log_{2}\left(1+\alpha_{k}\right)-\sum_{k\in\mathcal{K}}\frac{\alpha_{k}}{\ln
2}+\sum_{k\in\mathcal{K}}\frac{\left(1+\alpha_{k}\right)\mathrm{SINR}_{k}}{\left(1+\mathrm{SINR}_{k}\right)\ln
2}.$

Hence, problem $(\mathrm{P1})$ can be reformulated as

$\displaystyle(\mathrm{P1.1})\max_{\boldsymbol{P},\boldsymbol{\Theta}_{n},\boldsymbol{A},\boldsymbol{\alpha}}$
$\displaystyle~{}f_{1a}\left(\boldsymbol{P},\boldsymbol{\Theta}_{n},\boldsymbol{A},\boldsymbol{\alpha}\right),$
(12) $\displaystyle{\rm s.t.}~{}$
$\displaystyle(10\mathrm{a})-(10\mathrm{e}).$ (12a)

In problem $(\mathrm{P}1.1)$, when $\boldsymbol{P}$, $\boldsymbol{\Theta}_{n}$
and $\boldsymbol{A}$ are fixed, the optimal $\alpha_{k}$, denoted by
$\alpha_{k}^{opt}$, can be obtained by setting
${{\partial{f_{1a}}}\mathord{\left/{\vphantom{{\partial{f_{2}}}{\partial{\alpha_{k}}}}}\right.\kern-1.2pt}{\partial{\alpha_{k}}}}$
to zero. We have $\alpha_{k}^{opt}=\mathrm{SINR}_{k}$.

Note that given $\alpha_{k}$, only the last term of $f_{1a}$ is connected with
$\boldsymbol{P}$, $\boldsymbol{\Theta}_{n}$ and $\boldsymbol{A}$.
Consequently, the original problem can be further simplified as

$\displaystyle(\mathrm{P1.2})\max_{\boldsymbol{P},\boldsymbol{\Theta}_{n},\boldsymbol{A}}$
$\displaystyle~{}f_{1b}\left(\boldsymbol{P},\boldsymbol{\Theta}_{n},\boldsymbol{A}\right)=\sum_{k\in\mathcal{K}}\frac{\left(1+\alpha_{k}\right)\mathrm{SINR}_{k}}{\left(1+\mathrm{SINR}_{k}\right)},$
(13) $\displaystyle{\rm s.t.}~{}$
$\displaystyle(10\mathrm{a})-(10\mathrm{e}).$ (13a)

The above problem $(\mathrm{P1.2})$ can be further solved as shown in
subsection III-B, III-C and III-D.

###  III-B Active Beamforming Optimization at mBS

To facilitate the description, we define
$\hat{\boldsymbol{h}}_{n,k}^{H}=\boldsymbol{h}_{n,k}^{H}\boldsymbol{\Theta}_{n}^{H}\boldsymbol{H}_{n}$.
Substituting $\hat{\boldsymbol{h}}_{n,k}^{H}$ and (8) into (13), we rewrite
$f_{1b}$ with given $\boldsymbol{\alpha}$, $\boldsymbol{\Theta}_{n}$ and
$\boldsymbol{A}$ as

$\displaystyle f_{2}(\boldsymbol{P})$
$\displaystyle=\sum_{k\in\mathcal{K}}\frac{\left(1+\alpha_{k}\right)\operatorname{SINR}_{k}}{\left(1+\operatorname{SINR}_{k}\right)}$
(14)
$\displaystyle=\sum_{k\in\mathcal{K}}\frac{\left(1+\alpha_{k}\right)\left|\left(\sum_{n\in\mathcal{N}}a_{n,k}\hat{\boldsymbol{h}}_{n,k}^{H}\right)\boldsymbol{p}_{k}\right|^{2}}{\sum_{i\in\mathcal{K}}\left|\left(\sum_{n\in\mathcal{N}}a_{n,k}\hat{\boldsymbol{h}}_{n,k}^{H}\right)\boldsymbol{p}_{i}\right|^{2}+\sigma^{2}}.$

Thereby, the problem of optimizing $\boldsymbol{P}$ can be expressed as

$\displaystyle(\mathrm{P2})\max_{\boldsymbol{P}}$
$\displaystyle~{}f_{2}(\boldsymbol{P}),$ (15) $\displaystyle{\rm s.t.}~{}$
$\displaystyle(10\mathrm{a}),(10\mathrm{c}).$ (15a)

We see that $f_{2}$ is in a sum-of-ratio form, which can be tackled by
applying quadratic transform [17] as follows

$\displaystyle(\mathrm{P2.1})\max_{\boldsymbol{P},\boldsymbol{\varepsilon}}$
$\displaystyle~{}f_{2a}(\boldsymbol{P},\boldsymbol{\varepsilon}),$ (16)
$\displaystyle{\rm s.t.}~{}$ $\displaystyle(10\mathrm{a}),(10\mathrm{c}),$
(16a)

where
$\boldsymbol{\varepsilon}=\left[\mathcal{\varepsilon}_{1},\mathcal{\varepsilon}_{2},\cdots,\mathcal{\varepsilon}_{K}\right]^{T}$
is an auxiliary variable and the objective function
$f_{2a}(\boldsymbol{P},\boldsymbol{\varepsilon})$ is denoted by

$\displaystyle f_{2a}(\boldsymbol{P},\boldsymbol{\varepsilon})=$
$\displaystyle\sum_{k\in\mathcal{K}}2\sqrt{\left(1+\alpha_{k}\right)}\mathfrak{R}\left\\{\varepsilon_{k}^{*}\boldsymbol{H}_{k}^{H}\boldsymbol{p}_{k}\right\\}$
(17)
$\displaystyle-\sum_{k\in\mathcal{K}}\left|\varepsilon_{k}\right|^{2}\left(\sum_{i\in\mathcal{K}}\left|\boldsymbol{H}_{k}^{H}\boldsymbol{p}_{i}\right|^{2}+\sigma^{2}\right),$

and
$\boldsymbol{H}_{k}^{H}=\sum_{n\in\mathcal{N}}a_{n,k}\hat{\boldsymbol{h}}_{n,k}^{H}$.

Given $\boldsymbol{P}$, the optimal $\mathcal{\varepsilon}_{k}$ is obtained by
setting ${{\partial
f_{2a}}\mathord{\left/{\vphantom{{\partial{f_{5}}}{\partial{\varepsilon_{k}}}}}\right.\kern-1.2pt}{\partial{\varepsilon_{k}}}}$
to zero as

$\varepsilon_{k}^{opt}=\frac{\sqrt{\left(1+\alpha_{k}\right)}\boldsymbol{H}_{k}^{H}\boldsymbol{p}_{k}}{\left(\sum_{i\in\mathcal{K}}\left|\boldsymbol{H}_{k}^{H}\boldsymbol{p}_{i}\right|^{2}+\sigma^{2}\right)}.$
(18)

The following problem is optimizing $\boldsymbol{P}$ with given
$\boldsymbol{\varepsilon}$. Let
$\mathbf{v}_{k}=\sqrt{\left(1+\alpha_{k}\right)}\varepsilon_{k}\boldsymbol{H}_{k}$,
$\mathbf{z}=\sum_{k\in\mathcal{K}}\left|\varepsilon_{k}\right|^{2}\boldsymbol{H}_{k}\boldsymbol{H}_{k}^{H}$
and $g=\sum\limits_{k\in{\cal
K}}{{{\left|{{\varepsilon_{k}}}\right|}^{2}}{\sigma^{2}}}$. The $f_{2a}$ can
be simplified as

$f_{2b}(\boldsymbol{P})=-\boldsymbol{P}^{H}\mathbf{Z}\boldsymbol{P}+2\mathfrak{R}\left\\{\mathbf{V}^{H}\boldsymbol{P}\right\\}-g,$
(19)

where
$\mathbf{Z}=\operatorname{diag}\left({{\mathbf{z}_{1}},\cdots,{\mathbf{z}_{K}}}\right)$,
$\mathbf{z}_{k}=\mathbf{z},\forall k\in\mathcal{K}$ and
$\mathbf{V}={\left[{{\mathbf{v}_{1}^{T}},\cdots,{\mathbf{v}_{K}^{T}}}\right]^{T}}$.

Thereby, problem $(\mathrm{P2.1})$ can be further simplified as

$\displaystyle(\mathrm{P2.2})\max_{\boldsymbol{P}}$
$\displaystyle~{}f_{2b}(\boldsymbol{P})=-\boldsymbol{P}^{H}\mathbf{Z}\boldsymbol{P}+2\mathfrak{R}\left\\{\mathbf{V}^{H}\boldsymbol{P}\right\\}-g,$
(20) $\displaystyle{\rm s.t.}~{}$
$\displaystyle(10\mathrm{a}),(10\mathrm{c}).$ (20a)

Although the objective function $f_{2b}$ and constraint $(10\mathrm{a})$ are
convex, the problem $(\mathrm{P2.2})$ is non-convex due to the non-convex
constraint $(10\mathrm{c})$.

According to [18], by using second-order-cone programming (SOCP), we rewrite
constraint $(10\mathrm{c})$ as

$\sqrt{1+\frac{1}{{\operatorname{SINR}{{}_{\min}}}}}{\boldsymbol{H}_{k}^{H}}{\boldsymbol{p}_{k}}\geq\left\|{\begin{array}[]{*{20}{c}}{{\boldsymbol{H}_{k}^{H}}\boldsymbol{\bar{P}}}\\\
\sigma\end{array}}\right\|,$ (21)
${\boldsymbol{H}_{k}^{H}}{\boldsymbol{p}_{k}}\geq 0,\forall k\in{\cal K},$
(22)

where
$\boldsymbol{\bar{P}}={\left[{{\boldsymbol{p}_{1}},{\boldsymbol{p}_{2}},\cdots,{\boldsymbol{p}_{K}}}\right]}$.
Therefore, the problem of active beamforming optimization can be eventually
solved as following form

$\displaystyle(\mathrm{P2.3})\max_{\boldsymbol{P}}$
$\displaystyle~{}f_{2b}(\boldsymbol{P})=-\boldsymbol{P}^{H}\mathbf{Z}\boldsymbol{P}+2\mathfrak{R}\left\\{\mathbf{V}^{H}\boldsymbol{P}\right\\}-g,$
(23) $\displaystyle{\rm s.t.}~{}$ $\displaystyle(10\mathrm{a}),(21)\text{ and
}(22).$ (23a)

Problem $(\mathrm{P2.3})$ is convex, which can be solved by standard convex
optimization solvers such as CVX. Hence, the optimal $\boldsymbol{P}$ is
obtained by solving $(\mathrm{P2.3})$.

###  III-C Passive Beamforming Optimization at IRS

First of all, we rewrite
$\hat{\boldsymbol{h}}_{n,k}^{H}=\boldsymbol{h}_{n,k}^{H}\boldsymbol{\Theta}_{n}^{H}\boldsymbol{H}_{n}$
as

$\hat{\boldsymbol{h}}_{n,k}^{H}\boldsymbol{p}_{i}=\left(\boldsymbol{h}_{n,k}^{H}\boldsymbol{\Theta}_{n}^{H}\boldsymbol{H}_{n}\right)\boldsymbol{p}_{i}=\boldsymbol{\theta}_{n}^{H}\operatorname{diag}\left(\boldsymbol{h}_{n,k}^{H}\right)\boldsymbol{H}_{n}\boldsymbol{p}_{i},$
(24)

where
$\boldsymbol{\theta}_{n}=\left[\theta_{n,1},\cdots,\theta_{n,L}\right]^{T}$.

Let
$\boldsymbol{b}_{n,k,i}=\operatorname{diag}\left(\boldsymbol{h}_{n,k}^{H}\right)\boldsymbol{H}_{n}\boldsymbol{p}_{i}$,
then substituting $\boldsymbol{\theta}_{n}$ and $\boldsymbol{b}_{n,k,i}$ into
(14), $f_{2}$ is rewritten as

$f_{3}\left(\boldsymbol{\theta}_{n}\right)=\sum_{k\in\mathcal{K}}\frac{\left(1+\alpha_{k}\right)\left|\sum_{n\in\mathcal{N}}a_{n,k}\boldsymbol{\theta}_{n}^{H}\boldsymbol{b}_{n,k,k}\right|^{2}}{\sum_{i\in\mathcal{K}}\left|\sum_{n\in\mathcal{N}}a_{n,k}\boldsymbol{\theta}_{n}^{H}\boldsymbol{b}_{n,k,i}\right|^{2}+\sigma^{2}}.$
(25)

We define
$\boldsymbol{\Theta}=\left[\boldsymbol{\theta}_{1}^{T},\cdots,\boldsymbol{\theta}_{N}^{T}\right]^{T}$
and
$\boldsymbol{B}_{k,i}=\left[a_{1,k}\boldsymbol{b}_{1,k,i}^{T},\cdots,a_{N,k}\boldsymbol{b}_{N,k,i}^{T}\right]^{T}$.
Thereby, $f_{3}$ is equivalently transformed to a new function of
$\boldsymbol{\Theta}$

$f_{3a}(\boldsymbol{\Theta})=\sum_{k\in\mathcal{K}}\frac{\left(1+\alpha_{k}\right)\left|\boldsymbol{\Theta}^{H}\boldsymbol{B}_{k,k}\right|^{2}}{\sum_{i\in\mathcal{K}}\left|\boldsymbol{\Theta}^{H}\boldsymbol{B}_{k,i}\right|^{2}+\sigma^{2}}.$
(26)

Consequently, given $\boldsymbol{\alpha}$, $\boldsymbol{P}$ and
$\boldsymbol{A}$, the passive beamforming optimization problem is expressed as

$\displaystyle(\mathrm{P3})\max_{\boldsymbol{\Theta}}$
$\displaystyle~{}f_{3a}(\boldsymbol{\Theta}),$ (27) $\displaystyle{\rm
s.t.}~{}$ $\displaystyle(10\mathrm{b}),(10\mathrm{c}).$ (27a)

Similarly, introducing auxiliary variable
$\boldsymbol{\beta}=\left[\beta_{1},\cdots,\beta_{K}\right]^{T}$, $f_{3a}$ is
transformed as

$\displaystyle f_{3b}(\boldsymbol{\Theta},\boldsymbol{\beta})=$
$\displaystyle\sum_{k\in\mathcal{K}}2\sqrt{\left(1+\alpha_{k}\right)}\mathfrak{R}\left\\{\beta_{k}^{*}\boldsymbol{\Theta}^{H}\boldsymbol{B}_{k,k}\right\\}$
(28)
$\displaystyle-\sum_{k\in\mathcal{K}}\left|\beta_{k}\right|^{2}\left(\sum_{i\in\mathcal{K}}\left|\boldsymbol{\Theta}^{H}\boldsymbol{B}_{k,i}\right|^{2}+\sigma^{2}\right).$

Let ${{\partial f_{3b}}\mathord{\left/{\vphantom{{\partial
f_{3b}}{\partial{\beta_{k}}}}}\right.\kern-1.2pt}{\partial{\beta_{k}}}}$ to
zero, the optimal $\beta_{k}$ is obtained as

$\beta_{k}^{opt}=\frac{\sqrt{\left(1+\alpha_{k}\right)}\left(\boldsymbol{\Theta}^{H}\boldsymbol{B}_{k,k}\right)}{\sum_{i\in\mathcal{K}}\left|\boldsymbol{\Theta}^{H}\boldsymbol{B}_{k,i}\right|^{2}+\sigma^{2}}.$
(29)

Given $\boldsymbol{\beta}$, the optimization of $\boldsymbol{\Theta}$ can be
represented as follows

$\displaystyle(\mathrm{P3.1})\max_{\boldsymbol{\Theta}}$
$\displaystyle~{}f_{3c}(\boldsymbol{\Theta})=-\boldsymbol{\Theta}^{H}\boldsymbol{U}\boldsymbol{\Theta}+2\mathfrak{R}\left\\{\boldsymbol{\Theta}^{H}\boldsymbol{D}\right\\}-c,$
(30) $\displaystyle{\rm s.t.}~{}$
$\displaystyle(10\mathrm{b}),(10\mathrm{c}),$ (30a)

where $\boldsymbol{U}=\sum\limits_{k\in{\cal
K}}{{{\left|{{\beta_{k}}}\right|}^{2}}\left({\sum\limits_{i\in{\cal
K}}{{\boldsymbol{B}_{k,i}}\boldsymbol{B}_{k,i}^{H}}}\right)}$,
$c=\sum\limits_{k\in{\cal K}}{{{\left|{{\beta_{k}}}\right|}^{2}}{\sigma^{2}}}$
and
$\boldsymbol{D}=\sum_{k\in\mathcal{K}}\sqrt{\left(1+\alpha_{k}\right)}\beta_{k}^{*}\boldsymbol{B}_{k,k}$.

Since the constraints $(10\mathrm{b})$ and $(10\mathrm{c})$ are non-convex,
the problem $(\mathrm{P3.1})$ is non-convex. For constraint $(10\mathrm{b})$,
it is an unit-modulus constraint. We relax constraint $(10\mathrm{b})$, i.e.
$\left|\theta_{n,l}\right|^{2}\leq 1,\forall n\in{\cal N},\forall l\in{\cal
L}$. For constraint $(10\mathrm{c})$, similar to (21) and (22), we handle it
with SOCP method. Then the new constraint is denoted by

$\sqrt{\left({1+\frac{1}{{\operatorname{SINR}{{}_{\min}}}}}\right)}{\boldsymbol{\Theta}^{H}}{\boldsymbol{B}_{k,k}}\geq\left\|{\begin{array}[]{*{20}{c}}{{\boldsymbol{\Theta}^{H}}\boldsymbol{B}_{k}}\\\
\sigma\end{array}}\right\|,$ (31)
${\boldsymbol{\Theta}^{H}}{\boldsymbol{B}_{k,k}}\geq 0,\forall k\in{\cal K},$
(32)

where
$\boldsymbol{B}_{k}={\left[{{\boldsymbol{B}_{k,1}},{\boldsymbol{B}_{k,2}},\cdots,{\boldsymbol{B}_{k,K}}}\right]}$.

Finally, the passive beamforming optimization is reformulated as

$\displaystyle(\mathrm{P3.2})\max_{\boldsymbol{\Theta}}$
$\displaystyle~{}f_{3c}(\boldsymbol{\Theta})=-\boldsymbol{\Theta}^{H}\boldsymbol{U}\boldsymbol{\Theta}+2\mathfrak{R}\left\\{\boldsymbol{\Theta}^{H}\boldsymbol{D}\right\\}-c,$
(33) $\displaystyle{\rm s.t.}~{}$
$\displaystyle\left|\theta_{n,l}\right|^{2}\leq 1,\forall n\in{\cal N},\forall
l\in{\cal L},$ (33a) $\displaystyle(31),(32),$ (33b)

which is convex and can be solve by CVX.

###  III-D IRS selection Optimization

Given $\boldsymbol{\alpha}$, $\boldsymbol{P}$ and $\boldsymbol{\Theta}$, the
IRS selection optimization problem can be expressed as

$\displaystyle(\mathrm{P4})\max_{\boldsymbol{A}}$
$\displaystyle~{}f_{4}\left(\boldsymbol{A}\right)=\sum_{k\in\mathcal{K}}R_{k},$
(34) $\displaystyle{\rm s.t.}~{}$
$\displaystyle(10\mathrm{d}),(10\mathrm{e}).$ (34a)

Due to $a_{n,k}\in\\{0,1\\},\forall n\in{\cal N},\forall k\in{\cal K}$, and
the dimension of matrix $\boldsymbol{A}$ will not be very large, we can obtain
the optimal $\boldsymbol{A}$ by enumerating all possible combinations between
IRSs and users. In the worst case, the complexity of enumeration is
$\mathcal{O}\left(K^{N}\right)$.

Finally, we update $\boldsymbol{\alpha}$, $\boldsymbol{P}$,
$\boldsymbol{\Theta}$ and $\boldsymbol{A}$ in an alternate manner, until
problem $(\mathrm{P1})$ reaches a stable optimal solution.

##  IV Simulation and Discussions

To verify the performance of the proposed method, we consider a simulation
scenario that the mBS is located at the origin, $N=6$ IRSs are located at
$(60{\rm{m}},40{\rm{m}})$, $(60{\rm{m}},-40{\rm{m}})$,
$(100{\rm{m}},40{\rm{m}})$, $(100{\rm{m}},-40{\rm{m}})$,
$(140{\rm{m}},40{\rm{m}})$ and $(140{\rm{m}},-40{\rm{m}})$, respectively. The
users are randomly distributed in a circle at $(100{\rm{m}},0{\rm{m}})$ with a
radius of $10{\rm{m}}$. The number of antenna at mBS $M=40$. According to [14]
and [15], the channel gain is generated as
$\rho_{l}\sim\mathcal{CN}\left(0,10^{-0.1\kappa}\right)$ and
$\kappa=a+10b\log_{10}(d)+\xi$, where
$\xi\sim\mathcal{N}\left(0,\sigma_{\xi}^{2}\right)$. For LOS path, the values
of $a$, $b$ and $\sigma_{\xi}$ are respectively setting as $a=61.4$, $b=2$ and
$\sigma_{\xi}=5.8$ dB. For NLOS path, the values of $a$, $b$ and
$\sigma_{\xi}$ are setting as $a=72$, $b=2.92$ and $\sigma_{\xi}=8.7$ dB,
respectively. The generation of channel complex gains $\zeta_{l}$ and
$\varrho_{l}$ are similar to $\rho_{l}$. And other parameters are set as
follows: $N_{B-I}=5$, $N_{I-k}=1$, ${P_{\max}}=30$ dBm,
${\rm{SIN}}{{\rm{R}}_{\min}}=-20$ dB and ${\sigma^{2}}=-90$ dBm.

Figure 2: Convergence performance of the proposed method.

Figure 3: The sum rate versus ${P_{\max}}$ with $N=4$, $K=4$.

In order to evaluate the proposed scheme, we compare it with the following
three methods: 1) WIS: without consider the IRS selection; 2) RPS: a random
passive beamforming is used at IRS; 3) NIS: select the nearest IRS for
service.

To show the convergence of the proposed method, we plot the sum rate against
the number of iterations under different number of IRSs and users in Fig. 2.
We can see that with the increase of the number of IRSs $N$ and the number of
users $K$, the convergence speed becomes slower but the sum rate obtains
significant improvement. For example, when $N=K=4$, the optimal solution is
obtained within 10 iterations and the sum rate of convergence is about 9
bps/Hz, while when $N=6$ and $K=4$, the optimal solution needs about 25
iterations to converge and the corresponding rate can up to about 12 bps/Hz.
Besides, we notice that when $N$ is fixed, increasing $K$ has little effect on
the sum rate. But when $K$ is fixed, increasing $N$ will greatly improve the
rate performance. The reason is that we introduced the IRS selection in the
proposed method, so that the number of IRSs largely determines the number of
users that the system can serve simultaneously.

Figure 4: The sum rate versus $L$ with $N=4$, $K=4$.

Fig. 3 shows the impact of the maximum transmit power on the sum rate
performance. This figure shows that the sum rate increases with the transmit
power and the proposed method outperforms all other methods except the “WIS”.
For example, when $P_{\max}=50$ dBm, the sum rate of the proposed method can
reach about 27 bps/Hz, which is about 3 bps/Hz higher than the “NIS”.
Comparing the proposed method with the “WIS”, we observe that when
$P_{\max}\leq 30$ dBm, the performance gap between them is very small but
gradually becomes larger when $P_{\max}$ exceeds $30$ dBm. The main reason is
that, when the value of $P_{\max}$ is small, the reflected signals received by
the users far away from the IRS will be very weak and contribute little to the
gain of the sum rate. Although the IRS selection results in a slight
performance degradation compared to the “WIS”, the IRS selection simplifies
the design of the IRS phase shift matrix while providing higher data rate for
the user served.

The impact of the number of reflecting elements at each IRS is shown in Fig.
4. In general, the sum rate of the “WIS”, the “Proposed Method” and the “NIS”
increases with the number of reflecting elements $L$. And the sum rate of the
other two methods barely increases when the reflecting unit of IRS is
increased. The reason is that the more reflective elements, the stronger the
reflected signals, and the more effective the passive beamforming will be, so
that the system throughput can be improved.

##  V Conclusions

To explore utilizing IRS to tackle the problem of seriously path loss and
blockage of mmWave, a multi-IRS assisted mmWave communication system is
considered and we investigate a sum rate maximization problem by jointly
optimizing active and passive beamforming as well as the optimal IRS
selection. To tackle the challenging problem, an alternating iterative
approach is proposed, where the transmit beamforming of the mBS, the passive
beamforming of the IRS and the IRS selection matrix are updated in an
alternating manner. The simulation results show the feasibility and
effectiveness of the proposed method.

## References

  * [1] X. Wu, S. Ma, and X. Yang, “Tensor-based low-complexity channel estimation for mmwave massive MIMO-OTFS systems,” _Journal of Communications and Information Networks_ , vol. 5, no. 3, pp. 324–334, 2020.
  * [2] S. A. Busari, K. M. S. Huq, S. Mumtaz, L. Dai, and J. Rodriguez, “Millimeter-wave massive MIMO communication for future wireless systems: A survey,” _IEEE Communications Surveys $\&$ Tutorials_, vol. 20, no. 2, pp. 836–869, 2018.
  * [3] M. Kamel, W. Hamouda, and A. Youssef, “Ultra-dense networks: A survey,” _IEEE Communications Surveys $\&$ Tutorials_, vol. 18, no. 4, pp. 2522–2545, 2016.
  * [4] Q. Xue, Y. Sun, J. Wang, G. Feng, L. Yan, and S. Ma, “User-centric association in ultra-dense mmWave networks via deep reinforcement learning,” _IEEE Commun. Lett._ , vol. 25, no. 11, pp. 3594–3598, 2021.
  * [5] C. Pan, H. Ren, K. Wang, J. F. Kolb, M. Elkashlan, M. Chen, M. Di Renzo, Y. Hao, J. Wang, A. L. Swindlehurst, X. You, and L. Hanzo, “Reconfigurable intelligent surfaces for 6G systems: Principles, applications, and research directions,” _IEEE Communications Magazine_ , vol. 59, no. 6, pp. 14–20, 2021.
  * [6] Y.-C. Liang, R. Long, Q. Zhang, J. Chen, H. V. Cheng, and H. Guo, “Large intelligent surface/antennas (LISA): Making reflective radios smart,” _Journal of Communications and Information Networks_ , vol. 4, no. 2, pp. 40–50, 2019.
  * [7] Z. Zhang and L. Dai, “A joint precoding framework for wideband reconfigurable intelligent surface-aided cell-free network,” _IEEE Transactions on Signal Processing_ , vol. 69, pp. 4085–4101, 2021.
  * [8] Q. Wu and R. Zhang, “Intelligent reflecting surface enhanced wireless network via joint active and passive beamforming,” _IEEE Transactions on Wireless Communications_ , vol. 18, no. 11, pp. 5394–5409, 2019.
  * [9] K. Xu, J. Zhang, X. Yang, S. Ma, and G. Yang, “On the sum-rate of RIS-assisted MIMO multiple-access channels over spatially correlated rician fading,” _IEEE Transactions on Communications_ , vol. 69, no. 12, pp. 8228–8241, 2021.
  * [10] J. Zhang, J. Liu, S. Ma, C.-K. Wen, and S. Jin, “Large system achievable rate analysis of RIS-assisted MIMO wireless communication with statistical CSIT,” _IEEE Transactions on Wireless Communications_ , vol. 20, no. 9, pp. 5572–5585, 2021.
  * [11] H. Guo, Y.-C. Liang, J. Chen, and E. G. Larsson, “Weighted sum-rate maximization for intelligent reflecting surface enhanced wireless networks,” in _2019 IEEE GLOBECOM_ , 2019, pp. 1–6.
  * [12] S. Gong, S. Ma, Z. Yang, C. Xing, and J. An, “Throughput maximization for asynchronous RIS-aided hybrid powered communication networks,” _IEEE Trans. Wireless Commun._ , vol. 21, no. 6, pp. 4114–4132, 2022.
  * [13] H. Hashida, Y. Kawamoto, N. Kato, M. Iwabuchi, and T. Murakami, “Mobility-aware user association strategy for IRS-aided mm-Wave multibeam transmission towards 6G,” _IEEE Journal on Selected Areas in Communications_ , vol. 40, no. 5, pp. 1667–1678, 2022.
  * [14] D. Zhao, H. Lu, Y. Wang, and H. Sun, “Joint passive beamforming and user association optimization for IRS-assisted mmwave systems,” in _2020 GLOBECOM_ , 2020, pp. 1–6.
  * [15] P. Wang, J. Fang, X. Yuan, Z. Chen, and H. Li, “Intelligent reflecting surface-assisted millimeter wave communications: Joint active and passive precoding design,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 12, pp. 14 960–14 973, 2020.
  * [16] Y. Cao, T. Lv, and W. Ni, “Intelligent reflecting surface aided multi-user mmwave communications for coverage enhancement,” in _2020 IEEE 31st Annual International Symposium on Personal, Indoor and Mobile Radio Communications_ , 2020, pp. 1–6.
  * [17] K. Shen and W. Yu, “Fractional programming for communication systems part I: Power control and beamforming,” _IEEE Transactions on Signal Processing_ , vol. 66, no. 10, pp. 2616–2630, 2018.
  * [18] H. Xie, J. Xu, and Y.-F. Liu, “Max-min fairness in IRS-aided multi-cell MISO systems via joint transmit and reflective beamforming,” in _2020 ICC_ , 2020, pp. 1–6.