# Self-MI: Efficient Multimodal Fusion via Self-Supervised Multi-Task Learning
with Auxiliary Mutual Information Maximization

Cam-Van Thi Nguyen, Ngoc-Hoa Thi Nguyen,
Duc-Trong Le, Quang-Thuy Ha
Faculty of Information Technology
VNU - University of Engineering and Technology, Hanoi, Vietnam

###### Abstract

Multimodal representation learning poses significant challenges in capturing
informative and distinct features from multiple modalities. Existing methods
often struggle to exploit the unique characteristics of each modality due to
unified multimodal annotations. In this study, we propose Self-MI in the self-
supervised learning fashion, which also leverage Contrastive Predictive Coding
(CPC) as an auxiliary technique to maximize the Mutual Information (MI)
between unimodal input pairs and the multimodal fusion result with unimodal
inputs. Moreover, we design a label generation module, $ULG_{MI}$ for short,
that enables us to create meaningful and informative labels for each modality
in a self-supervised manner. By maximizing the Mutual Information, we
encourage better alignment between the multimodal fusion and the individual
modalities, facilitating improved multimodal fusion. Extensive experiments on
three benchmark datasets including CMU-MOSI, CMU-MOSEI, and SIMS, demonstrate
the effectiveness of Self-MI in enhancing the multimodal fusion task.

##  1 Introduction

Multimodal sentiment analysis (MSA) has emerged as a prominent research area,
garnering significant attention in recent years Bagher Zadeh et al. (2018);
Poria et al. (2023). It demonstrates remarkable enhancements compared to the
traditional unimodal analysis. Real-life multimodal data typically involves
three distinct channels: visual, acoustic, and textual. Effective
representation and information fusion from these diverse sources are essential
components in MSA.

In multimodal learning, Baltrušaitis et al. (2018) highlight five fundamental
challenges including alignment, translation, representation, fusion, and co-
learning. Among these challenges, representation learning is a significant and
challenging task. The goal is to obtain effective representations that capture
both the commonalities and distinctions across different modalities. An
effective representation should include two key aspects: consistency and
differentiation Poria et al. (2020). The representations should be consistent
across modalities to enable seamless integration and analysis of information
from various sources. Equally important, it should also capture the unique
characteristics of each modality to facilitate a comprehensive understanding
of the data.

Yu et al. (2021) point out the shortcomings of current methods in capturing
distinctive information due to their reliance on unified multimodal
annotation. However, incorporating additional unimodal annotations can be
burdensome in terms of time and human resource. As a solution for this
challenge, Yu et al. (2021) introduce a novel approach named Self-MM. The main
idea is to incorporate a measure based on the distance between modality
representations and the class centroids, which correlates positively with the
model’s output. In order to generate unimodal labels, the authors implement a
self-supervised learning strategy, of which the absolute distance calculation
between two modality representations in different spaces is intractable.
Dealing with this problem, Han et al. (2021) propose MultiModal InfoMax
(MMIM), which maximize the shared information in multimodal fusion via
enhancing two types of mutual information: between unimodal representations
and between fusion embeddings and unimodal representations.

Building upon the insights from MMIM Han et al. (2021) and addressing the
limitations of Self-MM Yu et al. (2021), this paper presents a novel approach
named Self-MI that computes the similarity between modality representations
using the maximize mutual information technique, while also incorporating
self-supervised learning for the MSA task. Additionally, designing a multitask
learning model using the hard parameter sharing strategy allows us to gain a
comprehensive understanding of how unimodal tasks impact multimodal tasks. To
be clear, our contributions can be summarized as the following:

  * •

We propose an unified model called Self-MI, which is based on Self-supervised
learning and Multi-task Learning with the auxiliary of Mutual Information
(MI).

  * •

We design a module named Mutual Information Maximization for Unimodal Labels
Generation ($ULG_{MI}$) that utilizes the Contrastive Predictive Coding (CPC)
method to estimate the correlation between multimodal representations.

  * •

We perform extensive experiments on three benchmark datasets including CMU-
MOSI, CMU-MOSEI, and CMU-MOSEI. Our model results in superior or comparable
results compared to state-of-the-art models.

##  2 Related Works

In this section, we literately review several classes of previous research
works related to the multimodal sentiment analysis task and our proposed model
Self-MI.

###  2.1 Multimodal Sentiment Analysis

Multimodal Sentiment Analysis (MSA) is a comprehensive approach that combines
verbal and non-verbal features to perform user sentiment analysis. This field
focuses on extracting emotions, interpretations, and feelings by analyzing
various sources such as language, facial expressions, speech, music, and
movements Kaur and Kautish (2022). The field of multimodal sentiment analysis
commonly utilizes various public datasets such as CMU-MOSI Zadeh et al.
(2016), CMU-MOSEI Bagher Zadeh et al. (2018), CH-SIMS Yu et al. (2020),
IEMOCAP Busso et al. (2008), and several others.

MSA research could be divided into four groups: 1) early multimodal fusion
methods like Tensor Fusion Network TFN Zadeh et al. (2017), Low-rank
Multimodal Fusion LMF Liu et al. (2018), and Multimodal Factorization Model
MFM Tsai et al. (2019b), and 2) the methods that fuse multimodality through
modeling modality interaction, such as multimodal Transformer MulT Tsai et al.
(2019a) and modal-temporal attention graph MTAG Yang et al. (2021) and 3) the
methods focusing on the consistency and the difference of modality, in which
MISA Hazarika et al. (2020), Self-MM Yu et al. (2021), and MMIM Han et al.
(2021). Our work focuses on the third group of methods, in line with previous
studies Yu et al. (2021); Han et al. (2021). We also use these models as the
baseline models in this study. We provide detailed descriptions of these
models in the corresponding Section 4.

###  2.2 Self-Supervised Multimodal Learning

Multimodal learning, which seeks to comprehend and analyze information from
various modalities, has made significant advancements in the supervised regime
in recent years. However, the reliance on paired data and costly human
annotations hinders the scalability of models. On the other hand, with the
abundance of unannotated data available in the wild, self-supervised learning
has emerged as an appealing strategy to address the annotation bottleneck Zong
et al. (2023). The objective functions for training self-supervised multimodal
algorithms can be categorized into three main types: instance discrimination,
clustering, and masked prediction. In this study, our model belongs to the
category of instance discrimination, specifically contrastive learning. By
leveraging the ULGM module in Self-MM Yu et al. (2021), we re-design this
module and name it $ULG_{MI}$ (Mutual Information Maximization for Unimodal
Labels Generation). The purpose of this module is to generate uni-modal
supervision values using multimodal annotations and modality representations.
In contrast to Self-MM Yu et al. (2021), we replace the Relative Distance
Value, which evaluates the relative distance from the modality representation
to the positive center and the negative center, with the computation of Mutual
Information Maximization, utilizing the auxiliary of Contrastive Predictive
Coding (CPC) Oord et al. (2018a).

###  2.3 Multi-task Learning

Multi-task learning aims to enhance the generalization performance of multiple
related tasks by leveraging the knowledge from different tasks Zhang and Yang
(2021). In contrast to single-task learning, multi-task learning faces two
primary challenges during the training stage. The first challenge is how to
effectively share network parameters, which can be achieved through hard-
sharing or soft-sharing methods. The second challenge is to balance the
learning process across different tasks. Multi-task learning has been widely
applied in multimodal sentiment analysis (MSA) Akhtar et al. (2019). In this
work, we propose the incorporation of unimodal subtasks to assist in the
process of learning modality-specific representations.

###  2.4 Mutual Information

In probability theory and information theory, Mutual Information (MI) is a
measure of the dependence between two random variables. It quantifies how much
knowing the value of one variable reduces uncertainty about the value of the
other variable. In the context of machine learning, maximizing mutual
information can help to learn informative representations and improve the
performance of tasks such as classification and generation.

$I(X;Y)=\mathbb{E}_{p(x,y)}[log\frac{p(x,y)}{p(x)p(y)}]$ (1)

Alemi et al. (2016) Alemi et al. (2016) were the pioneers in integrating
optimization techniques related to mutual information into deep learning
models. Since then, numerous studies Amjad and Geiger (2019); He et al. (2020)
have investigated and demonstrated the advantages of maximizing mutual
information principles in various contexts. However, estimating mutual
information directly in high-dimensional spaces is often considered
impractical or challenging.

##  3 Methodology

In this section, we present the overall architecture of Self-MI as well as its
mathematical formulation for the multimodal sentiment analysis task.

Figure 1: The overall architecture of our propose model Self-MI

###  3.1 Task setup

A MSA model takes unimodal raw sequences $X_{m}\in\mathbb{R}^{l_{m}\times
d_{m}}$ from the same video fragment as input, where $l_{m}$ is the sequence
length, and $d_{m}$ is the dimension of the representation vector for modality
$m$. Here, we consider three modalities denoted as $m\in\\{t,a,v\\}$,
representing text, visual, and acoustic modalities respectively. The main
objective of Self-MI is to integrate information from these input vectors, and
create a unified representation to accurately predict the sentiment intensity
reflected via the truth value $y$.

The overall architecture of Self-MI is illustrated in Figure 1. The model
comprises one multimodal task and three separate unimodal subtasks. Between
the multimodal task and the unimodal tasks, we exploit the MI (Mutual
Information) maximization with Contrastive Predictive Coding (CPC) as the
fusion module in our model.

###  3.2 Multimodal Representation

The multimodal sentiment analysis task is tackled as a classification model,
which consists of three key components: (1) The feature representation module,
(2) The feature fusion module, and (3) The classification module.

For text modality, the sequence is denoted as
$T=\\{w_{C},w_{0},\dots,w_{i},\dots,w_{S}\\}$, where $w_{C}$ and $w_{S}$
represent the special tokens [CLS] and [SEP] respectively. The pre-trained
12-layers BERT model is utilized to extract the global text representation and
the local text representation. Empirically, the first-word vector in the last
layer is identified as the optimal representation for the entire sentence
denoted as $X_{t}$:

$X^{t}=BERT(T;\theta_{t}^{bert})$ (2)

where $X^{t}=\\{x_{C}^{t},x_{0}^{t},\dots,h_{i}^{t},\dots,h^{t}_{S}\\}$,
$X^{t}\in\mathbb{R}^{l^{t}\times d^{t}}$; $l^{t}$ is the max length of the
text sequence; and $d_{t}$ is the dimension of the text representation.

For audio and visual modality, we use a pre-trained ToolKits to extract the
initial vector features, i.e., A and V. Inspired by previous works Hazarika et
al. (2020); Yu et al. (2021), a single directional Long Short-Term Memory
(sLSTM) model Hochreiter and Schmidhuber (1997) is employed to capture
temporal characteristics. The final step adopts the end-state hidden vectors
as the comprehensive sequence representations:

$X^{a}=sLSTM(A;\theta_{a}^{lstm})\in R^{d_{a}}$ (3)
$X^{v}=sLSTM(V;\theta_{v}^{lstm})\in R^{d_{v}}$ (4)

where $X^{a}=\\{x_{0}^{a},\dots,h_{j}^{a},\dots,h^{a}_{l_{a}}\\}$,
$X^{a}\in\mathbb{R}^{l^{a}\times d^{a}}$,
$X^{v}=\\{x_{0}^{v},\dots,h_{k}^{v},\dots,h^{v}_{l_{v}}\\}$,
$X^{v}\in\mathbb{R}^{l^{v}\times d^{v}}$; $l^{a}$ and $l^{v}$ are the sequence
length of audio and vision, respectively.

Subsequently, all uni-modal representations are concatenated and projected
into a lower-dimensional space $\mathbb{R}^{d_{m}}$.

$X^{m}=[X^{t},X^{a},X^{v}]$ (5)
$Z^{m}=ReLU({W_{l_{1}}^{m}}^{T}X^{m}+b_{l_{1}}^{m})$ (6)

where ${W_{l_{1}}^{m}}^{T}\in R^{(d_{t}+d_{a}+d_{v})\times d_{m}}$ and $ReLU$
is the relu activation function.

Finally, the fusion representation $Z_{m}$ is utilized to generate the
multimodal sentiment prediction as:

$\hat{y}_{m}={W_{l_{2}}^{m}}^{T}Z^{m}+b_{l_{2}}^{m}$ (7)

where ${W_{l_{2}}^{m}}^{T}\in R^{d_{m}\times 1}$.

###  3.3 Unimodal Task

The three unimodal tasks leverage the same modality representations as the
multimodal task. To reduce the dimensionality gap between different
modalities, we project the representations into a new feature space. Then,
linear regression is used to obtain the unimodal results as follows:

$\displaystyle Z^{s}$
$\displaystyle=ReLU({W_{l_{1}}^{s}}^{T}X^{s}+b_{l_{1}}^{s})$ (8)
$\displaystyle\hat{y}_{s}$
$\displaystyle={W_{l_{2}}^{s}}^{T}Z^{s}+b_{l_{2}}^{s}$ (9)

where $s\in\\{t,a,v\\}$. In order to guide the unimodal task’s training
process, we design a Unimodal Label Generation with Mutual Information Module
($ULG_{MI}$) to get unimodal labels. Section 3.4 delves into the specifics of
the $ULG_{MI}$:

${y}_{s}=ULG_{MI}(y_{m},Z^{m},Z^{s})$ (10)

Ultimately, the multimodal task and three unimodal tasks are jointly learned
using human-annotated multimodal labels, i.e., m-labels, and auto-generated
unimodal labels, i.e., u-labels. It should be emphasized that these unimodal
tasks only exist during the training stage. Consequently, $y^{m}$ is used as
the final output.

###  3.4 Mutual Information Maximization for Unimodel Labels Generation -
$ULG_{MI}$

Since there exits a generation path from $X^{s}$ to $Z^{m}$, we expect an
oposite path to construct $X^{s}$ with $s\in\\{t,a,v\\}$. The objective is to
create unimodal supervised data representations based on human-labeled
multimodal data and modality-specific representations.

Unimodal supervised data representations have a strong correlation with
multimodal data labels, hence, the Contrastive Predictive Coding (CPC) method,
a widely used unsupervised approach for high-dimensional data, is applied to
estimate the correlation between the multimodal representation. Oord et al.
(2018b) use CPC to measure the mutual information between contextual
information and future elements across a time horizon. In other words, CPC
seeks to capture the “slow features” that extend over multiple time steps,
ensuring their retention in the encoding process. CPC uses a contrastive loss
function based on estimated contrastive noise, called InfoNCE (Normalized
Mutual Information-based Contrastive Estimation). The InfoNCE loss function is
designed to measure the similarity between pairs of data samples by
calculating the mutual information between them. Specifically, InfoNCE
leverages the concepts of entropy and mutual information from information
theory to compute the distance between pairs of data samples. Therefore, when
using InfoNCE, we are essentially estimating the shared information and
maximizing it using gradient descent. Inspired by this idea, Self-MI
implements $Z^{m}$ to reversely predict representation across modalities so
that more modality-invariant information can be passed to $Z^{m}$.

As depicted in Figure 1, after extracting unimodal representations and
multimodal representations, we seek to maximize the correlation between the
two modality’s representations via CPC. The CPC score function is defined as
follows:

$\displaystyle\overline{G_{\phi}(Z^{m})}$
$\displaystyle=\frac{G_{\phi}(Z^{m})}{\parallel G_{\phi}(Z^{m})\parallel^{2}}$
(11) $\displaystyle\overline{Z^{s}}$ $\displaystyle=\frac{Z^{s}}{\parallel
Z^{s}\parallel^{2}}$ (12)

We utilizes Euclidean normalization to compute the unit-length vectors.
Specifically, $G_{\phi}$ represents a simple neural network with parameters
$\phi$.

$s(Z^{m},Z^{s})=\exp(\overline{Z^{s}}(\overline{G_{\phi}(Z^{m}}))^{T})$ (13)

$\mathcal{L}_{N}(Z^{m},Z^{s})=\mathbb{E}_{F}[log\frac{s(Z^{m},Z^{s})}{\sum_{j}^{N}s(Z^{m},Z^{s})}]$
(14)

Here, $\mathcal{L}_{N}$ representing the CPC loss function between two vectors
$Z^{m}$ and $Z^{s}$, where $s\in\\{t,a,v\\}$, and $N$ being the number of
samples in a batch, the sample index is the sum of elements within the same
batch, excluding the current element. From this, we can derive the following
relationship:

$\mathcal{L}_{CPC}=\mathcal{L}_{N}^{mt}+\mathcal{L}_{N}^{ma}+\mathcal{L}_{N}^{mv}$
(15)

The CPC output will be passed through a regression multilayer perceptron (MLP)
to generate the unimodal label $y^{s}$. Along with the human-generated
multimodal label $ym$, we can define the loss function as follows:

$\mathcal{L}_{task}=MAE({y_{s}},y_{m})$ (16)

The unimodal labels update policy is shown in Algorithm 1.

###  3.5 Optimization Objectives

Finally, we utilize L1Loss as a fundamental optimization method. For unimodal
tasks, we employ the discrepancy between the unimodal label u and the
multimodal label m as the weighting factor for the loss function. This
indicates that the system should pay more attention to samples with
significant differences.

$L=\frac{1}{N}\sum_{i}^{N}(|\hat{y_{m}^{i}}-y_{m}^{i}|+\sum_{s}^{\\{t,a,v\\}}W_{s}^{i}\cdot|\hat{y_{s}^{i}}-y_{s}^{i}|)$
(17)

where $N$ representing the number of samples used for training,
$W_{s}^{i}=\tanh(|y_{s}-y_{m}|)$ denotes the weight of the $i^{th}$ sample
supporting task $s$.

Algorithm 1 $ULG_{MI}$

Input: unimodal input $X_{t},X_{a},X_{v}$, m-labels $y_{m}$

Output: u-label $y_{t}^{(i)},y_{a}^{(i)},y_{v}^{(i)}$ where $i$ is the number
of training epochs

Initialize model parameters $M(\theta;x)$

Initialize u-labels $y_{t}^{(1)}=y_{m},y_{a}^{(1)}=y_{m},y_{v}^{(1)}=y_{m}$

Initialize global representations
$F^{g}_{t}=0,F^{g}_{a}=0,F^{g}_{v}=0,F^{g}_{m}=0$

for $n\in[1,end]$ do

for mini-batch in dataLoader do

Compute mini-batch modality representations $Z^{t},Z^{a},Z^{v},Z^{m}$

Compute loss $\mathcal{L}$ using Equation 16, 17

Compute parameters gradient $\frac{\mathnormal{v}L}{\mathnormal{v}\theta}$

Update model parameters:
$\theta=\theta-\mathnormal{n}\frac{\mathnormal{v}L}{\mathnormal{v}\theta}$

if $n\neq 1$ then

Compute CPC loss $\mathcal{L}_{CPC}$ using Eq. 14 and Eq.15.

Compute $y_{t},y_{a},y_{v}$ from mutual information learning from CPC model +
MLP.

end if

Update global representations $F^{g}_{s}$ using $Z^{s}$, where
$s\in\\{m,t,a,v\\}$

end for

end for

##  4 Experimental Settings

In this section, we present the details of extensive experiments, which
investigate the advantages of Self-MI compared to state-of-the-art baselines
in term of evaluation metrics.

###  4.1 Dataset

In this study, we utilize two publicly available multimodal sentiment analysis
datasets: CMU-MOSI Zadeh et al. (2016), and SIMS Yu et al. (2020). Table 1
provides an overview of the basic statistics for each dataset.

CMU-MOSI: The CMU-MOSI dataset Zadeh et al. (2016), is widely recognized as
one of the prominent benchmark datasets for Multimodal Sentiment Analysis
(MSA). Each sample in the dataset is annotated by human annotators with a
sentiment score ranging from -3 (strongly negative sentiment) to 3 (strongly
positive sentiment).

MOSEI: The CMU-MOSEI dataset Bagher Zadeh et al. (2018) provides a larger
number of utterances, increased sample variety, speakers, and topics compared
to CMU-MOSI. Similarly to MOSI, annotators in the CMU-MOSEI dataset label each
sample with a sentiment score ranging from -3 (strongly negative) to 3
(strongly positive).

SIMS: The SIMS dataset Yu et al. (2020) is a unique benchmark dataset for
Chinese Multimodal Sentiment Analysis that offers fine-grained annotations of
modality. Each sample in the dataset is annotated by human annotators with a
sentiment score ranging from -1 (strongly negative sentiment) to 1 (strongly
positive sentiment).

Table 1: Dataset statistics in CMU-MOSI, CMU-MOSEI, and SIMS.

Dataset | Train | Valid | Test | Total
---|---|---|---|---
MOSI | 1,284 | 229 | 686 | 2,199
SIMS | 1,368 | 456 | 457 | 2,281

###  4.2 Baselines

To thoroughly evaluate the performance of our model Self-MI, we conduct a
comprehensive comparison with state-of-the-art baselines for the MSA task as
follows:

  * •

Tensor Fusion Network Zadeh et al. (2017): TFN disentangles unimodal data into
tensors through a threefold Cartesian product and computes the outer product
of these tensors for fusion.

  * •

Low-rank Multimodal Fusion Liu et al. (2018): LMF decomposes stacked high-
order tensors into multiple low-rank factors and efficiently performs fusion
based on these factors.

  * •

Multimodal Transformer Tsai et al. (2019a): MulT constructs an architecture
with separate unimodal and crossmodal transformer networks and completes the
fusion process using attention mechanisms.

  * •

Modal-temporal Attention Graph Yang et al. (2021): MTAG is an interpretable
graph attention network model capable of both fusion and alignment.

  * •

MISA Hazarika et al. (2020): MISA projects features into separate two spaces
with specific constraints and performs fusion on these features.

  * •

Self-MM Yu et al. (2021): Self-MM assigns each modality a unimodal training
task with automatically generated labels, aiming to adjust the gradient
backpropagation.

  * •

MMIM Han et al. (2021): MMIM improves Multimodal Fusion with Hierarchical
Mutual Information Maximization.

###  4.3 Experimental Settings

The hyperparameters used in the experiments are described in Table 2.

Table 2: The details of Hyperparameter Setting

Hyper-parameters | CMU-MOSI | MOSEI | SIMS
---|---|---|---
batch size | 32 | 64 | 32
| learning rate
---
textual modality (BERT)
5e-4 | 1e-3 | 5e-3
BERT embedding size | 768 | 768 | 768
text dropout | 0.0 | 0.1 | 0.1
| learning rate
---
audio modality
0.001 | 0.001 | 0.005
audio embedding size | 16 | 16 | 16
audio dropout | 0.1 | 0.1 | 0.0
| learning rate
---
visual modality
1e-3 | 1e-3 | 0.005
visual embedding size | 16 | 64 | 16
visual dropout | 0.1 | 0.1 | 0.0
fusion dropout | 0.1 | 0.1 | 0.0
sentiment score range | -3 to 3 | -3 to 3 | -1 to 1

Table 3: Experimental results on CMU-MOSI and CMU-MOSEI dataset. For
F1$\\_$score and Acc$\\_$2, the “/” sign separates the value of “negative/non-
negative” or “negative/positive” performance. The $\\_$ sign is no
information. Equally important, the sign $\downarrow$ indicates “smaller is
better” while $\uparrow$ signifies “higher is better”. The bold style stands
for the best-performing model whilst the underline marks the second best.

Model | CMU-MOSI | MOSEI
---|---|---
MAE $\downarrow$ | Corr $\uparrow$ | Acc$\\_$2 $\uparrow$ | F1_score $\uparrow$ | MAE $\downarrow$ | Corr $\uparrow$ | Acc$\\_$2 $\uparrow$ | F1_score $\uparrow$
TFN | 0.901 | 0.698 | _/80.8 | _/80.7 | 0.593 | 0.677 | -/82.5 | -/82.1
LMF | 0.917 | 0.695 | _/82.5 | _/82.4 | 0.623 | 0.700 | -/82.0 | -/82.1
MFM | 0.877 | 0.706 | -/81.7 | -/81.6 | 0.568 | 0.703 | -/84.4 | -/84.3
MTAG | 0.866 | 0.722 | -/82.3 | -/82.1 | - | - | - | -
MulT | 0.861 | 0.711 | 81.50/84.10 | 80.60/83.90 | 0.580 | 0.713 | -/82.5 | -/82.3
MISA | 0.804 | 0.764 | 80.79/82.10 | 80.77/82.03 | 0.568 | 0.717 | 82.59/84.23 |  82.67/83.97
Self-MM |  0.718 |  0.791 | 82.56/84.48 | 82.46/84.44 | 0.530 | 0.765 | 82.81/85.17 | 82.53/85.30
MMIM | 0.72 | 0.78 |  82.8/84.76 |  82.63/84.66 | 0.526 |  0.772 | 82.24/85.97 | 82.66/85.94
Ours (Self-MI) | 0.699 | 0.80 | 83.09/85.37 | 82.95/85.3 |  0.527 | 0.773 | 83.86/85.45 |  83.90/85.37

###  4.4 Evaluation Metrics

Followed by Han et al. (2021), we consider various evaluation metrics namely
mean absolute error (MAE), Pearson correlation (Corr), binary classification
accuracy (Acc_2), F1 score computed for positive/negative and non-
negative/negative classification.

##  5 Results & Discussion

In this section, we provide a detailed analysis and discussion on the
experimental results, which are obtained from Self-MI and baselines, on three
datasets: CMU-MOSI, CMU-MOSEI, and SIMS.

###  5.1 Comparison With Baselines

Table 3 shows the comparative results on CMU-MOSI and CMU-MOSEI datasets.
Generally, Self-MI significantly outperforms baselines across all evaluation
metrics on the CMU-MOSI dataset. For the CMU-MOSEI dataset, Self-MI
outperforms the SOTA models on Corr, Acc_2, and F1 score, while being
comparable in MAE. These results provide initial evidence of the effectiveness
of our approach in MSA tasks.

As the SIMS dataset consists of unaligned data, we compare Self-MI with TFN,
LMF, and Self-MM methods. The experimental results are presented in Table 4.
It is clear that Self-MI has a sustainable performance over all baselines.

Table 4: Experimental results on SIMS dataset. The sign $\downarrow$ indicates
“smaller is better” while $\uparrow$ signifies “higher is better”. The bold
style stands for the best-performing model whilst the underline marks the
second best.

Model | MAE $\downarrow$ | Corr $\uparrow$ | Acc$\\_$2 $\uparrow$ | F1_score $\uparrow$
---|---|---|---|---
TFN | 0.428 | 0.605 | 79.86 | 80.15
LMF | 0.431 | 0.600 | 79.37 | 78.65
Self-MM | 0.415 | 0.608 | 78.21 | 78.2
Ours (Self-MI) | 0.402 | 0.6138 |  79.65 |  79.82

###  5.2 Ablation Study

####  5.2.1 Effectiveness of unimodal subtasks

Clearly, Self-MI contributes significantly to the overall performance of the
model when examining the effectiveness of combining various unimodal tasks.
The experimental results under different ablation settings are presented in
Table 5. In comparing to the single-task model with the incorporation of
unimodal subtasks, we observe a notable improvement in performance.
Interestingly, the combinations “M, T” and “M, T, V” achieve comparable or
even better results than the combination “M, T, A, V”. Additionally, the
subtasks “T” and “A” seem to have a more positive impact on the overall
performance compared to the subtask “V”. These findings offer valuable
insights into the effectiveness of Self-MI in leveraging different unimodal
tasks to enhance multimodal learning.

Table 5: Perspective analysis results from multimodal data with different
tasks on CMU-MOSI dataset. M, T, A, V are multimodal, text, audio and visual
tasks, respectively. For F1$\\_$score and Acc$\\_$2, the “/” sign separates
the value of “negative/non-negative” or “negative/positive” performance.
Equally important, the sign $\downarrow$ indicates “smaller is better” while
$\uparrow$ signifies “higher is better”. The bold style stands for the best-
performing model.

Task | MAE $\downarrow$ | Corr $\uparrow$ | Acc$\\_$2 $\uparrow$ | F1_score $\uparrow$
---|---|---|---|---
M | 0.7155 | 0.796 | 81.67/82.13 | 81.55/83.08
M,T | 0.7217 | 0.7911 | 81.34/83.23 | 81.32/83.27
M,A | 0.6955 | 0.8027 | 82.67/82.98 | 83.59/85.96
M,V | 0.6908 | 0.8051 | 83.53/85.12 | 83.42/85.8
M,T,A | 0.7215 | 0.801 | 83.61/85.15 | 84.52/86.21
M,T,V | 0.7172 | 0.8012 | 84.49/85.93 | 84.51/85.01
M,V,A | 0.7121 | 0.7962 | 82.36/8384 | 82.35/83.88
M,T,V,A | 0.6999 | 0.8014 | 83.09/85.37 | 82.95/85.3

####  5.2.2 Effectiveness of CPC loss

To highlight the advantages of the CPC loss functions in Self-MI, we conduct a
series of ablation experiments on the CMU-MOSI dataset. The results are listed
in Table 6, where we eliminated one of these loss terms
($\mathcal{L}^{ma}_{N},\mathcal{L}^{mv}_{N},\mathcal{L}^{mt}_{N}$) from the
total CPC loss. We observe the similar phenomena on other datasets.

Table 6: Results of CPC on CMU-MOSI. Symbol t, v, a, m represent text, images,
audio and video. For F1$\\_$score and Acc$\\_$2, the “/” sign separates the
value of “negative/non-negative” or “negative/positive” performance. Equally
important, the sign $\downarrow$ indicates “smaller is better” while
$\uparrow$ signifies “higher is better”. The bold style stands for the best-
performing model.

Loss | MAE $\downarrow$ | Corr $\uparrow$ | Acc$\\_$2 $\uparrow$ | F1$\\_$score $\uparrow$
---|---|---|---|---
w/o $\mathcal{L}_{N}^{mt}$ | 0.6957 | 0.8016 | 83.53/85.67 | 83.41/85.62
w/o $\mathcal{L}_{N}^{ma}$ | 0.6924 | 0.8019 | 83.24/85.52 | 83.05/85.42
w/o $\mathcal{L}_{N}^{mv}$ | 0.7014 | 0.7982 | 81.78/83.69 | 81.4/83.71
| $\mathcal{L}_{CPC}$
---
$\mathcal{L}_{N}^{mt},\mathcal{L}_{N}^{ma},\mathcal{L}_{N}^{mv}$
0.6999 | 0.8014 | 83.09/85.37 | 82.95/85.30

From the initial CPC loss function, there exits a performance degradation
after removing a portion of the CPC loss. Specifically, when using
$\mathcal{L}_{CPC}$, Self-MI show a stable performance with a slightly higher
MAE score compared to using only $\mathcal{L}_{N}^{mt}$ or
$\mathcal{L}_{N}^{ma}$, with insignificant differences of 0.0042 and 0.0075,
respectively. Similarly, in the Acc_2 metric, when using $\mathcal{L}_{CPC}$,
the performance was slightly lower than when using only $\mathcal{L}_{N}^{mt}$
or $\mathcal{L}_{N}^{ma}$, with insignificant differences of 0.3 and 0.15,
respectively. This demonstrates the effectiveness of maximizing mutual
information as proposed in Self-MI model.

##  6 Conclusion

In this paper, we introduce Self-MI, a model designed to enhance efficient
multimodal representation. Our method includes a label generation module based
on self-supervised learning, which enables us to obtain independent unimodal
supervisions. Moreover, we maximize the Mutual Information (MI) between
unimodal input pairs and between the multimodal fusion result and unimodal
inputs, utilizing the auxiliary of Contrastive Predictive Coding (CPC). This
comprehensive approach allows for better representation learning and more
effective multimodal fusion in MSA task. We thoroughly evaluate the
performance of our model on three benchmark datasets, and our comprehensive
ablation study further validates the effectiveness of our proposed approach.
We anticipate that our work will serve as a source of inspiration for
advancing representation learning and multimodal sentiment analysis in future
research.

##  7 Acknowledgement

Cam-Van Nguyen Thi was funded by the Master, PhD Scholarship Programme of
Vingroup Innovation Foundation (VINIF), code VINIF.2022.TS143.

## References

  * Akhtar et al. (2019) Md Shad Akhtar, Dushyant Chauhan, Deepanway Ghosal, Soujanya Poria, Asif Ekbal, and Pushpak Bhattacharyya. 2019.  Multi-task learning for multi-modal emotion recognition and sentiment analysis.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 370–379, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Alemi et al. (2016) Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. 2016.  Deep variational information bottleneck.  _arXiv preprint arXiv:1612.00410_.
  * Amjad and Geiger (2019) Rana Ali Amjad and Bernhard C Geiger. 2019.  Learning representations for neural network-based classification using the information bottleneck principle.  _IEEE transactions on pattern analysis and machine intelligence_ , 42(9):2225–2239.
  * Bagher Zadeh et al. (2018) AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018.  Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2236–2246, Melbourne, Australia. Association for Computational Linguistics.
  * Baltrušaitis et al. (2018) Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018.  Multimodal machine learning: A survey and taxonomy.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(2):423–443.
  * Busso et al. (2008) Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008.  Iemocap: Interactive emotional dyadic motion capture database.  _Language resources and evaluation_ , 42:335–359.
  * Han et al. (2021) Wei Han, Hui Chen, and Soujanya Poria. 2021.  Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9180–9192, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Hazarika et al. (2020) Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. 2020.  Misa: Modality-invariant and-specific representations for multimodal sentiment analysis.  In _Proceedings of the 28th ACM international conference on multimedia_ , pages 1122–1131.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9729–9738.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Kaur and Kautish (2022) Ramandeep Kaur and Sandeep Kautish. 2022.  Multimodal sentiment analysis: A survey and comparison.  _Research Anthology on Implementing Sentiment Analysis Across Multiple Disciplines_ , pages 1846–1870.
  * Liu et al. (2018) Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. 2018.  Efficient low-rank multimodal fusion with modality-specific factors.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2247–2256, Melbourne, Australia. Association for Computational Linguistics.
  * Oord et al. (2018a) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018a.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_.
  * Oord et al. (2018b) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018b.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_.
  * Poria et al. (2020) Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, and Rada Mihalcea. 2020.  Beneath the tip of the iceberg: Current challenges and new directions in sentiment analysis research.  _IEEE Transactions on Affective Computing_.
  * Poria et al. (2023) Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, and Rada Mihalcea. 2023.  Beneath the tip of the iceberg: Current challenges and new directions in sentiment analysis research.  _IEEE Trans. Affect. Comput._ , 14(1):108–132.
  * Tsai et al. (2019a) Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019a.  Multimodal transformer for unaligned multimodal language sequences.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6558–6569, Florence, Italy. Association for Computational Linguistics.
  * Tsai et al. (2019b) Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019b.  Learning factorized multimodal representations.  In _ICLR_.
  * Yang et al. (2021) Jianing Yang, Yongxin Wang, Ruitao Yi, Yuying Zhu, Azaan Rehman, Amir Zadeh, Soujanya Poria, and Louis-Philippe Morency. 2021.  MTAG: Modal-temporal attention graph for unaligned human multimodal language sequences.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1009–1021, Online. Association for Computational Linguistics.
  * Yu et al. (2020) Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. 2020.  CH-SIMS: A Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3718–3727, Online. Association for Computational Linguistics.
  * Yu et al. (2021) Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. 2021.  Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 35, pages 10790–10797.
  * Zadeh et al. (2017) Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017.  Tensor fusion network for multimodal sentiment analysis.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1103–1114, Copenhagen, Denmark. Association for Computational Linguistics.
  * Zadeh et al. (2016) Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016.  MOSI: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos.  _CoRR_ , abs/1606.06259.
  * Zhang and Yang (2021) Yu Zhang and Qiang Yang. 2021.  A survey on multi-task learning.  _IEEE Transactions on Knowledge and Data Engineering_ , 34(12):5586–5609.
  * Zong et al. (2023) Yongshuo Zong, Oisin Mac Aodha, and Timothy Hospedales. 2023.  Self-supervised multimodal learning: A survey.  _arXiv preprint arXiv:2304.01008_.


# Conference Paper Title*
*Note: Sub-titles are not captured in Xplore and should not be used ††thanks: This work was made possible by AICC03-0324-200005 from the Qatar National Research Fund (a member of the Qatar Foundation). The findings herein reflect the work, and are solely the responsibility, of the authors.

Mohammad Shamsesalehi†, Mahmoud Ahmadian Attari†, Mohammad Amin Maleki Sadr††,
Benoit Champagne‡, Marwa Qaraqe*
†Dept. ECE, K. N. Toosi University of Technology, Tehran, Iran,
<EMAIL_ADDRESS>
‡Dept. ECE, McGill University, Montréal, Canada, email:
<EMAIL_ADDRESS>
*College of Science and Engineering, Hamad Bin Khalifa University, Qatar Foundation, Doha, Qatar
email<EMAIL_ADDRESS>*<EMAIL_ADDRESS>
This publication was made possible by AICC03-0324-200005 from the Qatar
National Research Fund (a member of Qatar Foundation). The findings herein
reflect the work, and are solely the responsibility, of the authors.

# A BFF-Based Attention Mechanism for Trajectory Estimation in mmWave MIMO
Communications

Mohammad Shamsesalehi†, Mahmoud Ahmadian Attari†, Mohammad Amin Maleki Sadr††,
Benoit Champagne‡, Marwa Qaraqe*
†Dept. ECE, K. N. Toosi University of Technology, Tehran, Iran,
<EMAIL_ADDRESS>
‡Dept. ECE, McGill University, Montréal, Canada, email:
<EMAIL_ADDRESS>
*College of Science and Engineering, Hamad Bin Khalifa University, Qatar Foundation, Doha, Qatar
email<EMAIL_ADDRESS>*<EMAIL_ADDRESS>
This publication was made possible by AICC03-0324-200005 from the Qatar
National Research Fund (a member of Qatar Foundation). The findings herein
reflect the work, and are solely the responsibility, of the authors.

###### Abstract

This paper explores a novel Neural Network (NN) architecture suitable for
Beamformed Fingerprint (BFF) localization in a millimeter-wave (mmWave)
multiple-input multiple-output (MIMO) outdoor system. The mmWave frequency
bands have attracted significant attention due to their precise timing
measurements, making them appealing for applications demanding accurate device
localization and trajectory estimation. The proposed NN architecture captures
BFF sequences originating from various user paths, and through the application
of learning mechanisms, subsequently estimates these trajectories.
Specifically, we propose a method for trajectory estimation, employing a
transformer network (TN) that relies on attention mechanisms. This TN-based
approach estimates wireless device trajectories using BFF sequences recorded
within a mmWave MIMO outdoor system. To validate the efficacy of our proposed
approach, numerical experiments are conducted using a comprehensive dataset of
radio measurements in an outdoor setting, complemented with ray tracing to
simulate wireless signal propagation at 28 GHz. The results illustrate that
the TN-based trajectory estimator outperforms other methods from the existing
literature and possesses the ability to generalize effectively to new
trajectories outside the training dataset.

###### Index Terms:

Millimeter Wave, MIMO, Localization, Trajectory Estimation, Transformer
Networks.

##  I Introduction

Beyond 5G networks operating within the millimeter wave (mmWave) frequency
bands, will provide unprecedented bandwidth, and are expected to excel in
terms of throughput, latency, and reliability across various transmission
distances [1]. To achieve these impressive goals, it is required that both
transmitting and receiving equipment possess precise knowledge about each
other’s relative position and orientation [2]. Moreover, the availability of
accurate localization information holds great importance for a range of
critical applications, and especially online trajectory estimation [3, 4].

From the perspective of the radio environment, localization within mmWave
bands can be categorized in terms of indoor and outdoor scenarios. A specific
subset of notable studies focused on indoor localization includes the works in
[5, 6, 7]. In [5], a novel massive MIMO (m-MIMO) framework for precise
position estimation in the mmWave band is proposed. In [6], the authors
introduce a novel hybrid 3D indoor positioning scheme designed for mmWave
massive MIMO systems, which combines received signal strength and angle of
arrival. For the outdoor scenarios, which is the focus of this work, [8]
presents a fingerprinting-based cooperative localization approach tailored for
mmWave massive MIMO systems, while [9] introduces a method using Monte Carlo
(MC) dropout to capture uncertainty in a CNN-based mmWave MIMO localization
system. In [10], authors explore the statistical relationship between channel
state information (CSI) and specific characteristics of user devices,
including position, orientation, and clock offset.

From a design perspective, localization methods can be further categorized
based on their modeling approach, i.e., distinguishing between physical and
data-driven methods. The former methods rely on physical knowledge of the
radio environment and channel characteristics [11]. However, applying these
methods in dense urban environments at mmWave frequencies presents substantial
challenges due to the intricate nature of channel models and the multitude of
parameters involved [12]. In contrast, data-driven models adopt a different
strategy and estimate user positions by collecting radio data or fingerprints
from a broad range of representative locations. Subsequently, machine learning
(ML) algorithms are employed to predict the location of a new target based on
its radio imprints or fingerprints [13].

In [14], the authors introduce a localization method based on fingerprints
that utilize path-loss measurements for both training and prediction phases.
Another fingerprint-based method is presented in [15], where crowd-sourcing is
employed to collect probabilistic Radio Signal Strength (RSS) data from
numerous smartphones, forming a database of fingerprints. Among the
fingerprint-based techniques, the Beamformed Fingerprint (BFF) methods stand
out. In [16], a BFF method is proposed to achieve precise outdoor positioning
by capitalizing on mmWave MIMO transmissions to achieve steerable and highly
focused radiation patterns. Recent studies show that BFF methods can generate
data-rich information, allowing ML techniques to discern subtle details in
both localization and trajectory estimation [17].

While recurrent Neural Networks (RNNs) are a common choice for modeling
sequential data, they suffer from gradient-related issues which diminish their
effectiveness over time, especially their ability to retain historical
information. These issues can be mitigated to some extent by incorporating
gating mechanisms into RNN cells, with Long Short-Term Memory (LSTM) being the
most prevalent architecture for this purpose. In [18] an LSTM network is
successfully employed for mobile localization based on WiFi fingerprints in an
indoor setting. Recently, Transformer Networks (TN), also known simply as
transformers, have demonstrated remarkable performance and adaptability in
effectively capturing and modeling sequential signals, as outlined in [19].
However, to the best of our knowledge, TN have not yet been applied to the
timely problem of BFF in outdoor MIMO systems.

In this paper, we introduce a deep learning (DL) approach that relies on BFF
sequences for trajectory estimation within a mmWave MIMO outdoor system. Our
approach capitalizes on the unique capabilities of BFF data and employs a
specific TN-based architecture to capture long-term historical data for
trajectory estimation tasks. Our main contributions and distinguishing
features of our work are summarized as follows:

  * •

We develop a novel TN architecture based on attention mechanism to forecast
the trajectories of wireless devices utilizing BFF sequences recorded within a
mmWave MIMO outdoor system.

  * •

Unlike RNN, LSTM, or temporal convolutional networks (TCN), the use of an
encoder-decoder structure along with attention mechanism allows the proposed
TN to detect subtle changes in the users’ directions within lengthy sequences,
and to generalize effectively to novel trajectories not included in the
training set.

  * •

The TN architecture naturally lends itself to a parallel implementation,
thereby enhancing computational speed and processing efficiency across
different sequences.

  * •

We validate the effectiveness of our proposed TN-based approach through
numerical experiments, using a substantial dataset of radio measurements
collected in an outdoor environment, enhanced by incorporating ray tracing
techniques to replicate wireless propagation at 28 GHz frequency.

  * •

We show that our proposed method can achieve accuracy levels of nearly one
meter for moving vehicles and less than one meter for pedestrians, surpassing
the performance of benchmark methods from the recent literature.

The remainder of this paper is structured as follows: Section II presents the
system model for BFF-based localization within a mmWave MIMO outdoor system.
Section III introduces the TN architecture based on attention mechanism for
addressing the trajectory estimation problem. Section IV presents the
simulation methodology along with our key experimental findings. Finally,
Section V concludes the paper.

##  II System Model and Problem Statement

As outdoor 5G base stations (BSs) are typically positioned at elevated
locations within urban environments, buildings emerge as the primary obstacles
that remain stationary over extended periods. Under such conditions, the
received power delay profile (PDP) measurements, as detailed in [20], are
expected to be a valuable resource for radio localization. To maximize
coverage, BSs can employ a sequence of directive Beamforming (BF) patterns to
transmit signals, effectively covering all feasible transmission angles.
Consequently, receivers can capture multiple distinct PDPs. Because of the
intricate propagation mechanisms that occur when obstacles are present, it is
expected that the measured PDPs will exhibit significant variations or
discontinuities across the localization area. These abrupt variations offer
valuable spatial information to a trained localization model [17].

Figure 1:  System model of BFF-based MIMO mmWave localization

The process of obtaining the BFF data through a BF codebook at the BS and PDP
monitoring at the mobile user is illustrated in Fig. 1. At the BS, directional
sounding signals are generated to facilitate the extraction of BFF data from
the radio environment for both purpose of offline model training and online
prediction, i.e., trajectory estimation, as elabotated below.

During the training phase, a BS equipped with $N_{T}$ antennas transmits
signals within a specific frequency band $B$ using a predetermined set of $M$
transmit BF vectors from a codebook
$\mathcal{C}=\left\\{\textbf{f}_{1},\textbf{f}_{2},\cdots,\textbf{f}_{M}\right\\}$,
where $\textbf{f}_{i}\in\mathbb{C}^{N_{T}\times 1}$ denotes the $i$th BF
vector. A user terminal equipped with $N_{R}$ antennas and located at a known
position (to be varied) is used to capture the sequence of transmitted
signals. The received user signal from the $i$th codebook at a specific
sounding frequency bin $f\in B$, can be expressed as:

$r_{i}(f)=\textbf{w}^{H}\textbf{H}\textbf{f}_{i}s+\textbf{w}^{H}\textbf{z},\;\;\;i\in\left\\{1,\cdots,M\right\\},$
(1)

where $s\in\mathbb{C}$ is the sounding signal amplitude,
$\textbf{H}\in\mathbb{C}^{N_{R}\times N_{T}}$ is the complex channel gain
matrix between the BS and user device, $\textbf{w}\in\mathbb{C}^{N_{R}\times
1}$ is the BF vector at the receiver and $\textbf{z}\in\mathbb{C}^{N_{R}\times
1}$ is an additive noise term.

At the user terminal, through processing of the received signals $r_{i}(f)$
over all frequencies $f\in B$, the received power is measured and an overall
PDP $P_{i}(\tau)$ is calculated, where $\tau$ represents the (continuous)
delay variable. A discrete-time version of the PDP, denoted as $P_{i}[j]$, is
obtained by uniformly sampling $P_{i}(\tau)$ at frequency
$F_{s}={T_{s}}^{-1}$, where $T_{s}$ denotes the sampling interval [20]. The
PDP can be expressed as:

$P_{i}[j]=P_{i}(jT_{s}),\;\;\;\forall j\in\left\\{1,\cdots,N_{s}\right\\}.$
(2)

where $N_{s}=\frac{T}{T_{s}}$ represents the total number of samples and $T$
denotes the maximum excess delay. The PDP samples are represented by binary
values for each sampling time, i.e.,

$x_{i,j}=I(P_{i}[j]\geq\eta),$ (3)

where $I$ is the indicator function and $\eta$ is a positive threshold level.
By utilizing binary PDP values, implementation complexity and memory
requirement of the TN can be decreased. Furthermore, utilizing a binary
representation of the PDP can effectively handle high noise levels and improve
localization performance [17]. Binary PDP values can be used to construct a
feature matrix $\textbf{X}=\left[x_{i,j}\right]$ which is transmitted to the
BS for localization purposes.

During the training phase, PDP data is collected as above for a predetermined
grid consisting of user terminal locations. The locations are indexed by
$n\in\left\\{1,\cdots,\mathcal{N}\right\\}$, where $\mathcal{N}$ is the total
number of locations, and $\textbf{y}_{n}\in\mathbb{R}^{1\times 2}$ represents
the coordinates of the $n$th location in 2D space. For each one of these
locations, the PDP feature matrix $\textbf{X}_{n}\in\mathbb{R}^{M\times
N_{s}}$ and the corresponding position label
$\textbf{y}_{n}\in\mathbb{R}^{1\times 2}$ are collected as the input and
output data for the TN. These matrices and vectors are collectively referred
to as the BFF and are stored in the BS as a dataset
$\mathcal{D}=(\textbf{X}_{n},\textbf{y}_{n})$. This dataset is only used for
training purpose. In the prediction phase, the collected PDP data is used
instead as input to the trained model for trajectory estimation of the user
terminal.

##  III Attention-Based Mechanism for BFF Trajectory Estimation

###  III-A Utilization of BFFs for Trajectory Modeling

The previous section described the method by which we can convert a single BFF
into a position. After extracting the positions in the environment based on
BFFs, trajectories can be created. The dataset did not include any
segmentation of movement types or traffic rules. In a real-world dataset,
these aspects would likely be observed, which could further enhance the
accuracy of the estimator.

Each trajectory can be described as
$\textbf{Y}=[\textbf{y}^{(1)},\textbf{y}^{(2)},\cdots,\textbf{y}^{(L)}]\in\mathbb{R}^{1\times
2L}$, where $L$ is the trajectory length (or the size of the system’s memory),
and each y shows a 2D location in the environment which is describes in the
previous section by $\textbf{y}_{n}$. We assume that all paths or trajectories
are involved in a movement in the environment. The input of our models
consists of the trajectories of different users, represented as
$\textbf{Y}^{\prime}=[\textbf{Y}_{1},\textbf{Y}_{2},\cdots,\textbf{Y}_{N}]^{\text{T}}$,
where $\textbf{Y}^{\prime}\in\mathbb{R}^{N\times 2L}$ and row space of the
above matrix is considered as a 2D cartesian path. We observe the positions of
all paths from time step 1 to $\text{T}_{obs}$, and predict their positions
for the time step from $\text{T}_{obs}+1$ to $L$. The set of future
trajectories can be described as $\bar{\textbf{Y}}\in\mathbb{R}^{N\times 2}$
and our estimated trajectories through the models can be denoted as
$\hat{\textbf{Y}}\in\mathbb{R}^{N\times 2}$.

###  III-B Trajectory Estimation Using TNs

TNs have been successfully used to estimate datasets that are structured as
sequences. One of the main benefits of using TN is their ability to focus on
subtle details and recognize different patterns within long data sequences. In
addition, TN utilizes an encoder-decoder design to accurately map inputs to
the corresponding outputs.

The encoder-decoder structures are widely utilized in sequence-to-sequence
models. In this paper, the encoder block transforms the input sequence
$\textbf{Y}^{\prime}$ into a latent state
$\textbf{Z}=\left[\textbf{Z}_{1},\textbf{Z}_{2},\cdots,\textbf{Z}_{N}\right]$.
Given Z, $\hat{\textbf{Y}}$ is produced at each time step. Note that, in an
autoregressive manner, the previously generated element is used as an
additional input to generate the subsequent element. In the TN, both the
encoder and decoder blocks use stacked self-attention, position-wise, and
fully connected (FC) layers. These layers can be seen in the left and right
halves of Fig. 2, respectively. For an in-depth discussion of the internal
structures of TNs, we refer the reader to [21].

Figure 2:  Representation of a TN model applied to BFF data.

The TN mainly can capture the dependencies in sequential data and the non-
linear attributes of BFFs through its Attention module. This module is
composed of a self-attention mechanism and a multi-head attention sub-layer.
These sub-layers allow the model to focus on subtle details of the input
sequence, capturing various aspects of the information.

The self-attention takes these inputs: the query matrix Q, the key matrix K,
the value matrix V, and the dimension $d_{k}$ of the queries and keys. It
computes the dot product of Q and K, and then scales it by a factor of
$\frac{1}{\sqrt{d_{k}}}$. The weights of values are determined by applying a
softmax function. This scaled operation is particularly beneficial when
$d_{k}$ is large, as it prevents the softmax function from entering regions
where the gradients are extremely small.

$\text{Attention}(\textbf{Q},\textbf{K},\textbf{V})=\text{softmax}(\frac{\textbf{Q}\textbf{K}^{\text{T}}}{\sqrt{d_{k}}})\textbf{V}.$
(4)

Rather than learning a single attention function, it has been found
advantageous to map the queries, keys, and values $h$ times in order to learn
different contextual information for each [19]. The self-attention function
applies to each projected version of the queries, keys, and values. Following
this, the outputs are concatenated and projected once more to determine the
best and final value weights. Thus, by leveraging the multi-head attention
sub-layer, the TN is capable of simultaneously producing a comprehensive
latent feature of the trajectory data.

$\begin{split}\text{MultiHead}(\textbf{Q},\textbf{K},\textbf{V})=\text{concat}(\text{head}_{1},\cdots,\text{head}_{h})\boldsymbol{\textbf{W}}^{O},\\\
\text{head}_{i}=\text{Attention}(\textbf{Q}\boldsymbol{\textbf{W}}_{i}^{Q},\textbf{K}\boldsymbol{\textbf{W}}_{i}^{K},\textbf{V}\boldsymbol{\textbf{W}}_{i}^{V}).\end{split}$
(5)

The parameters matrices in queries, keys, values, and output are represented
by the projections
$\boldsymbol{\textbf{W}}_{i}^{Q},\boldsymbol{\textbf{W}}_{i}^{K},\boldsymbol{\textbf{W}}_{i}^{V}$
and $\boldsymbol{\textbf{W}}_{i}^{O}$, respectively.

Moreover, the fully connected Feed-Forward Networks are made up of two linear
transformations, with a ReLU activation function that is applied to each of
the attention sublayers.

$\text{Feedforward}(\boldsymbol{\lambda})=\text{max}(0,\boldsymbol{\lambda}\boldsymbol{\textbf{W}}_{1}+\textbf{b}_{1})\boldsymbol{\textbf{W}}_{2}+\textbf{b}_{2}.$
(6)

Similar to other sequence-based models, this particular model uses learned
embeddings to convert input and output tokens into vectors with a dimension of
$\text{T}_{obs}$. A learned linear transformation function is also applied to
the decoder output to generate predicted probabilities for the next token. The
two embedding layers share a common weight matrix. In addition, positional
encoding is a technique used in the TN to help the model capture the
sequential order of the input data. Unlike RNNs, Transformers do not have a
natural way of encoding the order of the sequence, because the model treats
all the input tokens as unordered elements. Therefore, Positional Encoding
provides a way to inject information about the sequence position into the
input embeddings. Positional Encoding works by adding a fixed vector to the
embedding vector of each input token, which encodes the token’s position
within the sequence. The added vector is calculated based on a set of
sinusoidal functions with different frequencies and phases [22]. Specifically,
the Positional Encoding for a token at position $pos$ and dimension $j$ is
given by:

$\text{PE}_{{(pos,j)}}=\begin{cases}\sin({pos}/10000^{2j/\text{T}_{obs}})&\text{if
$j$ is even}\\\ \cos({pos}/10000^{2(j-1)/\text{T}_{obs}})&\text{if $j$ is
odd}\end{cases}.$ (7)

##  IV Numerical Results

We conducted experiments to evaluate the trajectory estimation performance of
various DL architectures. Initially, we learned the trajectories using trained
deep neural networks based on BFF data. Following this, we introduced new,
unseen trajectories that are not part of the training set for the testing
stage. The performance of these networks is then evaluated. Ultimately, we
compared the performance of our proposed method with other state-of-the-art DL
methods, specifically looking at average error versus sequence length. The
parameters for the simulation settings are detailed in Table I. To provide a
clearer understanding of the results, we have divided them into two
subsections: Simulation Methodology and Results and Discussion.

TABLE I: The DL architectures hyperparameters. Parameters | All tracking-based approaches
---|---
Number of units | (256, 64)
MLP layers | 2
Regression output | 2 neurons (2D position)
#of training sequences | 320408
Epochs | Up to 100
Batch size | 64
Optimizer | ADAM
Learning rate | $10^{-3}$
Dropout | 0.01

###  IV-A Simulation Methodology

We utilize the Wireless InSite ray-tracing simulator to generate mmWave data
[23]. This allows us to analyze the propagation of wireless signals in complex
urban environments and assess the accuracy of our model. We examine a detailed
3D map of the New York University (NYU) region [24], which includes BFF data
from 160801 distinct two-dimensional locations (401 x 401 positions). In
accordance with the findings in [25], we employ the propagation parameters and
the ray-tracing simulations, which are found to be consistent with
experimental measurements [26]. To evaluate the performance of our beamforming
algorithm, we conduct simulations at a carrier frequency of $28$GHz with a
codebook size of $M=32$. For all potential transmit beamforming vectors, we
set the maximum received power to be $30$dBm. The transmitter is positioned at
coordinates $(0,0)$, and we add noise into the ray-tracing data using a log-
normal distribution with a standard deviation of $6$dB.

To create sequences for tracking-based methods, two types of synthetic
sequences are produced: pedestrian-like and vehicle-like sequences. The
pedestrian-like sequences are designed to allow sudden stops or quick changes
in direction at a low average speed of 5 km/h, while the vehicle-like
sequences have a higher average speed of 30 km/h and acceleration, with a
limited steering angle. The sequences in the dataset are sampled at a rate of
1 Hz, producing paths as shown in Fig. 3. With regard to sequences, the
training, validating, and testing paths are selected from separate groups of
trajectories in order to avoid memorization.

Figure 3:  The generated sequence in the NYU area

###  IV-B Results and Discussion

We utilized three DL models commonly used for tracking and trajectory
estimation for benchmarking against our proposed method. These include LSTM,
temporal convolutional network (TCN) [17], and RNN. Table II presents a
comparison of three significant attributes: the model size in terms of
learnable parameters, training time, and prediction time. As observed, TNs is
faster than LSTM and TCN models, as this desired method consider all inputs
simultaneously. Training LSTM and TCN networks poses a greater challenge than
TNs due to their significantly larger number of learnable parameters. Despite
this, TNs maintain competitive complexity and computational cost with RNN.

TABLE II: Number of learnable parameters, training and prediction time for sequence length 11 for the vehicles. DL architecture | L.parameters | T.time (hours) | P.Time (mins)
---|---|---|---
RNN | $6.525\times 10^{4}$ | $00:39:51$ | $00:45$
LSTM | $2.625\times 10^{5}$ | $2:41:50$ | $05:17$
TCN | $2.812\times 10^{5}$ | $2:31:03$ | $04:25$
Transformer | $6.9\times 10^{4}$ | $00:53:15$ | $01:07$

Fig. 4 illustrates the average error achieved by the DL models between
$\bar{\textbf{Y}}\in\mathbb{R}^{N\times 2}$ and
$\hat{\textbf{Y}}\in\mathbb{R}^{N\times 2}$ for the trajectory estimation task
of vehicles across different BFF sequences. It is evident that in our proposed
method outperforms the state-of-the-art methods in terms of average error,
maintaining precision even for longer trajectories. In addition, vehicle-like
sequences present distinctive characteristics, including higher average speeds
and rapid accelerations. Consequently, deep learning-based trajectory
estimation for these sequences tends to be slightly less precise than for
sequences with less complexity in movement. The increased complexity of
vehicle-like motions poses challenges for accurate modeling, leading to some
compromises in the performance of trajectory estimation.

Fig. 5 further depicts the average error versus sequence length for
pedestrians using the same DL methods. As observed, our proposed method
continues to surpass other methods. The inherent predictability and smoothness
of pedestrian-like sequences make them ideal for accurate modeling and
tracking.

Figure 4:  Prediction error comparison vs sequence length for vehicles,
considering an average noise value ($\sigma$ = 6 dB). Figure 5: Prediction
error comparison vs sequence length for pedestrians, considering an average
noise value ($\sigma$ = 6 dB).

Fig. 6 and Fig. 7 demonstrate $95^{th}$ percentile error as a tool to extract
additional accuracy across the proposed approach TN, TCN, and LSTM at noise
level 6 dB for vehicles and pedestrians, respectively. As can be seen, the TN
still outperforms in terms of $95^{th}$ percentile errors than the two other
state-of-the-art methods. The $95^{th}$ percentile basically says that 95
percent of the error is below this value and the other 5 percent of the error
exceeds that value. Moreover, we compare the performance of the models for a
sequence length = 7 and noise level $\sigma=9$dB in Table III. The table
demonstrates the superior performance of the TN model under these conditions.
Despite the increased noise level, the TN model consistently outperformed both
the LSTM and TCN models.

Figure 6: Comparative analysis of $95^{th}$ percentile errors vs sequence length for vehicles, considering an average noise value ($\sigma$ = 6 dB). Figure 7: Comparative analysis of $95^{th}$ percentile errors vs sequence length for pedestrians, considering an average noise value ($\sigma$ = 6 dB). TABLE III: Comparing the performance of the models for a sequence length = 7 and noise level $\sigma=9$ dB. DL architecture | Vehicles Avg Error(m) | Pedestrians Avg Error(m)
---|---|---
LSTM | $2.20712$ | $1.0045$
TCN | $1.69597$ | $0.982$
Transformer | $1.4104$ | $0.89$

##  V Conslusion

In this paper, we proposed and investigated a procedure for trajectory
estimation in mmWave MIMO outdoor systems. We show that TN architecture can
capture sequences of BFF signals originating from vehicles and pedestrians’
movements, and through the application of learning mechanisms, subsequently
estimate these trajectories. Our simulation results confirm that the TN
architecture based on the attention mechanism outperforms the other state-of-
the-art methods.

## References

  * [1] A. Shahmansoori, G. E. Garcia, G. Destino, G. Seco-Granados, and H. Wymeersch, “5G Position and Orientation Estimation through Millimeter Wave MIMO,” in _IEEE Globecom Workshops_ , 06-10 December 2015, pp. 1–6.
  * [2] F. Mogyorósi, P. Revisnyei, A. Pašić, Z. Papp, I. Törös, P. Varga, and A. Pašić, “Positioning in 5G and 6G Networks—a Survey,” _Sensors_ , vol. 22, no. 13, p. 4757, 2022.
  * [3] M. Yin, A. K. Veldanda, A. Trivedi, and etc., “Millimeter Wave Wireless Assisted Robot Navigation with Link State Classification,” _OJ-COMS_ , vol. 3, pp. 493–507, 2022.
  * [4] H. Kim, H. Wymeersch, N. Garcia, G. Seco-Granados, and S. Kim, “5G mmWave Vehicular Tracking,” in _IEEE 52nd Asilomar Conf. Signals Syst. Comput._ , 28-31 October 2018, pp. 541–547.
  * [5] N. Vukmirović, M. Janjić, P. M. Djurić, and M. Erić, “Position Estimation with a Millimeter-Wave Massive MIMO System based on Distributed Steerable Phased Antenna Arrays,” _Eurasip J. Adv. Signal Process_ , vol. 33, pp. 1–17, 2018.
  * [6] Z. Lin, T. Lv, and P. T. Mathiopoulos, “3-D Indoor Positioning for Millimeter-Wave Massive MIMO Systems,” _IEEE Trans. Commun._ , vol. 66, no. 6, pp. 2472–2486, 2018.
  * [7] A. Vashist, M. P. Li, and etc., “KF-Loc: A Kalman Filter and Machine Learning Integrated Localization System using Consumer-Grade Millimeter-Wave Hardware,” _IEEE Consum. Electron. Mag._ , vol. 11, no. 4, pp. 65–77, 2021.
  * [8] Z. Shen, K. Xu, and X. Xia, “2D Fingerprinting-based Localization for mmWave Cell-free Massive MIMO Systems,” _IEEE Commun. Lett._ , vol. 25, no. 11, pp. 3556–3560, 2021.
  * [9] M. A. M. Sadr, J. Gante, B. Champagne, G. Falcao, and L. Sousa, “Uncertainty Estimation via Monte Carlo Dropout in CNN-Based mmWave MIMO Localization,” _IEEE Signal Process. Lett._ , vol. 29, pp. 269–273, 2021\.
  * [10] R. Mendrzik, F. Meyer, G. Bauch, and M. Win, “Localization, Mapping, and Synchronization in 5G Millimeter Wave Massive MIMO Systems,” in _20th IEEE SPAWC_. IEEE, 2019, pp. 1–5.
  * [11] M. Koivisto, M. Costa, J. Werner, K. Heiska, J. Talvitie, K. Leppänen, V. Koivunen, and M. Valkama, “Joint Device Positioning and Clock Synchronization in 5G Ultra-Dense Networks,” _IEEE Trans. Wirel. Commun._ , vol. 16, no. 5, pp. 2866–2881, 2017.
  * [12] O. Kanhere and T. S. Rappaport, “Position Locationing for Millimeter Wave Systems,” in _IEEE GLOBECOM Conf._ IEEE, 9-13 December 2018, pp. 206–212.
  * [13] X. Sun, X. Gao, G. Y. Li, and W. Han, “Single-Site Localization Based on a New Type of Fingerprint for Massive MIMO-OFDM Systems,” _IEEE Trans. Veh. Technol._ , vol. 67, no. 7, pp. 6134–6145, 2018.
  * [14] J. Zhang, G. Han, N. Sun, and L. Shu, “Path-Loss-Based Fingerprint Localization Approach for Location-Based Services in Indoor Environments,” _IEEE Access_ , vol. 5, pp. 13 756–13 769, 2017.
  * [15] B. Zhou, Q. Li, Q. Mao, and W. Tu, “A Robust Crowdsourcing-Based Indoor Localization System,” _Sensors_ , vol. 17, no. 4, p. 864, 2017\.
  * [16] J. Gante, G. Falcão, and L. Sousa, “Beamformed Fingerprint Learning for Accurate Millimeter Wave Positioning,” in _88th IEEE VTC_. IEEE, 27-30 August 2018, pp. 1–5.
  * [17] J. Gante, G. Falcao, and L. Sousa, “Deep Learning Architectures for Accurate millimeter Wave Positioning in 5G,” _Neural Process. Lett._ , vol. 51, no. 1, pp. 487–514, 2020.
  * [18] S. Bai, M. Yan, Y. Luo, and Q. Wan, “Rfedrnn: An End-to-End Recurrent Neural Network for Radio Frequency Path Fingerprinting,” in _IEA/AIE_. Springer, 2018, pp. 560–571.
  * [19] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention Is All You Need,” _Adv. Neural Inf. Process._ , vol. 30, 2017.
  * [20] K. Guan, B. Peng, D. He, J. M. Eckhardt, H. Yi, S. Rey, B. Ai, Z. Zhong, and T. Kürner, “Channel Sounding and Ray Tracing for Intrawagon Scenario at mmWave and Sub-mmWave Bands,” _IEEE Trans. Antennas Propag._ , vol. 69, no. 2, pp. 1007–1019, 2020.
  * [21] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in _IEEE CVPR_ , 12 December 2016, pp. 770–778.
  * [22] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin, “Convolutional Sequence to Sequence Learning,” in _ICML_. PMLR, 12 May 2017, pp. 1243–1252.
  * [23] Remcom, _”Wireless Insite,”_ , Accessed: Jun. 2018. [Online]. Available:http://www.remcom.com/wireless-insite.
  * [24] _“Open Source New York 3D Building Model.”_ , Accessed: Apr. 9, 2021. [Online]. Available: http://www1.nyc.gov/site/doitt/initiatives/3d-building.
  * [25] Y. Azar, G. N. Wong, K. Wang, R. Mayzus, J. K. Schulz, H. Zhao, F. Gutierrez, D. Hwang, and T. S. Rappaport, “28 GHz Propagation Measurements for Outdoor Cellular Communications using Steerable Beam Antennas in New York City,” in _IEEE ICC_ , June 2013, pp. 5143–5147.
  * [26] J. Gante, G. Falciao, and L. Sousa, “Data-Aided Fast Beamforming Selection for 5G,” in _IEEE ICASSP_ , April 2018, pp. 1183–1187.


# Tubular surfaces whose Gauss map $N$ satisfies
$\Delta^{II}\boldsymbol{N}=\varLambda\boldsymbol{N}$

Hassan Al-Zoubi  Department of Mathematics, Al-Zaytoonah University of Jordan,
P.O. Box 130, Amman, Jordan 11733<EMAIL_ADDRESS>
###### Abstract.

In this paper, we consider tubes in the Euclidean 3-space whose Gauss map
$\boldsymbol{N}$ is of coordinate finite $II$-type, i.e., the position vector
$\boldsymbol{N}$ satisfies the relation
$\Delta^{II}\boldsymbol{N}=\varLambda\boldsymbol{N}$, where $\Delta^{II}$ is
the Laplace operator with respect to the second fundamental form $I$ of the
surface and $\varLambda$ is a square matrix of order 3. We show that circular
cylinders are the only class of surfaces mentioned above of coordinate finite
$I$-type Gauss map.

###### Key words and phrases:

Surfaces in the Euclidean 3-space, Surfaces of coordinate finite type, Laplace
operator, tubes in the Euclidean 3-space

###### 2010 Mathematics Subject Classification:

53A05, 47A75

##  1\. Introduction

Euclidean immersions of finite Chen type regarding to the first fundamental
form $I$ was introduced by B.-Y. Chen about four decades ago and it has been a
topic of active research by many differential geometers since then. Many
results in this area have been collected in [24].

Let r be an isometric immersion of a surface $S$ in the 3-dimensional
Euclidean space $\mathbb{E}^{3}$. We represent by $\Delta^{I}$ the Laplacian
operator of $S$ acting on the space of smooth functions $C^{\infty}(S)$. Then
$S$ is called surface of finite $I$-type, if the position vector r of $S$ can
be decomposed as a finite sum of eigenvectors of $\Delta^{I}$ of $S$, that is

$\textbf{{r}}=\textbf{{x}}_{0}+\textbf{{x}}_{1}+\textbf{{x}}_{2}+...+\textbf{{x}}_{k},$

where

$\Delta^{I}\textbf{{x}}_{i}=\lambda_{i}\,\textbf{{x}}_{i},\quad i=1,...,k,$

$\textbf{{y}}_{0}$ is a constant vector and
$\lambda_{1},\lambda_{2},...,\lambda_{k}$ are eigenvalues of the operator
$\Delta^{I}$.

Still now, the only known surfaces of finite type in the Euclidean 3-space are
the minimal surfaces, the circular cylinders and the spheres. So in [18] B.-Y.
Chen mentions the following problem

######  Problem 1.

Determine all surfaces of finite Chen $I$-type in $\mathbb{E}^{3}$.

In order to provide an answer to the above problem, important families of
surfaces were studied by different authors by proving that finite type ruled
surfaces, finite type quadrics, finite type tubes, finite type cyclides of
Dupin and finite type spiral surfaces are surfaces of the only known examples
in $\mathbb{E}^{3}$. However, for another classical families of surfaces, such
as surfaces of revolution, translation surfaces as well as helicoidal
surfaces, the classification of its finite type surfaces is not known yet.

Following this ideas, in [17] B.-Y. Chen and P. Piccini studied submanifolds
of which their Gauss map are of finite type with respect to the first
fundamental form, so, specifically for the surfaces in $\mathbb{E}^{3}$, we
are led to the following problem

######  Problem 2.

Classify all surfaces in $\mathbb{E}^{3}$ with finite type Gauss map
corresponding to the first fundamental form $I$.

Later in [22] O. Garay generalized Chen’s condition studied surfaces for which
surfaces in $\mathbb{E}^{3}$ satisfy the condition
$\Delta^{I}\boldsymbol{r}=A\boldsymbol{r}$ $({\ddagger})$ where
$A\in\mathbb{Re}^{3\times 3}$ or $\Delta^{I}\boldsymbol{N}=A\boldsymbol{N}$
where $\boldsymbol{N}$ is the Gauss map of $S$. It was shown that a surface
$S$ in $\mathbb{E}^{3}$ satisfies $({\ddagger})$ if and only if it is an open
part of a minimal surface, a sphere, or a circular cylinder. Surfaces
satisfying $({\ddagger})$ are said to be of coordinate finite type, meanwhile
Surfaces satisfying $\Delta^{I}\boldsymbol{N}=A\boldsymbol{N}$ are said to be
of coordinate finite type Gauss map.

In the framework of surfaces of finite type Stamatakis and Al-Zoubi in [29]
defined the notion of surfaces of finite type associated with the 2nd or 3rd
fundamental form. For results concerning this see ([1], [3], [5], [7], [9],
[11], [12], [15], [16]).

A general study which seems to be interesting can be made by investigating
families of surfaces in $\mathbb{E}^{3}$ whose position vector r satisfies the
relation

$\Delta^{J}\textbf{{r}}=A\textbf{{r}},\ \ \ J=II,III,$ (1.1)

where $A\in\mathbb{Re}^{3\times 3}$. Surfaces whose position vector r
satisfies the above relation are said to be of coordinate finite $J$-type.

In [14] H. Al-Zoubi and S. Stamatakis, studied the quadric surfaces satisfying

$\Delta^{III}\textbf{{r}}=A\textbf{{r}}.$ (1.2)

The same authors in [30] classified the class of surfaces of revolution
satisfying condition (1.2). Later in [13], the translation surfaces were
studied, and it was proved that Sherk’s surface is the only translation
surface satisfying (1.2). B. Senoussi and M. Bekkar, in [28] classified the
Helicoidal surfaces with $\Delta^{J}r=Ar,J=I,II,III$.

On the other hand, H. Al-Zoubi, T. Hamadneh in [11] studied the surfaces of
revolution satisfying

$\Delta^{II}\textbf{{r}}=A\textbf{{r}}.$

where $A\in\mathbb{Re}^{3\times 3}$.

Also, the notion of surfaces of finite type can be applied to any smooth map
not necessary the position vector of the surface, for example, its Gauss map.
For more details, the reader can be referred to ([2], [4], [6], [10], [17],
[18], [19], [20], [21], [27]).

In this research, we will focus on surfaces of finite $II$-type. Firstly, we
will define the second differential parameter of Beltrami regarded the 2nd
fundamental form of $S$. Then, we complete our study by proving our main
result.

##  2\. Preliminaries

We consider a $C^{m}$-surface $S$, $m\geq 3$, in the Euclidean 3-space
$\mathbb{E}^{3}$ defined by an injective $C^{m}$-immersion
$\boldsymbol{r}=\boldsymbol{r}(v^{1},v^{2})$ on a region
$W:=U\times\mathbb{R}\,\,\,(U\subset\mathbb{R}$ open interval) of
$\mathbb{R}^{2}$ with nonvanishing Gauss curvature. Denoting by

$I=g_{st}\,dv^{s}dv^{t},\quad II=b_{st}\,dv^{s}dv^{t}$,

$III=e_{st}\,dv^{s}dv^{t},\quad s,t=1,2$,

the 1st, 2nd and 3rd fundamental forms of $S$ respectively. For any two
sufficiently differentiable functions $p(v^{1},v^{2})$ and $q(v^{1},v^{2})$ on
$S$, the 1st differential parameter of Beltrami according to the fundamental
form $J$ is defined by [8]

$\nabla^{J}(p,q):=a^{st}p_{/s}q_{/t},\ \ \ J=I,II,III$ (2.1)

where $p_{/s}:=\frac{\partial p}{\partial v^{s}}$, and $(a^{st})$ stand as the
components of the inverse tensor of $(g_{st}),(b_{st})$ and $(e_{st})$
respectively. The 2nd differential parameter of Beltrami of $S$ is defined by
[8]

$\triangle^{J}p:=-a^{st}\nabla^{J}_{s}p_{/t},\ \ \ J=I,II,III,$ (2.2)

where $\nabla^{J}_{s}$ is the covariant derivative in the $v^{s}$ direction
corresponding to the fundamental form $J$.

After a long computations, one can find that [29]

$\Delta^{II}\boldsymbol{N}=\frac{1}{2K}grad^{I}(K)+2H\boldsymbol{N}.$ (2.3)

where $K,H$ are the Gauss and mean curvature of $S$ respectively. Next, we
want to investigate the tubular surfaces whose Gauss map $\boldsymbol{N}$
satisfies the following relation

$\Delta^{II}\textbf{{N}}=A\textbf{{N}}.$ (2.4)

##  3\. Tubes in $\mathbb{E}^{3}$

Let $\ell:\textbf{{a}}=\textbf{{a}}(\emph{u})$, $\mathit{u}\epsilon(a,b)$ be a
regular unit speed curve of finite length. We denote by
${\textbf{{t}},\textbf{{h}},\textbf{{b}}}$ the Frenet frame of the curve a and
with $\kappa>0$ its curvature. Then a regular parametric representation of a
tubular surface $\mathfrak{F}$ of radius $r$ satisfies
$0<r<min\frac{1}{|\kappa|}$ is given by

$\mathfrak{F}:\textbf{{x}}(u,\phi)=\textbf{{a}}+r\cos\phi\textbf{{h}}+r\sin\phi\textbf{{b}}.$

One can find that fundamental forms $I$ and $II$ of $\mathfrak{F}$ are given
by

$I=\big{(}\delta^{2}+r^{2}\tau^{2}\big{)}du^{2}+2r^{2}\tau
dud\phi+r^{2}d\phi^{2},$

$II=\big{(}-\kappa\delta\cos\phi+r\tau^{2}\big{)}du^{2}+2r\tau
dud\phi+rd\phi^{2},$

where $\delta:=(1-r\kappa\cos\phi)$ and $\tau$ is the torsion of the curve a.
The Gauss curvature of $\mathfrak{F}$ is given by

$K=-\frac{\kappa\cos\phi}{r\delta}$.

In the following we consider an open portion of the tube surface where $K\neq
0(\Leftrightarrow\cos\phi\neq 0)$. From (2.2) the Laplace operator
$\Delta^{II}$ of $\mathfrak{F}$ is (see also [1], formula (9))

$\displaystyle\Delta^{II}$ $\displaystyle=$
$\displaystyle\frac{1}{\kappa\delta\cos\phi}\Bigg{[}\frac{\partial^{2}}{\partial
u^{2}}-2\tau\frac{\partial^{2}}{\partial u\partial\phi}$
$\displaystyle+\Big{(}\tau^{2}-\frac{\kappa\delta\cos\phi}{r}\Big{)}\frac{\partial^{2}}{\partial\phi^{2}}+\frac{(1-2\delta)\beta}{2\kappa\delta\cos\phi}\frac{\partial}{\partial
u}$
$\displaystyle+\left(-\tau\acute{}+\frac{\tau\beta(2\delta-1)}{2\kappa\delta\cos\phi}+\frac{\kappa(2\delta-1)\sin\phi}{2r}\right)\frac{\partial}{\partial\phi}\Bigg{]},$

where $\beta:=\kappa\acute{}\cos\phi+\kappa\tau\sin\phi$ and
$\acute{}:=\frac{d}{du}$.

The Gauss map $\textbf{{N}}(u,\phi)$ of $\mathfrak{F}$ has parametric
representation

$\textbf{{N}}(u,\phi)=-\cos\phi\textbf{{h}}-\sin\phi\textbf{{b}}.$ (3.2)

Using (2.3) and (3), we obtain

$\displaystyle\Delta^{II}\textbf{{N}}$ $\displaystyle=$
$\displaystyle\frac{\beta}{2\kappa\delta^{2}\cos\phi}\textbf{{t}}+\bigg{(}\frac{\sin^{2}\phi}{2r\delta\cos\phi}+\frac{\cos\phi}{r\delta}-\frac{2\cos\phi}{r}\bigg{)}\textbf{{h}}$
(3.3) $\displaystyle+\frac{(1-4\delta)\sin\phi}{2r\delta}\textbf{{b}}.$

Let $(t_{i},h_{i},b_{i})$ and $N_{i},i=1,2,3$ be the coordinate functions of
$\boldsymbol{t},\boldsymbol{h},\boldsymbol{b},\boldsymbol{N}$ respectively,
and let $a_{ij}$ for $i,j=1,2,3$ be the entries of the matrix $A$. By virtue
of (2.4) and (3.3) one can find

$\displaystyle\frac{\beta}{2\kappa\delta^{2}\cos\phi}t_{i}+\bigg{(}\frac{\sin^{2}\phi}{2r\delta\cos\phi}+\frac{\cos\phi}{r\delta}-\frac{2\cos\phi}{r}\bigg{)}h_{i}$
$\displaystyle+\frac{(1-4\delta)\sin\phi}{2r\delta}b_{i}=$ $\displaystyle-
a_{i1}(h_{1}\cos\phi+b_{1}\sin\phi)-a_{i2}(h_{2}\cos\phi+b_{2}\sin\phi)$
$\displaystyle-a_{i3}(h_{3}\cos\phi+b_{3}\sin\phi).$ (3.4)

We rewrite the last equation as follows

$\displaystyle r\beta
t_{i}+\kappa\delta\big{(}1+(1-4\delta)\cos^{2}\phi\big{)}h_{i}$ (3.5)
$\displaystyle+\kappa\delta(1-4\delta)\cos\phi\sin\phi b_{i}$
$\displaystyle+2\kappa\delta^{2}\cos\phi\big{(}a_{i1}(h_{1}\cos\phi+b_{1}\sin\phi)$
$\displaystyle+a_{i2}(h_{2}\cos\phi+b_{2}\sin\phi)$
$\displaystyle+a_{i3}(h_{3}\cos\phi+b_{3}\sin\phi)\big{)}=0.$

We have the following two cases

Case I. $\beta\equiv 0$. Then $\kappa^{\prime}=0$ and $\kappa\tau=0$. Thus
$\tau=0$ and $\kappa$ = const. $\neq 0$, therefore the curve a is a plane
circle and so, $\mathfrak{F}$ is an anchor ring. In this case, the first
fundamental form becomes

$I=\delta^{2}du^{2}+r^{2}d\phi^{2}$,

while the second is

$II=-\kappa\delta\cos\phi du^{2}+rd\phi^{2}.$

Hence, the Laplacian is given by

$\Delta^{II}=\frac{1}{\kappa\delta\cos\phi}\frac{\partial^{2}}{\partial
u^{2}}-\frac{1}{r}\frac{\partial^{2}}{\partial\phi^{2}}+\frac{(2\delta-1)\sin\phi}{2r\delta\cos\phi}\frac{\partial}{\partial\phi}.$
(3.6)

For the Gauss map $\boldsymbol{N}$ of $\mathfrak{F}$, we have

$\boldsymbol{N}=\\{-\cos u\cos\phi,-\sin u\cos\phi,-\sin\phi\\}.$

Let $\boldsymbol{N}=(N_{1},N_{2},N_{3})$. By virtue of (3.6) we find

$\Delta^{II}N_{1}=\Bigg{[}\frac{1}{\kappa\delta}-\frac{\cos\phi}{r}+\frac{(2\delta-1)\sin^{2}\phi}{2r\delta\cos\phi}\Bigg{]}\cos
u,$ (3.7)

$\Delta^{II}N_{2}=\Bigg{[}\frac{1}{\kappa\delta}-\frac{\cos\phi}{r}+\frac{(2\delta-1)\sin^{2}\phi}{2r\delta\cos\phi}\Bigg{]}\sin
u,$ (3.8)

$\Delta^{II}N_{3}=-\frac{\sin\phi}{2r\delta}(4\delta-1).$ (3.9)

Let $a_{ij},i,j=1,2,3$ be the entries of the matrix $A$. From (2.4) and on
account of (3.7), (3.8) and (3.9) we get

$\displaystyle\Bigg{[}\frac{1}{\kappa\delta}-\frac{\cos\phi}{r}+\frac{(2\delta-1)\sin^{2}\phi}{2r\delta\cos\phi}\Bigg{]}\cos
u=$ (3.10) $\displaystyle-a_{11}\cos u\cos\phi-a_{12}\sin u\cos\phi-
a_{13}\sin\phi,$

$\displaystyle\Bigg{[}\frac{1}{\kappa\delta}-\frac{\cos\phi}{r}+\frac{(2\delta-1)\sin^{2}\phi}{2r\delta\cos\phi}\Bigg{]}\sin
u=$ (3.11) $\displaystyle-a_{21}\cos u\cos\phi-a_{22}\sin u\cos\phi-
a_{23}\sin\phi,$

$\displaystyle-\frac{\sin\phi}{2r\delta}(4\delta-1)=$ (3.12) $\displaystyle-
a_{31}\cos u\cos\phi-a_{32}\sin u\cos\phi-a_{33}\sin\phi.$

From the last equation it is easily verified that $a_{31}=a_{32}=0$. Hence
relation (3.12) becomes

$\frac{\sin\phi}{2r\delta}(4\delta-1)=a_{33}\sin\phi,$ (3.13)

then it follows that

$\frac{4\delta-1}{2r\delta}=a_{33}.$ (3.14)

In this case, the above relation is valid only when $a_{33}$ is function of
the parameters $u$ and $\phi$, Which contradicts our assumption. So we have

######  Corollary 1.

Every anchor ring in the Euclidean 3-space is of coordinate infinite type
Gauss map regarding the second fundamental form.

Case II. $\beta\neq 0$. Recalling equations (3.5) for $i=1,2,3$, we obtain
that these equations are polynomials in $\cos\phi,\sin\phi$ with coefficients
functions of $u$. Therefore we conclude that

$r\beta t_{i}=0,$ (3.15)
$\kappa\delta\bigg{(}1+(1-4\delta)\cos^{2}\phi\bigg{)}h_{i}=0,$ (3.16)
$\kappa\delta(1-4\delta)\cos\phi\sin\phi b_{i}=0,$ (3.17)
$2\kappa\delta^{2}\cos^{2}\phi\big{(}a_{i1}h_{1}+a_{i2}h_{2}+a_{i3}h_{3}\big{)}=0,$
(3.18)
$2\kappa\delta^{2}\cos\phi\sin\phi\big{(}a_{i1}b_{1}+a_{i2}b_{2}+a_{i3}b_{3}\big{)}=0.$
(3.19)

Since (3.15) is valid for $i=1,2,3$, so in vector notation it follows that

$r\beta\boldsymbol{t}=\boldsymbol{0},$ (3.20)

from which we must have $\beta=0$ and thus $\mathfrak{F}$ is an anchor ring.
So, according to Corollary 1, we have

######  Theorem 1.

All tubes in the Euclidean 3-space is of coordinate infinite type Gauss map
regarding the second fundamental form.

## References

  * [1] H. Al-Zoubi, Tubes of finite $II$-type in the Euclidean 3-space, WSEAS Trans. Math. 17 (2018), 1-5.
  * [2] H. Al-Zoubi, On the Gauss map of quadric surfaces, arXiv: 1905.00962v1, (2019).
  * [3] H. Al-Zoubi, W. Al Mashaleh, Surfaces of finite type with respect to the third fundamental form, IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology (JEEIT), Amman, April 9-11, (2019).
  * [4] H. Al-Zoubi, M. Al-Sabbagh, Anchor rings of finite type Gauss map in the Euclidean 3-space, Int. J. Math. and Comput. Methods 5 (2020), 9-13.
  * [5] H. Al-Zoubi, M. Al-Sabbagh, S. Stamatakis, On surfaces of finite Chen $III$-type, Bull. Belgian Math. Soc. 26 (2019), 177-187.
  * [6] H. Al-Zoubi, H. Alzaareer, T. Hamadneh, M. Al Rawajbeh, Tubes of coordinate finite type Gauss map in the Euclidean 3-space, Indian J. Math. 62 (2020), 171-182.
  * [7] H. Al-Zoubi, S. Al-Zu’bi, S. Stamatakis and H. Almimi, Ruled surfaces of finite Chen-type. J. Geom. Graph. 22 (2018), 15-20.
  * [8] H. Al-Zoubi, A. Dababneh, M. Al-Sabbagh, Ruled surfaces of finite $II$-type, WSEAS Trans. Math. 18 (2019), 1-5.
  * [9] H. Al-Zoubi, T. Hamadneh, Surfaces of revolution of finite $III$-type, arXiv: 1907.12390v2, Oct (2019).
  * [10] H. Al-Zoubi, T. Hamadneh, Quadric surfaces of coordinate finite type Gauss map, arXiv 2006.04529v1, May (2020).
  * [11] H. Al-Zoubi, T. Hamadneh, Surfaces of coordinate finite $II$-type, arXiv: 2005.05120v1, May (2020).
  * [12] H. Al-Zoubi, K. M. Jaber, S. Stamatakis, Tubes of finite Chen-type, Comm. Korean Math. Soc. 33 (2018), 581-590.
  * [13] H. Al-Zoubi, S. Stamatakis, W. Al Mashaleh and M. Awadallah, Translation surfaces of coordinate finite type, Indian J. Math. 59 (2017), 227-241.
  * [14] H. Al-Zoubi, S. Stamatakis, Ruled and quadric surfaces satisfying $\triangle^{III}\mathbf{x}=A\mathbf{x}$, J. Geom. Graph. 20 (2016), 147-157.
  * [15] H. Al-Zoubi, T. Hamadneh, M. Abu Hammad, and M. Al-Sabbagh, Tubular surfaces of finite type Gauss map, J. Geom. Graph. 25 (2021), 45-52.
  * [16] H. Al-Zoubi, F. Abdel-Fattah, M. Al-Sabbagh, Surfaces of finite $III$-type in the Euclidean 3-space, WSEAS Trans. Math. 20 (2021), 729-735.
  * [17] Ch. Baikoussis, D. E. Blair, On the Gauss map of Ruled Surfaces, Glasgow Math. J. 34 (1992), 355-359.
  * [18] Ch. Baikoussis, F. Denever, P. Emprechts, L. Verstraelen, On the Gauss map of the cyclides of Dupin, Soochow J. Math. 19 (1993), 417-428.
  * [19] Ch. Baikoussis, B.-Y. Chen, L. Verstraelen, Ruled Surfaces and tubes with finite type Gauss map, Tokyo J. Math. 16 (1993), 341-349.
  * [20] Ch. Baikoussis, L. Verstraelen, On the Gauss map of translation surfaces, Rend. Semi. Mat. Messina Ser II (in press).
  * [21] Ch. Baikoussis, L. Verstraelen, On the Gauss map of helicoidal surfaces, Rend. Semi. Mat. Messina Ser II 16 (1993), 31-42.
  * [22] B.-Y. Chen, Total mean curvature and submanifolds of finite type. Second edition, World Scientific Publisher, (2015).
  * [23] B.-Y. Chen, Some open problems and conjectures on submanifolds of finite type, Soochow J. Math. 17 (1991), 169-188.
  * [24] B.-Y. Chen, F. Dillen, L. Verstraelen, L. Vrancken, Ruled surfaces of finite type, Bull. Austral. Math. Soc. 42 (1990), 447-453.
  * [25] B.-Y. Chen, F. Dillen, Quadrics of finite type, J. of Geom. 38 (1990), 16-22.
  * [26] F. Defever, R. Deszcz and L. Verstraelen, The compact cyclides of Dupin and a conjecture of B.-Y. Chen. J. Geom. 46 (1993), 33-38.
  * [27] B. Senoussi, H. Al-Zoubi, Translation surfaces of finite type in Sol3, Comment. Math. Univ. Carolin. 61 (2020), 237–256
  * [28] B. Senoussi and M. Bekkar, Helicoidal surfaces with $\triangle^{J}r=Ar$ in 3-dimensional Euclidean space, Stud. Univ. Babes-Bolyai Math. 8 (2015), 437-448.
  * [29] S. Stamatakis, H. Al-Zoubi, On surfaces of finite Chen-type, Results. Math. 43 (2003), 181-190.
  * [30] S. Stamatakis, H. Al-Zoubi, Surfaces of revolution satisfying $\triangle^{III}\mathbf{x}=A\mathbf{x}$, J. Geom. Graph. 14 (2010), 181-186.