# Multimodal Infusion Tuning for Large Models

Hao Sun
Zhejiang University
Hangzhou, China
<EMAIL_ADDRESS>Yu Song
Ritsumeikan University
Shiga, Japan
<EMAIL_ADDRESS>Jihong Hu
Ritsumeikan University
Shiga, Japan
<EMAIL_ADDRESS>Xinyao Yu
Zhejiang University
Hangzhou, China
<EMAIL_ADDRESS>Jiaqing Liu
Ritsumeikan University
Shiga, Japan
<EMAIL_ADDRESS>Yen-Wei Chen
Ritsumeikan University
Shiga, Japan
<EMAIL_ADDRESS>Lanfen Lin
Zhejiang University
Hangzhou, China
<EMAIL_ADDRESS>
###### Abstract

Recent advancements in large-scale models have showcased remarkable
generalization capabilities in various tasks. However, integrating multimodal
processing into these models presents a significant challenge, as it often
comes with a high computational burden. To address this challenge, we
introduce a new parameter-efficient multimodal tuning strategy for large
models in this paper, referred to as Multimodal Infusion Tuning (MiT). MiT
leverages decoupled self-attention mechanisms within large language models to
effectively integrate information from diverse modalities such as images and
acoustics. In MiT, we also design a novel adaptive rescaling strategy at the
head level, which optimizes the representation of infused multimodal features.
Notably, all foundation models are kept frozen during the tuning process to
reduce the computational burden(only 2.5% parameters are tunable). We conduct
experiments across a range of multimodal tasks, including image-related tasks
like referring segmentation and non-image tasks such as sentiment analysis.
Our results showcase that MiT achieves state-of-the-art performance in
multimodal understanding while significantly reducing computational
overhead(10% of previous methods). Moreover, our tuned model exhibits robust
reasoning abilities even in complex scenarios.

Figure 1: The overview of our proposed MiT for different modality data and
tasks. For image-involved data, a pretrained CLIP encoder is employed to
extract the image representation(left). Otherwise, a trainable network is
employed to generate the modality embedding(such as acoustic and facial
features, right). The multimodal embedding is infused into the pretrained LLM
by MiT module. All of the pretrained models are kept frozen during the whole
procedure.

##  1 Introduction

Leveraging the vast scale of model sizes and training corpus, large language
models (LLMs) have showcased formidable capabilities across various real-world
tasks. The prominence of LLMs is evident in the widespread attention they have
garnered from both academia and industry, with notable examples being OPT [37]
and LLaMA [25]. However, training a new large model is highly resource-
intensive, leading to the development of numerous parameter-efficient fine-
tuning (PEFT) methods, such as LLaMA-adapter [36] and (IA)3 [19].

Currently, the majority of PEFT approaches are tailored for text-only data
[15, 19], posing a challenge in handling multimodal information. Empowering
LLMs with multimodal data, such as images and acoustics, remains a significant
challenge. To this end, some researchers have proposed integrating image
information into LLMs through PEFT, exemplified by approaches like FROMAGe
[12] and mPLUG-Owl [31]. Most of these methods tend to concatenate or prefix
the vision embedding with text tokens and feed them directly to the LLMs.
However, we identify two drawbacks in this approach. Firstly, the
concatenation of tokens significantly increases memory consumption, growing
quadratically due to the self-attention mechanism’s $L^{2}$ memory requirement
(where $L$ represents the length of tokens). Secondly, direct prefix
approaches are deemed coarse-grained, leading to insufficient interactions
among representations from multiple modalities.

To address these issues, we propose a new tuning framework termed multimodal
infusion tuning (MiT). Specifically, we decouple the self-attention of large
language models and progressively infuse the global representations of other
modalities into the keys ($K$) and values ($V$) of text embedding. The
infusion procedure is designed to be linear, ensuring low memory consumption.
Additionally, we introduce a novel adaptive rescaling strategy at the head
level to enhance collaboration between different modalities. This infusion
strategy allows us to endow LLMs with the capability to process multimodal
information. One major difference between MiT and other methods is that we
decouple self-attention in-depth, and inject multimodal information at a more
fine-grained aspect. We freeze the parameters in pretrained models and
exclusively fine-tune those newly introduced in MiT(around only 2.5% of all
parameters). Unlike previous approaches, MiT is versatile and can handle not
only image data but also other types of data, such as acoustic and facial
features. For various downstream tasks, we employ different head networks
during tuning, such as a lightweight decoder for referential segmentation and
a fully connected layer for sentiment analysis. We conduct experiments on
seven datasets encompassing three tasks, and the results demonstrate that MiT
achieves state-of-the-art performance while maintaining efficiency(our
proposed method requires only 0.47 TFLOPs).

In summary, our paper presents three key contributions:

  * •

We introduce a fine-grained tuning strategy for LLMs, named multimodal
infusion tuning. This strategy is designed to efficiently accommodate various
types of multimodal data, allowing for their progressive infusion into LLMs.

  * •

We develop an adaptive rescaling strategy at the head level, facilitating the
collaboration of information from different modalities to enhance interaction.

  * •

Our approach achieves state-of-the-art performance on seven evaluated
datasets, where the pretrained LLM is kept frozen during training and only
around 2.5% all parameters are trainable.

##  2 Related Works

###  2.1 Large Language Models

Large language models (LLMs) have demonstrated significant abilities in in-
context learning and long-term generation recently, exemplified by models such
as ChatGPT, OPT [37], and LLaMA [25]. To enhance the application of LLMs in
downstream tasks, various methods have been introduced, including adapter-
tuning [7] and prefix-tuning [15]. These methods primarily focus on fine-
tuning the model while preserving the original parameters, yielding two key
benefits: reduction in computational overhead and preservation of the LLM’s
proficiency trained on large corpora. More recently, novel Prompt Encoding for
Fine-Tuning (PEFT) methods have emerged, such as LLaMA-adapter [36] and (IA)3
[19]. Notably, (IA)3 introduces learnable vectors that are multiplied with the
keys (K) and values (V) in self-attention mechanisms, thereby reducing memory
usage. Our proposed method draws considerable inspiration from (IA)3, albeit
specifically tailored for multimodal data and tasks.

###  2.2 Multimodal Tuning of LLMs

While LLMs have demonstrated remarkable success in natural language
processing, their ability to perceive and understand other modalities, such as
images and acoustics, remains a challenge. Recent efforts have aimed to equip
LLMs with multimodal perception capabilities. For instance, Flamingo [2]
employs a frozen image encoder and integrates multimodal signals using gated
cross-attention, showcasing the potential of LLMs for understanding multiple
modalities. Building upon this, BLIP-2 [14] introduces the Q-Former module, a
sophisticated structure designed to align image and text embeddings by
extracting common semantics between modalities. This approach has found
widespread adoption in various studies due to its effectiveness. Furthermore,
Driess et al. [3] proposed PaLM-E, which directly incorporates multimodal
information as input and has exhibited significant efficacy across numerous
tasks. Similarly, FROMAGe [12] introduces linear transformations to ground
text features in the visual domain, facilitating seamless translation between
texts and images. With the release of LLaMA [25], efforts have been made to
adapt the model for multimodal tasks, as demonstrated by LLaMA-adapter [36].
While initially focused on text-based tasks, LLaMA-adapter has shown promising
multimodal performance by concatenating image embeddings with text tokens.
However, this approach leads to quadratic increases in memory consumption due
to the attention mechanism. In contrast, our method also equips LLMs with
multimodal understanding capabilities but employs a novel infusion tuning
strategy, resulting in linear memory consumption. Moreover, our approach is
versatile, capable of processing not only image data but also incorporating
information from other modalities, such as acoustic and facial features,
particularly beneficial in sentiment analysis.

##  3 Multimodal Infusion Tuning

Our method aims to seamlessly integrate multimodal information into a
pretrained LLM while keeping its parameters frozen. The overall pipeline is
illustrated in Figure 1. We introduce a tunable MiT module into the pretrained
LLM, facilitating the integration of representations from other modalities.
Through tuning, the LLM progressively acquires the capability to process
multimodal signals as MiT module is zero-initialized. The resulting interacted
textual features, combined with visual features in tasks involving
images(Figure 1-left), are then fed into various downstream networks for
predictions. Consequently, our method finds applications across diverse
domains, including referent segmentation, image-text classification, and
sentiment analysis.

Figure 2: The detailed architecture of MiT. We decouple the self-attention and
integrate the image representation LLM via element-wise multiplication and
addition. The integration is performed on both self-attention(left) and feed-
forward(right) module.

###  3.1 Architecture Design

As illustrated in Figure 2, we infuse the multimodal information in both self-
attention and feed-forward module of pretrained LLMs. We design MiT as a
plugable module, so we can leverage the capabilities of the LLM learnt from
large scale corpus.

Without loss of generality, assume that we have an global image representation
extracted by CLIP image encoder [22]: $I\in R^{d_{I}}$, where $d_{I}$ is the
corresponding feature dimension. We delve into textual self-attention and
integrate $I$ into key($K_{t}$) and value($V_{t}$). Take $V_{t}$ for example,
to fit it into the space of text representations, we employ two
transformations:

$\displaystyle I_{d}^{v}$ $\displaystyle=F^{v}_{d}(I)=I\cdot
W^{v}_{d}+b^{v}_{d},$ (1) $\displaystyle I_{a}^{v}$
$\displaystyle=F^{v}_{a}(I)=I\cdot W^{v}_{a}+b^{v}_{a},$

where $W^{v}_{d},W^{v}_{a}\in R^{d_{I}\times d_{T}}$, $b^{v}_{d},b^{v}_{a}\in
R^{d_{T}}$ are learnable vectors. Then we infuse the transformed
representation into $V_{t}$ via element-wise multiplication and addition:

$V_{t}^{{}^{\prime}}=V_{t}\cdot I_{d}^{v}+I_{a}^{v}.$ (2)

The same is true the infusion of $K_{t}$, which can be represented as:

$K_{t}^{{}^{\prime}}=K_{t}\cdot I_{d}^{k}+I_{a}^{k},$ (3)

where $I_{d}^{k}$ and $I_{a}^{k}$ are generated in the same approach as
Equation LABEL:equ:image_trans.

Numerical instabilities may emerge when integrating multimodal representations
into LLM due to the modality gap. To address this issue, we have implemented
an adaptive rescaling mechanism at the head-level. Given that current self-
attention mechanisms operate with multiple heads, the tensors
$V_{t}^{{}^{\prime}}$ and $K_{t}^{{}^{\prime}}$ are reshaped as $R^{h\times
d_{h}}$ for each token prior to computing the attention map, where $h$
represents the number of heads, and $d_{h}=d_{T}/h$ denotes the dimensionality
per head. Consequently, we have introduced a learnable rescaling vector $L\in
R^{h}$, which is combined with the similarity between the text value $V_{t}$
and the image embedding $I$ at the head level:

$\displaystyle L^{{}^{\prime}}=L+\frac{V_{t}\cdot I}{||V_{t}||\cdot||I||}.$
(4)

Then we element-wise multiply the normalized rescaling vector to the infused
value and key:

$\displaystyle V^{r}_{t}=V^{{}^{\prime}}_{T}\cdot\sigma(L^{{}^{\prime}}),$ (5)
$\displaystyle K^{r}_{t}=K^{{}^{\prime}}_{T}\cdot\sigma(L^{{}^{\prime}}),$

where $\sigma$ is the sigmoid function. The rescaled value and key, alongside
with query($Q_{T}$), are then employed to perform the self-attention:

$\displaystyle S=\mathrm{softmax}(Q_{t}K^{rT}_{t}/\sqrt{d_{T}})V^{r}_{t}.$ (6)

Finally after the self-attention, the integration is also performed in feed-
forward module:

$\displaystyle H_{t}^{{}^{\prime}}$ $\displaystyle=H_{t}\cdot
F^{f}(I)=H_{t}\cdot(I\cdot W^{f}+b^{f}),$ (7)

where $W^{f}\in R^{d_{I}\times d_{T^{\prime}}}$, $b^{f}\in
R^{d_{T^{\prime}}}$, $H_{t}\in R^{d_{T^{\prime}}}$ is the activated textual
hidden embedding in LLM, and $d_{T^{\prime}}$ is the corresponding hidden
size.

In this approach, we can seamlessly integrate multimodal information with a
linear computational burden, yet in a more fine-grained manner. Notably, all
learnable vectors to be added are initialized to 0 (e.g., $I_{d}^{v}$), and
all vectors to be multiplied are initialized to 1 (e.g., $I_{a}^{v}$) in MiT.
Therefore, the LLMs can progressively acquire the ability for multimodal
understanding with minimal computational load.

Figure 3: The formation of our employed dataset and tasks: referring
segmentation(left), image-text classification(middle), and sentiment
analysis(right). For different dataset and tasks, we design different
templates, so as to excavate the capabilities of LLMs obtained in pretraining.

###  3.2 Downstream Task Tuning

We fine-tune our MiT on three tasks for multimodal understanding: refer
segmentation, image-text classification, and multimodal sentiment analysis.
Among these tasks, sentiment analysis does not involve images, while the other
two require image data. For tasks involving images, we utilize a frozen CLIP
encoder to generate global semantic representations and features of images at
different levels, as illustrated in Figure 1. Alternatively, for tasks not
involving images, we employ a trainable transformer feature encoder to extract
multimodal embeddings. Since the LLMs employed in our approach are decoder-
only structures, we extract the semantic representations for downstream tasks
from the last token of the entire sequences.

For refer segmentation, the goal is to segment the object from the image
according to a descriptive. Under this scenario, we design a light-weight
decoder, which takes infused text embedding and multi-level image features as
input, and generate the predicted mask. The DICE loss [21] is employed in our
framework:

$\displaystyle\mathcal{L}_{seg}=\mathcal{L}_{DICE}=1-\frac{2|\hat{y}\cap
y_{gt}|}{|\hat{y}|+|y_{gt}|}$ (8)

where $\hat{y}$ is the predicted mask and $y_{gt}$ is the ground truth. For
image-text classification, the global image representation and infused text
embedding are concatenated, followed by a prediction head. Like general
approaches, the cross-entropy loss is employed for this task:

$\displaystyle\mathcal{L}_{cls}=\mathcal{L}_{CE}=-\sum_{i=1}^{n}y_{gt,i}\log(\hat{y}_{i})$
(9)

where $n$ is the number of training samples in a mini-batch. For multimodal
sentiment analysis, we only utilize the infused text embedding for prediction,
as text is dominant in this task(shown in Figure 1). As sentiment analysis is
a regression task(sentiment tendency), we employ the RMSE as the loss function
following previous works [6, 4]:

$\displaystyle\mathcal{L}_{msa}=\mathcal{L}_{RMSE}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(\hat{y}_{i}-y_{gt,i})}.$
(10)

###  3.3 Implementation Details

We evaluate our method on RefCOCO [32], RefCOCO+ [32], and G-Ref [20] for
refer segmentation. There are 19994, 19992, and 26711 images, with 50000,
49856, and 54822 references, respectively. For image-text classification, we
employ two datasets: UPMC-Food101 [27] and SNLI-VE [29], which contain 90840
and 565286 image-text pairs, respectively. UPMC-Food101 provides 101-category
classification task while SNLI-VE aims to reason the semantic
relationship(entailment, neutral, or contradiction) between text hypothesis
and image premise. As for the sentiment analysis, we use two popular dataset:
MOSI [33] and MOSEI [35], in which the sentiment tendency are annotated within
[-3, +3]. The goal is to predict the sentiment tendency based on corresponding
texts, acoustic and facial features.

We use the popular LLaMA [25] with 7B parameters as the textual foundation
model. For image branch, we employ the CLIP ViT-L-336 model [22] to generate
the global representations. In MiT, we infuse the multimodal representation
into last certain layers in LLM, and the detailed effect are discussed in
Section 5.1. There are three layers in the light-weight decoder for refer
segmentation to maintain efficiency. In sentiment analysis task, three-layer
transformer blocks are employed to encoder acoustic and facial features.
Inherited from pretrained models, $d_{I}$, $d_{T}$, and $d_{T^{\prime}}$ are
set to 768, 4096, and 11008, respectively.

Our models are built under the PyTorch framework and tuned with half-
precision(bfloat16 [1]). As only 2.5% parameters are tunable and no extra
memory are required by self-attention, MiT is memory and compute efficient. We
use the Adam optimizer and set the learning rate to 0.00004. We train the
model for 30 epochs and the learning rate is decayed by 0.1 every 10 epochs.
Our experiments are conducted on two NVIDIA RTX 4090 GPUs. The batch size is
set to 8 on each GPU. The whole tuning process takes 20 hours on RefCOCO and 1
hours on MOSI.

##  4 Experimental Results

To better utilize the capabilities of LLM obtained from pretraining, we format
the original inputs in the evaluated dataset, as shown in Figure 3. For
example, we format the input sentence as Segment the {description} according
to the text. #Segmentation: for referring segmentation. We use overall
Intersection-over-Unions(oIoU) as metrics for referring segmentation following
previous works. Accuracy and F1-score are utilized as metrics for
classification. For sentiment analysis, we use mean absolute error(MAE),
Pearson correlation coefficient(Corr) and corresponding accuracy as metrics.

Table 1: The results of our method on referring segmentation benchmarks: RefCOCO, RefCOCO+, and RefCOCOg. The metrics in the table is oIoU. LLM/TFLOPs means the LLM employed and the methods’ overall computation requirement. RefCOCOg(U) means the UMD partition of RefCOCOg dataset. Method | LLM/TFLOPs | RefCOCO | RefCOCO+ | RefCOCOg(U)
---|---|---|---|---
val | testA | testB | val | testA | testB | val | test
CRIS [28] | - | 70.5 | 73.2 | 66.1 | 62.3 | 68.1 | 53.7 | 59.9 | 60.4
LAVT [30] | - | 72.7 | 75.8 | 68.8 | 62.1 | 68.4 | 55.1 | 61.2 | 62.1
ReLA [18] | - | 73.8 | 76.5 | 70.2 | 66.0 | 71.0 | 57.7 | 65.0 | 66.0
SEEM [38] | - | - | - | - | - | - | - | 64.6 | -
LISA [13] | LLaMA2-13B/10.24 | 74.1 | 76.5 | 71.1 | 62.4 | 67.4 | 56.5 | 66.4 | 68.5
PixelLM [23] | LLaMA2-13B/6.65 | 73.0 | 76.5 | 68.2 | 66.3 | 71.7 | 58.3 | 69.3 | 70.5
MiT(ours) | LLaMA2-7B/0.47 | 73.1 | 77.1 | 68.4 | 66.9 | 72.2 | 57.6 | 65.4 | 68.5

###  4.1 Results on Referring Segmentation

Table 1 shows the results on referring segmentation benchmarks. We compare our
MiT with previous state-of-the-art methods, including LAVT [30] and LISA [13].
LAVT is composed of a vision transformer with a BERT model [8] for textual
embedding extraction, and the whole vision branch is trainable during
optimizing. LISA is based on large models, which integrates image embeddings
into language models via a <SEG>token. As we can see from the results, we can
reach competitive results with LISA, but with a much lower computational
burden. There are three reasons for this phenomenon: 1) LISA uses a larger
language foundation model than us(we use LLaMA-7B but LISA employs a 13B
model); 2) LISA employs the SAM model [11] for segmentation(SAM requires 10x
times memory than CLIP during inferring); 3) the linear consumption of
infusion during multimodal tuning. Some segmentation cases are shown in Figure
5, illustrating the effectiveness of of proposed method.

Table 2: This results of our method on image-text classification benchmarks. The accuracy is utilized as metrics in the table. Method | UPMC-Food101 | SNLI-VE
---|---|---
MMBT [10] | 92.10 | 74.69
MaPLe [9] | 90.80 | 71.52
PromptFuse [17] | 82.21 | 64.53
PMF [16] | 91.51 | 71.92
MiT(ours) | 93.88 | 74.71

###  4.2 Results on Image-text Classification

On image-text classification benchmarks, we can reach the best performance on
each metric, as illustrated in Table 2. Different from referring segmentation,
image-text classification does not have a decoder, but a prediction head
instead. We employ MaPLe [9] and PMF [16] as our baselines for this task. Both
MaPLe and PMF utilize existing large models while introducing new prompting
strategies, achieved through concatenation or prefixing. However, MiT, being
more fine-grained and delving into self-attention mechanisms, facilitates
superior multimodal interactions, resulting in enhanced performance on
evaluated benchmarks.

Table 3: The results of our method on MOSI and MOSEI datasets. Acc-2 and Acc-7
mean the accuracy of binary and seven-class classification. Acc-2 and F1 are
classification metrics. MAE, Corr, and Acc-7 are regression metrics.

Models | MOSI | MOSEI
---|---|---
Acc-2/F1($\uparrow$) | MAE($\downarrow$) | Corr($\uparrow$) | Acc-7($\uparrow$) | Acc-2/F1($\uparrow$) | MAE($\downarrow$) | Corr($\uparrow$) | Acc-7($\uparrow$)
TFN[34] | 73.9/73.4 | 0.970 | 0.633 | 32.1 | 82.5/82.1 | 0.593 | 0.700 | 50.2
MulT[26] | 83.0/82.8 | 0.871 | 0.698 | 40.0 | 82.5/82.3 | 0.580 | 0.703 | 51.8
MISA[26] | 82.1/82.0 | 0.817 | 0.748 | 41.4 | 84.9/84.8 | 0.557 | 0.748 | 51.7
BBFN[4] | 84.3/84.3 | 0.776 | 0.775 | 45.0 | 86.2/86.1 | 0.529 | 0.767 | 54.8
CubeMLP[24] | 85.6/85.5 | 0.770 | 0.767 | 45.5 | 85.1/84.5 | 0.529 | 0.760 | 54.9
MMIM[5] | 84.1/84.0 | 0.700 | 0.800 | 46.6 | 82.2/82.6 | 0.526 | 0.772 | 54.2
MiT(ours) |  86.5/86.5 | 0.632 | 0.858 | 49.0 |  87.2/87.2 | 0.509 | 0.788 | 55.2

###  4.3 Results on Sentiment Analysis

We conduct the sentiment analysis on MOSI and MOSEI dataset. Different from
image-involved tasks, sentiment analysis concentrates more on text. Therefore,
the textual embeddings are directly sent to predictors without features from
other modalities after the multimodal infusion. The results are shown in Table
3, our method is much better than previous methods by a large margin. One of
the important reasons is that we use a large language model, while previous
approaches(like MISA [6] and CubeMLP [24]) generally use models such as BERT
[8]. Despite the effectiveness of large language models, our ablation
experiments can prove that MiT also plays a decisive role(Section 5.1). The
experiments show that LLM can not only accept common image data, but also
process uncommon acoustic and facial information.

##  5 Analysis

To further study the effectiveness of MiT, we conduct some additional
experiments with corresponding analysis in this section.

Table 4: The ablation study of each module and employed vision encoder in MiT.
K/V-Inf means the infusion of K and V. FF-Inf means the infusion of feed-
forward part. A.R means the adaptive rescaling on head level. For vision
encoders, P16 means the patch size is 16. The ablations are conducted on
RefCOCO dataset. Ablation of MiT modules
---
K/V-Inf. | FF-Inf. | A.R. | val | testA | testB
|  |  | 51.4 | 60.1 | 50.0
| $\checkmark$ | $\checkmark$ | 69.4 | 72.1 | 65.0
$\checkmark$ |  | $\checkmark$ | 72.9 | 75.6 | 67.7
$\checkmark$ | $\checkmark$ |  | 72.2 | 75.4 | 67.4
$\checkmark$ | $\checkmark$ | $\checkmark$ | 73.1 | 77.1 | 68.4
Ablation of Vision Encoders
CLIP-Base-P16-224x | 71.1 | 75.3 | 66.1
CLIP-Large-P14-224x | 72.0 | 76.0 | 66.2
CLIP-Large-P14-336x | 73.1 | 77.1 | 68.4

###  5.1 Ablation Studies

Modules of MiT. There are several components in our proposed MiT, including
K/V infusion, adaptive rescaling, feed-forward infusion, etc. The ablation
study of each part on RefCOCO benchmark are shown in Table 4. As we can see
from the results, K/V infusion plays the most important role in MiT. When the
adaptive head rescaling is removed, the performance of the model will also
decrease accordingly, demonstrating the effect of this part. Besides, infusing
in feed-forward module also plays a positive role for the final results.

Selection of Image Encoders. In our method, image encoders are crucial for
segmentation, as they contains essential spatial information. Therefore, we
conduct the ablation of different image backbones. As we can draw from Table6,
larger backbones tends to get better performance. The performance reaches the
best when we use the CLIP-Large-P14-336x vision backbone. However, we cannot
perform the ablation on text foundation model because of the hardware
limitation(LLaMA-7B is the largest language model we can utilize). We believe
that after replacing the larger text foundation model, the performance will be
improved correspondingly.

Table 5: The ablation study of infusion layers. There are 32 layers in LLaMA-7B. The experiments are conducted on RefCOCO. The performance drops when infusing in the first 1/3 layers. Infusion layers | RefCOCO
---|---
val | testA | testB
1,9,17,25,32 | 69.1 | 74.2 | 66.0
9,17,25,32 | 71.9 | 75.5 | 66.8
21,25,29,32 | 72.6 | 76.8 | 67.9
17,21,25,29,32 | 72.7 | 76.7 | 68.0
13,17,21,25,29,32 | 73.1 | 77.1 | 68.4

Scales of infusion scales. In our proposed method, we can infuse the
multimodal information at arbitrary layers of LLM. There are 32 layers in
LLaMA-7B in all, and the ablation results are shown in Table 5. We find that
infusing in the last 1/3 layers of LLM achieves better results. When we
infusing image information in the first 12 layers, the performance drops
dramatically. The results reaches the best when we infusing the multimodal
information between layer 13 and 32 with interval 4. This phenomenon can be
explained by the fact that deeper layers contain richer semantic information
and are thus more suitable for modality fusion.

Table 6: The ablation of involved modalities for sentiment analysis. T, A, and
F indicate textual, acoustic, and facial modalities, respectively. The
experiments are conducted on MOSI dataset.

Involved modalities | MOSI
---|---
T. | A. | F. | MAE | Corr | Acc-2/F1 | Acc-7
$\checkmark$ |  |  | 0.701 | 0.744 | 82.2/82.2 | 45.3
$\checkmark$ |  | $\checkmark$ | 0.677 | 0.813 | 83.9/84.0 | 46.6
$\checkmark$ | $\checkmark$ |  | 0.670 | 0.832 | 85.2/85.0 | 47.2
$\checkmark$ | $\checkmark$ | $\checkmark$ | 0.632 | 0.858 | 86.5/86.5 | 49.0

Involved modalities for sentiment analysis. Although text dominates sentiment
analysis tasks, the influence of other modalities and interaction methods
cannot be ignored. The quantitative experimental results are illustrated in
Table 6 on MOSI dataset. We find the worst performance without acoustic or
facial signals, but still better than current state-of-the-art methods. When
MiT is removed, meaning multimodal signals are added directly to text
embedding, the performance is still behind the complete infusion method. These
results prove that our performance improvement in sentiment analysis comes not
only from LLM, but also from our proposed MiT.

Figure 4: The impact of text length to last-token and <SEG>-token schema on
referring segmentation. The experiments are conducted on the testA set of
RefCOCO.

Figure 5: The case visualization of complex reasoning(up) and multimodal in-
context understanding(down). In complex reasoning, we describe objects more
implicitly instead of identifying the object directly. For multimodal in-
context understanding, we conduct the experiments in a conversation manner,
which also get great segmentation results.

###  5.2 Difference from ¡TASK¿ Token in Multimodal Tuning

Although MiT reaches the state-of-the art performance, it still has some
difference with previous approaches. One of the main difference is the
<TASK>token, which has been used in previous works [6, 23]. Specifically, for
segmentation, they newly introduce a learnable token <SEG>to the pretrained
language vocabulary and use the corresponding embedding for segmentation.
Instead of using an entirely new token, we utilize the last token’s embedding
in decoder-only LLM for different tasks.

In order to explore the difference between the two methods, we also introduce
different task tokens(e.g., <SEG>for segmentation and <CLS>for classification)
and conduct respective experiments. Interestingly, we find that the
performance gap between them is sensitive to the length of text
inputs(including the formatted prompt). The results are shown in Figure 4.
Through the experiments, we find that when using shorter text input, the last-
token method is better; when using longer text input, the <SEG>-token method
performs better. Through our experiments, we observe that when utilizing
shorter text inputs, the last-token method yields superior performance,
whereas with longer text inputs, the <SEG>-token method performs better.
Despite the variations in performance, our proposed MiT demonstrates
effectiveness across different methods, highlighting its robustness. On one
hand, text lengths are relatively short in the benchmarks we evaluated, and on
the other hand, for simplicity, we adopt the last-token scheme for downstream
tasks.

###  5.3 Complex Scenario Reasoning

In our experiments, we observe that our fine-tuned model demonstrates notable
multimodal complex reasoning capabilities. Despite not receiving direct
descriptions of the objects to be segmented, the model showcases the ability
to segment desired parts through reasoning. For instance, as illustrated in
Figure 5(up), while we provide the model with the description human’s best
friend instead of explicitly stating dog, the model successfully segments the
dog. The capability is inherited from pretrained LLMs, and maintained by the
progressive tuning strategy. Additionally, leveraging half-precision training,
the fine-tuned model achieves inference on a single NVIDIA RTX 3090 GPU in
69ms, demonstrating efficiency in both time and memory usage.

To examine the multimodal in-context understanding ability of our model, we
further conduct the experiments in conversation scenarios. As shown in Figure
5(down), the model can understand the contextual multimodal information(down-
center). However, the performance seems worse than directly segmenting the
target(down-right), which is one the the drawback of our MiT. This may be due
to the fact that we did not introduce text supervision task(e.g., next-token
supervision). In the future, we plan to involve the text tasks into tuning, so
as to improve the ability of multimodal in-context understanding.

##  6 Conclusion

In this paper, we proposed a new tuning method, named as multimodal infusion
tuning. In our proposed MiT, we infuse the multimodal information into LLMs
with a linear complexity. In addition, we also introduced an adaptive
rescaling strategy to eliminate the numerical instabilities. We have conducted
the experiments on seven datasets with three tasks. We reach the state-of-the-
art performance but with a much lower time and memory consumption. Further
analysis also reveal the multimodal understanding ability of our tuned models,
including complex reasoning and multimodal in-context understanding. However,
one limitation in our work is the lack of complex text understanding, which
can further improve the in-context understanding capability. In the future, we
are going to continue this research and involve more complex understanding
tasks, including long-text reasoning and multitask tuning.

## Acknowledgments

This work was partially supported by JST SPRING to Yu Song, grant number
JPMJSP2101.

## References

  * Abadi et al. [2016] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al.  Tensorflow: Large-scale machine learning on heterogeneous distributed systems.  _arXiv preprint arXiv:1603.04467_ , 2016.
  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Driess et al. [2023] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al.  Palm-e: An embodied multimodal language model.  _arXiv preprint arXiv:2303.03378_ , 2023.
  * Han et al. [2021a] Wei Han, Hui Chen, Alexander Gelbukh, Amir Zadeh, Louis-philippe Morency, and Soujanya Poria.  Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis.  In _Proceedings of the 2021 International Conference on Multimodal Interaction_ , pages 6–15, 2021a.
  * Han et al. [2021b] Wei Han, Hui Chen, and Soujanya Poria.  Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9180–9192, 2021b.
  * Hazarika et al. [2020] Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria.  Misa: Modality-invariant and-specific representations for multimodal sentiment analysis.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pages 1122–1131, 2020.
  * Houlsby et al. [2019] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In _International Conference on Machine Learning_ , pages 2790–2799. PMLR, 2019.
  * Kenton and Toutanova [2019] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT_ , pages 4171–4186, 2019.
  * Khattak et al. [2023] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan.  Maple: Multi-modal prompt learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19113–19122, 2023.
  * Kiela et al. [2019] Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Ethan Perez, and Davide Testuggine.  Supervised multimodal bitransformers for classifying images and text.  _arXiv preprint arXiv:1909.02950_ , 2019.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Koh et al. [2023] Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried.  Grounding language models to images for multimodal inputs and outputs.  In _International Conference on Machine Learning_ , pages 17283–17300. PMLR, 2023.
  * Lai et al. [2023] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia.  Lisa: Reasoning segmentation via large language model.  _arXiv preprint arXiv:2308.00692_ , 2023.
  * Li et al. [2023a] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023a.
  * Li and Liang [2021] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, 2021.
  * Li et al. [2023b] Yaowei Li, Ruijie Quan, Linchao Zhu, and Yi Yang.  Efficient multimodal fusion via interactive prompting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2604–2613, 2023b.
  * Liang et al. [2022] Sheng Liang, Mengjie Zhao, and Hinrich Schütze.  Modular and parameter-efficient multimodal fusion with prompting.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2976–2985, 2022.
  * Liu et al. [2023] Chang Liu, Henghui Ding, and Xudong Jiang.  Gres: Generalized referring expression segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 23592–23601, 2023.
  * Liu et al. [2022] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.  _Advances in Neural Information Processing Systems_ , 35:1950–1965, 2022.
  * Mao et al. [2016] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy.  Generation and comprehension of unambiguous object descriptions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 11–20, 2016.
  * Milletari et al. [2016] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi.  V-net: Fully convolutional neural networks for volumetric medical image segmentation.  In _2016 fourth international conference on 3D vision (3DV)_ , pages 565–571. Ieee, 2016.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Ren et al. [2023] Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin.  Pixellm: Pixel reasoning with large multimodal model.  _arXiv preprint arXiv:2312.02228_ , 2023.
  * Sun et al. [2022] Hao Sun, Hongyi Wang, Jiaqing Liu, Yen-Wei Chen, and Lanfen Lin.  Cubemlp: An mlp-based model for multimodal sentiment analysis and depression estimation.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 3722–3729, New York, NY, USA, 2022. Association for Computing Machinery.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Tsai et al. [2019] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov.  Multimodal transformer for unaligned multimodal language sequences.  In _Proceedings of the conference. Association for Computational Linguistics. Meeting_ , pages 6558–6569. NIH Public Access, 2019.
  * Wang et al. [2015] Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord, and Frederic Precioso.  Recipe recognition with large multimodal food dataset.  In _2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)_, pages 1–6. IEEE, 2015.
  * Wang et al. [2022] Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu.  Cris: Clip-driven referring image segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 11686–11695, 2022.
  * Xie et al. [2018] Ning Xie, Farley Lai, Derek Doran, and Asim Kadav.  Visual entailment task for visually-grounded language learning.  _arXiv preprint arXiv:1811.10582_ , 2018.
  * Yang et al. [2022] Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr.  Lavt: Language-aware vision transformer for referring image segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18155–18165, 2022.
  * Ye et al. [2023] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al.  mplug-owl: Modularization empowers large language models with multimodality.  _arXiv preprint arXiv:2304.14178_ , 2023.
  * Yu et al. [2016] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg.  Modeling context in referring expressions.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14_ , pages 69–85. Springer, 2016.
  * Zadeh et al. [2016] Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency.  Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages.  _IEEE Intelligent Systems_ , 31(6):82–88, 2016.
  * Zadeh et al. [2017] Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency.  Tensor fusion network for multimodal sentiment analysis.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1103–1114, 2017.
  * Zadeh et al. [2018] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency.  Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2236–2246, 2018.
  * Zhang et al. [2023] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.  _arXiv preprint arXiv:2303.16199_ , 2023.
  * Zhang et al. [2022] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_ , 2022.
  * Zou et al. [2023] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee.  Segment everything everywhere all at once.  _arXiv preprint arXiv:2304.06718_ , 2023.


$\operatorname{Mod}(\Sigma_{g,1})$ on $\mathcal{X}(\mathbf{\Sigma}_{g}^{*})$,
$\widehat{\overline{\mathcal{X}}}(\mathbf{\Sigma}_{g}^{*})$ and
${\overline{\mathcal{X}}}(\mathbf{\Sigma}_{g}^{*})$ in such a way that
$\overline{\pi}\colon\widehat{\overline{\mathcal{X}}}(\mathbf{\Sigma}_{g}^{*})\to{\overline{\mathcal{X}}}(\mathbf{\Sigma}_{g}^{*})$
is equivariant. More explicitly, the action of
$\operatorname{Mod}(\Sigma_{g,1})$ on ${\mathcal{X}}(\mathbf{\Sigma}_{g}^{*})$
is given by

$(\phi\cdot\rho)(\gamma):=\rho(\phi(\gamma)),\qquad\text{for all
}\phi\in\operatorname{Mod}(\Sigma_{g,1}),\ \gamma\in\pi_{1}(\Sigma_{g,1}),\
\rho\colon\ \pi_{1}(\Sigma_{g,1})\to\operatorname{SL}_{2}.$

Since every mapping class in $\operatorname{Mod}(\Sigma_{g,1})$ leaves
$\gamma_{\partial}$ invariant, then $\mu(\phi\cdot\rho)=\mu(\rho)$. Recall
from equation (4.1) that an element of
$\widehat{\overline{\mathcal{X}}}(\mathbf{\Sigma}_{g}^{*})$ is a pair
$\widehat{\rho}=(\rho,z)$ where
$\rho\colon\pi_{1}(\Sigma_{g,1})\to\operatorname{SL}_{2}$,
$z\in\mathbb{C}^{*}$ are such that
$\rho(\gamma_{\partial})=\left(\begin{smallmatrix}0&-z^{-N}\\\
z^{N}&d\end{smallmatrix}\right)$ for some $d\in\mathbb{C}$.

Since the central boundary element $\alpha_{\partial}$ is also preserved by
every mapping class in $\operatorname{Mod}(\Sigma_{g,1})$, the action of
$\operatorname{Mod}(\Sigma_{g,1})$ on
$\widehat{\overline{\mathcal{X}}}(\mathbf{\Sigma}_{g}^{*})$ is given by the
formula

$\phi\cdot(\rho,z)=(\phi\cdot\rho,z)\qquad\text{for all
}\phi\in\operatorname{Mod}(\Sigma_{g,1}),\
(\rho,z)\in\widehat{\overline{\mathcal{X}}}(\mathbf{\Sigma}_{g}^{*}).$

###  5.2 Construction of mapping class groups representations

Let
$\mathcal{X}^{0}\subset\mathcal{X}_{\operatorname{SL}_{2}}(\mathbf{\Sigma}_{g}^{*})$
be the subset of representations
$\rho\colon\pi_{1}(\Sigma_{g,1})\to\operatorname{SL}_{2}$ such that
$\mu(\rho)\in\operatorname{SL}_{2}^{0}$, i.e., $\mathcal{X}^{0}$ is the
Azumaya locus of $\mathcal{S}_{A}(\mathbf{\Sigma}_{g}^{*})$ by Corollary 4.14.
For each $\rho\in\mathcal{X}^{0}$, fix an irreducible representation

$r_{\rho}\colon\
\mathcal{S}_{A}(\mathbf{\Sigma}_{g}^{*})\to\operatorname{End}(V(\rho))$

with induced central character $r_{\rho}$: so $r_{\rho}$ is unique up to
unique isomorphism by Corollary 4.14 and has dimension $N^{3g}$ by Theorem
2.16. For $\phi\in\operatorname{Mod}(\Sigma_{g,1})$, consider the
representation $\phi\bullet
r_{\rho}\colon{\mathcal{S}}_{A}(\mathbf{\Sigma}_{g}^{*})\to\operatorname{End}(V({\rho}))$
defined by

$(\phi\bullet r_{\rho})(X):=r_{\widehat{\rho}}(\phi\cdot X),\qquad\text{for
all }X\in{\mathcal{S}}_{A}(\mathbf{\Sigma}_{g}^{*}).$

Since the representation $\phi\bullet r_{\rho}$ has dimension $N^{3g}$, it is
irreducible and since its induced central character is $\phi\cdot{\rho}$,
there exists an isomorphism $L_{\rho}(\phi)\colon
V(\rho)\xrightarrow{\cong}V(\phi\cdot{\rho})$, unique up to multiplication by
an invertible scalar, such that:

$\phi\bullet
r_{\rho}(X)=L_{\rho}(\phi)(r_{\rho}(X))L_{\rho}(\phi)^{-1},\qquad\text{for all
}X\in{\mathcal{S}}_{A}(\mathbf{\Sigma}_{g}^{*}).$ (5.1)

Note that, for $\phi_{1},\phi_{2}\in\operatorname{Mod}(\Sigma_{g,1})$ then by
unicity, we have

$L_{\phi_{1}\cdot{\rho}}(\phi_{2})\circ
L_{\rho}(\phi_{1})=cL_{\rho}(\phi_{2}\circ\phi_{1}),\qquad\text{for some
}c\in\mathbb{C}^{*}.$ (5.2)

Similarly, for each
$\widehat{\rho}=(\rho,z)\in\widehat{\overline{\mathcal{X}}}(\mathbf{\Sigma}_{g}^{*})$,
we fix one irreducible representation

$r_{\widehat{\rho}}\colon\
\overline{\mathcal{S}}_{A}(\mathbf{\Sigma}_{g}^{*})\to\operatorname{End}(V(\widehat{\rho})),$

whose induced character on the center is $\widehat{\rho}$. By Theorem 1.1,
such a representation is unique up to isomorphism and has dimension $N^{3g-1}$
by Theorem 2.15. Define the intertwiner $L_{\widehat{\rho}}\colon
V(\widehat{\rho})\to V(\phi\cdot\widehat{\rho})$ in the same manner, i.e., by
the formula

$r_{\widehat{\rho}}(X^{\phi})=L_{\widehat{\rho}}(\phi)(r_{\rho}(X))L_{\widehat{\rho}}(\phi)^{-1},\qquad\text{for
all }X\in\overline{{\mathcal{S}}}_{A}(\mathbf{\Sigma}_{g}^{*}).$

######  Definition 5.1.

Let $G\subset\operatorname{Mod}(\Sigma_{g,1})$ be a subgroup and
$\mathcal{O}\subset\operatorname{Hom}(\pi_{1}(\Sigma_{g,1}),\operatorname{SL}_{2})$
a finite $G$-orbit included in some leaf $\mu^{-1}(g)$ for
$g=\left(\begin{smallmatrix}a&b\\\ c&d\end{smallmatrix}\right)$.

  * •

If $a\neq 0$, consider the finite-dimensional space

$W(\mathcal{O}):=\oplus_{{\rho}\in\mathcal{O}}V({\rho})$

and let $\pi\colon G\to\operatorname{PGL}(W(\mathcal{O}))$ be the projective
finite-dimensional representation defined by

$\pi(g):=\oplus_{\widehat{\rho}\in\mathcal{O}}L_{{\rho}}(g),\qquad\text{for
all }g\in G.$

  * •

If $a=0$, further choose a scalar $z\in\mathbb{C}^{*}$ such that $z^{N}=c$ and
consider the $G$ orbit
$\overline{\mathcal{O}}_{z}=\\{(\rho,z),\rho\in\mathcal{O}\\}\subset\widehat{\overline{\mathcal{X}}}(\mathbf{\Sigma}_{g}^{*})$.
Define the finite-dimensional space

$W(\widehat{\mathcal{O}}_{z}):=\oplus_{\widehat{\rho}\in\widehat{\mathcal{O}}_{z}}V(\widehat{\rho}).$

Let $\pi\colon
G\to\operatorname{PGL}\big{(}W(\widehat{\mathcal{O}}_{z})\big{)}$ be the
projective finite-dimensional representation defined by

$\pi(g):=\oplus_{\widehat{\rho}\in\widehat{\mathcal{O}}_{z}}L_{\widehat{\rho}}(g),\qquad\text{for
all }g\in G.$

######  Example 5.2.

  1. 1.

For instance, for $H\subset\operatorname{SL}_{2}$ a finite subgroup, then

$\mathcal{O}_{H}:=\operatorname{Hom}(\pi_{1}(\Sigma_{g,1}),H)\subset\operatorname{Hom}(\pi_{1}(\Sigma_{g,1}),\operatorname{SL}_{2})$

is a finite $\operatorname{Mod}(\Sigma_{g,1})$-orbit. In particular, taking
$H=\\{{\rm id}\\}$ for which $\mathcal{O}_{H}$ is the singleton formed by the
trivial representation, then we obtain representations of
$\operatorname{Mod}(\Sigma_{g,1})$ of dimension $N^{3g}$.

  2. 2.

Similarly, taking for $G$ the Torelli subgroup and for $\mathcal{O}$ a
singleton formed by a diagonal representation, i.e., a representation that
factorizes through the diagonal embedding as
$\rho\colon\pi_{1}(\Sigma_{g,1})\to\mathbb{C}^{*}\xrightarrow{\varphi}\operatorname{SL}_{2}$,
we get representations of the Torelli group of dimensions $N^{3g}$.

######  Lemma 5.3.

For a finite $G$ orbit $\mathcal{O}$ in some leaf $\mu^{-1}(g)$ with
$g\in\operatorname{SL}_{2}^{1}$, then the representation
$\big{(}\pi,W\big{(}\widehat{\mathcal{O}}_{z}\big{)}\big{)}$ does not depend
on the choice of $z$ up to isomorphism.

###### Proof.

Recall from Section 4.2 the toric action of $\mathbb{C}^{*}$ on
$\overline{\mathcal{S}}_{A}(\mathbf{\Sigma}_{g}^{*})$ given by the co-action

$\Delta_{\rm toric}^{L}\colon\
\overline{\mathcal{S}}_{A}(\mathbf{\Sigma}_{g}^{*})\xrightarrow{\Delta^{L}_{a}}\mathcal{O}_{q}[\operatorname{SL}_{2}]\otimes\overline{\mathcal{S}}_{A}(\mathbf{\Sigma}_{g}^{*})\xrightarrow{\varphi\otimes{\rm
id}}\mathbb{C}[X^{\pm
1}]\otimes\overline{\mathcal{S}}_{A}(\mathbf{\Sigma}_{g}^{*}).$

Its induced action on the center of
$\overline{\mathcal{S}}_{A}(\mathbf{\Sigma}_{g}^{*})$ is given by

$\lambda\bullet(\rho,z):=\left(\lambda^{N}\cdot\rho,\lambda
z\right),\qquad\text{for all }\lambda\in\mathbb{C}^{*},\
(\rho,z)\in\overline{\mathcal{X}}(\mathbf{\Sigma}_{g}^{*}),$

where

$\big{(}\lambda^{N}\cdot\rho\big{)}(\gamma)=\begin{pmatrix}\lambda^{N}&0\\\
0&\lambda^{-N}\end{pmatrix}\rho(\gamma)\begin{pmatrix}\lambda^{-N}&0\\\
0&\lambda^{N}\end{pmatrix}.$

For $\hat{\rho}=(\rho,z)\in\overline{\mathcal{X}}(\mathbf{\Sigma}_{g}^{*})$
and $\lambda\in\mathbb{C}^{*}$, the toric action of $\lambda$ on
$\overline{\mathcal{S}}_{A}(\mathbf{\Sigma}_{g}^{*})$ induces an isomorphism
of algebras

$F_{\lambda,\hat{\rho}}\colon\
\operatorname{End}(V(\hat{\rho}))\cong{\raisebox{1.99997pt}{$\overline{\mathcal{S}}_{A}(\mathbf{\Sigma}_{g}^{*})$}\left/\raisebox{-1.99997pt}{$\mathcal{I}_{\hat{\rho}}$}\right.}\xrightarrow{\lambda\bullet\cdot}{\raisebox{1.99997pt}{$\overline{\mathcal{S}}_{A}(\mathbf{\Sigma}_{g}^{*})$}\left/\raisebox{-1.99997pt}{$\mathcal{I}_{\lambda\bullet\hat{\rho}}$}\right.}\cong\operatorname{End}(V(\lambda\bullet\hat{\rho})).$

Since the group of outer morphisms of a matrix algebra is trivial, the
isomorphism $F_{\lambda,\hat{\rho}}$ is inner, so there exists
$f_{\lambda,\hat{\rho}}\colon
V(\hat{\rho})\xrightarrow{\cong}V(\lambda\bullet\hat{\rho})$ such that

$F_{\lambda,\hat{\rho}}(X)=f_{\lambda,\hat{\rho}}Xf_{\lambda,\hat{\rho}}^{-1},\qquad\text{for
all }X\in\operatorname{End}(V(\hat{\rho})).$

For $G\subset\operatorname{Mod}(\Sigma_{g,1})$ and
$\widehat{\mathcal{O}}\subset\overline{\mathcal{X}}(\mathbf{\Sigma}_{g}^{*})$
a finite $G$-orbit, define an isomorphism

$f_{\lambda,\widehat{\mathcal{O}}}\colon\
W\big{(}\widehat{\mathcal{O}}\big{)}\xrightarrow{\cong}W\big{(}\lambda\bullet\widehat{\mathcal{O}}\big{)},\qquad
f_{\lambda,\widehat{\rho}}:=\oplus_{\widehat{\rho}\in\widehat{\mathcal{O}}}f_{\lambda,\widehat{\mathcal{O}}}.$

By construction, the isomorphism $f_{\lambda,\widehat{\mathcal{O}}}$ is
$G$-equivariant, so it provides an isomorphism between the representations
$\big{(}\pi,W\big{(}\widehat{\mathcal{O}}\big{)}\big{)}$ and
$\big{(}\pi,W\big{(}\lambda\bullet\widehat{\mathcal{O}}\big{)}\big{)}$. In the
particular case where $\lambda$ belongs to the subgroup
$\nu(N)\subset\mathbb{C}^{*}$ of complexes such that $\lambda^{N}=1$, the
action reads $\lambda\bullet(\rho,z)=(\rho,\lambda z)$ and this action acts
transitively on the possible $z$ so the conclusion of the lemma follows. ∎

In virtue of Lemma 5.3 and by abuse of notations, we will now simply denote by
$(\pi,W(\mathcal{O}))$ one of the pairwise isomorphic representations
$(\pi,W(\widehat{\mathcal{O}}_{z}))$ associated to a reduced finite orbit
$\mathcal{O}$.

###  5.3 Kernel of the representations

The present paper was originally motivated by the hope to find faithful
finite-dimensional representations of the mapping class groups (modulo
center), it is thus natural to study the eventual kernel of the
representations defined in the previous subsection. We will not achieve this
goal but instead provide tools to prove that a given element does not belong
to the kernel.

The kernel of the Witten–Reshetikhin–Turaev representations

$\rho^{\rm WRT}\colon\
\operatorname{Mod}(\Sigma_{g})\to\operatorname{PGL}(V(\Sigma_{g}))$

is not known but it is easy to see that it contains the normal subgroup
generated by the elements $T_{\gamma}^{N}-{\rm id}$ where $T_{\gamma}$ are the
Dehn–Twists associated to simple closed curves $\gamma$. These representations
are defined using some (irreducible) representations $r^{\rm
WRT}\colon\mathcal{S}_{A}(\Sigma_{g})\to\operatorname{End}(V(\Sigma_{g}))$ of
the usual skein algebras, the representation $\rho^{\rm WRT}$ can be defined
from $r^{\rm WRT}$ by the same strategy than we used in the previous
subsection, that is using an analogue of equation (5.1). The fact that the
kernel of $\rho^{\rm WRT}$ is hard to compute is related to the fact that the
kernel of $r^{\rm WRT}$ is unknown (though it is proved in [12] that is
contains the elements $T_{N}([\gamma])+2{\rm id}$).

An advantage of our representations compared to $\rho^{\rm WRT}$ is the fact
that the kernel of the representations
$r_{\rho}\colon\mathcal{S}_{A}(\mathbf{\Sigma}_{g})\to\operatorname{End}(V(\rho))$
and
$r_{\hat{\rho}}\colon\overline{\mathcal{S}}_{A}(\mathbf{\Sigma}_{g})\to\operatorname{End}(V(\widehat{\rho}))$
are well known: they correspond to the ideals $\mathcal{I}_{\rho}$ and
$\mathcal{I}_{\widehat{\rho}}$ by definition. We can thus deduce

######  Proposition 5.4.

Let $G\subset\operatorname{Mod}(\Sigma_{g,1})$ and $\pi\colon
G\to\operatorname{PGL}(W(\mathcal{O}))$ a representation associated to a
finite orbit $\mathcal{O}$. Let $\phi\in G$ and suppose that there exists a
closed curve $\gamma$ and $\rho\in\mathcal{O}$ when the orbit is in the big
cell $($resp. $\rho\in\widehat{\mathcal{O}}_{z}$ when the orbit is reduced$)$
such that

$\phi(\gamma)\not\equiv z\gamma\pmod{\mathcal{I}_{\rho}},\qquad\text{for all
}z\in\mathbb{C}^{*}.$

Then $\pi(\phi)\neq{\rm id}$.

###### Proof.

This is an immediate consequence of the definitions. ∎

###  5.4 The use of quantum Teichmüller theory in the reduced case

The criterion found in Proposition 5.4 is not easy to use in practice. In this
subsection, we explain how the use of quantum traces and quantum Teichmüller
spaces might simplify this criterion for practical computations in the case
where the orbit is reduced. Recall that the triangle $\mathbb{T}$ is the
marked surface made of a disc with three boundary arcs.

######  Definition 5.5.

A marked surface $\mathbf{\Sigma}$ is triangulable if it can be obtained from
a finite disjoint union $\mathbf{\Sigma}_{\Delta}=\bigsqcup_{i}\mathbb{T}_{i}$
of triangles by gluing some pairs of edges. A triangulation $\Delta$ is then
the data of the disjoint union $\mathbf{\Sigma}_{\Delta}$ together with the
set of glued pair of edges.

The connected components $\mathbb{T}_{i}$ of $\mathbf{\Sigma}_{\Delta}$ are
called faces and their set is denoted $F(\Delta)$. The image in $\Sigma$ of
the edges of the faces $\mathbb{T}_{i}$ are called edges of $\Delta$ and their
set is denoted $\mathcal{E}(\Delta)$. Note that each boundary arc is an edge
in $\mathcal{E}(\Delta)$; the elements of the complementary
$\mathring{\mathcal{E}}(\Delta):=\mathcal{E}(\Delta)\setminus\mathcal{A}$ are
called inner edges. Figure 12 illustrates a triangulation $\Delta_{1}$ of
$\mathbf{\Sigma}_{1}^{*}$. Note that when $(\mathbf{\Sigma},\Delta)$ is a
triangulated marked surface, the fusion $\mathbf{\Sigma}_{1\circledast 2}$
admits a natural triangulation obtained from $\Delta$ by adding one face
corresponding to the added triangle. Therefore the triangulation $\Delta_{1}$
of $\mathbf{\Sigma}_{1}^{*}$ induces a triangulation $\Delta_{g}$ of
$\mathbf{\Sigma}_{g}^{*}$.

Figure 12: A triangulation of $\mathbf{\Sigma}_{1}^{*}$.

######  Definition 5.6.

  1. 1.

Consider a pair $\mathbb{E}=(E,(\cdot,\cdot))$ where $E$ is a free finite rank
$\mathbb{Z}$-module (so $E\cong\mathbb{Z}^{n}$) and $(\cdot,\cdot)\colon
E\times E\to\mathbb{Z}$ is a skew-symmetric pairing. The quantum torus
$\mathbb{T}_{q}(\mathbb{E})$ is the algebra generated by generators $Z_{e}$,
$e\in E$ with relations $Z_{a}Z_{b}=A^{-(a,b)/4}Z_{a+b}$. Said differently,
given $e=(e_{1},\dots,e_{n})$ a basis of $E$, the quantum torus
$\mathbb{T}_{q}(\mathbb{E})$ is isomorphic to the complex algebra generated by
invertible elements $Z_{e_{i}}^{\pm 1}$ with relations
$Z_{e_{i}}Z_{e_{j}}=A^{-(e_{i},e_{j})/2}Z_{e_{j}}Z_{e_{i}}$.

  2. 2.

Let $(\mathbf{\Sigma},\Delta)$ be a triangulated marked surface and denote by
$\mathcal{E}(\Delta)$ the set of edges of the triangulation. A map
$k\colon\mathcal{E}(\Delta)\to\mathbb{Z}$ is balanced if for any face
$\mathbb{T}$ of the triangulation with edges $a$, $b$, $c$ then
$k(a)+k(b)+k(c)$ is even. We denote by $K_{\Delta}$ the $\mathbb{Z}$-module of
balanced maps. For $e$ and $e^{\prime}$ two edges, denote by
$a_{e,e^{\prime}}$ the number of faces $\mathbb{T}\in F(\Delta)$ such that $e$
and $e^{\prime}$ are edges of $\mathbb{T}$ and such that we pass from $e$ to
$e^{\prime}$ in the counter-clockwise direction in $\mathbb{T}$. The
Weil–Petersson form $(\cdot,\cdot)^{\rm WP}\colon K_{\Delta}\times
K_{\Delta}\rightarrow\mathbb{Z}$ is the skew-symmetric form defined by
$(\mathbf{k}_{1},\mathbf{k}_{2})^{\rm
WP}:=\sum_{e,e^{\prime}}\mathbf{k}_{1}(e)\mathbf{k}_{2}(e^{\prime})(a_{e,e^{\prime}}-a_{e^{\prime},e})$.
The balanced Chekhov–Fock algebra is the quantum torus
$\mathcal{Z}_{q}(\mathbf{\Sigma},\Delta):=\mathbb{T}_{q}\big{(}K_{\Delta},(\cdot,\cdot)^{\rm
WP}\big{)}$.

  3. 3.

If $A$ is root of unity of order $N$, the Frobenius morphism is the central
embedding

$\operatorname{Fr}_{N}\colon\
\mathbb{T}_{+1}(\mathbb{E})\hookrightarrow\mathcal{Z}(\mathbb{T}_{q}(\mathbb{E})),\qquad\operatorname{Fr}_{N}(Z_{a}):=(Z_{a})^{N}=Z_{Na}.$

######  Theorem 5.7 (De Concini–Procesi [27, Proposition 7.2]).

If $A\in\mathbb{C}^{*}$ is a root of unity, then $\mathbb{T}_{q}(\mathbb{E})$
is Azumaya.

The quantum trace is an algebra embedding

$\operatorname{Tr}^{\Delta}_{A}\colon\
\overline{\mathcal{S}}_{A}(\mathbf{\Sigma})\hookrightarrow\mathcal{Z}_{q}(\mathbf{\Sigma},\Delta)$

defined by Bonahon–Wong for unmarked surfaces in [10] and extended to marked
surfaces by Lê in [51]. It is characterized as follows. First consider the
triangle $\mathbb{T}$ with edges $e_{1}$, $e_{2}$, $e_{3}$ and arcs
$\alpha_{i}$ as in Figure 13.

Figure 13: The triangle and some arcs.

For $i\in\mathbb{Z}/3\mathbb{Z}$, let $\mathbf{k}_{i}$ be the balanced map
sending $e_{i}$ to $0$ and $e_{i+1}$, $e_{i+2}$ to $1$. The quantum trace
$\operatorname{Tr}^{\mathbb{T}}_{A}\colon\overline{\mathcal{S}}_{A}(\mathbb{T})\xrightarrow{\cong}\mathcal{Z}_{q}(\mathbb{T})$
is the isomorphism characterized by the facts that it sends
$(\alpha_{i})_{-+}$ to $0$, $(\alpha_{i})_{++}$ to $Z^{\mathbf{k}_{i}}$ and
$(\alpha_{i})_{--}$ to $Z^{-\mathbf{k}_{i}}$.

Consider a general triangulated marked surface $(\mathbf{\Sigma},\Delta)$, so
$\mathbf{\Sigma}$ is obtained from a disjoint union of triangle
$\mathbf{\Sigma}_{\Delta}:=\bigsqcup_{\mathbb{T}\in F(\Delta)}\mathbb{T}$ by
gluing some pairs of edges. Let $\widehat{\mathcal{E}}$ be the set of edges
(boundary arcs) of the disjoint union $\mathbf{\Sigma}_{\Delta}$ and let
$\pi\colon\widehat{\mathcal{E}}\to\mathcal{E}$ be the induced surjective map.

Consider the embedding
$\theta^{\Delta}\colon\mathcal{Z}_{q}(\mathbf{\Sigma},\Delta)\hookrightarrow\otimes_{\mathbb{T}\in
F(\Delta)}\mathcal{Z}_{q}(\mathbb{T})$ sending $Z^{\mathbf{k}}$ to
$\otimes_{\mathbb{T}}Z^{\mathbf{k}_{\mathbb{T}}}$ where
$\mathbf{k}_{\mathbb{T}}(e)\allowbreak:=\mathbf{k}(\pi(e))$.

######  Definition 5.8 ([10, 51]).

The quantum trace $\operatorname{Tr}^{\Delta}_{A}$ is the unique algebra
morphism making commuting the following diagram

${\overline{\mathcal{S}}_{A}(\mathbf{\Sigma})}$${\otimes_{\mathbb{T}\in
F(\Delta)}\overline{\mathcal{S}}_{A}(\mathbb{T})}$${\mathcal{Z}_{q}(\mathbf{\Sigma},\Delta)}$${\otimes_{\mathbb{T}\in
F(\Delta)}\mathcal{Z}_{q}(\mathbb{T}).}$$\scriptstyle{\theta^{\Delta}}$$\scriptstyle{\operatorname{Tr}^{\Delta}_{A}}$$\scriptstyle{\cong}$$\scriptstyle{\otimes_{\mathbb{T}}\operatorname{Tr}^{\mathbb{T}}_{A}}$$\scriptstyle{\theta^{\Delta}}$

An easy consequence of the definition is the fact that
$\operatorname{Tr}^{\Delta}_{+1}\circ\operatorname{Ch}_{A}=\operatorname{Fr}_{N}\circ\operatorname{Tr}^{\Delta}_{A}$,
i.e., that the Chebyshev–Frobenius morphism $\operatorname{Ch}_{A}$ is the
restriction of the Frobenius $\operatorname{Fr}_{N}$ through the quantum trace
(this is how $\operatorname{Ch}_{A}$ was first defined in [10]). The centers
and PI-degrees of the balanced Chekhov–Fock algebras were computed in [13] for
unmarked surfaces and in [50] for marked surfaces. In the particular case of
$\mathbf{\Sigma}_{g}^{*}$, it is described as follows. Let us fix a
triangulation $\Delta$ of $\mathbf{\Sigma}_{g}^{*}$. Let
$\mathbf{k}_{\partial}\colon\mathcal{E}(\Delta)\to\mathbb{Z}$ be the balanced
map sending every $e\in\mathcal{E}(\Delta)$ to $\mathbf{k}_{\partial}(e):=2$.
Then $H_{\partial}:=Z^{\mathbf{k}_{\partial}}$ is central in
$\mathcal{Z}_{q}(\mathbf{\Sigma}_{g}^{*},\Delta)$ and we have
$\operatorname{Tr}^{\Delta}_{A}(\alpha_{\partial})=H_{\partial}$.

######  Theorem 5.9 ([50]).

Suppose that $A^{1/2}$ is a root of unity of odd order $N$.

  1. $1.$

The center of $\mathcal{Z}_{q}(\mathbf{\Sigma}_{g}^{*},\Delta)$ is generated
by the image of the Frobenius morphims $\operatorname{Fr}_{N}$ together with
the elements $H_{\partial}^{\pm 1}$. In particular the quantum trace
$\operatorname{Tr}^{\Delta}_{A}$ embeds the center of
$\overline{\mathcal{S}}_{A}(\mathbf{\Sigma})$ into the center of
$\mathcal{Z}_{q}(\mathbf{\Sigma}_{g}^{*},\Delta)$.

  2. $2.$

The PI-degree of $\mathcal{Z}_{q}(\mathbf{\Sigma}_{g}^{*},\Delta)$ is equal to
$N^{3g-1}$, i.e., to the PI-degree of
$\mathcal{S}_{A}(\mathbf{\Sigma}_{g}^{*})$.

Let $\mathcal{Z}$ be the center of
$\mathcal{Z}_{q}(\mathbf{\Sigma}_{g}^{*},\Delta)$ and consider the algebraic
tori

$\mathcal{Y}(\mathbf{\Sigma}_{g}^{*},\Delta):=\operatorname{Specm}(\mathcal{Z}_{+1}(\mathbf{\Sigma}_{g}^{*},\Delta))$

and
$\widehat{\mathcal{Y}}(\mathbf{\Sigma}_{g}^{*},\Delta):=\operatorname{Specm}(\mathcal{Z})$
and the natural regular covering
$\pi\colon\widehat{\mathcal{Y}}(\mathbf{\Sigma}_{g}^{*},\Delta)\to\mathcal{Y}(\mathbf{\Sigma}_{g}^{*},\Delta)$.
By Theorem 5.9, $\pi$ is a principal $(\mathbb{Z}/N\mathbb{Z})^{\times}$
bundle; more precisely,
$\widehat{\mathcal{Y}}(\mathbf{\Sigma}_{g}^{*},\Delta)$ is identified with the
set of pairs $\widehat{\mathbf{z}}=(\mathbf{z},h_{\partial})$ where
$h_{\partial}\in\mathbb{C}^{*}$ and
$\mathbf{z}\in\mathcal{Y}(\mathbf{\Sigma}_{g}^{*},\Delta)$ corresponds to a
character sending $H_{\partial}\in\mathcal{Z}_{+1}(\mathbf{\Sigma},\Delta)$ to
$h_{\partial}^{N}$.

The quantum trace in $A^{1/2}=+1$

$\operatorname{Tr}_{+1}^{\Delta}\colon\
\overline{\mathcal{S}}_{+1}(\mathbf{\Sigma}_{g}^{*})\hookrightarrow\mathcal{Z}_{+1}(\mathbf{\Sigma}_{g}^{*},\Delta)$

defines regular maps
$\widehat{\mathcal{NA}}\colon\widehat{\mathcal{Y}}(\mathbf{\Sigma}_{g}^{*},\Delta)\to\widehat{\overline{\mathcal{X}}}(\mathbf{\Sigma}_{g}^{*})$
and
$\mathcal{NA}\colon\mathcal{Y}(\mathbf{\Sigma}_{g}^{*},\Delta)\to\overline{\mathcal{X}}(\mathbf{\Sigma}_{g}^{*})$
named the non-abelianization maps. Since
$\operatorname{Tr}_{A}^{\Delta}(\alpha_{\partial})=H_{\partial}$,
$\widehat{\mathcal{NA}}$ is a morphism of principal
$(\mathbb{Z}/N\mathbb{Z})^{\times}$ bundles: it sends a pair
$(\mathbf{z},h_{\partial})$ to $(\mathcal{NA}(\mathbf{z}),h_{\partial})$.

Since $\operatorname{Tr}_{+1}^{\Delta}$ is injective, the map $\mathcal{NA}$
is dominant, so by irreducibility of
$\overline{\mathcal{X}}(\mathbf{\Sigma}_{g}^{*})$, the image of $\mathcal{NA}$
is dense. Unfortunately, a precise description of this image is still unknown.

######  Definition 5.10.

A point $\rho\in\overline{\mathcal{X}}(\mathbf{\Sigma}_{g}^{*})$ admits a
$\Delta$-lift if $\rho$ is in the image of $\mathcal{NA}$. An orbit
$\mathcal{O}\subset\overline{\mathcal{X}}(\mathbf{\Sigma}_{g}^{*})$ admits a
$\Delta$-lift if each of its points does.

For $\widehat{\mathbf{z}}\in\mathcal{Y}(\mathbf{\Sigma}_{g}^{*},\Delta)$
corresponding to a character $\chi_{\widehat{\mathbf{z}}}$ over the center of
$\mathcal{Z}_{q}(\mathbf{\Sigma}_{g}^{*},\Delta)$, let
$\mathcal{J}_{\widehat{\mathbf{z}}}\subset\mathcal{Z}_{q}(\mathbf{\Sigma}_{g}^{*},\Delta)$
be the ideal generated by the kernel of $\chi_{\widehat{\mathbf{z}}}$. Then,
writing $\widehat{\rho}:=\mathcal{NA}(\widehat{\mathbf{z}})$ and in virtue of
Theorems 5.7 and 5.9, we have

$\operatorname{End}(V(\widehat{\rho}))\cong{\raisebox{1.99997pt}{$\overline{\mathcal{S}}_{A}(\mathbf{\Sigma})$}\left/\raisebox{-1.99997pt}{$\mathcal{I}_{\widehat{\rho}}$}\right.}\xrightarrow[\cong]{\operatorname{tr}^{\Delta}_{A}}{\raisebox{1.99997pt}{$\mathcal{Z}_{q}(\mathbf{\Sigma}_{g}^{*},\Delta)$}\left/\raisebox{-1.99997pt}{$\mathcal{J}_{\widehat{\mathbf{z}}}$}\right.}.$
(5.3)

So a point $\rho\in\overline{\mathcal{X}}(\mathbf{\Sigma}_{g}^{*})$ admits a
$\Delta$-lift if and only if for one (or equivalently every) of its lifts
$\widehat{\rho}\in\widehat{\overline{\mathcal{X}}}(\mathbf{\Sigma}_{g}^{*})$,
the simple $\overline{\mathcal{S}}_{A}(\mathbf{\Sigma})$ module
$V(\widehat{\rho})$ extends (in a non necessarily unique way) to a simple
$\mathcal{Z}_{q}(\mathbf{\Sigma}_{g}^{*},\Delta)$ module through the quantum
trace. Equivalently, an orbit $\mathcal{O}$ admits a $\Delta$-lift if and only
if the $\overline{\mathcal{S}}_{A}(\mathbf{\Sigma})$ module $W(\mathcal{O})$
extends to a $\mathcal{Z}_{q}(\mathbf{\Sigma}_{g}^{*},\Delta)$ module.
Unfortunately, the author is not aware of any criterion to prove that a given
orbit $\mathcal{O}$ admits a $\Delta$-lift, so the usefulness of the methods
developed in this subsection is purely conjectural. The criterion of
Proposition 5.4 can be improved as follows.

######  Proposition 5.11.

Let $G\subset\operatorname{Mod}(\Sigma_{g,1})$ and $\pi\colon
G\to\operatorname{PGL}\big{(}W\big{(}\widehat{\mathcal{O}}\big{)}\big{)}$ a
representation associated to a finite orbit $\mathcal{O}$ in the reduced cell.
Let $\phi\in G$ and consider a simple closed curve $\gamma$ and
$\widehat{\rho}\in\widehat{\mathcal{O}}$ which admits a $\Delta$-lift, i.e.,
such that $\widehat{\rho}=\widehat{\mathcal{NA}}(\widehat{\mathbf{z}})$.
Suppose that

$\operatorname{Tr}_{A}^{\Delta}(\phi(\gamma))\not\equiv\lambda\operatorname{Tr}_{A}^{\Delta}(\gamma)\pmod{\mathcal{J}_{\widehat{\mathbf{z}}}},\qquad\text{for
all }\lambda\in\mathbb{C}^{*}.$

Then $\pi(\phi)\neq{\rm id}$.

###### Proof.

The proposition is a direct consequence of Proposition 5.4 and of equation
(5.3). ∎

In order to be able to use the criterion of Proposition 5.11 efficiently, we
need to find a way to prove that, given two distinct simple closed curves
$\alpha$ and $\beta$ in $\Sigma_{g,1}$ then

$\operatorname{Tr}_{A}^{\Delta}(\alpha)\not\equiv\lambda\operatorname{Tr}_{A}^{\Delta}(\beta)\pmod{\mathcal{J}_{\widehat{\mathbf{z}}}},\qquad\text{for
all }\lambda\in\mathbb{C}^{*}.$

To achieve this goal we need to find a basis of
$\mathcal{Z}_{q}(\mathbf{\Sigma}_{g}^{*},\Delta)$ as a module over its center.
The strategy is quite general: let $\mathbb{T}_{q}(\mathbb{E})$ be an
arbitrary quantum torus with $A^{1/2}$ a root of unity of odd order $N$ and
let $E^{0}\subset E$ be the submodule defined as the kernel:

$E^{0}:=\ker\big{(}E\otimes_{\mathbb{Z}}E\xrightarrow{(\cdot,\cdot)}\mathbb{Z}\to\mathbb{Z}/N\mathbb{Z}\big{)}.$

Then the center of $\mathbb{T}_{q}(\mathbb{E})$ is spanned by elements
$Z_{e_{0}}$ for $e_{0}\in E^{0}$ therefore, in order to find a basis $B$ of
$\mathbb{T}_{q}(\mathbb{E})$ seen as module over its center, we need to find a
representative set $E^{1}\subset E$ of the coset
${\raisebox{1.99997pt}{$E$}\left/\raisebox{-1.99997pt}{$E^{0}$}\right.}$ and
we can choose $B:=\operatorname{Span}\big{(}Z_{e},e\in E^{1}\big{)}$.

It was proved in [50] that the submodule $K_{\Delta}^{0}\subset K_{\Delta}$
defined by

$K_{\Delta}^{0}:=\ker\big{(}K_{\Delta}\otimes_{\mathbb{Z}}K_{\Delta}\xrightarrow{(\cdot,\cdot)^{\rm
WP}}\mathbb{Z}\to\mathbb{Z}/N\mathbb{Z}\big{)}$

is equal to

$K_{\Delta}^{0}=NK_{\Delta}+\mathbb{Z}\mathbf{k}_{\partial}.$ (5.4)

Recall that $\mathbf{k}_{\partial}$ is the balanced map sending every edge to
$+2$.

Let us now describe the image of simple closed curve by the quantum trace. Let
$\gamma\subset\Sigma_{g,1}$ be a simple closed curve isotoped such that it is
transversed to $\mathcal{E}(\Delta)$ with minimal number of intersection
points. A full state on $\gamma$ is a map
$\hat{s}\colon\mathcal{E}(\Delta)\cap\gamma\rightarrow\\{-,+\\}$. A pair
$(\gamma,\hat{s})$ induces on each face $\mathbb{T}\in F(\Delta)$ a stated
diagram in $\mathbb{T}$. A full state $\hat{s}$ is admissible if the
restriction of $(\gamma,\hat{s})$ on each face does not contain bad arcs. We
denote by ${\rm St}^{a}(\gamma)$ the set of admissible full states. For
$\hat{s}\in{\rm St}^{a}(\gamma)$, we denote by $\mathds{k}(\hat{s})\in
K_{\Delta}$ the balanced map defined by

$\mathds{k}(\hat{s})(e):=\sum_{v\in\gamma\cap e}\hat{s}(v),\qquad
e\in\mathcal{E}(\Delta).$

######  Lemma 5.12 ([47, Lemma 2.17]).

There exist some integers $n(\hat{s})\in\mathbb{Z}$ such that

$\operatorname{Tr}_{A}^{\Delta}([\gamma])=\sum_{\hat{s}\in{\rm
St}^{a}(\gamma)}A^{n(\hat{s})/2}Z^{\mathds{k}(\hat{s})}.$

We denote by $\underline{{\rm St}}^{a}(\gamma)\subset K_{\Delta}$ the set of
balanced maps of the form $\mathds{k}(\hat{s})$ for $\hat{s}\in{\rm
St}^{a}(\gamma)$. Let $\mathds{k}_{\gamma}\in K_{\Delta}$ be the balanced map
defined by

$\mathds{k}_{\gamma}(e):=|\gamma\cap e|,\qquad\text{for all
}e\in\mathcal{E}(\Delta).$

Recall that $\gamma$ is in minimal position with respect to
$\mathcal{E}(\Delta)$ so $|\gamma\cap e|$ is the geometric intersection of
$\gamma$ and $e$. Note also that
$\mathds{k}_{\gamma}=\mathds{k}(s^{+})\in\underline{{\rm St}}^{a}(\gamma)$
corresponds to the admissible state $s^{+}$ sending every point of
$\gamma\cap\mathcal{E}(\gamma)$ to $+$.

######  Lemma 5.13.

Let $\mathds{k}=\mathds{k}(\hat{s})\in\underline{{\rm St}}^{a}(\gamma)$.

  1. $1.$

For all $e\in\mathcal{E}(\Delta)$ then $-|\gamma\cap
e|\leq\mathds{k}(e)\leq|\gamma\cap e|$ and $\mathds{k}(e)$ has the same parity
than $|\gamma\cap e|$.

  2. $2.$

One has $\mathds{k}(a_{\partial})=0$.

###### Proof.

This is an immediate consequence of the definition of $\underline{{\rm
St}}^{a}(\gamma)$. ∎

######  Notations 5.14.

For $\mathbf{k}\in K_{\Delta}$ and $\gamma\subset\Sigma_{g,1}$ a simple closed
curve, let

$S(\gamma):=\big{\\{}\mathbf{k}_{0}\in K_{\Delta}^{0}\text{ such that
}\mathbf{k}+\mathbf{k}_{0}\in\underline{\operatorname{St}}^{a}(\gamma)\big{\\}}\subset
K_{\Delta}^{0}.$

For $\hat{\mathbf{z}}$ a character over the center of
$\mathcal{Z}_{q}(\mathbf{\Sigma},\Delta)$, write

$x_{\hat{\mathbf{z}},\mathbf{k}}(\gamma):=\sum_{\mathbf{k}_{0}\in
S(\gamma),\,\mathbf{k}+\mathbf{k}_{0}=\mathds{k}(\hat{s})}\hat{\mathbf{z}}(\mathbf{k}_{0})A^{n(\hat{s})/2}.$

If $B\subset K_{\Delta}$ is a set of representatives of the coset
${\raisebox{1.99997pt}{$K_{\Delta}$}\left/\raisebox{-1.99997pt}{$K_{\Delta}^{0}$}\right.}$
then by definition one has

$\operatorname{Tr}_{A}^{\Delta}([\gamma])\equiv\sum_{\mathbf{k}\in
B}x_{\hat{\mathbf{z}},\mathbf{k}}(\gamma)Z^{\mathbf{k}}\pmod{\mathcal{J}_{\hat{\mathbf{z}}}},$

where the family $\big{\\{}Z^{\mathbf{k}},\,\mathbf{k}\in B\big{\\}}$ is a
basis of the space
${\raisebox{1.99997pt}{$\mathcal{Z}_{q}(\mathbf{\Sigma},\Delta)$}\left/\raisebox{-1.99997pt}{$\mathcal{J}_{\hat{\mathbf{z}}}$}\right.}$.
We thus have proved

######  Lemma 5.15.

Let $\alpha,\beta\subset\Sigma_{g,1}$ be two simple closed curves and
$\lambda\in\mathbb{C}^{*}$. Then

$\operatorname{Tr}_{A}^{\Delta}(\alpha)\equiv\lambda\operatorname{Tr}_{A}^{\Delta}(\beta)\pmod{\mathcal{J}_{\widehat{\mathbf{z}}}}$

if and only if $x_{\hat{\mathbf{z}},\mathbf{k}}(\alpha)=\lambda
x_{\hat{\mathbf{z}},\mathbf{k}}(\beta)$ for all $\mathbf{k}\in B$.

In particular, if there exists $\mathbf{k}\in K_{\Delta}$ such that
$x_{\hat{\mathbf{z}},\mathbf{k}}(\alpha)=0$ and
$x_{\hat{\mathbf{z}},\mathbf{k}}(\beta)\neq 0$, then

$\operatorname{Tr}_{A}^{\Delta}(\alpha)\not\equiv\lambda\operatorname{Tr}_{A}^{\Delta}(\beta)\pmod{\mathcal{J}_{\widehat{\mathbf{z}}}},\qquad\text{for
all }\lambda\in\mathbb{C}^{*}.$

Lemma 5.15 together with Proposition 5.11 permit to prove that a given mapping
class $\phi\in G\subset\operatorname{Mod}(\Sigma_{g,1})$ does not belong to
the kernel of a representation $\pi\colon
G\to\operatorname{End}(W(\mathcal{O}))$ associated to a reduced orbit,
provided that we are able to compute the numbers
$x_{\hat{\mathbf{z}},\mathbf{k}}(\gamma)$. As an illustration, let us derive
from this criterion an easy consequence.

######  Lemma 5.16.

Let $\pi\colon
K_{\Delta}\to{\raisebox{1.99997pt}{$K_{\Delta}$}\left/\raisebox{-1.99997pt}{$K_{\Delta}^{0}$}\right.}$
be the quotient map. Suppose that $\gamma\subset\Sigma_{g,1}$ intersects each
edge of the triangulation $\Delta$ at most $N-1$ times. Then the restriction
$\pi\colon\underline{\operatorname{St}}^{a}(\gamma)\to{\raisebox{1.99997pt}{$K_{\Delta}$}\left/\raisebox{-1.99997pt}{$K_{\Delta}^{0}$}\right.}$
is injective.

###### Proof.

Let
$\mathbf{k}_{1},\mathbf{k}_{2}\in\underline{\operatorname{St}}^{a}(\gamma)$ be
such that $\mathbf{k}_{1}-\mathbf{k}_{2}\in K_{\Delta}^{0}$. By equation
(5.4), there exists $\mathbf{k}_{0}\in K_{\Delta}$ and $n\in\\{0,\dots,N-1\\}$
such that

$\mathbf{k}_{1}-\mathbf{k}_{2}=n\mathbf{k}_{\partial}+N\mathbf{k}_{0}.$

By the second item of Lemma 5.13, one has
$\mathbf{k}_{1}(a_{\partial})=\mathbf{k}_{2}(a_{\partial})=0$ so $n=0$. By the
first item of Lemma 5.13, for all $e\in\mathcal{E}(\Delta)$, one has
$-(N-1)\leq\mathbf{k}_{1}(e),\mathbf{k}_{2}(e)\leq N-1$ and
$\mathbf{k}_{1}(e)\equiv\mathbf{k}_{2}(e)\pmod{2}$ therefore
$\mathbf{k}_{0}(e)=0$. We thus have proved that
$\mathbf{k}_{1}=\mathbf{k}_{2}$. This concludes the proof. ∎

We eventually have the following criterion, which should be compared to
Costantino–Martelli criterion in [24, Lemma 7.1] for the Turaev–Viro
representations.

######  Theorem 5.17.

Let $\pi\colon G\to\operatorname{End}(W(\mathcal{O}))$ be a representation of
$G\subset\operatorname{Mod}(\Sigma_{g,1})$ associated to a reduced orbit and a
root of unity of odd order $N$. Consider a mapping class $\phi\in G$ and a
simple closed curve $\alpha\subset\Sigma_{g,1}$ such that $\alpha$ and
$\beta:=\phi(\alpha)$ are not isotopic. Suppose that there exists a
triangulation $\Delta$ of $\mathbf{\Sigma}_{g}^{*}$ such that

  1. $1)$

there exists $\rho\in\mathcal{O}$ which admits a $\Delta$-lift,

  2. $2)$

both $\alpha$ and $\beta$ intersect each edge of $\Delta$ at most $N-1$ times.

Then $\pi(\phi)\neq{\rm id}$.

###### Proof.

Let $\widehat{\mathbf{z}}$ be the character associated to an arbitrary lift of
$\rho$. If $\alpha\neq\beta$ then there exists $e\in\mathcal{E}(\Delta)$ such
that $|\alpha\cap e|\neq|\beta\cap e|$ so Lemma 5.13 implies that
$\underline{\operatorname{St}}^{a}(\alpha)\neq\underline{\operatorname{St}}^{a}(\beta)$.
Exchanging $\alpha$ and $\beta$ if necessary, we can suppose that there exists
$\mathbf{k}\in\underline{\operatorname{St}}^{a}(\beta)$ such that
$\mathbf{k}\notin\underline{\operatorname{St}}^{a}(\alpha)$. Lemma 5.16
implies that $x_{\hat{\mathbf{z}},\mathbf{k}}(\alpha)=0$ and
$x_{\hat{\mathbf{z}},\mathbf{k}}(\beta)\neq 0$ so Lemma 5.15 implies that

$\operatorname{Tr}_{A}^{\Delta}(\alpha)\not\equiv\lambda\operatorname{Tr}_{A}^{\Delta}(\beta)\pmod{\mathcal{J}_{\widehat{\mathbf{z}}}},\qquad\text{for
all }\lambda\in\mathbb{C}^{*}.$

We conclude using Proposition 5.11. ∎

###  5.5 The use of quantum Teichmüller theory in the non reduced case

We now extend all the tools developed in the previous subsection to the study
of representation $\pi\colon G\to\operatorname{End}(W(\mathcal{O}))$
associated to orbits in the big cell. Bonahon and Wong’s quantum trace permits
to embed the reduced stated skein algebras into quantum tori. In order to be
able to embed the non-reduced stated skein algebras into quantum tori as well,
Lê and Yu developed in [53] the following trick. Let
$\mathbf{\Sigma}_{g}^{**}=(\Sigma_{g,1},\\{a_{\partial}^{\prime},a_{\partial}^{\prime\prime}\\})$
be $\Sigma_{g,1}$ with two boundary arcs. Consider the marked surface
embedding
$j\colon\mathbf{\Sigma}_{g}^{*}\hookrightarrow\mathbf{\Sigma}_{g}^{**}$ which
is the identity outside a neighborhood of $a_{\partial}$ and which sends
$a_{\partial}$ to $a_{\partial}^{\prime}$ as illustrated in Figure 14.
Alternatively, one can think of $\mathbf{\Sigma}_{g}^{**}$ as been obtained
from $\mathbf{\Sigma}_{g}^{*}$ by gluing a triangle $\mathbb{T}_{0}$ with
edges $\\{a_{\partial},a_{\partial}^{\prime},a_{\partial}^{\prime\prime}\\}$
along the unique boundary arc of $\mathbf{\Sigma}_{g}^{*}$. In particular,
every triangulation $\Delta$ of $\mathbf{\Sigma}_{g}^{*}$ induces a
triangulation $\Delta^{*}$ of $\mathbf{\Sigma}_{g}^{**}$ obtained by adding
the face $\mathbb{T}_{0}$. An illustration in the genus $1$ case is given in
Figure 14.

Figure 14: An illustration of the embedding $j$.

Lê and Yu proved in [53] that the embedding $j$ induces an embedding

$j_{*}\colon\
\mathcal{S}_{A}(\mathbf{\Sigma}_{g}^{*})\hookrightarrow\overline{\mathcal{S}}_{A}(\mathbf{\Sigma}_{g}^{**})$

of the non-reduced stated skein algebra of $\mathbf{\Sigma}_{g}^{*}$ onto the
reduced stated skein algebra of $\mathbf{\Sigma}_{g}^{**}$. Therefore, by
composition, we obtain an embedding

$\phi\colon\
\mathcal{S}_{A}(\mathbf{\Sigma}_{g}^{*})\xrightarrow{j_{*}}\overline{\mathcal{S}}_{A}(\mathbf{\Sigma}_{g}^{**})\xrightarrow{\operatorname{Tr}_{q}^{\Delta^{*}}}\mathcal{Z}_{q}(\mathbf{\Sigma}_{g}^{**},\Delta^{*}).$

The image of $\phi$ actually belongs to a smaller quantum torus that we now
describe.

Let
$\overline{\mathcal{E}}:=\mathcal{E}(\Delta)\bigsqcup\\{\widehat{a_{\partial}}\\}$.
Let $\overline{K}_{\Delta}$ denote the set of maps
$\mathbf{k}\colon\overline{\mathcal{E}}(\Delta)\to\mathbb{Z}$ such that $(1)$
for any face of $\Delta$ with edges $a$, $b$, $c$, then
$\mathbf{k}(a)+\mathbf{k}(b)+\mathbf{k}(c)$ is even and $(2)$
$\mathbf{k}(\widehat{a_{\partial}})$ is even. Define an injective linear map
$i\colon\overline{K}_{\Delta}\hookrightarrow K_{\Delta^{*}}$ sending
$\mathbf{k}$ to $\mathbf{k}^{\prime}$ where
$\mathbf{k}^{\prime}(e)=\mathbf{k}(e)$ if $e\in\mathring{\mathcal{E}}(\Delta)$
and
$\mathbf{k}^{\prime}(a_{\partial})=\mathbf{k}(a_{\partial})+\mathbf{k}(\widehat{a_{\partial}})$,
$\mathbf{k}^{\prime}(a^{\prime}_{\partial})=-\mathbf{k}(a_{\partial})$ and
$\mathbf{k}^{\prime}(a^{\prime\prime}_{\partial})=0$. Let
$\langle\cdot,\cdot\rangle\colon\overline{\mathcal{E}}\otimes_{\mathbb{Z}}\overline{\mathcal{E}}\to\mathbb{Z}$
be the skew-symmetric form defined by $\langle x,y\rangle:=(i(x),i(y))^{\rm
WP}$. Let
$\overline{\mathcal{Z}}_{q}(\mathbf{\Sigma}_{g}^{*},\Delta)\subset\mathcal{Z}_{q}(\mathbf{\Sigma}^{**},\Delta^{*})$
be the submodule spanned by elements $Z^{i(\mathbf{k})}$ for
$\mathbf{k}\in\overline{K}_{\Delta}$. An easy computation shows that the map
$\phi$ takes values in the smaller quantum torus
$\overline{\mathcal{Z}}_{q}(\mathbf{\Sigma},\Delta)\cong\mathbb{T}_{q}(\overline{K}_{\Delta},\langle\cdot,\cdot\rangle)$.

######  Definition 5.18 (Lê–Yu [53]).

The injective morphism
$\phi\colon\mathcal{S}_{A}(\mathbf{\Sigma}_{g}^{*})\hookrightarrow\overline{\mathcal{Z}}_{q}(\mathbf{\Sigma}_{g}^{*},\Delta)$
will be referred to as the refined quantum trace.

Let $\overline{K}_{\Delta}^{0}$ be the kernel of the skew-symmetric form

$\overline{K}_{\Delta}\otimes_{\mathbb{Z}}\overline{K}_{\Delta}\xrightarrow{\langle\cdot,\cdot\rangle}\mathbb{Z}\to\mathbb{Z}/N\mathbb{Z}.$

Let $\mathbf{k}_{\widehat{a_{\partial}}}\in\overline{K}_{\Delta}$ be defined
by $\mathbf{k}_{\widehat{a_{\partial}}}(\widehat{a_{\partial}})=2$ and
$\mathbf{k}_{\widehat{a_{\partial}}}(e)=0$ for all $e\in\mathcal{E}(\Delta)$.
So by definition, one has
$\overline{K}_{\Delta}=K_{\Delta}\oplus\mathbb{Z}\mathbf{k}_{\widehat{a_{\partial}}}$.

######  Lemma 5.19.

One has

$\overline{K}_{\Delta}^{0}=K_{\Delta}^{0}\oplus
N\mathbb{Z}\mathbf{k}_{\widehat{a_{\partial}}}.$

Therefore the PI-degree of
$\overline{\mathcal{Z}}_{q}(\mathbf{\Sigma}_{g}^{*},\Delta)$ is equal to
$N^{3g}$, i.e., to the PI-degree of
$\mathcal{S}_{A}(\mathbf{\Sigma}_{g}^{*})$.

###### Proof.

Let
$\overline{\mathbf{k}}_{1}=\mathbf{k}_{1}+n_{1}\mathbf{k}_{\widehat{a_{\partial}}}$
and
$\overline{\mathbf{k}}_{2}=\mathbf{k}_{2}+n_{2}\mathbf{k}_{\widehat{a_{\partial}}}$
be two maps in
$\overline{K}_{\Delta}=K_{\Delta}\oplus\mathbb{Z}\mathbf{k}_{\widehat{a_{\partial}}}$.
Then a straightforward computation shows that

$\big{\langle}\overline{\mathbf{k}}_{1},\overline{\mathbf{k}}_{2}\big{\rangle}=\big{(}i\big{(}\overline{\mathbf{k}}_{1}\big{)},i\big{(}\overline{\mathbf{k}}_{2}\big{)}\big{)}^{\rm
WP}=(\mathbf{k}_{1},\mathbf{k}_{2})^{\rm
WP}+n_{1}\mathbf{k}_{1}(a_{\partial})-n_{2}\mathbf{k}_{2}(a_{\partial}).$

The equality $\overline{K}_{\Delta}^{0}=K_{\Delta}^{0}\oplus
N\mathbb{Z}\mathbf{k}_{\widehat{a_{\partial}}}$ follows. We deduce the index:

$\big{|}\overline{K}_{\Delta}:\overline{K}_{\Delta}^{0}\big{|}=N\times\big{|}K_{\Delta}:K_{\Delta}^{0}\big{|}=N\times
N^{3g-1}=N^{3g}.$

Therefore the PI-degree of
$\overline{\mathcal{Z}}_{q}(\mathbf{\Sigma}_{g}^{*},\Delta)$ is equal to
$N^{3g}$. ∎

Let
$\overline{\mathcal{Y}}(\mathbf{\Sigma}_{g}^{*},\Delta):=\operatorname{Specm}\big{(}\overline{\mathcal{Z}}_{+1}(\mathbf{\Sigma}_{g}^{*},\Delta)\big{)}$.
The refined quantum trace taken at $A^{1/2}=+1$ defines a dominant morphism

$\overline{\mathcal{NA}}\colon\
\overline{\mathcal{Y}}(\mathbf{\Sigma}_{g}^{*},\Delta)\to\mathcal{X}(\mathbf{\Sigma}_{g}^{*}).$

A point $\rho\in\mathcal{X}(\mathbf{\Sigma}_{g}^{*})$ is said to admit a
$\Delta$-lift if it is in the image of $\overline{\mathcal{NA}}$. The
dominance of $\overline{\mathcal{NA}}$ and the irreducibility of
$\mathcal{X}(\mathbf{\Sigma}_{g}^{*})$ show that this is a generic condition.
A point $\rho$ in the big cell
$\mathcal{X}^{0}=\mu^{-1}\big{(}\operatorname{SL}_{2}^{0}\big{)}$ admits a
$\Delta$-lift if and only if the representation
$r_{\rho}\colon\mathcal{S}_{A}(\mathbf{\Sigma}_{g}^{*})\to\operatorname{End}(V(\rho))$
extends through $\phi$ to a representation of
$\overline{\mathcal{Z}}_{q}(\mathbf{\Sigma}_{g}^{*},\Delta)$.

For $\gamma\subset\Sigma_{g,1}$ a simple closed curve, since $\gamma$ can be
isotoped outside of the triangle $\mathbb{T}_{0}$, the formula in Lemma 5.12
still holds, that is one has

$\phi([\gamma])=\sum_{\hat{s}\in{\rm
St}^{a}(\gamma)}A^{n(\hat{s})/2}Z^{\mathds{k}(\hat{s})}$

for the same integers $n(\hat{s})$ as in Lemma 5.12. Consider $K_{\Delta}$ as
the subset of $\overline{K}_{\Delta}$ of maps sending $\widehat{a}_{\partial}$
to $0$ and recall the definition of the set $\overline{St}^{a}(\gamma)\subset
K_{\Delta}\subset\overline{K}_{\Delta}$. For
$\mathbf{z}\in\overline{\mathcal{Y}}(\mathbf{\Sigma}_{g}^{*},\Delta)$ and
$\mathbf{k}\in K_{\Delta}$, define the numbers
$x_{\mathbf{z},\mathbf{k}}(\gamma)\in\mathbb{C}$ in the same manner than in
the previous subsection, i.e., by the formula

$x_{{\mathbf{z}},\mathbf{k}}(\gamma):=\sum_{\mathbf{k}_{0}\in
S(\gamma),\,\mathbf{k}+\mathbf{k}_{0}=\mathds{k}(\hat{s})}{\mathbf{z}}(\mathbf{k}_{0})A^{n(\hat{s})/2}.$

Let
$\mathcal{J}_{\mathbf{z}}\subset\overline{\mathcal{Z}}_{q}(\mathbf{\Sigma}_{g}^{*},\Delta)$
be the ideal generated by the kernel of $\mathbf{z}$. The following analogues
of Lemma 5.15, Proposition 5.11 and Theorem 5.17 hold in the non reduced case.

######  Proposition 5.20.

  1. $1.$

For $\mathbf{z}\in\overline{\mathcal{Y}}(\mathbf{\Sigma}_{g}^{*},\Delta)$ and
$\lambda\in\mathbb{C}^{*}$, one has
$\phi(\alpha)\equiv\lambda\phi(\beta)\pmod{\mathcal{J}_{\mathbf{z}}}$ if and
only if $x_{\mathbf{z},\mathbf{k}}(\alpha)=\lambda
x_{\mathbf{z},\mathbf{k}}(\beta)$ for all $\mathbf{k}\in K_{\Delta}$.

  2. $2.$

Let $\pi\colon G\to\operatorname{End}(W(\mathcal{O}))$ a representation
associated to an orbit in the big cell, $\phi\in G$ and $\alpha$ a simple
closed curve such that $\alpha$ and $\beta:=\phi(\alpha)$ are not isotopic.
Suppose that there exists $\rho\in\mathcal{O}$ which admits a $\Delta$-lift,
i.e., such that $\rho=\overline{\mathcal{NA}}(\mathbf{z})$. Further suppose
that

$\phi(\alpha)\not\equiv\lambda\phi(\beta)\pmod{\mathcal{J}_{\mathbf{z}}},\qquad\text{for
all }\lambda\in\mathbb{C}^{*}.$

Then $\pi(\phi)\neq{\rm id}$.

  3. $3.$

Let $\pi\colon G\to\operatorname{End}(W(\mathcal{O}))$ a representation
associated to an orbit in the big cell, $\phi\in G$ and $\alpha$ a simple
closed curve such that $\alpha$ and $\beta:=\phi(\alpha)$ are not isotopic. If
$(1)$ there exists $\rho\in\mathcal{O}$ which admits a $\Delta$-lift and $(2)$
both $\alpha$ and $\beta$ intersect the edges of $\Delta$ at most $N-1$ times,
then $\pi(\phi)\neq{\rm id}$.

###### Proof.

The proofs are straightforward adaptations of the proofs of Lemma 5.15,
Proposition 5.11 and Theorem 5.17, using Lemma 5.19, left to the reader. ∎

### Acknowledgements

The author thanks S. Baseilhac, D. Callaque, F. Costantino, A. Quesney, T.Q.T.
Lê and P. Safronov for valuable conversations. He also thanks the anonymous
referees for their very detailed reports which improved the quality and
readability of this paper. He acknowledges support from the Japanese Society
for Promotion of Sciences, from the Centre National de la Recherche
Scientifique and from the ERC DerSympApp (Grant 768679).

## References

  * [1] Alekseev A.Yu., Grosse H., Schomerus V., Combinatorial quantization of the Hamiltonian Chern–Simons theory. I, Comm. Math. Phys. 172 (1995), 317–358, arXiv:hep-th/9403066.
  * [2] Alekseev A.Yu., Grosse H., Schomerus V., Combinatorial quantization of the Hamiltonian Chern–Simons theory. II, Comm. Math. Phys. 174 (1996), 561–604, arXiv:hep-th/9408097.
  * [3] Alekseev A.Yu., Kosmann-Schwarzbach Y., Meinrenken E., Quasi-Poisson manifolds, Canad. J. Math. 54 (2002), 3–29, arXiv:math.DG/0006168.
  * [4] Alekseev A.Yu., Malkin A.Z., Symplectic structures associated to Lie–Poisson groups, Comm. Math. Phys. 162 (1994), 147–173, arXiv:hep-th/9303038.
  * [5] Alekseev A.Yu., Malkin A.Z., Symplectic structure of the moduli space of flat connection on a Riemann surface, Comm. Math. Phys. 169 (1995), 99–119, arXiv:hep-th/9312004.
  * [6] Alekseev A.Y., Schomerus V., Representation theory of Chern–Simons observables, Duke Math. J. 85 (1996), 447–510, arXiv:q-alg/9503016.
  * [7] Ben-Zvi D., Brochier A., Jordan D., Integrating quantum groups over surfaces, J. Topol. 11 (2018), 874–917, arXiv:1501.04652.
  * [8] Ben-Zvi D., Brochier A., Jordan D., Quantum character varieties and braided module categories, Selecta Math. (N.S.) 24 (2018), 4711–4748, arXiv:1606.04769.
  * [9] Blanchet C., Costantino F., Geer N., Patureau-Mirand B., Non-semi-simple TQFTs, Reidemeister torsion and Kashaev’s invariants, Adv. Math. 301 (2016), 1–78, arXiv:1404.7289.
  * [10] Bonahon F., Wong H., Quantum traces for representations of surface groups in ${\rm SL}_{2}(\mathbb{C})$, Geom. Topol. 15 (2011), 1569–1615, arXiv:1003.5250.
  * [11] Bonahon F., Wong H., Representations of the Kauffman bracket skein algebra I: invariants and miraculous cancellations, Invent. Math. 204 (2016), 195–243, arXiv:1206.1638.
  * [12] Bonahon F., Wong H., The Witten–Reshetikhin–Turaev representation of the Kauffman bracket skein algebra, Proc. Amer. Math. Soc. 144 (2016), 2711–2724, arXiv:1309.0921.
  * [13] Bonahon F., Wong H., Representations of the Kauffman bracket skein algebra II: Punctured surfaces, Algebr. Geom. Topol. 17 (2017), 3399–3434, arXiv:1206.1639.
  * [14] Brown K.A., Goodearl K.R., Lectures on algebraic quantum groups, Advanced Courses in Mathematics. CRM Barcelona, Birkhäuser Verlag, Basel, 2002.
  * [15] Brown K.A., Gordon I., The ramification of centres: Lie algebras in positive characteristic and quantised enveloping algebras, Math. Z. 238 (2001), 733–779, arXiv:math.RT/9911234.
  * [16] Brown K.A., Gordon I., Poisson orders, symplectic reflection algebras and representation theory, J. Reine Angew. Math. 559 (2003), 193–216, arXiv:math.RT/0201042.
  * [17] Brown K.A., Yakimov M.T., Azumaya loci and discriminant ideals of PI algebras, Adv. Math. 340 (2018), 1219–1255, arXiv:1702.04305.
  * [18] Buffenoir E., Roche P., Two-dimensional lattice gauge theory based on a quantum group, Comm. Math. Phys. 170 (1995), 669–698, arXiv:hep-th/9405126.
  * [19] Buffenoir E., Roche P., Link invariants and combinatorial quantization of Hamiltonian Chern–Simons theory, Comm. Math. Phys. 181 (1996), 331–365, arXiv:q-alg/9507001.
  * [20] Bullock D., A finite set of generators for the Kauffman bracket skein algebra, Math. Z. 231 (1999), 91–101.
  * [21] Bullock D., Frohman C., Kania-Bartoszyńska J., Topological interpretations of lattice gauge field theory, Comm. Math. Phys. 198 (1998), 47–81, arXiv:q-alg/9710003.
  * [22] Cooke J., Excision of skein categories and factorisation homology, arXiv:1910.02630.
  * [23] Costantino F., Lê T.T.Q., Stated skein algebras of surfaces, J. Eur. Math. Soc., to appear, arXiv:1907.11400.
  * [24] Costantino F., Martelli B., An analytic family of representations for the mapping class group of punctured surfaces, Geom. Topol. 18 (2014), 1485–1538, arXiv:1210.2666.
  * [25] De Concini C., Kac V.G., Representations of quantum groups at roots of $1$, in Operator Algebras, Unitary Representations, Enveloping Algebras, and Invariant Theory (Paris, 1989), Progr. Math., Vol. 92, Birkhäuser Boston, Boston, MA, 1990, 471–506.
  * [26] De Concini C., Lyubashenko V., Quantum function algebra at roots of $1$, Adv. Math. 108 (1994), 205–262.
  * [27] De Concini C., Procesi C., Quantum groups, in $D$-Modules, Representation Theory, and Quantum Groups (Venice, 1992), Lecture Notes in Math., Vol. 1565, Springer, Berlin, 1993, 31–140.
  * [28] De Renzi M., Gainutdinov A.M., Geer N., Patureau-Mirand B., Runkel I., Commun. Contemp. Math., to appear, arXiv:2010.14852.
  * [29] Drinfel’d V.G., On constant quasiclassical solutions of the Yang–Baxter equations, Soviet Math. Dokl. 28 (1983), 667–671.
  * [30] Faitg M., Holonomy and (stated) skein algebras in combinatorial quantization, arXiv:2003.08992.
  * [31] Fock V.V., Rosly A.A., Poisson structure on moduli of flat connections on Riemann surfaces and the $r$-matrix, in Moscow Seminar in Mathematical Physics, Amer. Math. Soc. Transl. Ser. 2, Vol. 191, Amer. Math. Soc., Providence, RI, 1999, 67–86, arXiv:math.QA/9802054.
  * [32] Frohman C., Kania-Bartoszynska J., Lê T., Unicity for representations of the Kauffman bracket skein algebra, Invent. Math. 215 (2019), 609–650, arXiv:1707.09234.
  * [33] Frohman C., Kania-Bartoszynska J., Lê T., Dimension and trace of the Kauffman bracket skein algebra, Trans. Amer. Math. Soc. Ser. B 8 (2021), 510–547, arXiv:1902.02002.
  * [34] Ganev I., Jordan D., Safronov P., The quantum Frobenius for character varieties and multiplicative quiver varieties, arXiv:1901.11450.
  * [35] Gunningham S., Jordan D., Safronov P., The finiteness conjecture for skein modules, arXiv:1908.05233.
  * [36] Guruprasad K., Huebschmann J., Jeffrey L., Weinstein A., Group systems, groupoids, and moduli spaces of parabolic bundles, Duke Math. J. 89 (1997), 377–412, arXiv:dg-ga/9510006.
  * [37] Habiro K., A note on quantum fundamental groups and quantum representation varieties for $3$-manifolds, RIMS Kōkyūroku 1777 (2012), 21–30.
  * [38] Haïoun B., Relating stated skein algebras and internal skein algebras, SIGMA 18 (2022), 042, 39 pages, arXiv:2104.13848.
  * [39] Hartshorne R., Algebraic geometry, Graduate Texts in Mathematics, Vol. 52, Springer-Verlag, New York – Heidelberg, 1977.
  * [40] Higgins V., Triangular decomposition of ${\rm SL}_{3}$ skein algebras, arXiv:2008.09419.
  * [41] Hodges T.J., Levasseur T., Primitive ideals of ${\bf C}_{q}[{\rm SL}(3)]$, Comm. Math. Phys. 156 (1993), 581–605.
  * [42] Korinman J., Decomposition of some Witten–Reshetikhin–Turaev representations into irreducible factors, SIGMA 15 (2019), 011, 25 pages, arXiv:1406.4389.
  * [43] Korinman J., Triangular decomposition of character varieties, arXiv:1904.09022.
  * [44] Korinman J., Finite presentations for stated skein algebras and lattice gauge field theory, Algebr. Geom. Topol., to appear, arXiv:2012.03237.
  * [45] Korinman J., Stated skein algebras and their representations, arXiv:2105.09563.
  * [46] Korinman J., Stated skein algebras and their representations, RIMS Kōkyūroku 2191 (2021), 52–71.
  * [47] Korinman J., Unicity for representations of reduced stated skein algebras, Topology Appl. 293 (2021), 107570, 28 pages, arXiv:2001.00969.
  * [48] Korinman J., Murakami J., Relating quantum character varieties and skein modules, in preparation.
  * [49] Korinman J., Quesney A., Classical shadows of stated skein representations at odd roots of unity, arXiv:1905.03441.
  * [50] Korinman J., Quesney A., The quantum trace as a quantum non-abelianization map, J. Knot Theory Ramifications 31 (2022), 2250032, 49 pages, arXiv:1907.01177.
  * [51] Lê T.T.Q., Triangular decomposition of skein algebras, Quantum Topol. 9 (2018), 591–632, arXiv:1609.04987.
  * [52] Lê T.T.Q., Yu T., Stated skein modules of marked 3-manifolds/surfaces, a survey, Acta Math. Vietnam. 46 (2021), 265–287, arXiv:2005.14577.
  * [53] Lê T.T.Q., Yu T., Quantum traces and embeddings of stated skein algebras into quantum tori, Selecta Math. (N.S.) 28 (2022), 66, 48 pages, arXiv:2012.15272.
  * [54] Majid S., Foundations of quantum group theory, Cambridge University Press, Cambridge, 1995.
  * [55] Przytycki J.H., Sikora A.S., Skein algebras of surfaces, Trans. Amer. Math. Soc. 371 (2019), 1309–1332, arXiv:1602.07402.
  * [56] Reiner I., Maximal orders, London Mathematical Society Monographs New Series, Vol. 28, The Clarendon Press, Oxford University Press, Oxford, 2003.
  * [57] Reshetikhin N., Turaev V.G., Invariants of $3$-manifolds via link polynomials and quantum groups, Invent. Math. 103 (1991), 547–597.
  * [58] Turaev V.G., Quantum invariants of knots and 3-manifolds, De Gruyter Studies in Mathematics, Vol. 18, Walter de Gruyter & Co., Berlin, 2010.