# The Power of Combined Modalities in Interactive Robot Learning

Helen Beierling1 and Anna-Lisa Vollmer1 This work was funded by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation): TRR 318/1 2021 –
438445824.1The authors are with the Medical School Ostwestfalen-Lippe and the
Center for Cognitive Interaction Technology, Bielefeld University, Bielefeld,
Germany<EMAIL_ADDRESS>
###### Abstract

This study contributes to the evolving field of robot learning in interaction
with humans, examining the impact of diverse input modalities on learning
outcomes. It introduces the concept of “meta-modalities,” which encapsulate
additional forms of feedback beyond the traditional preference and scalar
feedback mechanisms. Unlike prior research that focused on individual meta-
modalities, this work evaluates their combined effect on learning outcomes.
Through a study with human participants, we explore user preferences for these
modalities and their impact on robot learning performance. Our findings reveal
that while individual modalities are perceived differently, their combination
significantly improves learning behavior and usability. This research not only
provides valuable insights into the optimization of human-robot interactive
task learning but also opens new avenues for enhancing the interactive freedom
and scaffolding capabilities provided to users in such settings.

##  I Introduction

Artificial intelligence (AI) now permeates various aspects of our lives,
exemplified by virtual assistants like ChatGPT and suggestion algorithms on
social media platforms. AI systems often rely on user feedback to learn and
improve. This feedback can take various forms, from traditional star ratings
to more passive indicators like watch time.

There is a growing demand for intelligent agents that can simplify everyday
tasks, particularly in home environments characterized by diverse individual
preferences and tasks.

To address this, users, especially lay users, need to be empowered to teach
robots new actions according to their preferences. Interactive learning with
users in the loop, based on active feedback, is a common method for achieving
this, where scalar and preference feedback are popular options [1].

Figure 1: Study setup. Red depicts the used robot platform, green the task
relevant objects, purple the camera for study recording, and yellow the
objects the user gets in contact with.

Research suggests that humans are more adept at comparing options than rating
them on a scale, making preference feedback a foundational element [2].
Additionally, literature advocates for modalities that allow users to express
supplementary feedback to enhance the basis feedback quality and guide actions
effectively [3, 4, 5, 6, 7, 8, 9] ; we refer to these as ”meta-modalities”.
While the literature presents various beneficial meta-modalities, they have
been largely tested in isolation. Therefore, our study explores whether all
modalities are utilized and perceived equally, and whether their combined use
offers a learning benefit.

##  II Related Work

Human inputs in reinforcement scenarios are valuable, given that reward
functions customized for specific situations may underperform in diverse
contexts [1]. Additionally, it can be challenging to define such functions
rationally via reward engineering, especially for tasks based on personal
preferences. There is abundant literature spanning from older approaches [10,
11] to more contemporary ones [2, 12] regarding the utilization of human
feedback that were able to show positive results with various usage methods of
human feedback. Thus, the approach of incorporating human feedback in
reinforcement learning is current and relevant, particularly within the realm
of robotics.

Concerning the base modality, we concentrate on the input modalities
identified as most common. In the survey by Kaufmann et al. [1], examining
existing approaches for reinforcement learning with direct user feedback, they
are classified into preference-based and reward-based learning approaches.
Here, users provide numerical or binary evaluations of the agent’s actions,
reflecting evaluative feedback. In addition, emotions [13, 14], gestures [14],
or speech [15] can also be recognized and used as feedback individually or in
combination. However, this paper focuses on direct evaluative feedback
methods. Our decision to adopt preference as the base feedback method was
based on a recent study comparing preference and scalar feedback within a
reinforcement learning algorithm based on evolutionary black-box optimization
in a robot setup [2]. The findings indicated that users were better able to
compare two trajectories than provide scalar ratings within their own
contextual framework of past ratings for other trajectories.
In addition to the base modality, we identified six modalities from the
literature that build upon them.

Guidance: An earlier study by Thomaz et al. [3] highlights the importance of
providing users with the option to provide feedback not only for present
actions but also for future actions, giving guidance to the robot. This method
not only prevents users from misusing feedback options for guidance but also
enhances learning outcomes.

Correction: Liu et al. [4] suggested the opposite of guidance, namely
correction. The literature suggests that using corrections for human-in-the-
loop robot learning benefits learning [4, 5, 6, 7].

Demonstration: Another modality is the possibility to demonstrate an action to
the robot to assist or lead the learning process. Approaches like kinesthetic
teaching are increasingly popular [8].

Exploration: While the exploration versus exploitation dilemma is a
longstanding challenge in machine learning, it becomes particularly pronounced
in scenarios where the algorithm receives sparse rewards. In such cases,
random exploration risks wasting valuable rewards. However, this challenge can
be mitigated by leveraging demonstrations to guide exploration. Besides
demonstration which we already covered, one can directly incorporate the
user’s guidance for exploration [16, 9].

Speed: A modality closely aligned with the action advice concept outlined by
Arzate Cruz et al. [9] involves offering guidance on specific attributes of an
action. We selected execution speed as a simple attribute for user guidance.

Fallback: Lastly, there is the option to revert to a previous state of action
and resume learning from there. This modality, grounded in the concept of
correction or negative guidance, as outlined by Thomaz et al. [3]. In
instances where the agent converges to undesired behavior, a fallback modality
allows for retracing the explored path and restarting from the best action
achieved thus far.

While there is considerable research on individual meta-modalities, their
combination remains an open question. A survey focusing on interactive
robotics and human-robot interactions, examining trendsetting human feedback
methods highlights the open challenge of combining feedback types and their
application to reinforcement learning algorithms [9]. Similarly, the potential
of combining feedback methods in interactive reinforcement has been emphasized
[17].

Our work aims to address this research gap and study the combination of
multiple feedback modalities for robot learning in interaction with human
users. Our objective is to investigate whether learning behavior improves,
which modalities users utilize, and which modality they prefer.

##  III Methodology

###  III-A Implementation

The modalities were implemented on a Kinova Gen2/Jaco (Jaco2) robotic arm [18]
using ROS Noetic [19].

To represent movements, we utilized an implementation of probabilistic
movement primitives (ProMP) [20, 21]. ProMP depict movements based on
parameterized distributions, rendering them highly adaptable as they
encapsulate entire distributions rather than singular trajectories. Moreover,
they boast a compact design, necessitating adjustments solely to their
parameters to accommodate changes in trajectories. We implemented the PIBB
black-box algorithm for learning purposes [22]. The PIBB algorithm learns the
correlation between ProMP weights and provided feedback, thereby refining a
distribution of ProMP weights utilized to generate new, improved movements. In
our study, we also developed a graphical user interface (GUI) to showcase the
modalities available and to capture user interactions for our analysis. We
configured two versions of our GUI: one that incorporates only the basic
modality and another that, in addition, offers all meta modalities to the
participants, see Figure 2.

###  III-B Modalities

####  III-B1 Base Modality Preference

For the preference modality, two different behaviors are presented, from which
the participants the one they prefer. We used the function

$\begin{cases}(100,100)&\texttt{Both preferred}\\\ (100,-100)&\texttt{First
action preferred}\\\ (-100,100)&\texttt{Second action preferred}\\\
(-100,-100)&\texttt{None preferred}\end{cases}$

This approach not only enables the articulation of a preference for one option
but also accommodates the expression of preference for neither or both
options. Our choice for this implementation was driven, among other factors,
by the need for the algorithm to perform rapidly and sample-efficiently. To
amplify the differentiation in actions, we increased the disparity between
positive and negative rewards, addressing the issue that users struggle to
express their preferences when actions are too similar. We also opted for
samplewise updates to the sampling instead of batchwise, as users in direct
interaction expect immediate responses [23].

####  III-B2 Meta Modalities

The $PIBB^{2}$ algorithm calculates a weighted average, weighted by so called
$PIBB^{2}$ weights. These are produced by computing the norm from the $ProMP$
weights and the rewards they received. We additionally incorporated a decay
factor for past rewards $(=0.9)$, in addition to the covariance decay factor
$(=0.973)$, which diminishes the variance of the produced distribution. The
resulting rewards are then normalized by

$(r_{n})_{i}=-h*(1-((r_{i}-min(rewards))/reward\>range))$

with $h=10$ the eliteness factor of $PIBB^{2}$. The normalized rewards are
then exponentiated which makes them $PIBB^{2}$. Those are lastly normalized by
the sum of weights.

##### Guidance

was achieved by significantly increasing the influence of the action chosen as
guidance in the selection of future actions, while drastically diminishing the
impact of all other actions. This is achieved influencing the custom decay of
rewards so it only spare the last sample marked as guidance. In case, one of
the last preference samples was marked as guidance, additionally all other
rewards are decayed by a guidance decay factor of $(0.5)$. Furthermore, the
normalization is enhanced by multiplying the eliteness factor by $1.3$. This
intensifies their influence on the normalized rewards positively due to their
positive rewards. Lastly, they receive higher rewards of $150$ instead of
$100$.

##### Correction

, as the inverse of Guidance, is achieved by significantly reducing the impact
of the chosen action to deter its selection in the future. Unlike guidance
samples, correction samples do not induce a decay, as negative actions should
retain their influence in contrast to previously positive actions.
Furthermore, they are calculated differently during the normalization in the
same as the Guidance since they receive negative rewards they are negatively
enhanced. Correction samples also receive more negative rewards of $-150$
instead of $-100$.

##### Demonstrations

were done via kinesthetic teaching and served as a new average for generating
subsequent actions. Given that users anticipate behavior closely mirroring the
demonstration, past inputs are disregarded; otherwise, movements, particularly
in the later stages of the learning trajectory, would deviate significantly
from the demonstration due to the influence of previous actions.

##### Exploration

was facilitated by adjusting the base exploration factor upwards or downwards
on five levels, ensuring, however, that exploration consistently decreases
over time.

##### Speed

was realized as accelerating or decelerating all movements after trajectories
were generated, eliminating the need to incorporate an additional dimension
into the learning process.

##### Fallback

was implemented by marking a movement that can be loaded by the user as
needed. When a fallback is saved it is marked as a guidance as well. This
triggers all positive effects of a guidance since a fallback sample is marked
as the best so far by the user. If it is loaded it has the same effect as
providing a demonstration.

###  III-C Study Design

####  III-C1 Hypotheses

H1: Enhanced Learning through Multiple Modalities The learning process in a
human-in-the-loop reinforcement learning framework can be improved by
providing multiple modalities of direct user input.
H2: User Preferences and Modality Utilization Users do not utilize all
modalities equally but tend to show a preference for certain ones.
H3: Differential Benefits of Modalities Some modalities are perceived by users
to be more beneficial compared to others in facilitating the learning process.
H4: Enhanced Learning through Individual Modalities The use of individual
modalities positively correlate with the efficacy of the learning algorithm
when they are employed.

####  III-C2 Participants

The participant cohort for the study was randomly assigned to two groups.
Group 1 as the baseline condition only expressed their preference between two
movements and consisted of 15 individuals: 6 male, 8 female, and 1 identifying
as diverse. Group 2 was made available all additional meta-modalities and
comprised 18 participants: 9 male and 9 female. Across groups, there were 18
individuals with a higher education entrance qualification and 15 with a
higher education degree, totaling 33 participants. All 33 participants were
German-speaking, aligned with the German-only interface of the system used.
Participants were recruited on campus through flyers, posters, and mailing
lists. All subjects gave their informed consent for inclusion before they
participated in the study. The study was conducted in accordance with the
Declaration of Helsinki, and the protocol was approved by the Ethics Committee
of Bielefeld University.

####  III-C3 Task

The task designed for the study was a minigolf challenge without obstacles,
chosen for its balanced blend of enjoyment, clarity, and practicality. The
setup of the task is depicted in Figure 1. The simplicity of the objective
made the goal and proper technique apparent for evaluation. Furthermore, the
straightforward nature of the task allowed for easy reset between attempts. In
this process, the robot’s action pairs were evaluated 40 times, even if the
task had been completed beforehand. The participants were instructed to
succeed (i.e., the robot hits the hole) as often as possible.

####  III-C4 Procedure

The study began with a welcome and briefing on the task of teaching a robotic
arm to play minigolf. The protocol detailed the use of a user interface for
rating the robot’s task executions. The GUI for Group 1 was designed for
simple interaction. It displays two options, ’Bewegung 1’ for the first and
’Bewegung 2’ for the second movement, each with a replay button and a heart
symbol for expressing preference. At the bottom, there is a submit button for
confirming choices. The second GUI for Group 2 was configured to evaluating
robotic movements with all meta modalities, see Figure 2. In addition to the
features available to Group 1, it also have the option to save movements as
fallback. Furthermore, central sliders allow users to adjust exploration rates
and movement speed. The central button at the bottom facilitates the
demonstration of movements, and below on the right there is the option to load
a fallback movement.

Figure 2: This Figure showcases the interface for Group 2, equipped with all
meta-modalities and the base modality.

After the introduction to the corresponding GUI, the study protocol outlined
the recording of choices and timing for analysis purposes, concluding with a
post-interaction questionnaire described in the following and compensation
details.

####  III-C5 Data Acquisition and Analysis

Data collection focused according to the hypotheses and study aims on two
primary objectives: evaluating learning progress and assessing user
satisfaction and preference.

Learning progress was analyzed through video recordings. Our primary metric
for assessing learning progress was the completion of the task, specifically
measuring how often the task was completed successfully and the timing of the
first success (i.e., the first hit). We examined not only the total of
successful hits compared between the two groups but also across individual
trials (1-40), noting how many participants achieved a hit in each attempt.
Concerning user satisfaction, overall satisfaction with the system for both
groups was measured on a 5-point Likert scale. We calculated the significance
across the groups to analyze the data. Group 2 participants also responded to
a SUS questionnaire [24] for each meta-modality. From the calculated SUS
scores, the average was determined for each meta-modality and compared
pairwise. Preferences for the modalities were measured not only through SUS
evaluations but also directly via rankings of the meta modalities. We also
collected the reasons behind these rankings. For each option in the ranking,
we calculated the percentage frequency of its placement in each position. The
qualitative statements from participants were analyzed by categorizing them
into thematic groups and then evaluating these categories based on their size.
Lastly, we analyzed the relationship between learning success and modality
usage.

##  IV Results

Participants in Group 2 achieved their first hit on average after 17.56
($SD=13.65$) attempts, whereas participants in Group 1 required 34.60
($SD=27.93$) attempts to reach their first hit. The results from the Exact
Wilcoxon-Mann-Whitney Test suggests a trend, which does not reach statistical
significance of $p<0.05$ ($Z=1.7622$, $p=0.0796$).

Comparing the total number of hits between Group 1 and Group 2 yielded a
significant difference in the performance of the two groups, with Group 2
hitting more frequent ($Z=-3.24$, $p=0.0007$) as presented in Figure 3.

Figure 3: This graph illustrates the percentage of successful hits per trial
across all participants.

Concerning participant satisfaction, the overall satisfaction level of Group 1
was consistently moderate, with $M=3.31$ ($SD=1.14$). The mean scores for
overall satisfaction in Group 2 were both notably higher, standing at $M=4.17$
($SD=0.857$). The Asymptotic Kruskal-Wallis Test indicates a significant
difference in satisfaction values, $\chi^{2}(1,34)=5.3026$, $p<0.05$, with
$p=0.022$ , revealing that participants in Group 2 were more satisfied with
the system than those in Group 1. Due to the demonstration modality being
sparsely used, we excluded it from the SUS score evaluation, as the results
would not be very representative. With mean SUS scores of $M=79.17$
($SD=8.83$) for Guidance, $M=79.67$ ($SD=14.81$) for Speed, $M=74.22$
($SD=15.80$) for Correction, $M=72.67$ ($SD=12.12$) for Fallback, and
$M=68.82$ ($SD=16.94$) for Exploration, the system is generally perceived as
good in usability; Guidance and Speed are highly satisfactory, while
Correction and Fallback are seen as good but with room for improvement, and
Exploration, though still good, is identified as having the most noticeable
areas for enhancement.

The Friedman test suggests a significant difference in the SUS-Score-values,
with $\chi^{2}(4,19)=17.777,p<0.01$ and $p=0.0013$. A significant difference
was identified between Exploration and Guidance ($p=0.008$), with Guidance
being perceived as more usable. Between Exploration and Speed, a significant
difference was found ($p=0.0023$), with Speed being rated higher than
Exploration. A trend was observed between Fallback and Speed ($p=0.029$),
suggesting that Fallback was perceived as better. No significant differences
were found in the other pairwise comparisons.

We have illustrated the results of the ranking of meta-modalities in Figure 4,
showing the proportion of rankings for each modality every time it was ranked.
The results indicate a preference for the “Guidance” modality, which was
mostly placed on the first and second rank positions, showcasing its
favorability among participants. “Speed” is favored as well, with a similar
portion of its rankings in the first place, although it also shows a spread
across third to even fifth ranks in contrast to “Guidance”. The “Correction”
modality is ranked with moderate preference, evident from its distribution
mostly across second to fourth ranks, indicating a balanced reception.
Similarly, “Fallback” shows a slight leaning towards the first rank but also
to the fifth rank, thus its distribution suggests it is only moderately
preferred. “Exploration” is arguably the least preferred, as the majority of
its rankings are in the lower fourth and fifth ranks. Overall, “Guidance” and
“Speed” were the most liked, while “Exploration” is far behind in the ranking.

Figure 4: This diagram showcases the ranking distribution of the meta
modalities. It illustrates the percentage placement of each meta modality
across the ranking positions by Group 2 participants.

Concerning the usage frequency, we evaluated two key metrics: a) the total
usage means per participant where a modality, when utilized, is counted once;
b) the relative count which aggregates across all participants, measuring how
frequently each modality was used overall. The results are displayed in Figure
5.

Figure 5: This Figure illustrates, on one hand, the relative usage of
modalities across all clicks and, on the other, the usage among participants,
taking into account whether the modality was used at least once.

The “Demonstration” modality saw minimal use, with just $0.67\%$ of all
interactions and only $25\%$ of participants using it. Fallback was divided
into the save of a Fallback point and the load of such points. In contrast,
“Fallback load” was used by a higher percentage of participants ($50\%$),
despite accounting for only $3.53\%$ of all interactions. On the other hand,
“Fallback saved” was not only used by a majority ($87.5\%$) of participants
but also were a higher portion of interactions at $16.97\%$. “Speed” stood out
with the highest engagement, utilized by $93.75\%$ of participants and making
up $12.44\%$ of all interactions. “Exploration” was also popular, with
$81.25\%$ of participants engaging with it and $26.05\%$ share of total
modality use. Similarly, “Correction” was used by the same percentage of
participants as “Speed” ($93.75\%$), but with a greater interaction with
$22.52\%$ participants using it. Meanwhile, “Guidance” was used by $56.25\%$
of participants and $17.82\%$ of all interactions. Overall, the “Speed” and
“Correction” modalities lead in participant usage, with “Correction” also
notable for interaction frequency alongside “Exploration”, whereas
“Demonstration” lags behind with the lowest interaction rate and participant
engagement.
As previously described, we also captured the participants’ explanations for
the rankings analyzed above. These are depicted in Figure 6, categorized into
positive justifications for higher placements and more critical reasons for
lower rankings.

Figure 6: Presentation of Participant Rationale for Meta-Modality Rankings

Overall, the reason that was most often given for higher rankings was “Goal
Efficiency” and “Understandability”. Due to space restrictions, for all other
rankings, we refer the reader to the figure.

Spearman’s rank correlation was used to analyze the correlations of hits per
try and modalities used per try. A correlation between Exploration and hit
success unveiled a statistically significant, albeit weak, negative
correlation, with $(\rho=-0.21,p=0.025)$. Thus, with a decrease of Exploration
the likelihood of hitting the target marginally increases. The analysis of
Speed revealed a modest positive correlation $(\rho=0.2389221,p=0.03898)$.
This suggests that higher speeds correlate with a slightly improved chance of
hitting the target. Contrastingly, the correlation with load $(\rho=-0.01)$
and save $(\rho=0.007)$ of a fallback were not only very weak, but both also
lacked statistical significance with $(p=0.7)$ and $(p=0.8)$, indicating an
essentially negligible relationship between fallback and hit success. Guidance
also showed a weak negative correlation with hit success, with ($\rho=-0.03$),
the finding was as well not statistically significant $(p=0.3)$, suggesting
that guidance does not significantly influence hit success. Between Correction
and hit success a statistically significant negative correlation
($\rho=-0.07$, $p=0.01$) was found. However, the correlation is below $0.1$
thus can not even be considered as weak.

Overall, the analyses show that while an increase in speed and a decrease in
exploration positively influences hit success fallback, guidance and
correction do not show significant or notable correlations with hit success.

##  V Discussion

H1: Enhanced Learning through Multiple Modalities. Our results affirm that
providing multiple feedback modalities significantly enhances the learning
process, as evidenced by Group 2’s higher performance. This group consistently
hitting targets confirms that multiple modalities not only expedite initial
learning success but also fortify sustained learning progress.

H2: User Preferences and Modality Utilization. Clearly users do not engage
with all modalities equally, displaying marked preferences. These preferences
are mirrored in the higher overall satisfaction observed in Group 2,
suggesting that the success facilitated by preferred modalities positively
influences user perceptions. Participants expressed their preferences through
usage frequency, rankings, reasons for their choices, and SUS scores of the
individual meta-modalities. Notably, Guidance and Speed stand out for their
high usage and positive reception, attributed to their perceived efficacy
impact on the learning process and intuitiveness. Both also stand out in the
pairwise comparison regarding the SUS score compared to Exploration.
Conversely, Exploration and Demonstration were less favored, with the former
criticized for its ambiguous effects and the latter for its underuse, possibly
due to perceived effort or lack of necessity.

H3: Differential Benefits of Modalities. Our findings corroborate the
hypothesis that certain modalities are perceived as more beneficial than
others in enhancing the learning process. Guidance, with its high engagement
and favorable ranking, was seen as particularly effective, emphasizing its
role in achieving learning goals efficiently. Although Correction and Fallback
were recognized for their utility, their engagement levels and mixed rankings
suggest a nuanced perception of their effectiveness in learning enhancement.

H4: Enhanced Learning through Individual Modalities Exploration and Speed
positively impact hit success, with exploration reduced upon successful hits
to maintain performance and speed ensuring the ball’s reach. Guidance emerged
as the most valued by users, while Exploration was seen as least useful.
However, if its function was understood, it is enhancing learning success.
Movement attributes, perceived positively and frequently interacted with, and
thus seem to be a natural teaching method for participants even though they
are underestimated in the field.

In summary, our findings advocate for a broader adoption and integration of
multiple modalities within the domain of interactive robot learning.
Particular focus should be placed on clarifying exploration, since it
positively correlates with learning progress when used correctly, on
leveraging intuitive movement properties, and on emphasizing guidance, which
is a favorite among participants.

## References

  * [1] T. Kaufmann, P. Weng, V. Bengs, and E. Hüllermeier, “A survey of reinforcement learning from human feedback,” arXiv preprint arXiv:2312.14925, 2023.
  * [2] L. Hindemith, O. Bruns, A. M. Noller, N. Hemion, S. Schneider, and A.-L. Vollmer, “Interactive robot task learning: Human teaching proficiency with different feedback approaches,” IEEE Trans. Cognit. Dev. Syst., 2022.
  * [3] A. L. Thomaz, C. Breazeal, et al., “Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance,” in Aaai, vol. 6, pp. 1000–1005, Boston, MA, 2006.
  * [4] H. Liu, A. Chen, Y. Zhu, A. Swaminathan, A. Kolobov, and C.-A. Cheng, “Interactive robot learning from verbal correction,” arXiv preprint arXiv:2310.17555, 2023.
  * [5] M. Li, A. Canberk, D. P. Losey, and D. Sadigh, “Learning human objectives from sequences of physical corrections,” in 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 2877–2883, IEEE, 2021.
  * [6] A. Bajcsy, D. P. Losey, M. K. O’Malley, and A. D. Dragan, “Learning from physical human corrections, one feature at a time,” in Proc. of 2018 ACM/IEEE Int. Conf. on Human-Robot Interaction, pp. 141–149, 2018.
  * [7] C. Celemin and J. Kober, “Knowledge-and ambiguity-aware robot learning from corrective and evaluative feedback,” Neural Comput. Appl., pp. 1–19, 2023.
  * [8] H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard, “Recent advances in robot learning from demonstration,” Annu. Rev. Control Rob. Auton. Syst., vol. 3, pp. 297–330, 2020.
  * [9] C. Arzate Cruz and T. Igarashi, “A survey on interactive reinforcement learning: Design principles and open challenges,” in Proceedings of the 2020 ACM Designing Interactive Syst. Conf., pp. 1195–1209, 2020.
  * [10] C. Isbell, C. R. Shelton, M. Kearns, S. Singh, and P. Stone, “A social reinforcement learning agent,” in Proceedings of the fifth international conference on Autonomous agents, pp. 377–384, 2001.
  * [11] W. B. Knox and P. Stone, “Tamer: Training an agent manually via evaluative reinforcement,” in 2008 7th IEEE international conference on development and learning, pp. 292–297, IEEE, 2008.
  * [12] Z. Ding, Y. Chen, A. Z. Ren, S. S. Gu, H. Dong, and C. Jin, “Learning a universal human prior for dexterous manipulation from human preference,” arXiv preprint arXiv:2304.04602, 2023.
  * [13] C. Yu and A. Tapus, “Interactive robot learning for multimodal emotion recognition,” in Social Robotics: 11th International Conference, ICSR 2019, Madrid, Spain, November 26–29, 2019, Proceedings 11, pp. 633–642, Springer, 2019.
  * [14] H. Su, W. Qi, J. Chen, C. Yang, J. Sandoval, and M. A. Laribi, “Recent advancements in multimodal human–robot interaction,” Front. Neurorob., vol. 17, p. 1084000, 2023.
  * [15] H. Chen, M. C. Leu, and Z. Yin, “Real-time multi-modal human–robot collaboration using gestures and speech,” J. Manuf. Sci. Eng., vol. 144, no. 10, p. 101007, 2022.
  * [16] R. Arakawa, S. Kobayashi, Y. Unno, Y. Tsuboi, and S.-i. Maeda, “Dqn-tamer: Human-in-the-loop reinforcement learning with intractable feedback,” arXiv preprint arXiv:1810.11748, 2018.
  * [17] G. Li, R. Gomez, K. Nakamura, and B. He, “Human-centered reinforcement learning: A survey,” IEEE Trans. Hum.-Mach. Syst., vol. 49, no. 4, pp. 337–349, 2019.  Publisher: IEEE.
  * [18] K. inc., “Kinova assistiv.” https://assistive.kinovarobotics.com/product/ jaco-robotic-arm, 2023.
  * [19] O. Robotics, “Ros noetic ninjemys.” http://wiki.ros.org/noetic, 2020.
  * [20] A. Paraschos, C. Daniel, J. R. Peters, and G. Neumann, “Probabilistic movement primitives,” Adv. in neural information processing systems, vol. 26, 2013.
  * [21] J. K. Alexander Fabisch, “Movement primitives.” https://github.com/dfki-ric/movement_primitives, 2020.
  * [22] F. Stulp and O. Sigaud, “Policy improvement methods: Between black-box optimization and episodic reinforcement learning,” 2012.
  * [23] A.-L. Vollmer and N. J. Hemion, “A user study on robot skill learning without a cost function: Optimization of dynamic movement primitives via naive user feedback,” Front. Rob. AI, vol. 5, p. 77, 2018.
  * [24] J. Brooke, “Sus: A quick and dirty usability scale,” Usability Eval. Ind., vol. 189, 11 1995.


# Strong convergence for tensor GUE random matrices

Benoît Collins  Department of Mathematics, Graduate School of Science, Kyoto
University, Kyoto 606-8502, Japan<EMAIL_ADDRESS>and  Wangjun
Yuan  University of Luxembourg, Department of Mathematics, Maison du Nombre 6,
Avenue de la Fonte L-4364 Esch-sur-Alzette. Luxembourg.
<EMAIL_ADDRESS>
###### Abstract.

Haagerup and Thorbjørnsen proved that iid GUEs converge strongly to free
semicircular elements as the dimension grows to infinity. Motivated by
considerations from quantum physics – in particular, understanding nearest
neighbor interactions in quantum spin systems – we consider iid GUE acting on
multipartite state spaces, with a mixing component on some sites and identity
on the remaining sites. We show that under proper assumptions on the dimension
of the sites, strong asymptotic freeness still holds. Our proof relies on an
interpolation technology recently introduced by Bandeira, Boedihardjo and van
Handel.

Keywords. Gaussian unitary ensemble; strong asymptotic freeness; tensor

AMS 2020 Subject Classifications. 15B52, 60B20, 47A80

##  1\. Introduction and statement of main result

###  1.1. Motivating considerations

A typical problem in multi-matrix random matrix theory consists in considering
a sequence (indexed by a dimension) of a $d$-tuple of random matrices,
consider a fixed non-commutative function in $d$ abstract variables and apply
it to this sequence of random matrices, and study its properties in the large
dimension limit. Multiple breakthrough related to the study of such problems
have arisen from the use of free probability theory, after Voiculescu observed
in [Voi1991] in 1991, the asymptotic freeness of independent GUE as the
dimension goes to infinity.

In the case of asymptotic freeness, the convergence of the spectrum is
primarily being considered, however, many other “measurables” are also
considered, such as the operator norm. This has been the motivation of strong
asymptotic freeness with the first major achievement with Haagerup and
Thorbjørnsen [Haagerup2005], and subsequently, many other variants. Among
subsequent variants, let us mention [CMale] where it is proved that
independent unitaries are strongly asymptotically free from constant matrices.
Later, the authors of [Bordenave2019, Bordenave2023, Bordenave2024] proved
strong freeness involving random permutations and also large tensors . The
tensor setup is actually the one in which we are interested, motivated by
quantum mechanics.

Let us outline a motivating example based on the paper [Siddhardh2019],
capturin the typical random matrix theoretical problems of interest to quantum
many body system. We are interested in the spectrum of a matrix

$\displaystyle\sum_{i=1}^{k}X_{i},$ (1.1)

where $X$ is a random matrix (e.g. a GUE) acting on
$M_{n}(\mathbb{C})^{\otimes 2}$ and $X_{i}$ is the version of $X$ acting on
the leg $i,i+1$ of $M_{n}(\mathbb{C})^{\otimes k+1}$. We refer to (2.1) for
definitions.

It was observed by the authors of the above paper and also in
[Charlesworth2021] that epsilon-freeness is the right concept for such models
– in the case where $X_{i}$ emanate from a different i.i.d. copy of $X$ for
each $i$. However, strong convergence for models introduced in
[Charlesworth2021] in the context of epsilon-freeness and of the paper
[Siddhardh2019] remains elusive. On the other hand, and quite unexpectedly,
this turns out to be also an important problem in random geometry, see
[Magee2023].

It turns out that we are not able to handle the problem in such generality at
that point, because techniques that are available with large tensors
(developed by [Bordenave2024]) do not support large rank tensor product. The
only paper supporting in a robust way large rank tensor product is the
fundamental recent paper by [vH2024]. In this paper, they obtain strong
convergence for Gaussian models with completely explicit numerical bounds,
which are, under some assumptions, compatible with large matrix tensor rank.
This is the technology on which we rely. However, we had to restrict our
considerations in this manuscript to the following setup:

(1) Unlike in the original model from physics, the dimensions of the various
systems are not equal.

(2) Each summand in the model has to be independent from the others.

In another vein, it is interesting to compare our manuscript with [Mingo2019]
in which it is proved that a unitary matrix and its partial transpose are
asymptotically free, in the sense that we achieve freeness with less
randomness than initially required by Voiculescu (in [Mingo2019], taking the
transpose – a deterministic operation – yields freeness, whereas for us,
changing a small dimension leg yields strong asymptotic freeness). The common
point between [Mingo2019] and our manuscript is that in both cases there is a
tensor structure and a modification of the model on a subsystem.

###  1.2. Statement of the results and organization

Let $N,m$ be positive integers. We denote $[m]=\\{1,2,\ldots,m\\}$. For any
$J\subseteq[m]$, we denote by $\\{a_{1},\ldots a_{|J|}\\}$ the elements in $J$
with a certain order, and $|J|$ the number of elements in $J$. Let $X$ be a
Gaussian unitary ensemble (GUE) matrix on $\mathbb{M}_{N^{|J|}}(\mathbb{C})$.
Note that
$\mathbb{M}_{N^{|J|}}(\mathbb{C})=\mathbb{M}_{N}(\mathbb{C})^{\otimes|J|}$, we
have the following decomposition of $X_{J}$:

$\displaystyle X=\sum_{i_{1},\ldots,i_{|J|},j_{1},\ldots
j_{|J|}=1}^{N}x_{i_{1}\ldots i_{|J|}j_{1}\ldots
j_{|J|}}E_{i_{1}j_{1}}\otimes\ldots\otimes E_{i_{|J|}j_{|J|}},$

where the family $\\{E_{ij}:1\leq i,j\leq N\\}$ is a basis of
$\mathbb{M}_{N}(\mathbb{C})$. Now we introduce the notation
$X\tilde{\otimes}_{J}I_{N}^{\otimes(m-|J|)}$ as follows:

$\displaystyle
X\tilde{\otimes}_{J}I_{N}^{\otimes(m-|J|)}=\sum_{i_{1},\ldots,i_{|J|},j_{1},\ldots
j_{|J|}=1}^{N}x_{i_{1}\ldots i_{|J|}j_{1}\ldots
j_{|J|}}I_{N}\otimes\ldots\otimes E_{i_{1}j_{1}}\otimes\ldots\otimes
E_{i_{|J|}j_{|J|}}\otimes\ldots\otimes I_{N},$

where the base matrices $E_{i_{1}j_{1}},\ldots E_{i_{|J|}j_{|J|}}$ are in the
position $a_{1},\ldots,a_{|J|}$ respectively. We will introduce the notation
$\tilde{\otimes}_{J}$ with more details in Section 2.1.

For $1\leq i\leq k$, let $J_{i}$ be a subset of $[m]$ such that $|J_{i}|>m/2$.
We define

$\displaystyle\alpha=\min_{1\leq i\leq k}(2|J_{i}|-m).$

Let $X_{J_{i}}$ be a GUE matrix on $\mathbb{M}_{N^{|J_{i}|}}(\mathbb{C})$, and
$B_{i}$ be a deterministic $d\times d$ self-adjoint matrix. Consider the
following model

$\displaystyle
X_{N}=\sum_{i=1}^{k}B_{i}\otimes\left(X_{J_{i}}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right).$
(1.2)

Let $\\{s_{1},\ldots,s_{k}\\}$ be a family of free semicircular elements, and
define

$\displaystyle X_{\mathrm{free}}=\sum_{i=1}^{k}B_{i}\otimes s_{i}.$ (1.3)

Let $\iota$ be a mapping that maps a matrix to the column vector of its
entries. We have the following result on the spectrum of $X_{N}$ and
$X_{\mathrm{free}}$.

######  Theorem 1.

Assume that there exist parameters $\Gamma,\Theta$ that may depend on $N$,
such that

$\displaystyle\left\|\sum_{i=1}^{k}\left(B_{i}\right)^{2}\right\|\leq\Gamma,\quad\sum_{i=1}^{k}\left\|\iota(B_{i})\iota(B_{i})^{*}\right\|\leq\Theta.$

Then for all $\alpha,N,k$, for all $t\geq 0$, we have

$\displaystyle\mathbb{P}\left(\mathrm{sp}(X_{N})\subseteq\mathrm{sp}(X_{\mathrm{free}})+CN^{-\alpha/4}\Gamma^{1/4}\Theta^{1/4}\left(\ln^{3/4}\left(dN^{m}\right)+t\right)[-1,1]\right)\geq
1-e^{-t^{2}},$ (1.4)

and

$\displaystyle\mathbb{P}\left(\left\|X_{N}\right\|>\left\|X_{\mathrm{free}}\right\|+CN^{-\alpha/4}\Gamma^{1/4}\Theta^{1/4}\left(\ln^{3/4}\left(dN^{m}\right)+t\right)\right)\leq
e^{-t^{2}},$ (1.5)

for a universal positive constant $C$.

If we choose $t=\ln^{3/4}(dN^{m})$ in Theorem 1, we have the following
corollary:

######  Corollary 1.

Suppose that the assumptions in Theorem 1 hold. Then for all $\alpha,N,k$, we
have

$\displaystyle\mathbb{P}\left(\mathrm{sp}(X_{N})\subseteq\mathrm{sp}(X_{\mathrm{free}})+CN^{-\alpha/4}\Gamma^{1/4}\Theta^{1/4}\ln^{3/4}\left(dN^{m}\right)[-1,1]\right)\geq
1-e^{-\ln^{3/2}(dN^{m})},$

and

$\displaystyle\mathbb{P}\left(\left\|X_{N}\right\|>\left\|X_{\mathrm{free}}\right\|+CN^{-\alpha/4}\Gamma^{1/4}\Theta^{1/4}\ln^{3/4}\left(dN^{m}\right)\right)\leq
e^{-\ln^{3/2}(dN^{m})},$

for a universal positive constant $C$.

In particular, by Borel-Cantelli’s lemma, we have the next corollary:

######  Corollary 2.

Suppose that the assumptions in Theorem 1 hold. Assume that

$\displaystyle\lim_{N\to\infty}N^{-\alpha}\Gamma\Theta\ln^{3}\left(dN^{m}\right)=0,$

then for any $\epsilon>0$, we have almost surely,

$\displaystyle\mathrm{sp}(X_{N})\subseteq\mathrm{sp}(X_{\mathrm{free}})+(-\epsilon,\epsilon)\quad\mathrm{and}\quad\left\|X_{N}\right\|\leq\left\|X_{\mathrm{free}}\right\|+\epsilon,$
(1.6)

eventually as $N\to\infty$.

We would like to remark that Theorem 1, Corollary 1 and Corollary 2 allow $k$
to grow with $N$. If we fix $k$, we have the following strong asymptotic
freeness.

######  Theorem 2.

Let $k,d$ and $B_{1},\ldots B_{k}$ to be fixed. Assume that

$\displaystyle\lim_{N\to\infty}N^{-\alpha}m^{3}\ln^{3}N=0,$

then, for any non-commutative polynomial $P$, we have

$\displaystyle\lim_{N\to\infty}\left\|P\left(X_{J_{1}}\tilde{\otimes}_{J_{1}}I_{N}^{\otimes(m-|J_{1}|)},\ldots,X_{J_{k}}\tilde{\otimes}_{J_{k}}I_{N}^{\otimes(m-|J_{k}|)}\right)\right\|=\left\|P\left(s_{1},\ldots,s_{k}\right)\right\|$

almost surely, where $\\{s_{1},\ldots,s_{k}\\}$ be a family of free
semicircular elements.

The rest of the paper is organized as follows. We introduce some preliminaries
on tensors, free probability, and matrices in Section 2. Then we present the
proof of Theorem 1 and Theorem 2 in Section 3 and Section 4 respectively.

###  1.3. Acknowledgements

B. C. is supported by JSPS Grant-in-Aid Scientific Research (B) no. 21H00987,
and Challenging Research (Exploratory) no. 20K20882 and 23K17299.

W. Y. gratefully acknowledges the financial support of ERC Consolidator Grant
815703 ”STAMFORD: Statistical Methods for High Dimensional Diffusions”.

##  2\. Preliminaries

###  2.1. Tensor products

For two vector spaces $\mathcal{A}_{1}$ and $\mathcal{A}_{2}$, we define
$\mathcal{A}_{1}\otimes\mathcal{A}_{2}$ as the tensor product of
$\mathcal{A}_{1}$ and $\mathcal{A}_{2}$. Let $m,N\in\mathbb{N}$, we consider
the tensor $\mathcal{H}^{\otimes m}$ of $N$ dimensional vector space
$\mathcal{H}$. For any permutation $\sigma$ on $[m]$, we consider the linear
transformation $\mathcal{A}_{\sigma}:\mathcal{H}^{\otimes
m}\to\mathcal{H}^{\otimes m}$ given by

$\displaystyle\mathcal{A}_{\sigma}\left(x_{1}\otimes\ldots\otimes
x_{m}\right)=x_{\sigma(1)}\otimes\ldots\otimes x_{\sigma(m)},\quad\forall
x_{1},\ldots,x_{m}\in\mathcal{H}.$

Then $\mathcal{A}_{\sigma}$ is a unitary transformation. For any linear
transformation $\mathcal{X}:\mathcal{H}^{\otimes m}\to\mathcal{H}^{\otimes
m}$, we consider the linear transformation
$\mathcal{A}_{\sigma}\mathcal{X}\mathcal{A}_{\sigma^{-1}}$. If $\mathcal{X}$
is of the form
$\mathcal{X}=\mathcal{X}_{1}\otimes\ldots\otimes\mathcal{X}_{m}$, where
$\mathcal{X}_{i}$ are linear transformation on $\mathcal{H}$, then for any
$x_{1},\ldots,x_{m}\in\mathcal{H}$, we have

$\displaystyle\mathcal{A}_{\sigma}\mathcal{X}\mathcal{A}_{\sigma^{-1}}\left(x_{1}\otimes\ldots\otimes
x_{m}\right)=\mathcal{A}_{\sigma}\mathcal{X}\left(x_{\sigma^{-1}(1)}\otimes\ldots\otimes
x_{\sigma^{-1}(m)}\right)$
$\displaystyle=\mathcal{A}_{\sigma}\left(\mathcal{X}_{1}\left(x_{\sigma^{-1}(1)}\right)\otimes\ldots\otimes\mathcal{X}_{m}\left(x_{\sigma^{-1}(m)}\right)\right)=\mathcal{X}_{\sigma(1)}(x_{1})\otimes\ldots\otimes\mathcal{X}_{\sigma(m)}(x_{m}),$

which implies that
$\mathcal{A}_{\sigma}\mathcal{X}\mathcal{A}_{\sigma^{-1}}=\mathcal{X}_{\sigma(1)}\otimes\ldots\otimes\mathcal{X}_{\sigma(m)}$.
Therefore, we can define $\mathcal{L}_{\sigma}$ the linear transformation on
the space of linear transformation on $\mathcal{H}^{\otimes m}$ given by
$\mathcal{L}_{\sigma}(\mathcal{X})=\mathcal{A}_{\sigma}\mathcal{X}\mathcal{A}_{\sigma^{-1}}$.
We would like to remark that, if a basis of $\mathcal{H}$ is fixed, then any
vector in $\mathcal{H}$ corresponds to a vector in $\mathbb{C}^{N}$, and a
linear transformation on $\mathcal{H}^{\otimes m}$ is equivalent to a matrix
in $\mathbb{M}_{N}(\mathbb{C})^{\otimes m}=\mathbb{M}_{N^{m}}(\mathbb{C})$.
Hence, we may abuse the notation and view $\mathcal{L}_{\sigma}$ as a linear
transformation on matrix space $\mathbb{M}_{N}(\mathbb{C})^{\otimes m}$.

Let $J=\\{a_{1},\ldots,a_{|J|}\\}$ be an (ordered) subset of $[m]$, and let
$X\in\mathbb{M}_{N^{|J|}}(\mathbb{C})=\mathbb{M}_{N}(\mathbb{C})^{\otimes|J|}$
and
$Y\in\mathbb{M}_{N^{m-|J|}}(\mathbb{C})=\mathbb{M}_{N}(\mathbb{C})^{\otimes(m-|J|)}$.
Then $X\otimes Y$ is an element belonging to
$\mathbb{M}_{N}(\mathbb{C})^{\otimes m}$. We choose the permutation $\sigma$
on $[m]$ such that $\sigma(l)=a_{l}$ for $1\leq l\leq|J|$. Then we define

$\displaystyle X\tilde{\otimes}_{J}Y=\mathcal{L}_{\sigma}(X\otimes Y).$ (2.1)

######  Remark 1.

If $J=\\{1,\ldots,|J|\\}$ and $\sigma=\mathrm{Id}_{[m]}$, then the operator
$\tilde{\otimes}_{J}$ becomes the usual operator $\otimes$.

######  Remark 2.

The linear transformation $\mathcal{L}_{\sigma}$ depends on $\sigma$. However,
in our paper, we will use this notation for
$Y=\mathrm{Id}_{N}^{\otimes(m-|J|)}$. Hence, $L_{\sigma}$ only depends on $J$,
but not the value of $\sigma(|J|+1),\ldots,\sigma(m)$.

We have the following property:

######  Lemma 1.

For
$X,Y\in\mathbb{M}_{N^{|J|}}(\mathbb{C}),Z,W\in\mathbb{M}_{N^{m-|J|}}(\mathbb{C})$,
$a,b,c,d\in\mathbb{C}$, we have the following statements.

  1. (1)

$(aX+bY)\tilde{\otimes}_{J}(cZ+dW)=acX\tilde{\otimes}_{J}Z+adX\tilde{\otimes}_{J}W+bcY\tilde{\otimes}_{J}Z+bdY\tilde{\otimes}_{J}W$.

  2. (2)

$(X\tilde{\otimes}_{J}Z)(Y\tilde{\otimes}_{J}W)=(XY)\tilde{\otimes}_{J}(ZW)$.

###### Proof.

The proof is a direct verification using the definition above, noting that the
definition does not depend on the representation. ∎

Next, we turn to the relationship between $\tilde{\otimes}_{J},\otimes$ and
$\iota$. We have the following lemma.

######  Lemma 2.

Let
$X\in\mathbb{M}_{N^{|J|}}(\mathbb{C}),Y\in\mathbb{M}_{N^{m-|J|}}(\mathbb{C})$,
then there exists a permutation matrix $U\in\mathbb{M}_{N^{m}}(\mathbb{C})$
that only depends on $J$, such that

$\displaystyle\iota\left(X\tilde{\otimes}_{J}Y\right)=U\left(\iota(X)\otimes\iota(Y)\right).$
(2.2)

###### Proof.

(1). Using the fact that $A\otimes B$ and $B\otimes A$ have the same sets of
entries for any $A$ and $B$, one can check with the definition that
$X\tilde{\otimes}_{J}Y$ and $X\otimes Y$ have the same sets of entries, and
the positions of the entries depends only on $J$. Note that the set of entries
of $\iota(X\otimes Y)$ is the same as the set of components of the vector
$\iota(X)\otimes\iota(Y)$. Thus, there exists a permutation matrix $U$ that
only depends on $J$, such that (2.2) holds. ∎

###  2.2. Free probability

In this subsection, we recall some definitions and results in free
probability. We refer interested readers to [MingoSpeicher2017, NicaSpeicher]
for more details on free probability.

######  Definition 1.

Let $\mathcal{A}$ be a unital associative algebra over $\mathbb{C}$ and
$\tau:\mathcal{A}\to\mathbb{C}$ be a linear functional satisfying $\tau(1)=1$.
Then, the pair $(\mathcal{A},\tau)$ is called a _non-commutative probability
space_.

######  Definition 2.

Let $(\mathcal{A},\tau)$ be a non-commutative probability space.

  1. (1)

If $\tau(ab)=\tau(ba)$ for all $a,b\in\mathcal{A}$, then $(\mathcal{A},\tau)$
a _tracial non-commutative probability space_.

  2. (2)

If $\mathcal{A}$ is a $*$-algebra and $\tau(a^{*}a)\geq 0$ for all
$a\in\mathcal{A}$, then we call $\tau$ a _state_ and $(\mathcal{A},\tau)$ a
_$*$ -probability space_. In addition, if $\tau(a^{*}a)=0$ implies that $a=0$,
then we say $\tau$ is _faithful_.

######  Definition 3.

Let $(\mathcal{A},\tau)$ be a non-commutative probability space
($*$-probability space, resp.) and $I$ be an index set. Then the sub-algebras
$\\{\mathcal{A}_{i}:i\in I\\}$ of $\mathcal{A}$ are _free_ in
$(\mathcal{A},\tau)$ if

$\displaystyle\tau\left(\left(P_{1}(a_{i_{1}})-\tau\left(P_{1}(a_{i_{1}})\right)\right)\ldots\left(P_{k}(a_{i_{k}})-\tau\left(P_{k}(a_{i_{k}})\right)\right)\right)=0,$

for all

  * •

$k\in\mathbb{N}$,

  * •

$i_{1},\ldots,i_{k}\in I$ satisfying
$i_{1}\not=i_{2}\not=i_{3}\not=\ldots\not=i_{k-1}\not=i_{k}$,

  * •

polynomials (*-polynomials, resp.) $P_{1},\ldots,P_{k}$,

  * •

$a_{j}\in\mathcal{A}_{i_{j}}$ for all $1\leq j\leq k$.

A family of elements $\\{a_{i}\\}_{i\in I}$ in $\mathcal{A}$ are called _free_
if the unital sub-algebras generated by $\\{1,a_{i}\\}$
($\\{1,a_{i},a_{i}^{*}\\}$, resp.) are free.

######  Definition 4.

Let $(\mathcal{A}_{n},\tau_{n})$ be a sequence of non-commutative probability
space (resp. a $*$-probability space). A sequence
$a_{n,1},\ldots,a_{n,k}\in\mathcal{A}_{n}$ is _asymptotically free_ if

$\displaystyle\tau_{n}\left(\left(P_{1}(a_{n,i_{1}})-\tau\left(P_{1}(a_{n,i_{1}})\right)\right)\ldots\left(P_{m}(a_{n,i_{m}})-\tau\left(P_{m}(a_{n,i_{m}})\right)\right)\right)\to
0,$

as $n\to\infty$, for any $m\in\mathbb{N}$, $1\leq i_{1},\ldots,i_{m}\leq k$
satisfying $i_{1}\not=i_{2}\not=i_{3}\not=\ldots\not=i_{m-1}\not=i_{m}$, and
any polynomials (*-polynomials, resp.) $P_{1},\ldots,P_{k}$.

######  Definition 5.

Let $(\mathcal{A},\tau)$ be a *-probability space. A self-adjoint element
$s\in\mathcal{A}$ is called a semicircular element if it has moments

$\displaystyle\tau(s^{k})=\begin{cases}0,&k=2m-1\mathrm{\ for\ some\
}m\in\mathbb{N},\\\ \frac{1}{m+1}\binom{2m}{m},&k=2m\mathrm{\ for\ some\
}m\in\mathbb{N}.\end{cases}$

######  Remark 3.

The moments of a semicircular element $s$ are

$\displaystyle\tau\left(s^{k}\right)=\dfrac{1}{2\pi}\int_{-2}^{2}t^{k}\sqrt{4-t^{2}}dt,\quad\forall
k\in\mathbb{N}.$

Next, we introduce the weak convergence in the framework of free probability.

######  Definition 6.

Let $(\mathcal{A}_{n},\tau_{n})$ for every $n\in\mathbb{N}$ as well as
$(\mathcal{A},\tau)$ be non-commutative probability spaces. Let
$\\{b_{n}:n\in\mathbb{N}\\}$ be a sequence of elements with
$b_{n,1},\ldots,b_{n,k}\in\mathcal{A}_{n}$ for every $n\in N$ and let
$b_{1},\ldots,b_{k}\in\mathcal{A}$. We say that the family
$\\{b_{n,1},\ldots,b_{n,k}:n\in\mathbb{N}\\}$ converges in distribution to
$\\{b_{1},\ldots,b_{k}\\}$ if it holds that

$\displaystyle\lim_{n\to\infty}\tau_{n}\left(b_{n,i_{1}}^{a_{1}}\ldots
b_{n,i_{m}}^{a_{m}}\right)=\tau\left(b_{i_{1}}^{a_{1}}\ldots
b_{i_{m}}^{a_{m}}\right),\quad\forall m\in\mathbb{N},\forall 1\leq
i_{1},\ldots,i_{m}\leq k,\forall a_{1},\ldots,a_{m}\in\\{1,*\\}.$

Now we turn to the so-called $C^{*}$-probability space.

######  Definition 7.

Let $(\mathcal{A},\tau)$ be a $*$-probability space endowed with a norm
$\|\cdot\|:\mathcal{A}\to[0,+\infty)$ which makes it a complete normed vector
space and which satisfies

$\displaystyle\|ab\|\leq\|a\|\|b\|\quad\mathrm{and}\quad\|a^{*}a\|=\|a\|^{2},\forall
a,b\in\mathcal{A}.$

Then $(\mathcal{A},\tau)$ is a _$C^{*}$ -probability space_.

In the following, we collect some asymptotic freeness results on Gaussian
matrices from [BBvH2023], which will be used later.

For each $N\in\mathbb{N}$, let $\\{H_{i}^{N}:1\leq i\leq m\\}$ be a family of
independent self-adjoint $N$-dimensional random matrices satisfying
$\mathbb{E}[H_{i}^{N}]=0$ and $\mathbb{E}[(H_{i}^{N})^{2}]=I_{N}$ for all
$1\leq i\leq m$, and have the following decomposition

$\displaystyle H_{i}^{N}=\sum_{j=1}^{N_{i}}g_{i,j}^{N}B_{ij}^{N},$

for some $N_{i}\in\mathbb{N}$, where $\\{g_{i,j}^{N}:1\leq i\leq m,1\leq j\leq
N_{i}\\}$ is a family of i.i.d. standard real Gaussian random variables, and
$\\{B_{ij}^{N}:1\leq i\leq m,1\leq j\leq N_{i}\\}$ is a family of
$N$-dimensional self-adjoint complex matrices. Let $\\{A_{i}:1\leq i\leq m\\}$
be a family of $d$-dimensional self-adjoint matrices. Define

$\displaystyle\Theta_{N}=\sum_{i=1}^{m}A_{i}\otimes
H_{i}^{N}=\sum_{i=1}^{m}A_{i}\otimes\sum_{j=1}^{N_{i}}g_{i,j}^{N}B_{ij}^{N},$
$\displaystyle\Theta_{N,\mathrm{free}}=\sum_{i=1}^{m}A_{i}\otimes\sum_{j=1}^{N_{i}}B_{ij}^{N}\otimes
s_{i,j},$ $\displaystyle\Theta_{\mathrm{free}}=\sum_{i=1}^{m}A_{i}\otimes
s_{i},$

where $\\{s_{1},\ldots,s_{k}\\}$ and $\\{s_{i,j}:1\leq i\leq k,1\leq j\leq
N_{i}\\}$ are families of free semicircular elements.

The following lemma is the linearization argument of [Haagerup2005] and is a
direct consequence of [BBvH2023]*Theorem 7.7.

######  Lemma 3.

([BBvH2023]*Theorem 7.7) Assume that the following holds

$\displaystyle\mathrm{sp}\left(\Theta_{N}\right)\subseteq\mathrm{sp}\left(\Theta_{\mathrm{free}}\right)+[-\epsilon,\epsilon],\quad
a.s.$

eventually as $N\to\infty$, for any $\epsilon>0$. Then for any non-commutative
polynomial $P$, the following holds:

$\displaystyle\limsup_{N\to\infty}\left\|P\left(H_{1}^{N},\ldots,H_{m}^{N}\right)\right\|\leq\left\|P\left(s_{1},\ldots,s_{m}\right)\right\|,\quad
a.s.$

The following lemma is [BBvH2023]*Lemma 7.9.

######  Lemma 4.

([BBvH2023]*Lemma 7.9) For every $N\in\mathbb{N}$, we have

$\displaystyle(\mathrm{tr}\otimes\tau)\left(H_{j_{1}}^{N,\mathrm{free}}\ldots
H_{j_{q}}^{N,\mathrm{free}}\right)=\tau\left(s_{j_{1}}\ldots
s_{j_{q}}\right),\quad\forall q\in\mathbb{N},\ \forall 1\leq
j_{1},\ldots,j_{q}\leq m,$

where $H_{i}^{N,\mathrm{free}}=\sum_{j=1}^{N_{i}}B_{ij}^{N}\otimes s_{i,j}$.
In particular, it holds that

$\displaystyle\mathrm{sp}\left(\Theta_{N,\mathrm{free}}\right)=\mathrm{sp}\left(\Theta_{\mathrm{free}}\right).$

The next lemma is a combination of part a of [BBvH2023]*Theorem 2.10 and
[BBvH2023]*Lemma 7.11, which yields the weak asymptotic freeness.

######  Lemma 5.

([BBvH2023]*Theorem 2.10, [BBvH2023]*Lemma 7.11) Let the control parameter $v$
be defined in (2.4). If for all $1\leq i\leq m$,
$v(H_{i}^{N})=o_{N}(\ln^{-3/2}N)$, then for any non-commutative polynomial
$P$, it holds almost surely that

$\displaystyle\lim_{N\to\infty}\mathrm{tr}\left(P\left(H_{1}^{N},\ldots,H_{m}^{N}\right)\right)=\tau\left(P\left(s_{1},\ldots,s_{m}\right)\right).$

###  2.3. Estimates on spectrum and operator norm

For $N,k\in\mathbb{N}$, let $A_{0},\ldots,A_{k}\in\mathbb{M}_{N}(\mathbb{C})$
be self-adjoint matrices, let $\\{g_{1},\ldots,g_{k}\\}$ be a family of i.i.d.
real standard Gaussian variables, and let $\\{s_{1},\ldots,s_{k}\\}$ be a
family of free semicircular elements. We define

$\displaystyle X_{N}=A_{0}+\sum_{i=1}^{k}g_{i}A_{i},\quad\quad
X_{\mathrm{free}}=A_{0}\otimes{\bf 1}+\sum_{i=1}^{k}A_{i}\otimes s_{i}.$

We introduce the following control parameters:

$\displaystyle\sigma(X_{N})^{2}=\left\|\sum_{i=1}^{k}\left(A_{i}\right)^{2}\right\|$
(2.3)

and

$\displaystyle
v(X_{N})^{2}=\left\|\mathrm{Cov}(\iota(X_{N}))\right\|=\left\|\sum_{i=1}^{k}\iota(A_{i})\iota(A_{i})^{*}\right\|.$
(2.4)

We define

$\displaystyle u(X_{N})^{2}=\sigma(X_{N})v(X_{N}).$ (2.5)

The following lemma is a direct consequence of [BBvH2023]*Theorem 2.1,
Corollary 2.2.

######  Lemma 6.

([BBvH2023]*Theorem 2.1, Corollary 2.2) For all $t\geq 0$, we have

$\displaystyle\mathbb{P}\left(\mathrm{sp}(X_{N})\subseteq\mathrm{sp}(X_{\mathrm{free}})+Cu(X_{N})\left(\ln^{3/4}N+t\right)[-1,1]\right)\geq
1-e^{-t^{2}},$

and

$\displaystyle\mathbb{P}\left(\left\|X_{N}\right\|>\left\|X_{\mathrm{free}}\right\|+Cu(X_{N})\left(\ln^{3/4}N+t\right)\right)\leq
e^{-t^{2}},$

for a universal positive constant $C$.

###  2.4. Matrix inequalities

The following lemma is the matrix version of Cauchy-Schwarz inequality.

######  Lemma 7.

For any $n\in\mathbb{N}$, any family $\\{E_{i},F_{i}:1\leq i\leq n\\}$ of
$N\times N$ matrices, we have

$\displaystyle\left\|\sum_{i=1}^{n}E_{i}F_{i}\right\|^{2}\leq\left\|\sum_{i=1}^{n}E_{i}E_{i}^{*}\right\|\left\|\sum_{i=1}^{n}F_{i}^{*}F_{i}\right\|.$

###### Proof.

By Cauchy-Schwarz inequality, for any $x\in\mathbb{C}^{N}$ with $\|x\|=1$, any
family $\\{E_{i},F_{i}:1\leq i\leq n\\}$, we obtain

$\displaystyle
x^{*}\left(\sum_{i=1}^{n}E_{i}F_{i}\right)y=\sum_{i=1}^{n}x^{*}E_{i}F_{i}y\leq\left(\sum_{i=1}^{n}x^{*}E_{i}E_{i}^{*}x\right)^{1/2}\left(\sum_{i=1}^{n}y^{*}F_{i}^{*}F_{i}y\right)^{1/2}.$

Hence,

$\displaystyle\left\|\sum_{i=1}^{n}E_{i}F_{i}\right\|=\sup_{x,y\in\mathbb{C}^{N},\|x\|=\|y\|=1}x^{*}\left(\sum_{i=1}^{n}E_{i}F_{i}\right)y$
$\displaystyle\leq\sup_{x,y\in\mathbb{C}^{N},\|x\|=\|y\|=1}\left(\sum_{i=1}^{n}x^{*}E_{i}E_{i}^{*}x\right)^{1/2}\left(\sum_{i=1}^{n}y^{*}F_{i}^{*}F_{i}y\right)^{1/2}=\left\|\sum_{i=1}^{n}E_{i}E_{i}^{*}\right\|^{1/2}\left\|\sum_{i=1}^{n}F_{i}^{*}F_{i}\right\|^{1/2}.$

∎

##  3\. Proof of Theorem 1

We divide the proof into three steps.

Step 1. In this step, we introduce some notation. Let $\\{e_{i}^{(N)}:1\leq
i\leq N\\}$ be the canonical basis of $\mathbb{C}^{N}$. For $1\leq i\leq N$,
let

$\displaystyle E_{ii}^{(N)}=e_{i}^{(N)}\left(e_{i}^{(N)}\right)^{*},$

and for $1\leq i<j\leq N$ let

$\displaystyle
E_{ij}^{(N)}=\dfrac{1}{\sqrt{2}}\left(e_{i}^{(N)}\left(e_{j}^{(N)}\right)^{*}+e_{j}^{(N)}\left(e_{i}^{(N)}\right)^{*}\right),$
$\displaystyle\tilde{E}_{ij}^{(N)}=\dfrac{\sqrt{-1}}{\sqrt{2}}\left(e_{i}^{(N)}\left(e_{j}^{(N)}\right)^{*}-e_{j}^{(N)}\left(e_{i}^{(N)}\right)^{*}\right).$

Recall that for each $1\leq i\leq k$, $X_{J_{i}}$ is a GUE matrix of dimension
$N^{|J_{i}|}$. We have the decomposition

$\displaystyle
X_{J_{i}}=N^{-|J_{i}|/2}\sum_{j=1}^{N^{2|J_{i}|}}g_{i,j}A_{i,j},$

where $\\{A_{i,j}:1\leq j\leq N^{2|J_{i}|}\\}$ is the set

$\displaystyle\\{E_{ll}^{(N^{|J_{i}|})},E_{rs}^{(N^{|J_{i}|})},\tilde{E}_{rs}^{(N^{|J_{i}|})}:1\leq
l\leq N^{|J_{i}|},1\leq r<s\leq N^{|J_{i}|}\\},$

and $\\{g_{i,j}:1\leq i\leq k,1\leq j\leq N^{2|J_{i}|}\\}$ is a family of
i.i.d. real standard Gaussian variables. Then the matrix
$X_{J_{i}}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}$ has the following
representation

$\displaystyle
X_{J_{i}}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}=N^{-|J_{i}|/2}\sum_{j=1}^{N^{2|J_{i}|}}g_{i,j}A_{i,j}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}.$

Thus, by (1.2), we can write

$\displaystyle X_{N}$
$\displaystyle=\sum_{i=1}^{k}B_{i}\otimes\left(\sum_{j=1}^{N^{2|J_{i}|}}N^{-|J_{i}|/2}g_{i,j}A_{i,j}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right)$
$\displaystyle=\sum_{i=1}^{k}N^{-|J_{i}|/2}\sum_{j=1}^{N^{2|J_{i}|}}g_{i,j}B_{i}\otimes\left(A_{i,j}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right).$

Step 2. Let $X_{N,\mathrm{free}}$ be the object obtained from $X_{N}$ by
replacing the i.i.d. standard real Gaussian random variables by free
semicircular elements. That is,

$\displaystyle
X_{N,\mathrm{free}}=\sum_{i=1}^{k}N^{-|J_{i}|/2}\sum_{j=1}^{N^{2|J_{i}|}}B_{i}\otimes\left(A_{i,j}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right)\otimes
s_{i,j},$

where $\\{s_{i,j}:1\leq i\leq k,1\leq j\leq N^{2|J_{i}|}\\}$ is a family of
free semicircular elements. In this step, we will show the following lemma.

######  Lemma 8.

Under assumptions in Theorem 1, we have then for all $t\geq 0$, we have

$\displaystyle\mathbb{P}\left(\mathrm{sp}(X_{N})\subseteq\mathrm{sp}(X_{N,\mathrm{free}})+CN^{-\alpha/4}\Gamma^{1/4}\Theta^{1/4}\left(\ln^{3/4}\left(dN^{m}\right)+t\right)[-1,1]\right)\geq
1-e^{-t^{2}},$

and

$\displaystyle\mathbb{P}\left(\left\|X_{N}\right\|>\left\|X_{N,\mathrm{free}}\right\|+CN^{-\alpha/4}\Gamma^{1/4}\Theta^{1/4}\left(\ln^{3/4}\left(dN^{m}\right)+t\right)\right)\leq
e^{-t^{2}},$

for a universal positive constant $C$.

###### Proof.

By the definition (2.3), we obtain

$\displaystyle\sigma(X_{N})^{2}$
$\displaystyle=\left\|\sum_{i=1}^{k}N^{-|J_{i}|}\sum_{j=1}^{N^{2|J_{i}|}}\left(B_{i}\otimes\left(A_{i,j}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right)\right)^{2}\right\|$
$\displaystyle=\left\|\sum_{i=1}^{k}N^{-|J_{i}|}\left(B_{i}\right)^{2}\otimes\sum_{j=1}^{N^{2|J_{i}|}}\left(A_{i,j}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right)^{2}\right\|$
$\displaystyle=\left\|\sum_{i=1}^{k}N^{-|J_{i}|}\left(B_{i}\right)^{2}\otimes\left(\sum_{j=1}^{N^{2|J_{i}|}}\left(A_{i,j}\right)^{2}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right)\right\|,$
(3.1)

where we use Lemma 1 in the third line. For $1\leq i\leq k$, we have

$\displaystyle\left(E_{ll}^{(N^{|J_{i}|})}\right)^{2}=E_{ll}^{(N^{|J_{i}|})},\forall
1\leq l\leq N^{|J_{i}|},$

and

$\displaystyle\left(E_{rs}^{(N^{|J_{i}|})}\right)^{2}=\left(\tilde{E}_{rs}^{(N^{|J_{i}|})}\right)^{2}=\dfrac{1}{2}\left(E_{rr}^{(N^{|J_{i}|})}+E_{ss}^{(N^{|J_{i}|})}\right),\quad\forall
1\leq r<s\leq N^{|J_{i}|}.$

Hence, by Lemma 1, we obtain

$\displaystyle\sum_{j=1}^{N^{2|J_{i}|}}\left(A_{i,j}\right)^{2}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}=$
$\displaystyle\left(\sum_{j=1}^{N^{2|J_{i}|}}\left(A_{i,j}\right)^{2}\right)\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}$
$\displaystyle=$ $\displaystyle
N^{|J_{i}|}I_{N^{|J_{i}|}}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}=N^{|J_{i}|}I_{N^{m}}.$
(3.2)

Substituting (3) to (3), we get

$\displaystyle\sigma(X_{N})^{2}=\left\|\sum_{i=1}^{k}\left(B_{i}\right)^{2}\otimes
I_{N^{m}}\right\|=\left\|\sum_{i=1}^{k}\left(B_{i}\right)^{2}\right\|\leq\Gamma.$
(3.3)

By the definition (2.4), we have

$\displaystyle
v(X_{N})^{2}=\left\|\sum_{i=1}^{k}N^{-|J_{i}|}\sum_{j=1}^{N^{2|J_{i}|}}\iota\left(B_{i}\otimes\left(A_{i,j}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right)\right)\iota\left(B_{i}\otimes\left(A_{i,j}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right)\right)^{*}\right\|$
$\displaystyle=\left\|\sum_{i=1}^{k}N^{-|J_{i}|}\sum_{j=1}^{N^{2|J_{i}|}}\left(\iota\left(B_{i}\right)\otimes\iota\left(A_{i,j}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right)\right)\left(\iota\left(B_{i}\right)\otimes\iota\left(A_{i,j}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right)\right)^{*}\right\|$
$\displaystyle=\left\|\sum_{i=1}^{k}N^{-|J_{i}|}\left(\iota(B_{i})\iota(B_{i})^{*}\right)\otimes\sum_{j=1}^{N^{2|J_{i}|}}\iota\left(A_{i,j}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right)\iota\left(A_{i,j}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right)^{*}\right\|,$
(3.4)

where we use Lemma 2 in the second line with the permutation matrix being
absorbed by the operator norm.

Note that

$\displaystyle\iota\left(E_{ll}^{(N^{|J_{i}|})}\right)\iota\left(E_{ll}^{(N^{|J_{i}|})}\right)^{*}=E_{l^{\prime}l^{\prime}}^{(N^{2|J_{i}|})},$

and

$\displaystyle\iota\left(E_{rs}^{(N^{|J_{i}|})}\right)\iota\left(E_{rs}^{(N^{|J_{i}|})}\right)^{*}+\iota\left(\tilde{E}_{rs}^{(N^{|J_{i}|})}\right)\iota\left(\tilde{E}_{rs}^{(N^{|J_{i}|})}\right)^{*}=E_{r^{\prime}r^{\prime}}^{(N^{2|J_{i}|})}+E_{s^{\prime}s^{\prime}}^{(N^{2|J_{i}|})},$

where $l^{\prime}$ is the position of the 1 in the vector
$\iota\left(E_{ll}^{(N^{|J_{i}|})}\right)$, and $r^{\prime},s^{\prime}$ are
the positions of the 1 in the vectors
$\iota\left(e_{r}^{(N^{|J_{i}|})}\left(e_{s}^{(N^{|J_{i}|})}\right)^{*}\right)$
and
$\iota\left(e_{s}^{(N^{|J_{i}|})}\left(e_{r}^{(N^{|J_{i}|})}\right)^{*}\right)$.
Hence, we have

$\displaystyle\sum_{j=1}^{N^{2|J_{i}|}}\iota\left(A_{i,j}\right)\iota\left(A_{i,j}\right)^{*}=I_{N^{2|J_{i}|}}.$
(3.5)

Thus, by Lemma 2, there exists a permutation matrix $U_{i}$, such that

$\displaystyle\sum_{j=1}^{N^{2|J_{i}|}}\iota\left(A_{i,j}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right)\iota\left(A_{i,j}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right)^{*}$
$\displaystyle=\sum_{j=1}^{N^{2|J_{i}|}}U_{i}\left(\iota\left(A_{i,j}\right)\otimes\iota\left(I_{N}^{\otimes(m-|J_{i}|)}\right)\right)\left(\iota\left(A_{i,j}\right)\otimes\iota\left(I_{N}^{\otimes(m-|J_{i}|)}\right)\right)^{*}U_{i}^{*}$
$\displaystyle=U_{i}\left(\left(\sum_{j=1}^{N^{2|J_{i}|}}\iota\left(A_{i,j}\right)\iota\left(A_{i,j}\right)^{*}\right)\otimes\left(\iota\left(I_{N}^{\otimes(m-|J_{i}|)}\right)\iota\left(I_{N}^{\otimes(m-|J_{i}|)}\right)^{*}\right)\right)U_{i}^{*}$
$\displaystyle=U_{i}\left(I_{N^{2|J_{i}|}}\otimes\left(\iota\left(I_{N}^{\otimes(m-|J_{i}|)}\right)\iota\left(I_{N}^{\otimes(m-|J_{i}|)}\right)^{*}\right)\right)U_{i}^{*},$
(3.6)

where we substitute (3.5) in the last line.

Besides, applying Lemma 2, we obtain

$\displaystyle\left\|\iota\left(I_{N}^{\otimes(m-|J_{i}|)}\right)\iota\left(I_{N}^{\otimes(m-|J_{i}|)}\right)^{*}\right\|=\left\|\iota\left(I_{N}\right)^{\otimes(m-|J_{i}|)}\left(\iota\left(I_{N}\right)^{\otimes(m-|J_{i}|)}\right)^{*}\right\|$
$\displaystyle=\left\|\left(\iota\left(I_{N}\right)\iota\left(I_{N}\right)^{*}\right)^{\otimes(m-|J_{i}|)}\right\|=\left\|\iota\left(I_{N}\right)\iota\left(I_{N}\right)^{*}\right\|^{m-|J_{i}|}=N^{m-|J_{i}|},$
(3.7)

where the permutation matrix is absorbed by the operator norm. Substituting
(3) and (3) to (3), we obtain

$\displaystyle v(X_{N})^{2}$
$\displaystyle\leq\sum_{i=1}^{k}N^{-|J_{i}|}\left\|\iota(B_{i})\iota(B_{i})^{*}\right\|\left\|\sum_{j=1}^{N^{2|J_{i}|}}\iota\left(A_{i,j}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right)\iota\left(A_{i,j}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right)^{*}\right\|$
$\displaystyle\leq\sum_{i=1}^{k}N^{m-2|J_{i}|}\left\|\iota(B_{i})\iota(B_{i})^{*}\right\|\leq
N^{-\alpha}\Theta.$ (3.8)

where we use triangle inequality in the first line.

By the definition (2.5), we combine (3.3) and (3) to obtain

$\displaystyle u(X_{N})\leq N^{-\alpha/4}\Gamma^{1/4}\Theta^{1/4}.$ (3.9)

Therefore, applying Lemma 6, we have

$\displaystyle\mathbb{P}\left(\mathrm{sp}(X_{N})\subseteq\mathrm{sp}(X_{N,\mathrm{free}})+CN^{-\alpha/4}\Gamma^{1/4}\Theta^{1/4}\left(\ln^{3/4}\left(dN^{m}\right)+t\right)[-1,1]\right)$
$\displaystyle\geq\mathbb{P}\left(\mathrm{sp}(X_{N})\subseteq\mathrm{sp}(X_{N,\mathrm{free}})+Cu(X_{N})\left(\ln^{3/4}\left(dN^{m}\right)+t\right)[-1,1]\right)\geq
1-e^{-t^{2}},$

and

$\displaystyle\mathbb{P}\left(\left\|X_{N}\right\|>\left\|X_{N,\mathrm{free}}\right\|+CN^{-\alpha/4}\Gamma^{1/4}\Theta^{1/4}\left(\ln^{3/4}\left(dN^{m}\right)+t\right)\right)$
$\displaystyle\leq\mathbb{P}\left(\left\|X_{N}\right\|>\left\|X_{N,\mathrm{free}}\right\|+Cu(X_{N})\left(\ln^{3/4}\left(dN^{m}\right)+t\right)\right)\leq
e^{-t^{2}},$

for any $t\geq 0$. ∎

Step 3. In the last step, we conclude the proof via Lemma 4.

Note that for any $1\leq i\leq k$, $X_{J_{i}}$ is a GUE matrix with dimension
$N^{|J_{i}|}$, we have

$\displaystyle\mathbb{E}\left[X_{J_{i}}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right]=0$

and

$\displaystyle\mathbb{E}\left[\left(X_{J_{i}}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right)^{2}\right]=\mathbb{E}\left[\left(X_{J_{i}}\right)^{2}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right]=I_{N^{|J_{i}|}}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}=I_{N^{m}},$

where we use Lemma 1 for the second formula. Then we apply Lemma 4 to obtain

$\displaystyle\mathrm{sp}\left(X_{N,\mathrm{free}}\right)=\mathrm{sp}\left(X_{\mathrm{free}}\right).$

Then, the inequality (1.4) follows from this equality and the conclusion of
Step 2. The inequality (1.5) follows immediately from (1.4).

######  Remark 4.

Lemma 4 implies that $\mathrm{sp}(X_{N,\mathrm{free}})$ does not depends on
$N$. Hence, we obtain

$\displaystyle\mathbb{P}\left(\mathrm{sp}(X_{N})\subseteq\mathrm{sp}(X_{2,\mathrm{free}})+CN^{-\alpha/4}\Gamma^{1/4}\Theta^{1/4}\left(\ln^{3/4}\left(dN^{m}\right)+t\right)[-1,1]\right)\geq
1-e^{-t^{2}},$

and

$\displaystyle\mathbb{P}\left(\left\|X_{N}\right\|>\left\|X_{2,\mathrm{free}}\right\|+CN^{-\alpha/4}\Gamma^{1/4}\Theta^{1/4}\left(\ln^{3/4}\left(dN^{m}\right)+t\right)\right)\leq
e^{-t^{2}}.$

Comparing with $X_{\mathrm{free}}$, the quantity $X_{2,\mathrm{free}}$ may be
better in some cases as it reveals the relationship among the free
semicircular family in $X_{\mathrm{free}}$.

##  4\. Proof of Theorem 2

We use a similar idea as the proof of part b of [BBvH2023]*Theorem 2.10. We
sketch the proof below.

For any fixed $k,d$ and $d\times d$ self-adjoint deterministic matrices
$B_{1},\ldots,B_{k}$, we have $\Gamma=O_{N}(1)$ and $\Theta=O_{N}(1)$. Thus,
$N^{-\alpha/4}\Gamma^{1/4}\Theta^{1/4}\ln^{3/4}(dN^{m})=o_{N}(1)$. Hence, by
Corollary 2, for any $\epsilon>0$, we have

$\displaystyle\mathrm{sp}(X_{N})\subseteq\mathrm{sp}(X_{\mathrm{free}})+(-\epsilon,\epsilon),$

eventually as $N\to\infty$, almost surely. Then we apply Lemma 3 to obtain

$\displaystyle\limsup_{N\to\infty}\left\|P\left(X_{J_{1}}\tilde{\otimes}_{J_{1}}I_{N}^{\otimes(m-|J_{1}|)},\ldots,X_{J_{k}}\tilde{\otimes}_{J_{k}}I_{N}^{\otimes(m-|J_{k}|)}\right)\right\|\leq\left\|P\left(s_{1},\ldots,s_{k}\right)\right\|$
(4.1)

almost surely, for any non-commutative polynomial $P$.

We can use the computation of (3) for the case $k=d=1$ and $B_{1}=1$ to obtain
that for any $1\leq i\leq m$,

$\displaystyle
v\left(X_{J_{i}}\tilde{\otimes}_{J_{i}}I_{N}^{\otimes(m-|J_{i}|)}\right)\leq
N^{-\alpha/2}=o_{N}\left(\ln^{-3/2}\left(dN^{m}\right)\right).$

Hence, applying Lemma 5, we have the following almost surely convergence

$\displaystyle\lim_{N\to\infty}\mathrm{tr}\left(P\left(X_{J_{1}}\tilde{\otimes}_{J_{1}}I_{N}^{\otimes(m-|J_{1}|)},\ldots,X_{J_{k}}\tilde{\otimes}_{J_{k}}I_{N}^{\otimes(m-|J_{k}|)}\right)\right)=\tau\left(P\left(s_{1},\ldots,s_{k}\right)\right)$

for any non-commutative polynomial $P$. Thus, for any even positive integer
$r$, we have almost surely,

$\displaystyle\liminf_{N\to\infty}\left\|P\left(X_{J_{1}}\tilde{\otimes}_{J_{1}}I_{N}^{\otimes(m-|J_{1}|)},\ldots,X_{J_{k}}\tilde{\otimes}_{J_{k}}I_{N}^{\otimes(m-|J_{k}|)}\right)\right\|$
$\displaystyle\geq\liminf_{N\to\infty}\left(\mathrm{tr}\left(\left|P\left(X_{J_{1}}\tilde{\otimes}_{J_{1}}I_{N}^{\otimes(m-|J_{1}|)},\ldots,X_{J_{k}}\tilde{\otimes}_{J_{k}}I_{N}^{\otimes(m-|J_{k}|)}\right)\right|^{r}\right)\right)^{1/r}$
$\displaystyle=\left(\tau\left(\left|P\left(s_{1},\ldots,s_{k}\right)\right|^{r}\right)\right)^{1/r},$

noting that $|P|^{r}$ is a non-commutative polynomial. Hence,

$\displaystyle\liminf_{N\to\infty}\left\|P\left(X_{J_{1}}\tilde{\otimes}_{J_{1}}I_{N}^{\otimes(m-|J_{1}|)},\ldots,X_{J_{k}}\tilde{\otimes}_{J_{k}}I_{N}^{\otimes(m-|J_{k}|)}\right)\right\|$
$\displaystyle\geq\lim_{r\to\infty}\left(\tau\left(\left|P\left(s_{1},\ldots,s_{k}\right)\right|^{r}\right)\right)^{1/r}=\left\|P\left(s_{1},\ldots,s_{k}\right)\right\|.$
(4.2)

The proof is concluded by combining (4.1) and (4).

######  Remark 5.

For $1\leq i\leq k$, we define

$\displaystyle
S_{J_{i}}=2^{-|J_{i}|/2}\left(\sum_{l=1}^{2^{|J_{i}|}}E_{ll}^{(2^{|J_{i}|})}\otimes
s_{i,l}+\sum_{1\leq r<s\leq 2^{|J_{i}|}}\left(E_{rs}^{(2^{|J_{i}|})}\otimes
s_{i,rs}+\tilde{E}_{rs}^{(2^{|J_{i}|})}\otimes
s_{i,rs}^{\prime}\right)\right),$

where $\\{s_{i,l},s_{i,rs},s_{i,rs}^{\prime}:1\leq i\leq k,1\leq l\leq
2^{|J_{i}|},1\leq r<s\leq 2^{|J_{i}|}\\}$ is a family of free semicircular
elements. Then

$\displaystyle
X_{2,\mathrm{free}}=\sum_{i=1}^{k}B_{i}\otimes\left(S_{J_{i}}\tilde{\otimes}_{J_{i}}I_{2}^{\otimes(m-|J_{i}|)}\right).$

By Lemma 4, one can obtain that for any non-commutative polynomial $P$,

$\displaystyle\lim_{N\to\infty}\left\|P\left(X_{J_{1}}\tilde{\otimes}_{J_{1}}I_{N}^{\otimes(m-|J_{1}|)},\ldots,X_{J_{k}}\tilde{\otimes}_{J_{k}}I_{N}^{\otimes(m-|J_{k}|)}\right)\right\|=\left\|P\left(S_{J_{1}},\ldots,S_{J_{k}}\right)\right\|,$

almost surely.

##  5\. Open questions and perspectives

In this paper, we proved that strong asymptotic freeness holds in a case
beyond Haagerup and Thorbjørnsen’s result ([Haagerup2005]), after introducing
interaction – in the sense of quantum systems – with independent systems of
small size. Our results leave a handful of questions open, which we list up
below:

(1) Can one handle more general dimensions (less constraint on the size of
tensors)? Let us point out here that a very recent preprint [vH2024] developed
new highly original moment techniques to compute norm convergence. At first
sight it was not clear though that it could allow us to make straightforward
progress towards understanding our initial matrix model.

(2) Can one handle the case where $X_{i}$ are all the same but acting on
different legs, rather than i.i.d.? This question seems to be the original
question from the point of view of quantum mechanics. An application of
[BBvH2023] allowed us to compare with an operator algebraic object but
although we suspect freeness, we were not able to confirm it.

(3) Can we handle the case where some $X_{i}$, $X_{j}$ are “remote”, i.e.
commute? That would also be used to address satisfactorily the initial problem
in quantum physics.

Finally, it would be interesting to consider the same problem not only for
nearest-neighbor interaction, but for any kind of graph-type interaction.
While this looks challenging, this is the most natural question from the point
of view of epsilon-freeness, and it would allow to treat more models in
quantumm mechanics, such as nearest neighbors interaction not only in one
dimension, but in higher dimensions.

## References