# ANDROMEDA: An FPGA Based RISC-V MPSoC Exploration Framework

Farhad Merchant, Dominik Sisejkovic, Lennart M. Reimann,
Kirthihan Yasotharan, Thomas Grass, Rainer Leupers {farhad.merchant,
sisejkovic, reimannl, yasotharan, grass<EMAIL_ADDRESS>Institute for Communication Technologies and Embedded Systems, RWTH Aachen
University, Germany

###### Abstract

With the growing demands of consumer electronic products, the computational
requirements are increasing exponentially. Due to the applications’
computational needs, the computer architects are trying to pack as many cores
as possible on a single die for accelerated execution of the application
program codes. In a multiprocessor system-on-chip (MPSoC), striking a balance
among the number of cores, memory subsystems, and network-on-chip parameters
is essential to attain the desired performance. In this paper, we present
_ANDROMEDA_ , a RISC-V based framework that allows us to explore the different
configurations of an MPSoC and observe the performance penalties and gains. We
emulate the various configurations of MPSoC on the Synopsys HAPS-80D Dual FPGA
platform. Using STREAM, matrix multiply, and N-body simulations as benchmarks,
we demonstrate our framework’s efficacy in quickly identifying the right
parameters for efficient execution of these benchmarks.

###### Index Terms:

design space exploration, multiprocessor system-on-chip, RISC-V, performance
tuning

##  I Introduction

Multiprocessor system-on-chips (MPSoCs) are the common computing substrate for
application domains such as automotive, and internet-of-things (IoT) [1][2].
Recently, MPSoCs are heavily involved in service-oriented architectures due to
their capabilities to offer an order of speed-up over the state-of-the-art
[3]. The applications executed on the MPSoCs in these domains are compute and
communication-intensive, requiring complex platforms with deep memory
hierarchy and network infrastructure for reduced application run-time at
constrained energy and area footprints [4]. A right balance between system
parameters is required to attain the desired performance.

Exploring system-level parameters for run-time performance improvement is an
extensively studied topic in the literature [5]. However, most of the
literature solutions are incomplete and sometimes rely on a complex set of
parameters that can be simplified. Also, for many solutions, the rapid field-
programmable gate array (FPGA) prototyping is intricate due to complexities
involved in the intermediate tools, since the tools have non-standardized
tool-interfaces.

Recently, an open instruction-set architecture called RISC-V has
revolutionized the system design aspects due to its flexibility [6]. Due to
the momentum gained by the adoption of RISC-V, there is an increasing demand
for RISC-V based system design and prototyping. While there have been several
attempts to design and develop efficient single-core RISC-V compute platforms,
there are a handful of MPSoCs [7][8]. There is an increasing need for the
frameworks to support RISC-V based MPSoC exploration to enable the system
designers to develop efficient platforms for the next-generation computing
systems.

In this paper, we present _ANDROMEDA_ , a unified framework that facilitates
the design space exploration of FPGA based MPSoCs. The ANDROMEDA framework
helps in identifying the bottlenecks in application execution for system-level
optimizations. The major contributions in this paper are as follows:

  * •

A light-weight network-on-chip (NoC) for clustered RISC-V MPSoC platform
development.

  * •

ANDROMEDA, an FPGA-based MPSoC framework for early-stage exploration and
application execution bottleneck identification.

  * •

Evaluation of ANDROMEDA using STREAM, matrix multiply, and N-body simulations,
and identification of bottlenecks for the benchmarks.

The rest of the paper is organised as follows: In Section II, we discuss
background and the literature. A light-weight NoC implementation is discussed
in Section III along with the proposed ANDROMEDA framework. The experimental
setup and results are depicted in Section IV. We conclude our work in Section
V.

##  II Background and Related Work

###  II-A Background

####  II-A1 RocketChip Generator

The Rocket Chip is an open-source tool to instantiate the Rocket Core, and
synthesizable system-on-chip (SoC). The Rocket Chip generator tool is
implemented in the _Chisel_ hardware construction language. The major
advantage of the RocketChip generator is its configurability. The parameters
such as, number of cores, cache size, cache placement policies, arithmetic
units, pipeline stages, memory management unit, hardware performance counters,
interconnect, and others can be customised to attain the SoC.

####  II-A2 Rocket core

Rocket is a 5-stage single-issue in-order CPU which implements the RISC-V ISA.
It supports both RV32G and RV64G. Rocket features a non-blocking data cache, a
branch predictor, an MMU, and an FPU. Rocket can be configured to meet
individual requirements. Options include supported ISA extensions (e.g., M, A,
F, D) and cache sizes.

####  II-A3 Synopsys HAPS-80D Dual

Figure 1: Synopsys HAPS-80D Dual

Synopsys high-performance ASIC prototyping systems (HAPS) is a family of FPGA
based systems for prototyping ASICs and SoCs. HAPS solutions are shipped with
prototyping hardware and the supporting software tools, to enable faster
system validation and earlier software development.

Fig. 1 shows the overall system architecture of HAPS-80D Dual platform. The
platform consists of two UltraScale XCVU440 FPGAs. The FPGAs are connected via
several high-speed serial links. HAPS-80D Dual is equipped with 13 low-skew
clock networks. The global clocks are labeled from GCLK0 to GCLK2. GCLK0 is a
fixed 100 MHz clock reserved for system function. GCLK1-GCLK12 are available
for user designs, and in most cases, are sourced from the on-board phase-lock-
loops (PLLs). The HAPS-80D Dual contains an on-board DDR4 DRAM module with 8GB
capacity. It can be used as memory in user designs, or as a large sample
memory for signal debugging. HAPS-80D Dual features various I/O options, such
as GPIO, PMOD, JTAG and UART. The HAPS-80D Dual system connects to a host
computer via a high-speed USB-C cable. The Synopsys UMRBus protocol is used
for both configuring the system with the user design, and for subsequent
debugging.

###  II-B Related Work

There has been a plethora of MPSoC works in the literature focusing on early
stage design space exploration [5]. Some of the early works focused on design
automation for custom generation of MPSoCs, while a few of them focused on
industrial-grade design customization. Daedalus framework presented in [9] and
[10] introduces a system-level exploration, programming and prototyping
framework. The Daedalus framework takes a sequential application as an input
and translates it into an MPSoC implementation on FPGA. The Daedalus framework
considers only dataflow dominated applications.

The system-on-chip environment (SCE) is an interactive framework presented in
[11]. The SCE frameworks accepts high-level specifications as an input and
translates the specifications into hardware/software implementation. The major
bottleneck in the SCE design flow is the initial system-level specifications
for the desired hardware/software implementation. The SystemCoDesigner
framework presented in [12] has similar limitations.

Our proposed framework, ANDROMEDA, is an over-simplistic framework for RISC-V
based MPSoC prototyping on FPGA. The input to the framework are several high-
level system parameters that result in an MPSoC implementation on Synopsys
HAPS-80D Dual platform.

##  III Network-on-Chip and Andromeda

TABLE I: NoC configurations Name | Nodes | Proc. cores / node | Router | Flow-control
---|---|---|---|---
NOC_BASE | 16 | 4 | $S_{k}$ | Store-and-forward
NOC_SW | 16 | 4 | AXIS-Switch | Store-and-forward
NOC_SW_C | 16 | 4 | AXIS-Switch | Cut-through

A base system, NOC_BASE, was designed, which was used as a starting point for
design space exploration. Table I shows the parameters of NOC_BASE, together
with other configurations, which are explored in later section. In the
following, the hardware architecture of NOC_BASE is covered.

Figure 2: The RocketChip mesh NoC: numbers at top left corners of nodes
indicate node IDs; numbers inside nodes indicate core (hart) IDs. Figure 3:
Internal structure of a node in NOC_BASE

The system consists of 16 nodes, $N_{0}...N_{15}$, interconnected in a
$4\times 4$ 2-D mesh network, as shown in Fig. 2. Inside each node $N_{k}$, a
RocketChip SoC with $(n+1)$ cores, $C_{k0}...C_{kn}$ (hart 0 … hart $n$) is
located. Every RocketChip instance is created from the same generated
ExampleRocketSystem. Cores $C_{k0}...C_{k,(n-1)}$ are (general-purpose)
processing cores, while $C_{kn}=:S_{k}$ is a small RocketCore, which is also
coherently interconnected with the processing cores. The small core’s role is
to act as a co-processor to handle all network-related tasks, like routing and
switching. Though using a general-purpose core as a software router is likely
not the optimal solution, this route was initially chosen to get to a working
prototype system as quickly as possible. Furthermore, without the RocketChip
generator, the design of this particular approach would have been more
involved, too.

TABLE II: Data cache configurations Name | nSets | nWays | Size (KiB)
---|---|---|---
BASE | 64 | 4 | 16
C-64-8 | 64 | 8 | 32
C-64-16 | 64 | 16 | 64

The memory model of the system follows the distributed memory paradigm. Each
node contains BRAM as main memory (used for data and instructions), which is
only accessible by the node’s local cores, and not from other nodes. The size
of the BRAM is configurable in Vivado and was set to be 256 kB for all nodes,
except $N_{0}$, which has 2 MB. In total, the complete system can fit up to
5.75 MB of on-chip data. $N_{0}$ additionally incorporates an AXI UART Lite to
interface with a serial console on the host computer. Fig. 3 shows the
internal structure of a node. We explain usage of NoC and RocketChip to
incorporate a distributed memory system, ANDROMEDA for system level
exploration (see Fig. 4).

Figure 4: ANDROMEDA framework for FPGA based MPSoC exploration

ANDROMEDA is a simple framework that consisting of input system parameters
that are used for system configuration. The parameters are the number of
cores, cache size (see configuration in Table II), and coherency, floating-
point unit (FPU) and its pipeline stages, memory management unit, hardware
performance counters, and interconnect network and its parameters.

In ANDROMEDA, a designer sets the parameters that automatically generate the
desired multicore or manycore system. The generated system is later prototyped
on the HAPS-80D Dual FPGA prototyping platform, where the system is evaluated
using benchmarks written in the C programming language. If the performance
attained is not satisfactory, then the parameters are calibrated through the
designer intervention to arrive at a satisfactory performance. The proposed
ANDROMEDA framework enables rapid prototyping as the system parameters are at
a higher level of abstraction than in the literature. At this stage, automatic
tuning of the parameters is out of the scope of this work.

##  IV Experimental Setup and Results

###  IV-A Benchmarks

To evaluate the performance of ANDROMEDA, three benchmarks written in C have
been applied. In the following, the selected benchmarks are briefly explained.

####  IV-A1 STREAM

The STREAM benchmark [13] is used to measure the sustainable memory bandwidth
of computer systems. Though initially targeted for high-performance computing
systems, it can also be applied on personal or embedded computers. STREAM
executes four kernels on contiguous arrays a, b and c, and determines from the
execution time the resulting bandwidth. The kernels are: COPY (a[i] = b[i]),
SCALE (a[i] = q*b[i]), ADD (a[i] = b[i] + c[i]), and TRIAD (a[i] = b[i] +
q*c[i]).

To minimize cache effects, the sizes of the arrays should, in general, be
larger than the largest available cache. The data type of the arrays can be
selected in the benchmark, with the default type being double, which was also
used in this work.

####  IV-A2 Matrix Multiplication

As matrix multiplication (Matmul) is an operation found in many applications,
a Matmul benchmark was also implemented for RocketChip. It multiplies two
matrices, $A\ (N\times K)$ and $B\ (K\times M)$, and writes the result into a
third matrix, $C\ (N\times M)$. Matrix $A$ and $C$ are stored in row-major
format, while matrix $B$ is stored column-major. All matrices contain double-
precision values.

####  IV-A3 N-body Simulation

The N-body simulation (N-body) numerically solves the N-body problem, which is
a classic problem from orbital mechanics. Given are $N$ bodies with mass
$m_{i}$, initial position $\vec{r_{i}}(t=0)$ and initial velocity
$\vec{v_{i}}(t=0)$. The bodies exert forces on each other according to
Newton’s gravitation law. The net force exerted on body $i$ is then given by:

$\vec{F_{i}}(t)=\sum_{\begin{subarray}{c}j=0\\\ j\neq
i\end{subarray}}^{N-1}{\frac{Gm_{i}m_{j}}{\left\lVert\vec{r_{i}}(t)-\vec{r_{j}}(t)\right\rVert^{3}}(\vec{r_{i}}(t)-\vec{r_{j}}(t))}.$
(1)

The goal is to find the positions and velocities of each of the bodies after a
time $t$. The simulation is performed in several timesteps. In each timestep,
the net force exerted on each body is calculated according to equation 1. From
the net force, the acceleration experienced by that particular body is
computed. Then, based on the acceleration, the new position and velocity can
be derived. The computational complexity of an N-body simulation is
$\mathcal{O}(N^{2})$. All values (masses, positions, velocities) are stored in
single-precision floating-point format (float). The storage requirement scales
with $\mathcal{O}(N)$.

###  IV-B Parallelization Techniques

For parallelizing the benchmarks, the worksharing principle was applied. In
all three benchmarks (STREAM, Matmul, N-body), the work can be distributed
evenly among the cores. For bare-metal applications, some manual work is
required to realize the worksharing. In general, each core calculates the
start index of its designated partition based on its hart ID. After a parallel
region, synchronization may be necessary. The riscv-tests provide a barrier()
function, which can be used to synchronize the cores.

Matmul can be parallelized by dividing matrix $A$ into sub-matrices $A_{i}$.
Each core $i$ then calculates the corresponding partition $C_{i}$ of matrix
$C$ by multiplying $A_{i}$ with $B$.

N-body is parallelized by partitioning the arrays containing the body data
evenly among the cores. Each core then computes the new positions and
velocities of the bodies in its assigned partition. At the end of each
timestep, a barrier is required, to ensure that each core has updated its body
values before the next timestep starts.

###  IV-C Evaluation

This section evaluates the ANDROMEDA presented in section III for different
parameters and evaluates single node (BASE) resource utilization and benchmark
performance. BASE32 represents single node with 32 cores.

####  IV-C1 Number of cores

##### FPGA utilization

Table III shows the FPGA resource utilization of BASE for core counts from one
to 32. Synthesis for all versions was constrained to 10 MHz. As can be seen in
table III, the LUT counts seem to suggest sub-linear scaling. However, this is
primarily caused by modules inside the SoC that are independent of the number
of cores. A linear increase can be observed for the number of BRAM, which are
used for the L1 data and instruction caches, and DSP slices, which are, for
example, used to implement parts of the floating-point unit (FPU). The 32 core
system utilizes around 40% of the available LUTs.

TABLE III: FPGA utilization of BASE (f = 10 MHz) Number of cores | LUTs | BRAM | DSP
---|---|---|---
1 |  37,324 ($\times 1$) | 12 | 35
2 |  69,053 ($\times 1.9$) | 24 | 70
4 |  131,188 ($\times 3.5$) | 48 | 140
8 |  255,000 ($\times 6.8$) | 96 | 280
16 |  499,931 ($\times 13.4$) | 192 | 560
32 |  998,145 ($\times 26.7$) | 384 | 1120

##### STREAM

Figure 5a shows the memory bandwidth measurements made from the STREAM
benchmark. It was run on BASE32 with an array size of 128,000. The theoretical
peak memory bandwidth delivered by the outer memory bus (i.e., SmartConnect)
is 8 bytes per cycle, as the AXI data bus is 64 bit wide. As Fig. 5a suggests,
with increasing core count, a larger fraction of the available memory bus
bandwidth could be utilized. A linear scaling can be seen for one to four
cores. Beginning at eight cores, however, the sustainable bandwidth begins to
saturate at about 1.41 bytes/cycle for the COPY kernel, well below the
theoretical maximum.

(a)

(b)

(c)

Figure 5: (a) STREAM benchmark run on BASE32, size = 128,000 doubles, (b)
Matmul (N=K=M=128) run on BASE32, and (c) N-body simulation with N=4,096
bodies and 10 timesteps run on BASE32.

##### Matmul

Matmul with N=K=M=128 was run on BASE32. Fig. 5b shows the average execution
times (in cycles) for varying core counts. Until four cores, the speedup is
approximately equal to the number cores. At eight cores, the speedup drops to
about 6 (instead of ideally 8). The situation worsens for even larger core
counts. These results again show the limitation of shared memory
architectures.

##### N-body

Fig. 5c plots the execution times (in cycles) of an N-body simulation with
N=4,096 bodies run on BASE32. As can be seen from Fig. 5c, the cycle count
decreases with increasing core count, as expected. Up until 16 cores, close-
to-ideal speedup is achieved. For 32 cores, however, the speedup is only 20
(instead of ideally 32). At high core counts, the shared memory bus can still
limit the performance.

Overall it can be concluded that adding more cores, in general, improves
performance. However, if the number of cores sharing a bus gets too large ($>$
8), the limited bus bandwidth caps the performance.

The situation can be improved by reducing the pressure applied on the bus. One
method is to increase the size of the caches so that the probability of
finding data in the local cache is increased. The next section analyzes the
effects of the different cache configurations from Table II.

####  IV-C2 Cache subsystem

##### FPGA utilization

The synthesis results for the cache configurations from Table II are shown in
Table IV. The percentages indicate the relative increase in the respective
resource count compared to BASE. As expected, the usages of both LUTs and
BRAMs of the L1 data cache grow as the number of ways is doubled. The last
column, ”Total LUTs”, refers to the LUT usage of the complete
ExampleRocketSystem of a particular configuration. It can be seen that the
increase of LUTs is reasonable for both cache configurations.

TABLE IV: FPGA utilization of cache configs (f = 50 MHz) Config | L1 Data Cache | Total LUTs
---|---|---
LUTs | BRAM
BASE | 3065 | 4 | 131026
C-64-8 |  3579 ($+16.8\%$) |  8 ($+100\%$) |  133271 ($+1.7\%$)
C-64-16 |  4248 ($+38.6\%$) |  16 ($+300\%$) |  136078 ($+3.9\%$)

To compare the different L1 data cache sizes, Matmul and N-body were run on
the three configurations.

##### Matmul

Fig. 6a shows the execution time (in cycles) for a Matmul with N=K=M=128 run
on the three configurations from Table IV. It can be seen that in general, the
performance improves for larger cache sizes. For a cache size of 64 kb
(C-64-16), the execution time drops by over 8% for all core counts compared to
BASE. This result is achieved with an LUT overhead of less than 4%, but with
quadruple the amount of BRAMs per L1 data cache.

(a)

(b)

Figure 6: (a), Matmul (N=K=M=128) run on BASE, C-64-8 and C-64-16 for core
counts 1, 2 and 4, and (b) N-body simulation (N=2,048, 10 timesteps) run on
BASE, C-64-8 and C-64-16 for core counts 1, 2 and 4.

##### N-body

Fig. 6b shows the execution time (in cycles) for an N-body simulation with
N=2,048 bodies. As for Matmul, it can be observed that the execution time
reduces when more cache is available. A larger cache means that more bodies
can be stored in the local memories.

The results show that larger caches indeed improve performance, though
depending on the particular application’s working set size and access
patterns. The total area overhead is mainly devoted to on-chip memory.

###  IV-D Network-on-Chip

The NoC presented in Section III has been designed to evaluate the benefits of
distributing cores across multiple nodes, instead of having the cores share a
single bus. This section evaluates the explored NoC configurations from Table
I.

####  IV-D1 FPGA synthesis results

Table V summarizes the total resource usage of the three explored
configurations. Both NOC_SW and NOC_SW_C reduce the overall utilization
compared to NOC_BASE. NOC_SW and NOC_SW_C, both use hardware switches instead
of a software router. NOC_SW_C uses a switch with cut-through switching. The
primary savings in terms of LUTs and BRAMs are the consequence of removing
three AXI4-Stream memory-mapped FIFOs from each node. LUTs can be saved
because the AXI4-Stream Data FIFOs, added in NOC_SW and NOC_SW_C, require less
logic as they do not come with an MMIO interface.

TABLE V: FPGA utilization of NoC configs (f = 5 MHz) Config | LUTs | BRAM
---|---|---
NOC_BASE | 2,011,152 | 2,384
NOC_SW |  1,905,056 ($-5.3\%$) |  2,368 ($-0.7\%$)
NOC_SW_C |  1,901,234 ($-5.5\%$) |  2,368 ($-0.7\%$)
TABLE VI: FPGA utilization of NoC configs (f = 5 MHz) Config | LUTs
---|---
NOC_BASE | 52,992
NOC_SW |  39,016 ($-26.4\%$)
NOC_SW_C |  35,880 ($-32.3\%$)

####  IV-D2 Benchmarks

The performance and scaling of the NoC is analyzed using the previously
introduced Matmul and N-body benchmarks.

##### Matmul

For parallelizing Matmul on the NoC, the principles from the shared memory
parallel version presented earlier were extended to multiple nodes. Now, each
node $N_{k}$ is assigned a partition $A_{k}$. The partitions are distributed
by $N_{0}$ using the noc_scatter() function. Further, each node needs the
complete matrix $B$ to compute its respective partition $C_{k}$ of matrix $C$.
$B$ is distributed by $N_{0}$ using the noc_bcast() function. The calculated
partitions $C_{k}$ are finally collected by $N_{0}$ with the noc_gather()
function.

Fig. 7a shows the execution times (in cycles) for a Matmul with N=256, K=128
and M=32, run on different node/core arrangements. It can be observed that the
computation time decreases with increasing aggregate core count. This scaling
even holds for the maximum core count of 64. Recall, that in the shared memory
case, the cycle count began to increase already for core counts above eight.
Fig. 7b compares the execution times for a Matmul (N=512, K=128, M=32) between
two 16 core arrangements, (16,1) and (4,4), and a 16-core SMP system. It can
be seen that even though the 16-core SMP system suffers from the scaling
bottleneck as seen in Fig. 5b, it still performs better than the two compared
node/core configurations, which are dominated by the communication overheads.

In conclusion, the naive parallel Matmul is not well suited for execution on a
distributed memory system, where communication dominates computation. Though
exploiting intra-node parallelism improves performance in general, the gains
are quickly overshadowed by the communication overhead, as more nodes are
introduced. Lastly, it must be again stated that the potential of the NoC
designed in this work is likely limited by the small cores $S_{k}$.

(a)

(b)

(c)

Figure 7: (a) Matmul (N=256, K=128, M=32) run on NOC_SW_C. (b) Comparison of
Matmul (N=512, K=128, M=32) run on NOC_SW_C and 16-core BASE32, and (c) N-body
simulation (N=4,096) run on NOC_SW_C. The values at the top of each bar
indicate the speedup compared to the single core version from Fig. 5c.

##### N-body

As the N-body simulation is a computationally intensive application, it
appears to be a good candidate to benefit from a distributed system. The NoC
parallel version follows similar principles as the shared memory version.

Fig. 7c shows the execution times (in cycles) for an N-body simulation with
N=4,096. The speedup values compared to the single core version from Fig. 5c
are indicated at the top of each bar. It can be immediately noticed that the
computation to communication ratio is considerably larger than for Matmul.
Further, the speedup is superlinear up until arrangement (4,2) compared to the
speedups of the SMP version shown in Fig. 5c.

Fig. 8 compares N-body on NOC_SW_C with BASE32 for increasing body numbers.
The performance is measured in single-precision FLOPs/cycles. In Fig. 8a, the
performance is compared for 16 cores. It can be seen that the 16-core SMP
system performs better for a smaller number of bodies ($<$2,048).

In Fig. 8b, a total of 32 cores are compared. As was the case for 16 cores,
the SMP system performs better for smaller body counts. At body counts larger
than 1,024, the performance of the SMP system drops significantly due to the
fact that the working sets get larger than the available L1 data caches,
resulting in increased bus pressure. The distributed computation, on the other
hand, does not experience this bottleneck for the same number of bodies.

(a) 16 cores

(b) 32 cores

Figure 8: Comparison of N-body between NOC_SW_C and two SMP systems (BASE32)

In summary, it was shown that the benefit of employing a distributed system
largely depends on the computation to communication ratio. The naive Matmul
does not benefit much from distributing the workload across multiple nodes; in
fact, the performance worsens when more than eight nodes are used, due to the
heavy communication requirements. The N-body simulation can profit from the
NoC for large N, as the computation greatly dominates the communication.

Further, it was shown that the hybrid approach, i.e., having multiple cores
inside each node, can yield an additional performance gain, as intra-node
parallelism can be exploited. The other benefit of hybrid systems is that less
inter-node communication is required with the same total core count (e.g.,
(4,4) vs. (16,1)).

##  V Conclusion

We presented a light-weight NoC for distributed memory MPSoC. Later, we
presented ANDROMEDA, a RISC-V MPSoC exploration framework that incorporated
the light-weight NoC. The generated RISC-V MPSoCs are prototyped on the
Synopsys HAPS-80D Dual. The experimental evaluation demonstrated that the
ANDROMEDA framework is easy to use for early-stage system prototyping. The
user analyses of the benchmarks and applications help to identify the
performance bottleneck in the NoC-based MPSoCs. In the future, we plan to
extend support for other FPGA platforms and focus on RISC-V customization for
further run-time reduction.

## References

  * [1] C. E. Salloum _et al._ , “The across mpsoc – a new generation of multi-core processors designed for safety-critical embedded systems,” in _2012 15th Euromicro Conference on DSD_ , 2012, pp. 105–113.
  * [2] J. Zhou _et al._ , “Security-critical energy-aware task scheduling for heterogeneous real-time mpsocs in iot,” _IEEE Transactions on Services Computing_ , vol. 13, no. 4, pp. 745–758, 2020.
  * [3] C. Wang _et al._ , “Service-oriented architecture on fpga-based mpsoc,” _IEEE TPDS_ , vol. 28, no. 10, pp. 2993–3006, 2017.
  * [4] A. Kumar _et al._ , “An fpga design flow for reconfigurable network-based multi-processor systems on chip,” in _2007 DATE_ , 2007, pp. 1–6.
  * [5] A. Gerstlauer _et al._ , “Electronic system-level synthesis methodologies,” _IEEE TCAD_ , vol. 28, no. 10, pp. 1517–1530, 2009.
  * [6] A. Waterman _et al._ , _The RISC-V Instruction Set Manual, Volume II: Privileged Architecture, Document Version 20190608-Priv-MSU-Ratified_ , RISC-V Foundation, June 2019.
  * [7] A. Kamaleldin _et al._ , “Modular memory system for RISC-v based MPSoCs on xilinx FPGAs,” in _2019 IEEE 13th International Symposium on Embedded Multicore/Many-core Systems-on-Chip (MCSoC)_. IEEE, pp. 68–73. [Online]. Available: https://ieeexplore.ieee.org/document/8906519/
  * [8] F. Zaruba _et al._ , “The cost of application-class processing: Energy and performance analysis of a linux-ready 1.7ghz 64bit risc-v core in 22nm fdsoi technology,” _IEEE TVLSI_ , vol. 27, no. 11, pp. 2629–2640, 2019.
  * [9] M. Thompson _et al._ , “A framework for rapid system-level exploration, synthesis, and programming of multimedia mp-socs,” in _2007 5th IEEE/ACM/IFIP CODES+ISSS_ , 2007, pp. 9–14.
  * [10] H. Nikolov _et al._ , “Daedalus: Toward composable multimedia mp-soc design,” in _2008 45th ACM/IEEE DAC_ , 2008, pp. 574–579.
  * [11] R. Dömer _et al._ , “System-on-chip environment: A specc-based framework for heterogeneous mpsoc design,” _EURASIP J. Embed. Syst._ , vol. 2008, 2008. [Online]. Available: https://doi.org/10.1155/2008/647953
  * [12] J. Keinert _et al._ , “Systemcodesigner—an automatic esl synthesis approach by design space exploration and behavioral synthesis for streaming applications,” _ACM Trans. Des. Autom. Electron. Syst._ , vol. 14, no. 1, Jan. 2009. [Online]. Available: https://doi.org/10.1145/1455229.1455230
  * [13] J. D. McCalpin, “Stream: Sustainable memory bandwidth in high performance computers,” University of Virginia, Charlottesville, Virginia, Tech. Rep., 1991-2007, a continually updated technical report. http://www.cs.virginia.edu/stream/. [Online]. Available: http://www.cs.virginia.edu/stream/


# More than Vanilla Fusion: a Simple, Decoupling-free, Attention Module for
Multimodal Fusion Based on signal Theory

###### Abstract

The vanilla fusion methods still dominate a large percentage of mainstream
audio-visual tasks. However, the effectiveness of vanilla fusion from a
theoretical perspective is still worth discussing. Thus, this paper
reconsiders the signal fused in the multimodal case from a bionics perspective
and proposes a simple, plug-and-play, attention module for vanilla fusion
based on fundamental signal theory and uncertainty theory. In addition,
previous work on multimodal dynamic gradient modulation still relies on
decoupling the modalities. So, a decoupling-free gradient modulation scheme
has been designed in conjunction with the aforementioned attention module,
which has various advantages over the decoupled one. Experiment results show
that just a few lines of code can achieve up to 2.0% performance improvements
to several multimodal classification methods. Finally, quantitative evaluation
of other fusion tasks reveals the potential for additional application
scenarios.

Index Terms—  multimodal fusion, modulation scheme, signal theory

##  1 Introduction

In recent years, we have witnessed a large number of multimodal methods [1, 2,
3, 4], which comprehensively surpassed unimodal methods in multiple tasks. In
in-depth investigations of the audio-visual area, some studies addressed
imbalanced optimization and coverage rate as an obstacle in coarse-grained
multimodal learning [5, 6]. Thus, the potential of multiple modalities could
not be fully exploited [7]. To cope with this, extra uni-modal classifiers [6]
and auxiliary loss [6, 8] were introduced to cultivate better models.
Furthermore, studies [6, 9] have shown that gradient modulation could
alleviate the imbalance between modalities. As an effective modulation method,
on-the-fly gradient modulation [7] based on approximate uni-modal performance
was proposed to enable parameters of different modalities to update with
inconsistent learning rates. Then, the attention-based multi-modal fusion
strategies [2], allowing cross-modal information interaction during training,
alleviated the imbalance via information sharing.

The application of informatics knowledge has been utilized in the field of
deep learning for a long time. Informatics correlation was first applied to
the training loss of learning models based on canonical correlation analysis
(CCA) [10, 11]. As a competitive substitution, researchers [12, 13, 14]
proposed the deep learning implementation of HGR maximal correlation, Soft
HGR, which has been proven to learn better correlation and to hold lower
computing complexity than CCA. However, since the mechanism of auditory [15]
and visual attention [16] was well demonstrated in cognitive science, another
group of inference methods [17] based on bionic signal theory has emerged and
has been proven to be enlightening in the field. SimAM [18] proposed to treat
the weight of neurons as signals and found the importance of each neuron by
optimizing an energy function without adding extra parameters, which was found
also suitable for speech-related tasks [19]. Naturally, such an approach based
on informatics energy theory can be utilized for audio and visual separately,
so why not fusion?

The “vanilla” fusion method, which simply consists of the regular
transformation applied to multimodal inputs, can be described as a series of
discrete signals undergoing simple operations. So, inspired by bionic
thoughts, closed-form solutions based on signal theory are derived on the
basis of SimAM [18] to optimize energy and mutual energy. Based on the study
[20], we find that this optimization can be understood from both an energy
perspective and an uncertainty perspective, which further proves the
feasibility of the theory. Furthermore, based on derivation, the vanilla
fusion methods have been improved through just a few dozen lines of code,
which is named Simple Attention Module for Multimodal (SimAM2). Then to avoid
cumulative errors caused by uncoupling, we apply a method that does not
require the decoupling of fusion features to achieve convergence supervision
of different modalities. During the experiment, the methods above are
individually validated for effectiveness and work together to yield up to 2.0%
improvement in results across multiple audio-visual classification datasets
including CREMA-D [21], VGGSound [22]. Finally, extra experiments beyond
classification are elicited to demonstrate the possibility of other scenarios.

Fig. 1:  The pipeline of SimAM2 in vanilla fusion

##  2 Method

Here, two important assumptions are introduced. Firstly, in visual
neuroscience, the most informative neurons are usually the ones that show
distinctive stimulation patterns from surrounding neurons [18]. In other
words, the neurons displaying clear spatial suppression effects should be
given higher priority. Moreover, the same thing goes on in multimodality
processing. Secondly, when the current point is below the threshold of
triggering the action potential, all electrical signals generated by nerve
cells are collectively superimposed [23] as Eq.4 on the resting potential. So,
with multimodal stimuli, neurons need to make not only unimodal weight
adjustments based on unimodal signal but also comprehensive adjustments based
on the overall stimuli.

Therefore, we propose a series of modifications to vanilla fusion based on the
assumption of linear superposition of neuronal stimuli and make adaptations to
the overall training method. The elaboration on the overall method is divided
into 3 parts, Simple Attention Module for Multimodal in Sec.2.1, modification
on vanilla fusion in Sec.2.2, and coupling-free approach in Sec.2.3 to control
training equilibrium.

###  2.1 Simple Attention Module for Multimodal (SimAM2)

Since the spatial suppression effects assumption in unimodal neurons is
exactly the same as SimAM [18], we follow most of the derivations of SimAM
regarding the energy part of neurons. So the energy function of neurons still
follows it as

$\displaystyle e_{t}\left(w_{t},b_{t},\mathbf{y},x_{i}\right)=$
$\displaystyle\frac{1}{M-1}\sum_{i=1}^{M-1}\left(-1-\left(w_{t}x_{i}+b_{t}\right)\right)^{2}$
(1) $\displaystyle+\left(1-\left(w_{t}t+b_{t}\right)\right)^{2}+\lambda
w_{t}^{2}.$

Note, $t$ and $x_{i}$ are the target neuron and other neurons;
$\left(w_{t}x_{i}+b_{t}\right)$ and $\left(w_{t}t+b_{t}\right)$ are linear
transforms of $x_{i}$ and $t$; $M$ is the number of neurons on that channel
$\lambda$ is the regularization term coefficient. Here, $\lambda$, as is used
to limit the growth of the model, should be discussed later. The same fast
closed-form solution with respect to $w_{t}$ and $b_{t}$ is

$\displaystyle w_{t}$
$\displaystyle=\frac{2\left(t-\mu_{t}\right)}{\left(t-\mu_{t}\right)^{2}+2\sigma_{t}^{2}+2\lambda},$
(2) $\displaystyle b_{t}$
$\displaystyle=-\frac{1}{2}\left(t+\mu_{t}\right)w_{t}.$

while $\mu_{t}$ and $\sigma_{t}^{2}$ are mean and variance calculated over all
neurons $x_{i}$ except $t$ in that channel. The minimal energy can be
expressed as a function with respect to $\lambda$

$\vspace{-0.2cm}e_{t}^{*}(\lambda)=\frac{4\left(\hat{\sigma}^{2}+\lambda\right)}{(t-\hat{\mu})^{2}+2\hat{\sigma}^{2}+2\lambda},$
(3)

where $\hat{\sigma},\hat{\mu}$ are mean and variance calculated over all
neurons.

Interestingly, MMCosine [8] made the assumption that vanilla fusion could all
be represented as concatenation and then made adjustments to vanilla fusion
methods. However, based on the biological theory in Sec.2, we propose the
following hypothesis of linear superposition,

$\vspace{-0.2cm}U=\zeta X_{1}+(1-\zeta)X_{2}.$ (4)

$X_{1}$,$X_{2}$,$U$ are representations of two different modalities and the
fusion representation respectively. Here, $\zeta$ is the adjustment weight
obtained by a learnable module and has the same number of channels as
$X_{1}$,$X_{2}$, and $U$. After taking the expression for minimum energy into
the energy equation for the signal superimposition, the result is as follows

$\displaystyle e_{t,U}\left(\lambda\right)=$
$\displaystyle\zeta^{2}e_{t,x_{1}}\left(\lambda=0\right)+(1-\zeta)^{2}e_{t,x_{2}}\left(\lambda=0\right)$
(5) $\displaystyle+2\zeta(1-\zeta)E(x_{1},x_{2}).$

Here we do not introduce regularization coefficients ($\lambda$=$0$) for the
two modalities separately but instead add regularization to the fused
representation and overall regularization. Note, due to the non-linear
operation in calculating energy, the mutual energy $E(x_{1},x_{2})$ here
cannot be simply understood as a variable related to the linear correlation
coefficient.

Speaking of correlation, how to estimate modal correlations without decoupling
and introducing additional classifiers is essential in this derivation. We
forced the correlation based on the soft HGR maximum correlation coefficient
[13, 14] of the two modalities to be exactly equal to a particular value by
the MAE. Then the batch’s distribution of the $\zeta$ neurons in the Eq.7 is
observed after full training. A more scattered distribution of $\zeta$ is
recorded under larger and smaller correlation circumstances. So we consider

$\displaystyle
r=tanh(\frac{s\sum_{i}var(\zeta_{i})+\lambda}{\sum_{i}var_{max}(\zeta_{i})+\lambda}),$
(6)

as an indirect measurement to estimate correlations. $var(\zeta_{i})$ and
$var_{max}$ are the variance of the $i$-th $\zeta$ neuron within the batch and
the highest variance record of this neuron respectively. $\lambda$ is the same
regulation term, usually $1e-6$ in Eq.3, and $s$ is the scaling
hyperparameter. In other words, when two modalities are completely
uncorrelated or completely correlated, $\zeta$ is unable to learn which modal
signal is more important, resulting in a random distribution ($r\rightarrow
1$).

In a general sense, multimodal tasks are two modalities making reasonable
judgments with common information. This means both mutual energy
$E(x_{1},x_{2})$ and overall fusion energy $e_{t,U}$ need to be optimized. But
simply using $e_{t,U}$ will cause instability in extreme cases, which makes us
add an extra factor $r$ to form degradation term

$\displaystyle E^{*}=$ $\displaystyle
e_{t,U}-\zeta^{2}{e_{t,x_{1}}}-(1-\zeta)^{2}{e_{t,x_{2}}}+(1-r)e_{t,U},$ (7)

The above equation can be explained as the numerator of the transformed Eq.5
plus a degradation term (also analyzed in Sec.2.2) that limits the extreme
conditional degeneracy of the correlation.

The fusion weights $U$ are updated with the formula

$\displaystyle\bar{U}=$
$\displaystyle\operatorname{sigmoid}\left(E^{*}\right)\odot U,$ (8)

Then, the derivation of the update of the fusion feature $U$ is completed.

Interestingly, from the perspective of uncertainty estimation, it also
explains the feasibility of SimAM2. Q. Zhang [20] proposed and proved an
assumption that the estimated uncertainty of each modality is positively
correlated with its modal-specific loss logits. Meanwhile, Zhang used energy
scores calculated from the sum of logits to update the logits of each modality
in late fusion. Based on this conclusion of Zhang, using a single variable to
update the fused feature is effective in late fusion. Our proposed Eq.8 has
the same form as Zhang proposed. So from another perspective, Eq.8 can be seen
as considering the uncertainty of each neuron more meticulously, rather than
the whole feature as [20].

###  2.2 Improvement of Vanilla Fusion

Total 3 methods of vanilla fusion are discussed and 2 of them improved on the
grounded theory of SimAM2 (Sec.2.1).

Summation: There are two dominant methods of summation widely used. The
simplest method is the direct summation like [7], which does not adjust
$\zeta$ automatically. So the implementation of SimAM2 is just letting
$\zeta$=0.5 in Eq.4. As a second and slightly more complex approach, learnable
parameters are used to adjust the weights of both sides (learnable $\zeta$ in
Eq.4) [24]. In practice, the latter method seems to be more advantageous apart
from being more computationally intensive.

FiLM: An adaptive affine fusion proposed by FiLM [25] innovatively introduces
learnable feature modulation, but obviously using different major modality
will have a large impact on the results.

$\displaystyle\operatorname{FiLM}\left(F_{i,c}\mid\gamma_{i,c},\beta_{i,c}\right)=\gamma_{i,c}F_{i,c}+\beta_{i,c}.$
(9)

FiLM learns functions $f$ and $h$ which have input of $x_{i}$ and output
$\gamma_{i,c}=f_{c}\left(\boldsymbol{x}_{i}\right)$ and
$\beta_{i,c}=h_{c}\left(\boldsymbol{x}_{i}\right)$. In other words, swapping
the order of the two modalities in the calculation will give different
results. So $\gamma_{i,c}$ and $\beta_{i,c}$ modulate a neural network’s
activations $F_{i,c}$, whose subscripts refer to the $i^{th}$ input’s $c^{th}$
feature. Thinking about it another way in fact the first term of Eq.9 is just
a coupling of one modality on top of another modality weight. So the two terms
of Eq.9 still essentially have completely different modality information. At
the same time, for more general multimodal domains, we still want to use
$\zeta$ to balance the training process. So Eq.9 is transformed into the form
of Eq.4 as

$\displaystyle\operatorname{FiLM}\left(F_{i,c}\mid\gamma_{i,c},\beta_{i,c}\right)=\zeta\gamma_{i,c}F_{i,c}+(1-\zeta)\beta_{i,c}.$
(10)

After treating these terms as different modalities, the rest of the
calculation is the same as the summation. Later experiments demonstrated that
the introduction of $\zeta$ alleviated the problem generated by different
major modalities.

Concatenation: It is obvious that concatenation can be seen as the case where
two vectors are respectively padded at different positions and expressed as
Eq.4. However, the vectors after padding are clearly linearly uncorrelated. So
the degradation term and the mutual energy in Eq.5 will degrade to $0$ as
analysed in Sec.4. The reason why the mutual energy is not completely equal to
zero is that there have been nonlinear transformations on the features before.

###  2.3 Decoupling-free gradient modulation

A number of works [7, 8] have emerged in recent years to provide solutions to
the imbalance of multimodal training from the perspective of metric learning,
and backpropagation respectively. Let’s take OGM-GE [7] as an example. OGM-GE
needs to decouple the fused features and then calculate the logits of each
modality. However, the fusion involves non-linear layers such as sigmoid, and
more complex combinations of multi-layer convolution and activation layers,
which makes it hard to accurately calculate the logits of individual modality.
So, the final result is not as good as it could be.

According to OGM-GE, the discrepancy ratios are used to monitor the
discrepancy of their contribution to the learning objective, which exactly
fits our definition of the adjustment weight $\zeta$. So, naturally, we
propose the calculation of the discrepancy ratio without decoupling the fused
features as

$\vspace{-0.2cm}\rho_{t}^{v}=\frac{\sum_{i}\zeta_{i}}{\sum_{i}1-\zeta_{i}}.$
(11)

$\rho_{t}^{a}$ is accordingly defined as the reciprocal of $\rho_{t}^{v}$. The
rest of the propagation remains the same as for the OGM-GE. To analyze the
efficiency of the two methods, we performed a correlation analysis of the
discrepancy ratios obtained from the two different calculation methods during
the ablation experiments. It was found that the values from both methods are
indeed strongly correlated. So, avoiding the accumulation of decoupling errors
brings optimization to gradient modulation.

##  3 Experiment

Two mainstream audio-visual classification datasets were used to ensure the
method’s effectiveness. CREMA-D [21] is an audio-visual dataset utilizing face
and voice data to classify six basic emotional states, which contain 7,442
video clips of 2-3 seconds from 91 actors speaking several short words. This
dataset consists of the 6 most usual emotions. VGGSound [22] is a large-scale
video dataset that contains 309 classes, covering a wide range of audio events
in everyday life. The duration of each video is 10 seconds, and the division
of the dataset is the same as proposed in [22]. For CREMA-D and VGGSound, we
employ the exact same data-preprocessing, overall backbones, and losses as
OGM-GE [7] for Table.1. As for additional experiments, we insert our module
into 3 representative frameworks AGVA [26] and PSP [27] on AVE dataset [26],
and FOP[24] on Seen-Heard/Unseen-Unheard dataset [28] with adjustment of very
few parameters to explore the impacts.

For each task, the parameter $s$ in Eq.7 is roughly set to around $2.5$
without elaborate searching. And all the experiments are conducted on 4 NVIDIA
3090Ti GPUs.

##  4 Result and Analysis

Table 1:  Performance of multimodal classification

Method | Fusion | CREMA-D | VGGSound
---|---|---|---
Top1-Acc(%) | mAP(%) | Top1-Acc(%) | mAP(%)
Baseline | Concatenation$\dagger$ | 51.7 | 53.5 | 49.1 | 52.5
Summation | 51.5 | 53.5 | 49.1 | 52.4
FiLM | 50.6 | 52.1 | 48.5 | 51.6
+SimAM2 | Concatenation$\dagger$ | 51.3 | 53.0 | 49.2 | 52.7
Summation | 54.3 | 55.7 | 49.3 | 52.9
FiLM | 52.4 | 54.1 | 48.9 | 51.2
+OGM-GE[7] | Concatenation$\dagger$ | 61.9 | 63.9 | 50.6 | 53.9
Summation | 62.2 | 64.3 | 50.4 | 53.6
FiLM | 55.6 | 57.4 | 50.0 | 52.9
+OGM-GE +SimAM2 | Concatenation$\dagger$ | 62.1 | 64.0 | 51.0 | 54.2
Summation | 63.7 | 65.7 | 51.4 | 54.0
FiLM | 58.6 | 60.3 | 50.1 | 53.1
+Decouping-Free +SimAM2* | Concatenation$\dagger$ | 60.1 | 61.7 | 50.1 | 53.3
Summation | 63.9 | 66.1 | 51.6 | 54.4
FiLM | 59.8 | 61.4 | 50.4 |  53.5

  * *

To make clear comparison, learnable $\zeta$ is only applied to “+Decouping-
Free + SimAM2” experiment.

  * $\dagger$

The experiments on concatenation are only used to show the effect on the
results in the most extreme scenario.

Comparison of Classification: SimAM2 based on Sec.2.2 was applied directly to
the CREMA-D and VGGSound, and the improvement of 2% on average in accuracy was
obtained under the conditions of two different vanilla fusions. We first
conduct experiments on the baseline where the two modalities are directly co-
trained. The same problem of the unbalanced problem as described in [7] also
occurs in the baseline training process. So, likewise, OGM-GE is introduced to
equilibrium convergence to explore more comparisons. Finally, it is observed
in the experiment that the best results are obtained if the decoupling-free
strategy is applied along with SimAM2. The comparison of the three vanilla
fusion experiments on the CREMA-D and VGGSound shows that SimAM2 can indeed
introduce more robustness in the fusion process. Clearly, since the design of
SimAM2 is based on signal superposition theory, it can be observed that SimAM2
has a better performance gain in terms of direct summation. However, SimAM2 is
less of a boost for concatenation, due to the fact that the concatenation
method is essentially a simple summation of two strictly orthogonal signals as
described in Sec.2.2.

Comparison of decoupling-free gradient modulation: As mentioned above, for the
orthogonal reasons described in Sec.2.2, the discrepancy ratio obtained using
decoupling-free is not accurate when using the concatenation method for
fusion. However, it is worth mentioning that the concatenation method is the
easiest method in OGM-GE to decouple. So we prefer to keep the decoupling
method of OGM-GE in the concatenation method, but the remaining two fusion
methods still demonstrate stronger robustness in the case of decoupling-free
methods. In order to further analyze the relationship between the discrepancy
ratios obtained by the two methods, we calculate and record the two ratios at
the same time and observe a clear correlation. Again, it can be concluded from
the experimental results that the non-decoupled method is an optimization of
the decoupled method.

Table 2:  Performance of Audio-visual Event Localization

Method | Network | Acc(%)
---|---|---
baseline | AVGA[26] | 72.0
PSP[27] | 76.2
+SimAM2 | AVGA | 72.4
PSP | 76.7
+OGM-GE | AVGA | 72.8
PSP | 76.9
+SimAM2 +Decoupling-Free | AVGA | 73.0
PSP | 77.2

Comparison of fusion in other applications: In order to further validate the
versatility of SimAM2 in more general scenarios, the fusion of SimAM2 is
applied in multiple tasks based on the idea of plug-and-play. Firstly,
applying the SimAM2 method to audio-visual event localization [26, 27] in late
fusion, it is observed that general performance gains are achieved regardless
of the network used in Table.2. Another interesting experiment is on FOP [24]
in the face-voice association. This simple modification of fusion also seems
to yield slightly better results on unseen-unheard and seen-heard datasets
[28] in Table.3 as well. So, the improvement brought by the SimAM2 fusion
method demonstrates that our method still generally brings consistent
performance for fusion.

Table 3:  Performance of Face-Voice Association

Methods | Seen-Heard | Unseen-Unheard
---|---|---
EER(%) | AUC(%) | EER(%) | AUC(%)
FOP[24] | 19.3 | 89.3 | 24.9 | 83.5
FOP+SimAM2* | 18.5 | 89.8 | 24.6 | 83.7

  * *

Learnable $\zeta$ has already been applied to original FOP.

##  5 conculsion

In this paper, a novel attention method based on informatics energy theory is
reasoned. Then based on this approach, the vanilla fusion is improved along
with the decoupling-free gradient modulation scheme, allowing a general
improvement of multimodal tasks on different datasets and methods. In the
future, provable fusion based on informatics theory needs to be discovered.

## References

  * [1] Shentong Mo and Yapeng Tian,  “Audio-visual grouping network for sound localization from mixtures,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 10565–10574.
  * [2] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun,  “Attention bottlenecks for multimodal fusion,”  Advances in Neural Information Processing Systems, vol. 34, pp. 14200–14213, 2021.
  * [3] Yanmin Qian, Zhengyang Chen, and Shuai Wang,  “Audio-visual deep neural network for robust person verification,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1079–1092, 2021.
  * [4] Zhengyang Chen, Shuai Wang, and Yanmin Qian,  “Multi-modality matters: A performance leap on voxceleb.,”  in INTERSPEECH, 2020, pp. 2252–2256.
  * [5] Yake Wei, Di Hu, Yapeng Tian, and Xuelong Li,  “Learning in audio-visual context: A review, analysis, and new perspective,”  arXiv preprint arXiv:2208.09579, 2022.
  * [6] Weiyao Wang, Du Tran, and Matt Feiszli,  “What makes training multi-modal classification networks hard?,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 12695–12705.
  * [7] Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu,  “Balanced multimodal learning via on-the-fly gradient modulation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 8238–8247.
  * [8] Ruize Xu, Ruoxuan Feng, Shi-Xiong Zhang, and Di Hu,  “Mmcosine: Multi-modal cosine loss towards balanced audio-visual fine-grained learning,”  in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  * [9] Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer,  “Audiovisual slowfast networks for video recognition,”  arXiv preprint arXiv:2001.08740, 2020.
  * [10] Zhongkai Sun, Prathusha Sarma, William Sethares, and Yingyu Liang,  “Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2020, vol. 34, pp. 8992–8999.
  * [11] Xiaobin Chang, Tao Xiang, and Timothy M Hospedales,  “Scalable and effective deep cca via soft decorrelation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 1488–1497.
  * [12] Lichen Wang, Jiaxiang Wu, Shao-Lun Huang, Lizhong Zheng, Xiangxiang Xu, Lin Zhang, and Junzhou Huang,  “An efficient approach to informative feature extraction from multimodal data,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2019, vol. 33, pp. 5281–5288.
  * [13] Joshua Lee, Yuheng Bu, Prasanna Sattigeri, Rameswar Panda, Gregory Wornell, Leonid Karlinsky, and Rogerio Feris,  “A maximal correlation approach to imposing fairness in machine learning,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 3523–3527.
  * [14] Yihua Liang, Fei Ma, Yang Li, and Shao-Lun Huang,  “Person recognition with hgr maximal correlation on multimodal data,”  in 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021, pp. 1–8.
  * [15] Barbara G Shinn-Cunningham,  “Object-based auditory and visual attention,”  Trends in cognitive sciences, vol. 12, no. 5, pp. 182–186, 2008\.
  * [16] Marisa Carrasco,  “Visual attention: The past 25 years,”  Vision research, vol. 51, no. 13, pp. 1484–1525, 2011.
  * [17] Meng-Hao Guo, Tian-Xing Xu, Jiang-Jiang Liu, Zheng-Ning Liu, Peng-Tao Jiang, Tai-Jiang Mu, Song-Hai Zhang, Ralph R Martin, Ming-Ming Cheng, and Shi-Min Hu,  “Attention mechanisms in computer vision: A survey,”  Computational Visual Media, vol. 8, no. 3, pp. 331–368, 2022.
  * [18] Lingxiao Yang, Ru-Yuan Zhang, Lida Li, and Xiaohua Xie,  “Simam: A simple, parameter-free attention module for convolutional neural networks,”  in International conference on machine learning, 2021, pp. 11863–11874.
  * [19] Xiaoyi Qin, Na Li, Chao Weng, Dan Su, and Ming Li,  “Simple attention module based speaker verification with iterative noisy label detection,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6722–6726.
  * [20] Qingyang Zhang, Haitao Wu, Changqing Zhang, Qinghua Hu, Huazhu Fu, Joey Tianyi Zhou, and Xi Peng,  “Provable dynamic fusion for low-quality multimodal data,”  in International conference on machine learning, 2023, pp. 23318–23340.
  * [21] Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma,  “Crema-d: Crowd-sourced emotional multimodal actors dataset,”  IEEE transactions on affective computing, vol. 5, no. 4, pp. 377–390, 2014.
  * [22] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman,  “Vggsound: A large-scale audio-visual dataset,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 721–725.
  * [23] E.R. Kandel,  Principles of Neural Science, Fifth Edition,  McGraw-Hill’s AccessMedicine. McGraw-Hill Education, 2013.
  * [24] Muhammad Saad Saeed, Muhammad Haris Khan, Shah Nawaz, Muhammad Haroon Yousaf, and Alessio Del Bue,  “Fusion and orthogonal projection for improved face-voice association,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 7057–7061.
  * [25] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville,  “Film: Visual reasoning with a general conditioning layer,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2018, vol. 32.
  * [26] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu,  “Audio-visual event localization in unconstrained videos,”  in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 247–263.
  * [27] Jinxing Zhou, Liang Zheng, Yiran Zhong, Shijie Hao, and Meng Wang,  “Positive sample propagation along the audio-visual event line,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 8436–8444.
  * [28] Arsha Nagrani, Samuel Albanie, and Andrew Zisserman,  “Learnable pins: Cross-modal embeddings for person identity,”  in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 71–88.