# Hyperdimensional Computing Nanosystem

Abbas Rahimi<EMAIL_ADDRESS>Tony F. Wu<EMAIL_ADDRESS>Haitong Li
<EMAIL_ADDRESS>Jan M. Rabaey<EMAIL_ADDRESS>H.-S. Philip Wong
<EMAIL_ADDRESS>Max M. Shulaker<EMAIL_ADDRESS>Subhasish Mitra
<EMAIL_ADDRESS>ETH Zurich  UC Berkeley  MIT  Stanford

###### Abstract

One viable solution for continuous reduction in energy-per-operation is to
rethink functionality to cope with uncertainty by adopting computational
approaches that are inherently robust to uncertainty. It requires a novel look
at data representations, associated operations, and circuits, and at materials
and substrates that enable them. 3D integrated nanotechnologies combined with
novel brain-inspired computational paradigms that support fast learning and
fault tolerance could lead the way. Recognizing the very size of the brain’s
circuits, hyperdimensional (HD) computing can model neural activity patterns
with points in a HD space, that is, with _hypervectors_ as large randomly
generated patterns. At its very core, HD computing is about manipulating and
comparing these patterns inside memory. Emerging nanotechnologies such as
carbon nanotube field effect transistors (CNFETs) and resistive RAM (RRAM),
and their monolithic 3D integration offer opportunities for hardware
implementations of HD computing through tight integration of logic and memory,
energy-efficient computation, and unique device characteristics. We
experimentally demonstrate and characterize an end-to-end HD computing
nanosystem built using monolithic 3D integration of CNFETs and RRAM. With our
nanosystem, we experimentally demonstrate classification of 21 languages with
measured accuracy of up to 98% on $>$20,000 sentences (6.4 million
characters), training using one text sample ($\approx$100,000 characters) per
language, and resilient operation (98% accuracy) despite 78% hardware errors
in HD representation (outputs stuck at 0 or 1). By exploiting the unique
properties of the underlying nanotechnologies, we show that HD computing, when
implemented with monolithic 3D integration, can be up to 420$\times$ more
energy-efficient while using 25$\times$ less area compared to traditional
silicon CMOS implementations.

††journal: Memristive Devices for Brain-inspired Computing as a book chapter

##  1 Introduction

Over the past six decades, the semiconductor industry has been immensely
successful in providing exponentially increasing computational power at an
ever-reducing cost and energy footprint. Underlying this staggering evolution
is a set of well-defined abstraction layers: starting from robust switching
devices that support a deterministic Boolean algebra, to a scalable and stored
program architecture that is Turing complete and hence capable of tackling
(almost) any computational challenge. Unfortunately, this abstraction chain is
being challenged as scaling continues to nanometer dimensions, as well as by
exciting new applications that must support a myriad of new data types.
Maintaining the current deterministic computational model ultimately puts a
lower bound on the energy scaling that can be obtained, set in place by
fundamental physics that governs the operation, variability and reliability of
the underlying nanoscale devices [1, 2, 3].

At the same time, the nature of computing itself is evolving rapidly: for a
vast number of emerging applications, cognitive functions such as
classification, recognition, and learning are rapidly gaining importance. For
efficient information-extraction, these applications require a fundamental
departure from the traditional von Neumann architecture, where data has to be
transported to the processing unit and back, creating the infamous memory
wall. One of the most promising options for realizing such non-von Neumann
architectures is to exploit beyond silicon materials and substrates that allow
dense and 3D integration of memory and logic. However, such a dense and
layered 3D system increases the risk of failures within the chip, and system
must be fault-tolerant. As has been realized for a long time, this also
resembles the way brain computes. Hence, 3D integrated nanotechnologies
combined with brain-inspired computational paradigms that support fast
learning and fault-tolerance could lead the way [4].

Emerging hyperdimensional (HD) computing [5] is based on the understanding
that brains compute with _patterns of neural activity_ that are not readily
associated with scalar numbers. In fact, the brain’s ability to calculate with
numbers is feeble. However, due to the very size of the brain’s circuits, we
can model neural activity patterns with points of a HD space, that is, with
hypervectors. In this formalism, information is represented in hypervectors as
ultra-wide words. Such hypervectors can then be mathematically manipulated to
not only classify but also to bind, associate, and perform other types of
cognitive operations in a straightforward manner. In addition, these
mathematical operations also ensure that the resulting hypervector is unique
and thus the learning is one-shot or few-shot meaning that object categories
are learned from few examples in a _single pass_ over the training data [6, 7,
8, 9, 10]. Thus HD computing can substantially reduce the number of operations
needed by conventional learning algorithms, thereby providing tremendous
energy savings. Implementation of the HD computing in practical hardware needs
large arrays of non-volatile memory so that the learning is not “forgotten.”
Our approach is therefore focused on potential low-voltage, non-volatile
Resistive Random Access Memory (RRAM) that can be integrated at high density
with logic switches [11]. We further explore potential low-voltage approaches
to logic transistors such as the carbon nanotube field effect transistors
(CNFETs) so that the overall supply voltage requirement and hence the energy
dissipation can be lowered [12, 13].

In this book chapter, we present HD computing nanosystem by efficient
implementation of HD operations using emerging nanoscalable CNFETs and RRAM,
and their monolithic 3D integration. The rest of this book chapter is
organized as follows. In Section 2, we briefly introduce HD computing and
discuss some of its key properties including a well-defined set of arithmetic
operations (Section 2.1), generality and scalability (Section 2.2), robustness
(Section 2.3), and embarrassingly parallel operations (Section 2.4). In
Section 3, we describe an application of HD computing in language recognition,
and show how its operations can be used to solve various learning and
classification tasks. In Section 4, we present the emerging technology for HD
computing and describe how the principal operations can be efficiently
implemented in a 3D integrated architecture. Our experimental results for 3D
architecture regarding robustness and energy efficiency are described in
Section 5.

##  2 Background in HD Computing

The difference between traditional computing and HD computing is apparent in
the elements that the computer computes with. In traditional computing the
elements are Booleans, numbers, and memory pointers. In HD computing they are
multicomponent hypervectors, or tuples, where neither individual component nor
a subset thereof has a specific meaning: a component of a hypervector and the
entire hypervector represent the same thing. Furthermore, the hypervectors are
ultra-wide: the number of components is in the thousands and they are
independent and identically distributed (i.i.d.).

We will demonstrate the idea with a simple example from language [14, 15]. The
task is to identify the language of a sentence from its three-letter sequences
called _trigrams_. We compare the trigram profile of a test sentence to the
trigram profiles of 21 languages and chose the language with the most similar
profile. A profile is essentially a histogram of trigram frequencies in the
text in question.

The standard algorithm for computing the profile—the _baseline_ —scans through
the text and counts the trigrams. The Latin alphabet of 26 letters and the
space give rise to $27^{3}$ = 19,683 possible trigrams, and so we can
accumulate the trigram counts into a 19,683-dimensional vector and compare
such vectors to find the language with the most similar profile. This is
straightforward and simple with trigrams but it gets complicated with higher-
order $n$-grams when the number of possible $n$-grams grows into the millions
(the number of possible pentagrams is $27^{5}$ = 14,348,907). The standard
algorithm generalizes poorly.

The HD algorithm starts by choosing a set of 27 letter hypervectors at random.
They serve as _seed hypervectors,_ and the same seeds are used with all
training and test data. We have used 10,000-dimensional hypervectors of
equally probable $0$s and $1$s as seeds (aka binary spatter coding [16]). From
these we make trigram hypervectors by _rotating_ the first letter hypervector
twice, the second letter hypervector once, and use the third letter
hypervector as is, and then by _multiplying_ the three hypervectors component
by component. Such trigram hypervectors resemble the seed hypervectors in that
they are 10,000-$D$ with equally probable $1$s and $0$s, and they are random
relative to each other. A text’s profile is then the sum of all the trigrams
in the text: for each occurrence of a trigram in the text, we _add_ its
hypervector into the profile hypervector. The profile of a test sentence is
then compared to the language profiles and the most similar one is returned as
the system’s answer, as above. In contrast to the standard algorithm, the HD
algorithm generalizes readily to any $n$-gram size: the hypervectors remain
10,000-$D$.

###  2.1 Arithmetic Operations on Hypervectors

HD computing is based on the properties of hypervectors and operations on
them. We will review them with reference to $D$-bit hypervectors, where
$D$=10,000 for example [17]. There are $2^{D}$ such hypervectors, also called
_points_ , and they correspond to the corners of a $D$-dimensional unit cube.
The number of places at which two binary hypervectors differ is called the
_Hamming distance_ and it provides a measure of _similarity_ between
hypervectors. A peculiar property of HD spaces is that most points are
relatively far from any given point. Hence two $D$-bit hypervectors chosen at
random are dissimilar with near certainty: when referenced from the center of
the cube they are nearly _orthogonal_ to each other.

To combine hypervectors, HD computing uses three operations [5]: _addition_
(which can be weighted), _multiplication_ , and _permutation_ (more generally,
multiplication by a matrix). “Addition” and “multiplication” are meant in the
abstract algebra sense where the sum of binary vectors $[A+B+\ldots]$ is
defined as the componentwise majority function with ties broken at random, the
product is defined as the componentwise `XOR` (addition modulo 2 denoted by
$\mathbin{\oplus}$), and permutation ($\rho$) shuffles the components. All
these operations produce a $D$-bit hypervector, and we collectively call them
as Multiply-Add-Permute (MAP) operations [18].

The usefulness of HD computing comes from the nature of the MAP operations.
Specifically, addition produces a hypervector that is _similar_ to the
argument hypervectors—the inputs—whereas multiplication and random permutation
produce a _dissimilar_ hypervector; multiplication and permutation are
_invertible_ , addition is approximately invertible; multiplication
_distributes_ over addition; permutation distributes over both multiplication
and addition; multiplication and permutation _preserve similarity_ , meaning
that two similar hypervectors are mapped to equally similar hypervectors
elsewhere in the space.

Operations on hypervectors can produce results that are approximate or “noisy”
and need to be identified with the exact hypervectors. For that, we maintain a
list of known (noise-free) seed hypervectors in a so-called _item memory_ or
_clean-up memory._ When presented with a noisy hypervector, the item memory
outputs the most similar stored hypervector. High dimensionality is crucial to
make that work reliably [17]. With 10,000-bit hypervectors, $1/3$ of the bits
can be flipped at random and the resulting hypervector can still be identified
with the original stored one.

These operations make it possible to encode/decode and manipulate sets,
sequences, and lists—in essence, any data structure. Such packing and
unpacking operations are then viewed as mappings between points of the space
suggesting a mechanism for analogy, with the analogy mapping being computed
from examples. This enables to implement analogical reasoning to answer non-
trivial queries, e.g., “What’s the Dollar of Mexico?” [19].

Figure 1: An example of encoding and decoding of a data structure using HD
computing.

Figure 1 shows how a data record consisting of variables $x,y,z$ with values
$a,b,c$ can be encoded into a hypervector $H$ and the value of $x$ can be
extracted from it. We start with randomly chosen seed hypervectors
$X,Y,Z,A,B,C$ for the variable and the values and store them in the item
memory. We then encode the record by _binding_ the variables to their values
with multiplication and by adding together the bound pairs:

$H=[(X\oplus A)+(Y\oplus B)+(Z\oplus C)]$

To find the value of $x$ in $H$ we multiply it with the inverse of $X$, which
for `XOR` is $X$ itself: $A^{\prime}=X\oplus H$. The resulting hypervector
$A^{\prime}$ is given to the item memory which returns $A$ as the most-similar
stored hypervector. An analysis of this example would show how the properties
of the operations, as listed above, come to play. A thing to note about the
operations is that addition and multiplication approximate an algebraic
structure called a field, to which permutation gives further expressive power.

HD computing has been described above in terms of binary hypervectors.
However, the key properties are shared by hypervectors of many kinds, all of
which can serve as the computational infrastructure. They include Holographic
Reduced Representations (HRR) [20], Frequency-domain Holographic Reduced
Representations (FHRR) [20], Binary Spatter Codes (BSC) [16], Multiply-Add-
Permute (MAP) coding [18], Binary Sparse Distributed Codes (BSDC) [21], Matrix
Binding of Additive Terms (MBAT) [22], and Geometric Analogue of Holographic
Reduced Representations (GAHRR) [23]. Different representational schemes using
high-dimensional vectors and operations on them are generally referred to as
Vector Symbolic Architectures (VSA) [24] and the ultrahigh dimensionality is
referred to as _hyperdimensional_ [5].

###  2.2 General and Scalable Model of Computing

(a) Text analytics

(b) EMG signals

Figure 2: General and scalable HD computing for various cognitive tasks: (a) European languages recognition; (b) EMG-based hand gesture recognition. Table 1: List of applications based on HD computing with different number of inputs and classes. The last two columns compare the classification accuracy of HD versus the baseline in that application domain. Applications | Inputs (#) | Classes (#) | HD (%) | Baseline (%)
---|---|---|---|---
Language recognition [14, 15] | 1 | 21 | 96.7% | 97.9%
Text categorization [25] | 1 | 8 | 94.2% | 86.4%
Speech recognition [26] | 1 | 26 | 95.3% | 93.6%
EMG gesture recognition [6] | 4 | 5 | 97.8% | 89.7%
Flexible EMG gesture recognition [7] | 64 | 5 | 96.6% | 88.9%
EEG brain-machine interface [9, 10] | 64 | 2 | 74.5% | 69.5%
ECoG seizure detection [8] | 100 | 2 | 95.4% | 94.3%

HD computing is a complete computational paradigm that is easily applied to
learning problems. Its main difference from other paradigms is that it can
operate with data represented as approximate patterns, allowing it to scale to
large learning applications. HD computing has been used commercially since
2008 for making semantic vectors for words—semantic vectors have the property
that words with similar meaning are represented by similar vectors. The Random
Indexing (RI) [27] algorithm for making semantic hypervectors was developed as
an alternative to Latent Semantic Analysis (LSA) [28], which relies on
compute-heavy Singular Value Decomposition (SVD). The original experiment used
37,000 “documents” on 7 topics to compute 8,000-dimensional semantic
hypervectors of equal quality for 54,000 words. SDV-based LSA requires memory
in proportion to the product: ‘size of vocabulary’ $\times$ ‘number of
documents’. By contrast, RI requires memory in proportion to the size of the
vocabulary, and the statistics of documents/contexts is learned through simple
vector addition [27]. Thus, the complexity of the method grows linearly with
the size of the training corpus and scales easily to millions of documents.

Multiplication and permutation make it possible to encode causal relations and
grammar into these hypervectors, thereby capturing more and more of the
meaning in language [14, 29]. We have used HD computing successfully to
identify the language of test sentences, as described at the beginning of this
section [14, 15] (also with sparse hypervectors [30]), to categorize News
articles [25], and to classify DNA [31]; other applications to text include
common substrings search [32] and recognition of permuted words [33]. HD
computing has been also used in speech recognition [34, 26]. All these
applications have a single input stream (Figure 2(a)), while HD computing
provides a natural fit for applications with multiple sensory inputs, e.g.,
biosignal processing (Figure 2(b)). For instance, we have adapted the
architecture for text analytics to the classification of hand gestures, when
analog electromyography (EMG) signals are recorded simultaneously by four
sensors [6, 35, 36] or even a larger flexible electrode array [7]. The
template architecture is shown in Figure 2. The architecture was further
extended to operate on electroencephalography (EEG) [9, 10], and
electrocorticography (ECoG) [8] with up to 100 electrodes.

Notably, the learning and classification tasks are performed on the same
hardware construct by integrating the following three main steps: 1) mapping
to the HD space, 2) encoding with the MAP operations, and 3) associative
memory (see Figure 2). The only difference is that during training, the
associative memory updates the learned patterns with new hypervectors, while
during classification it computes distances between a query hypervector and
learned patterns. Hence, it is possible to build a general-purpose
computational engine based on these operations to cover a variety of tasks
with similar success rates. We show later in Section 5, how such computational
engine can be efficiently realized by using emerging nanotechnologies. In
addition, since the same hardware is used for learning and classification, the
architecture is ideal for incremental or online learning.

###  2.3 Robustness of Computations

HD computing is extremely robust. Its tolerance for low-precision and faulty
components is achieved by brain-inspired properties of hypervectors:
(pseudo)randomness, high-dimensionality, and fully distributed holographic
representations. Symbols represented with hypervectors begin with i.i.d.
components and when combined with the MAP operations, the resulting
hypervectors also appear as identically distributed random hypervectors, and
the independence of the individual components is mostly preserved. This means
that a failure in a component of a hypervectors is not “contagious.” At the
same time, failures in a subset of components are compensated for by the
holographic nature of the data representation, i.e., the error-free components
can still provide a useful representation that is similar enough to the
original hypervector. This inherent robustness eliminates the need for
asymmetric error protection in memory units. This makes HD data representation
suited for operation at low signal-to-noise ratios (SNR).

###  2.4 Memory-centric with Embarrassingly Parallel Operations

At its very core, HD computing is about manipulating and comparing large
patterns within the memory itself. The MAP operations allow a high degree of
parallelism by needing to communicate with only a local component or its
immediate neighbors. Other operations such as the distance computation can be
performed in a distributed fashion [37]. This is a fundamental difference from
traditional computational architectures, where data has to be transported to
the processing unit and back, creating the infamous memory wall. In HD
processing, logic is tightly integrated with the memory and all computations
are fully distributed. This translates into substantial energy savings, as
global interconnects are accessed at a relatively low frequency.

##  3 Case Study: Language Recognition

Figure 3: A 2D architecture of HD computing for language recognition task (see
also Figure 2(a)): mapping and encoding module, and search module.

As a concrete application of HD computing, let us look at an implementation of
the language recognition algorithm discussed in Section 2. The HD algorithm
generates trigram profiles as hypervectors and compares them for similarity.
As shown in Figure 3, the design is based on a memory-centric architecture
where logic is tightly integrated with the memory and all computations are
fully distributed. The HD architecture has two main parts: mapping and
encoding modules, and similarity search module (associative memory). The
mapping and encoding module embeds an input text, composed of a stream of
letters, to a hypervector in the HD space. Then this hypervector is broadcast
to the similarity-search module for comparison with a set of _learned_
language hypervectors. Finally, the search module returns the language that
has the closest match based on Hamming distance similarity.

###  3.1 Mapping and Encoding Module

This module accepts the text as a stream of letters and computes its
representation as a hypervector. The module has an item memory that holds a
random hypervector (the “letter” hypervector) for each of the 26 letters and
the space. The item memory is implemented as a lookup table that remains
constant. In the dense binary coding [16], a letter hypervector has an
approximately equal number of randomly placed $1$s and $0$s, hence the 27
hypervectors are approximately orthogonal to each other. As another
alternative, mapping to binary hypervectors can be realized by
_rematerialization_ [38] e.g., by using a cellular automaton exhibiting
exhibiting chaotic behaviour [39].

The module computes a hypervector for each block of 3 consecutive letters as
the text streams in. It consists of 3 stages in FIFO style, each of which
stores a letter hypervector. A trigram hypervector is created by successively
permuting the letter vectors based on their order and binding them together,
which creates a unique representation for each unique sequence of three
letters. For example, the trigram “abc” is represented by the hypervector
$\rho(\rho(A)\oplus B)\oplus C=\rho(\rho(A))\oplus\rho(B)\oplus C$. Use of
permutation and binding distinguishes the sequence “abc” from “acb”, since a
permuted hypervector is uncorrelated with all the other hypervectors.

The random permutation operation $\rho$ is fixed and is implemented as a
rotation to right by 1 position as shown in Figure 3. For instance, given the
trigram “abc”, the $A$ hypervector is rotated twice ($\rho(\rho(A))$), the $B$
hypervector is rotated once ($\rho(B)$), and there is no rotation for the $C$
hypervector. Once “c” is reached, its corresponding $C$ hypervector is fetched
from the item memory and is written directly to the first stage of the encoder
(i.e., Letter3 hypervector in Figure 3). The two previous letters are rotated
as they pass through the encoder and turn into $\rho\rho(A)$ and $\rho(B)$.
Componentwise bindings (i.e., multiplication) are then applied between these
three hypervectors to compute the trigram hypervector, i.e.,
$\rho\rho(A)\oplus\rho(B)\oplus C$. Since the trigram hypervector is binary,
the binding between two hypervectors is implemented with $D$ `XOR` gates.

The hypervector for the input text is computed by adding together the
hypervectors for all the trigrams in the text and by applying a threshold to
retain them in the binary space. An input text of length $k+2$ generates $k$
trigram vectors. We implement the componentwise addition with a set of $D$
accumulators (ACC in Figure 3), one for each dimension of the hypervector, and
count the number of $1$s in that component location. This componentwise
accumulation produces a $D$-dimensional hypervector of integers. To compute
the corresponding binary hypervector, the encoding module applies a threshold
of $k/2$ (implementing the majority function ($k$, $k/2$)) to every
accumulator value, where $k$ is the number of trigrams accumulated from the
input. Left side of Figure 3 shows such a dedicated accumulation and
thresholding for every hypervector component. The output of the module is the
binary text hypervector.

The mapping and encoding module is used for both training and testing. During
training when the language of the input text in known, we refer to the text
hypervector as a _language_ hypervector. Such language hypervectors are stored
in the search module as learned patterns. When the language of a text is
unknown, as it is during testing, we call the text hypervector a _query_
hypervector. The query hypervector is sent to the similarity search module to
identify its source language.

###  3.2 Similarity Search Module

The search module stores a set of language hypervectors that are precomputed
by the mapping and encoding module. These language hypervectors are formed in
exactly the same way as described above, by making the text hypervectors from
samples of a known language. Therefore, during the training phase, we feed
texts of a known language to the mapping and encoding module and save the
resulting text hypervector as a language hypervector in the search module. We
consider 21 European languages and at the end of training have 21 language
hypervectors, each stored in its own row of the search module.

The language of an unknown text is determined by comparing its query
hypervector to all the language hypervectors. This comparison is done in a
distributed fashion using an associative memory, and with the Hamming distance
as the similarity function. Hamming distance counts the number of components
at which two binary hypervectors disagree. The module uses a set of $D$ `XOR`
gates to identify mismatches between the two hypervectors. In this digital
implementation, the similarity-measurement block compares only one component
each clock cycle. Hence, it takes $O(D)$ cycles to compute the Hamming
distance between the two hypervectors [15]. This block is replicated 21 times
(the number of languages in our application) within the search module as shown
in Figure 3. The query hypervector is broadcast across the search module,
hence all the similarity-measurement blocks compute their distance
concurrently. Finally, a combinational comparison block selects the minimum
Hamming distance and returns its associated language as the language that the
unknown text has been written in.

##  4 Emerging Technologies for HD Computing

Several emerging nanotechnologies such as carbon nanotube field-effect
transistors, resistive RAM, and monolithic 3D integration have been
demonstrated to be particularly effective for implementation of HD computing
as well as other computing paradigms. One or more of these technologies has
been used in demonstrating HD computing operations [11] as well as in full
system demonstrations [12, 13]. Here, we introduce these technologies.

###  4.1 Carbon Nanotube Field-Effect Transistors

Carbon nanotube field-effect transistors (CNFETs) are an emerging transistor
technology which promises an order of magnitude improvement in energy-delay-
product (a metric of energy efficiency) for digital circuits [40]. CNFETs use
multiple carbon nanotubes (CNTs), which are cylindrical structures of carbon
atoms 1-2 nm in diameter, that act as channels. CNTs enable highly energy-
efficient digital logic circuits due to their high carrier mobility and
excellent electrostatic control in CNFETs [41]. High-performance complementary
logic has been demonstrated using CNFETs with an
I${}_{\textup{on}}$/I${}_{\textup{off}}$ ratio (i.e., the ratio of drive
current to the off-state leakage current) of about $10^{6}$ [42, 43]. They can
be built at scaled gate lengths (5 nm) [44] and without hysteresis [45].
CNFETs have even been fabricated as negative capacitance FETs with sub-55
mV/decade subthreshold swing at room temperature [46]. CNFETs can be
fabricated at low temperature ($\leq$ 250∘C), which is key to enabling
monolithic 3D integration (discussed in later in this chapter).

When designing circuits with CNFETs, imperfections inherent in CNTs, such as
mis-positioned CNTs (that can lead to stray conducting paths resulting in
incorrect functionality) and metallic CNTs (i.e., CNTs with little or no
bandgap), can be overcome using the imperfection-immune paradigm. The
imperfection-immune paradigm uses a combination of fabrication and design
techniques [47, 48, 49] to enable wafer-scale fabrication and VLSI-compatible
design of CNFET circuits. This paradigm has enabled experimental
demonstrations such as the first CNT computer, the first 3D nanosystem
consisting of over 2 million CNFETs on a single die, and the first full-system
demonstration of an HD computing nanosystem [50, 51, 12].

In addition to process variations that exist in silicon transistors (e.g.,
variations in threshold voltage, channel length, and oxide thickness), CNFETs
are subject to CNT-specific variations such as CNT count variations (i.e.,
variations in the number of CNTs in a CNFET). These variations cause drive
current variations, which can manifest as delay variations in digital
circuits. These variations can be suppressed using optimized process and
circuit design. However, these inherent variations can be utilized in HD
computing to generate the seed hypervectors, as demonstrated in [12] by
capitalizing on the variations in CNT count and threshold voltage. This means
that variability and randomness essentially become the sources for
computation.

###  4.2 Resistive RAM

Resistive RAM (RRAM) is an emerging memory technology that promises high
capacity, non-volatile data storage (10-year retention), and can be fabricated
at low temperature ($\leq$ 250∘C) [52, 53]. RRAM is fabricated as a metal
oxide switching layer (insulator) sandwiched between two metallic electrodes
and can be realized using various metal-insulator-metal material combinations.

Three main operations are typically performed on an RRAM cell: set, reset, and
read. The set operation transforms the cell from high-resistance state (HRS)
to low-resistance state (LRS) by applying a positive voltage (i.e., set
voltage) across the top and bottom electrodes [52]. A transistor is typically
used to limit the current for the set operation (called the compliance
current). This creates or lengthens a filament of oxygen vacancies from the
bottom electrode to the top electrode. As the length of the conductive
filament increases, the resistance of the RRAM decreases [52]. In most cases,
a higher set voltage (called forming voltage) is applied to form the filament
(once) after fabrication. However, forming-less RRAM cells have also been
demonstrated [52, 54]. A reset operation transforms the cell from LRS to HRS
by applying a negative voltage (i.e., reset voltage) across the top and bottom
electrodes, rupturing the filaments between the electrodes. RRAM cells with
$\leq$2 V set/reset voltage ($\approx$10 ns pulse duration) and 10-100 HRS/LRS
resistance ratio have been demonstrated [52, 55]. RRAM is also subject to
variations in its resistance, stemming from the stochastic size and shape of
the conductive filament after a set or reset operation. These variations can
also be exploited to generate seed hypervectors discussed in Section 5.1. A
read operation detects the state of the cell (e.g., HRS or LRS) by sensing the
current after applying a small voltage across the two electrodes. This voltage
is small enough to not change the resistance of the cell. Although RRAM has
limited write (i.e., set/reset) endurance ($10^{12}$ cycles at the cell level
[55] and $10^{5}$-$10^{7}$ cycles at the array level [56, 57]), HD computing
is shown to be robust against such endurance-related errors (see Section 5.1).

Many cell structures (e.g., 1 transistor-1 RRAM cell, 1 transistor-n RRAM
cell, 1 selector-1 RRAM cell [52]) may be used for RRAM, with each structure
exhibiting a trade of between cell density (i.e., the number of cells that can
be placed in a given area) and the controlability (of the resistance during
set or reset operations) or the ability to detect the state of the cell
reliably. For example, the 1 transistor-1 RRAM (1T-1R) cell configuration can
be effectively used to prevent current overshoot during the set operation and
provide exceptional selectivity between cells during the read operation but
has limited cell density due to each cell using a transistor (typically larger
than the RRAM cell itself) [52]. Array-level implementations using the 1T-1R
RRAM structure have been demonstrated up to 16 Gbits of capacity [58].
Moreover, RRAM can be vertically built (3D VRRAM) in a bit-cost scalable
manner to improve the cell density [59]. HD computing demonstrations have used
both the 1T-1R [12] configuration as well as 3D VRRAM [11].

A single RRAM cell can store a single bit or multiple bits [60]. To
demonstrate multi-bit storage in RRAM cells, one or a combination of set or
reset parameters are adjusted to change the resistance of the cell to an
intermediate value (between LRS and HRS): compliance current in the set
operation, reset voltage, and set or reset pulse duration. These parameters
can also be adjusted to gradually increase the RRAM cell resistance (i.e.,
increasing the resistance incrementally). This gradual increase in RRAM cell
resistance has been demonstrated for a variety of switching layers (i.e., the
material in which the filament forms, such as HfOX [61]) by using short pulses
during the reset operation. This behavior, called _gradual reset_ , can be
employed to realize addition operations in hardware [12].

The RRAM has been demonstrated as digital storage, and as incrementers using
gradual reset (i.e., the ability to increment the RRAM resistance in a fine-
grained manner), and for performing the bitwise operations necessary for HD
computing [12, 11].

###  4.3 Monolithic 3D Integration

Monolithic 3D integration is a process whereby tiers of circuits (i.e., a
layer of logic, memory, or sensors) are fabricated on top of each other on the
same substrate. Monolithic 3D integration uses inter-layer vias (ILVs),
standard vias used to connect adjacent metal layers in the interconnect stack
of today’s silicon CMOS technologies, to connect between tiers of circuits.
This is in contrast to chip stacking using through-silicon vias (TSVs) with
typical pitches of around 10 $\mu$m [62]. ILVs can have the same pitch as
metal interconnects (100 nm at 28 nm technology node [63]), enabling
significantly denser vertical connectivity compared to TSVs [64]—a key to
tight integration between logic and memory.

Monolithic 3D integration requires low temperature fabrication for upper tiers
of circuits ($\leq$400∘C) as higher temperatures can damage existing circuits
(transistors and interconnects) on the bottom tiers. While this is difficult
for traditional silicon CMOS technologies (e.g., high temperature requirements
for dopant activation $\geq$1,000∘C), it is naturally enabled by CNFETs and
RRAM due to their low temperature fabrication [52, 65]. In recent
demonstrations of HD computing, all CNFETs and RRAM were fabricated with a
maximum temperature of 200∘C. Monolithic 3D integration of CNFETs, RRAM, and
silicon transistors has been shown [42], demonstrating compatibility with
silicon CMOS. HD computing has been shown to provide up to 35$\times$ energy-
execution time product benefits when implemented using such monolithically
integrated CNFETs and RRAM compared to the standard silicon CMOS approach
[13].

##  5 Experimental Demonstrations for HD Computing

In this section, we describe several experimental hardware demonstrations for
HD computing, including in-memory MAP kernels using 3D VRRAM [11] and an end-
to-end HD system with CNFETs and RRAM using monolithic 3D integration [12,
13]. Electrical characterization results and system simulations will be
discussed to provide insights into how emerging device technologies can be
utilized towards efficient implementations of HD computing systems.

Figure 4: In-memory MAP kernels utilizing the unique 3D structure and non-
volatility of 3D VRRAM. Electrical measurement data are collected from 4-layer
3D VRRAM devices.

###  5.1 3D VRRAM Demonstration: In-Memory MAP Kernels

Figure 5: Evaluation of the impact of RRAM endurance constraints (switching-
induced stuck-at errors).

One novel approach to realize memory-intensive MAP operations is to directly
construct a set of native MAP kernels within a dense memory array without
moving data around. 3D VRRAM, with multiple layers vertically stacked in a
NAND-like fashion, has been found naturally suitable for the MAP kernel
implementation (see Figure 4). Binary bits in HD representation are
initialized and stored in RRAM cells in a non-volatile fashion, utilizing the
native stochastic switching behaviors of RRAM. With a 50% probability of
switching from high-resistance state (HRS, representing ‘0’) to low-resistance
state (LRS, representing ‘1’), seed hypervectors can be generated. The central
idea of performing MAP logic functions within 3D VRRAM is to utilize the 3D
vertical connectivity and the non-volatility during modified write and read
operations on 3D VRRAM. Figure 4 shows an example of a 4-layer 3D VRRAM (i.e.,
a vertical 1T-4R structure) that is used to demonstrate the MAP operations
[11].

To yield arbitrary Boolean logic functions (including the bit-wise `XOR`),
voltage division between the select transistor and multiple RRAM cells (in
this example, 4 RRAM cells) along a vertical pillar in 3D VRRAM is utilized
during pulsing operations. The voltage drop on certain RRAM cells is a
function of resistance states of the rest of RRAM cells, thus creating a logic
mapping from inputs to outputs. Detailed pulse operation schemes can be found
in [66]. In the following, we describe how this method can be exploited to
implement the MAP operations for HD computing (see also Figure 4):

  * 1.

For multiplication, after one-time logic mapping, the 3D non-volatile memories
serve as an `XOR` look-up table, where bit-wise `XOR` inputs are used to
decode and read out the `XOR` outputs stored in 3D VRRAM cells, eliminating
the need of re-writing any RRAM cell. Up to $10^{12}$ `XOR` evaluation cycles
are performed by electrical measurement without disturb errors in readout
results (Figure 4).

  * 2.

The addition kernel employs the analog-domain current summing property of 3D
VRRAM. Along a vertical pillar which is ‘shared’ by multiple layers of RRAM
cells, current summing can be enabled by performing ‘read’ operations in
parallel on multiple layers. Proof-of-concept 4-bit vectors are written into
multiple vertical pillars in 4-layer 3D VRRAM, and accumulated current during
readout measurements correspond to correct outputs, with up to $10^{11}$
addition cycles measured (Figure 4).

  * 3.

Implementing permutation within 3D VRRAM does not require separate read-out
and write-back processes. Similar to realizing Boolean functions, vertical
connectivity (and voltage division) can be exploited to perform bit-wise data
transfer among RRAM cells. Using pairs of VDD/GND pulses along vertical
pillars, data transfer is implemented to perform permutation for hypervectors
in a sequential fashion.

Since using non-volatile memory cells for both HD data representation and MAP
operations involves write operations, endurance constraint (total number of
write cycles before a hard error is produced) is also evaluated by conducting
simulations on the language recognition task. During the training and
inference on the task dataset, endurance failures (stuck at ‘1’ or ‘0’) may
occur on RRAM cells. Under different levels of endurance characteristics at
the device level, simulations show that certain degree of robustness can be
retained (Figure 5), owing to the robust HD representation. Future work is
needed on more realistic endurance modeling to enable device-system co-
optimization being aware of RRAM endurance trade-offs.

Figure 6: Schematic of Monolithic 3D HD system using CNFETs and RRAM.

###  5.2 System Demonstration using Monolithic 3D Integrated CNFETs and RRAM

HD computing can be realized in hardware using monolithic 3D integration of
CNFETs and RRAM [12, 13] and has demonstrated pairwise classification of 21
languages with measured mean accuracy of up to 98% on $>$20,000 sentences.
Unique properties of RRAM and CNFETs can be exploited to create area- and
energy-efficient monolithic 3D circuit blocks that combine CNFETs with fine-
grained access to RRAM memories (Figure 6):

  1. 1.

Circuits that embrace inherent variations in RRAM and CNFETs, with estimated
3$\times$ lower dynamic energy (vs. silicon CMOS implementations at the same
technology node) stemming from both the circuit topology and the use of
energy-efficient CNFETs.

  2. 2.

Approximate incrementer circuits using gradual RRAM reset operation, which can
use 30$\times$ fewer transistors vs. full-digital incrementer implementations.

  3. 3.

Ternary content addressable memory (TCAM) cells built using pairs of CNFETs
and RRAM, that use 19$\times$ lower energy (simulated vs. SRAM-based TCAM
cells) due to reduced leakage of non-volatile RRAM.

When such a system is implemented at smaller technology nodes (e.g., 28nm
node), it can simultaneously achieve lower energy and faster execution time
compared to conventional silicon CMOS approaches (e.g., 7.6$\times$ lower
energy and 4.6$\times$ faster execution time).

Figure 7: Delay cells exploiting inherent variations of CNFETs and RRAM Figure
8: Approximate incrementer using the gradual reset property of RRAM. All
transistors are CNFETs

To realize an item memory, with randomly generated seeds, to map input letters
to hypervectors, inherent variations in RRAM and CNFETs can be exploited
(Figure 7). Delay cells can be used to translate device-level variations such
as drive current variations resulting from variations in carbon nanotube (CNT)
count (i.e., the number of CNTs in a CNFET) or threshold voltage of CNFETs,
and resistance variations of RRAM to delay variations. To generate
hypervectors, each possible input (26 letters of the alphabet and the space
character) is mapped to a delay from a reference clock edge (time-encoded). To
calculate each bit of the hypervector, random delays are added to both the
input signal and the reference clock using delay cells. If the resulting
signals are coincident (the falling edges are close enough to set a SR latch)
the output is ‘1’. Before training the system, to initialize delay cells, the
RRAM resistance is first reset to a high-resistance state (HRS) and then set
to a low-resistance state (LRS).

To realize circuits to perform addition in hardware, an approximate
incrementer with thresholding which leverages the multiple values of RRAM
resistance that can be programmed by performing a gradual reset is used
(Figure 8). A digital buffer is used to transform (threshold) the sum to a
binary hypervector. Each such approximate incrementer uses 8 transistors and a
single RRAM cell. In contrast, a digital 7-bit incrementer may use 240
transistors. Thus, when $D$ (e.g. 10,000) accumulators are needed, the savings
can be significant.

The search module is implemented using 2T2R (2-CNFET transistor, 2-RRAM)
ternary content-addressable memory (TCAM) cells to form an associative memory
(Figure 6). During training, the matchline (i.e. ml0 or ml1) corresponding to
the language (e.g., ml0 for English and ml1 for Spanish) is set to a high
voltage to write to the RRAM cells (e.g., 3V), writing the query hypervector
into the RRAM cells connected to the matchline. During inference, the
matchlines (i.e. ml0 and ml1) are set to a low voltage (e.g. 0.5 V), and the
current on each matchline is read as an output. When the query hypervector bit
is equal to the value stored in a TCAM cell (match), the current is high.
Otherwise (mismatch), the current is low. Cell currents are summed on each
matchline. The line with the most current corresponds to the output class.

## References

  * [1] W. Dehaene, Sc1: Circuit design in advanced cmos technologies: How to design with lower supply voltages, in: 2015 IEEE International Solid-State Circuits Conference - (ISSCC) Digest of Technical Papers, 2015, pp. 1–2.  doi:10.1109/ISSCC.2015.7063154.
  * [2] S. Williams, E. P. DeBenedictis, OSTP nanotechnology-inspired grand challenge: Sensible machines (extended version 2.5).
  * [3] A. Rahimi, L. Benini, R. K. Gupta, Variability mitigation in nanometer cmos integrated systems: A survey of techniques from circuits to software, Proceedings of the IEEE 104 (7) (2016) 1410–1448.  doi:10.1109/JPROC.2016.2518864.
  * [4] A. Rahimi, S. Datta, D. Kleyko, E. P. Frady, B. Olshausen, P. Kanerva, J. M. Rabaey, High-dimensional computing as a nanoscalable paradigm, IEEE Transactions on Circuits and Systems I: Regular Papers 64 (9) (2017) 2508–2521.  doi:10.1109/TCSI.2017.2705051.
  * [5] P. Kanerva, Hyperdimensional computing: An introduction to computing in distributed representation with high-dimensional random vectors, Cognitive Computation 1 (2) (2009) 139–159.  doi:10.1007/s12559-009-9009-8.
URL http://dx.doi.org/10.1007/s12559-009-9009-8

  * [6] A. Rahimi, S. Benatti, P. Kanerva, L. Benini, J. M. Rabaey, Hyperdimensional biosignal processing: A case study for EMG-based hand gesture recognition, in: IEEE International Conference on Rebooting Computing, 2016.
  * [7] A. Moin, A. Zhou, A. Rahimi, S. Benatti, A. Menon, S. Tamakloe, J. Ting, N. Yamamoto, Y. Khan, F. Burghardt, L. Benini, A. C. Arias, J. M. Rabaey, An EMG gesture recognition system with flexible high-density sensors and brain-inspired high-dimensional classifier, IEEE International Symposium on Circuits and Systems, ISCAS.
  * [8] L. B. A. Burrello, K. Schindler, A. Rahimi, One-shot learning for iEEG seizure detection using end-to-end binary operations: Local binary patterns with hyperdimensional computing, Biomedical Circuits and Systems Conference (BioCAS), 2018 IEEE.
  * [9] A. Rahimi, P. Kanerva, J. del R. Millân, J. M. Rabaey, Hyperdimensional computing for noninvasive brain–computer interfaces: Blind and one-shot classification of EEG error-related potentials, 10th ACM/EAI International Conference on Bio-inspired Information and Communications Technologies (BICT).
  * [10] A. Rahimi, A. Tchouprina, P. Kanerva, J. d. R. Millán, J. M. Rabaey, Hyperdimensional computing for blind and one-shot classification of EEG error-related potentials, Mobile Networks and Applicationsdoi:10.1007/s11036-017-0942-6.
URL https://doi.org/10.1007/s11036-017-0942-6

  * [11] H. Li, T. F. Wu, A. Rahimi, K. S. Li, M. Rusch, C. H. Lin, J. L. Hsu, M. M. Sabry, S. B. Eryilmaz, J. Sohn, W. C. Chiu, M. C. Chen, T. T. Wu, J. M. Shieh, W. K. Yeh, J. M. Rabaey, S. Mitra, H. S. P. Wong, Hyperdimensional computing with 3d vrram in-memory kernels: Device-architecture co-design for energy-efficient, error-resilient language recognition, in: 2016 IEEE International Electron Devices Meeting (IEDM), 2016, pp. 16.1.1–16.1.4.  doi:10.1109/IEDM.2016.7838428.
  * [12] T. F. Wu, H. Li, P. Huang, A. Rahimi, J. M. Rabaey, H. . P. Wong, M. M. Shulaker, S. Mitra, Brain-inspired computing exploiting carbon nanotube fets and resistive ram: Hyperdimensional computing case study, in: 2018 IEEE International Solid - State Circuits Conference - (ISSCC), 2018, pp. 492–494.  doi:10.1109/ISSCC.2018.8310399.
  * [13] T. F. Wu, H. Li, P. Huang, A. Rahimi, G. Hills, B. Hodson, W. Hwang, J. M. Rabaey, H. . P. Wong, M. M. Shulaker, S. Mitra, Hyperdimensional computing exploiting carbon nanotube fets, resistive ram, and their monolithic 3d integration, IEEE Journal of Solid-State Circuits (2018) 1–14doi:10.1109/JSSC.2018.2870560.
  * [14] A. Joshi, J. T. Halseth, P. Kanerva, Language geometry using random indexing, in: J. A. de Barros, B. Coecke, E. Pothos (Eds.), Quantum Interaction: 10th International Conference, QI 2016, San Francisco, CA, USA, July 20-22, 2016, Revised Selected Papers, Springer International Publishing, Cham, 2017, pp. 265–274.  doi:10.1007/978-3-319-52289-0_21.
URL http://dx.doi.org/10.1007/978-3-319-52289-0_21

  * [15] A. Rahimi, P. Kanerva, J. M. Rabaey, A robust and energy efficient classifier using brain-inspired hyperdimensional computing, in: Low Power Electronics and Design (ISLPED), 2016 IEEE/ACM International Symposium on, 2016.
  * [16] P. Kanerva, Binary spatter-coding of ordered k-tuples, in: (Ed.), ICANN’96, Proceedings of the International Conference on Artificial Neural Networks, Vol. 1112 of Lecture Notes in Computer Science, Springer, 1996, pp. 869–873.
  * [17] P. Kanerva, Sparse Distributed Memory, The MIT Press, Cambridge, MA, USA, 1988.
  * [18] R. W. Gayler, Multiplicative binding, representation operators & analogy, in: Gentner, D., Holyoak, K. J., Kokinov, B. N. (Eds.), Advances in analogy research: Integration of theory and data from the cognitive, computational, and neural sciences, New Bulgarian University, Sofia, Bulgaria, 1998, pp. 1–4.
URL http://cogprints.org/502/

  * [19] P. Kanerva, What we mean when we say “what’s the dollar of mexico?”: Prototypes and mapping in concept space, in: AAAI Fall Symposium: Quantum Informatics for Cognitive, Social, and Semantic Processes, 2010, pp. 2–6.
  * [20] T. Plate, Holographic Reduced Representations, CLSI Publications, 2003.
  * [21] D. A. Rachkovskij, Representation and Processing of Structures with Binary Sparse Distributed Codes, IEEE Transactions on Knowledge and Data Engineering 3 (2) (2001) 261–276.
  * [22] S. I. Gallant, T. W. Okaywe, Representing objects, relations, and sequences, Neural Computation 25 (8) (2013) 2038–2078.
  * [23] D. Aerts, M. Czachor, B. De Moor, Geometric analogue of holographic reduced representation, Journal of Mathematical Psychology 53 (2009) 389–398.
  * [24] R. W. Gayler, Vector symbolic architectures answer jackendoff’s challenges for cognitive neuroscience, in: Proceedings of the Joint International Conference on Cognitive Science. ICCS/ASCS, 2003, pp. 133–138.
  * [25] F. R. Najafabadi, A. Rahimi, P. Kanerva, J. M. Rabaey, Hyperdimensional computing for text classification, Design, Automation Test in Europe Conference Exhibition (DATE), University Booth.
URL https://www.date-conference.com/system/files/file/date16/ubooth/37923.pdf

  * [26] M. Imani, D. Kong, A. Rahimi, T. Rosing, Voicehd: Hyperdimensional computing for efficient speech recognition, 2017 IEEE International Conference on Rebooting Computing (ICRC) (2017) 1–8doi:10.1109/ICRC.2017.8123650.
  * [27] P. Kanerva, J. Kristoferson, A. Holst, Random indexing of text samples for latent semantic analysis, in: Proceedings of the 22nd Annual Conference of the Cognitive Science Society, Erlbaum, 2000, p. 1036.
URL http://www.rni.org/kanerva/cogsci2k-poster.txt

  * [28] T. K. Landauer, S. T. Dutnais, A solution to plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge, Psychological Review 104 (2) (1997) 211–240.
  * [29] G. Recchia, M. Sahlgren, P. K. M. Jones, Encoding Sequential Information in Semantic Space Models. Comparing Holographic Reduced Representation and Random Permutation, Computational Intelligence and Neuroscience (2015) 1–18.
  * [30] M. Imani, J. Hwang, T. Rosing, A. Rahimi, J. M. Rabaey, Low-power sparse hyperdimensional encoder for language recognition, IEEE Design Test 34 (6) (2017) 94–101.  doi:10.1109/MDAT.2017.2740839.
  * [31] M. Imani, T. Nassar, A. Rahimi, T. Rosing, Hdna: Energy-efficient DNA sequencing using hyperdimensional computing, 2018 IEEE EMBS International Conference on Biomedical Health Informatics (BHI) (2018) 271–274doi:10.1109/BHI.2018.8333421.
  * [32] D. Kleyko, E. Osipov, On bidirectional transitions between localist and distributed representations: The case of common substrings search using vector symbolic architecture, Procedia Computer Science 41 (2014) 104–113.
  * [33] D. Kleyko, E. Osipov, R. W. Gayler, Recognizing permuted words with Vector Symbolic Architectures: A Cambridge test for machines, Procedia Computer Science 88 (2016) 169–175.
  * [34] O. Rasanen, Generating Hyperdimensional Distributed Representations from Continuous Valued Multivariate Sensory Input, in: Proceedings of the 37th Annual Meeting of the Cognitive Science Society, 2015, pp. 1943–1948.
  * [35] F. Montagna, A. Rahimi, S. Benatti, D. Rossi, L. Benini, Pulp-hd: Accelerating brain-inspired high-dimensional computing on a parallel ultra-low power platform, in: Proceedings of the 55th Annual Design Automation Conference, DAC ’18, ACM, New York, NY, USA, 2018, pp. 111:1–111:6.  doi:10.1145/3195970.3196096.
URL http://doi.acm.org/10.1145/3195970.3196096

  * [36] D. Kleyko, A. Rahimi, D. A. Rachkovskij, E. Osipov, J. M. Rabaey, Classification and recall with binary hyperdimensional computing: Tradeoffs in choice of density and mapping characteristics, IEEE Transactions on Neural Networks and Learning Systems (2018) 1–19doi:10.1109/TNNLS.2018.2814400.
  * [37] M. Imani, A. Rahimi, D. Kong, T. Rosing, J. M. Rabaey, Exploring hyperdimensional associative memory, 2017 IEEE International Symposium on High Performance Computer Architecture (HPCA) (2017) 445–456doi:10.1109/HPCA.2017.28.
  * [38] M. Schmuck, L. Benini, A. Rahimi, Hardware Optimizations of Dense Binary Hyperdimensional Computing: Rematerialization of Hypervectors, Binarized Bundling, and Combinational Associative Memory, ArXiv e-printsarXiv:1807.08583.
  * [39] O. Yilmaz, Symbolic computation using cellular automata-based hyperdimensional computing, Neural Computation 27 (12) (2015) 2661–2692, pMID: 26496041.  arXiv:https://doi.org/10.1162/NECO_a_00787, doi:10.1162/NECO\\_a\\_00787.
URL https://doi.org/10.1162/NECO_a_00787

  * [40] L. Chang, Iedm short course, 2012.
  * [41] J. Appenzeller, Carbon nanotubes for high-performance electronics—progress and prospect, Proceedings of the IEEE 96 (2) (2008) 201–211.  doi:10.1109/JPROC.2007.911051.
  * [42] M. M. Shulaker, G. Pitner, G. Hills, M. Giachino, H. . P. Wong, S. Mitra, High-performance carbon nanotube field-effect transistors, in: 2014 IEEE International Electron Devices Meeting, 2014, pp. 33.6.1–33.6.4.  doi:10.1109/IEDM.2014.7047164.
  * [43] Y. Yang, L. Ding, J. Han, Z. Zhang, L.-M. Peng, High-performance complementary transistors and medium-scale integrated circuits based on carbon nanotube thin films, ACS Nano 11 (4) (2017) 4124–4132, pMID: 28333433.  arXiv:https://doi.org/10.1021/acsnano.7b00861, doi:10.1021/acsnano.7b00861.
URL https://doi.org/10.1021/acsnano.7b00861

  * [44] C. Qiu, Z. Zhang, M. Xiao, Y. Yang, D. Zhong, L.-M. Peng, Scaling carbon nanotube complementary transistors to 5-nm gate lengths, Science 355 (6322) (2017) 271–276.  arXiv:http://science.sciencemag.org/content/355/6322/271.full.pdf, doi:10.1126/science.aaj1628.
URL http://science.sciencemag.org/content/355/6322/271

  * [45] R. S. Park, G. Hills, J. Sohn, S. Mitra, M. M. Shulaker, H.-S. P. Wong, Hysteresis-free carbon nanotube field-effect transistors, ACS Nano 11 (5) (2017) 4785–4791, pMID: 28463503\.  arXiv:https://doi.org/10.1021/acsnano.7b01164, doi:10.1021/acsnano.7b01164.
URL https://doi.org/10.1021/acsnano.7b01164

  * [46] T. Srimani, G. Hills, M. D. Bishop, U. Radhakrishna, A. Zubair, R. S. Park, Y. Stein, T. Palacios, D. Antoniadis, M. M. Shulaker, Negative capacitance carbon nanotube fets, IEEE Electron Device Letters 39 (2) (2018) 304–307.  doi:10.1109/LED.2017.2781901.
  * [47] J. Zhang, A. Lin, N. Patil, H. Wei, L. Wei, H. . P. Wong, S. Mitra, Carbon nanotube robust digital vlsi, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 31 (4) (2012) 453–471.  doi:10.1109/TCAD.2012.2187527.
  * [48] M. M. Shulaker, G. Hills, T. F. Wu, Z. Bao, H. . P. Wong, S. Mitra, Efficient metallic carbon nanotube removal for highly-scaled technologies, in: 2015 IEEE International Electron Devices Meeting (IEDM), 2015, pp. 32.4.1–32.4.4.  doi:10.1109/IEDM.2015.7409815.
  * [49] G. Hills, J. Zhang, M. M. Shulaker, H. Wei, C. Lee, A. Balasingam, H. . P. Wong, S. Mitra, Rapid co-optimization of processing and circuit design to overcome carbon nanotube variations, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 34 (7) (2015) 1082–1095.  doi:10.1109/TCAD.2015.2415492.
  * [50] M. M. Shulaker, G. Hills, N. Patil, H. Wei, H.-Y. Chen, H. S. P. Wong, S. Mitra, Carbon nanotube computer, Nature 501 (2013) 526 EP –.
  * [51] M. M. Shulaker, G. Hills, R. S. Park, R. T. Howe, K. Saraswat, H. S. P. Wong, S. Mitra, Three-dimensional integration of nanotechnologies for computing and data storage on a single chip, Nature 547 (2017) 74 EP –.
  * [52] H. S. P. Wong, H. Y. Lee, S. Yu, Y. S. Chen, Y. Wu, P. S. Chen, B. Lee, F. T. Chen, M. J. Tsai, Metal Oxide RRAM, Proceedings of the IEEE 100 (6) (2012) 1951–1970.  doi:10.1109/JPROC.2012.2190369.
  * [53] S. Lee, J. Sohn, H.-Y. Chen, H.-S. P. Wong, Metal Oxide Resistive Memory using Graphene Edge Electrode, Nature Communications.
  * [54] Z. Fang, H. Y. Yu, X. Li, N. Singh, G. Q. Lo, D. L. Kwong, $\hbox{HfO}_{x}/\hbox{TiO}_{x}/\hbox{HfO}_{x}/\hbox{TiO}_{x}$multilayer-based forming-free rram devices with excellent uniformity, IEEE Electron Device Letters 32 (4) (2011) 566–568.  doi:10.1109/LED.2011.2109033.
  * [55] Y. Kim, S. R. Lee, D. Lee, C. B. Lee, M. Chang, J. H. Hur, M. Lee, G. Park, C. J. Kim, U. Chung, I. Yoo, K. Kim, Bi-layered rram with unlimited endurance and extremely uniform switching, in: 2011 Symposium on VLSI Technology - Digest of Technical Papers, 2011, pp. 52–53.
  * [56] A. Grossi, E. Nowak, C. Zambelli, C. Pellissier, S. Bernasconi, G. Cibrario, K. E. Hajjam, R. Crochemore, J. F. Nodin, P. Olivo, L. Perniola, Fundamental variability limits of filament-based rram, in: 2016 IEEE International Electron Devices Meeting (IEDM), 2016, pp. 4.7.1–4.7.4.  doi:10.1109/IEDM.2016.7838348.
  * [57] Z. Chen, H. Wu, B. Gao, D. Wu, N. Deng, H. Qian, Z. Lu, B. Haukness, M. Kellam, G. Bronner, Performance improvements by sl-current limiter and novel programming methods on 16mb rram chip, in: 2017 IEEE International Memory Workshop (IMW), 2017, pp. 1–4.  doi:10.1109/IMW.2017.7939097.
  * [58] R. Fackenthal, M. Kitagawa, W. Otsuka, K. Prall, D. Mills, K. Tsutsui, J. Javanifard, K. Tedrow, T. Tsushima, Y. Shibahara, G. Hush, 19.7 a 16gb reram with 200mb/s write and 1gb/s read in 27nm technology, in: 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), 2014, pp. 338–339.  doi:10.1109/ISSCC.2014.6757460.
  * [59] H. Li, et al., Four-layer 3D vertical RRAM integrated with FinFET as a versatile computing unit for brain-inspired cognitive information processing, in: IEEE Symp. VLSI Technology, 2016.
  * [60] S. Sheu, M. Chang, K. Lin, C. Wu, Y. Chen, P. Chiu, C. Kuo, Y. Yang, P. Chiang, W. Lin, C. Lin, H. Lee, P. Gu, S. Wang, F. T. Chen, K. Su, C. Lien, K. Cheng, H. Wu, T. Ku, M. Kao, M. Tsai, A 4mb embedded slc resistive-ram macro with 7.2ns read-write random-access time and 160ns mlc-access capability, in: 2011 IEEE International Solid-State Circuits Conference, 2011, pp. 200–202.  doi:10.1109/ISSCC.2011.5746281.
  * [61] T. Cabout, J. Buckley, C. Cagli, V. Jousseaume, J.-F. Nodin, B. de Salvo, M. Bocquet, C. Muller, Role of ti and pt electrodes on resistance switching variability of hfo2-based resistive random access memory, Thin Solid Films 533 (2013) 19 – 23, eMRS 2012 Symposium L.  doi:https://doi.org/10.1016/j.tsf.2012.11.050.
URL http://www.sciencedirect.com/science/article/pii/S0040609012015477

  * [62] P. Leduc, L. D. Cioccio, B. Charlet, M. Rousseau, M. Assous, D. Bouchu, A. Roule, M. Zussy, P. Gueguen, A. Roman, O. Rozeau, M. Heitzmann, J. Nieto, L. Vandroux, P. Haumesser, R. Quenouillere, A. Toffoli, P. Sixt, S. Maitrejean, L. Clavelier, N. Sillon, Enabling technologies for 3d chip stacking, in: 2008 International Symposium on VLSI Technology, Systems and Applications (VLSI-TSA), 2008, pp. 76–78.  doi:10.1109/VTSA.2008.4530806.
  * [63] C. W. Liang, M. T. Chen, J. S. Jenq, W. Y. Lien, C. C. Huang, Y. S. Lin, B. J. Tzau, W. J. Wu, Z. H. Fu, I. C. Wang, P. Y. Chou, C. S. Fu, C. Y. Tzeng, K. L. Chiu, L. S. Huang, J. W. You, J. G. Hung, Z. M. Cheng, B. C. Hsu, H. Y. Wang, Y. H. Ye, J. Y. Wu, C. L. Yang, C. C. Huang, C. C. Chien, Y. R. Wang, C. C. Liu, S. F. Tzou, Y. H. Huang, C. C. Yu, J. H. Liao, C. L. Lin, D. F. Chen, S. C. Chien, I. C. Chen, A 28nm poly/sion cmos technology for low-power soc applications, in: 2011 Symposium on VLSI Technology - Digest of Technical Papers, 2011, pp. 38–39.
  * [64] P. Batude, M. Vinet, B. Previtali, C. Tabone, C. Xu, J. Mazurier, O. Weber, F. Andrieu, L. Tosti, L. Brevard, B. Sklenard, P. Coudrain, S. Bobba, H. B. Jamaa, P. Gaillardon, A. Pouydebasque, O. Thomas, C. L. Royer, J. . Hartmann, L. Sanchez, L. Baud, V. Carron, L. Clavelier, G. D. Micheli, S. Deleonibus, O. Faynot, T. Poiroux, Advances, challenges and opportunities in 3d cmos sequential integration, in: 2011 International Electron Devices Meeting, 2011, pp. 7.3.1–7.3.4.  doi:10.1109/IEDM.2011.6131506.
  * [65] N. Patil, A. Lin, E. R. Myers, K. Ryu, A. Badmaev, C. Zhou, H. . P. Wong, S. Mitra, Wafer-scale growth and transfer of aligned single-walled carbon nanotubes, IEEE Transactions on Nanotechnology 8 (4) (2009) 498–504.  doi:10.1109/TNANO.2009.2016562.
  * [66] H. Li, T. F. Wu, S. Mitra, H.-S. P. Wong, Resistive ram-centric computing: Design and modeling methodology, IEEE Transactions on Circuits and Systems I: Regular Papers 64 (9) (2017) 2263–2273.


#  Task-Agnostic Federated Learning with Imbalanced Data

Zhengtao Yao12, Hong Nguyen12, Ajitesh Srivastava2 1 Corresponding authors  2
University of Southern California, Los Angeles, CA 90089
<EMAIL_ADDRESS>
###### Abstract

In the realm of medical imaging, leveraging large-scale datasets from various
institutions is crucial for developing precise deep learning models, yet
privacy concerns frequently impede data sharing. Federated learning (FL)
emerges as a prominent solution for preserving privacy while facilitating
collaborative learning. However, its application in real-world scenarios faces
several obstacles, such as task & data heterogeneity, label scarcity, non-
identically distributed (non-IID) data, computational variation, etc. In real-
world, medical institutions may not want to disclose their tasks to FL server
and generalization challenge of out-of-network institutions with un-seen task
want to join the on-going federated system. This study addresses task-agnostic
and generalization problem on unseen tasks by adapting self-supervised FL
framework. Utilizing Vision Transformer (ViT) as consensus feature encoder for
self-supervised pre-training, no initial labels required, the framework
enables effective representation learning across diverse datasets and tasks.
Our extensive evaluations, using various real-world non-IID medical imaging
datasets, validate the approach’s efficiency. The proposed model retains 90%
of F1 accuracy of centralized approaches for classification task, and
outperforms it for segmentation task, which exhibits adaptability to out-of-
distribution data. The result indicates that federated learning architecture
can be a potential approach toward multi-task foundation modeling.

###### Index Terms:

federated learning, self-supervised learning, image classification, task
agnostic, vision transformer

##  I Introduction

Federated learning (FL) enables model training using data dispersed across
multiple locations without direct data sharing. In comparison to models
trained at individual sites, federated models can utilize a significantly
broader and larger dataset, potentially leading to enhanced performance and
greater generalizability. As a result, this training method has found
widespread adoption in crucial medical applications such as brain tumor
detection and COVID-19 diagnosis, and has been utilized across various data
types. However, In conventional FL system, the center model has to know what
task each sites want to achieve, e.g., predict brain ages, detect tumor,
classify Diabetic retinopathy, brain regional segmentation, etc. Thus, the
task-agnosticity limits the generalization of FL system.

Figure 1: Federated Learning system where clients’ tasks are anonymous between
each others and hidden from the server. $E_{i}$ is the $i_{th}$ feature
encoder and $D_{i}$ is the $i_{th}$ decoder.

Recently, advantage in unsupervised and self-supervised learning narrows the
gap in understanding medical images’ context, modality without knowing the
tasks. Self-supervised pre-training learns the intrinsic features of images in
local clients without labels, embodying less label-specific inductive bias and
thus, less susceptible to label distribution skewness. The proposed method
learns visual representations more effectively across non-IID clients, even
when data are limited at some clients. No task was reveal to server and still,
pretrained model learn semantic information of domain knowledge which can be
used for few-shot learning at each FL sites.

To the best of authors’ knowledge, many related works assume that the center
model knows all, or at least some tasks and possible labels of local
clients[31][32]. We address the challenges of data heterogeneity and tasks
anonymous by proposing a novel SSL-FL framework, following the unsupervised
federated learning setting in [14]. It tackles the non-iid data issue using
vision mamba to employ masked image modeling as the self-supervised task,
which are efficient in processing long sequences non-iid data because of its
convolutional and near-linear computation. The proposed pre-training scheme
significantly advances the capability of federated models over highly
heterogeneous data partitions. We also propose a novel client-specific fine
tuning framework using low rank adaptation, enabling the existing client to
achieve high prediction performance by fintuning only 2% of the pretrained
model. Moreover, We show that our framework is robust to merging new
institutes as well as new task. Our main contribution are summarized as
follows:

  * •

We define a new problem setting in federated learning involving task-agnostic
between clients and server. Along with that, we proposed a simple yet powerful
approach to address task-agnostic challenges at FL clients using self-
supervised pretraining. After the pretraining, we are able to transfer the
knowledge to the target tasks by a efficient fine-tuning implementation.

  * •

Using real-world datasets, we show that the combined framework robust to
multiple tasks as it achieves up 90% of SSL central training performance in
term of F1 and accuracy with only their own data and outperforms the central
model in segmentation tasks. It is found in the experiments that tasks with
less data benefits more from federated pretraining.

##  II Related Works

Client-Heterogeneity in Federated Learning. As a decentralized approach, FL
suffers from performance degradation due to client heterogeneity [36]. While
several research efforts [14] have been devoted to addressing the challenges
caused by data heterogeneity, task heterogeneity have not been well
investigated in literature. Moreover, the success of such models largely
relies on supervised ImageNet pre-training, which could suffer from domain
discrepancy when fine-tuning with medical images and can be further improved
by self-supervised pre-training on a centrally shared large-scale in-domain
medical dataset [14]. However, such centrally shared datasets rarely exist in
the medical domain due to privacy and ownership concerns. Therefore, it is
desired to build a self-supervised FL framework that collaboratively learns a
global model by leveraging all available unlabeled data without sharing data
among institutions.

Vision Foundation model and fine-tuning Understand medical images without
labels is a long-standing problem. At current state, label deficiency is a
common challenge in medical imaging. To address this issue, various approaches
such as semi-supervised and self-supervised learning methods [37][38][39][40]
have been proposed to allow models to learn from partially labeled or
unlabeled data. This work make use of advantage in SSL on FL systems for task-
agnostic settings.

Specifically, large model fine-tuning and the use of adapters are evolving
learning methods and pivotal strategies in the field of large language
model[15] [16], vision model[17] [18], and vision language model[19] [20].
These approaches are designed to tailor pre-trained models to specific tasks
or datasets with relatively minimal computational cost and data requirements.

##  III Problem Statement

Suppose there are $\mathcal{N}$ distributed clients and one centralized server
in a federated system. Each client has a private dataset $\mathcal{D}$ and
performs multiple private tasks
$\mathcal{T}=\\{\mathcal{T}_{1},...\mathcal{T}_{k}\\}$ with the dataset they
have. The server neither have access to $\mathcal{D}$ nor knows the
underlining tasks. Take advantage of un-labeled data from multi-modality and
anonymous tasks at clients, we want to boost the performance of each clients
compare with local supervision. There are similar works on multi-task
federated learning that aggregate observed tasks but the server must know
beforehand every possible tasks of clients. With identical SSL-FL
architecture, address non-IID and data heterogeneity problem, on a single
task. Unlike any previous works, we introduce a strictly constrained problem
involved task-related leakage concerns at federated server.

To address this problem, we assume that there is a unified block that is
compromised among clients and server, which have to be feature encoder. In
this work, we pre-trained the feature encoder at clients and aggregate it on
server to generalize the feature space with out-of-distribution data and
tasks.

##  IV Methodology

###  IV-A SSL-FL Pre-training

During the pre-training phase, each local model, indexed by $i$, functions as
an autoencoder with an encoder $E_{i}$ and a decoder $D_{i}$. This involves
masked image modeling, where some image patches are masked and the original
signals in these patches are reconstructed [33]. The encoder in a Vision
Transformer processes image patches using multiple layers of self-attention
and feed-forward networks to encode complex features and contextual
relationships. The decoder reconstructs the original image from these encoded
features, utilizing similar layers to generate predictions.

Each local encoder $E_{i}$ and decoder $D_{i}$ are trained with local data
$D^{i}$ to minimize the local objective function
$L_{i}(w)=\mathbb{E}_{x\sim\mathcal{X}^{i}}\big{[}l_{k}\big{(}w;\boldsymbol{x}\big{)}\big{]}$,
where $l_{i}$ is the mean squared error of the predicted pixel values of the
masked patches. The local loss function $l_{k}$ is given by:

$l_{k}=\sum_{j\in\mathcal{P}}\frac{1}{|\mathcal{P}|}((\boldsymbol{x}_{p}^{j}-\hat{\boldsymbol{x}}_{p}^{j})^{2};w)$

For pre-training, each local client update its local model $E_{k}$ and $D_{k}$
by minimizing its own loss $L_{k}$ on data $\mathcal{X}^{k}.$ Then, the server
takes a weighted average of all the resulting local models to update the
global model $E_{a}$ and $D_{a}$,which is further sent back to the local
clients for the next training iteration. Once pre-training is complete, the
final pre-trained global encoder $(E_{a}^{*})$ is saved.

###  IV-B Downstream fine-tuning

During the federated fine-tuning phase, depicted in Figure 2, we initialize
the local encoder $E_{k}$ of the $k_{th}$ client with the pretrained global
encoder $E_{a}$ acquired from the initial stage. Subsequently, we augment the
encoder with a linear downstream task head. The complete model is then fine-
tuned using the local labeled data.

Classification Task We freeze the weight of pre-trained VIT and add a
trainable LoRA adapter, as well as a fully-connected layer, which put on top
of pre-trained VIT, for specific classification tasks - Multilabel
classification task, Multiclass classification task and binary classification
task.

Segmentation Task The same pre-trained VIT is applied here for segmentation
task. We use of unetr framework[42] for our evaluation since unet-family allow
pre-trained encoder injected.

##  V Experiments Implementaion

###  V-A Datasets

In this work, we use six public fundus dataset in which the first four
datasets are used for in-distribution pretraining and task evaluation. The
last twos, LACDHS and JSIEC, act as out-of-distribution (ID) test dataset.
These two play a role as out-of-network (OOD) new clients joining federated
system. We want to evaluate how well own approach generalize to unseen task
and OOD data.

TABLE I: Statistics of six fundus-image datasets. Dataset | Train | Test | Task
---|---|---|---
RFMiD | 1920 | 640 | Multi-label Classification
DR | 29600 | 6800 | Severity Rating
EYEPACS | 8000 | 770 | Bianry Classification
REFUGE2 | 400 | 400 | Optic Disc Segmentation
LACDHS | 80 | 20 | Vessel Segmentation
JSIEC | 800 | 200 | Multi-class classification
TABLE II: Results on classification tasks Methods | Accuracy $\uparrow$ | Precision $\uparrow$ | Macro F1 $\uparrow$
---|---|---|---
$\mathrlap{\textit{Multilabel classification task:}}$ |
Local Supervision (Split 1) | 57.5 $\pm$ 0.6 | 13.9 $\pm$ 0.4 | 15.7 $\pm$ 0.3
Local Supervision (Split 2) | 59.7 $\pm$ 0.3 | 15.3 $\pm$ 0.2 | 15.9 $\pm$ 0.2
Centralized SSL (No FL) | 63.3 $\pm$ 0.5 | 16.9 $\pm$ 0.3 | 17.3 $\pm$ 0.3
SSL-FL (Split 1) | 58.9 $\pm$ 0.6 | 14.3 $\pm$ 0.5 | 14.8 $\pm$ 0.5
SSL-FL (Split 2) | 60.9 $\pm$ 0.6 | 15.3 $\pm$ 0.4 | 16.3 $\pm$ 0.3
$\mathrlap{\textit{Binary classification task:}}$ |  |  |
Local Supervision (Split 1) | 58.9 $\pm$ 0.4 | 55.3 $\pm$0.2 | 52.1 $\pm$ 0.2
Local Supervision (Split 2) | 60.4 $\pm$ 0.3 | 54.9 $\pm$ 0.3 | 55.4 $\pm$ 0.4
Centralized SSL (No FL) | 62.7 $\pm$ 0.3 | 58.7 $\pm$ 0.3 | 56.9 $\pm$ 0.2
SSL-FL (Split 1) | 59.8 $\pm$ 0.6 | 56.4 $\pm$ 0.3 | 54.2 $\pm$ 0.3
SSL-FL (Split 2) | 61.1 $\pm$ 0.3 | 57.1 $\pm$ 0.2 | 53.2 $\pm$ 0.2
$\mathrlap{\textit{Severity rating task:}}$ |  |  |
Local Supervision (Split 1) | 71.6 $\pm$ 0.3 | 17.2 $\pm$ 0.2 | 17.0 $\pm$ 0.3
Local Supervision (Split 2) | 73.3 $\pm$ 0.1 | 18.7 $\pm$ 0.0 | 18.5 $\pm$ 0.0
Centralized SSL (No FL) | 73.3 $\pm$ 0.1 | 18.3 $\pm$ 0.0 | 17.9 $\pm$ 0.0
SSL-FL (Split 1) | 71.5 $\pm$ 0.3 | 16.9 $\pm$ 0.1 | 17.2 $\pm$ 0.2
SSL-FL (Split 2) | 73.0 $\pm$ 0.3 | 18.4 $\pm$ 0.1 | 18.4 $\pm$ 0.1
TABLE III: Results on segmentation tasks (the lower the better) Methods | Dice Loss $\downarrow$ | Dice Focal $\downarrow$
---|---|---
$\mathrlap{\textit{Optic disc segmentation task on REFUGE dataset:}}$ |  |
Local Supervision (Scratch) | 94.9 $\pm$ 0.8 | 76.0 $\pm$ 0.5
Centralized SSL (No FL) | 92.4 $\pm$ 0.7 | 73.9 $\pm$ 1.2
SSL-FL (Split 1) | 93.7 $\pm$ 1.1 | 69.4 $\pm$ 0.8
SSL-FL (Split 2) | 91.1 $\pm$ 1.1 | 74.5 $\pm$ 2.3
$\mathrlap{\textit{Vessel segmentation task on JSIEC dataset:}}$ |  |
Local Supervision (Scratch) | 43.9 $\pm$ 0.9 | 64.1 $\pm$ 3.2
Centralized SSL (No FL) | 44.3 $\pm$ 1.8 | 66.4 $\pm$ 7.2
SSL-FL (Split 1) | 45.2 $\pm$ 1.7 | 61.4 $\pm$ 1.4
SSL-FL (Split 2) | 44.7 $\pm$ 1.1 | 60.7 $\pm$ 1.6

(a) Split 1

(b) Split 2

Figure 2: Data distribution of two split settings. In (a) clients have
different tasks and different datasets (modality) while clients in (b) share
the same amount of data from each datasets and do the same tasks for such
portion of dataset.

  * •

JSIEC Fundus dataset 1000 fundus images which belong to 39 classes are come
from the Joint Shantou International Eye Centre (JSIEC), Shantou city,
Guangdong province, China. The copyright of these images belongs to JSIEC.

  * •

RFMiD Small Dataset consists of 2560 fundus images captured using three
different fundus cameras with 46 conditions annotated through adjudicated
consensus of two senior retinal experts.

  * •

EyePACS The complete Rotterdam EyePACS AIROGS dataset, encompassing both
training and testing sets, comprises 8000 color fundus images sourced from
subjects across roughly 500 diverse sites with varying ethnic backgrounds.

  * •

REFUSE2 The Retinal Fundus Glaucoma Challenge 2nd Edition (REFUGE2) dataset
comprises 800 color fundus images accompanied by annotations for glaucoma
classification, optic disc/cup segmentation, and fovea localization.

  * •

DR Dataset Each image within the Diabetic retinopathy dataset is assessed by a
clinician who assigned a rating on a scale of 0 to 4, corresponding to the
absence of diabetic retinopathy (No DR), mild, moderate, severe, and
proliferative stages.

  * •

LACDHS Dataset The dataset comprises 100 fundus digital images of the retina
sourced from the Armed Forces Institute of Ophthalmology (AFIO) in Rawalpindi,
Pakistan. Included within the dataset are annotations detailing the retinal
blood vessel network, segmented artery/vein networks utilized for calculating
the Arteriovenous Ratio (AVR), as well as annotations of the Optic Nerve Head
(ONH).

###  V-B Experimental Settings

####  V-B1 Baseline

Since we derive a new problem setting, there are no other methods yield to
solve it. We used supervised learning on local labels as lower bound for
comparison. We assume that in best scenario, data will be visible from all
site so that we train a self-supervised encoder from all data and fine-tune it
on each tasks to get upper bound performance.

####  V-B2 Task and Data Heterogeneity Setup

We model Task imbalance (Split 1, data split by the dataset) and Task balance
(Split 2, data split by drawing same number of data from each dataset ) data
distributions of the six datasets, as is shown in 2. Centralized task is the
model pretrained on the six dataset together without aggregation. Local
supervision task represents pretraining and finetuning on its own task dataset
but not federatedly learn the data from other clients. Simulated data
partitions and task partitions allow for a more flexible and thorough
investigation of the model behavior, as they can be easily manipulated to test
data and task heterogeneity. All the different tasks are finetuned on their
own downstream task and the corresponding dataset.

####  V-B3 Self-supervised FL Pre-training and Supervised FL Fine-tuning and
Evaluation Metrics

Following [34], ViT-B [35] is chosen as the backbone for the proposed models.
Following the setup in MAE[33], the input is split into 16 × 16 patches for
MAE. In our main experiment, we randomly mask at most 60% of total image
patches for MAE. The downstream model is initialized using the pretrained
encoder and fine-tuned with a base learning rate starting at $3e^{-3}$ for all
tasks. We use accuracy, precision and F1 score as the evaluation metric for
classification on all the datasets.

##  VI results and discussion

###  VI-A Results for Classification

Table II shows the results from different training methodologies across three
classification tasks: multilabel classification, binary classification, and
severity rating. In all these classification tasks, the Centralized SSL (No
FL) method consistently delivers superior performance, suggesting the
advantage for federated learning with lack of data and data communication. The
federated learning part shows interesting results. In the multilabel
classification task, while SSL-FL (Split 1) shows a decrease in performance
compared to centralized SSL, SSL-FL (Split 2) regains some ground, suggesting
that the balanced nature of Split 2 may help mitigate some challenges
associated with federated settings. In the binary classification task, it is
worth to note that both federated learning approaches perform better than
local supervision alone, but not much for Split 2, which might due to the
great amount of training data for local supervisions in this case, leading to
strong representation of the encoder and bringing in information of other
dataset does not help much. Still Split 1 with unbalanced data scenarios poses
specific challenges that affect performance negatively compared to centralized
approaches and Split 2. The severity rating task results reveal a similar
pattern where Centralized SSL (No FL) achieves the best overall results. We
can see again Split 2 shows improved performance over Split 1, which echoes
the trend observed in the multilabel task. Also federated learning does not
help much when data is enough for local training. Overall, Centralized SSL
provides the best outcomes across all tasks and metrics. The performance
differences between SSL-FL splits emphasize the importance of data balancing
in federated settings, reducing disparities in model learning and
generalization and shows that clients with less data may benefit more from the
federated training.

###  VI-B Results for Segmentation and OOD Tasks

Figure 3: Example of segmentation results given by SSL-FL framework and local
supervision.

Table III presents the results of various methods applied to two different
tasks in medical image analysis: optic disc segmentation on the REFUGE dataset
and vessel segmentation on the JSIEC dataset. For the optic disc segmentation
task on the REFUGE dataset, the results show that Local Supervision achieves
the lowest performance with 94.9% Dice coefficient for optic disc segmentation
and 76.0% for cup segmentation. Centralized SSL without FL follows closely
behind with slightly higher performance. However, when SSL is combined with
Federated Learning (SSL-FL), there’s a increase in performance, especially
noticeable in the cup segmentation metric, with Split 1 achieving the highest
scores. Moving to the vessel segmentation task on the JSIEC dataset, similar
trends are observed. Local Supervision achieves the highest Dice coefficients
for vessel and background segmentation, with Centralized SSL without FL
trailing slightly behind. However, when SSL is coupled with Federated Learning
(SSL-FL), there’s again a increse in performance, particularly evident in
vessel segmentation for Split 2 because of task balance between clients. These
results suggest that while SSL can enhance segmentation performance, the
introduction of federated learning in this context might not consistently
improve results. The choice of data split also influences the effectiveness of
SSL with Federated Learning.

## Conclusion

In this paper, we propose a privacy-preserving and federated self-supervised
learning framework that collaboratively trains models on decentralized data
using masked image modeling as the heterogeneous self-supervised tasks, in
which Low Rank Adaptation(LoRA) is used in the fine tuning part. Our framework
is robust to non-IID data distribution across clients, and performs well under
severe task heterogeneity and data imbalance settings across diverse medical
datasets. Experiments show that while tasks(clients) with less data benefits
more from federated pretraining, all with different downstream task will
perform better than its local supervision.

## References

  * [1] J. Clerk Maxwell, A Treatise on Electricity and Magnetism, 3rd ed., vol. 2. Oxford: Clarendon, 1892, pp.68–73.
  * [2] I. S. Jacobs and C. P. Bean, “Fine particles, thin films and exchange anisotropy,” in Magnetism, vol. III, G. T. Rado and H. Suhl, Eds. New York: Academic, 1963, pp. 271–350.
  * [3] K. Elissa, “Title of paper if known,” unpublished.
  * [4] R. Nicole, “Title of paper with only first word capitalized,” J. Name Stand. Abbrev., in press.
  * [5] Y. Yorozu, M. Hirano, K. Oka, and Y. Tagawa, “Electron spectroscopy studies on magneto-optical media and plastic substrate interface,” IEEE Transl. J. Magn. Japan, vol. 2, pp. 740–741, August 1987 [Digests 9th Annual Conf. Magnetics Japan, p. 301, 1982].
  * [6] M. Young, The Technical Writer’s Handbook. Mill Valley, CA: University Science, 1989.
  * [7] Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. 1960
  * [8] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
  * [9] Albert Gu, Karan Goel, and Christopher Re. Efficiently ´ modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021.
  * [10] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Re. Combining recurrent, ´ convolutional, and continuous-time models with linear state space layers. In NeurIPS, 2021.
  * [11] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Re.´ On the parameterization and initialization of diagonal state space models. In NeurIPS, 2022.
  * [12] Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. In NeurIPS, 2022\.
  * [13] Zhu, L., Liao, B., Zhang, Q., Wang, X., Liu, W., Wang, X.: Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417
  * [14] Yan, Rui, et al. ”Label-efficient self-supervised federated learning for tackling data heterogeneity in medical imaging.” IEEE Transactions on Medical Imaging (2023).
  * [15] Mao, Y.; Mathias, L.; Hou, R.; Almahairi, A.; Ma, H.; Han, J.; Yih, W.-t.; and Khabsa, M. 2021. Unipelt: A unified framework for parameter-efficient language model tuning. arXiv preprint arXiv:2110.07577.
  * [16] Mosbach, M.; Andriushchenko, M.; and Klakow, D. 2020. On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines. In ICLR.
  * [17] Mostafa, H.; and Wang, X. 2019. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In ICML.
  * [18] He, Xuehai, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. ”Parameter-efficient fine-tuning for vision transformers.” arXiv preprint arXiv:2203.16329 3 (2022).
  * [19] Sun, Jingchen, Jiayu Qin, Zihao Lin, and Changyou Chen. ”Prompt tuning based adapter for vision-language model adaption.” arXiv preprint arXiv:2303.15234 (2023).
  * [20] Yu, Tao, Zhihe Lu, Xin Jin, Zhibo Chen, and Xinchao Wang. ”Task residual for tuning vision-language models.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10899-10909. 2023.
  * [21] Hu, Edward J., Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. ”Lora: Low-rank adaptation of large language models.” arXiv preprint arXiv:2106.09685 (2021
  * [22] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-Efficient Transfer Learning for NLP. arXiv:1902.00751 [cs, stat], June 2019.
  * [23] Guo, Yunhui, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. ”Spottune: transfer learning through adaptive fine-tuning.” In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4805-4814. 2019.
  * [24] Brian Lester, Rami Al-Rfou, and Noah Constant. The Power of Scale for Parameter-Efficient Prompt Tuning. arXiv:2104.08691 [cs], April 2021.
  * [25] Jia, Menglin, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. ”Visual prompt tuning.” In European Conference on Computer Vision, pp. 709-727. Cham: Springer Nature Switzerland, 2022.
  * [26] Guo, Ziyu, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, and Bin Cui. ”Calip: Zero-shot enhancement of clip with parameter-free attention.” In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 1, pp. 746-754. 2023.
  * [27] Cui, Baiyun, Yingming Li, Ming Chen, and Zhongfei Zhang. ”Fine-tune BERT with sparse self-attention mechanism.” In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp. 3548-3553. 2019
  * [28] Finn, Chelsea, Pieter Abbeel, and Sergey Levine. ”Model-agnostic meta-learning for fast adaptation of deep networks.” In International conference on machine learning, pp. 1126-1135. PMLR, 2017.
  * [29] Snell, Jake, Kevin Swersky, and Richard Zemel. ”Prototypical networks for few-shot learning.” Advances in neural information processing systems 30 (2017).
  * [30] Nichol, Alex, and John Schulman. ”Reptile: a scalable metalearning algorithm.” arXiv preprint arXiv:1803.02999 2, no. 3 (2018): 4.
  * [31] Itahara, Sohei, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, and Koji Yamamoto. ”Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data.” IEEE Transactions on Mobile Computing 22, no. 1 (2021): 191-205.
  * [32] Lin, Haowen, Jian Lou, Li Xiong, and Cyrus Shahabi. ”Semifed: Semi-supervised federated learning with consistency and pseudo-labeling.” arXiv preprint arXiv:2108.09412 (2021).
  * [33] He, Kaiming, et al. ”Masked autoencoders are scalable vision learners.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022
  * [34] Yan, Rui, et al. ”Label-efficient self-supervised federated learning for tackling data heterogeneity in medical imaging.” IEEE Transactions on Medical Imaging 42.7 (2023): 1932-1943.
  * [35] Dosovitskiy, Alexey, et al. ”An image is worth 16x16 words: Transformers for image recognition at scale.” arXiv preprint arXiv:2010.11929 (2020).
  * [36] Mitra, Aritra, et al. ”Linear convergence in federated learning: Tackling client heterogeneity and sparse gradients.” Advances in Neural Information Processing Systems 34 (2021): 14606-14619.
  * [37] Atito, Sara, Muhammad Awais, and Josef Kittler. ”Sit: Self-supervised vision transformer.” arXiv preprint arXiv:2104.03602 (2021).
  * [38] Baevski, Alexei, et al. ”Data2vec: A general framework for self-supervised learning in speech, vision and language.” International Conference on Machine Learning. PMLR, 2022.
  * [39] Weng, Zejia, et al. ”Semi-supervised vision transformers.” European conference on computer vision. Cham: Springer Nature Switzerland, 2022.
  * [40] Zhai, Xiaohua, et al. ”S4l: Self-supervised semi-supervised learning.” Proceedings of the IEEE/CVF international conference on computer vision. 2019.
  * [41] Smith, Virginia, et al. ”Federated multi-task learning.” Advances in neural information processing systems 30 (2017).
  * [42] Hatamizadeh, Ali, et al. ”Unetr: Transformers for 3d medical image segmentation.” Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2022.