# Capsule Network Projectors are Equivariant and Invariant Learners

Miles Everett $\;{}^{1}$ &Aiden Durrant$\mathsection$1 &Mingjun Zhong 1
&Georgios Leontidis 1,2 1 Department of Computing Science
University of Aberdeen, UK
&2 Interdisciplinary Centre for Data and AI
University of Aberdeen, UK
<EMAIL_ADDRESS>Co-
first authors

###### Abstract

Learning invariant representations has been the longstanding approach to self-
supervised learning. However, recently progress has been made in preserving
equivariant properties in representations, yet do so with highly prescribed
architectures. In this work, we propose an invariant-equivariant self-
supervised architecture that employs Capsule Networks (CapsNets) which have
been shown to capture equivariance with respect to novel viewpoints. We
demonstrate that the use of CapsNets in equivariant self-supervised
architectures achieves improved downstream performance on equivariant tasks
with higher efficiency and fewer network parameters. To accommodate the
architectural changes of CapsNets, we introduce a new objective function based
on entropy minimisation. This approach which we name CapsIE (Capsule Invariant
Equivariant Network) achieves state-of-the-art performance across all
invariant and equivariant downstream tasks on the 3DIEBench dataset, while
outperforming supervised baselines. Our results demonstrate the ability of
CapsNets to learn complex and generalised representations for large-scale,
multi-task datasets compared to previous CapsNet benchmarks. Code is available
at https://github.com/AberdeenML/CapsIE.

##  1 Introduction

Equivariance and invariance have become increasingly important properties and
objectives of deep learning in recent times, with precedence being largely
placed on the latter. The task of invariance, i.e. being able to classify a
specific object no matter the camera perspective or augmentation applied, has
driven progress in modern self-supervised learning approaches, specifically
those which follow a joint embedding architecture [1, 3, 7]. Equivariance on
the other hand is the task of capturing embeddings which equally reflect the
translations applied to the input space in the latent space. Equivariance thus
has become an important property to capture to enable the learning of high-
quality representations in the real world where transformations such as
viewpoint are essential.

Self-supervised learning predominantly owes its success to such invariant
objectives, where all recent progress, whether that is by contrastive [7],
information-maximisation [3, 33], or clustering based methods [5, 1] rely on
ensuring invariance in their representations under augmentation. This setting
typically ensures performance in classification based tasks, but when
employing the representations in alternative tasks, preservation of
information is essential to improve generalisation. To maintain properties of
the transformation one can predict the augmentations applied [8, 21], yet this
is typically not considered truly equivariant given the mapping of
transformations is not represented in the latent space. Methods that employ
such a prediction methodology are typically considered equivariant as the
transformation in the input space is directly preserved in the latent space.
Here prediction networks are employed to reconstruct the view prior to
transformation [30], learn symmetric representations [26], or predict the
latent representation of the transformed view from the representation of the
original view given the transformation parameters [14].

The above methods, although promising, enforce equivariance via objective
functions on vector representations, yet these methods fail to employ
architectural approaches that have shown to be capable of better capturing
these properties. Capsule Networks (CapsNets), which utilise a process called
routing [29, 17, 12, 15, 9, 22], are one such architecture, showing signs of
desirable properties that other state of the art (SOTA) architectures such as
Vision Transformers (ViTs) and CNNs do not. Specifically, CapsNets have shown
a natural ability to have strong viewpoint equivariance and viewpoint
invariance properties – they achieve this through their ability to capture
equivariance with respect to viewpoints in neural activities, and invariance
in the weights. In addition, viewpoint changes have nonlinear effects on
pixels but linear effects on object relationships [9, 17]. Ideally, these
properties could lead to the development of more sample-efficient models that
can exploit robust representations to better generalise to unseen cases and
new samples.

However, a common argument is that CapsNets have only shown these properties
on toy examples such as the smallNORB dataset [20], which many would consider
irrelevant for modern architectures. Despite this, small CapsNets outperform
much larger CNN and ViT counterparts [12]. In this work, we propose a novel
CapsNet formulation and corresponding objective function, achieving SOTA on
several experiments and ablations studies on the 3DIEBench dataset [14] which
has been created to specifically benchmark equivariant and invariant
properties of deep learning models. We prove that CapsNets retain their
desirable properties on this dataset which is considerably more difficult than
what has been previously achieved with CapsNets, while also establishing new
SOTA on these tasks.

(a) Schematic overview of the CapsIE architecture.

(b) Generalised visualisation of the CapsNet projector.

Figure 1: Left: Schematic overview of the proposed CapsIE architecture.
Representations are fed into a CapsNet projector, and the output embeddings
$Z_{\text{act}}$ and $Z_{\text{pose}}$ correspond to invariant and equivariant
embeddings respectively. Right: Generalised view of a Capsule projection head.
A CNN feature map is transformed via the primary capsules into initial poses
$u_{i}$, represented by cylinders, and activations $a_{i}$, represented by
circles. These poses are transformed into votes, which represent a lower-level
capsules prediction for each of the higher-level capsules. The routing process
then defines how well these votes match the concept represented by the upper-
level capsule, creating the coupling coefficients. Finally, these coupling
coefficients are used to create $u_{j}$ and $a_{j}$, the output of the capsule
projector head.

To summarise, our contributions are:

  * •

We propose a novel architecture based on a Capsule Network projection head
that utilises the key assumptions of capsule architectures to learn
equivariant and invariant representations which does not require the explicit
split of representations.

  * •

We design a new objective function to accommodate the employment of a CapsNet
projector, enforcing invariance through entropy minimisation.

  * •

We demonstrate that CapsNet projectors implicitly learn pose understanding in
a self-supervised setting.

  * •

We show state of the art performance on 3DIEBench classification for
equivariance and invariance benchmark tasks from our CapsNet based
architecture.

Figure 2: Visual depiction of the problem statement. Two images are
represented by subscripts $0,1$ while view under transformation $g$ is given
by ′. Arrows represent the construction of embeddings from an encoder network.
Top the invariance objective is to maximise the similarity between embeddings
of views originating from the same image. Bottom the equivariance objective
aim to learn the transformations $\rho_{X}(g)\cdot x$ applied to $x$.

##  2 Problem Statement

Typically, self-supervised learning maximises the similarity between
embeddings of two augmented views of an image such that they are invariant to
augmentations, and instead capture semantically meaningful information of the
original image. Views $x$ and $x^{\prime}$ are each transformed from an image
$d\in\mathbb{R}^{c\times h\times w}$ sampled from dataset $\mathcal{D}$ by
image augmentations $\tau,\tau^{\prime}\sim T$ sampled from a set of
augmentations $T$. Embeddings are obtained by feeding each view through an
encoder $f_{\theta}$, where the output representations $y,y^{\prime}$ are fed
through a projection head $h_{\phi}$ to produce embeddings $z,z^{\prime}$
whose similarity is maximised. However, it is detrimental in many settings
that $f$ is invariant to all transformations, instead in this work we are
focused on ensuring that $f$ is equivariant to viewpoint transformations. To
train for and evaluate such properties we base our study on the challenging
3DIEBench dataset and corresponding problem definition presented in [14].

First to define equivariance we begin by defining a Group consisting of a set
$G$ and a binary operation $\cdot$ on G, $\cdot:G\times G\rightarrow G$ such
that $\cdot$ are associative; there is an identity $e$ which satisfies $e\cdot
a=a=a\cdot e,\forall a\in G$; and for each $a\in G$ there exists an inverse
$a^{-1}$ such that $a\cdot a^{-1}=e=a^{-1}\cdot a$. Group actions are
concerned with how groups manipulate sets, where the left group action can be
defined as a function $\alpha$ of group $G$ and set $S$, $\alpha:G\times
S\rightarrow S$ such that $\alpha(e,s)=s,\forall s\in S$, and
$\alpha(g,\alpha(h,s))=\alpha(gh,s),\forall s\in S,\text{and }\forall g,h\in
G$. In our setting, we are concerned with group representations which are
linear group actions acting on vector space $V$, which we define as
$\rho:G\rightarrow GL(V)$ where $GL(V)$ is the general linear group on $V$.
Here $\rho(g)$ describes the transformation applied to both the input data $x$
and latent $x^{\prime}$ given parameters $g$ [26, 14].

Following this, we can define the function $f:X\rightarrow Y$ as being
equivariant with respect to a group $G$ with representations $\rho_{X}$ and
$\rho_{Y}$ if $\forall x\in X,$ and $\forall g\in G,$

$f(\rho_{X}(g)\cdot x)=\rho_{Y}(g)\cdot f(x).$ (1)

As defined in [14] the goal (visually depicted in Figure 2) is to learn $f$
and $\rho_{Y}$ to construct representations that are equivariant to viewpoint
transformations when $\rho_{X}$ is not known, but the group elements $g$ that
parameterise the transformations are. Further details of these transformations
and the benchmark 3DIEBench dataset are given in Section A.

##  3 Method

###  3.1 Architecture

Our method which we name CapsIE (Capsule Network Invariant Equivariant)
follows the general joint embedding architecture previously described in
Section 2 and extends those proposed by VICReg [3] and SIE [14]. Like previous
methods, we employ a ResNet-18 [16] encoder as the core feature extractor
$f_{\theta}$ of our network. Yet unlike SIE [14], we do not split the
representations, and therefore do not require the use of separate invariant
and equivariant projection heads. Instead, we employ a single CapsNet
(described in Section 3.2) which takes as input the full representation of the
encoder in place of the multi-layer perceptron (MLP) in the projection head
$h_{\phi}$. Given the architectural design of CapsNets, our projection head
outputs both an activation scalar, representing how active the capsule is, and
a $4\times 4$ pose for each capsule.

To align with the above problem statement and Equation 1, we aim to
simultaneously learn invariant $f$ and equivariant representations by
optimising our network with respect to the output activations and poses. In
this case, we consider the activation vector to capture existence of semantic
concepts/objects of the input, thus, the invariant information is preserved by
the transformation. The pose on the other hand is designed to encode
positional information related to each corresponding capsule [27] (i.e.
semantic concept), therefore it contains equivariant information that that was
changed by the transformation. Akin to the SIE method, we therefore, consider
two embedding vectors for each image view, $z_{\text{act}}$ and
$z_{\text{pose}}$, which correspond to the capsule activation vector and pose,
and invariant and equivariant components respectively.

To enforce our network to learn equivariant properties we utilise a prediction
network $p_{\psi,g}$ which takes as input the transformation $g$ and
$z_{\text{pose}}$ to predict $z^{\prime}_{\text{pose}}$, and hence learn
$\rho_{Y}(g)$. In our setting, $g\in\mathbb{R}^{3}$ corresponding to the
quaternions of the rotation applied. In this work, we employ the hypernetwork
approach taken by [14] which uses an MLP that takes as input the
transformation parameters $g$ to parameterise an MLP predictor. Such a network
avoids the case where the transformation parameters $g$ are ignored and the
predictor provides invariant solutions. We present more details of the
predictor network in the appendix, and visually depict the full CapsIE
architecture in Figure 1.

###  3.2 Capsule Network Projector

Figure 3: Simplified visual representation of CapsNet outputs. Activation
vector outputs probability of each capsule being activated. pose corresponds
to the object pose for each corresponding capsule.

#### Capsule Viewpoint Equivariance

CapsNets are designed to handle spatial hierarchies and recognise objects
regardless of their orientation or location, achieving equivariance through
their structure [28]. A capsule is a group of neurons – vector-based
representations – representing instantiation parameters such as position,
orientation, and size. Before any routing process begins, lower-level capsule
poses $u_{i}$ are transformed to $n$ upper-level capsule poses $u_{j|i}$ which
align with concepts represented by higher-level capsules, preserving spatial
relationships and hierarchical information. It is then determined through the
routing process how well these transformed poses correspond with the concept
represented by the upper-level capsule. CapsNets, unlike convolution, excel in
achieving viewpoint invariance and viewpoint equivariance as they can capture
equivariance with respect to viewpoints in neural activities, and invariance
in the network’s weights [27]. Consequently, capsule routing aims to detect
objects by looking for agreement between their parts, thereby performing
equivariant inference.

#### Self Routing Capsule

We use the Self Routing CapsNet (SRCaps) [15] based on the efficiency of its
non-iterative routing algorithm. We consider the trade-off of a small amount
of classification accuracy to be acceptable when comparing the performance of
SRCaps to other capsule architectures which require significantly more
resources to train. Based on the size of the 3DIEBench dataset, these other
routing algorithms would be unsuitable.

SRCaps calculates the coupling coefficients between each capsule in lower
layer $i$ with each capsule in upper layer $j$ to produce the coupling
coeffients $c_{ij}$. It does so by using a learnable routing matrix
$W^{route}$ multiplied with the lower capsule pose vector $u_{i}$, mimicking a
single layer perceptron to produce routing coefficients $b_{ij}$ which when
passed through a softmax function produce coupling coefficients $c_{ij}$.
Additionally, we determine the activation of upper-level capsules $a_{j}$ by
first multiplying $a_{i}$ by $c_{ij}$ to create votes and then dividing this
by $a_{i}$ to create weighted votes.

$c_{ij}=\text{softmax}(W^{route}_{i}u_{i})_{j},\quad
a_{j}=\frac{\sum_{i\in\Omega_{l}}c_{ij}a_{i}}{\sum_{i\in\Omega_{l}}a_{i}}$ (2)

The output pose of a capsule layer is calculated using learnable weight matrix
$W^{pose}$ which when multiplied with $u_{i}$ provides a capsule pose of each
lower-level capsule for each upper-layer capsule i.e $u_{j|i}$. Following the
same procedure as the activations, $u_{j}$ is the weighted sum of these poses
by $a_{j}$.

$\hat{u}_{j|i}=W_{ij}^{\text{pose}}u_{i},\quad
u_{j}=\frac{\sum_{i\in\Omega_{l}}c_{ij}a_{i}\hat{u}_{j|i}}{\sum_{i\in\Omega_{l}}c_{ij}a_{i}}.$
(3)

###  3.3 Objective Functions

#### Invariant Criterion.

To train our aforementioned architecture we first introduce an invariant
objective as the cross entropy between activation probability vectors,
$H(Z_{\text{act}},Z^{\prime}_{\text{act}})$ where $Z$ refers to the matrix
embeddings over a batch. The aims is to enforce embedding probability pairs
originating from the same image to be matched. To avoid trivial solutions and
collapse to a single capsule, we employ the mean entropy maximisation
regularisation [2, 1] on the same activation probability vectors to encourage
the model to utilise the full set of capsules over a batch. This
regularisation maximises the entropy of the mean probabilities
$H(\bar{Z}_{\text{act}})$ and $H(\bar{Z}^{\prime}_{\text{act}})$, where
$\bar{Z}_{\text{act}}=\frac{1}{B}\sum^{B}_{i=1}Z_{\text{act}}$ and $B$ is the
batch size.

#### Equivariant Criterion.

As previously stated in Section 2, our goal is to learn the predictor
$p_{\psi,g}$ to model $\rho_{Y}(g)$ as to enforce equivariant representations.
This is achieved by maximising the cosine similarity between the output vector
of the predictor $p_{\psi,g}(Z_{\text{pose}})$ given translation parameters
$g$ and equivariant representation $Z_{\text{pose}}$, and the augmented view’s
equivariant representation vector $Z^{\prime}_{\text{pose}}$. To avoid
collapse and improve training stability we also regularise the output of
$p_{\psi,g}(Z_{\text{pose}})$ by minimising the variance of the predicted
equivariance representation to avoid collapse. Whereas, SIE [14] finds this to
be an optional but recommended component, we found in practice, without such
regularisation the predictor would consistently collapse to trivial solutions.

As with the activation vector we employ variance-covariance regularisation on
the pose to ensure they do not collapse of representations to trivial
solutions. The variance objective $V$ ensures that all dimensions $d$ in the
embedding vector are equally utilised while the covariance objective $C$
decorrelates the dimensions to reduce redundancy across dimensions. The
regularisation for equivariant vectors $\mathcal{L}_{\text{reg}}$ is given by

$\mathcal{L}_{\text{reg}}(Z)=\lambda_{C}\;C(Z)+\lambda_{V}\;V(Z),\quad\text{where}$
(4) $C(Z)=\frac{1}{d}\sum_{i\neq j}Cov(Z)_{i,j}^{2}\quad\text{and}\quad
V(Z)=\frac{1}{d}\sum_{j=1}^{d}\max\left(0,1-\sqrt{Var(Z_{\cdot,j})}\right).$
(5)

The final objective function is given by the weighted sum of the individual
objectives

$\displaystyle\mathcal{L}(Z_{\text{act}},Z^{\prime}_{\text{act}},Z_{\text{pose}},Z^{\prime}_{\text{pose}})=\
$
$\displaystyle\lambda_{\text{inv}}H(Z_{\text{act}},Z^{\prime}_{\text{act}})+(H(\bar{Z}_{\text{act}})+H(\bar{Z}^{\prime}_{\text{act}}))\
+$ (6)
$\displaystyle\lambda_{\text{equi}}\frac{1}{N}\sum^{N}_{i=1}\|p_{\psi,g_{i}}(Z_{i,\text{pose}})-Z^{\prime}_{i,\text{pose}}\|^{2}_{2}\
+$ (7)
$\displaystyle\mathcal{L}_{\text{reg}}(Z_{\text{pose}})+\mathcal{L}_{\text{reg}}(Z^{\prime}_{\text{pose}})+\lambda_{V}V(p_{\psi,g_{i}}(Z_{i,\text{pose}})).$
(8)

##  4 Experimentation

Table 1: Evaluation of invariant properties on downstream classification task. We evaluate both the representations and the intermediate embeddings of the projection head when different numbers of capsules in the projection head is used. ‘-’ refers to non-compatible experiments. | Computational Load | Embedding Dims | Classification (Top-1%)
---|---|---|---
Method | Parameters | # Flops | Inv. | Equi. | All | Inv. | Equi.
Supervised |  |  |  |
ResNet-18 | 11.2M | 3.09G | - | - | 87.47 | - | -
SR-Caps - 16 | 11.0M | 3.16G | - | - | - | 73.85 | -
SR-Caps - 32 | 13.0M | 4.27G | - | - | - | 59.70 | -
SR-Caps - 64 | 18.7M | 8.22G | - | - | - | 69.45 | -
Encoder Representation |  |  |  |  |
SIE [14] | 20.1M | 13.07G | 512 | 512 | 82.94 | 82.08 | 80.32
CapsIE - 16 | 12.7M | 3.49G | 16 | 256 | 78.96 | - | -
CapsIE - 32 | 14.7M | 4.57G | 32 | 512 | 80.00 | - | -
CapsIE - 64 | 20.4M | 8.69G | 64 | 1024 | 80.26 | - | -
Capsule Projector - 1st Intermediate Embedding |  |  |  |  |
CapsIE - 16 | 12.7M | 3.49G | 16 | 256 | - | 82.96 | -
CapsIE - 32 | 14.7M | 4.57G | 32 | 512 | - | 83.49 | -
CapsIE - 64 | 20.4M | 8.69G | 64 | 1024 | - | 83.64 | -

###  4.1 Training Protocol

To directly compare with prior works employing the 3DIEBench dataset, we
follow an identical training protocol, as defined in [14]. All methods employ
a ResNet-18 encoder network ($f_{\theta}$), for the projection head
($h_{\phi}$) we compare a variety of hyperparameterisations, which we later
describe in the following sections. For primary bench-marking we train our
model for 2000 epochs using the Adam [19] optimiser with default settings, a
fixed learning rate of 1e-3 and batch size of 1024. For ablations and
sensitivity analyses we train for 500 epochs and employ a batch size of 512,
with other settings remaining unchanged. We find in practice that 500 epochs
presents a strong correlation of performance. By default the objective
function weighting are as follows,
$\lambda_{\text{inv}}=0.1,\;\lambda_{\text{equi}}=5,\;\lambda_{V}=10,\;\lambda_{C}=1$.
Each self-supervised 2000 epoch pretraining run took approximately 22 hours
using 3 Nvidia A100 80GB GPU’s, with 64 capsule models taking approximately 25
hours using 6 Nvidia A100 80GB GPU’s. All eval tasks are completed on a single
Nvidia A100 80GB GPU and take approximately 6 hours for angle prediction, and
3 hours for classification.

###  4.2 Downstream Evaluation

To evaluate the quality of representations learnt under self-supervision, we
use the standard benchmark approach of learning downstream task specific
networks with frozen representations as input. In our case we evaluate the
representations in three distinct tasks to evaluate both invariant and
equivariant properties, we use a linear evaluation training protocol of the
frozen representations. Further details of the evaluation protocol are given
in the appendix B.2.

Invariant Evaluation. To evaluate invariant properties of the representation
we train a classifier on either the frozen representations output from the
encoder network or the intermediate embeddings of the capsule network
projector. Given our advocation for CapsNets we evaluate using both the
standard linear classification and a capsule layer whose number of out
capsules is set to the number of classes. All methods are trained for 300
epochs by cross entropy.

Equivariant Evaluation. Evaluating equivariant properties is achieved through
a rotation prediction task in which a three layer MLP is trained to predict
the quaternions defining the rotation between two views of the same object. We
train for 300 epochs using MSE loss. Similarly to rotation prediction, we
evaluate the representation’s equivariant properties by regressing the the
colour hue of an object view. We train a single linear layer for 50 epochs
using MSE loss.

Representation Quality. The performance of CapsIE for both invariant and
equivariant benchmark tasks is given in Tables 1 and 2 respectively. We
evaluate both the representations produced by the ResNet-18 encoder and the
intermediate embeddings of the capsule layer projection head given different
values for the numbers of capsules. We observe that across all models that the
invariant properties captured within the representations marginally suffer
compared to the MLP projector of SIE. This observation is expected given the
significantly reduced number of embeddings employed in the invariant criterion
compared to SIE. However, the evaluation of equivariant properties captured by
the representations demonstrates that the use of a capsule projector in place
of a MLP can lead to vastly improved performance in rotation prediction
($\uparrow 0.05\;R^{2}$) advancing the prior state-of-the-art whist also
improving on the supervised baseline by a significant margin.

Interestingly, we also observe that the colour prediction task achieves a
performance close to that of the supervised setting even though our criteria
do not directly optimise for such equivariant properties. This suggests that
the CapsNet is responsible for implicitly capturing transformations of the
input whilst having little to no impact on the tasks directly optimised for.
In addition, to the improvement on SOTA our CapsIE model has the favourable
consequence of being far more parameter and computationally efficient.

Table 2: Evaluation of equivariant properties on downstream rotation prediction (left) and colour prediction (right) tasks. We evaluate both the representations and the intermediate embeddings of the projection head when different numbers of capsules in the projection head is used. | Rotation Prediction ($R^{2}$) | Colour Prediction ($R^{2}$)
---|---|---
Method | All | Inv. | Equi. | All | Inv. | Equi.
Supervised |  |  |
ResNet-18 | 0.76 | - | - | 0.99 | - | -
SR-Caps - 16 | - | - | 0.83 | - | - | 0.99
SR-Caps - 32 | - | - | 0.84 | - | - | 0.99
SR-Caps - 64 | - | - | 0.80 | - | - | 0.99
Encoder Representation |  |  |
SIE [14] | 0.73 | 0.23 | 0.73 | 0.07 | 0.05 | 0.02
CapsIE - 16 | 0.78 | - | - | 0.97 | - | -
CapsIE - 32 | 0.75 | - | - | 0.97 | - | -
CapsIE - 64 | 0.72 | - | - | 0.97 | - | -
Projector - 1st Intermediate Embedding |  |  |
CapsIE - 16 | - | - | 0.78 | - | - | 0.97
CapsIE - 32 | - | - | 0.77 | - | - | 0.97
CapsIE - 64 | - | - | 0.78 | - | - | 0.97

Intermediate Projector Embeddings. The role of the projector is primarily
employed to decorrelate the embeddings on which the objective function
operates on from the representations employed downstream. The premise is to
avoid representations that are over-fit to the self-supervised objective [4].
However, it has been well studied that it can be beneficial to maintain a
number of projector layers and instead utilise intermediate projector
embeddings for downstream tasks. In our case, the preservation of capsule
layers for downstream tasks ensures that the desirable equivariant and part-
whole properties are maintained. Specifically, the equivariant information
that has shown to be captured in the object pose [27].

We evaluate the intermediate embeddings output from the primary capsule layer
in the same manner as the representations, however the activations and pose
are given over a spatial region we perform average pooling to return an
activation vector and a $4\times 4$ pose for each capsule which we then
flatten into a vector. As with the representation evaluation we report our
invariant and equivariant task performance in Tables 1 and 2 respectively. We
find across all settings that evaluating the intermediate embeddings of the
capsule projector leads to improved performance on all tasks. We observe that
classification via the activation vector (i.e. invariant part) significantly
improves on SIE ($\uparrow 0.7\;\text{Top-1}\%$) while approaching performance
levels of explicitly invariant approaches such as VICReg (see appendix C.2).
The same increase in performance is seen for the rotation prediction task,
where evaluating on the pose of the intermediate embedding (equivariant part)
leads to improved performance across all capsule based models, extending
beyond supervised training.

##  5 Ablations & Sensitivity Analysis

###  5.1 Number of Capsules

Each capsule, in theory, should represent a unique concept, thus when the
number of capsules is increased, logically so should the networks
representation ability to capture an increasing number of semantic concepts.
Observing the invariant performance during training (Figure 5) and downstream
evaluation in Table 1, CapsIE gains slight improvement with the addition of
more capsules. This demonstrates that our model has better utilised the
additional representational power to improve performance. This pattern is also
observed when evaluating equivariant properties (shown in Figure 5), yet is
less pronounced. However, in Table 1 we also show that increasing the number
of capsules of a supervised SR-Caps model trained in a standard supervised
fashion is not an indicator of increased performance, aligning with prior
capsule research [12]. This differentiation in behaviour provides an
interesting direction for future research.

###  5.2 Implicit Viewpoint Equivariance

Table 3: Evaluation of SIE and CapsIE - 32 downstream rotation prediction on
representations by either a rotation or colour hue equivariant objective.
Colour Prediction ($R^{2}$)
---
Method | All
SIE - Rotation | 0.43
SIE - Colour | 0.29 ($\downarrow$ 0.14)
CapsIE - Rotation | 0.59
CapsIE - Colour | 0.48 ($\downarrow$ 0.11)

To investigate whether capsule models are implicitly learning equivariant
properties without any explicit enforcing criterion, we train our CapsIE model
to predict colour hue rather than rotation via the predictor network. Given
the observation in Table 2 that CapsIE is able to achieve near-perfect
prediction without any optimising criterion, we hypothesise that capsules are
implicitly capable of learning equivariant properties. The evaluation
performance of representations on the rotation task under the aforementioned
ablation pre-training settings in Table 5, demonstrates that CapsNets indeed
learns more implicit equivariant properties than the benchmark SIE by a
considerable margin. We do however observe that SIE is still able to capture
some equivariant information without being explicitly trained for, which does
not align with the findings of the setting where colour is evaluated [14].
Notably, this results is not a significant improvement over random
representations in which the $R^{2}$ lies approximately at 0.25. This
observation presents interesting study for future work, while the improvement
of the CapsIE model over SIE empirically demonstrates the viewpoint
equivariant assumptions of CapsNets.

Figure 4: Classification evaluation performance (top-1 %) during training for
given varying number of capsules.

Figure 5: Rotation prediction evaluation performance ($R^{2}$) during training
for given varying number of capsules.

##  6 Related Work

###  6.1 Equivariant Self-Supervised Learning

Self-supervised learning has seen the majority of its success in the invariant
setting by either contrastive [7], information maximisation [33, 3], or
clustering methods [5, 1]. All families of approach rely on training a network
to be invariant to transformations by increasing the similarity between
embeddings of the same image under augmentation. The differing approaches
emerge from alternative methods to avoid collapse, the phenomena where
embeddings fall into a lower-dimensional subspace rather than the entire
available embedding space resulting in a trivial solution [18]. Although these
methods differ, they all produce similarly performing representations, hence
we employ information maximisation methods as the basis of this work due to
their computational efficiency.

Learning to be invariant to transformations is typically useful for semantic
discrimination tasks, yet preserving information about the transformations can
be highly beneficial. Some approaches have attempted to capture specific
information regarding transformations by predicting the applied augmentation
parameters [21], preserving the strength of augmentations [31] and introducing
rotational transformations [8]. However, as stated in [14], these methods
provide no guarantee that a mapping is learnt in the latent space that
reflects the transformations in the input space. Hence, methods have been
employed that address this limitation, [10, 26, 14] all employ predictor
networks to predict the displacement representations in the latent space given
one view representation and the transformation parameters. The latter, SIE
[14], is the basis of our work, which further extends prior methods by
splitting representation vectors into invariant and equivariant parts to
better separate differing information.

###  6.2 Capsule Networks

Figure 6: Objects with similar semantic concepts (i.e. bus and car) have
corresponding capsule activation/invariant embeddings. Top-3 capsule
activations are highlighted.

CapsNets present an alternative architecture to CNNs, addressing their
limitations by explicitly preserving hierarchical spatial relationships
between features [29]. CapsNets replace scalar neurons with vector or matrix
poses, representing specific concepts at different levels of a parse tree as
the network goes deeper. The first layer (primary capsules) correspond to the
most basic parts, while capsules in deeper layers represent more complex
concepts made up of the simpler concepts as they get closer to the final layer
where each capsule corresponds to a specific class (depicted in Figure 6).

The key components of the CapsNet are the pose and the activation. The pose of
a capsule is an embedding vector or matrix which provides a representation for
the concept. The activation scalar is a value between 0 and 1 which represents
how certain the network is that the concept is present and can be calculated
directly from the values of the pose or via other means via the routing
mechanism.

The key novelty in CapsNets is the routing mechanism, which determines the
contributions of lower-level capsules to higher-level capsules. Numerous
routing algorithms, both iterative and non-iterative, have been proposed to
address the efficiency and effectiveness of this process [29, 23, 13, 28, 17,
9, 32, 11]. Among these, SRCaps [15] introduces a non-iterative routing
mechanism. This method maintains all of the desirable properties of CapsNets,
such as equivariance while largely mitigating the time cost of iterative
methods, at the price of a small amount of performance. However, SRCaps faces
the same limitations as other CapsNets where high resolution or high class
datasets are beyond the network’s abilities when trained in a standard
fashion. For a more detailed description of capsule routing mechanisms please
check this review [27].

##  7 Conclusion

Our proposed method demonstrates how CapsNets can be employed in self-
supervised learning to better learn equivariant properties without the need to
explicitly split representation vectors and train separate projector networks.
The resulting solution, CapsIE, achieves state-of-the-art performance in
equivariant and invariant downstream benchmarks with a significant improvement
of 0.05 $R^{2}$ on prior self-supervised rotation prediction tasks and 0.2
$R^{2}$ improvement over the supervised baseline. In addition, we observe
competitive performance of CapsIE on equivariant tasks not explicitly trained
for, further demonstrating the implicit equivariant properties of our capsule
architecture under standard invariant optimisation criteria akin to those of
VICReg [3], SimCLR [7], and MSN [1]. Our results contribute significantly to
the avocation of CapsNets in self-supervised representation learning,
introducing desirable properties with improved effectiveness over MLP
projectors.

Limitations & Broader Impact. This work aims to learn higher quality and more
applicable representations of images without human generated annotations,
therefore such methods can lead to positive societal impacts the development
of more accurate or informative models for a number of downstream tasks.
However, as is the case with all vision systems, there is potential for
exploitation and security concerns and one should take into consideration AI
misuse when extending our method.

In addition, the use of CapsNets has shown little improvement when scaled,
similar to the findings of [12, 24, 25] where the addition of more capsule
layers has either led to stagnated or decreased performance. We strongly feel
that our work should be revisited when a capsule network that can both scale
and retain the desirable properties of capsule networks is found.

As stated in the original dataset proposal, the problem setting and
methodology presented relies on the group elements being known. Hence, the
applicability of the proposed method is only possible in settings where group
elements are known. However, we present findings where alternative group
elements are provided that suggests the CapsNets are more capable than
previous invariant methods at capturing equivariant properties, thus opening
an intriguing direction for future work.

## Acknowledgments and Disclosure of Funding

We would like to thank the University of Aberdeen’s HPC facility for enabling
this work.

## References

  * [1] M. Assran, M. Caron, I. Misra, P. Bojanowski, F. Bordes, P. Vincent, A. Joulin, M. Rabbat, and N. Ballas.  Masked siamese networks for label-efficient learning.  In European Conference on Computer Vision, pages 456–473. Springer, 2022.
  * [2] M. Assran, M. Caron, I. Misra, P. Bojanowski, A. Joulin, N. Ballas, and M. Rabbat.  Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8443–8452, 2021.
  * [3] A. Bardes, J. Ponce, and Y. LeCun.  Vicreg: Variance-invariance-covariance regularization for self-supervised learning.  arXiv preprint arXiv:2105.04906, 2021.
  * [4] F. Bordes, R. Balestriero, Q. Garrido, A. Bardes, and P. Vincent.  Guillotine regularization: Improving deep networks generalization by removing their head.  arXiv preprint arXiv:2206.13378, 2022.
  * [5] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin.  Emerging properties in self-supervised vision transformers.  arXiv preprint arXiv:2104.14294, 2021.
  * [6] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  * [7] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [8] R. Dangovski, L. Jing, C. Loh, S. Han, A. Srivastava, B. Cheung, P. Agrawal, and M. Soljačić.  Equivariant self-supervised learning: Encouraging equivariance in representations.  In International Conference on Learning Representations, 2022.
  * [9] F. De Sousa Ribeiro, G. Leontidis, and S. Kollias.  Introducing routing uncertainty in capsule networks.  Advances in Neural Information Processing Systems, 33:6490–6502, 2020.
  * [10] A. Devillers and M. Lefort.  Equimod: An equivariance module to improve self-supervised learning.  arXiv preprint arXiv:2211.01244, 2022.
  * [11] M. Everett, M. Zhong, and G. Leontidis.  Masked capsule autoencoders.  arXiv preprint arXiv:2403.04724, 2024.
  * [12] M. A. Everett, M. Zhong, and G. Leontidis.  Protocaps: A fast and non-iterative capsule network routing method.  Transactions on Machine Learning Research, 2023.
  * [13] Y. Feng, J. Gao, and C. Xu.  Spatiotemporal orthogonal projection capsule network for incremental few-shot action recognition.  IEEE Transactions on Multimedia, 2024.
  * [14] Q. Garrido, L. Najman, and Y. Lecun.  Self-supervised learning of split invariant equivariant representations.  arXiv preprint arXiv:2302.10283, 2023.
  * [15] T. Hahn, M. Pyeon, and G. Kim.  Self-routing capsule networks.  Advances in neural information processing systems, 32, 2019.
  * [16] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [17] G. E. Hinton, S. Sabour, and N. Frosst.  Matrix capsules with em routing.  In International conference on learning representations, 2018.
  * [18] T. Hua, W. Wang, Z. Xue, S. Ren, Y. Wang, and H. Zhao.  On feature decorrelation in self-supervised learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9598–9608, 2021.
  * [19] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [20] Y. LeCun, F. J. Huang, L. Bottou, et al.  Learning methods for generic object recognition with invariance to pose and lighting.  In CVPR (2), pages 97–104. Citeseer, 2004.
  * [21] H. Lee, K. Lee, K. Lee, H. Lee, and J. Shin.  Improving transferability of representations via augmentation-aware self-supervision.  Advances in Neural Information Processing Systems, 34:17710–17722, 2021.
  * [22] Y. Liu, D. Cheng, D. Zhang, S. Xu, and J. Han.  Capsule networks with residual pose routing.  IEEE Transactions on Neural Networks and Learning Systems, 2024\.
  * [23] V. Mazzia, F. Salvetti, and M. Chiaberge.  Efficient-capsnet: Capsule network with self-attention routing.  Scientific reports, 11(1):14634, 2021.
  * [24] M. Mitterreiter, M. Koch, J. Giesen, and S. Laue.  Why capsule neural networks do not scale: Challenging the dynamic parse-tree assumption, 2023.
  * [25] P. Nair, R. Doshi, and S. Keselj.  Pushing the limits of capsule networks.  arXiv preprint arXiv:2103.08074, 2021.
  * [26] J. Y. Park, O. Biza, L. Zhao, J. W. van de Meent, and R. Walters.  Learning symmetric embeddings for equivariant world models.  arXiv preprint arXiv:2204.11371, 2022.
  * [27] F. D. S. Ribeiro, K. Duarte, M. Everett, G. Leontidis, and M. Shah.  Learning with capsules: A survey.  arXiv preprint arXiv:2206.02664, 2022.
  * [28] F. D. S. Ribeiro, G. Leontidis, and S. Kollias.  Capsule routing via variational bayes.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3749–3756, 2020.
  * [29] S. Sabour, N. Frosst, and G. E. Hinton.  Dynamic routing between capsules.  Advances in neural information processing systems, 30, 2017.
  * [30] R. Winter, M. Bertolini, T. Le, F. Noe, and D.-A. Clevert.  Unsupervised learning of group invariant and equivariant representations.  Advances in Neural Information Processing Systems, 35:31942–31956, 2022.
  * [31] Y. Xie, J. Wen, K. W. Lau, Y. A. U. Rehman, and J. Shen.  What should be equivariant in self-supervised learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4111–4120, 2022.
  * [32] J. Yang, P. Zhao, Y. Rong, C. Yan, C. Li, H. Ma, and J. Huang.  Hierarchical graph capsule network.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10603–10611, 2021.
  * [33] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  arXiv preprint arXiv:2103.03230, 2021.

##  Appendix A 3DIEBench Dataset

Figure 7: The 3DIEBench dataset, one 3d model is used to create 50 different
views in a synthetic environment, which are saved as images along with the
latent values by which they are transformed.

Typical equivariant datasets are generally handcrafted and simple, with a
small amount of classes and instances within each class. This is due to the
time needed in order to ensure correctness. While standard image datasets do
allow for testing invariance in the form of augmenting the same image in two
different ways, they do not allow for precise transformation of the subject.
Thus the need for a new, synthetic dataset.

We use the 3DIEBench [14] dataset111The full dataset and splits employed can
be found at https://github.com/facebookresearch/SIE, which has been created
specifically to be a hard yet controlled test-bed for invariant and
equivariant methods. The dataset consists of 52,472 3d objects across 55
classes of 3d objects from ShapeNetCorev2 [6] posed in 50 different views as
well as the latent information of the view, this can be seen in figure 7. For
training we then randomly select two views from each model in the training
set. The parameters by which the model could have been augmented can be seen
in table 4.

Table 4: Values of the factors of variation used for the generation of 3DIEBench. Each value is sampled uniformly from the given interval. Object rotation is generated as Tayt-Bryan angles using extrinsic rotations. Light position is expressed in spherical coordinates. This table is sourced from [14]. Parameter | Minimum value | Maximum value
---|---|---
Object rotation X | $-\frac{\pi}{2}$ | $\frac{\pi}{2}$
Object rotation Y | $-\frac{\pi}{2}$ | $\frac{\pi}{2}$
Object rotation Z | $-\frac{\pi}{2}$ | $\frac{\pi}{2}$
Floor hue | 0 | 1
Light hue | 0 | 1
Light $\theta$ | $0$ | $\frac{\pi}{4}$
Light $\phi$ | $0$ | $2\pi$

##  Appendix B Training protocols

###  B.1 CapsIE Pre-training

Our proposed CapIE model is comprised of a ResNet-18 encoder, SR-CapsNet
comprised of a primary capsule layer routed to a second capsule layer. The SR-
CapsNet projector takes as input the activation map output of the ResNet prior
to the final global average pooling. The predictor network employed is that
described in [14]. All details of the architectural design are given in the
main paper.

Training of our CapsIE model is done over 2000 epochs with a batch size of
1024, optimised via the Adam optimiser with learning rate 0.001, and default
parameters, $\beta_{1}=0.9,\beta_{2}=0.999$. By default the objective function
weighting are as follows,
$\lambda_{\text{inv}}=0.1,\;\lambda_{\text{equi}}=5,\;\lambda_{V}=10,\;\lambda_{C}=1$
where we empirically found these optimal for our setting. Further performance
gains could be achieved by the tuning of such parameters, however, we deemed
this unnecessary.

For ablation studies, where we explicitly state, we train for fewer epochs and
with a smaller batch size, 500 and 512 respectively. We find in practice this
setting is a strong proxy for full training performance and significantly save
computational resource.

Training time for 2000 epochs with batch size of 1024, as previously stated,
took approximately 22 hours using 3 Nvidia A100 80GB GPU’s, with 64 capsule
models taking approximately 25 hours using 6 Nvidia A100 80GB GPU’s.

###  B.2 Downstream Evaluation

In our work we perform evaluation on both the frozen ResNet-18 representations
and the representations from the primary capsules layer, which are evaluated
using either a linear classifier or additional capsule layer acting as a class
capsules, i.e the number of capsules is set to the number of classes and
activations are used as the logits. Here, we detail the exact training
protocols to ensure complete reproducability.

For our evaluations we use two different depths of MLP heads, these are: 1.
Deep MLP referring to an MLP with layers containing in_dim - 1024 - out_dim
neurons, with intermediate ReLU activations. 2. Shallow MLP referring to a
single MLP layer with in_dim in neurons and out_dim out neurons. When we
evaluate our primary capsules for angle and colour prediction, we average the
8x8 feature map so that we only have a single pose vector for the entire
image. For our Capsule Classification task, we do not have an in_dim as we do
not use a MLP, but instead use a capsule layer which operates on the primary
capsules pose and activations.

Table 5: Training settings for our evaluations. Settings are the same for all number of capsules. NC is used as shorthand for number of capsules. - denotes that this element is not used. * denotes multiplication. | Representations | Representations | Representations | Capsule | Capsule | Capsule
---|---|---|---|---|---|---
| Angle | Colour | Classification | Angle | Colour | Classification
Caps Head | - | - | - | - | - | Yes
MLP Head | Deep | Shallow | - | Deep | Shallow | -
in_dim | 512 | 512 | 512 |  NC * 16 |  NC * 16 | N/A
out_dim | 4 | 2 | 55 | 4 | 2 | 55
Optimizer | Adam | Adam | Adam | Adam | Adam | Adam
LR | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001
$\beta_{1}$ | 0.9 | 0.9 | 0.9 | 0.9 | 0.9 | 0.9
$\beta_{2}$ | 0.999 | 0.999 | 0.999 | 0.999 | 0.999 | 0.999
Batch Size | 256 | 256 | 64 | 256 | 256 | 256
Epochs | 300 | 50 | 300 | 300 | 50 | 300
Objective | MSE | MSE | Cross Entropy | MSE | MSE | Cross Entropy

###  B.3 Supervised Training of SR-Caps

In our work we train a Self Routing Capsule Network model in a supervised
fashion for the downstream tasks to evaluate whether our pretrained model
improves the quality of downstream evaluations. The training setting of these
runs can be found in table 6.

Deep MLP refers to an MLP with layers containing number_caps * 16 * 2 - 1024 -
4 neurons, with intermediate ReLU activations. Shallow MLP head refers to a
single MLP layer with number_caps * 16 * 2 in neurons and either 4 (for
rotation prediction) or 2 (for colour prediction) out neurons.

Table 6: Training settings for our supervised Self Routing Capsule Network model. Settings are the same for all number of capsules. - denotes that this element is not used. | Angle | Colour | Classification |
---|---|---|---|---
Caps Head | - | - | Yes |
MLP Head | Deep | Shallow | - |
Optimizer | Adam | Adam | Adam |
LR | 0.001 | 0.001 | 0.001 |
$\beta_{1}$ | 0.9 | 0.9 | 0.9 |
$\beta_{2}$ | 0.999 | 0.999 | 0.999 |
Batch Size | 256 | 256 | 64 |
Epochs | 300 | 50 | 300 |
Objective | MSE | MSE | Cross Entropy |

##  Appendix C Further Experimentation

###  C.1 Capsule Downstream Classification on Embeddings.

Table 7 provides the downstream classification evaluation for the frozen
output embeddings of the CapsNet projection head. The drop in performance is
an expected result inline with that demonstrated by [7, 4]. These results
signify the importance of the projection head and specifically its role in
decorrelating the embeddings from representations to avoid overfitting to the
pre-training objective. We do note however, that our observed performance drop
is inline or slightly less than that reported in [14].

Table 7: Capsule downstream invariance evaluation on projection head embeddings. | Embedding Dims | Classification (Top-1%)
---|---|---
Method | Inv. | Equi. | Representations | Embeddings
Supervised | - | - | 87.47 |
SIE | 512 | 512 | 82.94 |
Capsule Projector Naive Evaluation
CapsIE - 16 | 16 | 256 | 78.96 | 65.83
CapsIE - 32 | 32 | 512 | 80.00 | 69.12
CapsIE - 64 | 64 | 1024 | 80.26 | 56.64

###  C.2 Invariant and Equivariant SSL Benchmarks

We report below the classification (invariant, Table 8), rotation prediction,
and colour prediction (equivariant, Table 9) performance of baseline self-
supervised methods. The below baseline results are acquired from [14], with
exception to those denoted by ‘*’ which corresponds to our re-implementation.

Table 8: Evaluation of invariant properties on downstream classification task for baseline SSL methods. We evaluate both the representations and the intermediate embeddings of the projection head when different numbers of capsules in the projection head is used. ‘-’ refers to non-compatible experiments. | Embedding Dims | Classification (Top-1%)
---|---|---
Method | Inv. | Equi. | All | Inv. | Equi.
Encoder Representation |  |  |
VICReg | - | - | 84.74 | - | -
VICReg, $g$ kept identical | - | - | 72.81 | - | -
SimCLR | - | - | 86.73 | - | -
SimCLR, $g$ kept identical | - | - | 71.21 | - | -
SimCLR + AugSelf | - | - | 85.11 | - | -
EquiMod (Original predictor) | - | - | 87.19 | - | -
EquiMod (SIE predictor) | - | - | 87.19 | - | -
SIE [14] | 512 | 512 | 82.94 | 82.08 | 80.32
SIE * | 512 | 512 | 82.54 | 82.11 | 80.74
CapsIE - 16 | 16 | 256 | 78.96 | - | -
CapsIE - 32 | 32 | 512 | 80.00 | - | -
CapsIE - 64 | 64 | 1024 | 80.26 | - | -
Capsule Projector - 1st Intermediate Embedding |  |  |
CapsIE - 16 | 16 | 256 | - | 82.96 | -
CapsIE - 32 | 32 | 512 | - | 83.49 | -
CapsIE - 64 | 64 | 1024 | - | 83.64 | -
Table 9: Evaluation of equivariant properties on downstream rotation prediction (left) and colour prediction (right) tasks for baseline SSL methods. We evaluate both the representations and the intermediate embeddings of the projection head when different numbers of capsules in the projection head is used. | Rotation Prediction ($R^{2}$) | Colour Prediction ($R^{2}$)
---|---|---
Method | All | Inv. | Equi. | All | Inv. | Equi.
Encoder Representation |  |  |
VICReg | 0.41 | - | - | 0.06 | - | -
VICReg, $g$ kept identical | 0.56 | - | - | 0.25 | - | -
SimCLR | 0.50 | - | - | 0.30 | - | -
SimCLR, $g$ kept identical | 0.54 | - | - | 0.83 | - | -
SimCLR + AugSelf | 0.75 | - | - | 0.12 | - | -
EquiMod (Original predictor) | 0.47 | - | - | 0.21 | - | -
EquiMod (SIE predictor) | 0.60 | - | - | 0.13 | - | -
SIE [14] | 0.73 | 0.23 | 0.73 | 0.07 | 0.05 | 0.02
SIE * | 0.72 | 0.21 | 0.71 | 0.06 | 0.05 | 0.03
CapsIE - 16 | 0.78 | - | - | 0.97 | - | -
CapsIE - 32 | 0.75 | - | - | 0.97 | - | -
CapsIE - 64 | 0.72 | - | - | 0.97 | - | -
Projector - 1st Intermediate Embedding |  |  |
CapsIE - 16 | - | - | 0.78 | - | - | 0.97
CapsIE - 32 | - | - | 0.77 | - | - | 0.97
CapsIE - 64 | - | - | 0.78 | - | - | 0.97