Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Perceiver IO's decoding procedure uses an attention mechanism to map from latents to arbitrarily sized and structured outputs using a querying system that can flexibly specify the semantics needed for outputs on a wide range of domains, including dense and multitask settings.

The review comment to be evaluated is:

#### Strength
- The idea of perceiver IO is novel and solid -- a general architecture capable of handling general-purpose inputs and outputs across different tasks and modalities. This is very promising to simplify the construction of highly tuned task-specific neural pipelines and improve the multimodal and multi-task problems.

- Each component in perceiver IO is necessary and well defined for the proposed tasks. 2D byte array input enables the task agnostic input for different modalities. Perceiver network that scales linearly with the size of input and output, and non-auto regressive decoding make the decoding to raw output possible.

- The proposed architecture is tested on massive experiments including language understanding tasks, optical flow, video audio class autoencoding, image classification, and starcraft II and achieves superior performance. Each task is supported with a detailed ablation study to shed light on future research.

#### Weakness
- In Table 1, the WNLI task is excluded from the GLUE benchmark, I wonder what is the reason this task is removed?

- The non-autoregressive decoding enables fast decoding for large outputs. However, for tasks that require more information on prior decoding context (machine translation or image generation tasks), does the proposed model can still perform well on those tasks?

- Although perceiver IO removes the task-specific pre-processing (tokenization, patch embedding, etc.), the model still requires huge engineering efforts to adapt the model for different tasks. For example, the model hyper-parameters are quite different for different tasks.

- One huge benefit of perceiver IO is to train different tasks together and explore the transfer between different tasks/modalities, which is not explored in this paper.

The evidence to be evaluated is:Several groups have proposed to use attention to manipulate the size of arrays or to introduce bottlenecks in processing. Set Transformers and related work (Lee et al., 2019; Goyal et al., 2022) use a learned query ("inducing points") to induce local bottlenecks by mapping a set back and forth from a set with fewer elements and learned decoder queries ("seed vectors") to map to outputs ("pooling by multiheaded attention").

Several works use attention to process latent spaces that interface with input/output data using task- or domain-specific architectures (Carion et al., 2020; Locatello et al., 2020; Wang et al., 2021), and cross-attention itself is widely used to produce outputs in of a different size or structure from inputs (Dai et al., 2019; Desai and Johnson, 2021; Miech et al., 2021; Vaswani et al., 2017; Raffel et al., 2020; Santoro et al., 2018; Hudson and Zitnick, 2021; Ma et al., 2021). Perceiver IO builds on this body of work to produce a general purpose architecture that can be easily and widely applied.

Rather than output a single category, Perceiver IO aims to have the same level of generality with respect to its _outputs_ as the Perceiver has with respect to its _inputs_: that is, it should produce arbitrary output arrays. We can predict each element of the output array using another attention module by _querying_ the latent array using a query feature vector unique to the desired output element. In other words, we define a query array with the same number of elements as the desired output. The queries may be hand-designed, learned embeddings, or a simple function of the input. They attend to the latents to yield an output array of the desired shape.

Finally, we **decode** by applying an attention module that maps latent arrays to output arrays \(y\in\mathbb{R}^{O\times E}\).

Each of these modules applies a global query-key-value (QKV) attention operation followed by a multi-layer perceptron (MLP).

Both encoder and decoder take in two input arrays, the first used as input to the module's key and value networks, and the second used as input to the module's query network. The module's output has the same index dimension (the same number of elements) as the query input.

Perceiver IO uses attention non-homogeneously by mapping inputs to a latent space, processing in that latent space, and decoding to an output space. Perceiver IO has no quadratic dependence on the input or output size: encoder and decoder attention modules depend linearly on the input and output size (respectively), while latent attention is independent of both input and output sizes (Sec. E.2).

This architecture can be applied to inputs of any shape or spatial layout including inputs or outputs with different spatial structure (e.g. sound and video).

To decode this information, we query for it using cross-attention.

We produce an output of this size by querying the decoder with an array of index dimension \(O\). To capture the structure of the output space, we use queries containing the appropriate information for each output point, e.g. its spatial position or its modality. We construct queries by combining (concatenating or adding) a set of vectors into a query vector containing all of the information relevant for one of the \(O\) desired outputs.

We illustrate the query structure for the tasks we consider here in Fig. 3. For tasks with simple outputs, such as classification, these queries can be reused for every example and can be learned from scratch. For outputs with a spatial or sequence structure, we include a position encoding (e.g. a learned positional encoding or a Fourier feature) representing the position to be decoded in the output. For outputs with a multi-task or multimodal structure, we learn a single query for each task or for each modality: this information allows the network to distinguish one task or modality query from the others, much as positional encodings allow attention to distinguish one position from another. For other tasks, the output should reflect the content of the input at the query location: for instance, for flow we find it helpful to include the input feature at the point being queried, and for StarCraft II we use the unit information to associate the model's output with the corresponding unit. We find that even very simple query features can produce good results, suggesting that the latent attention process is able to learn to organize the relevant information in a way that's easy to query. Each output point depends only on its query and the latent array, allowing us to decode outputs in parallel. This property allows us to amortize model training on datasets of very large output size.

To probe the generality of Perceiver IO, we evaluate it on several domains including language understanding (Wikipedia+C4 masked language modeling), visual understanding (Sintel/KITTI optical flow and ImageNet classification), multi-modal (Kinetics autoencoding and AudioSet classification) & multi-task settings (multi-task GLUE), and symbolic representations for games (StarCraft II).

But unlike Transformer-based models such as BERT (Devlin et al., 2019) or XLNet (Yang et al., 2019), Perceiver IO scales linearly with input length.

Perceiver IO produces one output vector per masked input by using learnable position-dependent vectors to query the output of the final latent processing layer.

We first observe that Perceiver IO applied on SentencePiece tokenized input sequences slightly outperforms a strong BERT baseline applied on the same inputs (81.2 vs 81.1).

**Multitask Perceiver IO.** We use multitask queries as described in Sec. 3.2 to finetune on all 8 GLUE tasks simultaneously using the UTF-8 byte model (results in Tab. 2).

We observe that our multitask approach outperforms single-task approaches and matches the approach that uses 8 task-specific input tokens. Our approach is more generic as it decouples the output array from the input array by not relying on [CLS] tokens.

To decode, we query the latent representation using the input encoding.

We explore using Perceiver IO for audio-video-label multimodal autoencoding on the Kinetics-700-2020 dataset (Smaira et al., 2020). The goal of multimodal autoencoding is to learn a model that can accurately reconstruct multimodal inputs in the the presence of a bottleneck induced by an architecture.

With Perceiver IO, we pad the inputs with modality-specific embeddings, serialize them into a single 2D input array and query outputs using queries containing position encodings (for video and audio) and modality embeddings.

We decode directly into pixels, raw audio, and the one-hot label without any post-processing.

This strongly suggests that Perceiver IO can jointly represent modalities with very different properties.

(1) on ImageNet, Perceiver IO surpasses 80% top-1 accuracy (84.5% top-1) without using 2D convolutions after pretraining on JFT.

In this work we introduce Perceiver IO, an architecture capable of handling general purpose inputs and outputs while scaling linearly in both input and output sizes. As we show, this architecture achieves good results in a wide variety of settings, making it a promising candidate for a _general purpose_ neural network architecture. Perceiver IO leverages the expressive power of latent attention and uses learned queries to expose a simple and unified interface that can handle multimodal and multitask settings. Overall, Perceiver IO offers a promising way to simplify the construction of sophisticated neural pipelines and facilitate progress on multimodal and multitask problems.

For ease of reference and comparison across domains, we describe the input and output size and processing used in all experiments in Tab. 5 and provide details of input key/value, position encoding, and output queries used in all experiments in Tab. 6. On all domains but StarCraft II, we include experiments with several input configurations, ranging from no domain adaptation (e.g. tokenizer-free language, flow from raw pixels, ImageNet with no convolutional or patch-based preprocessing and fully learned position encodings) to moderate domain adaptation (e.g. SentencePiece language understanding, flow from conv+maxpool-processed images and with RAFT upsampling, ImageNet with conv+maxpool-preprocessing and 2D Fourier features). These results demonstrate the unprecedented generality of Perceiver IO, the simplicity that this architecture unlocks in handling a range of tasks, and its flexibility to work as part of a domain-adapted system.

We use an architecture with weight sharing in depth: the latent (processing) component of the architecture includes 8 blocks of 6 attention modules each, and weights are shared between the corresponding modules in each block.

At its core, AlphaStar represents the units in the game as a discrete, unordered set of symbols (the "units"). These units are represented by a vector of properties including unit type, position, and health. At each timestep, the architecture encodes units with an entity encoder, which in the original model was parameterized using a vanilla Transformer. The entity encoder takes as input a set of 512 entities (referred to as embedded_entity in Vinyals et al. (2019)) and produces as output an embedding for each entity (entity_embeddings) and a 1D embedding reduced over entities (embedded_entity). These 512 entities represent the units and other entities that are present in the game: unused entity slots are masked. entity_embeddings is produced by passing the outputs of the entity encoder through a ReLU and a 1D convolution with 256 channels.

For this reason, we view AlphaStar as an important test case for Perceiver IO's ability to function as a general-purpose tool for processing symbolic or set-valued data. If the question is "can Perceiver IO serve as a replacement for a well-tuned Transformer as a symbolic processing engine?" then the answer is yes: We obtained StarCraft II results by using Perceiver IO instead of a Transformer for the AlphaStar entity encoder. We replaced the Transformer with a Perceiver IO with a latent of index dimension 32, keeping the input and output size of 512 units.

We seek to confirm that the the attention-based decoder helps even on classification, where the original Perceiver's decoder could be used.

As in the ImageNet experiments, We compare the performance of Perceiver and Perceiver IO using models that are matched except for the decoder (we use an average + project decoder for Perceiver and a query-based attention decoder for Perceiver IO, see Sec. E.3 and Fig. 6).

We find that as in the ImageNet experiments, using the attention-based decoder leads to small but consistent improvements over the less generally applicable average + project decoder. Because Perceiver IO introduces no domain assumptions not present in the original Perceiver, this is evidence that Perceiver IO is a strictly more general model.

Perceiver IO is constructed from GPT-2-style (Radford et al., 2019) Transformer attention modules, which consist of QKV attention followed by an MLP, along with linear projection layers to ensure inputs to and outputs from the QKV attention and MLP take on desired sizes. Using the array sizes of the encoder attention, the QKV attention takes in two two-dimensional arrays, a key-value input array \(X_{KV}\in\mathbb{R}^{M\times C}\) and a query input array \(X_{Q}\in\mathbb{R}^{N\times D}\), and maps them to an array \(X_{QKV}\in\mathbb{R}^{N\times D}\), sharing the shape of the query input (after projection).

The view of attention that we describe encompasses both cross-attention and self-attention, both of which are specific ways of using QKV-attention. Perceiver IO uses cross-attention for encoder and decoder attention modules and uses self-attention for the latent processing modules.

QKV attention takes in two two-dimensional arrays, a query input \(X_{Q}\in\mathbb{R}^{N\times D}\) and a key-value input \(X_{KV}\in\mathbb{R}^{M\times C}\). The output of QKV attention is an array with the same index (first) dimension as the query input and a channel (second) dimension determined by an output projection: \[Q=f_{Q}(X_{Q}); K=f_{K}(X_{KV}); V=f_{V}(X_{KV}) \tag{1}\] \[X_{QK}=\text{softmax}(QK^{T}/\sqrt{F})\] (2) \[\text{Attn}(X_{Q},X_{KV})=X_{QKV}=f_{O}(X_{QK}V), \tag{3}\] where \(X_{QK}\) is an array of attention maps \(\in\mathbb{R}^{N\times M}\), and \(X_{QKV}\) is an array \(\in\mathbb{R}^{N\times D}\). The functions \(f_{\{Q,K,V\}}\) are linear layers mapping each input to a shared feature dimension \(F\) and \(f_{O}\) is a linear layer projecting the output to a target channel dimension, which is often the same size as \(X_{Q}\)'s.

\[X_{QKV} =\text{Attn}(\text{layerNorm}(X_{Q}),\text{layerNorm}(X_{KV})) \tag{4}\] \[X_{QKV} =X_{QKV}+X_{Q}\] (5) \[X_{QKV} =X_{QKV}+\text{MLP}(\text{layerNorm}(X_{QKV})), \tag{6}\]

"Attn" refers to QKV as described above.

In other words, Perceiver IO has complexity linear in the size of the input and output arrays and it decouples the depth of the latent transformer from the input and output sizes. Both of these properties contribute to Perceiver IO's efficiency: while many proposals for efficient attention modules or architectures include linear or sub-quadratic scaling with input/output size, Perceiver IO is unusual in also decoupling depth from input/output size (without requiring domain-specific strategies like 2D convolution).

As we show in ImageNet and AudioSet experiments, the attentional decoder used here can be used in settings where standard average + project decoders are applicable. We find that the attentional decoder typically produces somewhat better results than the standard decoder. This likely occurs because attentional decoding is more expressive than average + project decoding.

Both decoders can be viewed as first averaging thelations and then projecting them to a target shape, but decoder attention uses more expressive modules for each of these operations. Instead of uniformly weighting each input in the averaging operation, decoder attention uses the attention scores as data-dependent weights for each input point. Instead of projecting the raw averaged input to a target dimensionality, decoder attention first projects inputs via a value layer and then processes them with an MLP. In addition to its greater expressivity, decoder attention has the advantage of being easily generalizable to dense outputs (by increasing the number of queries) and of reusing the same architectural pattern used for the encoder and processor modules.

In order to decode, we use learned queries of the same dimension of the input array (Tab. 11). We have as many output queries as inputs to be able to predict the masked token at all positions in the sentence (\(M\)=\(O\)). To get an insight into the learnt queries we visualize the attention weights in the first cross attention layer on a small paragraph (Fig. 7). We discover that the model has learnt both position and content based look-ups. The position-based look-ups can be either very sparse and precise or more distributed and periodic. This second mode appears somewhat less often and is more efficient because more data is being attended to at the same time, but also more distributed, since the values are subsequently averaged:

For single-task experiments, we do not require a [CLS] token as we use a single decoding query vector. In both single-task and multi-task experiments an extra 2-layer MLP with a hidden size of \(E\) and a tanh activation is used to map the the Perceiver IO outputs to the class logits (or regression target for STS-B).

After feeding this through the Perceiver IO architecture (including querying with the same convolutional features used as input), we have an output a feature grid with stride 4 and 64 channels, on top of which we apply a RAFT upsampling layer.

The decoder queries are also constructed from Fourier position embeddings for video and audio and a learned positional embedding for label: 387 features for video, 385 features for audio, and 1024 learned features for the label. We pad the queries for each modality with a different learned vector for each modality, so that the final feature size for the queries is 1026.

Perceiver IO is able to capture the structure of both video and audio inputs, despite encoding both jointly with a single network.

Input: Figure 1: The Perceiver IO architecture can be used on domains with a wide variety of input and output spaces, including multi-task language understanding, dense visual tasks like optical flow, hybrid dense/sparse multimodal tasks such as video+audio+class autoencoding, and tasks with symbolic outputs like StarCraft II. See Tables 5 and 6 for details of all domains considered here.

Input: Figure 5: Schematic depiction of encode, process, and decode attention. Each attention module uses the same operations, but differs in which inputs are used to generate key/values or queries and in the output shape. Encode attention can be viewed as mapping an input to a latent space, typically with a smaller index dimension (fewer elements). Decode attention can be viewed as mapping a latent to an output space, often with a larger index dimension (more elements). Both of these are forms of cross-attention. Process attention (self-attention) preserves the input index dimension (same elements). Red and blue dashed lines are used to highlight the two matrix multiplications used in QKV attention, as described in the text.

Input: Figure 2: The Perceiver IO architecture. Perceiver IO maps arbitrary input arrays to arbitrary output arrays in a domain agnostic process. The bulk of the computation happens in a latent space whose size is typically smaller than the inputs and outputs, which makes the process computationally tractable even for very large inputs & outputs. See Fig. 5 for a more detailed look at encode, process, and decode attention.

Input: Table 5: Details of each of the tasks we use to evaluate Perceiver IO here. The positional and task embeddings appended to inputs for each case are listed in Tab. 6.

Input: Table 6: Table best viewed on a screen. The structure and size of the positional and task embeddings used to construct Perceiver IO’s encoder key-value inputs and decoder query inputs, for each domain described in the main text. “[x, y]” indicates that x’s and y’s features are concatenated, while “x + y” indicates that x’s and y’s features are added to produce the full featurization. “FF” = Fourier features as in Jaegle et al. (2021).

Input: Figure 6: Single-query attention decoder (left), as used in Perceiver IO for classification tasks and a standard average + project decoder (right), as used in Jaegle et al. (2021). Both modules can be seen as first aggregating latents by weighted averaging (learned, data-dependent weighting for the attention decoder; uniform weights for the average + project decoder) and then projecting to an output channel dimension (linear value projection + MLP for the attention decoder; simple linear projection by the average + project decoder). Attentional decoding is more expressive than average + project decoding and follows the same architectural template as encoder and processor modules.

Input: Figure 4: Multimodal audio-video-label autoencoding with 88x compression. Side-by-side: inputs on left, reconstructions right. See the supplemental material for example output video and audio.

Input: Figure 3: We construct queries with output-specific features to produce outputs with different semantics. For settings where each output point differs only in its position, like language, a position embedding can be used. Input features for the target output can also be used to query, either alone (as for StarCraft II) or alongside position features (as for flow). For multi-{task, modal} settings we use one embedding for each {task, modality} instead of each position. A single learned embedding suffices for simple classification tasks, like ImageNet. For tasks with heterogeneous outputs like multimodal autoencoding, features that are specific to some queries (like xy position) can be combined with modality embeddings, which also pad embeddings to fixed length.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim accurately describes Perceiver IO's decoding mechanism and its capabilities. The evidence extensively documents how the architecture uses attention-based querying to flexibly map latent representations to outputs of arbitrary size and structure across diverse domains. The technical details, experimental results, and architectural diagrams all support the claim's statements about the querying system's flexibility and wide domain applicability. The claim is well-stated without exaggeration, as it aligns closely with the demonstrated capabilities and implementation details provided in the evidence.

<score>1</score>

Justification:
- The evidence provides detailed explanations of how the attention-based decoder works with query vectors to specify output semantics (Figures 2, 3, 5)
- Multiple examples demonstrate successful application across domains including language, vision, audio, and games
- The reviewer comments reinforce the architecture's generality and capability to handle different tasks/modalities
- The claim's scope matches what is demonstrated in the extensive experimental results
- No overstatement of capabilities - the claim accurately reflects what is shown in the evidence


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

This decoder allows Perceiver IO to serve as a drop-in replacement for a wide range of specialist networks currently in use on a set of challenging domains, while improving performance on tasks like classification that could be handled by the Perceiver.

The evidence to be evaluated is:Similar to our work, several groups have proposed to solve tasks in multiple domains (e.g. Kaiser et al. 2017; Alyraca et al. 2020; Akbari et al. 2021), but typically across a fixed and predefined set of modalities by means of domain-specific networks. Although single-task specialist networks remain dominant in vision, multi-task learning has become popular (Misra et al., 2016; Doersch and Zisserman, 2017; Kokkinos, 2017; Zamir et al., 2018) and individual models achieve generality in a restricted domain: e.g. Mask-RCNN (He et al., 2017) handles object detection, segmentation, and pose estimation.

Several groups have demonstrated that Transformers (originally designed for language) can be used or adapted to non-language tasks (e.g. Chen et al. 2020; Lu et al. 2021), but the limited scalability of Transformers limits their usefulness as general-purpose architectures.

The focus of our work is developing an architecture that is efficient and also performs well in many settings with a wide range of inputs and outputs.

Perceiver IO builds on this body of work to produce a general purpose architecture that can be easily and widely applied. The Perceiver IO architecture builds on the Perceiver (Jaegle et al., 2021), which achieved its cross-domain generality by assuming that its input is a simple 2D byte array: a set of elements (which might be pixels or patches in vision, characters or words in language, or some form of embedding, learned or otherwise), each described by a feature vector. The model then encodes information about the input array using a smaller number of latent feature vectors, using Transformer-style attention, followed by iterative processing and a final aggregation down to a category label. Rather than output a single category, Perceiver IO aims to have the same level of generality with respect to its _outputs_ as the Perceiver has with respect to its _inputs_: that is, it should produce arbitrary output arrays.

The queries may be hand-designed, learned embeddings, or a simple function of the input. They attend to the latents to yield an output array of the desired shape.

We first **encode** by applying an attention module that maps input arrays \(x\in\mathbb{R}^{M\times C}\) to arrays in a latent space \(z\in\mathbb{R}^{N\times D}\). We next **process** the latents \(z\) by applying a series of modules that take in and return arrays in this latent space. Finally, we **decode** by applying an attention module that maps latent arrays to output arrays \(y\in\mathbb{R}^{O\times E}\). \(M\), \(C\), \(O\), and \(E\) are properties of the task data and can be very large (Tab. 5), while \(N\) and \(D\) are hyperparameters and can be chosen to make model computation tractable. Following the design of the Perceiver, we implement each of the architecture's components using Transformer-style attention modules. Each of these modules applies a global query-key-value (QKV) attention operation followed by a multi-layer perceptron (MLP).

The Perceiver IO architecture builds on primitives similar to those in Transformers.

Perceiver IO uses attention non-homogeneously by mapping inputs to a latent space, processing in that latent space, and decoding to an output space. Perceiver IO has no quadratic dependence on the input or output size: encoder and decoder attention modules depend linearly on the input and output size (respectively), while latent attention is independent of both input and output sizes (Sec. E.2). Because of the corresponding reduction in compute and memory requirements, Perceiver IO scales to much larger inputs and outputs. While Transformers are typically used in settings with data preprocessed to contain at most a few thousand dimensions (Brown et al., 2020; Raffel et al., 2020), we show good results on domains with hundreds of thousands of dimensions. This architecture can be applied to inputs of any shape or spatial layout including inputs or outputs with different spatial structure (e.g. sound and video).

To decode this information, we query for it using cross-attention.

To capture the structure of the output space, we use queries containing the appropriate information for each output point, e.g. its spatial position or its modality. We construct queries by combining (concatenating or adding) a set of vectors into a query vector containing all of the information relevant for one of the \(O\) desired outputs.

For tasks with simple outputs, such as classification, these queries can be reused for every example and can be learned from scratch. For outputs with a spatial or sequence structure, we include a position encoding (e.g. a learned positional encoding or a Fourier feature) representing the position to be decoded in the output. For outputs with a multi-task or multimodal structure, we learn a single query for each task or for each modality: this information allows the network to distinguish one task or modality query from the others, much as positional encodings allow attention to distinguish one position from another.

for instance, for flow we find it helpful to include the input feature at the point being queried, and for StarCraft II we use the unit information to associate the model's output with the corresponding unit. We find that even very simple query features can produce good results, suggesting that the latent attention process is able to learn to organize the relevant information in a way that's easy to query.

To probe the generality of Perceiver IO, we evaluate it on several domains including language understanding (Wikipedia+C4 masked language modeling), visual understanding (Sintel/KITTI optical flow and ImageNet classification), multi-modal (Kinetics autoencoding and AudioSet classification) & multi-task settings (multi-task GLUE), and symbolic representations for games (StarCraft II).

But unlike Transformer-based models such as BERT (Devlin et al., 2019) or XLNet (Yang et al., 2019), Perceiver IO scales linearly with input length.

We find that at a given FLOPs budget, Perceiver IO trained without tokenization matches the performance of a strong Transformer-based model trained with SentencePiece tokenization (Sennrich et al., 2016; Kudo and Richardson, 2018).

We first observe that Perceiver IO applied on SentencePiece tokenized input sequences slightly outperforms a strong BERT baseline applied on the same inputs (81.2 vs 81.1). As a result of the reduced latent size of \(256\) we can train a much deeper network with 26 processing layers compared to BERT Base (12 layers) while maintaining a similar FLOPs budget.

Next, we show that we can leverage Perceiver IO to run on much longer sequences than a regular Transformer. Rather than using a fixed, handcrafted vocabulary, our model works directly with the raw byte inputs: we simply feed in and predict the UTF-8 bytes of the input string. Perceiver IO significantly outperforms a byte-level BERT baseline at the same FLOPs

Despite its greater depth, Perceiver IO is also faster than the Transformer-based BERT baselines in real wall-clock terms – by over a factor of 2 for the byte-based models – as shown in Tab. 1.

Our work scales to byte-level inputs without making any assumptions about the structure of the input, which allows it to be used beyond language as shown in the following sections.

We observe that our multitask approach outperforms single-task approaches and matches the approach that uses 8 task-specific input tokens. Our approach is more generic as it decouples the output array from the input array by not relying on [CLS] tokens. This is especially appealing when the tasks are many or inhomogeneous, as we show in Sec. 4.3.

To generalize to real data, the learned procedure needs to work for objects and textures that weren't seen in the training data.

State of the art algorithms, such as PWCNet (Sun et al., 2018), RAFT (Teed and Deng, 2020) or GMA (Jiang et al., 2021), use explicit machinery to ensure each of these steps is performed correctly even on out-of-domain data.

In contrast, we apply Perceiver IO to flow in a straightforward manner.

With traditional autoencoding models like convolutional encoder-decoders, it is not obvious how to combine these modalities, because each uses data of different dimensions - 3D (video), 1D (raw audio), and 0D (class labels) - and with wildly different numbers of elements. With Perceiver IO, we pad the inputs with modality-specific embeddings, serialize them into a single 2D input array and query outputs using queries containing position encodings (for video and audio) and modality embeddings. We train on 16 frames at \(224\times 224\) resolution, preprocessed into 50k 4x4 patches as well as 30k raw audio samples, producing a total of 1920 16-d vectors and one 700-d one-hot class label. We decode directly into pixels, raw audio, and the one-hot label without any post-processing. To prevent the model from encoding the label directly into one of the latent variables, we mask the class label 50% of the time in training. Due to the scale of inputs and outputs in this task we subsample decoding in training, while fully decoding in testing: we sampled 512 audio samples and 512 pixels and the class label for every training example. This allows us to directly decode to a video-sized array, which would otherwise be infeasible given memory constraints.

By masking the classification label during evaluation, our autoencoding model becomes a Kinetics 700 classifier.

By putting stronger weight on the class loss, we can reach 45% top-1 accuracy while maintaining 20.7 PSNR for video (Sec. I). This strongly suggests that Perceiver IO can jointly represent modalities with very different properties.

(1) on ImageNet, Perceiver IO surpasses 80% top-1 accuracy (84.5% top-1) without using 2D convolutions after pretraining on JFT. (2) When used to replace AlphaStar's entity Transformer, Perceiver IO obtains a \(\sim 3.5\times\) reduction in FLOPs while preserving StarCraft II 87 % win rate and parameter count, after only 3 experimental runs. (3) On AudioSet, Perceiver IO consistently outperforms the original Perceiver when using the same training protocol on multimodal video + audio classification.

In this work we introduce Perceiver IO, an architecture capable of handling general purpose inputs and outputs while scaling linearly in both input and output sizes. As we show, this architecture achieves good results in a wide variety of settings, making it a promising candidate for a _general purpose_ neural network architecture. Perceiver IO leverages the expressive power of latent attention and uses learned queries to expose a simple and unified interface that can handle multimodal and multitask settings. Overall, Perceiver IO offers a promising way to simplify the construction of sophisticated neural pipelines and facilitate progress on multimodal and multitask problems.

On all domains but StarCraft II, we include experiments with several input configurations, ranging from no domain adaptation (e.g. tokenizer-free language, flow from raw pixels, ImageNet with no convolutional or patch-based preprocessing and fully learned position encodings) to moderate domain adaptation (e.g. SentencePiece language understanding, flow from conv+maxpool-processed images and with RAFT upsampling, ImageNet with conv+maxpool-preprocessing and 2D Fourier features). These results demonstrate the unprecedented generality of Perceiver IO, the simplicity that this architecture unlocks in handling a range of tasks, and its flexibility to work as part of a domain-adapted system. Perceiver did well on ImageNet (Deng et al., 2009) classification without using 2D structure in the design of the architecture, but generated class scores using a simple average + project decoder (see Sec. E.3 and Fig. 6 for a diagram illustrating the difference between the two forms of decoder). We now evaluate the effect of this more general decoder.

While neither the Perceiver and Perceiver IO incorporate any 2D spatial structure architecturally, they use positional features that inject 2D spatial information (Sec. 3.2 and Appendix sec. D of Jaegle et al. 2021). By replacing these 2D position features with a fully learned position encoding as used on language, we can learn an image classification model that is given no privileged information about the structure of images.

We replace only the final linear layer of the decoder to produce the required 18,000 classes.

As in other experiments, we find that incorporating an attention-based decoder (Perceiver IO) leads to better results than averaging and pooling the output (Perceiver).

To further demonstrate Perceiver IO's capabilities on discrete modalities and as a drop-in replacement for Transformers, we plug in Perceiver IO in place of AlphaStar's Transformer. AlphaStar (Vinyals et al., 2019) is the state-of-the-art system for the challenging real-time strategy game of StarCraft II. At its core, AlphaStar represents the units in the game as a discrete, unordered set of symbols (the "units"). These units are represented by a vector of properties including unit type, position, and health. At each timestep, the architecture encodes units with an entity encoder, which in the original model was parameterized using a vanilla Transformer. The entity encoder takes as input a set of 512 entities (referred to as embedded_entity in Vinyals et al. (2019)) and produces as output an embedding for each entity (entity_embeddings) and a 1D embedding reduced over entities (embedded_entity). These 512 entities represent the units and other entities that are present in the game: unused entity slots are masked. entity_embeddings is produced by passing the outputs of the entity encoder through a ReLU and a 1D convolution with 256 channels.

For this reason, we view AlphaStar as an important test case for Perceiver IO's ability to function as a general-purpose tool for processing symbolic or set-valued data. If the question is "can Perceiver IO serve as a replacement for a well-tuned Transformer as a symbolic processing engine?" then the answer is yes: We obtained StarCraft II results by using Perceiver IO instead of a Transformer for the AlphaStar entity encoder. We replaced the Transformer with a Perceiver IO with a latent of index dimension 32, keeping the input and output size of 512 units.

Perceiver IO works out of the box. We observed that the resulting agent reached the same level of performance as the original AlphaStar agent, reaching an 87% win-rate versus the Elite bot after behavioral cloning (Pomerleau, 1989) on human data, while also leading to a 3\(\times\) decrease in FLOPs (Tab. 9). We replaced this Transformer with a 3-layer Perceiver IO with a latent of index dimension 32.

We seek to confirm that the the attention-based decoder helps even on classification, where the original Perceiver's decoder could be used. We show that the trend identified on ImageNet holds more generally, by revisiting the multimodal AudioSet classification domain. AudioSet is a large-scale event classification dataset containing 1.7 million training examples, each consisting of 10s long video and audio. Each example is labeled with several labels drawn from 527 classes.

We find that as in the ImageNet experiments, using the attention-based decoder leads to small but consistent improvements over the less generally applicable average + project decoder. Because Perceiver IO introduces no domain assumptions not present in the original Perceiver, this is evidence that Perceiver IO is a strictly more general model.

Perceiver IO is constructed from GPT-2-style (Radford et al., 2019) Transformer attention modules, which consist of QKV attention followed by an MLP, along with linear projection layers to ensure inputs to and outputs from the QKV attention and MLP take on desired sizes.

The view of attention that we describe encompasses both cross-attention and self-attention, both of which are specific ways of using QKV-attention. Perceiver IO uses cross-attention for encoder and decoder attention modules and uses self-attention for the latent processing modules.

As we show in ImageNet and AudioSet experiments, the attentional decoder used here can be used in settings where standard average + project decoders are applicable. We find that the attentional decoder typically produces somewhat better results than the standard decoder. This likely occurs because attentional decoding is more expressive than average + project decoding.

Instead of uniformly weighting each input in the averaging operation, decoder attention uses the attention scores as data-dependent weights for each input point. Instead of projecting the raw averaged input to a target dimensionality, decoder attention first projects inputs via a value layer and then processes them with an MLP. In addition to its greater expressivity, decoder attention has the advantage of being easily generalizable to dense outputs (by increasing the number of queries) and of reusing the same architectural pattern used for the encoder and processor modules.

Finally, to verify that Perceiver IO performs well on real-world data despite being trained only on synthetic imagery, we applied it to a small number (roughly 10) real videos taken from Getty images (www.gettyimages.com). Perceiver IO typically performs very well out-of-domain, although some failure cases remain: for instance, shadows tend to be interpreted as objects (Autoflow contains no shadows), and large regions with compression artifacts but no other texture may result in hallucinated flow (Autoflow contains no video compression artifacts). We include three challenging examples in the supplementary zip file, each of which depict complex motion and small objects. Perceiver IO can pick up on remarkably small objects such as the water droplets thrown by the girl's shoe in pigeon.mp4 or the confetti in thai_dance.mp4.

The decoder outputs \(16\times 224\times 224+16\times 1920/16+1\) vectors with 512 channels, that is, one element for each pixel in the video, one element for each audio patch, and one element for the classification label. These are then linearly projected to the appropriate channel size for each modality: 3 for videos, 16 for audio and 700 for classification (the logits for each of the 700 classes in Kinetics700).

We use a 387 dimensional 3D Fourier position embedding for each input video patch and a 385 dimensional 1D Fourier position embedding for each audio patch (385 to ensure the input dimensions to Perceiver IO match for all elements). In addition, we pad all input elements with a learned vector representing the modality; inputs from the same modality share the same token. In particular, we add a 317 dimensional modality embedding to video elements, a 319 dimensional modality embedding to audio elements, and a 4 dimensional modality embedding to the label, so that all elements have 704 features. The decoder queries are also constructed from Fourier position embeddings for video and audio and a learned positional embedding for label: 387 features for video, 385 features for audio, and 1024 learned features for the label. We pad the queries for each modality with a different learned vector for each modality, so that the final feature size for the queries is 1026.

To help verify the quality of Perceiver IO's outputs on real-world data, we applied it a small number of real videos (\(\sim\)10) with audio taken from Getty Images. Perceiver IO is able to capture the structure of both video and audio inputs, despite encoding both jointly with a single network.

Although decoding can be done in parallel, allowing us to decode very large output arrays in sequential batches, Perceiver IO requires all points are encoded simultaneously.

Input: Figure 1: The Perceiver IO architecture can be used on domains with a wide variety of input and output spaces, including multi-task language understanding, dense visual tasks like optical flow, hybrid dense/sparse multimodal tasks such as video+audio+class autoencoding, and tasks with symbolic outputs like StarCraft II. See Tables 5 and 6 for details of all domains considered here.

Input: Figure 5: Schematic depiction of encode, process, and decode attention. Each attention module uses the same operations, but differs in which inputs are used to generate key/values or queries and in the output shape. Encode attention can be viewed as mapping an input to a latent space, typically with a smaller index dimension (fewer elements). Decode attention can be viewed as mapping a latent to an output space, often with a larger index dimension (more elements). Both of these are forms of cross-attention. Process attention (self-attention) preserves the input index dimension (same elements). Red and blue dashed lines are used to highlight the two matrix multiplications used in QKV attention, as described in the text.

Input: Figure 2: The Perceiver IO architecture. Perceiver IO maps arbitrary input arrays to arbitrary output arrays in a domain agnostic process. The bulk of the computation happens in a latent space whose size is typically smaller than the inputs and outputs, which makes the process computationally tractable even for very large inputs & outputs. See Fig. 5 for a more detailed look at encode, process, and decode attention.

Input: Table 5: Details of each of the tasks we use to evaluate Perceiver IO here. The positional and task embeddings appended to inputs for each case are listed in Tab. 6.

Input: Table 6: Table best viewed on a screen. The structure and size of the positional and task embeddings used to construct Perceiver IO’s encoder key-value inputs and decoder query inputs, for each domain described in the main text. “[x, y]” indicates that x’s and y’s features are concatenated, while “x + y” indicates that x’s and y’s features are added to produce the full featurization. “FF” = Fourier features as in Jaegle et al. (2021).

Input: Figure 3: We construct queries with output-specific features to produce outputs with different semantics. For settings where each output point differs only in its position, like language, a position embedding can be used. Input features for the target output can also be used to query, either alone (as for StarCraft II) or alongside position features (as for flow). For multi-{task, modal} settings we use one embedding for each {task, modality} instead of each position. A single learned embedding suffices for simple classification tasks, like ImageNet. For tasks with heterogeneous outputs like multimodal autoencoding, features that are specific to some queries (like xy position) can be combined with modality embeddings, which also pad embeddings to fixed length.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the extensive evidence provided, I'll evaluate the claim: "This decoder allows Perceiver IO to serve as a drop-in replacement for a wide range of specialist networks currently in use on a set of challenging domains, while improving performance on tasks like classification that could be handled by the Perceiver."

The evidence strongly supports this claim through multiple demonstrations:
- Successfully replaced BERT for language tasks
- Replaced AlphaStar's Transformer while maintaining 87% win rate
- Handled complex multimodal tasks (video+audio+classification)
- Improved upon Perceiver's classification performance on ImageNet and AudioSet
- Scaled effectively to various domains without major architectural changes

The claim appears well-stated and accurately reflects the demonstrated capabilities without exaggeration.

<score>1</score>

Justification:
1. The evidence provides concrete examples and performance metrics across diverse domains showing successful drop-in replacement capabilities
2. Performance improvements are documented with specific metrics (e.g., 3.5x FLOP reduction in StarCraft II while maintaining win rate)
3. The claim's scope aligns with the breadth of experimental validation
4. No overstatement of capabilities - limitations are acknowledged (e.g., need for all points to be encoded simultaneously)
5. The evidence quantitatively supports both the "drop-in replacement" and "improving performance" aspects


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

This decoder allows Perceiver IO to serve as a drop-in replacement for a wide range of specialist networks currently in use on a set of challenging domains, while improving performance on tasks like classification that could be handled by the Perceiver.

The review comment to be evaluated is:

#### Strengths

- Major contribution is ability to scale the output to arbitrary length using an output query array. This enables supporting different types of output and scales well to different tasks.
- Byte level performance of Perceiver IO is impressive compared to BERT baseline. Byte level embeddings can be fed to Perceiver IO without any tokenization and this setting gets comparable performance with BERT sentence piece tokenization.
- Shows good results with multitask learning, using both single and task specific tokens.
- Authors do a thorough evaluation of this architecture on a wide variety of tasks, and achieves comparable results on multiple tasks in language, vision, multimodal, RL domains.

#### Weaknesses

- Novelty over Perceiver : Although I find this work quite impressive as it scales linearly with output embedding sizes as well as arbitrary types of outputs, the overall architecture seems incremental compared to Perceiver. One important change is there is no cross attention in the intermediate layers. I would love to see more detailed analysis why that is removed in this architecture and empirical evidence to support this change.

- How does choosing values for N,D affect the performance on various tasks. If we decouple input representation, output representation and base architecture, how does model performance scale with depth, N, D values.  Ablations on these hyper-parameters are required for more clarity.

- Given Perceiver can handle multiple modalities, I would be interested to see how it performs on multimodal tasks like VQA, Image/Text Retrieval, Hateful Memes etc and comparison to multimodal models like CLIP[1], ALIGN[2], SimVL[3].

  - [1] Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry et al. "Learning transferable visual models from natural language supervision." arXiv preprint arXiv:2103.00020 (2021).
  - [2] Jia, Chao, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. "Scaling up visual and vision-language representation learning with noisy text supervision." arXiv preprint arXiv:2102.05918 (2021).
  - [3] Wang, Zirui, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. "SimVLM: Simple visual language model pretraining with weak supervision." arXiv preprint arXiv:2108.10904 (2021).

The evidence to be evaluated is:Similar to our work, several groups have proposed to solve tasks in multiple domains (e.g. Kaiser et al. 2017; Alyraca et al. 2020; Akbari et al. 2021), but typically across a fixed and predefined set of modalities by means of domain-specific networks. Although single-task specialist networks remain dominant in vision, multi-task learning has become popular (Misra et al., 2016; Doersch and Zisserman, 2017; Kokkinos, 2017; Zamir et al., 2018) and individual models achieve generality in a restricted domain: e.g. Mask-RCNN (He et al., 2017) handles object detection, segmentation, and pose estimation.

Several groups have demonstrated that Transformers (originally designed for language) can be used or adapted to non-language tasks (e.g. Chen et al. 2020; Lu et al. 2021), but the limited scalability of Transformers limits their usefulness as general-purpose architectures.

The focus of our work is developing an architecture that is efficient and also performs well in many settings with a wide range of inputs and outputs.

Perceiver IO builds on this body of work to produce a general purpose architecture that can be easily and widely applied. The Perceiver IO architecture builds on the Perceiver (Jaegle et al., 2021), which achieved its cross-domain generality by assuming that its input is a simple 2D byte array: a set of elements (which might be pixels or patches in vision, characters or words in language, or some form of embedding, learned or otherwise), each described by a feature vector. The model then encodes information about the input array using a smaller number of latent feature vectors, using Transformer-style attention, followed by iterative processing and a final aggregation down to a category label. Rather than output a single category, Perceiver IO aims to have the same level of generality with respect to its _outputs_ as the Perceiver has with respect to its _inputs_: that is, it should produce arbitrary output arrays.

The queries may be hand-designed, learned embeddings, or a simple function of the input. They attend to the latents to yield an output array of the desired shape.

We first **encode** by applying an attention module that maps input arrays \(x\in\mathbb{R}^{M\times C}\) to arrays in a latent space \(z\in\mathbb{R}^{N\times D}\). We next **process** the latents \(z\) by applying a series of modules that take in and return arrays in this latent space. Finally, we **decode** by applying an attention module that maps latent arrays to output arrays \(y\in\mathbb{R}^{O\times E}\). \(M\), \(C\), \(O\), and \(E\) are properties of the task data and can be very large (Tab. 5), while \(N\) and \(D\) are hyperparameters and can be chosen to make model computation tractable. Following the design of the Perceiver, we implement each of the architecture's components using Transformer-style attention modules. Each of these modules applies a global query-key-value (QKV) attention operation followed by a multi-layer perceptron (MLP).

The Perceiver IO architecture builds on primitives similar to those in Transformers.

Perceiver IO uses attention non-homogeneously by mapping inputs to a latent space, processing in that latent space, and decoding to an output space. Perceiver IO has no quadratic dependence on the input or output size: encoder and decoder attention modules depend linearly on the input and output size (respectively), while latent attention is independent of both input and output sizes (Sec. E.2). Because of the corresponding reduction in compute and memory requirements, Perceiver IO scales to much larger inputs and outputs. While Transformers are typically used in settings with data preprocessed to contain at most a few thousand dimensions (Brown et al., 2020; Raffel et al., 2020), we show good results on domains with hundreds of thousands of dimensions. This architecture can be applied to inputs of any shape or spatial layout including inputs or outputs with different spatial structure (e.g. sound and video).

To decode this information, we query for it using cross-attention.

To capture the structure of the output space, we use queries containing the appropriate information for each output point, e.g. its spatial position or its modality. We construct queries by combining (concatenating or adding) a set of vectors into a query vector containing all of the information relevant for one of the \(O\) desired outputs.

For tasks with simple outputs, such as classification, these queries can be reused for every example and can be learned from scratch. For outputs with a spatial or sequence structure, we include a position encoding (e.g. a learned positional encoding or a Fourier feature) representing the position to be decoded in the output. For outputs with a multi-task or multimodal structure, we learn a single query for each task or for each modality: this information allows the network to distinguish one task or modality query from the others, much as positional encodings allow attention to distinguish one position from another.

for instance, for flow we find it helpful to include the input feature at the point being queried, and for StarCraft II we use the unit information to associate the model's output with the corresponding unit. We find that even very simple query features can produce good results, suggesting that the latent attention process is able to learn to organize the relevant information in a way that's easy to query.

To probe the generality of Perceiver IO, we evaluate it on several domains including language understanding (Wikipedia+C4 masked language modeling), visual understanding (Sintel/KITTI optical flow and ImageNet classification), multi-modal (Kinetics autoencoding and AudioSet classification) & multi-task settings (multi-task GLUE), and symbolic representations for games (StarCraft II).

But unlike Transformer-based models such as BERT (Devlin et al., 2019) or XLNet (Yang et al., 2019), Perceiver IO scales linearly with input length.

We find that at a given FLOPs budget, Perceiver IO trained without tokenization matches the performance of a strong Transformer-based model trained with SentencePiece tokenization (Sennrich et al., 2016; Kudo and Richardson, 2018).

We first observe that Perceiver IO applied on SentencePiece tokenized input sequences slightly outperforms a strong BERT baseline applied on the same inputs (81.2 vs 81.1). As a result of the reduced latent size of \(256\) we can train a much deeper network with 26 processing layers compared to BERT Base (12 layers) while maintaining a similar FLOPs budget.

Next, we show that we can leverage Perceiver IO to run on much longer sequences than a regular Transformer. Rather than using a fixed, handcrafted vocabulary, our model works directly with the raw byte inputs: we simply feed in and predict the UTF-8 bytes of the input string. Perceiver IO significantly outperforms a byte-level BERT baseline at the same FLOPs

Despite its greater depth, Perceiver IO is also faster than the Transformer-based BERT baselines in real wall-clock terms – by over a factor of 2 for the byte-based models – as shown in Tab. 1.

Our work scales to byte-level inputs without making any assumptions about the structure of the input, which allows it to be used beyond language as shown in the following sections.

We observe that our multitask approach outperforms single-task approaches and matches the approach that uses 8 task-specific input tokens. Our approach is more generic as it decouples the output array from the input array by not relying on [CLS] tokens. This is especially appealing when the tasks are many or inhomogeneous, as we show in Sec. 4.3.

To generalize to real data, the learned procedure needs to work for objects and textures that weren't seen in the training data.

State of the art algorithms, such as PWCNet (Sun et al., 2018), RAFT (Teed and Deng, 2020) or GMA (Jiang et al., 2021), use explicit machinery to ensure each of these steps is performed correctly even on out-of-domain data.

In contrast, we apply Perceiver IO to flow in a straightforward manner.

With traditional autoencoding models like convolutional encoder-decoders, it is not obvious how to combine these modalities, because each uses data of different dimensions - 3D (video), 1D (raw audio), and 0D (class labels) - and with wildly different numbers of elements. With Perceiver IO, we pad the inputs with modality-specific embeddings, serialize them into a single 2D input array and query outputs using queries containing position encodings (for video and audio) and modality embeddings. We train on 16 frames at \(224\times 224\) resolution, preprocessed into 50k 4x4 patches as well as 30k raw audio samples, producing a total of 1920 16-d vectors and one 700-d one-hot class label. We decode directly into pixels, raw audio, and the one-hot label without any post-processing. To prevent the model from encoding the label directly into one of the latent variables, we mask the class label 50% of the time in training. Due to the scale of inputs and outputs in this task we subsample decoding in training, while fully decoding in testing: we sampled 512 audio samples and 512 pixels and the class label for every training example. This allows us to directly decode to a video-sized array, which would otherwise be infeasible given memory constraints.

By masking the classification label during evaluation, our autoencoding model becomes a Kinetics 700 classifier.

By putting stronger weight on the class loss, we can reach 45% top-1 accuracy while maintaining 20.7 PSNR for video (Sec. I). This strongly suggests that Perceiver IO can jointly represent modalities with very different properties.

(1) on ImageNet, Perceiver IO surpasses 80% top-1 accuracy (84.5% top-1) without using 2D convolutions after pretraining on JFT. (2) When used to replace AlphaStar's entity Transformer, Perceiver IO obtains a \(\sim 3.5\times\) reduction in FLOPs while preserving StarCraft II 87 % win rate and parameter count, after only 3 experimental runs. (3) On AudioSet, Perceiver IO consistently outperforms the original Perceiver when using the same training protocol on multimodal video + audio classification.

In this work we introduce Perceiver IO, an architecture capable of handling general purpose inputs and outputs while scaling linearly in both input and output sizes. As we show, this architecture achieves good results in a wide variety of settings, making it a promising candidate for a _general purpose_ neural network architecture. Perceiver IO leverages the expressive power of latent attention and uses learned queries to expose a simple and unified interface that can handle multimodal and multitask settings. Overall, Perceiver IO offers a promising way to simplify the construction of sophisticated neural pipelines and facilitate progress on multimodal and multitask problems.

On all domains but StarCraft II, we include experiments with several input configurations, ranging from no domain adaptation (e.g. tokenizer-free language, flow from raw pixels, ImageNet with no convolutional or patch-based preprocessing and fully learned position encodings) to moderate domain adaptation (e.g. SentencePiece language understanding, flow from conv+maxpool-processed images and with RAFT upsampling, ImageNet with conv+maxpool-preprocessing and 2D Fourier features). These results demonstrate the unprecedented generality of Perceiver IO, the simplicity that this architecture unlocks in handling a range of tasks, and its flexibility to work as part of a domain-adapted system. Perceiver did well on ImageNet (Deng et al., 2009) classification without using 2D structure in the design of the architecture, but generated class scores using a simple average + project decoder (see Sec. E.3 and Fig. 6 for a diagram illustrating the difference between the two forms of decoder). We now evaluate the effect of this more general decoder.

While neither the Perceiver and Perceiver IO incorporate any 2D spatial structure architecturally, they use positional features that inject 2D spatial information (Sec. 3.2 and Appendix sec. D of Jaegle et al. 2021). By replacing these 2D position features with a fully learned position encoding as used on language, we can learn an image classification model that is given no privileged information about the structure of images.

We replace only the final linear layer of the decoder to produce the required 18,000 classes.

As in other experiments, we find that incorporating an attention-based decoder (Perceiver IO) leads to better results than averaging and pooling the output (Perceiver).

To further demonstrate Perceiver IO's capabilities on discrete modalities and as a drop-in replacement for Transformers, we plug in Perceiver IO in place of AlphaStar's Transformer. AlphaStar (Vinyals et al., 2019) is the state-of-the-art system for the challenging real-time strategy game of StarCraft II. At its core, AlphaStar represents the units in the game as a discrete, unordered set of symbols (the "units"). These units are represented by a vector of properties including unit type, position, and health. At each timestep, the architecture encodes units with an entity encoder, which in the original model was parameterized using a vanilla Transformer. The entity encoder takes as input a set of 512 entities (referred to as embedded_entity in Vinyals et al. (2019)) and produces as output an embedding for each entity (entity_embeddings) and a 1D embedding reduced over entities (embedded_entity). These 512 entities represent the units and other entities that are present in the game: unused entity slots are masked. entity_embeddings is produced by passing the outputs of the entity encoder through a ReLU and a 1D convolution with 256 channels.

For this reason, we view AlphaStar as an important test case for Perceiver IO's ability to function as a general-purpose tool for processing symbolic or set-valued data. If the question is "can Perceiver IO serve as a replacement for a well-tuned Transformer as a symbolic processing engine?" then the answer is yes: We obtained StarCraft II results by using Perceiver IO instead of a Transformer for the AlphaStar entity encoder. We replaced the Transformer with a Perceiver IO with a latent of index dimension 32, keeping the input and output size of 512 units.

Perceiver IO works out of the box. We observed that the resulting agent reached the same level of performance as the original AlphaStar agent, reaching an 87% win-rate versus the Elite bot after behavioral cloning (Pomerleau, 1989) on human data, while also leading to a 3\(\times\) decrease in FLOPs (Tab. 9). We replaced this Transformer with a 3-layer Perceiver IO with a latent of index dimension 32.

We seek to confirm that the the attention-based decoder helps even on classification, where the original Perceiver's decoder could be used. We show that the trend identified on ImageNet holds more generally, by revisiting the multimodal AudioSet classification domain. AudioSet is a large-scale event classification dataset containing 1.7 million training examples, each consisting of 10s long video and audio. Each example is labeled with several labels drawn from 527 classes.

We find that as in the ImageNet experiments, using the attention-based decoder leads to small but consistent improvements over the less generally applicable average + project decoder. Because Perceiver IO introduces no domain assumptions not present in the original Perceiver, this is evidence that Perceiver IO is a strictly more general model.

Perceiver IO is constructed from GPT-2-style (Radford et al., 2019) Transformer attention modules, which consist of QKV attention followed by an MLP, along with linear projection layers to ensure inputs to and outputs from the QKV attention and MLP take on desired sizes.

The view of attention that we describe encompasses both cross-attention and self-attention, both of which are specific ways of using QKV-attention. Perceiver IO uses cross-attention for encoder and decoder attention modules and uses self-attention for the latent processing modules.

As we show in ImageNet and AudioSet experiments, the attentional decoder used here can be used in settings where standard average + project decoders are applicable. We find that the attentional decoder typically produces somewhat better results than the standard decoder. This likely occurs because attentional decoding is more expressive than average + project decoding.

Instead of uniformly weighting each input in the averaging operation, decoder attention uses the attention scores as data-dependent weights for each input point. Instead of projecting the raw averaged input to a target dimensionality, decoder attention first projects inputs via a value layer and then processes them with an MLP. In addition to its greater expressivity, decoder attention has the advantage of being easily generalizable to dense outputs (by increasing the number of queries) and of reusing the same architectural pattern used for the encoder and processor modules.

Finally, to verify that Perceiver IO performs well on real-world data despite being trained only on synthetic imagery, we applied it to a small number (roughly 10) real videos taken from Getty images (www.gettyimages.com). Perceiver IO typically performs very well out-of-domain, although some failure cases remain: for instance, shadows tend to be interpreted as objects (Autoflow contains no shadows), and large regions with compression artifacts but no other texture may result in hallucinated flow (Autoflow contains no video compression artifacts). We include three challenging examples in the supplementary zip file, each of which depict complex motion and small objects. Perceiver IO can pick up on remarkably small objects such as the water droplets thrown by the girl's shoe in pigeon.mp4 or the confetti in thai_dance.mp4.

The decoder outputs \(16\times 224\times 224+16\times 1920/16+1\) vectors with 512 channels, that is, one element for each pixel in the video, one element for each audio patch, and one element for the classification label. These are then linearly projected to the appropriate channel size for each modality: 3 for videos, 16 for audio and 700 for classification (the logits for each of the 700 classes in Kinetics700).

We use a 387 dimensional 3D Fourier position embedding for each input video patch and a 385 dimensional 1D Fourier position embedding for each audio patch (385 to ensure the input dimensions to Perceiver IO match for all elements). In addition, we pad all input elements with a learned vector representing the modality; inputs from the same modality share the same token. In particular, we add a 317 dimensional modality embedding to video elements, a 319 dimensional modality embedding to audio elements, and a 4 dimensional modality embedding to the label, so that all elements have 704 features. The decoder queries are also constructed from Fourier position embeddings for video and audio and a learned positional embedding for label: 387 features for video, 385 features for audio, and 1024 learned features for the label. We pad the queries for each modality with a different learned vector for each modality, so that the final feature size for the queries is 1026.

To help verify the quality of Perceiver IO's outputs on real-world data, we applied it a small number of real videos (\(\sim\)10) with audio taken from Getty Images. Perceiver IO is able to capture the structure of both video and audio inputs, despite encoding both jointly with a single network.

Although decoding can be done in parallel, allowing us to decode very large output arrays in sequential batches, Perceiver IO requires all points are encoded simultaneously.

Input: Figure 1: The Perceiver IO architecture can be used on domains with a wide variety of input and output spaces, including multi-task language understanding, dense visual tasks like optical flow, hybrid dense/sparse multimodal tasks such as video+audio+class autoencoding, and tasks with symbolic outputs like StarCraft II. See Tables 5 and 6 for details of all domains considered here.

Input: Figure 5: Schematic depiction of encode, process, and decode attention. Each attention module uses the same operations, but differs in which inputs are used to generate key/values or queries and in the output shape. Encode attention can be viewed as mapping an input to a latent space, typically with a smaller index dimension (fewer elements). Decode attention can be viewed as mapping a latent to an output space, often with a larger index dimension (more elements). Both of these are forms of cross-attention. Process attention (self-attention) preserves the input index dimension (same elements). Red and blue dashed lines are used to highlight the two matrix multiplications used in QKV attention, as described in the text.

Input: Figure 2: The Perceiver IO architecture. Perceiver IO maps arbitrary input arrays to arbitrary output arrays in a domain agnostic process. The bulk of the computation happens in a latent space whose size is typically smaller than the inputs and outputs, which makes the process computationally tractable even for very large inputs & outputs. See Fig. 5 for a more detailed look at encode, process, and decode attention.

Input: Table 5: Details of each of the tasks we use to evaluate Perceiver IO here. The positional and task embeddings appended to inputs for each case are listed in Tab. 6.

Input: Table 6: Table best viewed on a screen. The structure and size of the positional and task embeddings used to construct Perceiver IO’s encoder key-value inputs and decoder query inputs, for each domain described in the main text. “[x, y]” indicates that x’s and y’s features are concatenated, while “x + y” indicates that x’s and y’s features are added to produce the full featurization. “FF” = Fourier features as in Jaegle et al. (2021).

Input: Figure 3: We construct queries with output-specific features to produce outputs with different semantics. For settings where each output point differs only in its position, like language, a position embedding can be used. Input features for the target output can also be used to query, either alone (as for StarCraft II) or alongside position features (as for flow). For multi-{task, modal} settings we use one embedding for each {task, modality} instead of each position. A single learned embedding suffices for simple classification tasks, like ImageNet. For tasks with heterogeneous outputs like multimodal autoencoding, features that are specific to some queries (like xy position) can be combined with modality embeddings, which also pad embeddings to fixed length.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the extensive evidence provided, I find the claim is well-stated (score of 1). The paper demonstrates through multiple experiments that Perceiver IO successfully functions as a drop-in replacement across diverse domains while often improving performance. Key evidence includes:

- Direct replacement of BERT for language tasks with better performance (81.2 vs 81.1)
- 3.5x FLOPS reduction while maintaining 87% win rate when replacing StarCraft II's Transformer
- Outperforming original Perceiver on AudioSet classification
- Strong results on ImageNet (84.5% top-1), optical flow, and multimodal tasks
- Demonstrated ability to handle arbitrary input/output spaces through a unified architecture

The claim's scope aligns well with the comprehensive empirical validation across multiple challenging domains.

<score>1</score>