Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Perceiver IO does this using a fully attentional read-process-write architecture: inputs are encoded (read) to a latent space, the latent representation is refined (process) via many layers of processing, and the latent space is decoded (write) to produce outputs.

The review comment to be evaluated is:

#### Strength
- The idea of perceiver IO is novel and solid -- a general architecture capable of handling general-purpose inputs and outputs across different tasks and modalities. This is very promising to simplify the construction of highly tuned task-specific neural pipelines and improve the multimodal and multi-task problems.

- Each component in perceiver IO is necessary and well defined for the proposed tasks. 2D byte array input enables the task agnostic input for different modalities. Perceiver network that scales linearly with the size of input and output, and non-auto regressive decoding make the decoding to raw output possible.

- The proposed architecture is tested on massive experiments including language understanding tasks, optical flow, video audio class autoencoding, image classification, and starcraft II and achieves superior performance. Each task is supported with a detailed ablation study to shed light on future research.

#### Weakness
- In Table 1, the WNLI task is excluded from the GLUE benchmark, I wonder what is the reason this task is removed?

- The non-autoregressive decoding enables fast decoding for large outputs. However, for tasks that require more information on prior decoding context (machine translation or image generation tasks), does the proposed model can still perform well on those tasks?

- Although perceiver IO removes the task-specific pre-processing (tokenization, patch embedding, etc.), the model still requires huge engineering efforts to adapt the model for different tasks. For example, the model hyper-parameters are quite different for different tasks.

- One huge benefit of perceiver IO is to train different tasks together and explore the transfer between different tasks/modalities, which is not explored in this paper.

The evidence to be evaluated is:Several groups have proposed to use attention to manipulate the size of arrays or to introduce bottlenecks in processing. Set Transformers and related work (Lee et al., 2019; Goyal et al., 2022) use a learned query ("inducing points") to induce local bottlenecks by mapping a set back and forth from a set with fewer elements and learned decoder queries ("seed vectors") to map to outputs ("pooling by multiheaded attention").

The Perceiver IO architecture builds on the Perceiver (Jaegle et al., 2021), which achieved its cross-domain generality by assuming that its input is a simple 2D byte array: a set of elements (which might be pixels or patches in vision, characters or words in language, or some form of embedding, learned or otherwise), each described by a feature vector. The model then encodes information about the input array using a smaller number of latent feature vectors, using Transformer-style attention, followed by iterative processing and a final aggregation down to a category label. Rather than output a single category, Perceiver IO aims to have the same level of generality with respect to its _outputs_ as the Perceiver has with respect to its _inputs_: that is, it should produce arbitrary output arrays. We can predict each element of the output array using another attention module by _querying_ the latent array using a query feature vector unique to the desired output element.

We first **encode** by applying an attention module that maps input arrays \(x\in\mathbb{R}^{M\times C}\) to arrays in a latent space \(z\in\mathbb{R}^{N\times D}\). We next **process** the latents \(z\) by applying a series of modules that take in and return arrays in this latent space. Finally, we **decode** by applying an attention module that maps latent arrays to output arrays \(y\in\mathbb{R}^{O\times E}\).

Following the design of the Perceiver, we implement each of the architecture's components using Transformer-style attention modules.

The Perceiver IO architecture builds on primitives similar to those in Transformers.

Perceiver IO uses attention non-homogeneously by mapping inputs to a latent space, processing in that latent space, and decoding to an output space. Perceiver IO has no quadratic dependence on the input or output size: encoder and decoder attention modules depend linearly on the input and output size (respectively), while latent attention is independent of both input and output sizes (Sec. E.2).

This architecture can be applied to inputs of any shape or spatial layout including inputs or outputs with different spatial structure (e.g. sound and video). In contrast to latent spaces typically used in vision (e.g. Ronneberger et al., 2015) the latent does not explicitly share the structure (spatial or otherwise) of the inputs. To decode this information, we query for it using cross-attention. Our goal is to produce a final output array of size \(O\times E\), given a latent representation of size \(N\times D\). We produce an output of this size by querying the decoder with an array of index dimension \(O\). To capture the structure of the output space, we use queries containing the appropriate information for each output point, e.g. its spatial position or its modality. We construct queries by combining (concatenating or adding) a set of vectors into a query vector containing all of the information relevant for one of the \(O\) desired outputs.

We find that even very simple query features can produce good results, suggesting that the latent attention process is able to learn to organize the relevant information in a way that's easy to query. Each output point depends only on its query and the latent array, allowing us to decode outputs in parallel.

Perceiver IO produces one output vector per masked input by using learnable position-dependent vectors to query the output of the final latent processing layer.

As a result of the reduced latent size of \(256\) we can train a much deeper network with 26 processing layers compared to BERT Base (12 layers) while maintaining a similar FLOPs budget.

In contrast, we apply Perceiver IO to flow in a straightforward manner.

We concatenate a fixed position encoding to these features and then apply Perceiver IO. To decode, we query the latent representation using the input encoding.

With Perceiver IO, we pad the inputs with modality-specific embeddings, serialize them into a single 2D input array and query outputs using queries containing position encodings (for video and audio) and modality embeddings.

We decode directly into pixels, raw audio, and the one-hot label without any post-processing.

In this work we introduce Perceiver IO, an architecture capable of handling general purpose inputs and outputs while scaling linearly in both input and output sizes.

Perceiver IO leverages the expressive power of latent attention and uses learned queries to expose a simple and unified interface that can handle multimodal and multitask settings.

These results demonstrate the unprecedented generality of Perceiver IO, the simplicity that this architecture unlocks in handling a range of tasks, and its flexibility to work as part of a domain-adapted system.

While neither the Perceiver and Perceiver IO incorporate any 2D spatial structure architecturally, they use positional features that inject 2D spatial information (Sec. 3.2 and Appendix sec. D of Jaegle et al. 2021).

We use an architecture with weight sharing in depth: the latent (processing) component of the architecture includes 8 blocks of 6 attention modules each, and weights are shared between the corresponding modules in each block.

Unlike in the other ImageNet experiments, we do not share weights in the latent self-attention process modules, but use a 16-layer latent network with no weight sharing in depth. Unlike the other ImageNet experiments, the process-module MLPs use a hidden layer with 4\(\times\) the number of channels (rather than 1\(\times\) as on other ImageNet experiments).

At each timestep, the architecture encodes units with an entity encoder, which in the original model was parameterized using a vanilla Transformer. The entity encoder takes as input a set of 512 entities (referred to as embedded_entity in Vinyals et al. (2019)) and produces as output an embedding for each entity (entity_embeddings) and a 1D embedding reduced over entities (embedded_entity).

We obtained StarCraft II results by using Perceiver IO instead of a Transformer for the AlphaStar entity encoder. We replaced the Transformer with a Perceiver IO with a latent of index dimension 32, keeping the input and output size of 512 units.

Perceiver IO is constructed from GPT-2-style (Radford et al., 2019) Transformer attention modules, which consist of QKV attention followed by an MLP, along with linear projection layers to ensure inputs to and outputs from the QKV attention and MLP take on desired sizes. Using the array sizes of the encoder attention, the QKV attention takes in two two-dimensional arrays, a key-value input array \(X_{KV}\in\mathbb{R}^{M\times C}\) and a query input array \(X_{Q}\in\mathbb{R}^{N\times D}\), and maps them to an array \(X_{QKV}\in\mathbb{R}^{N\times D}\), sharing the shape of the query input (after projection). \(X_{QKV}\) is used as input to an MLP, which is applied independently to each element of the index dimension (i.e., convolving the MLP with its input along the first dimension), producing a final array \(X_{MLP}\in\mathbb{R}^{N\times D}\).

Perceiver IO uses cross-attention for encoder and decoder attention modules and uses self-attention for the latent processing modules.

QKV attention takes in two two-dimensional arrays, a query input \(X_{Q}\in\mathbb{R}^{N\times D}\) and a key-value input \(X_{KV}\in\mathbb{R}^{M\times C}\).

\[Q=f_{Q}(X_{Q}); K=f_{K}(X_{KV}); V=f_{V}(X_{KV}) \tag{1}\] \[X_{QK}=\text{softmax}(QK^{T}/\sqrt{F})\] (2) \[\text{Attn}(X_{Q},X_{KV})=X_{QKV}=f_{O}(X_{QK}V), \tag{3}\] where \(X_{QK}\) is an array of attention maps \(\in\mathbb{R}^{N\times M}\), and \(X_{QKV}\) is an array \(\in\mathbb{R}^{N\times D}\). The functions \(f_{\{Q,K,V\}}\) are linear layers mapping each input to a shared feature dimension \(F\) and \(f_{O}\) is a linear layer projecting the output to a target channel dimension, which is often the same size as \(X_{Q}\)'s.

The full module has the following structure: \[X_{QKV} =\text{Attn}(\text{layerNorm}(X_{Q}),\text{layerNorm}(X_{KV})) \tag{4}\] \[X_{QKV} =X_{QKV}+X_{Q}\] (5) \[X_{QKV} =X_{QKV}+\text{MLP}(\text{layerNorm}(X_{QKV})), \tag{6}\]

The computational complexity of each attention module is dominated by the two matrix multiplications in QKV attention. Still using the shapes of the encoder attention, these two matrix multiplies involve matrices of shape \(M\times F\) and \(N\times F\) and \(M\times N\) and \(N\times F\), giving overall time and memory complexity of \(\mathcal{O}(MNF)\). Let \(M\), \(N\), and \(O\) be the index dimensions for the input, latent, and output arrays, and to simplify the analysis let \(F\) be the feature size for all layers.

In other words, Perceiver IO has complexity linear in the size of the input and output arrays and it decouples the depth of the latent transformer from the input and output sizes. Both of these properties contribute to Perceiver IO's efficiency: while many proposals for efficient attention modules or architectures include linear or sub-quadratic scaling with input/output size, Perceiver IO is unusual in also decoupling depth from input/output size (without requiring domain-specific strategies like 2D convolution).

Instead of uniformly weighting each input in the averaging operation, decoder attention uses the attention scores as data-dependent weights for each input point.

In order to decode, we use learned queries of the same dimension of the input array (Tab. 11). We have as many output queries as inputs to be able to predict the masked token at all positions in the sentence (\(M\)=\(O\)).

After feeding this through the Perceiver IO architecture (including querying with the same convolutional features used as input), we have an output a feature grid with stride 4 and 64 channels, on top of which we apply a RAFT upsampling layer.

We use a latent array with 2048 elements and 512 channels and 24 self-attention modules, each with 16 self-attention heads, unless otherwise noted.

For these experiments, the output of the perceiver decoder's cross attend is 64 channels, which is fed into a RAFT-style upsampling operation.

The decoder outputs \(16\times 224\times 224+16\times 1920/16+1\) vectors with 512 channels, that is, one element for each pixel in the video, one element for each audio patch, and one element for the classification label. These are then linearly projected to the appropriate channel size for each modality: 3 for videos, 16 for audio and 700 for classification (the logits for each of the 700 classes in Kinetics700). Finally, we un-patch the audio to arrive at the output audio. We note that we read and generate the audio waveform directly in the time domain; we do not transform first to a spectrogram. We use a 387 dimensional 3D Fourier position embedding for each input video patch and a 385 dimensional 1D Fourier position embedding for each audio patch (385 to ensure the input dimensions to Perceiver IO match for all elements). In addition, we pad all input elements with a learned vector representing the modality; inputs from the same modality share the same token. In particular, we add a 317 dimensional modality embedding to video elements, a 319 dimensional modality embedding to audio elements, and a 4 dimensional modality embedding to the label, so that all elements have 704 features. The decoder queries are also constructed from Fourier position embeddings for video and audio and a learned positional embedding for label: 387 features for video, 385 features for audio, and 1024 learned features for the label. We pad the queries for each modality with a different learned vector for each modality, so that the final feature size for the queries is 1026.

Perceiver IO is able to capture the structure of both video and audio inputs, despite encoding both jointly with a single network.

Although decoding can be done in parallel, allowing us to decode very large output arrays in sequential batches, Perceiver IO requires all points are encoded simultaneously.

Input: Figure 5: Schematic depiction of encode, process, and decode attention. Each attention module uses the same operations, but differs in which inputs are used to generate key/values or queries and in the output shape. Encode attention can be viewed as mapping an input to a latent space, typically with a smaller index dimension (fewer elements). Decode attention can be viewed as mapping a latent to an output space, often with a larger index dimension (more elements). Both of these are forms of cross-attention. Process attention (self-attention) preserves the input index dimension (same elements). Red and blue dashed lines are used to highlight the two matrix multiplications used in QKV attention, as described in the text.

Input: Figure 2: The Perceiver IO architecture. Perceiver IO maps arbitrary input arrays to arbitrary output arrays in a domain agnostic process. The bulk of the computation happens in a latent space whose size is typically smaller than the inputs and outputs, which makes the process computationally tractable even for very large inputs & outputs. See Fig. 5 for a more detailed look at encode, process, and decode attention.

Input: Figure 4: Multimodal audio-video-label autoencoding with 88x compression. Side-by-side: inputs on left, reconstructions right. See the supplemental material for example output video and audio.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that it accurately describes Perceiver IO's architecture and operation. The evidence, particularly Figures 2 and 5, directly illustrates the claimed read-process-write architecture through encode (read), process, and decode (write) attention modules. The technical details in the text explicitly confirm this structure and explain how each component functions. The description is precise without exaggeration, matching what is demonstrated in both the architectural diagrams and mathematical formulations. The claim sticks to the fundamental architectural description without overstating capabilities or performance.

<score>1</score>

Justification:
1. Direct evidence from figures showing the exact three-part architecture described
2. Detailed mathematical formulations confirming the attentional mechanisms at each stage
3. Clear technical explanations of how inputs are encoded to latent space, processed, and decoded to outputs
4. No exaggerated claims about performance or capabilities - focuses purely on architectural description
5. Review comments support the architectural description while separately discussing strengths/limitations

The claim is well-stated and accurately reflects the core architectural design without overreaching.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

This approach allows us to decouple the size of elements used for the bulk of the computation (the latent) from the size of the input and output spaces, while making minimal assumptions about the spatial or locality structure of the input and output.

The evidence to be evaluated is:Several groups have proposed to use attention to manipulate the size of arrays or to introduce bottlenecks in processing. Set Transformers and related work (Lee et al., 2019; Goyal et al., 2022) use a learned query ("inducing points") to induce local bottlenecks by mapping a set back and forth from a set with fewer elements and learned decoder queries ("seed vectors") to map to outputs ("pooling by multiheaded attention"). Each layer of these networks has complexity linear in the input size, while Perceivers use a deep latent network with complexity independent of the input and output.

The Perceiver IO architecture builds on the Perceiver (Jaegle et al., 2021), which achieved its cross-domain generality by assuming that its input is a simple 2D byte array: a set of elements (which might be pixels or patches in vision, characters or words in language, or some form of embedding, learned or otherwise), each described by a feature vector. The model then encodes information about the input array using a smaller number of latent feature vectors, using Transformer-style attention, followed by iterative processing and a final aggregation down to a category label. Rather than output a single category, Perceiver IO aims to have the same level of generality with respect to its _outputs_ as the Perceiver has with respect to its _inputs_: that is, it should produce arbitrary output arrays.

We first **encode** by applying an attention module that maps input arrays \(x\in\mathbb{R}^{M\times C}\) to arrays in a latent space \(z\in\mathbb{R}^{N\times D}\). We next **process** the latents \(z\) by applying a series of modules that take in and return arrays in this latent space. Finally, we **decode** by applying an attention module that maps latent arrays to output arrays \(y\in\mathbb{R}^{O\times E}\). \(M\), \(C\), \(O\), and \(E\) are properties of the task data and can be very large (Tab. 5), while \(N\) and \(D\) are hyperparameters and can be chosen to make model computation tractable.

Perceiver IO uses attention non-homogeneously by mapping inputs to a latent space, processing in that latent space, and decoding to an output space. Perceiver IO has no quadratic dependence on the input or output size: encoder and decoder attention modules depend linearly on the input and output size (respectively), while latent attention is independent of both input and output sizes (Sec. E.2).

This architecture can be applied to inputs of any shape or spatial layout including inputs or outputs with different spatial structure (e.g. sound and video). In contrast to latent spaces typically used in vision (e.g. Ronneberger et al., 2015) the latent does not explicitly share the structure (spatial or otherwise) of the inputs.

Our goal is to produce a final output array of size \(O\times E\), given a latent representation of size \(N\times D\). We produce an output of this size by querying the decoder with an array of index dimension \(O\). To capture the structure of the output space, we use queries containing the appropriate information for each output point, e.g. its spatial position or its modality. We construct queries by combining (concatenating or adding) a set of vectors into a query vector containing all of the information relevant for one of the \(O\) desired outputs.

For outputs with a spatial or sequence structure, we include a position encoding (e.g. a learned positional encoding or a Fourier feature) representing the position to be decoded in the output.

We find that even very simple query features can produce good results, suggesting that the latent attention process is able to learn to organize the relevant information in a way that's easy to query. Each output point depends only on its query and the latent array, allowing us to decode outputs in parallel.

Although Transformers were originally developed for language, their quadratic complexity makes them difficult to use on language inputs without tokenization, which typically shortens the length of input sequences by a factor of \(\sim\)4. But unlike Transformer-based models such as BERT (Devlin et al., 2019) or XLNet (Yang et al., 2019), Perceiver IO scales linearly with input length.

As a token contains many bytes on average, we need to increase the sequence length to input a similar amount of text: we use input sequence lengths of \(512\) SentencePiece tokens or \(2048\) UTF-8 bytes.

For the byte-level models, the vocabulary size is much smaller: 256 bytes and 4 special tokens ([PAD], [MASK], [CLS], [SEP]).

We first observe that Perceiver IO applied on SentencePiece tokenized input sequences slightly outperforms a strong BERT baseline applied on the same inputs (81.2 vs 81.1). As a result of the reduced latent size of \(256\) we can train a much deeper network with 26 processing layers compared to BERT Base (12 layers) while maintaining a similar FLOPs budget.

Next, we show that we can leverage Perceiver IO to run on much longer sequences than a regular Transformer. Rather than using a fixed, handcrafted vocabulary, our model works directly with the raw byte inputs: we simply feed in and predict the UTF-8 bytes of the input string.

Clark et al. (2022) also uses a bottleneck architecture to scale to longer text inputs, but their upsampling strategy differs from ours: they concatenate raw inputs with their aligned downsampled latent representation, apply a 1D convolution and then run a shallow transformer stack on the resulting upsampled sequence.

Our work scales to byte-level inputs without making any assumptions about the structure of the input, which allows it to be used beyond language as shown in the following sections.

Our approach is more generic as it decouples the output array from the input array by not relying on [CLS] tokens.

In contrast, we apply Perceiver IO to flow in a straightforward manner. We concatenate the frames along the channel dimension and extract a \(3\times 3\) patch around each pixel (leading to \(3\times 3\times 3\times 2=54\) values for each pixel).

To decode, we query the latent representation using the input encoding.

With traditional autoencoding models like convolutional encoder-decoders, it is not obvious how to combine these modalities, because each uses data of different dimensions - 3D (video), 1D (raw audio), and 0D (class labels) - and with wildly different numbers of elements. With Perceiver IO, we pad the inputs with modality-specific embeddings, serialize them into a single 2D input array and query outputs using queries containing position encodings (for video and audio) and modality embeddings.

We used a latent array with 512 channels and 784, 392, and 196 latents, resulting in compression ratios of 88x, 176x, and 352x respectively.

Latent variables are shared across modalities, so the quality of reconstructions for each modality is sensitive to the weight of its loss term and other training hyperparameters.

This strongly suggests that Perceiver IO can jointly represent modalities with very different properties.

In this work we introduce Perceiver IO, an architecture capable of handling general purpose inputs and outputs while scaling linearly in both input and output sizes. As we show, this architecture achieves good results in a wide variety of settings, making it a promising candidate for a _general purpose_ neural network architecture. Perceiver IO leverages the expressive power of latent attention and uses learned queries to expose a simple and unified interface that can handle multimodal and multitask settings. Overall, Perceiver IO offers a promising way to simplify the construction of sophisticated neural pipelines and facilitate progress on multimodal and multitask problems.

These results demonstrate the unprecedented generality of Perceiver IO, the simplicity that this architecture unlocks in handling a range of tasks, and its flexibility to work as part of a domain-adapted system.

While neither the Perceiver and Perceiver IO incorporate any 2D spatial structure architecturally, they use positional features that inject 2D spatial information (Sec. 3.2 and Appendix sec. D of Jaegle et al. 2021). By replacing these 2D position features with a fully learned position encoding as used on language, we can learn an image classification model that is given no privileged information about the structure of images.

ImageNet networks that use this positional encoding are given no information about 2D image structure. For these experiments, we additionally use a 1D convolutional network to project the RGB at each point to 256 before concatenating it with the learned positional encoding.

To our knowledge, this is the best result by any model on ImageNet without 2D architectural or feature information.

We use an architecture with weight sharing in depth: the latent (processing) component of the architecture includes 8 blocks of 6 attention modules each, and weights are shared between the corresponding modules in each block.

Unlike in the other ImageNet experiments, we do not share weights in the latent self-attention process modules, but use a 16-layer latent network with no weight sharing in depth. Unlike the other ImageNet experiments, the process-module MLPs use a hidden layer with 4\(\times\) the number of channels (rather than 1\(\times\) as on other ImageNet experiments).

In other image settings discussed here, we optionally use simple pre- and post-processing steps to reduce the size of very large inputs and outputs.

Incorporating a single convolution + max pooling leads to a moderate improvement in the performance of the architecture: this is perhaps unsurprising, as convolutional pre-processing injects information about the 2D structure of images into the architecture. By comparison ViT first processes images by applying a 2D convolution with matched kernel and stride to downsample its inputs (referred to as a "linear projection of flattened patches" in that work and throughout the ViT literature).

The entity encoder takes as input a set of 512 entities (referred to as embedded_entity in Vinyals et al. (2019)) and produces as output an embedding for each entity (entity_embeddings) and a 1D embedding reduced over entities (embedded_entity).

We obtained StarCraft II results by using Perceiver IO instead of a Transformer for the AlphaStar entity encoder. We replaced the Transformer with a Perceiver IO with a latent of index dimension 32, keeping the input and output size of 512 units.

We replaced this Transformer with a 3-layer Perceiver IO with a latent of index dimension 32. We tuned only the size of the index dimension (sweeping values of 32 and 64), but otherwise used the same hyperparameters as ImageNet.

All models use an architecture with 12 processor modules and a latent index dimension \(N\) of 512 (we omit the repeated cross-attends used in Jaegle et al. (2021)).

For all four model settings, we swept the number of latent channels (using \(D\in\{512,\,1024\}\)) and report the best value for each setting.

Perceiver IO is constructed from GPT-2-style (Radford et al., 2019) Transformer attention modules, which consist of QKV attention followed by an MLP, along with linear projection layers to ensure inputs to and outputs from the QKV attention and MLP take on desired sizes. Using the array sizes of the encoder attention, the QKV attention takes in two two-dimensional arrays, a key-value input array \(X_{KV}\in\mathbb{R}^{M\times C}\) and a query input array \(X_{Q}\in\mathbb{R}^{N\times D}\), and maps them to an array \(X_{QKV}\in\mathbb{R}^{N\times D}\), sharing the shape of the query input (after projection). \(X_{QKV}\) is used as input to an MLP, which is applied independently to each element of the index dimension (i.e., convolving the MLP with its input along the first dimension), producing a final array \(X_{MLP}\in\mathbb{R}^{N\times D}\).

QKV attention takes in two two-dimensional arrays, a query input \(X_{Q}\in\mathbb{R}^{N\times D}\) and a key-value input \(X_{KV}\in\mathbb{R}^{M\times C}\). The output of QKV attention is an array with the same index (first) dimension as the query input and a channel (second) dimension determined by an output projection: \[Q=f_{Q}(X_{Q}); K=f_{K}(X_{KV}); V=f_{V}(X_{KV}) \tag{1}\] \[X_{QK}=\text{softmax}(QK^{T}/\sqrt{F})\] (2) \[\text{Attn}(X_{Q},X_{KV})=X_{QKV}=f_{O}(X_{QK}V), \tag{3}\] where \(X_{QK}\) is an array of attention maps \(\in\mathbb{R}^{N\times M}\), and \(X_{QKV}\) is an array \(\in\mathbb{R}^{N\times D}\). The functions \(f_{\{Q,K,V\}}\) are linear layers mapping each input to a shared feature dimension \(F\) and \(f_{O}\) is a linear layer projecting the output to a target channel dimension, which is often the same size as \(X_{Q}\)'s.

The computational complexity of each attention module is dominated by the two matrix multiplications in QKV attention. Still using the shapes of the encoder attention, these two matrix multiplies involve matrices of shape \(M\times F\) and \(N\times F\) and \(M\times N\) and \(N\times F\), giving overall time and memory complexity of \(\mathcal{O}(MNF)\). Let \(M\), \(N\), and \(O\) be the index dimensions for the input, latent, and output arrays, and to simplify the analysis let \(F\) be the feature size for all layers.

A model with \(L\) latent attention blocks has complexity \(\mathcal{O}([M+O+LN]NF)\). In other words, Perceiver IO has complexity linear in the size of the input and output arrays and it decouples the depth of the latent transformer from the input and output sizes. Both of these properties contribute to Perceiver IO's efficiency: while many proposals for efficient attention modules or architectures include linear or sub-quadratic scaling with input/output size, Perceiver IO is unusual in also decoupling depth from input/output size (without requiring domain-specific strategies like 2D convolution).

In addition to its greater expressivity, decoder attention has the advantage of being easily generalizable to dense outputs (by increasing the number of queries) and of reusing the same architectural pattern used for the encoder and processor modules.

For the BERT (matching FLOPs) model trained on bytes, we reduce the model width from \(768\) to \(512\), the feed-forward hidden size from \(3072\) to \(2048\), the number of layers from \(12\) to \(6\) and the number of attention heads from \(12\) to \(8\). Given the longer sequence length of \(2048\) bytes, this model has about the same number of inference FLOPs as a BERT Base model on a sequence length of \(512\) tokens.

For a given FLOPs budget, there is a trade off between the number of latents \(N\) and the width \(D\) of the latents. We ablate this in Tab. 15 by varying the number of latents between 128, 256 (best), and 512. We adapt the latent dimension accordingly to match the FLOPs budget.

\(1{\times}1\) means that only a single pixel (or pair of pixels) is used for each input element. 'Separate frames' means that the frames are not concatenated, but rather, input array elements are extracted independently from the two frames (thereby doubling the number of input elements). In the case of separate frames, \(1\times 1\) means essentially no preprocessing: each pixel becomes its own element with no spatio-temporal context whatsoever. We also performed experiments with a less expensive input model which uses a \(7\times 7\) convolution to 64 channels, followed by a max pool, similar to the one used in our ImageNet experiments. After feeding this through the Perceiver IO architecture (including querying with the same convolutional features used as input), we have an output a feature grid with stride 4 and 64 channels, on top of which we apply a RAFT upsampling layer.

We then upsample this flow for a given pixel in the high-resolution flow map by applying attention over a neighboring 3x3 block of the low-resolution flow map, following the uppsampling approach in RAFT (Teed & Deng, 2020).

Convolutional downsampling and RAFT upsampling provide even more spatial context for both the input features and the queries, but this doesn't make up for the loss of resolution and overall performs slightly worse than using the full resolution.

We use a latent array with 2048 elements and 512 channels and 24 self-attention modules, each with 16 self-attention heads, unless otherwise noted.

The decoder outputs \(16\times 224\times 224+16\times 1920/16+1\) vectors with 512 channels, that is, one element for each pixel in the video, one element for each audio patch, and one element for the classification label. These are then linearly projected to the appropriate channel size for each modality: 3 for videos, 16 for audio and 700 for classification (the logits for each of the 700 classes in Kinetics700).

Perceiver IO is able to capture the structure of both video and audio inputs, despite encoding both jointly with a single network.

this may be partially attributable to the preprocessing, which included coarse patching (Tab. 5) for both modalities due to the very high computational cost of processing raw video and audio inputs (which amount to over 2 million raw points).

Input: Figure 1: The Perceiver IO architecture can be used on domains with a wide variety of input and output spaces, including multi-task language understanding, dense visual tasks like optical flow, hybrid dense/sparse multimodal tasks such as video+audio+class autoencoding, and tasks with symbolic outputs like StarCraft II. See Tables 5 and 6 for details of all domains considered here.

Input: Figure 5: Schematic depiction of encode, process, and decode attention. Each attention module uses the same operations, but differs in which inputs are used to generate key/values or queries and in the output shape. Encode attention can be viewed as mapping an input to a latent space, typically with a smaller index dimension (fewer elements). Decode attention can be viewed as mapping a latent to an output space, often with a larger index dimension (more elements). Both of these are forms of cross-attention. Process attention (self-attention) preserves the input index dimension (same elements). Red and blue dashed lines are used to highlight the two matrix multiplications used in QKV attention, as described in the text.

Input: Figure 2: The Perceiver IO architecture. Perceiver IO maps arbitrary input arrays to arbitrary output arrays in a domain agnostic process. The bulk of the computation happens in a latent space whose size is typically smaller than the inputs and outputs, which makes the process computationally tractable even for very large inputs & outputs. See Fig. 5 for a more detailed look at encode, process, and decode attention.

Input: Table 5: Details of each of the tasks we use to evaluate Perceiver IO here. The positional and task embeddings appended to inputs for each case are listed in Tab. 6.

Input: Table 6: Table best viewed on a screen. The structure and size of the positional and task embeddings used to construct Perceiver IO’s encoder key-value inputs and decoder query inputs, for each domain described in the main text. “[x, y]” indicates that x’s and y’s features are concatenated, while “x + y” indicates that x’s and y’s features are added to produce the full featurization. “FF” = Fourier features as in Jaegle et al. (2021).

Input: Figure 6: Single-query attention decoder (left), as used in Perceiver IO for classification tasks and a standard average + project decoder (right), as used in Jaegle et al. (2021). Both modules can be seen as first aggregating latents by weighted averaging (learned, data-dependent weighting for the attention decoder; uniform weights for the average + project decoder) and then projecting to an output channel dimension (linear value projection + MLP for the attention decoder; simple linear projection by the average + project decoder). Attentional decoding is more expressive than average + project decoding and follows the same architectural template as encoder and processor modules.

Input: Figure 3: We construct queries with output-specific features to produce outputs with different semantics. For settings where each output point differs only in its position, like language, a position embedding can be used. Input features for the target output can also be used to query, either alone (as for StarCraft II) or alongside position features (as for flow). For multi-{task, modal} settings we use one embedding for each {task, modality} instead of each position. A single learned embedding suffices for simple classification tasks, like ImageNet. For tasks with heterogeneous outputs like multimodal autoencoding, features that are specific to some queries (like xy position) can be combined with modality embeddings, which also pad embeddings to fixed length.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "This approach allows us to decouple the size of elements used for the bulk of the computation (the latent) from the size of the input and output spaces, while making minimal assumptions about the spatial or locality structure of the input and output."

The evidence strongly supports this claim. The architecture demonstrates:
1. Linear scaling with input/output size while computation happens in a smaller latent space (O([M+O+LN]NF))
2. Successful application across diverse domains without domain-specific modifications
3. Flexibility in handling different input/output structures (text, video, audio, etc.)
4. No assumptions about spatial relationships (works even without 2D positional encodings for images)

The claim accurately reflects capabilities demonstrated in the evidence without exaggeration. The technical details and experimental results consistently validate the decoupling property and domain-agnostic nature described.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

This approach allows us to decouple the size of elements used for the bulk of the computation (the latent) from the size of the input and output spaces, while making minimal assumptions about the spatial or locality structure of the input and output.

The review comment to be evaluated is:

#### Strengths

- Major contribution is ability to scale the output to arbitrary length using an output query array. This enables supporting different types of output and scales well to different tasks.
- Byte level performance of Perceiver IO is impressive compared to BERT baseline. Byte level embeddings can be fed to Perceiver IO without any tokenization and this setting gets comparable performance with BERT sentence piece tokenization.
- Shows good results with multitask learning, using both single and task specific tokens.
- Authors do a thorough evaluation of this architecture on a wide variety of tasks, and achieves comparable results on multiple tasks in language, vision, multimodal, RL domains.

#### Weaknesses

- Novelty over Perceiver : Although I find this work quite impressive as it scales linearly with output embedding sizes as well as arbitrary types of outputs, the overall architecture seems incremental compared to Perceiver. One important change is there is no cross attention in the intermediate layers. I would love to see more detailed analysis why that is removed in this architecture and empirical evidence to support this change.

- How does choosing values for N,D affect the performance on various tasks. If we decouple input representation, output representation and base architecture, how does model performance scale with depth, N, D values.  Ablations on these hyper-parameters are required for more clarity.

- Given Perceiver can handle multiple modalities, I would be interested to see how it performs on multimodal tasks like VQA, Image/Text Retrieval, Hateful Memes etc and comparison to multimodal models like CLIP[1], ALIGN[2], SimVL[3].

  - [1] Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry et al. "Learning transferable visual models from natural language supervision." arXiv preprint arXiv:2103.00020 (2021).
  - [2] Jia, Chao, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. "Scaling up visual and vision-language representation learning with noisy text supervision." arXiv preprint arXiv:2102.05918 (2021).
  - [3] Wang, Zirui, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. "SimVLM: Simple visual language model pretraining with weak supervision." arXiv preprint arXiv:2108.10904 (2021).

The evidence to be evaluated is:Several groups have proposed to use attention to manipulate the size of arrays or to introduce bottlenecks in processing. Set Transformers and related work (Lee et al., 2019; Goyal et al., 2022) use a learned query ("inducing points") to induce local bottlenecks by mapping a set back and forth from a set with fewer elements and learned decoder queries ("seed vectors") to map to outputs ("pooling by multiheaded attention"). Each layer of these networks has complexity linear in the input size, while Perceivers use a deep latent network with complexity independent of the input and output.

The Perceiver IO architecture builds on the Perceiver (Jaegle et al., 2021), which achieved its cross-domain generality by assuming that its input is a simple 2D byte array: a set of elements (which might be pixels or patches in vision, characters or words in language, or some form of embedding, learned or otherwise), each described by a feature vector. The model then encodes information about the input array using a smaller number of latent feature vectors, using Transformer-style attention, followed by iterative processing and a final aggregation down to a category label. Rather than output a single category, Perceiver IO aims to have the same level of generality with respect to its _outputs_ as the Perceiver has with respect to its _inputs_: that is, it should produce arbitrary output arrays.

We first **encode** by applying an attention module that maps input arrays \(x\in\mathbb{R}^{M\times C}\) to arrays in a latent space \(z\in\mathbb{R}^{N\times D}\). We next **process** the latents \(z\) by applying a series of modules that take in and return arrays in this latent space. Finally, we **decode** by applying an attention module that maps latent arrays to output arrays \(y\in\mathbb{R}^{O\times E}\). \(M\), \(C\), \(O\), and \(E\) are properties of the task data and can be very large (Tab. 5), while \(N\) and \(D\) are hyperparameters and can be chosen to make model computation tractable.

Perceiver IO uses attention non-homogeneously by mapping inputs to a latent space, processing in that latent space, and decoding to an output space. Perceiver IO has no quadratic dependence on the input or output size: encoder and decoder attention modules depend linearly on the input and output size (respectively), while latent attention is independent of both input and output sizes (Sec. E.2).

This architecture can be applied to inputs of any shape or spatial layout including inputs or outputs with different spatial structure (e.g. sound and video). In contrast to latent spaces typically used in vision (e.g. Ronneberger et al., 2015) the latent does not explicitly share the structure (spatial or otherwise) of the inputs.

Our goal is to produce a final output array of size \(O\times E\), given a latent representation of size \(N\times D\). We produce an output of this size by querying the decoder with an array of index dimension \(O\). To capture the structure of the output space, we use queries containing the appropriate information for each output point, e.g. its spatial position or its modality. We construct queries by combining (concatenating or adding) a set of vectors into a query vector containing all of the information relevant for one of the \(O\) desired outputs.

For outputs with a spatial or sequence structure, we include a position encoding (e.g. a learned positional encoding or a Fourier feature) representing the position to be decoded in the output.

We find that even very simple query features can produce good results, suggesting that the latent attention process is able to learn to organize the relevant information in a way that's easy to query. Each output point depends only on its query and the latent array, allowing us to decode outputs in parallel.

Although Transformers were originally developed for language, their quadratic complexity makes them difficult to use on language inputs without tokenization, which typically shortens the length of input sequences by a factor of \(\sim\)4. But unlike Transformer-based models such as BERT (Devlin et al., 2019) or XLNet (Yang et al., 2019), Perceiver IO scales linearly with input length.

As a token contains many bytes on average, we need to increase the sequence length to input a similar amount of text: we use input sequence lengths of \(512\) SentencePiece tokens or \(2048\) UTF-8 bytes.

For the byte-level models, the vocabulary size is much smaller: 256 bytes and 4 special tokens ([PAD], [MASK], [CLS], [SEP]).

We first observe that Perceiver IO applied on SentencePiece tokenized input sequences slightly outperforms a strong BERT baseline applied on the same inputs (81.2 vs 81.1). As a result of the reduced latent size of \(256\) we can train a much deeper network with 26 processing layers compared to BERT Base (12 layers) while maintaining a similar FLOPs budget.

Next, we show that we can leverage Perceiver IO to run on much longer sequences than a regular Transformer. Rather than using a fixed, handcrafted vocabulary, our model works directly with the raw byte inputs: we simply feed in and predict the UTF-8 bytes of the input string.

Clark et al. (2022) also uses a bottleneck architecture to scale to longer text inputs, but their upsampling strategy differs from ours: they concatenate raw inputs with their aligned downsampled latent representation, apply a 1D convolution and then run a shallow transformer stack on the resulting upsampled sequence.

Our work scales to byte-level inputs without making any assumptions about the structure of the input, which allows it to be used beyond language as shown in the following sections.

Our approach is more generic as it decouples the output array from the input array by not relying on [CLS] tokens.

In contrast, we apply Perceiver IO to flow in a straightforward manner. We concatenate the frames along the channel dimension and extract a \(3\times 3\) patch around each pixel (leading to \(3\times 3\times 3\times 2=54\) values for each pixel).

To decode, we query the latent representation using the input encoding.

With traditional autoencoding models like convolutional encoder-decoders, it is not obvious how to combine these modalities, because each uses data of different dimensions - 3D (video), 1D (raw audio), and 0D (class labels) - and with wildly different numbers of elements. With Perceiver IO, we pad the inputs with modality-specific embeddings, serialize them into a single 2D input array and query outputs using queries containing position encodings (for video and audio) and modality embeddings.

We used a latent array with 512 channels and 784, 392, and 196 latents, resulting in compression ratios of 88x, 176x, and 352x respectively.

Latent variables are shared across modalities, so the quality of reconstructions for each modality is sensitive to the weight of its loss term and other training hyperparameters.

This strongly suggests that Perceiver IO can jointly represent modalities with very different properties.

In this work we introduce Perceiver IO, an architecture capable of handling general purpose inputs and outputs while scaling linearly in both input and output sizes. As we show, this architecture achieves good results in a wide variety of settings, making it a promising candidate for a _general purpose_ neural network architecture. Perceiver IO leverages the expressive power of latent attention and uses learned queries to expose a simple and unified interface that can handle multimodal and multitask settings. Overall, Perceiver IO offers a promising way to simplify the construction of sophisticated neural pipelines and facilitate progress on multimodal and multitask problems.

These results demonstrate the unprecedented generality of Perceiver IO, the simplicity that this architecture unlocks in handling a range of tasks, and its flexibility to work as part of a domain-adapted system.

While neither the Perceiver and Perceiver IO incorporate any 2D spatial structure architecturally, they use positional features that inject 2D spatial information (Sec. 3.2 and Appendix sec. D of Jaegle et al. 2021). By replacing these 2D position features with a fully learned position encoding as used on language, we can learn an image classification model that is given no privileged information about the structure of images.

ImageNet networks that use this positional encoding are given no information about 2D image structure. For these experiments, we additionally use a 1D convolutional network to project the RGB at each point to 256 before concatenating it with the learned positional encoding.

To our knowledge, this is the best result by any model on ImageNet without 2D architectural or feature information.

We use an architecture with weight sharing in depth: the latent (processing) component of the architecture includes 8 blocks of 6 attention modules each, and weights are shared between the corresponding modules in each block.

Unlike in the other ImageNet experiments, we do not share weights in the latent self-attention process modules, but use a 16-layer latent network with no weight sharing in depth. Unlike the other ImageNet experiments, the process-module MLPs use a hidden layer with 4\(\times\) the number of channels (rather than 1\(\times\) as on other ImageNet experiments).

In other image settings discussed here, we optionally use simple pre- and post-processing steps to reduce the size of very large inputs and outputs.

Incorporating a single convolution + max pooling leads to a moderate improvement in the performance of the architecture: this is perhaps unsurprising, as convolutional pre-processing injects information about the 2D structure of images into the architecture. By comparison ViT first processes images by applying a 2D convolution with matched kernel and stride to downsample its inputs (referred to as a "linear projection of flattened patches" in that work and throughout the ViT literature).

The entity encoder takes as input a set of 512 entities (referred to as embedded_entity in Vinyals et al. (2019)) and produces as output an embedding for each entity (entity_embeddings) and a 1D embedding reduced over entities (embedded_entity).

We obtained StarCraft II results by using Perceiver IO instead of a Transformer for the AlphaStar entity encoder. We replaced the Transformer with a Perceiver IO with a latent of index dimension 32, keeping the input and output size of 512 units.

We replaced this Transformer with a 3-layer Perceiver IO with a latent of index dimension 32. We tuned only the size of the index dimension (sweeping values of 32 and 64), but otherwise used the same hyperparameters as ImageNet.

All models use an architecture with 12 processor modules and a latent index dimension \(N\) of 512 (we omit the repeated cross-attends used in Jaegle et al. (2021)).

For all four model settings, we swept the number of latent channels (using \(D\in\{512,\,1024\}\)) and report the best value for each setting.

Perceiver IO is constructed from GPT-2-style (Radford et al., 2019) Transformer attention modules, which consist of QKV attention followed by an MLP, along with linear projection layers to ensure inputs to and outputs from the QKV attention and MLP take on desired sizes. Using the array sizes of the encoder attention, the QKV attention takes in two two-dimensional arrays, a key-value input array \(X_{KV}\in\mathbb{R}^{M\times C}\) and a query input array \(X_{Q}\in\mathbb{R}^{N\times D}\), and maps them to an array \(X_{QKV}\in\mathbb{R}^{N\times D}\), sharing the shape of the query input (after projection). \(X_{QKV}\) is used as input to an MLP, which is applied independently to each element of the index dimension (i.e., convolving the MLP with its input along the first dimension), producing a final array \(X_{MLP}\in\mathbb{R}^{N\times D}\).

QKV attention takes in two two-dimensional arrays, a query input \(X_{Q}\in\mathbb{R}^{N\times D}\) and a key-value input \(X_{KV}\in\mathbb{R}^{M\times C}\). The output of QKV attention is an array with the same index (first) dimension as the query input and a channel (second) dimension determined by an output projection: \[Q=f_{Q}(X_{Q}); K=f_{K}(X_{KV}); V=f_{V}(X_{KV}) \tag{1}\] \[X_{QK}=\text{softmax}(QK^{T}/\sqrt{F})\] (2) \[\text{Attn}(X_{Q},X_{KV})=X_{QKV}=f_{O}(X_{QK}V), \tag{3}\] where \(X_{QK}\) is an array of attention maps \(\in\mathbb{R}^{N\times M}\), and \(X_{QKV}\) is an array \(\in\mathbb{R}^{N\times D}\). The functions \(f_{\{Q,K,V\}}\) are linear layers mapping each input to a shared feature dimension \(F\) and \(f_{O}\) is a linear layer projecting the output to a target channel dimension, which is often the same size as \(X_{Q}\)'s.

The computational complexity of each attention module is dominated by the two matrix multiplications in QKV attention. Still using the shapes of the encoder attention, these two matrix multiplies involve matrices of shape \(M\times F\) and \(N\times F\) and \(M\times N\) and \(N\times F\), giving overall time and memory complexity of \(\mathcal{O}(MNF)\). Let \(M\), \(N\), and \(O\) be the index dimensions for the input, latent, and output arrays, and to simplify the analysis let \(F\) be the feature size for all layers.

A model with \(L\) latent attention blocks has complexity \(\mathcal{O}([M+O+LN]NF)\). In other words, Perceiver IO has complexity linear in the size of the input and output arrays and it decouples the depth of the latent transformer from the input and output sizes. Both of these properties contribute to Perceiver IO's efficiency: while many proposals for efficient attention modules or architectures include linear or sub-quadratic scaling with input/output size, Perceiver IO is unusual in also decoupling depth from input/output size (without requiring domain-specific strategies like 2D convolution).

In addition to its greater expressivity, decoder attention has the advantage of being easily generalizable to dense outputs (by increasing the number of queries) and of reusing the same architectural pattern used for the encoder and processor modules.

For the BERT (matching FLOPs) model trained on bytes, we reduce the model width from \(768\) to \(512\), the feed-forward hidden size from \(3072\) to \(2048\), the number of layers from \(12\) to \(6\) and the number of attention heads from \(12\) to \(8\). Given the longer sequence length of \(2048\) bytes, this model has about the same number of inference FLOPs as a BERT Base model on a sequence length of \(512\) tokens.

For a given FLOPs budget, there is a trade off between the number of latents \(N\) and the width \(D\) of the latents. We ablate this in Tab. 15 by varying the number of latents between 128, 256 (best), and 512. We adapt the latent dimension accordingly to match the FLOPs budget.

\(1{\times}1\) means that only a single pixel (or pair of pixels) is used for each input element. 'Separate frames' means that the frames are not concatenated, but rather, input array elements are extracted independently from the two frames (thereby doubling the number of input elements). In the case of separate frames, \(1\times 1\) means essentially no preprocessing: each pixel becomes its own element with no spatio-temporal context whatsoever. We also performed experiments with a less expensive input model which uses a \(7\times 7\) convolution to 64 channels, followed by a max pool, similar to the one used in our ImageNet experiments. After feeding this through the Perceiver IO architecture (including querying with the same convolutional features used as input), we have an output a feature grid with stride 4 and 64 channels, on top of which we apply a RAFT upsampling layer.

We then upsample this flow for a given pixel in the high-resolution flow map by applying attention over a neighboring 3x3 block of the low-resolution flow map, following the uppsampling approach in RAFT (Teed & Deng, 2020).

Convolutional downsampling and RAFT upsampling provide even more spatial context for both the input features and the queries, but this doesn't make up for the loss of resolution and overall performs slightly worse than using the full resolution.

We use a latent array with 2048 elements and 512 channels and 24 self-attention modules, each with 16 self-attention heads, unless otherwise noted.

The decoder outputs \(16\times 224\times 224+16\times 1920/16+1\) vectors with 512 channels, that is, one element for each pixel in the video, one element for each audio patch, and one element for the classification label. These are then linearly projected to the appropriate channel size for each modality: 3 for videos, 16 for audio and 700 for classification (the logits for each of the 700 classes in Kinetics700).

Perceiver IO is able to capture the structure of both video and audio inputs, despite encoding both jointly with a single network.

this may be partially attributable to the preprocessing, which included coarse patching (Tab. 5) for both modalities due to the very high computational cost of processing raw video and audio inputs (which amount to over 2 million raw points).

Input: Figure 1: The Perceiver IO architecture can be used on domains with a wide variety of input and output spaces, including multi-task language understanding, dense visual tasks like optical flow, hybrid dense/sparse multimodal tasks such as video+audio+class autoencoding, and tasks with symbolic outputs like StarCraft II. See Tables 5 and 6 for details of all domains considered here.

Input: Figure 5: Schematic depiction of encode, process, and decode attention. Each attention module uses the same operations, but differs in which inputs are used to generate key/values or queries and in the output shape. Encode attention can be viewed as mapping an input to a latent space, typically with a smaller index dimension (fewer elements). Decode attention can be viewed as mapping a latent to an output space, often with a larger index dimension (more elements). Both of these are forms of cross-attention. Process attention (self-attention) preserves the input index dimension (same elements). Red and blue dashed lines are used to highlight the two matrix multiplications used in QKV attention, as described in the text.

Input: Figure 2: The Perceiver IO architecture. Perceiver IO maps arbitrary input arrays to arbitrary output arrays in a domain agnostic process. The bulk of the computation happens in a latent space whose size is typically smaller than the inputs and outputs, which makes the process computationally tractable even for very large inputs & outputs. See Fig. 5 for a more detailed look at encode, process, and decode attention.

Input: Table 5: Details of each of the tasks we use to evaluate Perceiver IO here. The positional and task embeddings appended to inputs for each case are listed in Tab. 6.

Input: Table 6: Table best viewed on a screen. The structure and size of the positional and task embeddings used to construct Perceiver IO’s encoder key-value inputs and decoder query inputs, for each domain described in the main text. “[x, y]” indicates that x’s and y’s features are concatenated, while “x + y” indicates that x’s and y’s features are added to produce the full featurization. “FF” = Fourier features as in Jaegle et al. (2021).

Input: Figure 6: Single-query attention decoder (left), as used in Perceiver IO for classification tasks and a standard average + project decoder (right), as used in Jaegle et al. (2021). Both modules can be seen as first aggregating latents by weighted averaging (learned, data-dependent weighting for the attention decoder; uniform weights for the average + project decoder) and then projecting to an output channel dimension (linear value projection + MLP for the attention decoder; simple linear projection by the average + project decoder). Attentional decoding is more expressive than average + project decoding and follows the same architectural template as encoder and processor modules.

Input: Figure 3: We construct queries with output-specific features to produce outputs with different semantics. For settings where each output point differs only in its position, like language, a position embedding can be used. Input features for the target output can also be used to query, either alone (as for StarCraft II) or alongside position features (as for flow). For multi-{task, modal} settings we use one embedding for each {task, modality} instead of each position. A single learned embedding suffices for simple classification tasks, like ImageNet. For tasks with heterogeneous outputs like multimodal autoencoding, features that are specific to some queries (like xy position) can be combined with modality embeddings, which also pad embeddings to fixed length.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review, the claim makes a reasonable assertion about Perceiver IO's ability to decouple computation size from input/output spaces. The evidence strongly supports this through:

1. Mathematical analysis showing linear complexity in input/output sizes while maintaining independence in latent processing
2. Demonstrated success across diverse domains (text, video, audio, games) with varying input/output structures
3. Empirical results showing competitive performance despite minimal assumptions about spatial/locality structure
4. Clear architectural design choices that enable this decoupling through the encode-process-decode pattern

The claim avoids overstating the capabilities and accurately reflects what the evidence demonstrates about the architecture's properties. The review comments also support this assessment, highlighting the scalability benefits while noting some remaining questions about parameter choices.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

This approach allows us to decouple the size of elements used for the bulk of the computation (the latent) from the size of the input and output spaces, while making minimal assumptions about the spatial or locality structure of the input and output.

The review comment to be evaluated is:

The main strength of Perceiver IO is its high flexibility to various input domains. While keeping the main architecture, we can focus on engineering the input query preprocessing with domain-specific knowledge. Perceiver IO not only shows flexibility but also shows high performance. The results are comparable to or sometimes outperform previous domain-specific methods. For example

- Table 1 shows that Perceiver IO works well even without tokenization, while BERT + UTF-8 bytes show a significant performance drop (81.0 -> 71.5). It may show that Perceiver IO can perform well even without minimal domain knowledge (tokenization).
- Table 2 shows that the input query engineering can improve the target task (e.g., single-task query to 8 multi-task queries).
- Table 5 shows that the small input preprocessing engineering can improve the performance a lot. For example, Perceiver IO shows 82.1 top-1 accuracies for conv preprocessing while Perceiver IO with Fourier features shows 79.0.

I have minor concerns about ImageNet experiments.

- Although this paper is not aiming to state-of-the-art ImageNet performances, I think Perceiver IO is not yet comparable to state-of-the-art models on ImageNet image classification (as described in the last paragraph of Section 1). Perceiver IO still needs tremendous FLOPs compared to vision-specific models (Table 17). For example, ResNet shows 78.6 top-1 accuracy with 4.1 GFLOPs, while Perceiver IO Fourier shows 79.0 top-1 accuracy with 407 GFLOPs (99 times larger FLOPs). Stronger Perceiver IO (conv) shows 82.1 top-1 accuracy with 369 GFLOPs, where CaiT-M48 448 shows 86.5 top-1 accuracy and 329.6 GFLOPs.
- "Perceiver IO (pretrained)" and "Perceiver IO" in 2D Fourier features are different models. For example, "Perceiver IO (pretrained)" does not use weight sharing, and uses larger channel sizes. As a result, Perceiver IO (pretrained) has 4.38 times larger parameters compared to Perceiver IO (but half GFLOPs). The current text can make a reader mislead that "Perceiver IO (pretrained)" is the same model as "Perceiver IO", but the only difference is the JFT pretraining, which is not true.

> Unlike in the other ImageNet experiments, we do not share weights in the latent self-attention process modules, but use a 16-layer latent network with no weight sharing in depth. Unlike the other ImageNet experiments, the process-module MLPs use a hidden layer with 4× the number of channels (rather than 1× as on other ImageNet experiments)

To avoid confusion, I suggest replacing Table 5 with Table 17. Also, I would like to suggest to clarify that "Perceiver IO (pretrained)" and "Perceiver IO" are not the same model.

## Questions

- Perceiver IO shows better performances consistently compared to Perceiver IO baseline on GLUE (Table 13), Audio Set (Table 7), and ImageNet (Table 17). To my knowledge, there is no specific reason to the proposed output array cross-attention works better than the conventional classification head. I wonder what is the opinion by the authors what is the source of the improvements.
- In Table 5, I found that using 2D + maxpool preprocessing harms Perceiver performances (78.6 -> 77.4), but it helps Perceiver IO a lot (79.0 -> 82.1). As far as the reviewer understood, the main difference between Perceiver and Perceiver IO is the last decoding layer, not the input processing. I wonder what is the opinion of the authors why Perceiver and Perceiver IO show contradictory results on ImageNet Fourier features and conv preprocessing.
- I wonder why there is no Perceiver IO (conv) + pretraining result. Do the authors have any plan to add Perceiver IO (conv + pretrained)? (this is not a mandatory experiment for the rebuttal)

## Minor comments

- Many details are missing in the main text. To understand the method, the readers should read through the heavy appendix and the previous Perceiver paper. I fully understand that this is because of the page limitation, but this paper is somewhat hard to understand at first glance without any prior knowledge.
- In my opinion, it would be very helpful to readers if Table 8 and 9 are in the main text.
- pg 3. Typo -- unnecessary space *( "inducing points")*
- pg 29. Missing hyperlink (Tab. ??)
- In my opinion, it will be very interesting if there is one unified Perceiver model pretrained on various domain data. For example, one can train a Perceiver model trained by language and vision modalities, where each modality has different input and output arrays.

The evidence to be evaluated is:Several groups have proposed to use attention to manipulate the size of arrays or to introduce bottlenecks in processing. Set Transformers and related work (Lee et al., 2019; Goyal et al., 2022) use a learned query ("inducing points") to induce local bottlenecks by mapping a set back and forth from a set with fewer elements and learned decoder queries ("seed vectors") to map to outputs ("pooling by multiheaded attention"). Each layer of these networks has complexity linear in the input size, while Perceivers use a deep latent network with complexity independent of the input and output.

The Perceiver IO architecture builds on the Perceiver (Jaegle et al., 2021), which achieved its cross-domain generality by assuming that its input is a simple 2D byte array: a set of elements (which might be pixels or patches in vision, characters or words in language, or some form of embedding, learned or otherwise), each described by a feature vector. The model then encodes information about the input array using a smaller number of latent feature vectors, using Transformer-style attention, followed by iterative processing and a final aggregation down to a category label. Rather than output a single category, Perceiver IO aims to have the same level of generality with respect to its _outputs_ as the Perceiver has with respect to its _inputs_: that is, it should produce arbitrary output arrays.

We first **encode** by applying an attention module that maps input arrays \(x\in\mathbb{R}^{M\times C}\) to arrays in a latent space \(z\in\mathbb{R}^{N\times D}\). We next **process** the latents \(z\) by applying a series of modules that take in and return arrays in this latent space. Finally, we **decode** by applying an attention module that maps latent arrays to output arrays \(y\in\mathbb{R}^{O\times E}\). \(M\), \(C\), \(O\), and \(E\) are properties of the task data and can be very large (Tab. 5), while \(N\) and \(D\) are hyperparameters and can be chosen to make model computation tractable.

Perceiver IO uses attention non-homogeneously by mapping inputs to a latent space, processing in that latent space, and decoding to an output space. Perceiver IO has no quadratic dependence on the input or output size: encoder and decoder attention modules depend linearly on the input and output size (respectively), while latent attention is independent of both input and output sizes (Sec. E.2).

This architecture can be applied to inputs of any shape or spatial layout including inputs or outputs with different spatial structure (e.g. sound and video). In contrast to latent spaces typically used in vision (e.g. Ronneberger et al., 2015) the latent does not explicitly share the structure (spatial or otherwise) of the inputs.

Our goal is to produce a final output array of size \(O\times E\), given a latent representation of size \(N\times D\). We produce an output of this size by querying the decoder with an array of index dimension \(O\). To capture the structure of the output space, we use queries containing the appropriate information for each output point, e.g. its spatial position or its modality. We construct queries by combining (concatenating or adding) a set of vectors into a query vector containing all of the information relevant for one of the \(O\) desired outputs.

For outputs with a spatial or sequence structure, we include a position encoding (e.g. a learned positional encoding or a Fourier feature) representing the position to be decoded in the output.

We find that even very simple query features can produce good results, suggesting that the latent attention process is able to learn to organize the relevant information in a way that's easy to query. Each output point depends only on its query and the latent array, allowing us to decode outputs in parallel.

Although Transformers were originally developed for language, their quadratic complexity makes them difficult to use on language inputs without tokenization, which typically shortens the length of input sequences by a factor of \(\sim\)4. But unlike Transformer-based models such as BERT (Devlin et al., 2019) or XLNet (Yang et al., 2019), Perceiver IO scales linearly with input length.

As a token contains many bytes on average, we need to increase the sequence length to input a similar amount of text: we use input sequence lengths of \(512\) SentencePiece tokens or \(2048\) UTF-8 bytes.

For the byte-level models, the vocabulary size is much smaller: 256 bytes and 4 special tokens ([PAD], [MASK], [CLS], [SEP]).

We first observe that Perceiver IO applied on SentencePiece tokenized input sequences slightly outperforms a strong BERT baseline applied on the same inputs (81.2 vs 81.1). As a result of the reduced latent size of \(256\) we can train a much deeper network with 26 processing layers compared to BERT Base (12 layers) while maintaining a similar FLOPs budget.

Next, we show that we can leverage Perceiver IO to run on much longer sequences than a regular Transformer. Rather than using a fixed, handcrafted vocabulary, our model works directly with the raw byte inputs: we simply feed in and predict the UTF-8 bytes of the input string.

Clark et al. (2022) also uses a bottleneck architecture to scale to longer text inputs, but their upsampling strategy differs from ours: they concatenate raw inputs with their aligned downsampled latent representation, apply a 1D convolution and then run a shallow transformer stack on the resulting upsampled sequence.

Our work scales to byte-level inputs without making any assumptions about the structure of the input, which allows it to be used beyond language as shown in the following sections.

Our approach is more generic as it decouples the output array from the input array by not relying on [CLS] tokens.

In contrast, we apply Perceiver IO to flow in a straightforward manner. We concatenate the frames along the channel dimension and extract a \(3\times 3\) patch around each pixel (leading to \(3\times 3\times 3\times 2=54\) values for each pixel).

To decode, we query the latent representation using the input encoding.

With traditional autoencoding models like convolutional encoder-decoders, it is not obvious how to combine these modalities, because each uses data of different dimensions - 3D (video), 1D (raw audio), and 0D (class labels) - and with wildly different numbers of elements. With Perceiver IO, we pad the inputs with modality-specific embeddings, serialize them into a single 2D input array and query outputs using queries containing position encodings (for video and audio) and modality embeddings.

We used a latent array with 512 channels and 784, 392, and 196 latents, resulting in compression ratios of 88x, 176x, and 352x respectively.

Latent variables are shared across modalities, so the quality of reconstructions for each modality is sensitive to the weight of its loss term and other training hyperparameters.

This strongly suggests that Perceiver IO can jointly represent modalities with very different properties.

In this work we introduce Perceiver IO, an architecture capable of handling general purpose inputs and outputs while scaling linearly in both input and output sizes. As we show, this architecture achieves good results in a wide variety of settings, making it a promising candidate for a _general purpose_ neural network architecture. Perceiver IO leverages the expressive power of latent attention and uses learned queries to expose a simple and unified interface that can handle multimodal and multitask settings. Overall, Perceiver IO offers a promising way to simplify the construction of sophisticated neural pipelines and facilitate progress on multimodal and multitask problems.

These results demonstrate the unprecedented generality of Perceiver IO, the simplicity that this architecture unlocks in handling a range of tasks, and its flexibility to work as part of a domain-adapted system.

While neither the Perceiver and Perceiver IO incorporate any 2D spatial structure architecturally, they use positional features that inject 2D spatial information (Sec. 3.2 and Appendix sec. D of Jaegle et al. 2021). By replacing these 2D position features with a fully learned position encoding as used on language, we can learn an image classification model that is given no privileged information about the structure of images.

ImageNet networks that use this positional encoding are given no information about 2D image structure. For these experiments, we additionally use a 1D convolutional network to project the RGB at each point to 256 before concatenating it with the learned positional encoding.

To our knowledge, this is the best result by any model on ImageNet without 2D architectural or feature information.

We use an architecture with weight sharing in depth: the latent (processing) component of the architecture includes 8 blocks of 6 attention modules each, and weights are shared between the corresponding modules in each block.

Unlike in the other ImageNet experiments, we do not share weights in the latent self-attention process modules, but use a 16-layer latent network with no weight sharing in depth. Unlike the other ImageNet experiments, the process-module MLPs use a hidden layer with 4\(\times\) the number of channels (rather than 1\(\times\) as on other ImageNet experiments).

In other image settings discussed here, we optionally use simple pre- and post-processing steps to reduce the size of very large inputs and outputs.

Incorporating a single convolution + max pooling leads to a moderate improvement in the performance of the architecture: this is perhaps unsurprising, as convolutional pre-processing injects information about the 2D structure of images into the architecture. By comparison ViT first processes images by applying a 2D convolution with matched kernel and stride to downsample its inputs (referred to as a "linear projection of flattened patches" in that work and throughout the ViT literature).

The entity encoder takes as input a set of 512 entities (referred to as embedded_entity in Vinyals et al. (2019)) and produces as output an embedding for each entity (entity_embeddings) and a 1D embedding reduced over entities (embedded_entity).

We obtained StarCraft II results by using Perceiver IO instead of a Transformer for the AlphaStar entity encoder. We replaced the Transformer with a Perceiver IO with a latent of index dimension 32, keeping the input and output size of 512 units.

We replaced this Transformer with a 3-layer Perceiver IO with a latent of index dimension 32. We tuned only the size of the index dimension (sweeping values of 32 and 64), but otherwise used the same hyperparameters as ImageNet.

All models use an architecture with 12 processor modules and a latent index dimension \(N\) of 512 (we omit the repeated cross-attends used in Jaegle et al. (2021)).

For all four model settings, we swept the number of latent channels (using \(D\in\{512,\,1024\}\)) and report the best value for each setting.

Perceiver IO is constructed from GPT-2-style (Radford et al., 2019) Transformer attention modules, which consist of QKV attention followed by an MLP, along with linear projection layers to ensure inputs to and outputs from the QKV attention and MLP take on desired sizes. Using the array sizes of the encoder attention, the QKV attention takes in two two-dimensional arrays, a key-value input array \(X_{KV}\in\mathbb{R}^{M\times C}\) and a query input array \(X_{Q}\in\mathbb{R}^{N\times D}\), and maps them to an array \(X_{QKV}\in\mathbb{R}^{N\times D}\), sharing the shape of the query input (after projection). \(X_{QKV}\) is used as input to an MLP, which is applied independently to each element of the index dimension (i.e., convolving the MLP with its input along the first dimension), producing a final array \(X_{MLP}\in\mathbb{R}^{N\times D}\).

QKV attention takes in two two-dimensional arrays, a query input \(X_{Q}\in\mathbb{R}^{N\times D}\) and a key-value input \(X_{KV}\in\mathbb{R}^{M\times C}\). The output of QKV attention is an array with the same index (first) dimension as the query input and a channel (second) dimension determined by an output projection: \[Q=f_{Q}(X_{Q}); K=f_{K}(X_{KV}); V=f_{V}(X_{KV}) \tag{1}\] \[X_{QK}=\text{softmax}(QK^{T}/\sqrt{F})\] (2) \[\text{Attn}(X_{Q},X_{KV})=X_{QKV}=f_{O}(X_{QK}V), \tag{3}\] where \(X_{QK}\) is an array of attention maps \(\in\mathbb{R}^{N\times M}\), and \(X_{QKV}\) is an array \(\in\mathbb{R}^{N\times D}\). The functions \(f_{\{Q,K,V\}}\) are linear layers mapping each input to a shared feature dimension \(F\) and \(f_{O}\) is a linear layer projecting the output to a target channel dimension, which is often the same size as \(X_{Q}\)'s.

The computational complexity of each attention module is dominated by the two matrix multiplications in QKV attention. Still using the shapes of the encoder attention, these two matrix multiplies involve matrices of shape \(M\times F\) and \(N\times F\) and \(M\times N\) and \(N\times F\), giving overall time and memory complexity of \(\mathcal{O}(MNF)\). Let \(M\), \(N\), and \(O\) be the index dimensions for the input, latent, and output arrays, and to simplify the analysis let \(F\) be the feature size for all layers.

A model with \(L\) latent attention blocks has complexity \(\mathcal{O}([M+O+LN]NF)\). In other words, Perceiver IO has complexity linear in the size of the input and output arrays and it decouples the depth of the latent transformer from the input and output sizes. Both of these properties contribute to Perceiver IO's efficiency: while many proposals for efficient attention modules or architectures include linear or sub-quadratic scaling with input/output size, Perceiver IO is unusual in also decoupling depth from input/output size (without requiring domain-specific strategies like 2D convolution).

In addition to its greater expressivity, decoder attention has the advantage of being easily generalizable to dense outputs (by increasing the number of queries) and of reusing the same architectural pattern used for the encoder and processor modules.

For the BERT (matching FLOPs) model trained on bytes, we reduce the model width from \(768\) to \(512\), the feed-forward hidden size from \(3072\) to \(2048\), the number of layers from \(12\) to \(6\) and the number of attention heads from \(12\) to \(8\). Given the longer sequence length of \(2048\) bytes, this model has about the same number of inference FLOPs as a BERT Base model on a sequence length of \(512\) tokens.

For a given FLOPs budget, there is a trade off between the number of latents \(N\) and the width \(D\) of the latents. We ablate this in Tab. 15 by varying the number of latents between 128, 256 (best), and 512. We adapt the latent dimension accordingly to match the FLOPs budget.

\(1{\times}1\) means that only a single pixel (or pair of pixels) is used for each input element. 'Separate frames' means that the frames are not concatenated, but rather, input array elements are extracted independently from the two frames (thereby doubling the number of input elements). In the case of separate frames, \(1\times 1\) means essentially no preprocessing: each pixel becomes its own element with no spatio-temporal context whatsoever. We also performed experiments with a less expensive input model which uses a \(7\times 7\) convolution to 64 channels, followed by a max pool, similar to the one used in our ImageNet experiments. After feeding this through the Perceiver IO architecture (including querying with the same convolutional features used as input), we have an output a feature grid with stride 4 and 64 channels, on top of which we apply a RAFT upsampling layer.

We then upsample this flow for a given pixel in the high-resolution flow map by applying attention over a neighboring 3x3 block of the low-resolution flow map, following the uppsampling approach in RAFT (Teed & Deng, 2020).

Convolutional downsampling and RAFT upsampling provide even more spatial context for both the input features and the queries, but this doesn't make up for the loss of resolution and overall performs slightly worse than using the full resolution.

We use a latent array with 2048 elements and 512 channels and 24 self-attention modules, each with 16 self-attention heads, unless otherwise noted.

The decoder outputs \(16\times 224\times 224+16\times 1920/16+1\) vectors with 512 channels, that is, one element for each pixel in the video, one element for each audio patch, and one element for the classification label. These are then linearly projected to the appropriate channel size for each modality: 3 for videos, 16 for audio and 700 for classification (the logits for each of the 700 classes in Kinetics700).

Perceiver IO is able to capture the structure of both video and audio inputs, despite encoding both jointly with a single network.

this may be partially attributable to the preprocessing, which included coarse patching (Tab. 5) for both modalities due to the very high computational cost of processing raw video and audio inputs (which amount to over 2 million raw points).

Input: Figure 1: The Perceiver IO architecture can be used on domains with a wide variety of input and output spaces, including multi-task language understanding, dense visual tasks like optical flow, hybrid dense/sparse multimodal tasks such as video+audio+class autoencoding, and tasks with symbolic outputs like StarCraft II. See Tables 5 and 6 for details of all domains considered here.

Input: Figure 5: Schematic depiction of encode, process, and decode attention. Each attention module uses the same operations, but differs in which inputs are used to generate key/values or queries and in the output shape. Encode attention can be viewed as mapping an input to a latent space, typically with a smaller index dimension (fewer elements). Decode attention can be viewed as mapping a latent to an output space, often with a larger index dimension (more elements). Both of these are forms of cross-attention. Process attention (self-attention) preserves the input index dimension (same elements). Red and blue dashed lines are used to highlight the two matrix multiplications used in QKV attention, as described in the text.

Input: Figure 2: The Perceiver IO architecture. Perceiver IO maps arbitrary input arrays to arbitrary output arrays in a domain agnostic process. The bulk of the computation happens in a latent space whose size is typically smaller than the inputs and outputs, which makes the process computationally tractable even for very large inputs & outputs. See Fig. 5 for a more detailed look at encode, process, and decode attention.

Input: Table 5: Details of each of the tasks we use to evaluate Perceiver IO here. The positional and task embeddings appended to inputs for each case are listed in Tab. 6.

Input: Table 6: Table best viewed on a screen. The structure and size of the positional and task embeddings used to construct Perceiver IO’s encoder key-value inputs and decoder query inputs, for each domain described in the main text. “[x, y]” indicates that x’s and y’s features are concatenated, while “x + y” indicates that x’s and y’s features are added to produce the full featurization. “FF” = Fourier features as in Jaegle et al. (2021).

Input: Figure 6: Single-query attention decoder (left), as used in Perceiver IO for classification tasks and a standard average + project decoder (right), as used in Jaegle et al. (2021). Both modules can be seen as first aggregating latents by weighted averaging (learned, data-dependent weighting for the attention decoder; uniform weights for the average + project decoder) and then projecting to an output channel dimension (linear value projection + MLP for the attention decoder; simple linear projection by the average + project decoder). Attentional decoding is more expressive than average + project decoding and follows the same architectural template as encoder and processor modules.

Input: Figure 3: We construct queries with output-specific features to produce outputs with different semantics. For settings where each output point differs only in its position, like language, a position embedding can be used. Input features for the target output can also be used to query, either alone (as for StarCraft II) or alongside position features (as for flow). For multi-{task, modal} settings we use one embedding for each {task, modality} instead of each position. A single learned embedding suffices for simple classification tasks, like ImageNet. For tasks with heterogeneous outputs like multimodal autoencoding, features that are specific to some queries (like xy position) can be combined with modality embeddings, which also pad embeddings to fixed length.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the claim about decoupling computation size from input/output spaces while making minimal assumptions about spatial/locality structure is well-supported by the evidence. The architecture demonstrates this through:

1. Linear scaling with input/output size while computation happens in a smaller latent space
2. Successful application across diverse domains without domain-specific architectural changes
3. Explicit mathematical complexity analysis showing O([M+O+LN]NF) scaling
4. Demonstrated flexibility in handling different spatial structures (1D audio, 2D images, 3D video, etc.)
5. Works even without built-in spatial assumptions, as shown by ImageNet results with learned positional encodings

The claim appears to be precise and accurately reflects the capabilities supported by the evidence without overstatement.

<score>1</score>