Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Perceiver IO allows us to simplify pipelines and remove domain-specific assumptions: we process language without tokenizers without a performance or speed hit, fine-tune on multiple classification tasks simultaneously and without the need for [CLS] tokens (Sec. 4.1), estimate optical flow without relying on explicit architectural features for multiscale correspondence (Sec. 4.2), learn joint representations of video, audio, and labels without separate network trunks (Sec. 4.3), and perform image classification with no information about the 2D structure of images (Sec. A).

The review comment to be evaluated is:

The main strength of Perceiver IO is its high flexibility to various input domains. While keeping the main architecture, we can focus on engineering the input query preprocessing with domain-specific knowledge. Perceiver IO not only shows flexibility but also shows high performance. The results are comparable to or sometimes outperform previous domain-specific methods. For example

- Table 1 shows that Perceiver IO works well even without tokenization, while BERT + UTF-8 bytes show a significant performance drop (81.0 -> 71.5). It may show that Perceiver IO can perform well even without minimal domain knowledge (tokenization).
- Table 2 shows that the input query engineering can improve the target task (e.g., single-task query to 8 multi-task queries).
- Table 5 shows that the small input preprocessing engineering can improve the performance a lot. For example, Perceiver IO shows 82.1 top-1 accuracies for conv preprocessing while Perceiver IO with Fourier features shows 79.0.

I have minor concerns about ImageNet experiments.

- Although this paper is not aiming to state-of-the-art ImageNet performances, I think Perceiver IO is not yet comparable to state-of-the-art models on ImageNet image classification (as described in the last paragraph of Section 1). Perceiver IO still needs tremendous FLOPs compared to vision-specific models (Table 17). For example, ResNet shows 78.6 top-1 accuracy with 4.1 GFLOPs, while Perceiver IO Fourier shows 79.0 top-1 accuracy with 407 GFLOPs (99 times larger FLOPs). Stronger Perceiver IO (conv) shows 82.1 top-1 accuracy with 369 GFLOPs, where CaiT-M48 448 shows 86.5 top-1 accuracy and 329.6 GFLOPs.
- "Perceiver IO (pretrained)" and "Perceiver IO" in 2D Fourier features are different models. For example, "Perceiver IO (pretrained)" does not use weight sharing, and uses larger channel sizes. As a result, Perceiver IO (pretrained) has 4.38 times larger parameters compared to Perceiver IO (but half GFLOPs). The current text can make a reader mislead that "Perceiver IO (pretrained)" is the same model as "Perceiver IO", but the only difference is the JFT pretraining, which is not true.

> Unlike in the other ImageNet experiments, we do not share weights in the latent self-attention process modules, but use a 16-layer latent network with no weight sharing in depth. Unlike the other ImageNet experiments, the process-module MLPs use a hidden layer with 4× the number of channels (rather than 1× as on other ImageNet experiments)

To avoid confusion, I suggest replacing Table 5 with Table 17. Also, I would like to suggest to clarify that "Perceiver IO (pretrained)" and "Perceiver IO" are not the same model.

## Questions

- Perceiver IO shows better performances consistently compared to Perceiver IO baseline on GLUE (Table 13), Audio Set (Table 7), and ImageNet (Table 17). To my knowledge, there is no specific reason to the proposed output array cross-attention works better than the conventional classification head. I wonder what is the opinion by the authors what is the source of the improvements.
- In Table 5, I found that using 2D + maxpool preprocessing harms Perceiver performances (78.6 -> 77.4), but it helps Perceiver IO a lot (79.0 -> 82.1). As far as the reviewer understood, the main difference between Perceiver and Perceiver IO is the last decoding layer, not the input processing. I wonder what is the opinion of the authors why Perceiver and Perceiver IO show contradictory results on ImageNet Fourier features and conv preprocessing.
- I wonder why there is no Perceiver IO (conv) + pretraining result. Do the authors have any plan to add Perceiver IO (conv + pretrained)? (this is not a mandatory experiment for the rebuttal)

## Minor comments

- Many details are missing in the main text. To understand the method, the readers should read through the heavy appendix and the previous Perceiver paper. I fully understand that this is because of the page limitation, but this paper is somewhat hard to understand at first glance without any prior knowledge.
- In my opinion, it would be very helpful to readers if Table 8 and 9 are in the main text.
- pg 3. Typo -- unnecessary space *( "inducing points")*
- pg 29. Missing hyperlink (Tab. ??)
- In my opinion, it will be very interesting if there is one unified Perceiver model pretrained on various domain data. For example, one can train a Perceiver model trained by language and vision modalities, where each modality has different input and output arrays.

The evidence to be evaluated is:Similar to our work, several groups have proposed to solve tasks in multiple domains (e.g. Kaiser et al. 2017; Alyraca et al. 2020; Akbari et al. 2021), but typically across a fixed and predefined set of modalities by means of domain-specific networks. Although single-task specialist networks remain dominant in vision, multi-task learning has become popular (Misra et al., 2016; Doersch and Zisserman, 2017; Kokkinos, 2017; Zamir et al., 2018) and individual models achieve generality in a restricted domain: e.g. Mask-RCNN (He et al., 2017) handles object detection, segmentation, and pose estimation. In language, training or evaluation on multiple tasks has also become common (Collobert and Weston, 2008; Luong et al., 2016; Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020). Several groups have demonstrated that Transformers (originally designed for language) can be used or adapted to non-language tasks (e.g. Chen et al. 2020; Lu et al. 2021), but the limited scalability of Transformers limits their usefulness as general-purpose architectures.

The focus of our work is developing an architecture that is efficient and also performs well in many settings with a wide range of inputs and outputs.

Perceiver IO builds on this body of work to produce a general purpose architecture that can be easily and widely applied. The Perceiver IO architecture builds on the Perceiver (Jaegle et al., 2021), which achieved its cross-domain generality by assuming that its input is a simple 2D byte array: a set of elements (which might be pixels or patches in vision, characters or words in language, or some form of embedding, learned or otherwise), each described by a feature vector.

Rather than output a single category, Perceiver IO aims to have the same level of generality with respect to its _outputs_ as the Perceiver has with respect to its _inputs_: that is, it should produce arbitrary output arrays.

We first **encode** by applying an attention module that maps input arrays \(x\in\mathbb{R}^{M\times C}\) to arrays in a latent space \(z\in\mathbb{R}^{N\times D}\). We next **process** the latents \(z\) by applying a series of modules that take in and return arrays in this latent space. Finally, we **decode** by applying an attention module that maps latent arrays to output arrays \(y\in\mathbb{R}^{O\times E}\).

Following the design of the Perceiver, we implement each of the architecture's components using Transformer-style attention modules. Each of these modules applies a global query-key-value (QKV) attention operation followed by a multi-layer perceptron (MLP).

Transformers scale very poorly in both compute and memory (Tay et al., 2020). Because Transformers deploy attention modules homogeneously throughout its architecture, using its full input to generate queries and keys at every layer. This means each layer scales quadratically in compute and memory, which makes it impossible to apply Transformers on high-dimensional data like images without some form of preprocessing. Even on domains like language where Transformers shine, preprocessing (e.g. tokenization) is often needed to scale beyond short input sequences. Perceiver IO uses attention non-homogeneously by mapping inputs to a latent space, processing in that latent space, and decoding to an output space. Perceiver IO has no quadratic dependence on the input or output size: encoder and decoder attention modules depend linearly on the input and output size (respectively), while latent attention is independent of both input and output sizes (Sec. E.2).

While Transformers are typically used in settings with data preprocessed to contain at most a few thousand dimensions (Brown et al., 2020; Raffel et al., 2020), we show good results on domains with hundreds of thousands of dimensions. This architecture can be applied to inputs of any shape or spatial layout including inputs or outputs with different spatial structure (e.g. sound and video). In contrast to latent spaces typically used in vision (e.g. Ronneberger et al., 2015) the latent does not explicitly share the structure (spatial or otherwise) of the inputs.

For tasks with simple outputs, such as classification, these queries can be reused for every example and can be learned from scratch. For outputs with a spatial or sequence structure, we include a position encoding (e.g. a learned positional encoding or a Fourier feature) representing the position to be decoded in the output. For outputs with a multi-task or multimodal structure, we learn a single query for each task or for each modality: this information allows the network to distinguish one task or modality query from the others, much as positional encodings allow attention to distinguish one position from another.

We find that even very simple query features can produce good results, suggesting that the latent attention process is able to learn to organize the relevant information in a way that's easy to query. Each output point depends only on its query and the latent array, allowing us to decode outputs in parallel.

To probe the generality of Perceiver IO, we evaluate it on several domains including language understanding (Wikipedia+C4 masked language modeling), visual understanding (Sintel/KITTI optical flow and ImageNet classification), multi-modal (Kinetics autoencoding and AudioSet classification) & multi-task settings (multi-task GLUE), and symbolic representations for games (StarCraft II).

Although Transformers were originally developed for language, their quadratic complexity makes them difficult to use on language inputs without tokenization, which typically shortens the length of input sequences by a factor of \(\sim\)4. But unlike Transformer-based models such as BERT (Devlin et al., 2019) or XLNet (Yang et al., 2019), Perceiver IO scales linearly with input length. Our experiments focus on showing that Perceiver IO performs as well as or better than Transformers for masked language modeling (MLM) while removing tokenization (which is hard to maintain, introduces engineering overhead, and adds needless complexity to language models (Bostrom and Durrett, 2020; Clark et al., 2022)).

We find that at a given FLOPs budget, Perceiver IO trained without tokenization matches the performance of a strong Transformer-based model trained with SentencePiece tokenization (Sennrich et al., 2016; Kudo and Richardson, 2018).

**Perceiver IO on SentencePiece tokens.** We first observe that Perceiver IO applied on SentencePiece tokenized input sequences slightly outperforms a strong BERT baseline applied on the same inputs (81.2 vs 81.1). As a result of the reduced latent size of \(256\) we can train a much deeper network with 26 processing layers compared to BERT Base (12 layers) while maintaining a similar FLOPs budget.

Rather than using a fixed, handcrafted vocabulary, our model works directly with the raw byte inputs: we simply feed in and predict the UTF-8 bytes of the input string. Perceiver IO significantly outperforms a byte-level BERT baseline at the same FLOPs

Despite its greater depth, Perceiver IO is also faster than the Transformer-based BERT baselines in real wall-clock terms – by over a factor of 2 for the byte-based models – as shown in Tab. 1.

Our work scales to byte-level inputs without making any assumptions about the structure of the input, which allows it to be used beyond language as shown in the following sections.

We use multitask queries as described in Sec. 3.2 to finetune on all 8 GLUE tasks simultaneously using the UTF-8 byte model (results in Tab. 2).

We observe that our multitask approach outperforms single-task approaches and matches the approach that uses 8 task-specific input tokens. Our approach is more generic as it decouples the output array from the input array by not relying on [CLS] tokens.

Optical flow is a decades-old open problem in computer vision (Lucas and Kanade, 1981; Horn and Schunck, 1981). Given two images of the same scene (e.g. two consecutive frames of a video), the task is to estimate the 2D displacement for each pixel in the first image.

Optical flow is challenging for neural networks for two reasons. First, optical flow relies on finding correspondence: a single frame provides no information about flow, and images with extremely different appearance can produce the same flow.

Algorithms for optical flow thus must learn to accomplish several steps in a way that transfers from synthetic to real data. First, the algorithm must find correspondence between points.

We explore using Perceiver IO for audio-video-label multimodal autoencoding on the Kinetics-700-2020 dataset (Smaira et al., 2020).

With traditional autoencoding models like convolutional encoder-decoders, it is not obvious how to combine these modalities, because each uses data of different dimensions - 3D (video), 1D (raw audio), and 0D (class labels) - and with wildly different numbers of elements. With Perceiver IO, we pad the inputs with modality-specific embeddings, serialize them into a single 2D input array and query outputs using queries containing position encodings (for video and audio) and modality embeddings.

**Please read the Appendix for results on ImageNet (Sec. A), StarCraft II (Sec. B), and AudioSet (Sec. C).**

(1) on ImageNet, Perceiver IO surpasses 80% top-1 accuracy (84.5% top-1) without using 2D convolutions after pretraining on JFT.

In this work we introduce Perceiver IO, an architecture capable of handling general purpose inputs and outputs while scaling linearly in both input and output sizes. As we show, this architecture achieves good results in a wide variety of settings, making it a promising candidate for a _general purpose_ neural network architecture. Perceiver IO leverages the expressive power of latent attention and uses learned queries to expose a simple and unified interface that can handle multimodal and multitask settings. Overall, Perceiver IO offers a promising way to simplify the construction of sophisticated neural pipelines and facilitate progress on multimodal and multitask problems.

On all domains but StarCraft II, we include experiments with several input configurations, ranging from no domain adaptation (e.g. tokenizer-free language, flow from raw pixels, ImageNet with no convolutional or patch-based preprocessing and fully learned position encodings) to moderate domain adaptation (e.g. SentencePiece language understanding, flow from conv+maxpool-processed images and with RAFT upsampling, ImageNet with conv+maxpool-preprocessing and 2D Fourier features). These results demonstrate the unprecedented generality of Perceiver IO, the simplicity that this architecture unlocks in handling a range of tasks, and its flexibility to work as part of a domain-adapted system. Perceiver did well on ImageNet (Deng et al., 2009) classification without using 2D structure in the design of the architecture, but generated class scores using a simple average + project decoder (see Sec. E.3 and Fig. 6 for a diagram illustrating the difference between the two forms of decoder).

By replacing these 2D position features with a fully learned position encoding as used on language, we can learn an image classification model that is given no privileged information about the structure of images.

ImageNet networks that use this positional encoding are given no information about 2D image structure.

To our knowledge, this is the best result by any model on ImageNet without 2D architectural or feature information.

Because ImageNet data points are relatively small (Tab. 5), we are able to process full images without convolutional pre- and post-processing. Consequently, we can use this dataset to probe the sensitivity of the model to convolutional pre-processing. Incorporating a single convolution + max pooling leads to a moderate improvement in the performance of the architecture: this is perhaps unsurprising, as convolutional pre-processing injects information about the 2D structure of images into the architecture. By comparison ViT first processes images by applying a 2D convolution with matched kernel and stride to downsample its inputs (referred to as a "linear projection of flattened patches" in that work and throughout the ViT literature). As in other experiments, we find that incorporating an attention-based decoder (Perceiver IO) leads to better results than averaging and pooling the output (Perceiver).

For this reason, we view AlphaStar as an important test case for Perceiver IO's ability to function as a general-purpose tool for processing symbolic or set-valued data. If the question is "can Perceiver IO serve as a replacement for a well-tuned Transformer as a symbolic processing engine?" then the answer is yes: We obtained StarCraft II results by using Perceiver IO instead of a Transformer for the AlphaStar entity encoder. We replaced the Transformer with a Perceiver IO with a latent of index dimension 32, keeping the input and output size of 512 units.

Perceiver IO works out of the box. We observed that the resulting agent reached the same level of performance as the original AlphaStar agent, reaching an 87% win-rate versus the Elite bot after behavioral cloning (Pomerleau, 1989) on human data, while also leading to a 3\(\times\) decrease in FLOPs (Tab. 9).

We show that the trend identified on ImageNet holds more generally, by revisiting the multimodal AudioSet classification domain. AudioSet is a large-scale event classification dataset containing 1.7 million training examples, each consisting of 10s long video and audio. Each example is labeled with several labels drawn from 527 classes. We perform experiments using the protocol described in Jaegle et al. (2021), training models for 100 epochs using 32-frame clips at train time and 16 overlapping 32-frame clips at test time.

Perceiver IO is constructed from GPT-2-style (Radford et al., 2019) Transformer attention modules, which consist of QKV attention followed by an MLP, along with linear projection layers to ensure inputs to and outputs from the QKV attention and MLP take on desired sizes.

The view of attention that we describe encompasses both cross-attention and self-attention, both of which are specific ways of using QKV-attention. Perceiver IO uses cross-attention for encoder and decoder attention modules and uses self-attention for the latent processing modules.

In other words, Perceiver IO has complexity linear in the size of the input and output arrays and it decouples the depth of the latent transformer from the input and output sizes. Both of these properties contribute to Perceiver IO's efficiency: while many proposals for efficient attention modules or architectures include linear or sub-quadratic scaling with input/output size, Perceiver IO is unusual in also decoupling depth from input/output size (without requiring domain-specific strategies like 2D convolution).

One application of Perceiver IO is byte-level language processing, which has concurrently been addressed by several other groups. Clark et al. (2022) trains models on Unicode code points and shows results competitive with subword-based models on a multilingual question answering dataset. Tay et al. (2022) trains on UTF-8 bytes directly by introducing a hand-designed module that is trained end-to-end to perform subword tokenization and produces results on-par with and sometimes better than subword-based models. Xue et al. (2022) trains encoder-decoder T5 models on UTF-8 bytes directly and shows that making the encoder 3x deeper than the decoder leads to comparable performance with subword baselines.

For single-task experiments, we do not require a [CLS] token as we use a single decoding query vector.

Perceiver IO is somewhat slower on traditional GPUs than our baseline RAFT model, but we find that the trend reverses on TPUs, which is the target architecture for our work.

Our most expensive model achieves approximately 0.8 frames/sec on a 2017 TITAN Xp, and our lightweight model (with conv downsampling and RAFT-style upsampling) achieves 3.3 frames/sec, which is not far from the 10 frames per second reported for RAFT (Teed & Deng, 2020). On the publicly-available TPU v3, however, our most expensive model achieves 4.4 frames/sec on a single TPU core, and 17.8 frames/sec for the lightweight model.

We suspect that the difference is due to the gather operations required for RAFT but not for Perceivers, which are slow on TPU due to their poor memory locality properties.

We see that the algorithm is capable of dealing with heavy occlusion, and can propagate optical flow across large regions with very little texture. The network can also deal with very large motions and very small objects. Finally, to verify that Perceiver IO performs well on real-world data despite being trained only on synthetic imagery, we applied it to a small number (roughly 10) real videos taken from Getty images (www.gettyimages.com). Perceiver IO typically performs very well out-of-domain, although some failure cases remain:

Perceiver IO can pick up on remarkably small objects such as the water droplets thrown by the girl's shoe in pigeon.mp4 or the confetti in thai_dance.mp4.

For the multimodal autoencoding experiments, we patch preprocessing for both images and audio, and we embed the labels as one-hot labels. The patch size is \(1\times 4\times 4\) for video and 16 for audio. The audio is sampled at 48kHz, or 1920 samples per frame. The decoder outputs \(16\times 224\times 224+16\times 1920/16+1\) vectors with 512 channels, that is, one element for each pixel in the video, one element for each audio patch, and one element for the classification label. These are then linearly projected to the appropriate channel size for each modality: 3 for videos, 16 for audio and 700 for classification (the logits for each of the 700 classes in Kinetics700).

We note that we read and generate the audio waveform directly in the time domain; we do not transform first to a spectrogram. We use a 387 dimensional 3D Fourier position embedding for each input video patch and a 385 dimensional 1D Fourier position embedding for each audio patch (385 to ensure the input dimensions to Perceiver IO match for all elements). In addition, we pad all input elements with a learned vector representing the modality; inputs from the same modality share the same token. In particular, we add a 317 dimensional modality embedding to video elements, a 319 dimensional modality embedding to audio elements, and a 4 dimensional modality embedding to the label, so that all elements have 704 features. The decoder queries are also constructed from Fourier position embeddings for video and audio and a learned positional embedding for label: 387 features for video, 385 features for audio, and 1024 learned features for the label. We pad the queries for each modality with a different learned vector for each modality, so that the final feature size for the queries is 1026.

To help verify the quality of Perceiver IO's outputs on real-world data, we applied it a small number of real videos (\(\sim\)10) with audio taken from Getty Images. Perceiver IO is able to capture the structure of both video and audio inputs, despite encoding both jointly with a single network.

Input: Figure 1: The Perceiver IO architecture can be used on domains with a wide variety of input and output spaces, including multi-task language understanding, dense visual tasks like optical flow, hybrid dense/sparse multimodal tasks such as video+audio+class autoencoding, and tasks with symbolic outputs like StarCraft II. See Tables 5 and 6 for details of all domains considered here.

Input: Figure 8: Qualitative examples of optical flow. For each image pair, we show the two frames (top), and then the estimated flow (bottom left) and the ground-truth flow (bottom right). In the left example, we see one person under heavy occlusion where the correct flow is propagated into a region with few details. Another person in the foreground has clothes with little texture and substantial blur, and yet the algorithm can propagate the flow across the entire region. In the center example, we see very large motions from both the dragon and the person, yet many fine structures are preserved like the pole. On the right, we see a forest scene with a few extremely small objects with very subtle motions (circled) which our algorithm is able to detect and segment correctly.

Input: Figure 2: The Perceiver IO architecture. Perceiver IO maps arbitrary input arrays to arbitrary output arrays in a domain agnostic process. The bulk of the computation happens in a latent space whose size is typically smaller than the inputs and outputs, which makes the process computationally tractable even for very large inputs & outputs. See Fig. 5 for a more detailed look at encode, process, and decode attention.

Input: Table 5: Details of each of the tasks we use to evaluate Perceiver IO here. The positional and task embeddings appended to inputs for each case are listed in Tab. 6.

Input: Table 6: Table best viewed on a screen. The structure and size of the positional and task embeddings used to construct Perceiver IO’s encoder key-value inputs and decoder query inputs, for each domain described in the main text. “[x, y]” indicates that x’s and y’s features are concatenated, while “x + y” indicates that x’s and y’s features are added to produce the full featurization. “FF” = Fourier features as in Jaegle et al. (2021).

Input: Figure 4: Multimodal audio-video-label autoencoding with 88x compression. Side-by-side: inputs on left, reconstructions right. See the supplemental material for example output video and audio.

Input: Figure 3: We construct queries with output-specific features to produce outputs with different semantics. For settings where each output point differs only in its position, like language, a position embedding can be used. Input features for the target output can also be used to query, either alone (as for StarCraft II) or alongside position features (as for flow). For multi-{task, modal} settings we use one embedding for each {task, modality} instead of each position. A single learned embedding suffices for simple classification tasks, like ImageNet. For tasks with heterogeneous outputs like multimodal autoencoding, features that are specific to some queries (like xy position) can be combined with modality embeddings, which also pad embeddings to fixed length.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After careful review, I find the claim to be somewhat overstated when evaluated against the evidence. While Perceiver IO does demonstrate impressive flexibility across domains and removes some domain-specific assumptions, there are important caveats:

1. The removal of tokenizers comes with significant computational costs (99x more FLOPs than ResNet for comparable ImageNet performance)
2. The reviewer points out that "Perceiver IO (pretrained)" and "Perceiver IO" are different models, which isn't clearly disclosed
3. The claim about "no performance or speed hit" when removing tokenizers is not fully supported, as there are tradeoffs in computational efficiency
4. Some domain-specific preprocessing is still beneficial for performance (e.g., convolutional preprocessing for images)

The architecture shows promise in simplifying pipelines, but the claim overstates the degree of domain-agnosticism achieved.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Perceiver IO allows us to simplify pipelines and remove domain-specific assumptions: we process language without tokenizers without a performance or speed hit, fine-tune on multiple classification tasks simultaneously and without the need for [CLS] tokens (Sec. 4.1), estimate optical flow without relying on explicit architectural features for multiscale correspondence (Sec. 4.2), learn joint representations of video, audio, and labels without separate network trunks (Sec. 4.3), and perform image classification with no information about the 2D structure of images (Sec. A).

The review comment to be evaluated is:

Pros:

1. This paper introduced a generic architecture for coping with different tasks with various input and output lengths. It is different from Transformer architecture and Perceiver architecture in that it uses a latent process to compress the larger number of input tokens into a hidden space, and then decode the output from this hidden space. This way it can decouple the inputs and outputs which has the potential to significantly reduce the cost brought by a large number of input and output tokens.

2. The authors presented a thorough study of Perceiver IO across different application scenarios, such as language, vision, multi-modality. Extensive experiments demonstrate that the proposed method can achieve comparable or even better performance than the baselines. To facilitate the adaptation, the authors suggested a number of good ways to convert the inputs/outputs into certain formats.

Cons:

1. The authors should list the sizes of inputs, latent, and output arrays for each of the downstream tasks to give the audience a better sense of the complexity. For some of the tasks, such as ImageNet classification, the authors should also report the FLOPs in comparisons with prior arts, e.g., ViTs.

2. The main merit of Perceiver IO compared with Perceiver is that it further introduces a cross-attention-based output decoder on top of the latent arrays. Though Perceiver was originally proposed for classification tasks, it can be also used to cope with structured outputs. For some of the tasks except for ImageNet classification listed in this paper, I am curious whether Perceiver can be applied and how it performs compared with Perceiver IO.

3. Perceiver IO is proposed as a generic architecture for various tasks by modeling them as a read-process-write process. One question is that whether we should reformulate all these tasks as the way shown in this paper. This paper does show some encouraging results on various tasks compared with the baselines. However, they are only compared with baseline methods and still underperforms established methods in specific domains. As we know, different tasks can still benefit a lot from the specific domain knowledge (e.g., 2D spatial for image recognition). Actually, we can also think of the Transformer encoder as a generic architecture in numerous domains. Then, do we need to build up a higher-level generic architecture upon the Transformer encoder?

4. A good part of unifying architectures for various tasks is that we can train the same set of parameters using different tasks. In this paper, the authors demonstrated the effectiveness of Perceiver IO across different settings, I am wondering whether such a generic architecture can be used for multi-task training so that it can leverage the training data from different tasks.

The evidence to be evaluated is:Similar to our work, several groups have proposed to solve tasks in multiple domains (e.g. Kaiser et al. 2017; Alyraca et al. 2020; Akbari et al. 2021), but typically across a fixed and predefined set of modalities by means of domain-specific networks. Although single-task specialist networks remain dominant in vision, multi-task learning has become popular (Misra et al., 2016; Doersch and Zisserman, 2017; Kokkinos, 2017; Zamir et al., 2018) and individual models achieve generality in a restricted domain: e.g. Mask-RCNN (He et al., 2017) handles object detection, segmentation, and pose estimation. In language, training or evaluation on multiple tasks has also become common (Collobert and Weston, 2008; Luong et al., 2016; Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020). Several groups have demonstrated that Transformers (originally designed for language) can be used or adapted to non-language tasks (e.g. Chen et al. 2020; Lu et al. 2021), but the limited scalability of Transformers limits their usefulness as general-purpose architectures.

The focus of our work is developing an architecture that is efficient and also performs well in many settings with a wide range of inputs and outputs.

Perceiver IO builds on this body of work to produce a general purpose architecture that can be easily and widely applied. The Perceiver IO architecture builds on the Perceiver (Jaegle et al., 2021), which achieved its cross-domain generality by assuming that its input is a simple 2D byte array: a set of elements (which might be pixels or patches in vision, characters or words in language, or some form of embedding, learned or otherwise), each described by a feature vector.

Rather than output a single category, Perceiver IO aims to have the same level of generality with respect to its _outputs_ as the Perceiver has with respect to its _inputs_: that is, it should produce arbitrary output arrays.

We first **encode** by applying an attention module that maps input arrays \(x\in\mathbb{R}^{M\times C}\) to arrays in a latent space \(z\in\mathbb{R}^{N\times D}\). We next **process** the latents \(z\) by applying a series of modules that take in and return arrays in this latent space. Finally, we **decode** by applying an attention module that maps latent arrays to output arrays \(y\in\mathbb{R}^{O\times E}\).

Following the design of the Perceiver, we implement each of the architecture's components using Transformer-style attention modules. Each of these modules applies a global query-key-value (QKV) attention operation followed by a multi-layer perceptron (MLP).

Transformers scale very poorly in both compute and memory (Tay et al., 2020). Because Transformers deploy attention modules homogeneously throughout its architecture, using its full input to generate queries and keys at every layer. This means each layer scales quadratically in compute and memory, which makes it impossible to apply Transformers on high-dimensional data like images without some form of preprocessing. Even on domains like language where Transformers shine, preprocessing (e.g. tokenization) is often needed to scale beyond short input sequences. Perceiver IO uses attention non-homogeneously by mapping inputs to a latent space, processing in that latent space, and decoding to an output space. Perceiver IO has no quadratic dependence on the input or output size: encoder and decoder attention modules depend linearly on the input and output size (respectively), while latent attention is independent of both input and output sizes (Sec. E.2).

While Transformers are typically used in settings with data preprocessed to contain at most a few thousand dimensions (Brown et al., 2020; Raffel et al., 2020), we show good results on domains with hundreds of thousands of dimensions. This architecture can be applied to inputs of any shape or spatial layout including inputs or outputs with different spatial structure (e.g. sound and video). In contrast to latent spaces typically used in vision (e.g. Ronneberger et al., 2015) the latent does not explicitly share the structure (spatial or otherwise) of the inputs.

For tasks with simple outputs, such as classification, these queries can be reused for every example and can be learned from scratch. For outputs with a spatial or sequence structure, we include a position encoding (e.g. a learned positional encoding or a Fourier feature) representing the position to be decoded in the output. For outputs with a multi-task or multimodal structure, we learn a single query for each task or for each modality: this information allows the network to distinguish one task or modality query from the others, much as positional encodings allow attention to distinguish one position from another.

We find that even very simple query features can produce good results, suggesting that the latent attention process is able to learn to organize the relevant information in a way that's easy to query. Each output point depends only on its query and the latent array, allowing us to decode outputs in parallel.

To probe the generality of Perceiver IO, we evaluate it on several domains including language understanding (Wikipedia+C4 masked language modeling), visual understanding (Sintel/KITTI optical flow and ImageNet classification), multi-modal (Kinetics autoencoding and AudioSet classification) & multi-task settings (multi-task GLUE), and symbolic representations for games (StarCraft II).

Although Transformers were originally developed for language, their quadratic complexity makes them difficult to use on language inputs without tokenization, which typically shortens the length of input sequences by a factor of \(\sim\)4. But unlike Transformer-based models such as BERT (Devlin et al., 2019) or XLNet (Yang et al., 2019), Perceiver IO scales linearly with input length. Our experiments focus on showing that Perceiver IO performs as well as or better than Transformers for masked language modeling (MLM) while removing tokenization (which is hard to maintain, introduces engineering overhead, and adds needless complexity to language models (Bostrom and Durrett, 2020; Clark et al., 2022)).

We find that at a given FLOPs budget, Perceiver IO trained without tokenization matches the performance of a strong Transformer-based model trained with SentencePiece tokenization (Sennrich et al., 2016; Kudo and Richardson, 2018).

**Perceiver IO on SentencePiece tokens.** We first observe that Perceiver IO applied on SentencePiece tokenized input sequences slightly outperforms a strong BERT baseline applied on the same inputs (81.2 vs 81.1). As a result of the reduced latent size of \(256\) we can train a much deeper network with 26 processing layers compared to BERT Base (12 layers) while maintaining a similar FLOPs budget.

Rather than using a fixed, handcrafted vocabulary, our model works directly with the raw byte inputs: we simply feed in and predict the UTF-8 bytes of the input string. Perceiver IO significantly outperforms a byte-level BERT baseline at the same FLOPs

Despite its greater depth, Perceiver IO is also faster than the Transformer-based BERT baselines in real wall-clock terms – by over a factor of 2 for the byte-based models – as shown in Tab. 1.

Our work scales to byte-level inputs without making any assumptions about the structure of the input, which allows it to be used beyond language as shown in the following sections.

We use multitask queries as described in Sec. 3.2 to finetune on all 8 GLUE tasks simultaneously using the UTF-8 byte model (results in Tab. 2).

We observe that our multitask approach outperforms single-task approaches and matches the approach that uses 8 task-specific input tokens. Our approach is more generic as it decouples the output array from the input array by not relying on [CLS] tokens.

Optical flow is a decades-old open problem in computer vision (Lucas and Kanade, 1981; Horn and Schunck, 1981). Given two images of the same scene (e.g. two consecutive frames of a video), the task is to estimate the 2D displacement for each pixel in the first image.

Optical flow is challenging for neural networks for two reasons. First, optical flow relies on finding correspondence: a single frame provides no information about flow, and images with extremely different appearance can produce the same flow.

Algorithms for optical flow thus must learn to accomplish several steps in a way that transfers from synthetic to real data. First, the algorithm must find correspondence between points.

We explore using Perceiver IO for audio-video-label multimodal autoencoding on the Kinetics-700-2020 dataset (Smaira et al., 2020).

With traditional autoencoding models like convolutional encoder-decoders, it is not obvious how to combine these modalities, because each uses data of different dimensions - 3D (video), 1D (raw audio), and 0D (class labels) - and with wildly different numbers of elements. With Perceiver IO, we pad the inputs with modality-specific embeddings, serialize them into a single 2D input array and query outputs using queries containing position encodings (for video and audio) and modality embeddings.

**Please read the Appendix for results on ImageNet (Sec. A), StarCraft II (Sec. B), and AudioSet (Sec. C).**

(1) on ImageNet, Perceiver IO surpasses 80% top-1 accuracy (84.5% top-1) without using 2D convolutions after pretraining on JFT.

In this work we introduce Perceiver IO, an architecture capable of handling general purpose inputs and outputs while scaling linearly in both input and output sizes. As we show, this architecture achieves good results in a wide variety of settings, making it a promising candidate for a _general purpose_ neural network architecture. Perceiver IO leverages the expressive power of latent attention and uses learned queries to expose a simple and unified interface that can handle multimodal and multitask settings. Overall, Perceiver IO offers a promising way to simplify the construction of sophisticated neural pipelines and facilitate progress on multimodal and multitask problems.

On all domains but StarCraft II, we include experiments with several input configurations, ranging from no domain adaptation (e.g. tokenizer-free language, flow from raw pixels, ImageNet with no convolutional or patch-based preprocessing and fully learned position encodings) to moderate domain adaptation (e.g. SentencePiece language understanding, flow from conv+maxpool-processed images and with RAFT upsampling, ImageNet with conv+maxpool-preprocessing and 2D Fourier features). These results demonstrate the unprecedented generality of Perceiver IO, the simplicity that this architecture unlocks in handling a range of tasks, and its flexibility to work as part of a domain-adapted system. Perceiver did well on ImageNet (Deng et al., 2009) classification without using 2D structure in the design of the architecture, but generated class scores using a simple average + project decoder (see Sec. E.3 and Fig. 6 for a diagram illustrating the difference between the two forms of decoder).

By replacing these 2D position features with a fully learned position encoding as used on language, we can learn an image classification model that is given no privileged information about the structure of images.

ImageNet networks that use this positional encoding are given no information about 2D image structure.

To our knowledge, this is the best result by any model on ImageNet without 2D architectural or feature information.

Because ImageNet data points are relatively small (Tab. 5), we are able to process full images without convolutional pre- and post-processing. Consequently, we can use this dataset to probe the sensitivity of the model to convolutional pre-processing. Incorporating a single convolution + max pooling leads to a moderate improvement in the performance of the architecture: this is perhaps unsurprising, as convolutional pre-processing injects information about the 2D structure of images into the architecture. By comparison ViT first processes images by applying a 2D convolution with matched kernel and stride to downsample its inputs (referred to as a "linear projection of flattened patches" in that work and throughout the ViT literature). As in other experiments, we find that incorporating an attention-based decoder (Perceiver IO) leads to better results than averaging and pooling the output (Perceiver).

For this reason, we view AlphaStar as an important test case for Perceiver IO's ability to function as a general-purpose tool for processing symbolic or set-valued data. If the question is "can Perceiver IO serve as a replacement for a well-tuned Transformer as a symbolic processing engine?" then the answer is yes: We obtained StarCraft II results by using Perceiver IO instead of a Transformer for the AlphaStar entity encoder. We replaced the Transformer with a Perceiver IO with a latent of index dimension 32, keeping the input and output size of 512 units.

Perceiver IO works out of the box. We observed that the resulting agent reached the same level of performance as the original AlphaStar agent, reaching an 87% win-rate versus the Elite bot after behavioral cloning (Pomerleau, 1989) on human data, while also leading to a 3\(\times\) decrease in FLOPs (Tab. 9).

We show that the trend identified on ImageNet holds more generally, by revisiting the multimodal AudioSet classification domain. AudioSet is a large-scale event classification dataset containing 1.7 million training examples, each consisting of 10s long video and audio. Each example is labeled with several labels drawn from 527 classes. We perform experiments using the protocol described in Jaegle et al. (2021), training models for 100 epochs using 32-frame clips at train time and 16 overlapping 32-frame clips at test time.

Perceiver IO is constructed from GPT-2-style (Radford et al., 2019) Transformer attention modules, which consist of QKV attention followed by an MLP, along with linear projection layers to ensure inputs to and outputs from the QKV attention and MLP take on desired sizes.

The view of attention that we describe encompasses both cross-attention and self-attention, both of which are specific ways of using QKV-attention. Perceiver IO uses cross-attention for encoder and decoder attention modules and uses self-attention for the latent processing modules.

In other words, Perceiver IO has complexity linear in the size of the input and output arrays and it decouples the depth of the latent transformer from the input and output sizes. Both of these properties contribute to Perceiver IO's efficiency: while many proposals for efficient attention modules or architectures include linear or sub-quadratic scaling with input/output size, Perceiver IO is unusual in also decoupling depth from input/output size (without requiring domain-specific strategies like 2D convolution).

One application of Perceiver IO is byte-level language processing, which has concurrently been addressed by several other groups. Clark et al. (2022) trains models on Unicode code points and shows results competitive with subword-based models on a multilingual question answering dataset. Tay et al. (2022) trains on UTF-8 bytes directly by introducing a hand-designed module that is trained end-to-end to perform subword tokenization and produces results on-par with and sometimes better than subword-based models. Xue et al. (2022) trains encoder-decoder T5 models on UTF-8 bytes directly and shows that making the encoder 3x deeper than the decoder leads to comparable performance with subword baselines.

For single-task experiments, we do not require a [CLS] token as we use a single decoding query vector.

Perceiver IO is somewhat slower on traditional GPUs than our baseline RAFT model, but we find that the trend reverses on TPUs, which is the target architecture for our work.

Our most expensive model achieves approximately 0.8 frames/sec on a 2017 TITAN Xp, and our lightweight model (with conv downsampling and RAFT-style upsampling) achieves 3.3 frames/sec, which is not far from the 10 frames per second reported for RAFT (Teed & Deng, 2020). On the publicly-available TPU v3, however, our most expensive model achieves 4.4 frames/sec on a single TPU core, and 17.8 frames/sec for the lightweight model.

We suspect that the difference is due to the gather operations required for RAFT but not for Perceivers, which are slow on TPU due to their poor memory locality properties.

We see that the algorithm is capable of dealing with heavy occlusion, and can propagate optical flow across large regions with very little texture. The network can also deal with very large motions and very small objects. Finally, to verify that Perceiver IO performs well on real-world data despite being trained only on synthetic imagery, we applied it to a small number (roughly 10) real videos taken from Getty images (www.gettyimages.com). Perceiver IO typically performs very well out-of-domain, although some failure cases remain:

Perceiver IO can pick up on remarkably small objects such as the water droplets thrown by the girl's shoe in pigeon.mp4 or the confetti in thai_dance.mp4.

For the multimodal autoencoding experiments, we patch preprocessing for both images and audio, and we embed the labels as one-hot labels. The patch size is \(1\times 4\times 4\) for video and 16 for audio. The audio is sampled at 48kHz, or 1920 samples per frame. The decoder outputs \(16\times 224\times 224+16\times 1920/16+1\) vectors with 512 channels, that is, one element for each pixel in the video, one element for each audio patch, and one element for the classification label. These are then linearly projected to the appropriate channel size for each modality: 3 for videos, 16 for audio and 700 for classification (the logits for each of the 700 classes in Kinetics700).

We note that we read and generate the audio waveform directly in the time domain; we do not transform first to a spectrogram. We use a 387 dimensional 3D Fourier position embedding for each input video patch and a 385 dimensional 1D Fourier position embedding for each audio patch (385 to ensure the input dimensions to Perceiver IO match for all elements). In addition, we pad all input elements with a learned vector representing the modality; inputs from the same modality share the same token. In particular, we add a 317 dimensional modality embedding to video elements, a 319 dimensional modality embedding to audio elements, and a 4 dimensional modality embedding to the label, so that all elements have 704 features. The decoder queries are also constructed from Fourier position embeddings for video and audio and a learned positional embedding for label: 387 features for video, 385 features for audio, and 1024 learned features for the label. We pad the queries for each modality with a different learned vector for each modality, so that the final feature size for the queries is 1026.

To help verify the quality of Perceiver IO's outputs on real-world data, we applied it a small number of real videos (\(\sim\)10) with audio taken from Getty Images. Perceiver IO is able to capture the structure of both video and audio inputs, despite encoding both jointly with a single network.

Input: Figure 1: The Perceiver IO architecture can be used on domains with a wide variety of input and output spaces, including multi-task language understanding, dense visual tasks like optical flow, hybrid dense/sparse multimodal tasks such as video+audio+class autoencoding, and tasks with symbolic outputs like StarCraft II. See Tables 5 and 6 for details of all domains considered here.

Input: Figure 8: Qualitative examples of optical flow. For each image pair, we show the two frames (top), and then the estimated flow (bottom left) and the ground-truth flow (bottom right). In the left example, we see one person under heavy occlusion where the correct flow is propagated into a region with few details. Another person in the foreground has clothes with little texture and substantial blur, and yet the algorithm can propagate the flow across the entire region. In the center example, we see very large motions from both the dragon and the person, yet many fine structures are preserved like the pole. On the right, we see a forest scene with a few extremely small objects with very subtle motions (circled) which our algorithm is able to detect and segment correctly.

Input: Figure 2: The Perceiver IO architecture. Perceiver IO maps arbitrary input arrays to arbitrary output arrays in a domain agnostic process. The bulk of the computation happens in a latent space whose size is typically smaller than the inputs and outputs, which makes the process computationally tractable even for very large inputs & outputs. See Fig. 5 for a more detailed look at encode, process, and decode attention.

Input: Table 5: Details of each of the tasks we use to evaluate Perceiver IO here. The positional and task embeddings appended to inputs for each case are listed in Tab. 6.

Input: Table 6: Table best viewed on a screen. The structure and size of the positional and task embeddings used to construct Perceiver IO’s encoder key-value inputs and decoder query inputs, for each domain described in the main text. “[x, y]” indicates that x’s and y’s features are concatenated, while “x + y” indicates that x’s and y’s features are added to produce the full featurization. “FF” = Fourier features as in Jaegle et al. (2021).

Input: Figure 4: Multimodal audio-video-label autoencoding with 88x compression. Side-by-side: inputs on left, reconstructions right. See the supplemental material for example output video and audio.

Input: Figure 3: We construct queries with output-specific features to produce outputs with different semantics. For settings where each output point differs only in its position, like language, a position embedding can be used. Input features for the target output can also be used to query, either alone (as for StarCraft II) or alongside position features (as for flow). For multi-{task, modal} settings we use one embedding for each {task, modality} instead of each position. A single learned embedding suffices for simple classification tasks, like ImageNet. For tasks with heterogeneous outputs like multimodal autoencoding, features that are specific to some queries (like xy position) can be combined with modality embeddings, which also pad embeddings to fixed length.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence, I'll evaluate the claim that "Perceiver IO allows simplifying pipelines and removing domain-specific assumptions."

The evidence shows mixed support for this claim. While Perceiver IO demonstrates flexibility across multiple domains and can handle raw inputs (e.g., bytes instead of tokens for language), the results often improve when domain-specific preprocessing is included (e.g., convolutional preprocessing for images). The paper acknowledges using "moderate domain adaptation" in most experiments. Performance sometimes lags behind specialized architectures, and reviewer comments note that domain-specific knowledge still provides benefits. The claim somewhat overstates the degree of pipeline simplification achieved.

<score>3</score>

Justification:
- Language: Successfully removes tokenizers but still benefits from them
- Classification: Works without [CLS] tokens but performance is comparable, not superior
- Optical flow: Removes some architectural assumptions but still uses domain-specific upsampling
- Multimodal: Demonstrates flexibility but requires careful input formatting and preprocessing
- Evidence shows trade-offs between generality and performance rather than unqualified simplification


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Perceiver IO allows us to simplify pipelines and remove domain-specific assumptions: we process language without tokenizers without a performance or speed hit, fine-tune on multiple classification tasks simultaneously and without the need for [CLS] tokens (Sec. 4.1), estimate optical flow without relying on explicit architectural features for multiscale correspondence (Sec. 4.2), learn joint representations of video, audio, and labels without separate network trunks (Sec. 4.3), and perform image classification with no information about the 2D structure of images (Sec. A).

The review comment to be evaluated is:

#### Strength
- The idea of perceiver IO is novel and solid -- a general architecture capable of handling general-purpose inputs and outputs across different tasks and modalities. This is very promising to simplify the construction of highly tuned task-specific neural pipelines and improve the multimodal and multi-task problems.

- Each component in perceiver IO is necessary and well defined for the proposed tasks. 2D byte array input enables the task agnostic input for different modalities. Perceiver network that scales linearly with the size of input and output, and non-auto regressive decoding make the decoding to raw output possible.

- The proposed architecture is tested on massive experiments including language understanding tasks, optical flow, video audio class autoencoding, image classification, and starcraft II and achieves superior performance. Each task is supported with a detailed ablation study to shed light on future research.

#### Weakness
- In Table 1, the WNLI task is excluded from the GLUE benchmark, I wonder what is the reason this task is removed?

- The non-autoregressive decoding enables fast decoding for large outputs. However, for tasks that require more information on prior decoding context (machine translation or image generation tasks), does the proposed model can still perform well on those tasks?

- Although perceiver IO removes the task-specific pre-processing (tokenization, patch embedding, etc.), the model still requires huge engineering efforts to adapt the model for different tasks. For example, the model hyper-parameters are quite different for different tasks.

- One huge benefit of perceiver IO is to train different tasks together and explore the transfer between different tasks/modalities, which is not explored in this paper.

The evidence to be evaluated is:Similar to our work, several groups have proposed to solve tasks in multiple domains (e.g. Kaiser et al. 2017; Alyraca et al. 2020; Akbari et al. 2021), but typically across a fixed and predefined set of modalities by means of domain-specific networks. Although single-task specialist networks remain dominant in vision, multi-task learning has become popular (Misra et al., 2016; Doersch and Zisserman, 2017; Kokkinos, 2017; Zamir et al., 2018) and individual models achieve generality in a restricted domain: e.g. Mask-RCNN (He et al., 2017) handles object detection, segmentation, and pose estimation. In language, training or evaluation on multiple tasks has also become common (Collobert and Weston, 2008; Luong et al., 2016; Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020). Several groups have demonstrated that Transformers (originally designed for language) can be used or adapted to non-language tasks (e.g. Chen et al. 2020; Lu et al. 2021), but the limited scalability of Transformers limits their usefulness as general-purpose architectures.

The focus of our work is developing an architecture that is efficient and also performs well in many settings with a wide range of inputs and outputs.

Perceiver IO builds on this body of work to produce a general purpose architecture that can be easily and widely applied. The Perceiver IO architecture builds on the Perceiver (Jaegle et al., 2021), which achieved its cross-domain generality by assuming that its input is a simple 2D byte array: a set of elements (which might be pixels or patches in vision, characters or words in language, or some form of embedding, learned or otherwise), each described by a feature vector.

Rather than output a single category, Perceiver IO aims to have the same level of generality with respect to its _outputs_ as the Perceiver has with respect to its _inputs_: that is, it should produce arbitrary output arrays.

We first **encode** by applying an attention module that maps input arrays \(x\in\mathbb{R}^{M\times C}\) to arrays in a latent space \(z\in\mathbb{R}^{N\times D}\). We next **process** the latents \(z\) by applying a series of modules that take in and return arrays in this latent space. Finally, we **decode** by applying an attention module that maps latent arrays to output arrays \(y\in\mathbb{R}^{O\times E}\).

Following the design of the Perceiver, we implement each of the architecture's components using Transformer-style attention modules. Each of these modules applies a global query-key-value (QKV) attention operation followed by a multi-layer perceptron (MLP).

Transformers scale very poorly in both compute and memory (Tay et al., 2020). Because Transformers deploy attention modules homogeneously throughout its architecture, using its full input to generate queries and keys at every layer. This means each layer scales quadratically in compute and memory, which makes it impossible to apply Transformers on high-dimensional data like images without some form of preprocessing. Even on domains like language where Transformers shine, preprocessing (e.g. tokenization) is often needed to scale beyond short input sequences. Perceiver IO uses attention non-homogeneously by mapping inputs to a latent space, processing in that latent space, and decoding to an output space. Perceiver IO has no quadratic dependence on the input or output size: encoder and decoder attention modules depend linearly on the input and output size (respectively), while latent attention is independent of both input and output sizes (Sec. E.2).

While Transformers are typically used in settings with data preprocessed to contain at most a few thousand dimensions (Brown et al., 2020; Raffel et al., 2020), we show good results on domains with hundreds of thousands of dimensions. This architecture can be applied to inputs of any shape or spatial layout including inputs or outputs with different spatial structure (e.g. sound and video). In contrast to latent spaces typically used in vision (e.g. Ronneberger et al., 2015) the latent does not explicitly share the structure (spatial or otherwise) of the inputs.

For tasks with simple outputs, such as classification, these queries can be reused for every example and can be learned from scratch. For outputs with a spatial or sequence structure, we include a position encoding (e.g. a learned positional encoding or a Fourier feature) representing the position to be decoded in the output. For outputs with a multi-task or multimodal structure, we learn a single query for each task or for each modality: this information allows the network to distinguish one task or modality query from the others, much as positional encodings allow attention to distinguish one position from another.

We find that even very simple query features can produce good results, suggesting that the latent attention process is able to learn to organize the relevant information in a way that's easy to query. Each output point depends only on its query and the latent array, allowing us to decode outputs in parallel.

To probe the generality of Perceiver IO, we evaluate it on several domains including language understanding (Wikipedia+C4 masked language modeling), visual understanding (Sintel/KITTI optical flow and ImageNet classification), multi-modal (Kinetics autoencoding and AudioSet classification) & multi-task settings (multi-task GLUE), and symbolic representations for games (StarCraft II).

Although Transformers were originally developed for language, their quadratic complexity makes them difficult to use on language inputs without tokenization, which typically shortens the length of input sequences by a factor of \(\sim\)4. But unlike Transformer-based models such as BERT (Devlin et al., 2019) or XLNet (Yang et al., 2019), Perceiver IO scales linearly with input length. Our experiments focus on showing that Perceiver IO performs as well as or better than Transformers for masked language modeling (MLM) while removing tokenization (which is hard to maintain, introduces engineering overhead, and adds needless complexity to language models (Bostrom and Durrett, 2020; Clark et al., 2022)).

We find that at a given FLOPs budget, Perceiver IO trained without tokenization matches the performance of a strong Transformer-based model trained with SentencePiece tokenization (Sennrich et al., 2016; Kudo and Richardson, 2018).

**Perceiver IO on SentencePiece tokens.** We first observe that Perceiver IO applied on SentencePiece tokenized input sequences slightly outperforms a strong BERT baseline applied on the same inputs (81.2 vs 81.1). As a result of the reduced latent size of \(256\) we can train a much deeper network with 26 processing layers compared to BERT Base (12 layers) while maintaining a similar FLOPs budget.

Rather than using a fixed, handcrafted vocabulary, our model works directly with the raw byte inputs: we simply feed in and predict the UTF-8 bytes of the input string. Perceiver IO significantly outperforms a byte-level BERT baseline at the same FLOPs

Despite its greater depth, Perceiver IO is also faster than the Transformer-based BERT baselines in real wall-clock terms – by over a factor of 2 for the byte-based models – as shown in Tab. 1.

Our work scales to byte-level inputs without making any assumptions about the structure of the input, which allows it to be used beyond language as shown in the following sections.

We use multitask queries as described in Sec. 3.2 to finetune on all 8 GLUE tasks simultaneously using the UTF-8 byte model (results in Tab. 2).

We observe that our multitask approach outperforms single-task approaches and matches the approach that uses 8 task-specific input tokens. Our approach is more generic as it decouples the output array from the input array by not relying on [CLS] tokens.

Optical flow is a decades-old open problem in computer vision (Lucas and Kanade, 1981; Horn and Schunck, 1981). Given two images of the same scene (e.g. two consecutive frames of a video), the task is to estimate the 2D displacement for each pixel in the first image.

Optical flow is challenging for neural networks for two reasons. First, optical flow relies on finding correspondence: a single frame provides no information about flow, and images with extremely different appearance can produce the same flow.

Algorithms for optical flow thus must learn to accomplish several steps in a way that transfers from synthetic to real data. First, the algorithm must find correspondence between points.

We explore using Perceiver IO for audio-video-label multimodal autoencoding on the Kinetics-700-2020 dataset (Smaira et al., 2020).

With traditional autoencoding models like convolutional encoder-decoders, it is not obvious how to combine these modalities, because each uses data of different dimensions - 3D (video), 1D (raw audio), and 0D (class labels) - and with wildly different numbers of elements. With Perceiver IO, we pad the inputs with modality-specific embeddings, serialize them into a single 2D input array and query outputs using queries containing position encodings (for video and audio) and modality embeddings.

**Please read the Appendix for results on ImageNet (Sec. A), StarCraft II (Sec. B), and AudioSet (Sec. C).**

(1) on ImageNet, Perceiver IO surpasses 80% top-1 accuracy (84.5% top-1) without using 2D convolutions after pretraining on JFT.

In this work we introduce Perceiver IO, an architecture capable of handling general purpose inputs and outputs while scaling linearly in both input and output sizes. As we show, this architecture achieves good results in a wide variety of settings, making it a promising candidate for a _general purpose_ neural network architecture. Perceiver IO leverages the expressive power of latent attention and uses learned queries to expose a simple and unified interface that can handle multimodal and multitask settings. Overall, Perceiver IO offers a promising way to simplify the construction of sophisticated neural pipelines and facilitate progress on multimodal and multitask problems.

On all domains but StarCraft II, we include experiments with several input configurations, ranging from no domain adaptation (e.g. tokenizer-free language, flow from raw pixels, ImageNet with no convolutional or patch-based preprocessing and fully learned position encodings) to moderate domain adaptation (e.g. SentencePiece language understanding, flow from conv+maxpool-processed images and with RAFT upsampling, ImageNet with conv+maxpool-preprocessing and 2D Fourier features). These results demonstrate the unprecedented generality of Perceiver IO, the simplicity that this architecture unlocks in handling a range of tasks, and its flexibility to work as part of a domain-adapted system. Perceiver did well on ImageNet (Deng et al., 2009) classification without using 2D structure in the design of the architecture, but generated class scores using a simple average + project decoder (see Sec. E.3 and Fig. 6 for a diagram illustrating the difference between the two forms of decoder).

By replacing these 2D position features with a fully learned position encoding as used on language, we can learn an image classification model that is given no privileged information about the structure of images.

ImageNet networks that use this positional encoding are given no information about 2D image structure.

To our knowledge, this is the best result by any model on ImageNet without 2D architectural or feature information.

Because ImageNet data points are relatively small (Tab. 5), we are able to process full images without convolutional pre- and post-processing. Consequently, we can use this dataset to probe the sensitivity of the model to convolutional pre-processing. Incorporating a single convolution + max pooling leads to a moderate improvement in the performance of the architecture: this is perhaps unsurprising, as convolutional pre-processing injects information about the 2D structure of images into the architecture. By comparison ViT first processes images by applying a 2D convolution with matched kernel and stride to downsample its inputs (referred to as a "linear projection of flattened patches" in that work and throughout the ViT literature). As in other experiments, we find that incorporating an attention-based decoder (Perceiver IO) leads to better results than averaging and pooling the output (Perceiver).

For this reason, we view AlphaStar as an important test case for Perceiver IO's ability to function as a general-purpose tool for processing symbolic or set-valued data. If the question is "can Perceiver IO serve as a replacement for a well-tuned Transformer as a symbolic processing engine?" then the answer is yes: We obtained StarCraft II results by using Perceiver IO instead of a Transformer for the AlphaStar entity encoder. We replaced the Transformer with a Perceiver IO with a latent of index dimension 32, keeping the input and output size of 512 units.

Perceiver IO works out of the box. We observed that the resulting agent reached the same level of performance as the original AlphaStar agent, reaching an 87% win-rate versus the Elite bot after behavioral cloning (Pomerleau, 1989) on human data, while also leading to a 3\(\times\) decrease in FLOPs (Tab. 9).

We show that the trend identified on ImageNet holds more generally, by revisiting the multimodal AudioSet classification domain. AudioSet is a large-scale event classification dataset containing 1.7 million training examples, each consisting of 10s long video and audio. Each example is labeled with several labels drawn from 527 classes. We perform experiments using the protocol described in Jaegle et al. (2021), training models for 100 epochs using 32-frame clips at train time and 16 overlapping 32-frame clips at test time.

Perceiver IO is constructed from GPT-2-style (Radford et al., 2019) Transformer attention modules, which consist of QKV attention followed by an MLP, along with linear projection layers to ensure inputs to and outputs from the QKV attention and MLP take on desired sizes.

The view of attention that we describe encompasses both cross-attention and self-attention, both of which are specific ways of using QKV-attention. Perceiver IO uses cross-attention for encoder and decoder attention modules and uses self-attention for the latent processing modules.

In other words, Perceiver IO has complexity linear in the size of the input and output arrays and it decouples the depth of the latent transformer from the input and output sizes. Both of these properties contribute to Perceiver IO's efficiency: while many proposals for efficient attention modules or architectures include linear or sub-quadratic scaling with input/output size, Perceiver IO is unusual in also decoupling depth from input/output size (without requiring domain-specific strategies like 2D convolution).

One application of Perceiver IO is byte-level language processing, which has concurrently been addressed by several other groups. Clark et al. (2022) trains models on Unicode code points and shows results competitive with subword-based models on a multilingual question answering dataset. Tay et al. (2022) trains on UTF-8 bytes directly by introducing a hand-designed module that is trained end-to-end to perform subword tokenization and produces results on-par with and sometimes better than subword-based models. Xue et al. (2022) trains encoder-decoder T5 models on UTF-8 bytes directly and shows that making the encoder 3x deeper than the decoder leads to comparable performance with subword baselines.

For single-task experiments, we do not require a [CLS] token as we use a single decoding query vector.

Perceiver IO is somewhat slower on traditional GPUs than our baseline RAFT model, but we find that the trend reverses on TPUs, which is the target architecture for our work.

Our most expensive model achieves approximately 0.8 frames/sec on a 2017 TITAN Xp, and our lightweight model (with conv downsampling and RAFT-style upsampling) achieves 3.3 frames/sec, which is not far from the 10 frames per second reported for RAFT (Teed & Deng, 2020). On the publicly-available TPU v3, however, our most expensive model achieves 4.4 frames/sec on a single TPU core, and 17.8 frames/sec for the lightweight model.

We suspect that the difference is due to the gather operations required for RAFT but not for Perceivers, which are slow on TPU due to their poor memory locality properties.

We see that the algorithm is capable of dealing with heavy occlusion, and can propagate optical flow across large regions with very little texture. The network can also deal with very large motions and very small objects. Finally, to verify that Perceiver IO performs well on real-world data despite being trained only on synthetic imagery, we applied it to a small number (roughly 10) real videos taken from Getty images (www.gettyimages.com). Perceiver IO typically performs very well out-of-domain, although some failure cases remain:

Perceiver IO can pick up on remarkably small objects such as the water droplets thrown by the girl's shoe in pigeon.mp4 or the confetti in thai_dance.mp4.

For the multimodal autoencoding experiments, we patch preprocessing for both images and audio, and we embed the labels as one-hot labels. The patch size is \(1\times 4\times 4\) for video and 16 for audio. The audio is sampled at 48kHz, or 1920 samples per frame. The decoder outputs \(16\times 224\times 224+16\times 1920/16+1\) vectors with 512 channels, that is, one element for each pixel in the video, one element for each audio patch, and one element for the classification label. These are then linearly projected to the appropriate channel size for each modality: 3 for videos, 16 for audio and 700 for classification (the logits for each of the 700 classes in Kinetics700).

We note that we read and generate the audio waveform directly in the time domain; we do not transform first to a spectrogram. We use a 387 dimensional 3D Fourier position embedding for each input video patch and a 385 dimensional 1D Fourier position embedding for each audio patch (385 to ensure the input dimensions to Perceiver IO match for all elements). In addition, we pad all input elements with a learned vector representing the modality; inputs from the same modality share the same token. In particular, we add a 317 dimensional modality embedding to video elements, a 319 dimensional modality embedding to audio elements, and a 4 dimensional modality embedding to the label, so that all elements have 704 features. The decoder queries are also constructed from Fourier position embeddings for video and audio and a learned positional embedding for label: 387 features for video, 385 features for audio, and 1024 learned features for the label. We pad the queries for each modality with a different learned vector for each modality, so that the final feature size for the queries is 1026.

To help verify the quality of Perceiver IO's outputs on real-world data, we applied it a small number of real videos (\(\sim\)10) with audio taken from Getty Images. Perceiver IO is able to capture the structure of both video and audio inputs, despite encoding both jointly with a single network.

Input: Figure 1: The Perceiver IO architecture can be used on domains with a wide variety of input and output spaces, including multi-task language understanding, dense visual tasks like optical flow, hybrid dense/sparse multimodal tasks such as video+audio+class autoencoding, and tasks with symbolic outputs like StarCraft II. See Tables 5 and 6 for details of all domains considered here.

Input: Figure 8: Qualitative examples of optical flow. For each image pair, we show the two frames (top), and then the estimated flow (bottom left) and the ground-truth flow (bottom right). In the left example, we see one person under heavy occlusion where the correct flow is propagated into a region with few details. Another person in the foreground has clothes with little texture and substantial blur, and yet the algorithm can propagate the flow across the entire region. In the center example, we see very large motions from both the dragon and the person, yet many fine structures are preserved like the pole. On the right, we see a forest scene with a few extremely small objects with very subtle motions (circled) which our algorithm is able to detect and segment correctly.

Input: Figure 2: The Perceiver IO architecture. Perceiver IO maps arbitrary input arrays to arbitrary output arrays in a domain agnostic process. The bulk of the computation happens in a latent space whose size is typically smaller than the inputs and outputs, which makes the process computationally tractable even for very large inputs & outputs. See Fig. 5 for a more detailed look at encode, process, and decode attention.

Input: Table 5: Details of each of the tasks we use to evaluate Perceiver IO here. The positional and task embeddings appended to inputs for each case are listed in Tab. 6.

Input: Table 6: Table best viewed on a screen. The structure and size of the positional and task embeddings used to construct Perceiver IO’s encoder key-value inputs and decoder query inputs, for each domain described in the main text. “[x, y]” indicates that x’s and y’s features are concatenated, while “x + y” indicates that x’s and y’s features are added to produce the full featurization. “FF” = Fourier features as in Jaegle et al. (2021).

Input: Figure 4: Multimodal audio-video-label autoencoding with 88x compression. Side-by-side: inputs on left, reconstructions right. See the supplemental material for example output video and audio.

Input: Figure 3: We construct queries with output-specific features to produce outputs with different semantics. For settings where each output point differs only in its position, like language, a position embedding can be used. Input features for the target output can also be used to query, either alone (as for StarCraft II) or alongside position features (as for flow). For multi-{task, modal} settings we use one embedding for each {task, modality} instead of each position. A single learned embedding suffices for simple classification tasks, like ImageNet. For tasks with heterogeneous outputs like multimodal autoencoding, features that are specific to some queries (like xy position) can be combined with modality embeddings, which also pad embeddings to fixed length.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence, the claim appears moderately overstated. While Perceiver IO does demonstrate impressive versatility across multiple domains and tasks, some key limitations and caveats are not acknowledged in the claim:

1. The evidence shows significant task-specific engineering/hyperparameter tuning is still needed (noted in reviewer comments)
2. The claim of "without a performance or speed hit" is only partially supported - there are speed tradeoffs depending on hardware (TPU vs GPU)
3. The removal of domain-specific preprocessing still requires careful consideration of input/output configurations
4. Some capabilities mentioned are demonstrated but with important qualifications (e.g., multimodal performance varies by task)

<score>3</score>