Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

As highlights, Perceiver IO outperforms a Transformer-based BERT baseline on the GLUE language benchmark despite removing input tokenization and achieves state-of-the-art performance on Sintel optical flow estimation with no explicit mechanisms for multiscale correspondence.

The evidence to be evaluated is:As hardware grew more powerful, neural nets led to breakthroughs in image understanding (Krizhevsky et al., 2012; Zeiler and Fergus, 2014; Szegedy et al., 2015) and interest intensified: autoregressive models that could process and complete samples of handwriting were developed (Graves, 2013), and new convolutional network designs led to good results in structured output spaces like semantic segmentation (Farabet et al., 2012; Long et al., 2015; Ronneberger et al., 2015), pose estimation (Toshev and Szegedy, 2014), detection (Sermanet et al., 2014), captioning (You et al., 2016), and optical flow (Fischer et al., 2015). At the same time, natural language applications research has made extensive progressive in capturing the structured nature of language, typically via autoregressive models (Collobert et al., 2011; Sutskever et al., 2014; Vaswani et al., 2017; Radford et al., 2019; Brown et al., 2020) or context prediction (Mikolov et al., 2013; Pennington et al., 2014; Devlin et al., 2019).

Several groups have demonstrated that Transformers (originally designed for language) can be used or adapted to non-language tasks (e.g. Chen et al. 2020; Lu et al. 2021), but the limited scalability of Transformers limits their usefulness as general-purpose architectures.

Our work uses attention over inputs and outputs of different sizes in part to produce an efficient attention architecture, and several other efficient attention architectures have been proposed, largely for language or small-scale problems (e.g. Xiong et al. 2021; Wang et al. 2020; Tay et al. 2021a; Beltagy et al. 2020 and see Tay et al. 2021b). The focus of our work is developing an architecture that is efficient and also performs well in many settings with a wide range of inputs and outputs.

Perceiver IO builds on this body of work to produce a general purpose architecture that can be easily and widely applied. The Perceiver IO architecture builds on the Perceiver (Jaegle et al., 2021), which achieved its cross-domain generality by assuming that its input is a simple 2D byte array: a set of elements (which might be pixels or patches in vision, characters or words in language, or some form of embedding, learned or otherwise), each described by a feature vector. The model then encodes information about the input array using a smaller number of latent feature vectors, using Transformer-style attention, followed by iterative processing and a final aggregation down to a category label. Rather than output a single category, Perceiver IO aims to have the same level of generality with respect to its _outputs_ as the Perceiver has with respect to its _inputs_: that is, it should produce arbitrary output arrays.

We first **encode** by applying an attention module that maps input arrays \(x\in\mathbb{R}^{M\times C}\) to arrays in a latent space \(z\in\mathbb{R}^{N\times D}\). We next **process** the latents \(z\) by applying a series of modules that take in and return arrays in this latent space. Finally, we **decode** by applying an attention module that maps latent arrays to output arrays \(y\in\mathbb{R}^{O\times E}\).

Following the design of the Perceiver, we implement each of the architecture's components using Transformer-style attention modules. Each of these modules applies a global query-key-value (QKV) attention operation followed by a multi-layer perceptron (MLP).

The Perceiver IO architecture builds on primitives similar to those in Transformers.

Even on domains like language where Transformers shine, preprocessing (e.g. tokenization) is often needed to scale beyond short input sequences. Perceiver IO uses attention non-homogeneously by mapping inputs to a latent space, processing in that latent space, and decoding to an output space. Perceiver IO has no quadratic dependence on the input or output size: encoder and decoder attention modules depend linearly on the input and output size (respectively), while latent attention is independent of both input and output sizes (Sec. E.2).

To probe the generality of Perceiver IO, we evaluate it on several domains including language understanding (Wikipedia+C4 masked language modeling), visual understanding (Sintel/KITTI optical flow and ImageNet classification), multi-modal (Kinetics autoencoding and AudioSet classification) & multi-task settings (multi-task GLUE), and symbolic representations for games (StarCraft II).

We first compare Perceiver IO to standard Transformers for language. Although Transformers were originally developed for language, their quadratic complexity makes them difficult to use on language inputs without tokenization, which typically shortens the length of input sequences by a factor of \(\sim\)4. But unlike Transformer-based models such as BERT (Devlin et al., 2019) or XLNet (Yang et al., 2019), Perceiver IO scales linearly with input length. Our experiments focus on showing that Perceiver IO performs as well as or better than Transformers for masked language modeling (MLM) while removing tokenization (which is hard to maintain, introduces engineering overhead, and adds needless complexity to language models (Bostrom and Durrett, 2020; Clark et al., 2022)).

We evaluate the quality of the learned representation on the GLUE benchmark (Wang et al., 2019) and report our results in Tab. 1. We find that at a given FLOPs budget, Perceiver IO trained without tokenization matches the performance of a strong Transformer-based model trained with SentencePiece tokenization (Sennrich et al., 2016; Kudo and Richardson, 2018).

**Finetuning.** We finetune Perceiver IO on the GLUE Benchmark Wang et al. (2019), reporting the best performance on the dev set for a fixed size sweep of finetuning hyperparameters.

We first observe that Perceiver IO applied on SentencePiece tokenized input sequences slightly outperforms a strong BERT baseline applied on the same inputs (81.2 vs 81.1). As a result of the reduced latent size of \(256\) we can train a much deeper network with 26 processing layers compared to BERT Base (12 layers) while maintaining a similar FLOPs budget.

Rather than using a fixed, handcrafted vocabulary, our model works directly with the raw byte inputs: we simply feed in and predict the UTF-8 bytes of the input string. Perceiver IO significantly outperforms a byte-level BERT baseline at the same FLOPs

Despite its greater depth, Perceiver IO is also faster than the Transformer-based BERT baselines in real wall-clock terms – by over a factor of 2 for the byte-based models – as shown in Tab. 1.

Our work scales to byte-level inputs without making any assumptions about the structure of the input, which allows it to be used beyond language as shown in the following sections.

We use multitask queries as described in Sec. 3.2 to finetune on all 8 GLUE tasks simultaneously using the UTF-8 byte model (results in Tab. 2).

We observe that our multitask approach outperforms single-task approaches and matches the approach that uses 8 task-specific input tokens.

State of the art algorithms, such as PWCNet (Sun et al., 2018), RAFT (Teed and Deng, 2020) or GMA (Jiang et al., 2021), use explicit machinery to ensure each of these steps is performed correctly even on out-of-domain data.

In contrast, we apply Perceiver IO to flow in a straightforward manner. We concatenate the frames along the channel dimension and extract a \(3\times 3\) patch around each pixel (leading to \(3\times 3\times 3\times 2=54\) values for each pixel).

We also test a version with convolutional downsampling and RAFT-style upsampling, which performs only slightly worse while improving computation time. It may seem counter-intuitive to append the images along the channel dimension, as large motions might result in pixels on entirely different objects being concatenated. However, this kind of operation isn't unprecedented: one of the earliest optical flow algorithms, Lucas-Kanade (Lucas & Kanade, 1981), makes explicit use of the _temporal_ image gradient, which is approximated by the difference in intensities at a given pixel across two frames. The algorithm uses the fact that the temporal gradient of the image approximates the spatial gradient times the spatial velocity, if lighting effects are ignored. The approximation is even better for image regions with very little texture.

On Sintel (Butler et al., 2012), our results are slightly better than RAFT on Sintel on the outperform PWCNet on KITTI (Meaze & Geiger, 2015). As far as we are aware, this result is state of the art on Sintel.final (GMA Jiang et al. (2021)) to PWCNet and RAFT baselines trained by the AutoFlow authors. On Sintel (Butler et al., 2012), our results are slightly better than RAFT on Sintel and outperform PWCNet on KITTI (Meaze & Geiger, 2015). As far as we are aware, this result is state of the art on Sintel.final (GMA Jiang et al. (2021)) propagate motion across image regions with little texture.

(1) on ImageNet, Perceiver IO surpasses 80% top-1 accuracy (84.5% top-1) without using 2D convolutions after pretraining on JFT.

On all domains but StarCraft II, we include experiments with several input configurations, ranging from no domain adaptation (e.g. tokenizer-free language, flow from raw pixels, ImageNet with no convolutional or patch-based preprocessing and fully learned position encodings) to moderate domain adaptation (e.g. SentencePiece language understanding, flow from conv+maxpool-processed images and with RAFT upsampling, ImageNet with conv+maxpool-preprocessing and 2D Fourier features). These results demonstrate the unprecedented generality of Perceiver IO, the simplicity that this architecture unlocks in handling a range of tasks, and its flexibility to work as part of a domain-adapted system.

While neither the Perceiver and Perceiver IO incorporate any 2D spatial structure architecturally, they use positional features that inject 2D spatial information (Sec. 3.2 and Appendix sec. D of Jaegle et al. 2021). By replacing these 2D position features with a fully learned position encoding as used on language, we can learn an image classification model that is given no privileged information about the structure of images.

Perceiver IO is constructed from GPT-2-style (Radford et al., 2019) Transformer attention modules, which consist of QKV attention followed by an MLP, along with linear projection layers to ensure inputs to and outputs from the QKV attention and MLP take on desired sizes.

The view of attention that we describe encompasses both cross-attention and self-attention, both of which are specific ways of using QKV-attention. Perceiver IO uses cross-attention for encoder and decoder attention modules and uses self-attention for the latent processing modules.

In other words, Perceiver IO has complexity linear in the size of the input and output arrays and it decouples the depth of the latent transformer from the input and output sizes. Both of these properties contribute to Perceiver IO's efficiency: while many proposals for efficient attention modules or architectures include linear or sub-quadratic scaling with input/output size, Perceiver IO is unusual in also decoupling depth from input/output size (without requiring domain-specific strategies like 2D convolution).

One application of Perceiver IO is byte-level language processing, which has concurrently been addressed by several other groups. Clark et al. (2022) trains models on Unicode code points and shows results competitive with subword-based models on a multilingual question answering dataset. Tay et al. (2022) trains on UTF-8 bytes directly by introducing a hand-designed module that is trained end-to-end to perform subword tokenization and produces results on-par with and sometimes better than subword-based models. Xue et al. (2022) trains encoder-decoder T5 models on UTF-8 bytes directly and shows that making the encoder 3x deeper than the decoder leads to comparable performance with subword baselines.

Perceiver IO is somewhat slower on traditional GPUs than our baseline RAFT model, but we find that the trend reverses on TPUs, which is the target architecture for our work.

Our most expensive model achieves approximately 0.8 frames/sec on a 2017 TITAN Xp, and our lightweight model (with conv downsampling and RAFT-style upsampling) achieves 3.3 frames/sec, which is not far from the 10 frames per second reported for RAFT (Teed & Deng, 2020). On the publicly-available TPU v3, however, our most expensive model achieves 4.4 frames/sec on a single TPU core, and 17.8 frames/sec for the lightweight model. An efficient Tensorflow implementation of RAFT (Sun et al., 2020) (received courtesy of the authors) achieves only 1.6 frames/sec on the same hardware.

We see that the algorithm is capable of dealing with heavy occlusion, and can propagate optical flow across large regions with very little texture. The network can also deal with very large motions and very small objects. Finally, to verify that Perceiver IO performs well on real-world data despite being trained only on synthetic imagery, we applied it to a small number (roughly 10) real videos taken from Getty images (www.gettyimages.com). Perceiver IO typically performs very well out-of-domain, although some failure cases remain: for instance, shadows tend to be interpreted as objects (Autoflow contains no shadows), and large regions with compression artifacts but no other texture may result in hallucinated flow (Autoflow contains no video compression artifacts). We include three challenging examples in the supplementary zip file, each of which depict complex motion and small objects. Perceiver IO can pick up on remarkably small objects such as the water droplets thrown by the girl's shoe in pigeon.mp4 or the confetti in thai_dance.mp4.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence carefully, I find the claim is somewhat overstated (score of 4). Here's why:

1. Regarding GLUE performance: While Perceiver IO slightly outperforms BERT baseline (81.2 vs 81.1) when using SentencePiece tokenization, the claim implies superior performance without tokenization. However, the evidence only shows it outperforms a "byte-level BERT baseline" without specifying the margin or baseline's performance level.

2. Regarding Sintel optical flow: The claim states "state-of-the-art performance" but the evidence is more nuanced, stating results are "slightly better than RAFT on Sintel" and there's some ambiguity in the text about whether it truly achieves SOTA (mentions GMA as SOTA).

Additionally, the claim doesn't acknowledge important caveats mentioned in the evidence:
- Hardware-specific performance differences (slower on GPUs)
- Limited real-world testing (only ~10 Getty videos)
- Existence of failure cases with shadows and compression artifacts

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

As highlights, Perceiver IO outperforms a Transformer-based BERT baseline on the GLUE language benchmark despite removing input tokenization and achieves state-of-the-art performance on Sintel optical flow estimation with no explicit mechanisms for multiscale correspondence.

The review comment to be evaluated is:

#### Strengths

- Major contribution is ability to scale the output to arbitrary length using an output query array. This enables supporting different types of output and scales well to different tasks.
- Byte level performance of Perceiver IO is impressive compared to BERT baseline. Byte level embeddings can be fed to Perceiver IO without any tokenization and this setting gets comparable performance with BERT sentence piece tokenization.
- Shows good results with multitask learning, using both single and task specific tokens.
- Authors do a thorough evaluation of this architecture on a wide variety of tasks, and achieves comparable results on multiple tasks in language, vision, multimodal, RL domains.

#### Weaknesses

- Novelty over Perceiver : Although I find this work quite impressive as it scales linearly with output embedding sizes as well as arbitrary types of outputs, the overall architecture seems incremental compared to Perceiver. One important change is there is no cross attention in the intermediate layers. I would love to see more detailed analysis why that is removed in this architecture and empirical evidence to support this change.

- How does choosing values for N,D affect the performance on various tasks. If we decouple input representation, output representation and base architecture, how does model performance scale with depth, N, D values.  Ablations on these hyper-parameters are required for more clarity.

- Given Perceiver can handle multiple modalities, I would be interested to see how it performs on multimodal tasks like VQA, Image/Text Retrieval, Hateful Memes etc and comparison to multimodal models like CLIP[1], ALIGN[2], SimVL[3].

  - [1] Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry et al. "Learning transferable visual models from natural language supervision." arXiv preprint arXiv:2103.00020 (2021).
  - [2] Jia, Chao, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. "Scaling up visual and vision-language representation learning with noisy text supervision." arXiv preprint arXiv:2102.05918 (2021).
  - [3] Wang, Zirui, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. "SimVLM: Simple visual language model pretraining with weak supervision." arXiv preprint arXiv:2108.10904 (2021).

The evidence to be evaluated is:As hardware grew more powerful, neural nets led to breakthroughs in image understanding (Krizhevsky et al., 2012; Zeiler and Fergus, 2014; Szegedy et al., 2015) and interest intensified: autoregressive models that could process and complete samples of handwriting were developed (Graves, 2013), and new convolutional network designs led to good results in structured output spaces like semantic segmentation (Farabet et al., 2012; Long et al., 2015; Ronneberger et al., 2015), pose estimation (Toshev and Szegedy, 2014), detection (Sermanet et al., 2014), captioning (You et al., 2016), and optical flow (Fischer et al., 2015). At the same time, natural language applications research has made extensive progressive in capturing the structured nature of language, typically via autoregressive models (Collobert et al., 2011; Sutskever et al., 2014; Vaswani et al., 2017; Radford et al., 2019; Brown et al., 2020) or context prediction (Mikolov et al., 2013; Pennington et al., 2014; Devlin et al., 2019).

Several groups have demonstrated that Transformers (originally designed for language) can be used or adapted to non-language tasks (e.g. Chen et al. 2020; Lu et al. 2021), but the limited scalability of Transformers limits their usefulness as general-purpose architectures.

Our work uses attention over inputs and outputs of different sizes in part to produce an efficient attention architecture, and several other efficient attention architectures have been proposed, largely for language or small-scale problems (e.g. Xiong et al. 2021; Wang et al. 2020; Tay et al. 2021a; Beltagy et al. 2020 and see Tay et al. 2021b). The focus of our work is developing an architecture that is efficient and also performs well in many settings with a wide range of inputs and outputs.

Perceiver IO builds on this body of work to produce a general purpose architecture that can be easily and widely applied. The Perceiver IO architecture builds on the Perceiver (Jaegle et al., 2021), which achieved its cross-domain generality by assuming that its input is a simple 2D byte array: a set of elements (which might be pixels or patches in vision, characters or words in language, or some form of embedding, learned or otherwise), each described by a feature vector. The model then encodes information about the input array using a smaller number of latent feature vectors, using Transformer-style attention, followed by iterative processing and a final aggregation down to a category label. Rather than output a single category, Perceiver IO aims to have the same level of generality with respect to its _outputs_ as the Perceiver has with respect to its _inputs_: that is, it should produce arbitrary output arrays.

We first **encode** by applying an attention module that maps input arrays \(x\in\mathbb{R}^{M\times C}\) to arrays in a latent space \(z\in\mathbb{R}^{N\times D}\). We next **process** the latents \(z\) by applying a series of modules that take in and return arrays in this latent space. Finally, we **decode** by applying an attention module that maps latent arrays to output arrays \(y\in\mathbb{R}^{O\times E}\).

Following the design of the Perceiver, we implement each of the architecture's components using Transformer-style attention modules. Each of these modules applies a global query-key-value (QKV) attention operation followed by a multi-layer perceptron (MLP).

The Perceiver IO architecture builds on primitives similar to those in Transformers.

Even on domains like language where Transformers shine, preprocessing (e.g. tokenization) is often needed to scale beyond short input sequences. Perceiver IO uses attention non-homogeneously by mapping inputs to a latent space, processing in that latent space, and decoding to an output space. Perceiver IO has no quadratic dependence on the input or output size: encoder and decoder attention modules depend linearly on the input and output size (respectively), while latent attention is independent of both input and output sizes (Sec. E.2).

To probe the generality of Perceiver IO, we evaluate it on several domains including language understanding (Wikipedia+C4 masked language modeling), visual understanding (Sintel/KITTI optical flow and ImageNet classification), multi-modal (Kinetics autoencoding and AudioSet classification) & multi-task settings (multi-task GLUE), and symbolic representations for games (StarCraft II).

We first compare Perceiver IO to standard Transformers for language. Although Transformers were originally developed for language, their quadratic complexity makes them difficult to use on language inputs without tokenization, which typically shortens the length of input sequences by a factor of \(\sim\)4. But unlike Transformer-based models such as BERT (Devlin et al., 2019) or XLNet (Yang et al., 2019), Perceiver IO scales linearly with input length. Our experiments focus on showing that Perceiver IO performs as well as or better than Transformers for masked language modeling (MLM) while removing tokenization (which is hard to maintain, introduces engineering overhead, and adds needless complexity to language models (Bostrom and Durrett, 2020; Clark et al., 2022)).

We evaluate the quality of the learned representation on the GLUE benchmark (Wang et al., 2019) and report our results in Tab. 1. We find that at a given FLOPs budget, Perceiver IO trained without tokenization matches the performance of a strong Transformer-based model trained with SentencePiece tokenization (Sennrich et al., 2016; Kudo and Richardson, 2018).

**Finetuning.** We finetune Perceiver IO on the GLUE Benchmark Wang et al. (2019), reporting the best performance on the dev set for a fixed size sweep of finetuning hyperparameters.

We first observe that Perceiver IO applied on SentencePiece tokenized input sequences slightly outperforms a strong BERT baseline applied on the same inputs (81.2 vs 81.1). As a result of the reduced latent size of \(256\) we can train a much deeper network with 26 processing layers compared to BERT Base (12 layers) while maintaining a similar FLOPs budget.

Rather than using a fixed, handcrafted vocabulary, our model works directly with the raw byte inputs: we simply feed in and predict the UTF-8 bytes of the input string. Perceiver IO significantly outperforms a byte-level BERT baseline at the same FLOPs

Despite its greater depth, Perceiver IO is also faster than the Transformer-based BERT baselines in real wall-clock terms – by over a factor of 2 for the byte-based models – as shown in Tab. 1.

Our work scales to byte-level inputs without making any assumptions about the structure of the input, which allows it to be used beyond language as shown in the following sections.

We use multitask queries as described in Sec. 3.2 to finetune on all 8 GLUE tasks simultaneously using the UTF-8 byte model (results in Tab. 2).

We observe that our multitask approach outperforms single-task approaches and matches the approach that uses 8 task-specific input tokens.

State of the art algorithms, such as PWCNet (Sun et al., 2018), RAFT (Teed and Deng, 2020) or GMA (Jiang et al., 2021), use explicit machinery to ensure each of these steps is performed correctly even on out-of-domain data.

In contrast, we apply Perceiver IO to flow in a straightforward manner. We concatenate the frames along the channel dimension and extract a \(3\times 3\) patch around each pixel (leading to \(3\times 3\times 3\times 2=54\) values for each pixel).

We also test a version with convolutional downsampling and RAFT-style upsampling, which performs only slightly worse while improving computation time. It may seem counter-intuitive to append the images along the channel dimension, as large motions might result in pixels on entirely different objects being concatenated. However, this kind of operation isn't unprecedented: one of the earliest optical flow algorithms, Lucas-Kanade (Lucas & Kanade, 1981), makes explicit use of the _temporal_ image gradient, which is approximated by the difference in intensities at a given pixel across two frames. The algorithm uses the fact that the temporal gradient of the image approximates the spatial gradient times the spatial velocity, if lighting effects are ignored. The approximation is even better for image regions with very little texture.

On Sintel (Butler et al., 2012), our results are slightly better than RAFT on Sintel on the outperform PWCNet on KITTI (Meaze & Geiger, 2015). As far as we are aware, this result is state of the art on Sintel.final (GMA Jiang et al. (2021)) to PWCNet and RAFT baselines trained by the AutoFlow authors. On Sintel (Butler et al., 2012), our results are slightly better than RAFT on Sintel and outperform PWCNet on KITTI (Meaze & Geiger, 2015). As far as we are aware, this result is state of the art on Sintel.final (GMA Jiang et al. (2021)) propagate motion across image regions with little texture.

(1) on ImageNet, Perceiver IO surpasses 80% top-1 accuracy (84.5% top-1) without using 2D convolutions after pretraining on JFT.

On all domains but StarCraft II, we include experiments with several input configurations, ranging from no domain adaptation (e.g. tokenizer-free language, flow from raw pixels, ImageNet with no convolutional or patch-based preprocessing and fully learned position encodings) to moderate domain adaptation (e.g. SentencePiece language understanding, flow from conv+maxpool-processed images and with RAFT upsampling, ImageNet with conv+maxpool-preprocessing and 2D Fourier features). These results demonstrate the unprecedented generality of Perceiver IO, the simplicity that this architecture unlocks in handling a range of tasks, and its flexibility to work as part of a domain-adapted system.

While neither the Perceiver and Perceiver IO incorporate any 2D spatial structure architecturally, they use positional features that inject 2D spatial information (Sec. 3.2 and Appendix sec. D of Jaegle et al. 2021). By replacing these 2D position features with a fully learned position encoding as used on language, we can learn an image classification model that is given no privileged information about the structure of images.

Perceiver IO is constructed from GPT-2-style (Radford et al., 2019) Transformer attention modules, which consist of QKV attention followed by an MLP, along with linear projection layers to ensure inputs to and outputs from the QKV attention and MLP take on desired sizes.

The view of attention that we describe encompasses both cross-attention and self-attention, both of which are specific ways of using QKV-attention. Perceiver IO uses cross-attention for encoder and decoder attention modules and uses self-attention for the latent processing modules.

In other words, Perceiver IO has complexity linear in the size of the input and output arrays and it decouples the depth of the latent transformer from the input and output sizes. Both of these properties contribute to Perceiver IO's efficiency: while many proposals for efficient attention modules or architectures include linear or sub-quadratic scaling with input/output size, Perceiver IO is unusual in also decoupling depth from input/output size (without requiring domain-specific strategies like 2D convolution).

One application of Perceiver IO is byte-level language processing, which has concurrently been addressed by several other groups. Clark et al. (2022) trains models on Unicode code points and shows results competitive with subword-based models on a multilingual question answering dataset. Tay et al. (2022) trains on UTF-8 bytes directly by introducing a hand-designed module that is trained end-to-end to perform subword tokenization and produces results on-par with and sometimes better than subword-based models. Xue et al. (2022) trains encoder-decoder T5 models on UTF-8 bytes directly and shows that making the encoder 3x deeper than the decoder leads to comparable performance with subword baselines.

Perceiver IO is somewhat slower on traditional GPUs than our baseline RAFT model, but we find that the trend reverses on TPUs, which is the target architecture for our work.

Our most expensive model achieves approximately 0.8 frames/sec on a 2017 TITAN Xp, and our lightweight model (with conv downsampling and RAFT-style upsampling) achieves 3.3 frames/sec, which is not far from the 10 frames per second reported for RAFT (Teed & Deng, 2020). On the publicly-available TPU v3, however, our most expensive model achieves 4.4 frames/sec on a single TPU core, and 17.8 frames/sec for the lightweight model. An efficient Tensorflow implementation of RAFT (Sun et al., 2020) (received courtesy of the authors) achieves only 1.6 frames/sec on the same hardware.

We see that the algorithm is capable of dealing with heavy occlusion, and can propagate optical flow across large regions with very little texture. The network can also deal with very large motions and very small objects. Finally, to verify that Perceiver IO performs well on real-world data despite being trained only on synthetic imagery, we applied it to a small number (roughly 10) real videos taken from Getty images (www.gettyimages.com). Perceiver IO typically performs very well out-of-domain, although some failure cases remain: for instance, shadows tend to be interpreted as objects (Autoflow contains no shadows), and large regions with compression artifacts but no other texture may result in hallucinated flow (Autoflow contains no video compression artifacts). We include three challenging examples in the supplementary zip file, each of which depict complex motion and small objects. Perceiver IO can pick up on remarkably small objects such as the water droplets thrown by the girl's shoe in pigeon.mp4 or the confetti in thai_dance.mp4.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me assess the claim based on the evidence and review comments:

1. Summary evaluation:
The claim makes two major assertions:
- Perceiver IO outperforms BERT baseline on GLUE without tokenization
- Achieves SOTA on Sintel optical flow without explicit multiscale correspondence

For the first assertion, the evidence confirms that Perceiver IO outperforms byte-level BERT baseline and matches/slightly exceeds tokenized BERT performance (81.2 vs 81.1). For the second assertion, while results are strong on Sintel, the evidence is more nuanced - it shows "slightly better than RAFT" performance and uses some domain adaptations in certain configurations. The claim is generally accurate but slightly overstates the optical flow achievements by not acknowledging the qualifications around domain adaptation and preprocessing options.

<score>2</score>

Justification:
1. Language performance claim is well-supported:
- Evidence clearly shows Perceiver IO matches/exceeds BERT on GLUE when using tokenization (81.2 vs 81.1)
- Significantly outperforms byte-level BERT baseline
- Review comments confirm "byte level performance is impressive"

2. Optical flow claim needs minor qualification:
- Evidence shows strong but not definitively SOTA results ("slightly better than RAFT")
- Some configurations use domain adaptations like "convolutional downsampling and RAFT-style upsampling"
- Model exhibits some limitations on real-world data (shadows, compression artifacts)

The claim is largely accurate but would benefit from slightly more precise wording regarding the optical flow results. The deviation from perfect accuracy is minor enough to warrant a score of 2 rather than 1, but not severe enough to push it higher on the overstatement scale.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

As highlights, Perceiver IO outperforms a Transformer-based BERT baseline on the GLUE language benchmark despite removing input tokenization and achieves state-of-the-art performance on Sintel optical flow estimation with no explicit mechanisms for multiscale correspondence.

The review comment to be evaluated is:

The main strength of Perceiver IO is its high flexibility to various input domains. While keeping the main architecture, we can focus on engineering the input query preprocessing with domain-specific knowledge. Perceiver IO not only shows flexibility but also shows high performance. The results are comparable to or sometimes outperform previous domain-specific methods. For example

- Table 1 shows that Perceiver IO works well even without tokenization, while BERT + UTF-8 bytes show a significant performance drop (81.0 -> 71.5). It may show that Perceiver IO can perform well even without minimal domain knowledge (tokenization).
- Table 2 shows that the input query engineering can improve the target task (e.g., single-task query to 8 multi-task queries).
- Table 5 shows that the small input preprocessing engineering can improve the performance a lot. For example, Perceiver IO shows 82.1 top-1 accuracies for conv preprocessing while Perceiver IO with Fourier features shows 79.0.

I have minor concerns about ImageNet experiments.

- Although this paper is not aiming to state-of-the-art ImageNet performances, I think Perceiver IO is not yet comparable to state-of-the-art models on ImageNet image classification (as described in the last paragraph of Section 1). Perceiver IO still needs tremendous FLOPs compared to vision-specific models (Table 17). For example, ResNet shows 78.6 top-1 accuracy with 4.1 GFLOPs, while Perceiver IO Fourier shows 79.0 top-1 accuracy with 407 GFLOPs (99 times larger FLOPs). Stronger Perceiver IO (conv) shows 82.1 top-1 accuracy with 369 GFLOPs, where CaiT-M48 448 shows 86.5 top-1 accuracy and 329.6 GFLOPs.
- "Perceiver IO (pretrained)" and "Perceiver IO" in 2D Fourier features are different models. For example, "Perceiver IO (pretrained)" does not use weight sharing, and uses larger channel sizes. As a result, Perceiver IO (pretrained) has 4.38 times larger parameters compared to Perceiver IO (but half GFLOPs). The current text can make a reader mislead that "Perceiver IO (pretrained)" is the same model as "Perceiver IO", but the only difference is the JFT pretraining, which is not true.

> Unlike in the other ImageNet experiments, we do not share weights in the latent self-attention process modules, but use a 16-layer latent network with no weight sharing in depth. Unlike the other ImageNet experiments, the process-module MLPs use a hidden layer with 4× the number of channels (rather than 1× as on other ImageNet experiments)

To avoid confusion, I suggest replacing Table 5 with Table 17. Also, I would like to suggest to clarify that "Perceiver IO (pretrained)" and "Perceiver IO" are not the same model.

## Questions

- Perceiver IO shows better performances consistently compared to Perceiver IO baseline on GLUE (Table 13), Audio Set (Table 7), and ImageNet (Table 17). To my knowledge, there is no specific reason to the proposed output array cross-attention works better than the conventional classification head. I wonder what is the opinion by the authors what is the source of the improvements.
- In Table 5, I found that using 2D + maxpool preprocessing harms Perceiver performances (78.6 -> 77.4), but it helps Perceiver IO a lot (79.0 -> 82.1). As far as the reviewer understood, the main difference between Perceiver and Perceiver IO is the last decoding layer, not the input processing. I wonder what is the opinion of the authors why Perceiver and Perceiver IO show contradictory results on ImageNet Fourier features and conv preprocessing.
- I wonder why there is no Perceiver IO (conv) + pretraining result. Do the authors have any plan to add Perceiver IO (conv + pretrained)? (this is not a mandatory experiment for the rebuttal)

## Minor comments

- Many details are missing in the main text. To understand the method, the readers should read through the heavy appendix and the previous Perceiver paper. I fully understand that this is because of the page limitation, but this paper is somewhat hard to understand at first glance without any prior knowledge.
- In my opinion, it would be very helpful to readers if Table 8 and 9 are in the main text.
- pg 3. Typo -- unnecessary space *( "inducing points")*
- pg 29. Missing hyperlink (Tab. ??)
- In my opinion, it will be very interesting if there is one unified Perceiver model pretrained on various domain data. For example, one can train a Perceiver model trained by language and vision modalities, where each modality has different input and output arrays.

The evidence to be evaluated is:As hardware grew more powerful, neural nets led to breakthroughs in image understanding (Krizhevsky et al., 2012; Zeiler and Fergus, 2014; Szegedy et al., 2015) and interest intensified: autoregressive models that could process and complete samples of handwriting were developed (Graves, 2013), and new convolutional network designs led to good results in structured output spaces like semantic segmentation (Farabet et al., 2012; Long et al., 2015; Ronneberger et al., 2015), pose estimation (Toshev and Szegedy, 2014), detection (Sermanet et al., 2014), captioning (You et al., 2016), and optical flow (Fischer et al., 2015). At the same time, natural language applications research has made extensive progressive in capturing the structured nature of language, typically via autoregressive models (Collobert et al., 2011; Sutskever et al., 2014; Vaswani et al., 2017; Radford et al., 2019; Brown et al., 2020) or context prediction (Mikolov et al., 2013; Pennington et al., 2014; Devlin et al., 2019).

Several groups have demonstrated that Transformers (originally designed for language) can be used or adapted to non-language tasks (e.g. Chen et al. 2020; Lu et al. 2021), but the limited scalability of Transformers limits their usefulness as general-purpose architectures.

Our work uses attention over inputs and outputs of different sizes in part to produce an efficient attention architecture, and several other efficient attention architectures have been proposed, largely for language or small-scale problems (e.g. Xiong et al. 2021; Wang et al. 2020; Tay et al. 2021a; Beltagy et al. 2020 and see Tay et al. 2021b). The focus of our work is developing an architecture that is efficient and also performs well in many settings with a wide range of inputs and outputs.

Perceiver IO builds on this body of work to produce a general purpose architecture that can be easily and widely applied. The Perceiver IO architecture builds on the Perceiver (Jaegle et al., 2021), which achieved its cross-domain generality by assuming that its input is a simple 2D byte array: a set of elements (which might be pixels or patches in vision, characters or words in language, or some form of embedding, learned or otherwise), each described by a feature vector. The model then encodes information about the input array using a smaller number of latent feature vectors, using Transformer-style attention, followed by iterative processing and a final aggregation down to a category label. Rather than output a single category, Perceiver IO aims to have the same level of generality with respect to its _outputs_ as the Perceiver has with respect to its _inputs_: that is, it should produce arbitrary output arrays.

We first **encode** by applying an attention module that maps input arrays \(x\in\mathbb{R}^{M\times C}\) to arrays in a latent space \(z\in\mathbb{R}^{N\times D}\). We next **process** the latents \(z\) by applying a series of modules that take in and return arrays in this latent space. Finally, we **decode** by applying an attention module that maps latent arrays to output arrays \(y\in\mathbb{R}^{O\times E}\).

Following the design of the Perceiver, we implement each of the architecture's components using Transformer-style attention modules. Each of these modules applies a global query-key-value (QKV) attention operation followed by a multi-layer perceptron (MLP).

The Perceiver IO architecture builds on primitives similar to those in Transformers.

Even on domains like language where Transformers shine, preprocessing (e.g. tokenization) is often needed to scale beyond short input sequences. Perceiver IO uses attention non-homogeneously by mapping inputs to a latent space, processing in that latent space, and decoding to an output space. Perceiver IO has no quadratic dependence on the input or output size: encoder and decoder attention modules depend linearly on the input and output size (respectively), while latent attention is independent of both input and output sizes (Sec. E.2).

To probe the generality of Perceiver IO, we evaluate it on several domains including language understanding (Wikipedia+C4 masked language modeling), visual understanding (Sintel/KITTI optical flow and ImageNet classification), multi-modal (Kinetics autoencoding and AudioSet classification) & multi-task settings (multi-task GLUE), and symbolic representations for games (StarCraft II).

We first compare Perceiver IO to standard Transformers for language. Although Transformers were originally developed for language, their quadratic complexity makes them difficult to use on language inputs without tokenization, which typically shortens the length of input sequences by a factor of \(\sim\)4. But unlike Transformer-based models such as BERT (Devlin et al., 2019) or XLNet (Yang et al., 2019), Perceiver IO scales linearly with input length. Our experiments focus on showing that Perceiver IO performs as well as or better than Transformers for masked language modeling (MLM) while removing tokenization (which is hard to maintain, introduces engineering overhead, and adds needless complexity to language models (Bostrom and Durrett, 2020; Clark et al., 2022)).

We evaluate the quality of the learned representation on the GLUE benchmark (Wang et al., 2019) and report our results in Tab. 1. We find that at a given FLOPs budget, Perceiver IO trained without tokenization matches the performance of a strong Transformer-based model trained with SentencePiece tokenization (Sennrich et al., 2016; Kudo and Richardson, 2018).

**Finetuning.** We finetune Perceiver IO on the GLUE Benchmark Wang et al. (2019), reporting the best performance on the dev set for a fixed size sweep of finetuning hyperparameters.

We first observe that Perceiver IO applied on SentencePiece tokenized input sequences slightly outperforms a strong BERT baseline applied on the same inputs (81.2 vs 81.1). As a result of the reduced latent size of \(256\) we can train a much deeper network with 26 processing layers compared to BERT Base (12 layers) while maintaining a similar FLOPs budget.

Rather than using a fixed, handcrafted vocabulary, our model works directly with the raw byte inputs: we simply feed in and predict the UTF-8 bytes of the input string. Perceiver IO significantly outperforms a byte-level BERT baseline at the same FLOPs

Despite its greater depth, Perceiver IO is also faster than the Transformer-based BERT baselines in real wall-clock terms – by over a factor of 2 for the byte-based models – as shown in Tab. 1.

Our work scales to byte-level inputs without making any assumptions about the structure of the input, which allows it to be used beyond language as shown in the following sections.

We use multitask queries as described in Sec. 3.2 to finetune on all 8 GLUE tasks simultaneously using the UTF-8 byte model (results in Tab. 2).

We observe that our multitask approach outperforms single-task approaches and matches the approach that uses 8 task-specific input tokens.

State of the art algorithms, such as PWCNet (Sun et al., 2018), RAFT (Teed and Deng, 2020) or GMA (Jiang et al., 2021), use explicit machinery to ensure each of these steps is performed correctly even on out-of-domain data.

In contrast, we apply Perceiver IO to flow in a straightforward manner. We concatenate the frames along the channel dimension and extract a \(3\times 3\) patch around each pixel (leading to \(3\times 3\times 3\times 2=54\) values for each pixel).

We also test a version with convolutional downsampling and RAFT-style upsampling, which performs only slightly worse while improving computation time. It may seem counter-intuitive to append the images along the channel dimension, as large motions might result in pixels on entirely different objects being concatenated. However, this kind of operation isn't unprecedented: one of the earliest optical flow algorithms, Lucas-Kanade (Lucas & Kanade, 1981), makes explicit use of the _temporal_ image gradient, which is approximated by the difference in intensities at a given pixel across two frames. The algorithm uses the fact that the temporal gradient of the image approximates the spatial gradient times the spatial velocity, if lighting effects are ignored. The approximation is even better for image regions with very little texture.

On Sintel (Butler et al., 2012), our results are slightly better than RAFT on Sintel on the outperform PWCNet on KITTI (Meaze & Geiger, 2015). As far as we are aware, this result is state of the art on Sintel.final (GMA Jiang et al. (2021)) to PWCNet and RAFT baselines trained by the AutoFlow authors. On Sintel (Butler et al., 2012), our results are slightly better than RAFT on Sintel and outperform PWCNet on KITTI (Meaze & Geiger, 2015). As far as we are aware, this result is state of the art on Sintel.final (GMA Jiang et al. (2021)) propagate motion across image regions with little texture.

(1) on ImageNet, Perceiver IO surpasses 80% top-1 accuracy (84.5% top-1) without using 2D convolutions after pretraining on JFT.

On all domains but StarCraft II, we include experiments with several input configurations, ranging from no domain adaptation (e.g. tokenizer-free language, flow from raw pixels, ImageNet with no convolutional or patch-based preprocessing and fully learned position encodings) to moderate domain adaptation (e.g. SentencePiece language understanding, flow from conv+maxpool-processed images and with RAFT upsampling, ImageNet with conv+maxpool-preprocessing and 2D Fourier features). These results demonstrate the unprecedented generality of Perceiver IO, the simplicity that this architecture unlocks in handling a range of tasks, and its flexibility to work as part of a domain-adapted system.

While neither the Perceiver and Perceiver IO incorporate any 2D spatial structure architecturally, they use positional features that inject 2D spatial information (Sec. 3.2 and Appendix sec. D of Jaegle et al. 2021). By replacing these 2D position features with a fully learned position encoding as used on language, we can learn an image classification model that is given no privileged information about the structure of images.

Perceiver IO is constructed from GPT-2-style (Radford et al., 2019) Transformer attention modules, which consist of QKV attention followed by an MLP, along with linear projection layers to ensure inputs to and outputs from the QKV attention and MLP take on desired sizes.

The view of attention that we describe encompasses both cross-attention and self-attention, both of which are specific ways of using QKV-attention. Perceiver IO uses cross-attention for encoder and decoder attention modules and uses self-attention for the latent processing modules.

In other words, Perceiver IO has complexity linear in the size of the input and output arrays and it decouples the depth of the latent transformer from the input and output sizes. Both of these properties contribute to Perceiver IO's efficiency: while many proposals for efficient attention modules or architectures include linear or sub-quadratic scaling with input/output size, Perceiver IO is unusual in also decoupling depth from input/output size (without requiring domain-specific strategies like 2D convolution).

One application of Perceiver IO is byte-level language processing, which has concurrently been addressed by several other groups. Clark et al. (2022) trains models on Unicode code points and shows results competitive with subword-based models on a multilingual question answering dataset. Tay et al. (2022) trains on UTF-8 bytes directly by introducing a hand-designed module that is trained end-to-end to perform subword tokenization and produces results on-par with and sometimes better than subword-based models. Xue et al. (2022) trains encoder-decoder T5 models on UTF-8 bytes directly and shows that making the encoder 3x deeper than the decoder leads to comparable performance with subword baselines.

Perceiver IO is somewhat slower on traditional GPUs than our baseline RAFT model, but we find that the trend reverses on TPUs, which is the target architecture for our work.

Our most expensive model achieves approximately 0.8 frames/sec on a 2017 TITAN Xp, and our lightweight model (with conv downsampling and RAFT-style upsampling) achieves 3.3 frames/sec, which is not far from the 10 frames per second reported for RAFT (Teed & Deng, 2020). On the publicly-available TPU v3, however, our most expensive model achieves 4.4 frames/sec on a single TPU core, and 17.8 frames/sec for the lightweight model. An efficient Tensorflow implementation of RAFT (Sun et al., 2020) (received courtesy of the authors) achieves only 1.6 frames/sec on the same hardware.

We see that the algorithm is capable of dealing with heavy occlusion, and can propagate optical flow across large regions with very little texture. The network can also deal with very large motions and very small objects. Finally, to verify that Perceiver IO performs well on real-world data despite being trained only on synthetic imagery, we applied it to a small number (roughly 10) real videos taken from Getty images (www.gettyimages.com). Perceiver IO typically performs very well out-of-domain, although some failure cases remain: for instance, shadows tend to be interpreted as objects (Autoflow contains no shadows), and large regions with compression artifacts but no other texture may result in hallucinated flow (Autoflow contains no video compression artifacts). We include three challenging examples in the supplementary zip file, each of which depict complex motion and small objects. Perceiver IO can pick up on remarkably small objects such as the water droplets thrown by the girl's shoe in pigeon.mp4 or the confetti in thai_dance.mp4.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I can provide the following assessment:

Summary: The claim has two main parts: (1) Perceiver IO outperforming BERT on GLUE without tokenization and (2) achieving state-of-the-art on Sintel optical flow. The evidence supports that Perceiver IO matches/slightly outperforms BERT baseline while removing tokenization (81.2 vs 81.1 with tokenization, and significantly better than byte-level BERT). For Sintel, while results are strong, the evidence is more nuanced - showing "slightly better than RAFT" rather than definitively state-of-the-art. The review comment provides important context about performance tradeoffs, particularly regarding computational costs.

<score>2</score>

Justification:
1. Language/GLUE claim:
- Evidence confirms Perceiver IO performs well without tokenization (matches/slightly exceeds BERT)
- Direct quote: "Perceiver IO significantly outperforms a byte-level BERT baseline at the same FLOPs"
- Review validates this, noting "Perceiver IO works well even without tokenization"

2. Optical flow claim:
- Slightly overstated as "state-of-the-art" when evidence shows more modest "slightly better than RAFT"
- Quote: "our results are slightly better than RAFT on Sintel"
- Shows good performance but with tradeoffs in computational cost
- Review provides important context about computational efficiency tradeoffs

While there is some slight overstatement regarding the optical flow results, the core claims are generally well-supported by the evidence, with appropriate qualifications about tradeoffs. The score of 2 reflects that while not perfectly stated, the claim remains reasonably grounded in the evidence.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

As highlights, Perceiver IO outperforms a Transformer-based BERT baseline on the GLUE language benchmark despite removing input tokenization and achieves state-of-the-art performance on Sintel optical flow estimation with no explicit mechanisms for multiscale correspondence.

The review comment to be evaluated is:

Pros:

1. This paper introduced a generic architecture for coping with different tasks with various input and output lengths. It is different from Transformer architecture and Perceiver architecture in that it uses a latent process to compress the larger number of input tokens into a hidden space, and then decode the output from this hidden space. This way it can decouple the inputs and outputs which has the potential to significantly reduce the cost brought by a large number of input and output tokens.

2. The authors presented a thorough study of Perceiver IO across different application scenarios, such as language, vision, multi-modality. Extensive experiments demonstrate that the proposed method can achieve comparable or even better performance than the baselines. To facilitate the adaptation, the authors suggested a number of good ways to convert the inputs/outputs into certain formats.

Cons:

1. The authors should list the sizes of inputs, latent, and output arrays for each of the downstream tasks to give the audience a better sense of the complexity. For some of the tasks, such as ImageNet classification, the authors should also report the FLOPs in comparisons with prior arts, e.g., ViTs.

2. The main merit of Perceiver IO compared with Perceiver is that it further introduces a cross-attention-based output decoder on top of the latent arrays. Though Perceiver was originally proposed for classification tasks, it can be also used to cope with structured outputs. For some of the tasks except for ImageNet classification listed in this paper, I am curious whether Perceiver can be applied and how it performs compared with Perceiver IO.

3. Perceiver IO is proposed as a generic architecture for various tasks by modeling them as a read-process-write process. One question is that whether we should reformulate all these tasks as the way shown in this paper. This paper does show some encouraging results on various tasks compared with the baselines. However, they are only compared with baseline methods and still underperforms established methods in specific domains. As we know, different tasks can still benefit a lot from the specific domain knowledge (e.g., 2D spatial for image recognition). Actually, we can also think of the Transformer encoder as a generic architecture in numerous domains. Then, do we need to build up a higher-level generic architecture upon the Transformer encoder?

4. A good part of unifying architectures for various tasks is that we can train the same set of parameters using different tasks. In this paper, the authors demonstrated the effectiveness of Perceiver IO across different settings, I am wondering whether such a generic architecture can be used for multi-task training so that it can leverage the training data from different tasks.

The evidence to be evaluated is:As hardware grew more powerful, neural nets led to breakthroughs in image understanding (Krizhevsky et al., 2012; Zeiler and Fergus, 2014; Szegedy et al., 2015) and interest intensified: autoregressive models that could process and complete samples of handwriting were developed (Graves, 2013), and new convolutional network designs led to good results in structured output spaces like semantic segmentation (Farabet et al., 2012; Long et al., 2015; Ronneberger et al., 2015), pose estimation (Toshev and Szegedy, 2014), detection (Sermanet et al., 2014), captioning (You et al., 2016), and optical flow (Fischer et al., 2015). At the same time, natural language applications research has made extensive progressive in capturing the structured nature of language, typically via autoregressive models (Collobert et al., 2011; Sutskever et al., 2014; Vaswani et al., 2017; Radford et al., 2019; Brown et al., 2020) or context prediction (Mikolov et al., 2013; Pennington et al., 2014; Devlin et al., 2019).

Several groups have demonstrated that Transformers (originally designed for language) can be used or adapted to non-language tasks (e.g. Chen et al. 2020; Lu et al. 2021), but the limited scalability of Transformers limits their usefulness as general-purpose architectures.

Our work uses attention over inputs and outputs of different sizes in part to produce an efficient attention architecture, and several other efficient attention architectures have been proposed, largely for language or small-scale problems (e.g. Xiong et al. 2021; Wang et al. 2020; Tay et al. 2021a; Beltagy et al. 2020 and see Tay et al. 2021b). The focus of our work is developing an architecture that is efficient and also performs well in many settings with a wide range of inputs and outputs.

Perceiver IO builds on this body of work to produce a general purpose architecture that can be easily and widely applied. The Perceiver IO architecture builds on the Perceiver (Jaegle et al., 2021), which achieved its cross-domain generality by assuming that its input is a simple 2D byte array: a set of elements (which might be pixels or patches in vision, characters or words in language, or some form of embedding, learned or otherwise), each described by a feature vector. The model then encodes information about the input array using a smaller number of latent feature vectors, using Transformer-style attention, followed by iterative processing and a final aggregation down to a category label. Rather than output a single category, Perceiver IO aims to have the same level of generality with respect to its _outputs_ as the Perceiver has with respect to its _inputs_: that is, it should produce arbitrary output arrays.

We first **encode** by applying an attention module that maps input arrays \(x\in\mathbb{R}^{M\times C}\) to arrays in a latent space \(z\in\mathbb{R}^{N\times D}\). We next **process** the latents \(z\) by applying a series of modules that take in and return arrays in this latent space. Finally, we **decode** by applying an attention module that maps latent arrays to output arrays \(y\in\mathbb{R}^{O\times E}\).

Following the design of the Perceiver, we implement each of the architecture's components using Transformer-style attention modules. Each of these modules applies a global query-key-value (QKV) attention operation followed by a multi-layer perceptron (MLP).

The Perceiver IO architecture builds on primitives similar to those in Transformers.

Even on domains like language where Transformers shine, preprocessing (e.g. tokenization) is often needed to scale beyond short input sequences. Perceiver IO uses attention non-homogeneously by mapping inputs to a latent space, processing in that latent space, and decoding to an output space. Perceiver IO has no quadratic dependence on the input or output size: encoder and decoder attention modules depend linearly on the input and output size (respectively), while latent attention is independent of both input and output sizes (Sec. E.2).

To probe the generality of Perceiver IO, we evaluate it on several domains including language understanding (Wikipedia+C4 masked language modeling), visual understanding (Sintel/KITTI optical flow and ImageNet classification), multi-modal (Kinetics autoencoding and AudioSet classification) & multi-task settings (multi-task GLUE), and symbolic representations for games (StarCraft II).

We first compare Perceiver IO to standard Transformers for language. Although Transformers were originally developed for language, their quadratic complexity makes them difficult to use on language inputs without tokenization, which typically shortens the length of input sequences by a factor of \(\sim\)4. But unlike Transformer-based models such as BERT (Devlin et al., 2019) or XLNet (Yang et al., 2019), Perceiver IO scales linearly with input length. Our experiments focus on showing that Perceiver IO performs as well as or better than Transformers for masked language modeling (MLM) while removing tokenization (which is hard to maintain, introduces engineering overhead, and adds needless complexity to language models (Bostrom and Durrett, 2020; Clark et al., 2022)).

We evaluate the quality of the learned representation on the GLUE benchmark (Wang et al., 2019) and report our results in Tab. 1. We find that at a given FLOPs budget, Perceiver IO trained without tokenization matches the performance of a strong Transformer-based model trained with SentencePiece tokenization (Sennrich et al., 2016; Kudo and Richardson, 2018).

**Finetuning.** We finetune Perceiver IO on the GLUE Benchmark Wang et al. (2019), reporting the best performance on the dev set for a fixed size sweep of finetuning hyperparameters.

We first observe that Perceiver IO applied on SentencePiece tokenized input sequences slightly outperforms a strong BERT baseline applied on the same inputs (81.2 vs 81.1). As a result of the reduced latent size of \(256\) we can train a much deeper network with 26 processing layers compared to BERT Base (12 layers) while maintaining a similar FLOPs budget.

Rather than using a fixed, handcrafted vocabulary, our model works directly with the raw byte inputs: we simply feed in and predict the UTF-8 bytes of the input string. Perceiver IO significantly outperforms a byte-level BERT baseline at the same FLOPs

Despite its greater depth, Perceiver IO is also faster than the Transformer-based BERT baselines in real wall-clock terms – by over a factor of 2 for the byte-based models – as shown in Tab. 1.

Our work scales to byte-level inputs without making any assumptions about the structure of the input, which allows it to be used beyond language as shown in the following sections.

We use multitask queries as described in Sec. 3.2 to finetune on all 8 GLUE tasks simultaneously using the UTF-8 byte model (results in Tab. 2).

We observe that our multitask approach outperforms single-task approaches and matches the approach that uses 8 task-specific input tokens.

State of the art algorithms, such as PWCNet (Sun et al., 2018), RAFT (Teed and Deng, 2020) or GMA (Jiang et al., 2021), use explicit machinery to ensure each of these steps is performed correctly even on out-of-domain data.

In contrast, we apply Perceiver IO to flow in a straightforward manner. We concatenate the frames along the channel dimension and extract a \(3\times 3\) patch around each pixel (leading to \(3\times 3\times 3\times 2=54\) values for each pixel).

We also test a version with convolutional downsampling and RAFT-style upsampling, which performs only slightly worse while improving computation time. It may seem counter-intuitive to append the images along the channel dimension, as large motions might result in pixels on entirely different objects being concatenated. However, this kind of operation isn't unprecedented: one of the earliest optical flow algorithms, Lucas-Kanade (Lucas & Kanade, 1981), makes explicit use of the _temporal_ image gradient, which is approximated by the difference in intensities at a given pixel across two frames. The algorithm uses the fact that the temporal gradient of the image approximates the spatial gradient times the spatial velocity, if lighting effects are ignored. The approximation is even better for image regions with very little texture.

On Sintel (Butler et al., 2012), our results are slightly better than RAFT on Sintel on the outperform PWCNet on KITTI (Meaze & Geiger, 2015). As far as we are aware, this result is state of the art on Sintel.final (GMA Jiang et al. (2021)) to PWCNet and RAFT baselines trained by the AutoFlow authors. On Sintel (Butler et al., 2012), our results are slightly better than RAFT on Sintel and outperform PWCNet on KITTI (Meaze & Geiger, 2015). As far as we are aware, this result is state of the art on Sintel.final (GMA Jiang et al. (2021)) propagate motion across image regions with little texture.

(1) on ImageNet, Perceiver IO surpasses 80% top-1 accuracy (84.5% top-1) without using 2D convolutions after pretraining on JFT.

On all domains but StarCraft II, we include experiments with several input configurations, ranging from no domain adaptation (e.g. tokenizer-free language, flow from raw pixels, ImageNet with no convolutional or patch-based preprocessing and fully learned position encodings) to moderate domain adaptation (e.g. SentencePiece language understanding, flow from conv+maxpool-processed images and with RAFT upsampling, ImageNet with conv+maxpool-preprocessing and 2D Fourier features). These results demonstrate the unprecedented generality of Perceiver IO, the simplicity that this architecture unlocks in handling a range of tasks, and its flexibility to work as part of a domain-adapted system.

While neither the Perceiver and Perceiver IO incorporate any 2D spatial structure architecturally, they use positional features that inject 2D spatial information (Sec. 3.2 and Appendix sec. D of Jaegle et al. 2021). By replacing these 2D position features with a fully learned position encoding as used on language, we can learn an image classification model that is given no privileged information about the structure of images.

Perceiver IO is constructed from GPT-2-style (Radford et al., 2019) Transformer attention modules, which consist of QKV attention followed by an MLP, along with linear projection layers to ensure inputs to and outputs from the QKV attention and MLP take on desired sizes.

The view of attention that we describe encompasses both cross-attention and self-attention, both of which are specific ways of using QKV-attention. Perceiver IO uses cross-attention for encoder and decoder attention modules and uses self-attention for the latent processing modules.

In other words, Perceiver IO has complexity linear in the size of the input and output arrays and it decouples the depth of the latent transformer from the input and output sizes. Both of these properties contribute to Perceiver IO's efficiency: while many proposals for efficient attention modules or architectures include linear or sub-quadratic scaling with input/output size, Perceiver IO is unusual in also decoupling depth from input/output size (without requiring domain-specific strategies like 2D convolution).

One application of Perceiver IO is byte-level language processing, which has concurrently been addressed by several other groups. Clark et al. (2022) trains models on Unicode code points and shows results competitive with subword-based models on a multilingual question answering dataset. Tay et al. (2022) trains on UTF-8 bytes directly by introducing a hand-designed module that is trained end-to-end to perform subword tokenization and produces results on-par with and sometimes better than subword-based models. Xue et al. (2022) trains encoder-decoder T5 models on UTF-8 bytes directly and shows that making the encoder 3x deeper than the decoder leads to comparable performance with subword baselines.

Perceiver IO is somewhat slower on traditional GPUs than our baseline RAFT model, but we find that the trend reverses on TPUs, which is the target architecture for our work.

Our most expensive model achieves approximately 0.8 frames/sec on a 2017 TITAN Xp, and our lightweight model (with conv downsampling and RAFT-style upsampling) achieves 3.3 frames/sec, which is not far from the 10 frames per second reported for RAFT (Teed & Deng, 2020). On the publicly-available TPU v3, however, our most expensive model achieves 4.4 frames/sec on a single TPU core, and 17.8 frames/sec for the lightweight model. An efficient Tensorflow implementation of RAFT (Sun et al., 2020) (received courtesy of the authors) achieves only 1.6 frames/sec on the same hardware.

We see that the algorithm is capable of dealing with heavy occlusion, and can propagate optical flow across large regions with very little texture. The network can also deal with very large motions and very small objects. Finally, to verify that Perceiver IO performs well on real-world data despite being trained only on synthetic imagery, we applied it to a small number (roughly 10) real videos taken from Getty images (www.gettyimages.com). Perceiver IO typically performs very well out-of-domain, although some failure cases remain: for instance, shadows tend to be interpreted as objects (Autoflow contains no shadows), and large regions with compression artifacts but no other texture may result in hallucinated flow (Autoflow contains no video compression artifacts). We include three challenging examples in the supplementary zip file, each of which depict complex motion and small objects. Perceiver IO can pick up on remarkably small objects such as the water droplets thrown by the girl's shoe in pigeon.mp4 or the confetti in thai_dance.mp4.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I'll assess the claim: "Perceiver IO outperforms a Transformer-based BERT baseline on the GLUE language benchmark despite removing input tokenization and achieves state-of-the-art performance on Sintel optical flow estimation with no explicit mechanisms for multiscale correspondence."

Summary:
The claim has two main parts:
1. GLUE benchmark performance: The evidence shows Perceiver IO slightly outperforms BERT baseline (81.2 vs 81.1) with tokenization, and significantly outperforms byte-level BERT without tokenization.
2. Sintel optical flow: While the evidence indicates strong performance "slightly better than RAFT on Sintel," and claims state-of-the-art results, this is somewhat qualified by implementation-specific details and hardware dependencies.

The claim is mostly well-supported but slightly overstated in its certainty about "state-of-the-art" performance, given the nuanced hardware-dependent performance trade-offs discussed in the evidence.

<score>2</score>

Justification:
1. For GLUE performance:
- Evidence directly supports the claim with specific numbers
- Shows success both with and without tokenization
- Clear comparative metrics against BERT baseline

2. For Sintel performance:
- Results are strong but more nuanced than simply "state-of-the-art"
- Performance varies by hardware (slower on GPUs, faster on TPUs)
- The "no explicit mechanisms" part is accurate as the paper describes a straightforward application

The score of 2 reflects that while the claim is generally accurate, it slightly oversimplifies the optical flow results' context-dependent nature.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

As highlights, Perceiver IO outperforms a Transformer-based BERT baseline on the GLUE language benchmark despite removing input tokenization and achieves state-of-the-art performance on Sintel optical flow estimation with no explicit mechanisms for multiscale correspondence.

The review comment to be evaluated is:

#### Strength
- The idea of perceiver IO is novel and solid -- a general architecture capable of handling general-purpose inputs and outputs across different tasks and modalities. This is very promising to simplify the construction of highly tuned task-specific neural pipelines and improve the multimodal and multi-task problems.

- Each component in perceiver IO is necessary and well defined for the proposed tasks. 2D byte array input enables the task agnostic input for different modalities. Perceiver network that scales linearly with the size of input and output, and non-auto regressive decoding make the decoding to raw output possible.

- The proposed architecture is tested on massive experiments including language understanding tasks, optical flow, video audio class autoencoding, image classification, and starcraft II and achieves superior performance. Each task is supported with a detailed ablation study to shed light on future research.

#### Weakness
- In Table 1, the WNLI task is excluded from the GLUE benchmark, I wonder what is the reason this task is removed?

- The non-autoregressive decoding enables fast decoding for large outputs. However, for tasks that require more information on prior decoding context (machine translation or image generation tasks), does the proposed model can still perform well on those tasks?

- Although perceiver IO removes the task-specific pre-processing (tokenization, patch embedding, etc.), the model still requires huge engineering efforts to adapt the model for different tasks. For example, the model hyper-parameters are quite different for different tasks.

- One huge benefit of perceiver IO is to train different tasks together and explore the transfer between different tasks/modalities, which is not explored in this paper.

The evidence to be evaluated is:As hardware grew more powerful, neural nets led to breakthroughs in image understanding (Krizhevsky et al., 2012; Zeiler and Fergus, 2014; Szegedy et al., 2015) and interest intensified: autoregressive models that could process and complete samples of handwriting were developed (Graves, 2013), and new convolutional network designs led to good results in structured output spaces like semantic segmentation (Farabet et al., 2012; Long et al., 2015; Ronneberger et al., 2015), pose estimation (Toshev and Szegedy, 2014), detection (Sermanet et al., 2014), captioning (You et al., 2016), and optical flow (Fischer et al., 2015). At the same time, natural language applications research has made extensive progressive in capturing the structured nature of language, typically via autoregressive models (Collobert et al., 2011; Sutskever et al., 2014; Vaswani et al., 2017; Radford et al., 2019; Brown et al., 2020) or context prediction (Mikolov et al., 2013; Pennington et al., 2014; Devlin et al., 2019).

Several groups have demonstrated that Transformers (originally designed for language) can be used or adapted to non-language tasks (e.g. Chen et al. 2020; Lu et al. 2021), but the limited scalability of Transformers limits their usefulness as general-purpose architectures.

Our work uses attention over inputs and outputs of different sizes in part to produce an efficient attention architecture, and several other efficient attention architectures have been proposed, largely for language or small-scale problems (e.g. Xiong et al. 2021; Wang et al. 2020; Tay et al. 2021a; Beltagy et al. 2020 and see Tay et al. 2021b). The focus of our work is developing an architecture that is efficient and also performs well in many settings with a wide range of inputs and outputs.

Perceiver IO builds on this body of work to produce a general purpose architecture that can be easily and widely applied. The Perceiver IO architecture builds on the Perceiver (Jaegle et al., 2021), which achieved its cross-domain generality by assuming that its input is a simple 2D byte array: a set of elements (which might be pixels or patches in vision, characters or words in language, or some form of embedding, learned or otherwise), each described by a feature vector. The model then encodes information about the input array using a smaller number of latent feature vectors, using Transformer-style attention, followed by iterative processing and a final aggregation down to a category label. Rather than output a single category, Perceiver IO aims to have the same level of generality with respect to its _outputs_ as the Perceiver has with respect to its _inputs_: that is, it should produce arbitrary output arrays.

We first **encode** by applying an attention module that maps input arrays \(x\in\mathbb{R}^{M\times C}\) to arrays in a latent space \(z\in\mathbb{R}^{N\times D}\). We next **process** the latents \(z\) by applying a series of modules that take in and return arrays in this latent space. Finally, we **decode** by applying an attention module that maps latent arrays to output arrays \(y\in\mathbb{R}^{O\times E}\).

Following the design of the Perceiver, we implement each of the architecture's components using Transformer-style attention modules. Each of these modules applies a global query-key-value (QKV) attention operation followed by a multi-layer perceptron (MLP).

The Perceiver IO architecture builds on primitives similar to those in Transformers.

Even on domains like language where Transformers shine, preprocessing (e.g. tokenization) is often needed to scale beyond short input sequences. Perceiver IO uses attention non-homogeneously by mapping inputs to a latent space, processing in that latent space, and decoding to an output space. Perceiver IO has no quadratic dependence on the input or output size: encoder and decoder attention modules depend linearly on the input and output size (respectively), while latent attention is independent of both input and output sizes (Sec. E.2).

To probe the generality of Perceiver IO, we evaluate it on several domains including language understanding (Wikipedia+C4 masked language modeling), visual understanding (Sintel/KITTI optical flow and ImageNet classification), multi-modal (Kinetics autoencoding and AudioSet classification) & multi-task settings (multi-task GLUE), and symbolic representations for games (StarCraft II).

We first compare Perceiver IO to standard Transformers for language. Although Transformers were originally developed for language, their quadratic complexity makes them difficult to use on language inputs without tokenization, which typically shortens the length of input sequences by a factor of \(\sim\)4. But unlike Transformer-based models such as BERT (Devlin et al., 2019) or XLNet (Yang et al., 2019), Perceiver IO scales linearly with input length. Our experiments focus on showing that Perceiver IO performs as well as or better than Transformers for masked language modeling (MLM) while removing tokenization (which is hard to maintain, introduces engineering overhead, and adds needless complexity to language models (Bostrom and Durrett, 2020; Clark et al., 2022)).

We evaluate the quality of the learned representation on the GLUE benchmark (Wang et al., 2019) and report our results in Tab. 1. We find that at a given FLOPs budget, Perceiver IO trained without tokenization matches the performance of a strong Transformer-based model trained with SentencePiece tokenization (Sennrich et al., 2016; Kudo and Richardson, 2018).

**Finetuning.** We finetune Perceiver IO on the GLUE Benchmark Wang et al. (2019), reporting the best performance on the dev set for a fixed size sweep of finetuning hyperparameters.

We first observe that Perceiver IO applied on SentencePiece tokenized input sequences slightly outperforms a strong BERT baseline applied on the same inputs (81.2 vs 81.1). As a result of the reduced latent size of \(256\) we can train a much deeper network with 26 processing layers compared to BERT Base (12 layers) while maintaining a similar FLOPs budget.

Rather than using a fixed, handcrafted vocabulary, our model works directly with the raw byte inputs: we simply feed in and predict the UTF-8 bytes of the input string. Perceiver IO significantly outperforms a byte-level BERT baseline at the same FLOPs

Despite its greater depth, Perceiver IO is also faster than the Transformer-based BERT baselines in real wall-clock terms – by over a factor of 2 for the byte-based models – as shown in Tab. 1.

Our work scales to byte-level inputs without making any assumptions about the structure of the input, which allows it to be used beyond language as shown in the following sections.

We use multitask queries as described in Sec. 3.2 to finetune on all 8 GLUE tasks simultaneously using the UTF-8 byte model (results in Tab. 2).

We observe that our multitask approach outperforms single-task approaches and matches the approach that uses 8 task-specific input tokens.

State of the art algorithms, such as PWCNet (Sun et al., 2018), RAFT (Teed and Deng, 2020) or GMA (Jiang et al., 2021), use explicit machinery to ensure each of these steps is performed correctly even on out-of-domain data.

In contrast, we apply Perceiver IO to flow in a straightforward manner. We concatenate the frames along the channel dimension and extract a \(3\times 3\) patch around each pixel (leading to \(3\times 3\times 3\times 2=54\) values for each pixel).

We also test a version with convolutional downsampling and RAFT-style upsampling, which performs only slightly worse while improving computation time. It may seem counter-intuitive to append the images along the channel dimension, as large motions might result in pixels on entirely different objects being concatenated. However, this kind of operation isn't unprecedented: one of the earliest optical flow algorithms, Lucas-Kanade (Lucas & Kanade, 1981), makes explicit use of the _temporal_ image gradient, which is approximated by the difference in intensities at a given pixel across two frames. The algorithm uses the fact that the temporal gradient of the image approximates the spatial gradient times the spatial velocity, if lighting effects are ignored. The approximation is even better for image regions with very little texture.

On Sintel (Butler et al., 2012), our results are slightly better than RAFT on Sintel on the outperform PWCNet on KITTI (Meaze & Geiger, 2015). As far as we are aware, this result is state of the art on Sintel.final (GMA Jiang et al. (2021)) to PWCNet and RAFT baselines trained by the AutoFlow authors. On Sintel (Butler et al., 2012), our results are slightly better than RAFT on Sintel and outperform PWCNet on KITTI (Meaze & Geiger, 2015). As far as we are aware, this result is state of the art on Sintel.final (GMA Jiang et al. (2021)) propagate motion across image regions with little texture.

(1) on ImageNet, Perceiver IO surpasses 80% top-1 accuracy (84.5% top-1) without using 2D convolutions after pretraining on JFT.

On all domains but StarCraft II, we include experiments with several input configurations, ranging from no domain adaptation (e.g. tokenizer-free language, flow from raw pixels, ImageNet with no convolutional or patch-based preprocessing and fully learned position encodings) to moderate domain adaptation (e.g. SentencePiece language understanding, flow from conv+maxpool-processed images and with RAFT upsampling, ImageNet with conv+maxpool-preprocessing and 2D Fourier features). These results demonstrate the unprecedented generality of Perceiver IO, the simplicity that this architecture unlocks in handling a range of tasks, and its flexibility to work as part of a domain-adapted system.

While neither the Perceiver and Perceiver IO incorporate any 2D spatial structure architecturally, they use positional features that inject 2D spatial information (Sec. 3.2 and Appendix sec. D of Jaegle et al. 2021). By replacing these 2D position features with a fully learned position encoding as used on language, we can learn an image classification model that is given no privileged information about the structure of images.

Perceiver IO is constructed from GPT-2-style (Radford et al., 2019) Transformer attention modules, which consist of QKV attention followed by an MLP, along with linear projection layers to ensure inputs to and outputs from the QKV attention and MLP take on desired sizes.

The view of attention that we describe encompasses both cross-attention and self-attention, both of which are specific ways of using QKV-attention. Perceiver IO uses cross-attention for encoder and decoder attention modules and uses self-attention for the latent processing modules.

In other words, Perceiver IO has complexity linear in the size of the input and output arrays and it decouples the depth of the latent transformer from the input and output sizes. Both of these properties contribute to Perceiver IO's efficiency: while many proposals for efficient attention modules or architectures include linear or sub-quadratic scaling with input/output size, Perceiver IO is unusual in also decoupling depth from input/output size (without requiring domain-specific strategies like 2D convolution).

One application of Perceiver IO is byte-level language processing, which has concurrently been addressed by several other groups. Clark et al. (2022) trains models on Unicode code points and shows results competitive with subword-based models on a multilingual question answering dataset. Tay et al. (2022) trains on UTF-8 bytes directly by introducing a hand-designed module that is trained end-to-end to perform subword tokenization and produces results on-par with and sometimes better than subword-based models. Xue et al. (2022) trains encoder-decoder T5 models on UTF-8 bytes directly and shows that making the encoder 3x deeper than the decoder leads to comparable performance with subword baselines.

Perceiver IO is somewhat slower on traditional GPUs than our baseline RAFT model, but we find that the trend reverses on TPUs, which is the target architecture for our work.

Our most expensive model achieves approximately 0.8 frames/sec on a 2017 TITAN Xp, and our lightweight model (with conv downsampling and RAFT-style upsampling) achieves 3.3 frames/sec, which is not far from the 10 frames per second reported for RAFT (Teed & Deng, 2020). On the publicly-available TPU v3, however, our most expensive model achieves 4.4 frames/sec on a single TPU core, and 17.8 frames/sec for the lightweight model. An efficient Tensorflow implementation of RAFT (Sun et al., 2020) (received courtesy of the authors) achieves only 1.6 frames/sec on the same hardware.

We see that the algorithm is capable of dealing with heavy occlusion, and can propagate optical flow across large regions with very little texture. The network can also deal with very large motions and very small objects. Finally, to verify that Perceiver IO performs well on real-world data despite being trained only on synthetic imagery, we applied it to a small number (roughly 10) real videos taken from Getty images (www.gettyimages.com). Perceiver IO typically performs very well out-of-domain, although some failure cases remain: for instance, shadows tend to be interpreted as objects (Autoflow contains no shadows), and large regions with compression artifacts but no other texture may result in hallucinated flow (Autoflow contains no video compression artifacts). We include three challenging examples in the supplementary zip file, each of which depict complex motion and small objects. Perceiver IO can pick up on remarkably small objects such as the water droplets thrown by the girl's shoe in pigeon.mp4 or the confetti in thai_dance.mp4.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "Perceiver IO outperforms a Transformer-based BERT baseline on the GLUE language benchmark despite removing input tokenization and achieves state-of-the-art performance on Sintel optical flow estimation with no explicit mechanisms for multiscale correspondence."

Analysis:
1. GLUE Benchmark claim:
- Evidence shows Perceiver IO outperforms BERT baseline when using tokenization (81.2 vs 81.1)
- With byte-level (no tokenization) input, it "significantly outperforms a byte-level BERT baseline"
- However, the exact performance numbers for byte-level comparison aren't provided

2. Sintel optical flow claim:
- Results are described as "slightly better than RAFT on Sintel"
- Claimed as "state of the art on Sintel.final"
- While it doesn't use explicit multiscale mechanisms, it does use some domain adaptations (conv downsampling and RAFT-style upsampling in some versions)

The claim is slightly overstated (score 2) because:
1. While the GLUE performance claim is supported, the magnitude of improvement isn't quantified for the tokenization-free case
2. The optical flow performance is presented more definitively in the claim than in the evidence, which shows more nuanced results and mentions domain adaptations

<score>2</score>