Question:
You are a highly experienced, conscientious, and fair academic reviewer, please help me review this paper. The review should be organized into nine sections:
1. Summary: A summary of the paper in 100-150 words.
2. Strengths/Weaknesses/Questions: The Strengths/Weaknesses/Questions of paper, which should be listed in bullet points, with each point supported by specific examples from the article where possible.
3. Soundness/Contribution/Presentation: Rate the paper's Soundness/Contribution/Presentation, and match this score to the corresponding description from the list below and provide the result. The possible scores and their descriptions are:
    1 poor
    2 fair
    3 good
    4 excellent
4. Rating: Give this paper an appropriate rating, match this rating to the corresponding description from the list below and provide the result. The possible Ratings and their descriptions are:
    1 strong reject
    2 reject, significant issues present
    3 reject, not good enough
    4 possibly reject, but has redeeming facets
    5 marginally below the acceptance threshold
    6 marginally above the acceptance threshold
    7 accept, but needs minor improvements
    8 accept, good paper
    9 strong accept, excellent work
    10 strong accept, should be highlighted at the conference
5. Paper Decision: It must include the Decision itself(Accept or Reject) and the reasons for this decision, based on the criteria of originality, methodological soundness, significance of results, and clarity and logic of presentation.

Here is the template for a review format, you must follow this format to output your review result:
**Summary:**
Summary content

**Strengths:**
- Strength 1
- Strength 2
- ...

**Weaknesses:**
- Weakness 1
- Weakness 2
- ...

**Questions:**
- Question 1
- Question 2
- ...

**Soundness:**
Soundness result

**Presentation:**
Presentation result

**Contribution:**
Contribution result

**Rating:**
Rating result

**Paper Decision:**
- Decision: Accept/Reject
- Reasons: reasons content

Please ensure your feedback is objective and constructive. The paper is as follows:

# Im-Promptu: In-Context Composition

from Image Prompts

Bhishma Dedhia

Department of Electrical and Computer Engineering, Princeton University

Michael Chang

Department of Computer Science, University of California Berkeley

Jake C. Snell

Department of Computer Science, Princeton University

Thomas L. Griffiths

Niraj K. Jha

Department of Psychology, Princeton University

###### Abstract

Large language models are few-shot learners that can solve diverse tasks from a handful of demonstrations. This implicit understanding of tasks suggests that the attention mechanisms over word tokens may play a role in analogical reasoning. In this work, we investigate whether analogical reasoning can enable in-context composition over composable elements of visual stimuli. First, we introduce a suite of three benchmarks to test the generalization properties of a visual in-context learner. We formalize the notion of an analogy-based in-context learner and use it to design a meta-learning framework called _Im-Promptu_. Whereas the requisite token granularity for language is well established, the appropriate compositional granularity for enabling in-context generalization in visual stimuli is usually unspecified. To this end, we use _Im-Promptu_ to train multiple agents with different levels of compositionality, including vector representations, patch representations, and object slots. Our experiments reveal tradeoffs between extrapolation abilities and the degree of compositionality, with non-compositional representations extending learned composition rules to unseen domains but performing poorly on combinatorial tasks. Patch-based representations require patches to contain entire objects for robust extrapolation. At the same time, object-centric tokenizers coupled with a cross-attention module generate consistent and high-fidelity solutions, with these inductive biases being particularly crucial for compositional generalization. Lastly, we demonstrate a use case of _Im-Promptu_ as an intuitive programming interface for image generation.

## 1 Introduction

No thought can be formed that isn't informed by the past; or, more precisely, we think only thanks to analogies that link our present to our past.

Humans represent complex concepts by combining and organizing simpler concepts [2, 3]. This endows us with an uncanny skill for constructing an unlimited number of concepts by composing a relatively small set of previously learned basic building blocks, an ability more formally called _compositional generalization_. For example, we can generate sentences by combining a dictionary of words in different ways and solve a wide range of mathematical problems using a small set of basic arithmetic operations and variables. In the case of the visual world, objects form naturallycomposable entities and object-centric primitives can be flexibly combined using abstract syntactic rules to yield novel and counterfactual scenes.

The growing body of work in object-centric learning methods [4, 5, 6] enable the extraction of latent object representations from perceptual scenes. However, slot composition methods have been limited to _ad-hoc_ composition rules based on hard-coding slots into concept libraries [7, 8]. A central challenge in learning a generative model that explicitly models the compositional grammar of the visual world is that the seemingly unlimited number of composition rules that undergird real-life visual entities prevents the _explicit_ specification of each rule. For example, a warm spring day consists of blooming flowers and freshly verdant expanses, but come autumn, the same scene stipulates the presence of golden headovers carpeted with fallen leaves. State-of-the-art text-to-image models [9, 10, 11, 12] circumvent this bottleneck by grounding images in natural language. This aligns the representations of entities in an image to the compositional structure of language. While the visual quality of these models is stunning, they often require engineering complex language prompts to elicit the desired output. They also suffer from limited understanding of object relations [13]. Moreover, studies [14] have shown that infants as young as six months old can extract compositional structure from sequences of visual events, even before they acquire language. Such compositional abstraction independent of language faculty exhibited by humans remains elusive for contemporary models.

Large language models (LLMs) [15, 16, 17, 18, 19, 20], on the other hand, demonstrate _implicit_ composition skills. Pre-trained on large text corpora, these models have impressive emergent few-shot learning capabilities. They can solve novel tasks on the fly without gradient updates by following examples shown in the instruction prompts, a process known as _in-context learning_. Under the hood, in-context learning uses the inherently compositional nature of the input text to implicitly infer the task structure and how to apply the task structure to a query [21]. In the most simple form, the prompt-based few-shot learning paradigm can be formalized as _analogy solving_ of the type \(A:B::C:D\), where \(A\) represents an example task, \(B\) represents the example solution, and \(C\) denotes the query. The generated solution is the analogy completion \(D\). Analogies over compositional elements, therefore, provide a direct isomorphism between in-context learning for language and learning implicit composition rules for visual objects, as depicted in Fig. 1. While LLMs use pre-specified tokenizers [22, 23, 24], the compositional granularity for _implicit_ visual understanding is unspecified. On the other hand, slot methods induce object-level representations but fail to learn composition rules. In this work, we then ask:

_Can analogy-solving enable in-context generalization over object-centric compositional elements of visual entities?_

While many linguistic corpora have been extracted from the Internet for text-based tasks, to the best of our knowledge, no datasets exist for measuring in-context composition capability from visual prompts. To this end, we introduce a suite of three diverse benchmarks in Section 3 built on top of rich combinatorial spaces to systematically test image-based in-context learning abilities: (a) 3D Shapes [25], (b) BitMoji Faces, and (c) CLEVr Objects [26]. If the attention over word tokens enables analogical understanding in LLMs, can we transfer this mechanism to visual analogies? We formulate a unifying framework in Section 4.1 to formalize this notion. We present a visual analogy-based meta-learning algorithm called _Im-Promptu_ in Section 4.2 and use it to train agents with a cross-attention module. How does one tokenize visual inputs? To answer this, we model several generative agents in Section 5, encompassing a continuum ranging from strong compositional learners using object-centric abstractions to non-compositional monolithic representation learners. Extensive experiments presented in Section 6 show the dependence of robust in-context learning for visual stimuli on object-centric slots as visual tokenizers and cross-attention. We demonstrate the use

Figure 1: LLMs perform in-context understanding of a task from a few examples in an analogical manner (left). We use analogy-making to implicitly understand the composition rules over visual stimuli made of object-like entities (right).

of our visual analogy framework in task extrapolation (Section 6.2), combinatorial generalization (Section 6.3), and generating counterfactual images (Section 6.5).

## 2 Related Work

In this section, we discuss prior works and draw connections to relevant research areas.

### Object-Centric Learning:

This growing body of work targets unsupervised decomposition of scenes into a set of object-centric vectors from raw perception data. A common approach involves modeling an autoencoder consisting of a latent factorization of object slots that are independently decoded to reconstruct the scene [4, 5, 6, 27, 28, 29]. These methods effectively segment scenes into representations of underlying objects but do not learn the flexible combination of object representations. On the other hand, the authors in [7] replace the independent slot decoding bias with an Image-GPT [30] and demonstrate novel slot composition. While this work largely answers how to compose a _given_ set of visual entities, their slot selection is performed manually through organization of slots into clustered libraries. In our work, we use a meta-learning approach to help the learner implicitly understand the composition task. Recently, the authors of [31] recast the iterative slot refinement problem as finding of fixed points through implicit differentiation to improve the stability and computational cost of the slot-attention (SA) technique. The authors of [32] use specific inductive biases to disentangle object contents from spatial information that enables relational reasoning. Our work produces a more generalized composition beyond spatial relations.

### Analogical Reasoning:

The ability to make analogies has been shown to have important implications in various domains, such as problem-solving, creativity, learning, and counterfactual thinking [1, 33, 34, 35, 36]. Various models have been proposed to explain the underlying mechanisms of analogy-making. Popular symbolic approaches include ARGUS [37], Transfer Frames [38], and Structural Mapping Engine [39], connectionist models include LISA [40] and Star-1 [41], and models that occupy a middle ground like Copycat [42]. Contemporary deep-learning models [43, 44, 45] attempt to solve Raven's Progressive Matrices and Bongard Problems, types of visual analogies present in IQ tests. The authors of [46] propose a generative deep visual analogy-making model based on a parallelogram regularizer on the latent space to disentangle variation factors and traverse transformation manifolds. Most similar to us, a recent work [47] reformulates the analogy-making task as Inpainting in an image grid and proposes an auto-encoder pre-training task to that end.

### Compositional Generative Models:

Prior works [48, 49] use energy-based models to represent composable factors of variation of a scene. However, these methods use concept labels or are grounded in text and are limited to a few concept composition operations. A follow-up work [50] describes unsupervised energy-based concepts; however, the re-composition is performed via manually-picked concepts. Recent works in text-to-image models use language grounding to induce compositionality. DALL-E [9] uses language modeling over the joint space of the text and image tokens to facilitate image generation. The latest text-to-image models [10, 11, 12] use diffusion modeling to guide the image generated from the text. A growing body of work [51, 52, 53] explores text-grounded controllable image editing via prompt engineering, where most parts of the image are preserved, but specific objects are re-composed.

### Modular Representations and Attention:

A swath of recent works proposes neural methods that combine modular primitives with recurrent mechanisms and cross-attention. The authors of [54] propose the Recurrent Independent Mechanisms (RIMs) to learn weakly coupled dynamical systems with independent compositional neural modules that communicate via a sparse attention bottleneck. Another work [55] uses object-centric abstractions as 'files' to store factorized declarative knowledge and the dynamics of the environment. A follow-up work [56] introduces neural abstractions of the classical Production Systems framework to learn rule-based templates, thereby facilitating compositional dynamics of the environment that can quickly adapt to novel stimuli. Most recently, the authors in [57] propose an attention-based architecture to learn in-context causal relationships between entities.

## 3 Benchmarks

Humans can contextualize past visual stimuli in the face of everyday experiences and even use them for imagining counterfactuals. Having seen an oak table, we can, with little difficulty, look at a metal armchair and understand the concept of an oak armchair. How do we then systematically test such abilities of a visual in-context learner? In this work, we posit that an answer to this question lies in combinatorial visual spaces with controllable factors of variation, which provide a _semantically rich, simple, scalable, and composable_ avenue for setting up such tasks. We create a suite of three benchmarks (Fig. 2) from compositional image creators that include (a) 3D Shapes [25], (b) BitMoji Faces, and (c) CLEVr Objects [26]. Each benchmark has a split of primitive training tasks and out-of-distribution test tasks. The _primitive_ tasks (Fig. 2, left) constitute an analogy where a _single_ composable element of the visual stimulus is modified in isolation. The description of each benchmark with its primitive set of tasks is given next:

**3.1 3D Shapes Image Prompts:** This dataset consists of static scenes of various objects lying on a colored floor in front of a colored wall viewed from different orientations. To set up primitive tasks, we consider the four properties \(P\) = {object color, wall color, floor color, scene orientation}, where each property can take multiple values within a domain. For each such task, the latent relation \(r(.)\) for the analogy prompt takes a property \(p\in P\) and modifies it from a _source_ domain value to a _target_ value. This benchmark comprises \(80000\) such tasks with a maximum of four supporting image pairs for each instance and with a roughly equal split across properties.

**3.2 BitMoji Faces Image Prompts:** BitMoji is an avatar creator service for social media users that allows them to create intricate cartoon faces. We queried the BitMoji API [58] to collect faces with four underlying dynamic elements \(P=\{\texttt{skin tone},\texttt{facial hair style},\texttt{hair style},\texttt{eyewear}\}\). Much like the previous case, primitive tasks are generated by intervening on a _source_ value of a property and modifying it to a _target_ value. We populate the training set with 80000 tasks, with four demonstrations available per task.

**3.3 CLEVr Objects Image Prompts:** CLEVr is a popular visual question-answering dataset with the visual component consisting of multiple objects lying in a scene. We use the CLEVr rendering engine to set up primitive tasks that include adding and deleting the same object across various scenes. In this fashion, we generate \(60000\) tasks with three examples per task.

A composable image space allows the concurrent modification of individual visual elements. One can construct _composite_ tasks (Fig. 2, right) out of a combination of primitives shown in the training set. A \(k\)-composite task is denoted as \(R^{k}(.)=r_{1}\circ r_{2}\circ\cdots\circ r_{k}\) where \(r_{1},\cdots,r_{k}\in\mathcal{R}\). We provide details of the \(k\)-compositeness test of each of our agents in Section 6.3. Finer details about each

Figure 2: We use combinatorial image spaces to set up visual in-context learning tasks. Each task requires the generation of a solution from a query and supporting examples. The latent description of the task is described within the square parentheses. Left: Primitive tasks modify a composable element in isolation. Right: Composite tasks combine primitives to test the combinatorial generalization skills of the learner.

benchmark, along with visual examples from every task, are detailed in Appendix A. We will release the complete dataset of primitive and composite tasks upon the publication of this article.

## 4 Methods

To learn compositional structure over visual stimuli, we begin by formulating a unified mechanistic interpretation of an in-context compositional learner using insights from design elements of LLMs.

### In-Context Learning as Analogy Completion:

A general _compositional_ in-context learner \(\mathcal{M}_{\phi,\alpha,\theta}(.)\) can be formalized via the following three key components:

* An encoder \(\mathcal{E}_{\phi}(.)\) that maps the input space \(\mathcal{X}\) to the compositional space \(\mathcal{C}\triangleq\mathcal{E}_{\phi}(.):\mathcal{X}\longmapsto\mathcal{C}\)
* An executor \(\mathcal{T}_{\alpha}(.\,,.\,,.)\) that maps compositional entities from example tasks, example solutions, and a query task to the query solution \(\triangleq\mathcal{T}_{\alpha}(.,.,.):\mathcal{C}\times\mathcal{C}\times \mathcal{C}\longmapsto\mathcal{C}\)
* A decoder \(\mathcal{D}_{\theta}(.)\) that maps the compositional space back to the input space \(\triangleq\mathcal{D}_{\theta}(.):\mathcal{C}\longmapsto\mathcal{X}\)

Coupling the above functions together yields the learner:

\[\mathcal{M}\triangleq\mathcal{D}_{\theta}\left(\mathcal{T}_{\alpha}\left( \mathcal{E}_{\phi}(.),\mathcal{E}_{\phi}(.),\mathcal{E}_{\phi}(.)\right) \right):\mathcal{X}\times\mathcal{X}\times\mathcal{X}\longmapsto\mathcal{X} \tag{1}\]

The inclusion of \(\mathcal{E}_{\phi}(.)\) and \(\mathcal{D}_{\theta}(.)\) is crucial for compositional abstraction since the input space may not be _a priori_ composable as in the case of images. Beyond this, we do not place any strong parametric constraints on the model. Let \(\mathcal{R}\) denote a task set. For any task \(r\in\mathcal{R}\), define the task analogy \(\zeta\) over input pairs \(x_{1},x_{2}\) as \(x_{1}:r(x_{1})::x_{2}:r(x_{2})\triangleq A:B::C:D\). An effective in-context learner over \(\mathcal{R}\) then satisfies the following property [59]:

\[\forall r\in\mathcal{R},\;\mathbb{E}_{\zeta\sim\mathcal{P}(r,\mathcal{X})} \left[\mathcal{L}\left(D;\mathcal{M}_{\phi,\alpha,\theta}(A,B,C)\right)\right] \leq\epsilon \tag{2}\]

Here, \(\mathcal{P}\) denotes the distribution of the analogies resulting from the input space \(\mathcal{X}\) and task \(r(.)\), \(\mathcal{L}(.)\) denotes a loss metric, and \(\epsilon\) is an error bound. The above property simply states that a good in-context learner solves analogies defined over a task set within some finite error bound. In Appendix B, we describe LLMs as an instantiation of this framework.

### Im-Promptu Learning:

Pre-training LLMs on large-scale text data leads to the emergence of highly transferable internal representations that allow them to adjust to novel tasks by simple priming on a few demonstrations. However, it is still unclear whether the underlying structure of the training data, architecture, or inductive biases instilled by choice of pre-training learning algorithms cause these intriguing properties to emerge. How can we encourage the emergence of such a general-purpose representation for visual data? In this work, we take an explicit approach based on the formalism presented in the previous section. Having established the desideratum (Eq. 2), we set up a learning procedure called _Im-Promptu_ that trains the model to generate analogy completions from image prompts. To this end, given a task set \(\mathcal{R}\), in each minibatch, we sample an analogy \(A:B::C:D\) that follows a latent primitive \(r(.)\in\mathcal{R}\). Given a loss criterion, \(\mathcal{L}\), we make the following stochastic gradient updates of the model parameters:

\[\{\phi^{t+1},\alpha^{t+1},\theta^{t+1}\}=\{\phi^{t},\alpha^{t},\theta^{t}\}\;- \frac{\alpha}{N}\sum_{i=1}^{N}\nabla_{\theta^{t},\alpha^{t},\phi^{t}} \mathcal{L}\left(D,\mathcal{M}_{\phi^{t},\alpha^{t},\theta^{t}}\left(A,B,C \right)\right) \tag{3}\]

where the step size \(\alpha\) is a hyperparameter. The full _Im-Promptu_ algorithm is laid out in Algorithm 1.

``` Input:\(\mathcal{R}\), \(\mathcal{R}\), \(\mathcal{R}

The encoder \(e_{\phi}(.)\) and decoder \(d_{\theta}(.)\) are convolutional and deconvolutional networks, respectively. The inference network \(f_{\alpha}(.)\) and executor network \(h_{\beta}(.)\) are modeled as multi-layered perceptrons. Here, \([\ ]\) denotes the concatenation operator. This architecture is detailed in Appendix D.1.

```
Require: Task set \(\mathcal{R}\), Step Size \(\alpha\)  Initialize parameters \(\theta,\alpha,\phi\) of model \(\mathcal{M}\) while not done do  Sample primitive task \(r(.)\sim\mathcal{R}\)  Sample \(N\) input image pairs \(X_{1}^{1:N},X_{2}^{1:N}\sim\mathcal{P}_{train}\) for\(i\) in \(1\cdots N\)do  Sample analogy \(\zeta^{i}\triangleq x_{1}^{i}:r(x_{1}^{i})::x_{2}^{i}:r(x_{2}^{i})\triangleq A :B::C:D\) endfor  Update \(\{\theta,\alpha,\phi\}=\{\theta,\alpha,\phi\}-\frac{\alpha}{N}\nabla_{\theta, \alpha,\phi}\sum_{i}\mathcal{L}_{\mathcal{M}}(\zeta^{i})\) endwhile
```

**Algorithm 1** Im-Prompt Learning Algorithm

### Dipainting Model:

Inspired by recent work on inpainting for In-Context learning [47], this agent (Inpaint., visualized in Appendix D.2) is modeled via an autoencoder architecture that fills in missing patches of an image. The architecture is composed of a discrete variational autoencoder (dVAE) that discretizes the input patches and a Vision Transformer [60] auto-encoder to impute missing values. While the monolithic agent separately encodes the components of the visual analogy, this model jointly represents them as a \(2\times 2\) image grid. The model is pre-trained using the masked autoencoder reconstruction task [61]. Subsequently, at inference time, the analogy solution is represented as a missing image in the grid. Details have been given in Appendix D.2.

### Object-Centric Learner (OCL):

This agent, as depicted in Fig. 3, is a strongly compositional learner that uses object-centric inductive biases. The encoder and decoder of the framework in Section 4.1 are parametrized using Slot Attention Transformer (SLATE) [7] (see Appendix C.2) as follows:

\[\mathcal{E}_{\phi}(.)\triangleq SA_{\phi}(f_{\phi}^{\text{dVAE}}(.)),\ \mathcal{D}_{\theta}(.)\triangleq g_{\theta}^{dVAE}(g_{\theta}^{GPT}(.)) \tag{5}\]

A Context Encoder Transformer (CET) \(\triangleq\mathcal{T}_{\alpha}(.\,.\,.)\) (.) with interleaved layers of: \((a)\) cross-attention on the context slots and \((b)\) self-attention, induces sparse modifications on the query slots to complete the analogy.

\[S_{1:N}^{A}=\mathcal{E}_{\phi}(A),\ S_{1:N}^{B}=\mathcal{E}_{\phi}(B),\ S_{1:N}^{C}=\mathcal{E}_{\phi}(C) \tag{6}\]

\[S_{1:N}^{D}=CET\left(\text{query}=S_{1:N}^{C},\text{keys, values}=S_{1:N}^{A},S_{1:N}^{B}\right) \tag{7}\]

The latent sequence \(\hat{Z}_{D}\) predicted by Image-GPT is mapped by the dVAE back to the pixel space \(\hat{D}\).

\[\hat{Z}_{D}=g_{\theta}^{GPT}(S_{1:N}^{D}),\ \hat{D}=g_{\theta}^{dVAE}(\hat{Z}_{D}) \tag{8}\]

The CET forward pass is shown in Appendix D.3.

### Sequential Prompter:

This (Seq., visualized in Appendix D.4) is an ablation over the OCL that replaces CET with an LLM-like sequence prompt obtained from a simple concatenation of slots of images \(A\), \(B\), and \(C\), injected with position embeddings \(p_{i}\).

\[\mathcal{T}_{\alpha}(.,.,.)\triangleq[S_{1:N}^{A}\,S_{1:N}^{B},\ S_{1:N}^{C}]+p_{i} \tag{9}\]

While the OCL explicitly encodes the entire context into a fixed-length slot sequence via the CET, the ablated agent models a longer concatenated sequence.

### Patch Learner:

This agent straddles the compositionality extremes of Monolithic Learners and OCLs, and uses image patch abstractions. The SA module from OCL is ablated to get a discrete sequence of \(4\times 4\) patch latents as the degree of compositional granularity \(\mathcal{E}_{\phi}(.)\triangleq f_{\phi}^{\text{dVAE}}(.)\). A larger image implies longer sequences for this modeling choice.

## 6 Experiments

In this section, we set up experiments to answer (1) whether analogical reasoning enables in-context generalization, (2) does such a generalization require key mechanisms from LLMs, namely compositionality and attention, (3) if compositionality is a key ingredient, what is the correct granularity for visual tokens, and (4) what are the limits to the generalization.

**6.1**: **Training Setup:** We trained each agent on the primitive set of tasks across the three benchmarks using _Im-Promptu_ (Section 4.2). The SA-based object-centric encoders used in the OCL and Sequential Prompter were pre-trained as a SLATE autoencoder, as described by the authors in [7]. On the other hand, the Patch Learner was trained completely from scratch. The loss metric \(\mathcal{L}\) used to train the agents was cross-entropy (CE) loss between the true latent sequence \(Z_{D}\) and the predicted solution \(\hat{Z}_{D}\) obtained from Image-GPT, i.e., \(\mathcal{L}_{impromptu}=CE(Z_{D},\hat{Z}_{D})\).

In addition to the above loss, the dVAE was trained using the mean-squared error (MSE) loss over the raw pixel space to yield the full loss function \(\mathcal{L}=\mathcal{L}_{impromptu}+MSE(D,\hat{D})\). For inference, answers of the transformer-based agents were sampled from the Image-GPT decoder using top-\(k\) nucleus sampling [62]. Hyperparameters for training and inference have been laid out in Appendix E.

**6.2**: **Primitive Task Extrapolation:** In the first out-of-distribution paradigm, we tested the ability of agents to apply learned rules to different domain pairs. In order to do this, we held out \(20\%\) of source-target pairs (see Section 3 for the definition of source and target) from the training primitives and tested the ability of agents to generalize from learned rules to unseen source-target pairs. For example, the object color property in the 3D Shapes benchmark can take \(10\) unique values and \(10\times 9=90\) source-target combinations. Thus, an agent was trained on only \(90\times 0.8=72\) pairs for object-hue primitives. We made sure that each value in the domain set was shown at least once as either the target or the source.

Fig. 4(a) plots scores of different agents across benchmarks against two key metrics: (1) MSE (lower is better) that quantitatively compares the construction against the ground truth and (2) Frechet inception distance (FID, lower is better) score to measure the perceptual quality of the composition. We make several interesting observations:

**(R1.1) Simple pixel manipulation produces implausible outputs.** The pixel baseline has poor MSE scores (\(\sim\) 138 for 3D Shapes, \(\sim\) 895 for BitMoji) with clearly apparent artifacts in the output. While linear pixel transformations are sufficient when an object is independent of the remaining entities, it is unable to model complex dependencies (see Fig. 4(b),(d) under 'Pixel').

**(R1.2) Inpainting model struggles to generalize beyond i.i.d. inputs.** We observed that the Inpainting model is able to cogently complete analogies on the i.i.d. validation set (see Appendix F.1 for outputs). However, when tested on extrapolated primitives, it only fills in the high-level structure (see Fig. 4 (b), (c)) and, in the case of more complex inputs, produces entirely incoherent outputs (Fig. 4 (d)). We posit that a top-down method like inpainting pre-ordains larger datasets for improved generalization and, as such, suffers from sample inefficiency.

**(R1.3) Monolithic Learners are reasonably effective at generalizing from learned rules on structured stimuli.** These agents have the lowest MSE scores on 3D Shapes and BitMoji Faces, both spatially consistent datasets. However, the output deteriorated over CLEVr Objects where the

Figure 3: The OCL uses compositional object vectors to answer image prompts. (**1**) The encoder \(\mathcal{E}_{\phi}(.)\) is a pre-trained SLATE [7] encoder that uses a dVAE to obtain a latent discrete sequence for the input image and runs SA [6] over the sequence to extract object-centric slots. (**2**) The CET modifies query slots \(S^{\mathcal{C}}_{1:N}\) via cross-attention over the support prompt slots \(S^{A}_{1:N},S^{B}_{1:N}\). (**3**) The modified slots are used to prompt an Image-GPT [30] that outputs the latent discrete sequence of the task solution \(D\) that is decoded by the dVAE decoder.

latent scene configuration is diverse. The edges of the generated shapes and lighting details were significantly blurred, and occluded objects were often fused together (FID \(\sim\) 92, worse than Pixel baseline, Fig. 4(d) under 'Monolithic').

**(R1.4) Patch Learner and Sequential Prompter benefit from additional examples.** For these agents, the addition of context examples was markedly apparent from the significant MSE drops for \(>\) 1 shot (see Fig. 4(a)). We posit that since these models learn from longer contexts, additional examples are particularly effective in regularizing the output sequence manifold of Image-GPT.

### Composite Task Extrapolation:

In this generalization paradigm, we probed the compositional generalization capabilities of the agents. A \(k\)-composite task (\(k\geq 2\), see Section 3) is an unordered composite function of \(k\) primitives. The value of \(k\) is varied across each benchmark, where higher values of \(k\) yield visually complex contexts. To solve such a task, the agent must identify the underlying primitive relations and combinatorially modify the query.

In Fig. 5(a), we note (R1.1) and (R1.3) from the previous setup in this extrapolation paradigm as well, but we make additional key observations next:

**(R2.1) The effect of object-centric biases is strong.** This is indicated by the widening MSE gap between the OCL and monolithic agent for increasing values of \(k\) across all three benchmarks (see Fig. 5(a)-(c)). Moreover, the OCL and the Sequential Prompter generate outputs with consistent perceptual quality (lower FID scores).

**(R2.2) Monolithic and Patch Learners apply shortcuts.** The monolithic agent tends to produce a Frankenstein-like image, as if a superposition of context images (see Fig. 5(d)-(f) under 'Monolithic'). In the case of the CLEVr Objects dataset, both the Monolithic and Patch Learners often simply filled in 'blobs' agnostic to the color and shape of objects (Fig. 5(f), Appendix F.4), with missing shadows leading to lower MSE scores but poor visual quality.

### Analysis:

Next, we discuss key elements required for in-context composition.

**(A1) In-context generalization critically hinges on compositional granularity.** As evident from (R1.2), monolithic representations suffice for reliably learning primitive composition rules over simple contexts. However, they suffer from fidelity issues when the visual space becomes more complex. Beyond primitive generalization, patch abstractions are effective when the underlying space

Figure 4: Primitive task extrapolation: (a) Plot showing scores where the left y-axis and the bars represent the MSE scores, while the right y-axis and the dotted line denote the FID scores. (b)-(d) Comparison of agent solutions with the dotted red circle representing anomalies with respect to the ground truth.

has object-patch equivalence, as evident from the CLEVr Objects dataset (Fig. 5(c)). However, even with object-patch equivalence, the Patch Learner omits the inter-object dependencies (e.g., object occlusions and shadows). It further fails to learn complex and diverse objects spanning multiple patches reliably (does poorly on the BitMoji Faces dataset with diverse facial components). Slot-based compositionality with the CET not only enables the generalization of the learned primitives but also enables combinatorial generalization to composite tasks that require abstracting out the composition rule over each entity in isolation as well as composing the resultant interplay between them to generate a globally consistent output.

**(A2) Encoding the context via cross-attention leads to sample efficiency.** Longer slot sequences in the Sequential Prompter require multi-shot contexts across all benchmarks. The sample inefficiency was particularly observed on the CLEVr Objects dataset that uses six slots and, hence, much longer sequences. Cross-attention via CET is essential for implicit inference, enabling the learner to encode specific objects from the context.

### 'Counterfactual Prompt Engineering' with Im-Promptu:

LLMs have spurred interest in several priming techniques [63, 64] that can 'engineer' the model into following user instructions. While this natural-language-based programming language provides an intuitive interface for tasks that have a fundamental degree of language abstraction (writing code, summarization, symbolic

Figure 5: Composite task extrapolation: (a)-(c) Plots showing scores across benchmarks where the left y-axis and the bars represent the MSE scores, while the right y-axis and the dotted line denote the FID scores. \(k\) denotes the level of compositeness. (d)-(f) Comparison of agent solutions with the dotted red circle representing anomalies with respect to the ground truth.

manipulation, etc.), it is not _a priori_ obvious that textual descriptions should also scaffold image generators. Coming up with natural-language prompts for complex visual entities is often tedious and unintuitive. Instead, explaining an image is naturally easier from its entities and composition over them1. To this end, _Im-Promptu_ provides an avenue for generating counterfactuals via image prompts. In Fig. 6(a), we demonstrate the creation of a scene from its object components via the OCL. As a point of comparison, in Fig. 6(b), using image-inpainting and natural language prompts with DALL-E [9] yields an unreliable output with distorted objects. More practically, the OCL reduces human effort significantly in rendering images with desired properties by using existing visual assets. The procedure to generate scenes from OCL can be found in Appendix G.

Footnote 1: “No one is an artist unless he carries his picture in his head before painting it, and is sure of his method and composition” – Claude Monet

## 7 Conclusion

This work investigated whether analogy-solving can enable in-context compositional generalization over visual entities. We first designed a new test suite. We probed if attention and compositionality, key ingredients in LLMs, also play a crucial role in the visual domain. We transported these ingredients from language to model several agents. Our experiments showed that non-compositional agents do not generalize well beyond primitive task extrapolation. Even with compositionality baked in, the performance of patch-based learners depends on idiosyncrasies of the visual structure. However, we found that object-centric biases consistently facilitate an implicit understanding of composition rules and generate outputs with global semantic consistency. Our ablation of the CET suggested that cross-attention plays a crucial role, just like language models, which posits the importance of analogy and attention in understanding the underpinnings of in-context learning. Future research challenges include collecting real-world primitives from photo-realistic graphic engines or human labels, improving object-centric inductive biases for real-life entities, and designing better prompting techniques. We hope that our work will spur further research on visual in-context learning.

## Acknowledgments and Disclosure of Funding

We would like to thank Sreejan Kumar for helpful discussions and insights throughout the course of this work. The experiments reported in this article were performed on the computational resources managed and supported by Princeton Research Computing at Princeton University. This project was funded by NSF Grant No. CCF-2203399 and ONR Grant No. N00014-18-1-2873. JS gratefully acknowledges financial support from the Schmidt DataX Fund at Princeton University made possible through a major gift from the Schmidt Futures Foundation.

Figure 6: Scene creation from object components: (a) OCL trained via _Im-Promptu_ is used to generate a scene of objects. Image-Prompts engineered using existing assets reliably generate the scene object by object. (b) The same scene is generated via language grounding and in-painting features of DALL-E [9]. Object properties are distorted and specifying the location of objects is tedious via language.

## References

* [1] Douglas R. Hofstadter and Emmanuel Sander. _Surfaces and Essences: Analogy as the Fuel and Fire of Thinking_. Basic Books, 2013.
* [2] Brenden M. Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua B. Tenenbaum. One Shot Learning of Simple Visual Concepts. _Cognitive Science_, 33, 2011.
* [3] Brenden M. Lake, Tal Linzen, and Marco Baroni. Human Few-Shot Learning of Compositional Instructions. _CoRR_, abs/1901.04587, 2019.
* [4] Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised Scene Decomposition and Representation. _CoRR_, abs/2303.08774, 2019.
* [5] Klaus Greff, Raphael Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-Object Representation Learning with Iterative Variational Inference. _CoRR_, abs/1903.00450, 2019.
* [6] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-Centric Learning with Slot Attention. _CoRR_, abs/2006.15055, 2020.
* [7] Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate DALL-E Learns to Compose. _CoRR_, abs/2110.11405, 2021.
* [8] Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-Centric Slot Diffusion. _CoRR_, abs/2303.10834, 2023.
* [9] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-Shot Text-to-Image Generation. _CoRR_, abs/2102.12092, 2021.
* [10] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text-Conditional Image Generation with CLIP Latents. _CoRR_, abs/2204.06125, 2022.
* [11] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. _CoRR_, abs/2205/11487, 2022.
* [12] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. _CoRR_, abs/2112.10752, 2021.
* [13] Gary Marcus, Ernest Davis, and Scott Aaronson. A Very Preliminary Analysis of DALL-E 2. _CoRR_, abs/2202.13807, 2022.
* [14] Fei Xu, Elizabeth S. Spelke, and Sydney Goddard. Number Sense in Human Infants. _Developmental Science_, 8(1):88-101, 2005.
* [15] Tom B. Brown et al. Language Models are Few-Shot Learners. _CoRR_, abs/2005.14165, 2020.
* [16] Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax), May 2021.
* [17] Aakanksha Chowdhery et al. PaLM: Scaling Language Modeling with Pathways. _CoRR_, abs/2204.02311, 2022.
* [18] Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. Jurassic-1: Technical Details and Evaluation. [https://uploads-ssl.webflow.com/60fd4503684b466578c0d307/61138924626a6981ee09caf6_jurassic_tech_paper.pdf](https://uploads-ssl.webflow.com/60fd4503684b466578c0d307/61138924626a6981ee09caf6_jurassic_tech_paper.pdf), 2022.
* [19] Teven Le Scao et al. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. _CoRR_, abs/2211.05100, 2022.
* [20] OpenAI. GPT-4 Technical Report. _CoRR_, abs/2303.08774, 2023.

* [21] Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. Larger Language Models do In-context Learning Differently. _CoRR_, abs/2303.03846, 2023.
* [22] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural Machine Translation of Rare Words with Subword Units. _CoRR_, abs/1508.07909, 2016.
* [23] Mike Schuster and Kaisuke Nakajima. Japanese and Korean Voice Search. In _IEEE International Conference on Acoustics, Speech and Signal Processing_, pages 5149-5152, 2012.
* [24] Taku Kudo and John Richardson. SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing. _CoRR_, abs/1808.06226, 2018.
* [25] Chris Burgess and Hyunjik Kim. 3D Shapes Dataset. [https://github.com/deepmind/3dshapes-dataset/](https://github.com/deepmind/3dshapes-dataset/), 2018.
* [26] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. _CoRR_, abs/1612.06890, 2016.
* [27] Klaus Greff, Sjoerd van Steenkiste, and Jurgen Schmidhuber. On the Binding Problem in Artificial Neural Networks. _CoRR_, abs/2012.05208, 2020.
* [28] Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations. _CoRR_, abs/1907.13052, 2019.
* [29] Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement. _CoRR_, abs/2104/09958, 2021.
* [30] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative Pretraining From Pixels. In Hal Daume III and Aarti Singh, editors, _Proceedings of the 37th International Conference on Machine Learning_, volume 119 of _Proceedings of Machine Learning Research_, pages 1691-1703. PMLR, 13-18 July 2020.
* [31] Michael Chang, Thomas L. Griffiths, and Sergey Levine. Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation. _CoRR_, abs/2207.00787, 2022.
* [32] James C. R. Whittington, Rishabh Kabra, Loic Matthey, Christopher P. Burgess, and Alexander Lerchner. Constellation: Learning Relational Abstractions over Objects for Compositional Imagination. _CoRR_, abs/2107.11153, 2021.
* [33] J. P. Guilford. Creativity. _American Psychologist_, 5(9):444, 1950.
* [34] Dedre Gentner. Structure-Mapping: A Theoretical Framework for Analogy. _Cognitive Science_, 7(2):155-170, 1983.
* [35] Arthur B. Markman and Dedre Gentner. Structural Alignment in Analogy and Similarity. _American Psychologist_, 48(1):40, 1993.
* [36] Keith J. Holyoak and Paul Thagard. _Mental Leaps: Analogy in Creative Thought_. MIT Press, 1995.
* [37] W. R. Reitman. _Cognition and Thought: An Information Processing Approach_. John Wiley and Sons, 1965.
* [38] Patrick H. Winston. Learning by Creatifying Transfer Frames. _Artificial Intelligence_, 10(2):147-172, 1978.
* [39] Brian Falkenhainer, Kenneth D. Forbus, and Dedre Gentner. The Structure-Mapping Engine: Algorithm and Examples. _Artificial Intelligence_, 41(1):1-63, 1989.
* [40] Jeffrey E. Hummel and Keith J. Holyoak. Distributed Representations of Structure: A Theory of Analogical Access and Mapping. _Psychological Review_, 104(3):427-466, 1997.

* [41] Graeme S. Halford, William H. Wilson, Jian Guo, Ross W. Gayler, Janet Wiles, and J. E. M. Stewart. _Connectionist Implications for Processing Capacity Limitations in Analogies_, pages 363-415. Ablex Publishing, 1994.
* [42] Douglas R. Hofstadter. _Fluid Concepts and Creative Analogies: Computer Models of the Fundamental Mechanisms of Thought_. Basic Books, 1995.
* [43] Stefan Depeweg, Constantin A. Rothkopf, and Frank Jakel. Solving Bongard Problems with a Visual Language and Pragmatic Reasoning. _CoRR_, abs/1804.04452, 2018.
* [44] Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. RAVEN: A Dataset for Relational and Analogical Visual rEasoNing. _CoRR_, abs/1903.02741, 2019.
* [45] Felix Hill, Adam Santoro, David G. T. Barrett, Ari S. Morcos, and Timothy Lillicrap. Learning to Make Analogies by Contrasting Abstract Relational Structure. _CoRR_, abs/1902.00120, 2019.
* [46] Scott E. Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep Visual Analogy-Making. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, _Advances in Neural Information Processing Systems_, volume 28. Curran Associates, Inc., 2015.
* [47] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei A. Efros. Visual Prompting via Image Inpainting. _CoRR_, abs/22098.00647, 2022.
* [48] Yilun Du, Shuang Li, and Igor Mordatch. Compositional Visual Generation with Energy Based Models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_, volume 33, pages 6637-6647. Curran Associates, Inc., 2020.
* [49] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. Compositional Visual Generation with Composable Diffusion Models. _CoRR_, abs/2206.01714, 2022.
* [50] Yilun Du, Shuang Li, Yash Sharma, Joshua B. Tenenbaum, and Igor Mordatch. Unsupervised Learning of Compositional Energy Concepts. _CoRR_, abs/2111.03042, 2021.
* [51] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aherman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-Prompt Image Editing with Cross Attention Control. _CoRR_, abs/2208.01626, 2022.
* [52] Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan. UniTune: Text-Driven Image Editing by Fine Tuning an Image Generation Model on a Single Image. _CoRR_, abs/2210.09477, 2022.
* [53] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-Based Real Image Editing with Diffusion Models. _CoRR_, abs/2210.09276, 2022.
* [54] Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Scholkopf. Recurrent Independent Mechanisms. _CoRR_, abs/1909.10893, 2020.
* [55] Anirudh Goyal, Alex Lamb, Phanideep Gampa, Philippe Beaudoin, Sergey Levine, Charles Blundell, Yoshua Bengio, and Michael Mozer. Object Files and Schemata: Factorizing Declarative and Procedural Knowledge in Dynamical Systems. _CoRR_, abs/2006.16225, 2020.
* [56] Anirudh Goyal, Aniket Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael Mozer, and Yoshua Bengio. Neural Production Systems: Learning Rule-Governed Visual Dynamics. _CoRR_, abs/2103.01937, 2022.
* [57] Nan Rosemary Ke, Silvia Chiappa, Jane Wang, Anirudh Goyal, Jorg Bornschein, Melanie Rey, Theophane Weber, Matthew Botvinic, Michael Mozer, and Danilo Jimenez Rezende. Learning to Induce Causal Structure. _CoRR_, abs/2204.04875, 2022.
* [58] Snap Inc. Bitmoji. [https://www.bitmoji.com/](https://www.bitmoji.com/), accessed 2023.
* [59] Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What Can Transformers Learn In-Context? A Case Study of Simple Function Classes. _CoRR_, abs/2208.01066, 2022.

* [60] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. _CoRR_, abs/2010.11929, 2021.
* [61] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. _CoRR_, abs/2111.06377, 2021.
* [62] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The Curious Case of Neural Text Degeneration. _CoRR_, abs/1904.09751, 2020.
* [63] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. _CoRR_, abs/2107.13586, 2021.
* [64] Simon Willison. In Defense of Prompt Engineering. [https://simonwillison.net/2023/Feb/21/in-defense-of-prompt-engineering/](https://simonwillison.net/2023/Feb/21/in-defense-of-prompt-engineering/), February 2023.
* [65] Jason Tyler Rolfe. Discrete Variational Autoencoders. _CoRR_, abs/1609.02200, 2016.
* [66] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural Discrete Representation Learning. _CoRR_, abs/1711.00937, 2017.

Response:
## Review

### Summary

This paper investigates the potential of analogical reasoning to enable in-context compositional generalization in visual modalities. It introduces a framework called Im-Promptu and constructs three benchmarks focused on visual analogy-making, including 3D shapes, BitMoji Faces, and CLEVR. The authors explore various visual representations, such as pixel-based and object-centric methods, to test the efficacy of their approach. The findings highlight the interaction between the degree of compositionality and extrapolation capabilities, ultimately aiming to contribute valuable insights to the field of visual in-context learning.

### Strengths

- The paper constructs three benchmarks to test compositional generalization in visual analogy-making, which is a novel area of study in computer vision.
- The introduction of the Im-Promptu framework is a significant contribution, providing a new approach for in-context learning.
- The experimental evaluations demonstrate the effectiveness of object-centric representations for compositional generalization.
- The work emphasizes the importance of in-context learning in visual modalities, which is an under-explored area compared to language models.

### Weaknesses

- The novelty of the proposed Im-Promptu framework is limited, as it resembles existing methods in visual analogy-making without clear differentiation.
- The benchmarks primarily utilize synthetic data, raising concerns about the generalizability of the results to real-world scenarios.
- The paper lacks comprehensive evaluations across a wider range of tasks beyond the three benchmarks presented.
- The clarity of the paper could be improved, as some sections are dense with technical details and may confuse readers.

### Questions

- What are the fundamental differences between the proposed Im-Promptu learning and existing visual analogy-making frameworks?
- What is the main technical contribution of this paper?
- Is there any ablation study regarding the number of in-context demonstrations used?
- Can the output of tasks be text rather than images in the proposed framework?
- How is the position-preserving functionality in the generated images realized by the Object-Centric Learner?

### Soundness

**Score:** 3

**Description:** 3 = good; the methodology is sound but lacks sufficient evaluation on real-world data to support its claims.

### Presentation

**Score:** 3

**Description:** 3 = good; while the paper is generally well-structured, some sections are overly complex and could benefit from clearer explanations.

### Contribution

**Score:** 3

**Description:** 3 = good; the paper presents novel ideas and frameworks but lacks clear articulation of its unique contributions compared to existing works.

### Rating

**Score:** 7

**Description:** 7 = accept, but needs minor improvements; the paper has a solid foundation with significant contributions but requires more comprehensive evaluations and clarifications.

### Paper Decision

**Decision:** Accept (poster)

**Reasons:** The paper demonstrates originality and soundness in exploring in-context learning in visual modalities. While it shows significant potential, particularly in addressing a novel area, it requires further evaluation on real-world datasets and clearer presentation of its contributions to fully establish its impact.