Chain of Ideas: Revolutionizing Research in Idea Development with LLM Agents
Under review as a conference paper at ICLR 2025

CHAIN OF IDEAS: REVOLUTIONIZING RESEARCH IN
NOVEL IDEA DEVELOPMENT WITH LLM AGENTS

Anonymous authors
Paper under double-blind review

ABSTRACT

Effective research ideation is a critical step for scientific research. However, the
exponential increase in scientific literature makes it challenging for researchers to
stay current with recent advances and identify meaningful research directions. Re-
cent developments in large language models (LLMs) suggest a promising avenue
for automating the generation of novel research ideas. However, existing methods
for idea generation either trivially prompt LLMs or directly expose LLMs to ex-
tensive literature without indicating useful information. Inspired by the research
process of human researchers, we propose a Chain-of-Ideas (CoI) agent, an LLM-
based agent that organizes relevant literature in a chain structure to effectively
mirror the progressive development in a research domain. This organization facil-
itates LLMs to capture the current advancements in research, thereby enhancing
their ideation capabilities. Furthermore, we propose Idea Arena, an evaluation
protocol that can comprehensively evaluate idea generation methods from dif-
ferent perspectives, aligning closely with the preferences of human researchers.
Experimental results indicate that the CoI agent consistently outperforms other
methods and shows comparable quality as humans in research idea generation.
Moreover, our CoI agent is budget-friendly, with a minimum cost of $0.50 to gen-
erate a candidate idea and its corresponding experimental design1.

1

INTRODUCTION

Idea generation is a crucial aspect of scientific research for driving technological innovations and
breakthroughs. Traditionally, this process has been predominantly human-driven, necessitating ex-
pert researchers to review extensive literature, identify limitations in existing solutions, and propose
new research directions. However, the complexity and vastness of scientific literature, coupled with
rapid technological advancements, have rendered this task increasingly challenging for researchers.

Recent advancements in large language models (LLMs) (Achiam et al., 2023; Dubey et al., 2024;
Yang et al., 2024a) have enabled these models to exceed human experts in various scientific tasks,
including mathematics (Yu et al., 2023), theorem proving (Yang et al., 2023), and coding (Chen
et al., 2021). Building on this robust scientific foundation, one may hypothesize that LLMs could
support a more abstract and creative research idea-generation task. Notably, Si et al. (2024); Kumar
et al. (2024) have validated this hypothesis, highlighting its substantial potential to expedite the
discovery of novel concepts and uncharted research avenues.

Existing methods seek to address two key challenges to improve the quality of generated ideas:
curating pertinent literature for LLMs to gain inspiration and ensuring the novelty of generated ideas.
To address the first challenge, previous research enhances traditional academic retrieval systems,
which typically depend on textual similarity, with academic knowledge graphs (Baek et al., 2024;
Wang et al., 2023). For the second challenge, existing approaches either apply predefined criteria
such as novelty to guide the idea generation process (Baek et al., 2024) or iteratively refine ideas
until they demonstrate low embedding similarities with existing papers (Wang et al., 2023).

However, in existing attempts, LLMs are presented with an extensive volume of research literature
when asked to generate ideas. This makes LLMs vulnerable to the influence of less relevant works,

1We will make our code and data publicly available

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Figure 1: Comparison between the vanilla retrieval augmented generation (RAG) research agent and
our Chain-of-Ideas agent on the idea generation task.

potentially resulting in ideas that lack logical coherence and technological innovation. As shown in
the upper part of Figure 1, the LLM borrows an idea from GraphGPT (Tang et al., 2024) and applies
it into GoT framework (Besta et al., 2024) to generate what they interpret as a “novel idea”. However,
the resultant idea conflates two concepts: GoT is a prompting method while GraphGPT is a fine-
tuning method leveraging graph neural network architecture (Zhou et al., 2020). In contrast, human
researchers often trace the evolution of a research field by analyzing its progression from founda-
tional works to the most recent advancements. This comprehensive perspective provides valuable
insights into the key factors driving developments within the domain. Such an understanding enables
researchers to critically assess the limitations of earlier studies while identifying emerging trends.
Therefore, they are better grounded in devising innovative and impactful research ideas.

Motivated by the human practices in conducting research, we introduce a novel Chain-of-Ideas (CoI)
agent framework to address the previously identified logical inconsistencies in the ideation processes
of LLMs. As shown in the bottom part of Figure 1, CoI agent aims to provide a clear landscape of
current research topics by systematically selecting and organizing the relevant papers and their ideas
in a chain structure. CoI agent offers several distinctive advantages: Firstly, it minimizes the risk
of interference from less relevant literature via carefully selecting papers (i.e. from CoT (Wei et al.,
2022) to GoT). Second, LLMs are demonstrated with human practice to craft a novel idea. For
example, SC (Wang et al., 2022) emerges as a novel idea derived from CoT. This can be viewed
as a form of few-shot prompting strategy, which has been proven to enhance the overall LLM’s
generation capability (Brown et al., 2020). Third, CoI exemplifies a global progression in research
development. As a result, LLMs can gain a deep understanding of the motivations behind these
developmental trends, facilitating the identification of promising future research directions.

Specifically, CoI agent first retrieves an anchor paper of the given research topic. Instead of indis-
criminately aggregating all papers within the citation network of the anchor, as done in (Baek et al.,
2024), we construct the CoI by selecting relevant and important literature from both the anchor’s
references and its subsequent works, thereby extending the chain backward and forward from the
anchor. We then prompt the constructed CoI to an LLM for idea generation and experiment design.
During idea generation, we require the LLM to predict possible future trends. This prognostic result
facilitates the gradual consolidation of the idea, beginning with the motivation for the proposed idea,
progressing through an assessment of its potential impact, and culminating in the realization. How-
ever, as the evolution of scientific discovery can emerge from multiple perspectives, a single CoI
may be insufficient to capture the most promising direction. Additionally, there is no guarantee that
the generated ideas will be novel. To address these issues, we construct multiple CoI branches for
different perspectives of a research topic. Additionally, a novelty-checker agent iteratively evaluates
the draft idea against existing literature and refines it if substantial similarity is identified.

We compare our CoI agent against existing baselines on idea generation in the artificial intelligence
(AI) field. To do this, we develop an arena-style evaluation framework called Idea Arena where
participant methods compete in pairs, which demonstrates high agreement with human evaluation.

2

Topic: Enhancing Large Language Model Problem-solving CapabilityChain of IdeasVanilla RAGTitle:EnhancingProblem-SolvingthroughMulti-ModalIntegrationforGoTPromptingMotivation:GoTfocusesontextualinputs,leavingthemulti-modalitydataunexplored.Thisworkexploreshowmulti-modalinputscanbeintegratedwithintheGoTprompting…Method:•Multi-ModalDataConversion toGraphNodes:Convertvisual,auditoryandtextualdataintographnodes…•GraphConstructionandIntegration:MotivatedbyGraphGPT,wecanemployGNNssuchasGraphSAGEorGATtoaggregateinformationfromthesemultimodalnodes…CoTSCToTGoTRAGRoGECOIGraphGPTSAASORALawyerGPT((Title:DynamicProblem-SpecificThoughtNetworkforEnhancingLLM’sProblem-SolvingMotivation:Thepre-definedstructuralconstraints(linear,tree,orgraph)maynotalwaysalignwiththenatureoftheproblembeingtackled.Therefore,amoreadaptableapproachthatdynamicallyadjustsitsstructurebasedontheproblemathandisneeded…Method:•ProblemAnalysis:Decidetheinitialreasoningstructureusingtheproblemdescription…•DynamicAdjustment:Monitorsthereasoningprocessanddynamicallyadjuststhestructurebasedonintermediateresultsandproblem-specificheuristics…CoTSCToTGoTRAGRoGECOIGraphGPTSAASORALawyerGPTUnder review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Figure 2: Our proposed CoI agent framework. The process consists of three stages: (1) Construct
CoIs based on the retrieved literature; (2) Develop potential ideas based on the CoIs; and (3) Design
the corresponding experiments for the proposed idea.

The experimental results show that CoI agent consistently ranks first among all automated baselines,
surpassing the second-best one by 56 ELO scores in human evaluation. CoI agent can generate ideas
as novel as those of human experts. Our analysis further shows that for LLMs to generate novel
ideas, a clear developmental trend analysis is more pivotal than the quantity of related literature.

Our contributions are summarized as follows: 1) We propose the CoI agent to enhance LLMs’ ca-
pability in idea generation. CoI agent organizes relevant literature in a chain structure to effectively
mirror the progressive nature of research development, allowing LLMs to better grasp the current re-
search advancements. 2) We propose Idea Arena for a comprehensive evaluation of idea-generation
methods, which shows high agreement with human researchers. 3) Extensive experiments demon-
strate the effectiveness of our CoI agent in generating ideas that are comparable to human creativity.

2 METHOD

2.1 FRAMEWORK: CHAIN-OF-IDEAS AGENT

In this section, we detail our CoI agent framework, as illustrated in Figure 2, which consists of
three stages: (1) CoI Construction, (2) Idea Generation, and (3) Experiment Design. First, given a
research topic, the CoI agent constructs multiple CoIs from existing literature, reflecting different
trends within the domain. Then, for each CoI, the LLM predicts future research directions, and
crafts ideas through step-by-step consolidation and iterative novelty checks. The best idea is then
selected. Lastly, the LLM generates and refines an experiment design to implement the final idea.

2.2 COI CONSTRUCTION

Generating novel research ideas requires a profound comprehension of the respective research do-
main, coupled with a rigorous reasoning process. Previous endeavors (Lu et al., 2024; Baek et al.,
2024) have sought to augment LLMs with relevant papers to facilitate the ideation process. However,
these methods simply mix these papers into the prompt without effective organization. This scenario
is akin to dropping an LLM at a chaotic intersection with no map in sight, leaving it uncertain about
which path to take. To address this issue, we propose a Chain-of-Ideas agent framework.

As shown in Figure 2, a CoI, represented as {I−M → · · · → I0 → · · · → IN }, is a sequence consist-
ing of M + N + 1 ideas extracted from M + N + 1 research papers respectively, where they together

3

Topic: Enhancing Large Language Model Problem-solving CapabilitySemanticScholarPaper 1(ToT)Paper 2Paper 3Stage 1: CoI ConstructionToTCoTGoTSCCurrent Trends: oCoT to SC: The progression from CoT to SC is marked by addressing the limitations of greedy decoding in complex reasoning tasks …oSC to ToT: …oToT to GoT: …Stage 2: Idea GenerationFuture Trend Prediction:Potential directions include adapting the task-solving framework according to the nature of the problem and reducing the computational costs of inference.Entities:oCoT Entities: …oSC Entities:…oToT Entities:…oGoT Entities:…CoI:Novel?NoIdea Consolidation:Title:Dynamic Problem-Specific Thought Network (DPSTN) …Motivation:The pre-defined structures (linear, tree, graph) may not align with the nature of the problem. Thus, we propose to dynamically adjusts its task-solving structure of problemMethods:…Final idea inspired by the CoIof Paper 1Final idea inspired by the CoI of Paper 2Final idea inspired by the CoI of Paper 3Final IdeaEntities:oCoT Entities: …oSC Entities:…oToT Entities:…oGoT Entities:…Previous Exp.:oCoT Exp.: …oSC Exp.:…oToT Exp.:…oGoT Exp.:…Designing:Step 1: Define Baselines:1. CoT prompting2. CoT with self-consistency…Step 2: Dataset Preparation…Step 3: Implement DPSTN …Clear?Supportive?FinalExperiment DesignStage 3: Experiment DesignYesNo⚓Current Trends:CoT→SC→ToT→GoTYes75%50%25%oIdea:Prompt LLM with reasoning steps…oExperiment:Appy CoTon arithmetic…oEntities:§GPT4: A strong LLM used in recent papers …oIdea:…oExperiment: …oEntities:…oIdea:…oExperiment: …oEntities:…oIdea:…oExperiment: …oEntities:…Under review as a conference paper at ICLR 2025

show the evolution progress within a given research field. Specifically, given an initial research
topic, we prompt the LLM to generate multiple queries, [q1, . . . , qK], that reflect K different per-
spectives of this topic. The prompt is given in Table 7 of Appendix. Unless otherwise specified, all
prompts of our framework are presented in the Appendix tables. The K queries are used to construct
K branches of CoI. This reduces the reliance on a single CoI that may be insufficient to capture the
most significant development and direction. For each query qk, we use it to retrieve a top-ranked
paper, which we call anchor paper Pk
0. In Figure 2, ToT (Yao et al., 2024) is an illustrative example
of an anchor paper. An anchor paper serves as the foundation for constructing a CoI. Specifically,
a CoI is constructed by extending from the corresponding anchor paper to related papers in both
directions: forward, tracing the progression of ideas, and backward, tracing their origins.

In the forward direction, starting from Pk
0, we identify subsequent papers that directly cite it by
leveraging the Semantic Scholar API2. We use OpenAI’s text-embedding-3-large3 to rank
these papers based on their cosine similarities to the concatenation of the initial research topic and
the abstract of the anchor paper. Subsequently, we select the highest-ranked paper as Pk
1 to extend
the CoI in the forward direction (e.g. GoT in Figure 2). This process is repeated iteratively from Pk
i
to Pk
i+1, until either the length of the CoI reaches a preset value or the LLM finds that there is no
valuable follow-up work (Table 8).

0 directly built upon, 2) references that serve as baselines in Pk

In the backward direction, starting from the anchor paper Pk
0, we instruct an LLM to thoroughly
review the full paper and to identify candidate references based on the following criteria: 1) refer-
ences that Pk
0, and 3) references that
tackle the same topic as Pk
0. With those candidate references, we ask the LLM to determine the most
relevant one to the anchor paper (Tables 9 and 10), denoted as Pk
−1 (e.g. SC in Figure 2), to extend
the CoI backward. This backward extension is also carried out iteratively from Pk
−(i+1) to
identify preceding papers (e.g. tracing backward from SC to CoT in Figure 2). It terminates when
the length of CoI reaches a preset value or we encounter a milestone paper (defined as one with over
1,000 citations), indicating that the idea from the milestone paper could serve as a strong starting
point for the CoI. Additionally, we instruct the LLM to terminate the search if no reference relevant
to the original research topic is found (Table 8).

−i to Pk

−M k → · · · → Ik

−M k → · · · → Pk

After we collect K paper chains, denoted as {Pk
k=1, we ask the
LLM to extract ideas from these papers and inherit the progressive relation of the paper chains to
form our CoIs {Ik
N k }K
k=1 (Tables 9 and 10). Then for each CoI, we
ask the LLM to summarize the existing research trends by analyzing the evolution between any two
adjacent ideas (Table 11). For example, the upper part of Figure 2 shows the evolution process from
CoT to GoT step-by-step. Additionally, we extract experiment designs and the definition of key
entities from these papers (Tables 9 and 10). The above information including CoIs and the derived
knowledge will be used in the following idea generation and experiment design stages.

0 → · · · → Pk

0 → · · · → Ik

N k }K

2.3

IDEA GENERATION

In this section, we use the above-constructed CoIs and their developing trends to guide the generation
of a novel idea. For each generated CoI, the first step is to predict possible future trends. As shown
in the lower-left section of Figure 2, we prompt the LLM with the CoI, the developing trends of
existing works, and the key entities extracted from existing literature, as described in Sec. 2.2
(Tables 12 and 13). These entities comprise relevant datasets and potential baseline models, which
are important to clarify the concepts mentioned in the existing literature. After obtaining the future
trend, we continue to prompt the LLM to articulate its motivation, novelty, and methodology, finally
consolidate the idea (Tables 14 and 15). Through this step-by-step manner, COI can produce a more
detailed idea. Following the previous practice (Wang et al., 2023; Lu et al., 2024), we also use a
novelty-check agent to evaluate candidate ideas. It retrieves relevant papers and prompts another
LLM to assess the similarity between the generated idea and the retrieved papers (Table 16). Based
on this assessment, our framework determines if another round of generation is necessary. Finally,
we pairwisely compare the generated ideas from all CoI branches and select the one with the highest

2https://www.semanticscholar.org/product/api
3https://openai.com/index/new-embedding-models-and-api-updates/

4

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

winning rate as the final idea for the experiment design. This pairwise comparison follows the same
method as Idea Arena, refer to Sec. 3.4 for details.

2.4 EXPERIMENT DESIGN

While our primary goal is to generate novel ideas, it is also useful to develop experimental plans that
help users implement these ideas. Thus, we extended the CoI agent to include experiment design.
As shown in the lower-right of Figure 2, we prompt the LLM with experiments from existing works
obtained from Sec. 2.2 as few-shot examples, along with the proposed idea and key entities, to guide
the LLM in designing experiments for our ideas (Table 17).

We also employ a review agent to assess the candidate experiment designs.
Its main role is to
evaluate the clarity and comprehensiveness of the protocol, ensuring all key elements—such as
datasets and models—are clearly specified. Additionally, it checks if the design provides enough
detail for practical implementation (Table 18). The review agent provides critical feedback on these
aspects, subsequently utilizing this information to conduct further searches for relevant literature
(Table 19) to help the LLM refine and enhance its previous experiment design (Table 20). Through
this iterative process of review and refinement, we arrive at a final experiment design.

3 EXPERIMENTAL SETUPS

3.1

IMPLEMENTATIONS

In our CoI agent, we primarily use GPT-4o (05-13) as our LLM implementation. For some modules
that require full-paper understanding, we use GPT-4o-mini (07-18) to read the paper and summarize
the core contents due to its lower price and good summarization capability. We use Semantic Scholar
as our academic search engine. For the main experimental results, the maximum length of the CoI
is set to 5 and the number of CoI branches is set to 3, and their analysis results are given later. The
iteration number of self-refinement in the experiment design stage is set to 1 for cost saving.

3.2 DATA

To evaluate our CoI agent’s ability to generate novel ideas, we collect recent research topics from
Hugging Face’s Daily Papers4, known for its timely updates and the high quality of the featured
papers. We select papers submitted between August 1 and September 15, 2024, ensuring that the
topics are sufficiently new and the time frame is after the data cutoff of the LLM. We ask 10 skilled
researchers (All have publications in top-tier conferences and major in AI-related topics, such as
computer vision, embodied intelligence, and natural language processing) to identify papers that
capture their interests. Subsequently, we prompt GPT-4o to extract research topics, proposed ideas,
and their corresponding experiment designs from these selected papers (Tables 21, Table 22 and
23). The extracted topics will then be returned to the researchers for validation, ensuring that the
extracted topics are valid and reasonable within their research domains. The extracted ideas and
experiment designs will be utilized as our Real Paper baseline, as described in Section 3.3. Due
to the substantial costs associated with generating and evaluating ideas and experiment designs, we
adhere to the assessment scale of Lu et al. (2024); Wang et al. (2023) to collect 50 research topics in
total for evaluation.

3.3 BASELINES

We compare our CoI agent with recent works on idea generation and experiment design. To ensure
a fair comparison, we employ GPT-4o and Semantic Scholar as the LLM and academic retriever
implementations, respectively, across all baseline methods. Furthermore, we unify the output format
of the generated ideas and experiment designs to minimize evaluation preference towards more
structured outputs (Chiang et al., 2024). We compare with the following baselines:

• RAG: This is a vanilla retrieval augmented generation approach (Lewis et al., 2020), where we

directly prompt the LLM with retrieved literature for idea generation and experiment design.

4https://huggingface.co/papers

5

Under review as a conference paper at ICLR 2025

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Figure 3: Evaluation results of idea gen-
eration with LLM as a judge.

Figure 4: Evaluation results of idea gen-
eration with human as judges.

• ResearchAgent (Baek et al., 2024): This work leverages an additional academic knowledge graph
for enhancing the literature retrieval and adopts a multi-agent framework to refine ideas through
peer discussions iteratively. We follow the original paper to reproduce this baseline.

• GPT-Researcher (Assafelovic, 2023): GPT-Researcher is an agent framework specifically de-
signed for the research domain. The agent is enhanced with plan-and-solve and RAG capabilities.
• AI-Scientist (Lu et al., 2024): This work originally aims to generate the entire paper with the
idea, methods, and experimental results. We extract the components related to idea generation
and experiment design to serve as our baseline.

• Real Paper: Note that, in Sec. 3.2, we extract topics from existing research papers. Therefore,
the ideas and the experiment designs from these papers serve as a natural baseline to quantify the
gap between model-generated ideas and genuine human ideas.

3.4 EVALUATION: IDEA ARENA

Model-based Evaluation. The open-ended nature of idea generation poses challenges for automatic
evaluation. Prior work primarily uses LLM-based Likert scale system to score ideas (Baek et al.,
2024; Lu et al., 2024). However, Si et al. (2024) show this method poorly aligns with human
preferences. Instead, they show LLMs perform better in ranking ideas. To obtain reliable scores for
evaluation, we propose Idea Arena, a pairwise evaluation system using a Round-Robin tournament
to compute ELO scores for each idea-generation method. For a given topic, we require the LLM
judge to rank the ideas generated by any pair of methods (Table 24). We evaluate each pair twice
with order reversed to reduce the position bias. To comprehensively evaluate an idea from multiple
perspectives, we incorporate criteria from ICML 2020 review guidelines 5, and those in Si et al.
(2024), which consist of Novelty, Significance, Clarity, Feasibility, and Expected Effectiveness.
Finally, the resultant win-loss-tie records are utilized to calculate the ELO scores for each method,
following the practices outlined in Zheng et al. (2024); Zhao et al. (2024). We also evaluate the
experiment design in the same pairwise way, focusing on Feasibility, Technical Quality, and Clarity.
Refer to Definitions for all metrics in Tables 5 and 6 of the Appendix.

Human Evaluation. The 10 AI researchers who review the extracted topics are asked to rank two
ideas and experiment designs based on the same pairwise criteria as the model-based evaluation. To
ensure fairness, we anonymize the source of the ideas by concealing the method identity.

4 RESULTS

4.1

IDEA GENERATION

Main results. Figures 3 and 4 present the results of idea generation evaluated by both a LLM
(specifically, GPT-4o) and human researchers. Detailed scores are in Table 26 of Appendix. Over-

5https://icml.cc/Conferences/2020/ReviewerGuidelines

6

AverageNoveltySignificanceClarityFeasibilityEffectiveness700800900100011001200CoI Agent (ours)Real PaperResearchAgentGPT-ResearcherAI-ScientistRAGAverageNoveltySignificanceClarityFeasibilityEffectiveness700800900100011001200CoI Agent (ours)Real PaperResearchAgentGPT-ResearcherAI-ScientistRAGUnder review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

all, our CoI agent performs better than all other automated methods in both model- and human-based
evaluations. Notably, It substantially outperforms the second-best baselines, GPT-Researcher and
RAG, by margins of 108 and 56 ELO scores, respectively, in the two evaluation settings. Our CoI
agent’s performance is on par with that of the Real Paper baseline and even excels in the metrics of
Novelty and Significance. These results highlight its exceptional capabilities in idea generation. Fur-
thermore, CoI demonstrates superior performance in Clarity, Feasibility, and Expected Effectiveness
compared to other automated methods in human evaluation. Nevertheless, it still lags considerably
behind the Real Paper in these areas. This substantial gap between automatic methods and Real
Paper is expected, as Real Paper ideas undergo extensive experimental validation. Additionally,
AI-Scientist’s performance is especially low, likely due to its original design, which focuses on
generating full papers from executable code. When given only a research topic, its simplistic idea
generation framework limits its ability to produce novel and feasible ideas.

Table 1: Agreement between the human and GPT-4o judges in all evaluated dimensions.

Novelty

Significance Clarity Feasibility Effectiveness Average

Agreement

66.5%

71.0%

76.3%

70.2%

71.0%

70.8%

Human-Model Agreements of Idea Arena. To assess
the reliability of our model-based evaluation within Idea
Arena, we analyze the agreements between the prefer-
ences of the human judges and the LLM judges. We
follow Zheng et al. (2024) to compute the agreement,
which is defined as the probability that two judges agree
on the winner of one specific arena match. Figure 5
shows the pairwise agreement between humans and sev-
eral state-of-the-art LLMs, including GPT-4o, Gemini-
1.5-Pro-Exp-08276, and Claude-3.5-Sonnet7. We observe
an average agreement of 70.8% between GPT-4o and hu-
mans. This finding indicates a strong alignment between
human-based and model-based evaluations , approaching
the level of agreement seen in human-to-human evalua-
tions (Si et al., 2024), thereby highlighting the robustness
of Idea Arena in evaluating the quality of generated re-
search ideas (More correlation results can be found in
Figure 8 and Figure 9). Moreover, GPT-4o demonstrates
the highest level of agreement with humans among all tested LLMs. Therefore, we will utilize GPT-
4o as the LLM judge for subsequent analytical experiments. Additionally, we present the agreement
on individual criteria between GPT-4o and human evaluators in Table 1. The results indicate a
consistently high level of agreement across all assessed criteria.

Figure 5: Agreements between human
and LLM judges.

4.2 ABLATION STUDIES FOR IDEA GENERATION

We conduct an ablation study to assess the contributions of each component of the CoI Agent to idea
generation quality. The following variants are examined: 1) – CoI: Excludes the CoI construction
stage, directly using all retrieved literature without progressive relation mining. 2) – Future Trend:
Omits the Future Trend Prediction module, prompting the LLM to consolidate ideas directly based
on the provided input information. 3) – Entities: Skips inputting entity definitions during idea
generation.To ensure fair comparison, each variant is scored against the full CoI Agent, with 2/1/0
points for win/tie/lose in 50 matches, for a maximum of 100 points.

Results in Table 2 show that all variants negatively affect idea quality. Excluding the CoI con-
struction stage has the most significant impact, emphasizing the importance of organizing literature
based on progressive relationships to enhance the LLM’s understanding of trends. Removing the
Future Trend Prediction reduces novelty, as the LLM lacks insight into potential forward-thinking
ideas. Although slight improvements in clarity and feasibility are observed, these are not substantial,

6https://ai.google.dev/gemini-api/docs/models/experimental-models
7https://www.anthropic.com/news/claude-3-5-sonnet

7

HumanGPT-4oGemini-1.5-proClaude-3.5HumanGPT-4oGemini-1.5-proClaude-3.5100.0%70.8%69.3%70.1%70.8%100.0%90.1%92.9%69.3%90.1%100.0%91.8%70.1%92.9%91.8%100.0%0.20.00.20.40.60.81.0Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Table 2: Ablation study on the design of CoI agent. The original CoI agent gets 50 points because it
receives 50 ties after battling with itself.

Novelty

Significance Clarity

Feasibility Effectiveness Average

CoI Agent

– CoI
– Future Trend
– Entities

50

41
40
46

50

39
43
49

50

44
51
42

50

49
53
47

50

39
44
43

50

42.4
46.2
45.4

likely due to evaluation variability. Finally, omitting entity information reduces clarity and effec-
tiveness, as the LLM generates more abstract ideas without grounding in specific concepts. This
highlights the value of entity information in enhancing the clarity and practical relevance of ideas.

4.3 CASE STUDY

We present an intriguing case study in Table 3 with the same topic of our paper – generating novel
research ideas using LLMs. Given the input topic, our CoI agent first constructs the chain of ideas,
extending I0 (Baek et al., 2024) in both forward and backward directions. Then the agent analyzes
current research trends for any two adjacent ideas. For instance, it identifies that the core develop-
ment from I−1 to I0 is the generation of ideas rather than hypotheses. After digesting the existing
trends, the CoI agent realizes that LLMs have great potential in idea generation but are limited in
novelty and diversity. Therefore, it proposes an evolutionary algorithm, which specifically models
the variations between parents and children, as a possible future trend for novel and diverse idea
generation. Finally, the agent consolidates its final idea by drawing on future trends and with practi-
cal implementations, such as crossover and mutation, to ensure effective realization. Therefore, the
generated idea is viable and novel, deserving further exploration in our future work.

4.4 EXPERIMENT DESIGN

As a byproduct of idea generation, we also require these
baselines to develop potential experiment designs for re-
alizing their proposed ideas. Table 4 presents the arena-
style results for experiment designs for both model-based
and human-based evaluations. Our CoI Agent demon-
strates superior performance across all evaluated criteria
in two evaluation settings, achieving the highest scores
among all automated methods. Notably,
it surpasses
RAG, the second-best automated method, by 70 ELO
points in human evaluation. Furthermore, there is a high
degree of model-human agreement in the experimental
designs. Despite the clarity and reasonable technical de-
tails of the experiment designs produced by the CoI Agent
in support of the proposed ideas, they tend to be less fea-
sible compared to those designs in the existing literature.
This phenomenon is also observed during the idea gener-
ation phase. Consequently, feasibility represents a signifi-
cant bottleneck in automatic idea generation, highlighting
the need for future research to address this challenge.

4.5 LENGTH OF COI

To examine the impact of the CoI length on the quality
of generated ideas, we constructed variants with differing
maximum chain lengths. Furthermore, we also adopt the
“- CoI” variant in Sec. 4.2 as a 0-length variant, which
uses 5 retrieved papers but does not organize them in a
chain structure. Figure 6 presents the idea arena results

8

Table 4: Results of experiment design
of both model and human evaluations,
as well as their agreements. Tech. refers
to the Technical Quality criterion.

Feasibility Tech. Clarity Average

n
o
i
t
a
u
l
a
v
E

l
e
d
o
M

n
o
i
t
a
u
l
a
v
E
n
a
m
u
H

Real Paper

CoI Agent (ours)

RAG

ResearchAgent

GPT-Researcher

AI-Scientist

Real Paper

CoI Agent (ours)

RAG

GPT-Researcher

ResearchAgent

AI-Scientist

1100

1029

1022

960

1001

888

1138

1092

1035

988

939

809

1122

1096

970

1020

965

827

1090

1043

1016

980

992

879

1111

1111

1123

1041

977

959

788

1121

1048

971

964

785

1103

1056

1003

987

986

865

1120

1112

1042

978

954

794

Agreement

70.7% 75.9% 72.1% 73.0%

Figure 6: Length analysis of the CoI.

03456Length94096098010001020ELO ScoresUnder review as a conference paper at ICLR 2025

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Table 3: Case study for the entire idea generation pipeline of our CoI agent.

Input topic: Using LLM agent to generate novel and original research ideas without human participation

Chain of ideas:
• I−3 (Kim et al., 2021): It addresses the challenge of discovering new materials through molecular generation.

It introduces GCT, a Transformer with a variational autoencoder, to generate SMILES strings . . .

• I−2 (Boiko et al., 2023): It explores the capabilities of LLM in designing, and executing experiments for
scientific research. This work presents a multi-LLM agent to autonomously execute complex scientific ex-
periments via internet browsing, documentation searching, and hands-on experimentation . . .

• I−1 (Yang et al., 2024b): It proposes a new dataset for social science hypotheses and develops a MOOSE

framework with LLM prompting and feedback mechanisms to facilitate hypothesis generation . . .

• I0 (Baek et al., 2024): It proposes a ResearchAgent framework for automatic idea generation. ResearchA-
gent combines LLMs with an entity-centric knowledge graph and iterative feedback from reviewing agents,
creating a structured and dynamic process for generating and refining research ideas . . .

• I1 (Si et al., 2024): The paper explores the capabilities of LLMs in generating novel research ideas and
presents a large-scale comparison between LLM-generated ideas and those produced by 100 NLP expert
researchers, revealing that LLMs can produce ideas deemed more novel than human-generated ideas . . .

Current Trends:
• I−3 → I−2: The progression from I−3 to I−2 marks a significant shift from the application of neural models

for molecular generation to the broader scope of automating scientific research using LLMs . . .

• I−2 → I−1: The transition from I−2 to I−1 focuses on refining the autonomous induction capabilities of

LLMs, specifically in generating novel and valid scientific hypotheses . . .

• I−1 → I0: I0 builds on the advancements made in I−1 by further extending the process of generating

hypotheses to generating and refining research ideas autonomously . . .

• I0 → I1: The transition from I0 to I1 emphasizes the importance of empirical validation of LLMs in

generating novel research ideas and highlights the potential of LLMs to contribute to ideation . . .

Future Trend Prediction: Given the previous research’s progression and the identified gaps, a promising
direction is to unleash the potential of LLM in ideation. We can develop a multi-agent system that leverages
evolutionary algorithms to enhance the diversity and novelty of LLM-generated research ideas . . .

Final Idea: EvoResearchAgent: Enhancing Diversity and Novelty in Idea Generation with Evolution
• Motivation: Using LLMs for idea generation has shown promising advancements. However, challenges
persist, particularly concerning the diversity and novelty of LLM-generated ideas. Si et al. (2024) show that
while LLMs can produce novel ideas, they often lack a broad range of perspectives and diversity. Ad-
ditionally, Baek et al. (2024) have emphasized the need for a more systematic approach to improving
the quality of generated ideas. To address these issues, we propose EvoResearchAgent, a multi-agent
system that leverages evolutionary algorithms to enhance the diversity and novelty of generated ideas . . .

• Method:

◦ Idea Initialize: An LLM generates some initial ideas as the start point of the evolutionary process . . .
◦ Metrics: Propose automatic metrics like topic diversity and novelty to evaluate the range of ideas . . .
◦ Evolution Integration:

1. Selection: Select the top ideas based on predefined novelty and diversity metrics.
2. Crossover: Combine elements of two high-scoring ideas to create new hybrid ideas.
3. Mutation: Introduce small changes to existing ideas for new possibilities and diversity.
4. Iteration: Repeat the selection, crossover, and mutation process iteratively . . .

among these length variants. We observe a substantial improvement of idea-generation quality when
we increase the length from 0 to 3. This indicates a clear developmental trend analysis is more
pivotal than the quantity of related literature. Furthermore, the quality of generated ideas continues
to improve as the length of the CoI increases. Longer CoIs offer more reliable and comprehensive
insights into the evolving trends within the current research domain, thereby enabling the LLM to
better capture future development trends. The quality of generated ideas levels off after reaching a
maximum length of 5. This saturation point indicates that this length is sufficient to capture relevant
trends, with additional literature offering diminishing returns.

4.6 WIDTH OF COI

We also assess the impact of the width of CoI (i.e., the branch number K) on the quality of
generated ideas. Figure 7 shows the trend of average ELO scores with varying branch num-
bers. Generally, increasing the branch numbers shows a positive correlation with idea quality.

9

Under review as a conference paper at ICLR 2025

However, the disparity in ELO scores across different
branch numbers is small. This phenomenon is likely at-
tributed to the fact that generating multiple chains primar-
ily helps reduce the impact of any single CoI performing
poorly. Fortunately, such low-quality CoIs are rare.

5 RELATED WORKS

Figure 7: Width analysis of the CoI.

Scientific Research Idea Generation. Idea generation is a fundamental step in scientific research.
Due to its innovative nature, idea generation has been primarily a human-driven activity. However,
recent studies indicate that LLMs can generate plausibly novel and feasible ideas as those of human
researchers (Si et al., 2024; Kumar et al., 2024). To investigate the potential of LLMs in idea gen-
eration, prior works begin with the task of scientific hypothesis discovery (Yang et al., 2024b; Qi
et al., 2023; Wang et al., 2023), which aims to elucidate relationships between two scientific vari-
ables. Despite its utility, scientific hypothesis discovery may not fully capture the complexity and
multifaceted nature of real-world problems. To address this limitation, projects like GPT-Researcher
(Assafelovic, 2023) and ResearchAgent (Baek et al., 2024) have adopted a more open-ended idea
generation scenario including the underlying methodologies and experimental designs. They lever-
age agent-based systems to enhance the quality of idea generation. Beyond ideation, numerous
studies also explore the use of LLMs for executing experiments (Huang et al., 2024; Tian et al.,
2024) or combining both idea generation and experimental execution (Li et al., 2024; Lu et al.,
2024). However, these approaches often make minor modifications to existing ideas for drafting
their ideas, which often lack depth and creativity.

Align LLMs with Human Cognitive Patterns. As LLMs are trained with vast amounts of human
data (Brown et al., 2020), this may enable them to internalize human cognitive patterns. Firstly,
CoT (Wei et al., 2022) indicates that LLMs can enhance their reasoning abilities when provided
with step-by-step guidance. Further research supports this notion by showing that simply prompting
LLMs to engage in step-by-step reasoning can trigger better reasoning capability (Kojima et al.,
2022). Additionally, Fu et al. (2022) reveals that in-depth reasoning of LLMs can be achieved with
more elaborate prompts. As a result, a prompting strategy that closely emulates human cognition is
likely to elicit more insightful responses from these models. Motivated by this, we propose CoI to
better mimic the progressive cognitive patterns of humans when generating new research ideas.

6 ETHIC DISCUSSION

The misuse of AI-generated research ideas could present a risk to our society. We believe this is a
fundamental limitation inherent in all generative models, not just an issue specific to our CoI. Con-
sequently, we advocate for the continuation of safety research specifically focused on the academic
domain. As for this paper, our primary goal is to enhance effectiveness, while safety issues are re-
ally out of this scope. Nevertheless, we still try to test the safety capability of our framework. The
analysis, detailed in Appendix A.2, shows that CoI does not compromise the safety alignment of
existing LLMs, thereby making it a safe and reliable framework for idea generation.

7 CONCLUSIONS

In this paper, we introduce Chain of Ideas (CoI) agent, a framework designed to enhance the capa-
bility of LLMs in generating research ideas. The CoI agent offers a promising and concise solution
by organizing ideas into a chain structure, effectively mirroring the progressive development within
a given research domain. It facilitates LLMs to digest the current advancements in research, thereby
enhancing their ideation capabilities.p To comprehensively evaluate the capability of automated idea
generation methods, we also propose Idea Arena, an evaluation system that requires the participant
methods to compete in pairs about their generated ideas for the research topics, which demonstrates
high agreement with human evaluation. Experimental results indicate that the CoI agent consistently
outperforms other methods and is capable of generating ideas comparable to human creativity.

10

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

1234Numbers94096098010001020ELO ScoresUnder review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale-
man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical
report. arXiv preprint arXiv:2303.08774, 2023.

Assafelovic.

gpt-researcher, 2023.

URL: https://github.com/assafelovic/

gpt-researcher.

Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative
research idea generation over scientific literature with large language models. arXiv preprint
arXiv:2404.07738, 2024.

Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gian-
inazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of
thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI
Conference on Artificial Intelligence, volume 38, pp. 17682–17690, 2024.

Daniil A Boiko, Robert MacKnight, and Gabe Gomes. Emergent autonomous scientific research

capabilities of large language models. arXiv preprint arXiv:2304.05332, 2023.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari-
wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar-
wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh,
Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz
Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec
Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners.
In
H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu-
ral Information Processing Systems, volume 33, pp. 1877–1901. Curran Associates, Inc.,
URL https://proceedings.neurips.cc/paper_files/paper/2020/
2020.
file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared
Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large
language models trained on code. arXiv preprint arXiv:2107.03374, 2021.

Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li,
Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica.
Chatbot arena: An open platform for evaluating llms by human preference, 2024.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha
Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.
arXiv preprint arXiv:2407.21783, 2024.

Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting
for multi-step reasoning. In The Eleventh International Conference on Learning Representations,
2022.

Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentbench: Evaluating language
agents on machine learning experimentation. In Forty-first International Conference on Machine
Learning, 2024. URL https://openreview.net/forum?id=1Fs1LvjYQW.

Hyunseung Kim, Jonggeol Na, and Won Bo Lee. Generative chemical transformer: neural machine
learning of molecular geometric structures from chemical language via attention. Journal of
chemical information and modeling, 61(12):5804–5814, 2021.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large
language models are zero-shot reasoners. Advances in neural information processing systems,
35:22199–22213, 2022.

Sandeep Kumar, Tirthankar Ghosal, Vinayak Goyal, and Asif Ekbal. Can large language models

unlock novel scientific research ideas? arXiv preprint arXiv:2409.06185, 2024.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal,
Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented genera-
tion for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:
9459–9474, 2020.

Ruochen Li, Teerth Patel, Qingyun Wang, and Xinya Du. Mlr-copilot: Autonomous machine learn-
ing research based on large language models agents. arXiv preprint arXiv:2408.14033, 2024.

Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scien-
tist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292,
2024.

Biqing Qi, Kaiyan Zhang, Haoxiang Li, Kai Tian, Sihang Zeng, Zhang-Ren Chen, and Bowen Zhou.
Large language models are zero shot hypothesis proposers. arXiv preprint arXiv:2311.05965,
2023.

Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-

scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109, 2024.

Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang.
Graphgpt: Graph instruction tuning for large language models. In Proceedings of the 47th In-
ternational ACM SIGIR Conference on Research and Development in Information Retrieval, pp.
491–500, 2024.

Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland
Haas, Pan Ji, Kittithat Krongchon, Yao Li, et al. Scicode: A research coding benchmark curated
by scientists. arXiv preprint arXiv:2407.13168, 2024.

Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. Scimon: Scientific inspiration machines

optimized for novelty. arXiv preprint arXiv:2305.14259, 2023.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh-
ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.
arXiv preprint arXiv:2203.11171, 2022.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny
Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in
neural information processing systems, 35:24824–24837, 2022.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li,
arXiv preprint

Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.
arXiv:2407.10671, 2024a.

Kaiyu Yang, Aidan M Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil,
Ryan Prenger, and Anima Anandkumar. Leandojo: Theorem proving with retrieval-augmented
In Thirty-seventh Conference on Neural Information Processing Systems
language models.
Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum?id=
g7OX2sOJtn.

Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. Large lan-
guage models for automated open-domain scientific hypotheses discovery. In Lun-Wei Ku, An-
dre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Lin-
guistics ACL 2024, pp. 13545–13565, Bangkok, Thailand and virtual meeting, August 2024b.
Association for Computational Linguistics. URL https://aclanthology.org/2024.
findings-acl.804.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik
Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad-
vances in Neural Information Processing Systems, 36, 2024.

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhen-
guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions
for large language models. arXiv preprint arXiv:2309.12284, 2023.

12

Under review as a conference paper at ICLR 2025

Ruochen Zhao, Wenxuan Zhang, Yew Ken Chia, Deli Zhao, and Lidong Bing. Auto arena of llms:
Automating llm evaluations with agent peer-battles and committee discussions. arXiv preprint
arXiv:2405.20267, 2024.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang,
Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and
chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.

Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang,
Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applica-
tions. AI open, 1:57–81, 2020.

13

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Under review as a conference paper at ICLR 2025

A APPENDIX

A.1 EVALUATION METRICS

Evaluation criteria for generated ideas include several key aspects. Novelty and Significance are
adapted from the ICML 2020 reviewer guidelines, with specific experimental evaluation standards
removed. Effectiveness is assessed with reference to AI-Researcher Si et al. (2024), while Feasi-
bility is tailored specifically for the task of Idea generation. Clarity is also sourced from the ICML
2020 reviewer guidelines. For the evaluation of experiment design, the criteria consist of Quality,
extracted from the Technical Quality section of the ICML 2020 guidelines with specific results-
oriented standards omitted, as well as Clarity, again based on ICML 2020 guidelines. Feasibility is
designed specifically for the task of experiment design generation.

Metric

Novelty

Significance

Clarity

Feasibility

Table 5: Evaluation metrics of ideas.

Definition

Are the problems or approaches new? Is this a novel combination of
familiar techniques? Is it clear how this work differs from previous con-
tributions? Is related work adequately referenced?

Are the idea important? Are other people (practitioners or researchers)
likely to use these ideas or build on them? Does the idea address a dif-
ficult problem in a better way than previous research? Does it provide a
unique theoretical or pragmatic approach?

Is the paper clearly written? Is it well-organized? Does it adequately
inform the reader?

Can the idea be realized with existing technology or methods? Are there
any technical difficulties or bottlenecks? Is the idea clear and logical?
Is there any obvious error or unreasonable part in the idea, and can the
experiment be designed normally according to this idea.

Expected Effectiveness How likely the proposed idea is going to work well (e.g., better than

existing baselines).

Table 6: Evaluation metrics of experiment design.

Metric

Definition

Feasibility Can the experiment be realized with existing technology or methods? Are there any
technical difficulties or bottlenecks? Is the experimental plan detailed and feasible?
Are the experimental steps clear and logical? Is there any obvious error or unreason-
able part in the experiment. Consider the rationality of its steps and the possibility that
the idea can be successfully implemented.

Quality

Clarity

Is there a clear rationale for each step of the experimental design? Are the baseline
and evaluation metrics chosen appropriately? Has the design taken into account the
potential advantages and limitations of the methods used? Can this experimental de-
sign effectively support the claims made in the idea.

Is the experimental plan clearly written? Dose it provide enough information for the
expert reader to understand the experiment? Is it well organized? Does it adequately
inform the reader?

A.2 ETHIC RESULTS

To test if CoI will generate unsafe research ideas, we try two unsafe topics: "Artificial intelligence
weaponization", and "Development of highly addictive and lethal drugs". For each topic, we gener-
ate 10 ideas.

14

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Under review as a conference paper at ICLR 2025

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Among 10 ideas about "artificial intelligence weaponization", four of them focus on the ethical
issues surrounding AI weapons, such as establishing guidelines for their use, enhancing account-
ability and oversight mechanisms, and preventing ethical dilemmas. Another four ideas address the
enhancement of safety in the use of AI weapons, including methods to distinguish between civilians
and combatants, increase human involvement, and build robustness against errors. The remaining
two ideas discuss ways to increase the transparency of AI weapons and improve their interpretability
to ensure compliance with international humanitarian law.

Among 10 ideas about "Development of Highly Addictive and Lethal Drugs", six ideas focus on
researches on predicting and preventing addictive behaviors. The remaining four ideas concentrate
on predicting and preventing substance abuse among youth in the community and treating addictive
behaviors.

It can be observed that even when CoI is presented with potentially unsafe topics, it consistently
suggests safe and reliable ideas. This is partly because most current LLMs have undergone safety
alignment. Additionally, the construction process of CoI involves searching for publicly available
research papers on the internet and conducting further research based on them. The majority of
accessible papers tend to present positive perspectives, which in turn guides CoI to propose ideas
that are more in line with ethical standards.

A.3 SPECIFIC PROMPTS

Here are the prompts used in this paper.

• Prompts used in CoI construction

– Prompt used to convert a topic into a search query for literature retrieval (Table 7)
– Prompt used to evaluate whether a paper is relevant to the topic (Table 8)
– Prompt used to extract idea, experiment, entities and references from paper (Table 9)

and 10

– Prompt used to summarize current trends of CoI (Table 11)

• Prompts used in idea generation

– Prompt used to predict future trend (Table 12 and 13)
– Prompt used to generate idea (Table 14 and 15)
– Prompt used to check the novelty of the idea (Table 16)

• Prompts used in experiment design

– Prompt used to generate experiment design (Table 17)
– Prompt used to review experiment design (Table 18)
– Prompt used to get queries for search paper to refine experiment design (Table 19)
– Prompt used to refine experiment (Table 20)

• Prompts used in benchmark construction

– Prompt used to extract topic from real paper (Table 21)
– Prompt used to extract the idea from real paper (Table 22)
– Prompt used to extract the experiment design from real paper (Table 23)

• Prompts used in idea arena

– Prompt used to compare two ideas (Table 24)
– Prompt used to compare two experiment designs (Table 25)

A.4 ADDITIONAL EXPERIMENT RESULTS

We present the evaluation results of idea generation for both model-based evaluation (including
GPT-4o, Gemini-1.5-Pro-Exp-0827, and Claude-3.5-Sonnet) and human-based evaluation in Table
26.

We also conducted a consistency analysis of Spearman and Pearson correlation coefficients. Specif-
ically, we utilized the ELO scores/rankings assigned by two judges to these baselines to compute

15

Under review as a conference paper at ICLR 2025

the Pearson and Spearman correlations for each evaluated dimension. We then averaged the scores
across all dimensions to determine the final correlation between the two judges. The detailed results
are illustrated in figure 8 and figure 9.

Table 7: Prompt used to convert a topic into a search query for literature retrieval

You are a master of literature searching, tasked with finding
relevant research literature based on a specific topic.

Currently, we would like to study the following topic: [Topic]
Please provide the literature search queries you would use to
search for papers related to the topic and idea.
Each query should be a string and should be enclosed in double
quotes.
other queries representing different aspects of the whole.

It is best to output one query representing the whole and

Output strictly in the following format:
Queries:

...

Table 8: Prompt used to evaluate whether a paper is relevant to the topic

You are an expert researcher tasked with evaluating whether a given
paper is relevant to our research topic based on its title and
abstract.

[Title]

Below are the details of the paper you need to assess:
Title:
Abstract: [Abstract]
The topic is: [Topic]
If the paper title and abstract are related to the topic, output
1; otherwise, output 0.
reference value for your question, you can use it to help you study
the topic, it does not need to be completely consistent in topic.

As long as you feel that this article has

Please follow the strict format below:
Think:
Relevant:

...

0/1

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

Table 9: Prompt used to extract idea, experiment, entities and references from paper (part I)

You are a scientific research expert, tasked with extracting and
summarizing information from provided paper content relevant to
the topic: [Topic].
references, extracted entities, a detailed summary, and the
experimental design.

Your deliverables will include pertinent

Format the entities with a name followed by a brief

Ensure all entities are relevant to the specified topic

Identify unique entities mentioned in the paper, such as model

The topic you are studying is: [Topic] (Ensure that the references
are pertinent to this topic.)
Extraction Requirements:
Entities:
1.
names, datasets, metrics, and specialized terminology.
2.
description.
3.
([Topic]).
Summary Idea:
1.
outlining the starting point of this paper.
2.
this paper in comparison to prior work.
3.
theory and functions of each core component.
4.
chosen methods are effective, including implementation details for
further research.
5.
Continue to next table →

Describe the main innovations and contributions of

Elaborate on the task’s context and previous work,

Explain the primary methods used, detailing the

Discuss current shortcomings of the approach.

Provide a thorough explanation of why the

Detail Reason:

Contribution:

Background:

Limitation:

Novelty:

Figure 8: Pearson correlation coefficient of
evaluation results of different judges

Figure 9: Spearman correlation coefficient of
evaluation results of different judges

17

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Under review as a conference paper at ICLR 2025

Table 10: Prompt used to extract idea, experiment, entities and references from paper (part II)

Baseline:

Verification:

Clarity of Plan:

Method Relevance:

Experimental Process:

Detail the entire experimental

Describe any specific technologies

State your experimental plan concisely to

Explain how your experimental design assists in

Elaborate on the baseline used, comparative methods,

Experimental Content:
1.
procedure, from dataset construction to specific steps, ensuring
clarity and thoroughness.
2.
Technical Details:
involved, providing detailed implementation processes.
3.
facilitate understanding without unnecessary complexity.
4.
and experimental design, illustrating how these support and
validate the conclusions drawn.
5.
verifying the core idea and ensure it is detailed and feasible.
Relevance Criteria:
1.
paper’s methodology, indicating improvements or modifications.
2.
if methods differ, better have the same topic [Topic]
3.
the methods discussed in the paper.
4.
publication years, formatted as titles only.
The paper content is as follows: [Paper content]
Please provide the entities, summary idea, experimental design,
and the three most relevant references (Sort by relevance, with
priority given to new ones with the same level of relevance, do not
reference the original paper.)
Note:
studying: [Topic].
[].

Ensure the references are pertinent to the topic you are

References should address the same task, even

References must directly correlate with the

If there are no relevant references, output

Provide references without author names or

References should serve as baselines for

based on the paper’s content.

Baseline Relevance:

Task Relevance:

Output Format:

...

Now please output strictly in the following format:
Entities:
Idea:
Experiment:
References:

...
...

...

18

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Under review as a conference paper at ICLR 2025

Table 11: Prompt used to get trends of CoI

You are a scientific research expert tasked with summarizing the
historical progression of research related to our current topic,
based on the literature we have reviewed.

: [Topic]

Here are the entities you need to know : [Entities]
The topic you are studying is:
The literature from early to late: [Idea chain]
Your objective is to outline the historical evolution of the
research in light of current trends.
requirements:
Analysis of Published Viewpoints:
across the identified papers.
to the next--for instance, how Paper 0 leads to Paper 1, and so
forth.
in Paper 0. Elaborate on specific advancements made, including
proposed modules, their designs, and the rationale behind their
effectiveness in addressing previous challenges.
analytical approach to each paper in the sequence.

Please follow these

Detail how each paper transitions

Apply this

Focus on understanding how Paper 1 builds upon the concepts

Examine the progression of ideas

Please present your findings in the following format:
Trends:
Paper 0 to Paper 1:
Paper 1 to Paper 2:
...

...
...

Table 12: Prompt used to predict future trend (Part I)

You are a scientific expert tasked with formulating a novel and
innovative research idea based on your comprehensive literature
review.
could significantly advance the field.

Your objective is to propose a feasible approach that

Here are the entities you need to know : [Entities]
The literature you have studied is as follows: [Chain of ideas]
The following section delineates the progressive relationships
among the previously summarized research papers: [Trend]
Based on previous research, analyze how human experts think
and transition from previous methods to subsequent approaches.
Focus on their reasoning logic and the sources of their thought
processes.
develop and guide your own research direction in a natural and
coherent manner.
Additionally, you are encouraged to adopt the following three modes
of thinking:
Continue to next table →

Learn to emulate their reasoning patterns to further

19

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Under review as a conference paper at ICLR 2025

Table 13: Prompt used to predict future trend (Part II)

Encourage

Reflection:

Think creatively

Consider whether there are

Consider potential solutions

Explore these solutions and adapt

Reflect on scenarios where a specific method

Some methods may present specific approaches to

Analogy: Identify a specific problem you are currently

1.
encounters significant challenges.
that could effectively address these issues, make the solutions
sounds reasonable, novel and amazing.
2.
facing and research existing solutions that have successfully
tackled similar challenges.
key principles and strategies to your situation.
about how tools and approaches from other domains can be
re-imagined to devise a novel strategy for your issue.
you to actively explore methods in other fields to solve your
current problems.
3.
Deep Dive:
addressing a particular problem.
aspects that could be modified to enhance their rationale and
effectiveness.
Note:Each article’s limitations are specific to that particular
piece and should not be applied to others.
task at hand and analyze the potential issues you might encounter
if you proceed with your original approach, reflecting on the
challenges previously faced.
address these issues effectively.
You are encouraged to apply human reasoning strategies to identify
future research directions based on prior studies.
in-depth analysis rather than mere integration of existing ideas.
Please avoid introducing unfamiliar information, ensuring that
the trends you present are both authentic and reasonable. Before
proposing any trends, take a moment to reflect on the principles
underlying the methods you’re employing and assess their relevance
to your research area.
The future research direction should be related to the topic:
[Topic]
Please present the future research direction in the following
format:
Future direction:

Then, think critically about how to

Carefully consider the

Aim for

...

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

20

Under review as a conference paper at ICLR 2025

Table 14: Prompt used to generate idea (part I)

Please avoid this situation as

You can continue to make in-depth innovations,

Your objective is to propose a feasible approach that

Distinguish your proposed method from existing methods

Present a detailed description of your idea, focusing on the

You are a scientific expert tasked with formulating a novel and
innovative research idea based on your comprehensive literature
review.
could significantly advance the field.
The following are examples of ideas you have proposed in the past
that are similar to real papers.
much as possible.
but avoid plagiarism: [Bad case]
Here are the entities you need to know: [Entities]
The topic you are studying is: [Topic]
The literature you have studied is as follows: [Chain of ideas]
Your idea is composed of the following components:
Motivation:
Provide a background for your idea, summarizing relevant work.
1.
2.
Identify shortcomings in previous research and highlight the
specific problems that remain unsolved and that you aim to address.
Novelty:
1.
(preferably by naming specific approaches).
Detail the improvements of your method compared to past work.
2.
3.
Clearly outline at least three contributions your idea offers
to the field, including the problems it resolves and the benefits
it delivers.
Method:
1.
core method, the specific problem it solves, and enhancements over
earlier research (citing relevant literature with titles).
2.
of each module and the rationale for why this approach effectively
addresses previous challenges.
Please adhere to the following guidelines:
1.
Your research idea should be innovative, feasible, and
contribute meaningfully to the field. Please carefully examine
the idea you have proposed, avoid immediate perception, and try to
be different from the previous methods as much as possible.
2.
to implement.
3.
limited background knowledge in the subject.
technical jargon, but when professional terms are necessary,
provide thorough explanations.
4.
prevent proposing ideas that may be incorrect or impractical.
5.
the cited papers.
6.
the trends you present are both authentic and reasonable.
proposing any trends, take a moment to reflect on the principles
underlying the methods you’re employing and assess their relevance
to your research area.
Continue to next table →

Logic should underpin your reasoning.
Write in clear, concise language aimed at an audience with

Please avoid introducing unfamiliar information, ensuring that

When referencing other research, please include the titles of

Explain the step-by-step methodology, including the functions

Ensure your proposal is solid, clearly defined, and practical

Refrain from introducing concepts from uncertain fields to

Avoid complex

Before

21

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

Under review as a conference paper at ICLR 2025

Table 15: Prompt used to generate idea (part II)

Carefully consider the

Then, think critically about how to

Each article’s limitations are specific to that particular

7.
piece and should not be applied to others.
task at hand and analyze the potential issues you might encounter
if you proceed with your original approach, reflecting on the
challenges previously faced.
address these issues effectively.
The following section delineates the progressive relationships
among the previously summarized research papers: [Trend]
The following section outlines the potential future research
directions based on the literature you have studied: [Future
direction]
Please output your motivation,novelty,method firstly and then
output your final idea.The final idea should clearly explain the
origins, motivation, and challenges of your idea, detailing how you
overcame these hurdles.

...

Please present the final idea in the following format:
Motivation:
Novelty:
Method:
Final idea:

...

...

...

Table 16: Prompt used to check the novelty of the idea

Your

You are a scientific research expert tasked with evaluating the
similarity between a specified idea and existing research.
objective is to determine if the target idea closely resembles any
findings in the provided papers.
The target idea you need to check is as follows: [Idea]
The relevant papers you need to refer to are as follows:[Content of
retrieved papers]
Here are your guidelines:
Comparison Process: Begin by thoroughly comparing each
1.
paper’s ideas with the target idea. Consider the methodologies,
conclusions, and underlying concepts in each paper in your
analysis.
2.
similarities with any existing research to the extent that they can
be considered identical, classify this as plagiarism.
3.
the similarity assessment, a summary of the target idea, and the ID
of the most relevant similar paper.
Please output strictly in the following format:
Think:
Similar:
Summary of the idea:
Similar paper id:

Your output should provide a clear thought process,

If the target idea shares fundamental

Similarity Assessment:

...
0 to n

Output:

...

0/1

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

22

Under review as a conference paper at ICLR 2025

Table 17: Prompt used to generate experiment

You are a scientific expert tasked with designing rigorous,
feasible experiments based on specified scientific questions
and the methodologies derived from the idea I provide, along
with relevant past research.
in systematically testing hypotheses and validating innovative
discoveries that could significantly advance their fields.

Your goal is to assist researchers

Structure

Experimental Design:

Develop rigorous experiments to

For any critical concepts utilized, provide thorough

Implementation of Technologies/Methods: If your experimental

Past Related Research Experiments: [Past experiments]
Here are the entities you need to know: [Entities]
Here is the idea you need to design an experiment for: [Idea]
Please propose a detailed experimental plan addressing the
following points:
1.
Provide
ensure the reliability and validity of your results.
a comprehensive explanation of the baseline used, comparative
methods, ablation study design, and criteria for data analysis
and result evaluation. Clarify how these components collectively
reinforce and validate the conclusions of your research.
your experimental design in a clear, logical, and step-by-step
manner, ensuring each step is well-defined and easy to understand.
2.
design involves specific technologies or methodologies, describe
the implementation process in detail, including key technical
aspects.
explanations.
detail its construction, components, and functionality.
Feasibility Assessment: Ensure your experimental plan is
3.
realistic, considering technological availability, timelines,
resources, and personnel.
propose strategies for addressing them.
4.
literature, include titles and pertinent details of the original
papers.
your experimental design.
5.
illustrate the implementation process.
pseudo code to detail the core algorithm or the model architecture,
or employ a flowchart to map out the experimental procedure and
data flow.
6.
your methods, assuming the reader may have limited knowledge of
the subject matter.
terminology.
clear and detailed explanations.

Strive to use as many references as necessary to support

References to Previous Studies: When citing related

If professional terms are necessary, please provide

For instance, if you propose a modular approach,

If useful, provide pseudo code or a flowchart to

Avoid complex jargon and utilize accessible

Use straightforward language to describe

Identify potential challenges and

For example, you can use

Clarity of Language:

Visual Aids:

Please output strictly in the following format:
Experiment:
...
Step1:
Step2:
...
...

23

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

Under review as a conference paper at ICLR 2025

Table 18: Prompt used to review experiment

You are an expert in paper review.
a given experiment can effectively verify a specific idea, as well
as assess the detail and feasibility of the experiment.

Your task is to analyze whether

Are there specific experimental procedures that are confusing

Here are the related entities you need to know: [Entities]
The idea presented is: [Idea]
The corresponding experiment designed for this idea is:
[Experiment]
Please conduct your analysis based on the following criteria:
Can the experiment validate the idea? If not, identify
1.
the issues and suggest improvements to enhance its verification
capability and feasibility.
2.
or poorly designed?
uncertainties in constructing the dataset, or a lack of explanation
regarding the implementation of certain methods.
3.
of the experimental design.
4.
shortcomings identified in your analysis.
5.
altering the original idea.
6.
specific.

Evaluate the clarity, detail, reasonableness, and feasibility

Provide suggestions for improving the experiment based on the

Ensure that your suggestions are constructive, concise, and

Focus solely on the experiment design; please refrain from

Discuss any methods that may not be feasible,

Please strictly follow the following format for output:
Suggestion:

...

Table 19: Prompt used to get query for search paper to refine experiment

You are a research expert tasked with refining and improving an
experimental plan based on the feedback received.

The experimental plan you proposed is as follows: [Experiment]
You have received the following suggestions for improvement:
[Suggestions]
Please decide whether you need to search for relevant papers to
obtain relevant knowledge to improve your experiment.
If you need to search for relevant papers, please provide a search
query for literature search, else provide "".
For example:
if suggestions say that the dynamic query additional
information and update knowledge graph described in the experiment
is not clearly described, so you need to output "dynamic knowledge
graph update".

Please output strictly in the following format:
Query:...

24

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

Under review as a conference paper at ICLR 2025

Table 20: Prompt used to refine experiment

You are a research expert tasked with refining and improving an
experimental plan based on the feedback received.

Structure

If your experimental

Feasibility Assessment:

[Searched paper information]

Implementation of Technologies/Methods:

For instance, if you propose a modular approach,

For any critical concepts utilized, provide thorough

Experimental Design: Develop rigorous experiments to

The information of the literature you maybe need to refer to are
as follows:
The experimental plan you proposed is as follows: [Experiment]
Please propose a detailed experimental plan addressing the
following points:
1.
ensure the reliability and validity of your results.
Provide
a comprehensive explanation of the baseline used, comparative
methods, ablation study design, and criteria for data analysis
and result evaluation. Clarify how these components collectively
reinforce and validate the conclusions of your research.
your experimental design in a clear, logical, and step-by-step
manner, ensuring each step is well-defined and easy to understand.
2.
design involves specific technologies or methodologies, describe
the implementation process in detail, including key technical
aspects.
explanations.
detail its construction, components, and functionality.
3.
Ensure your experimental plan is
realistic, considering technological availability, timelines,
resources, and personnel.
propose strategies for addressing them.
4.
References to Previous Studies:
literature, include titles and pertinent details of the original
papers.
your experimental design.
5.
illustrate the implementation process. For example, you can use
pseudo code to detail the core algorithm or the model architecture,
or employ a flowchart to map out the experimental procedure and
data flow.
6.
your methods, assuming the reader may have limited knowledge of
the subject matter.
terminology.
clear and detailed explanations.
You have received the following suggestions for
improvement:[Suggestions]
Please refine your experimental plan based on the feedback
provided.
and addresses the feedback you received.

Clarity of Language: Use straightforward language to describe

Strive to use as many references as necessary to support

Ensure your refined plan is feasible, clearly defined,

If professional terms are necessary, please provide

If useful, provide pseudo code or a flowchart to

Avoid complex jargon and utilize accessible

Identify potential challenges and

When citing related

Visual Aids:

Please output strictly in the following format:
Experiment:

...

25

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

Under review as a conference paper at ICLR 2025

1350
1351

1352
1353
1354
1355
1356
1357

1358
1359
1360
1361
1362
1363

1364
1365
1366
1367
1368

1369
1370
1371
1372
1373
1374

1375
1376
1377
1378
1379
1380

1381
1382
1383
1384
1385
1386

1387
1388
1389
1390
1391

1392
1393
1394
1395
1396
1397

1398
1399
1400
1401
1402
1403

Table 21: Prompt used to extract topic from real paper

You are a research expert tasked with extracting the main topic
from the provided paper information.

The main topic should encompass broad fields such as "Retrieve
augment generation" or "using diffusion models for video
generation". However, it should also include a relevant task to
the topic, formatted as "topic:...
Please read the provided paper and extract only the topic, which
should follow this structure.
The paper’s title is [Title]
The paper’s abstract is as follows: [Abstract]
The paper’s introduction is as follows: [Introduction]

task:...".

Please output strictly in the following format:
topic:

...

26

Under review as a conference paper at ICLR 2025

Table 22: Prompt used to extract idea from real paper

You are a research expert tasked with extracting the main idea from
the provided paper information.

Explain the differences between the method and the

Explain the background of the idea and past related

Provide a detailed description of your idea, including the

The main idea should encompass the motivation, solved problem,
novelty, method of the paper.
Please read the provided paper and extract the main idea from the
paper.
The paper content is as follows: [Content]
Idea is composed of the following components:
Motivation:
work, identify the shortcomings of past work, identify the problems
that need improvement, and identify the issues the paper want to
address.
Novelty:
current method (preferably list specific methods), explain what
improvements the paper have made to the previous method, and then
identify the problems that can be solved and the benefits that can
be gained from these improvements.
Method:
core method, the problem it solves, and the improvement compared
with previous work(Cite the previous work with the title of the
paper).
Explain the specific steps of the method, the specific
functions of each module, and the specific reasons why this method
can solve the previous problem.
Here are some tips for extracting the main idea:
1.
to describe, assuming the reader is someone who has few knowledge
of the subject, avoid using complex technical terms, and try to
use easy-to-understand terms to explain.If the paper use some
professional terms, please explain them in detail.
2.
the original paper.
The final idea should be detailed and specific, clearly explain
the origins, motivation, novelty, challenge, solved problem and
method of the paper, and detail how the overcame these hurdles.
Ensure your approach is innovative, specifying how this innovation
is reflected in your experimental design.
The final idea should be double-blind, i.e. no experimental
results or codes should be shown.

When the paper cite other papers, please indicate the title of

Make idea easy to understand, use clear and concise language

Please output strictly in the following format:
Final idea:

...

27

1404
1405

1406
1407
1408
1409
1410
1411

1412
1413
1414
1415
1416
1417

1418
1419
1420
1421
1422

1423
1424
1425
1426
1427
1428

1429
1430
1431
1432
1433
1434

1435
1436
1437
1438
1439
1440

1441
1442
1443
1444
1445

1446
1447
1448
1449
1450
1451

1452
1453
1454
1455
1456
1457

Under review as a conference paper at ICLR 2025

Table 23: Prompt used to extract experiment from real paper

You are a research expert tasked with extracting the specific
experiment steps from the provided paper information.

Describe the entire

Detail the Experimental Process:

The specific experiment steps should include the specific methods
for each step.
Please read the provided paper and extract specific experiment
steps from the paper.
The paper content is as follows: [Content]
There are some tips for extracting the experiment steps:
1.
experimental process, including how to construct the dataset and
each specific experimental step.
method is clearly and thoroughly detailed.
2.
If specific technologies are involved in the experimental
design, describe the implementation process in as much detail as
possible (i.e., technical details)
3.
be easily understood by others,should not be too complicated.
4.
the paper, the comparative methods, the ablation design and the
experimental design.
collectively support and validate the conclusions drawn in your
research.
5.
idea and how the experiment is detailed and feasible.

Please provide a detailed explanation of the baseline used in

Explain how your experimental design can help you verify the

Make sure your experimental plan is concise and clear, and can

Ensure that each experimental

Specifically, elaborate on how these elements

Now please output strictly in the following format:
Experiment:
...
Step1:
...
Step2:
...

28

1458
1459

1460
1461
1462
1463
1464
1465

1466
1467
1468
1469
1470
1471

1472
1473
1474
1475
1476

1477
1478
1479
1480
1481
1482

1483
1484
1485
1486
1487
1488

1489
1490
1491
1492
1493
1494

1495
1496
1497
1498
1499

1500
1501
1502
1503
1504
1505

1506
1507
1508
1509
1510
1511

Under review as a conference paper at ICLR 2025

Table 24: Prompt used to compare two ideas

You are a judge in a competition.
better.

You have to decide which idea is

Can the idea be realized with existing technology

Does the idea address a difficult problem in a better way

Significance:

Are other people

Please write a short paragraph

Novelty: Are the problems or approaches new? Is this a novel

Are the idea important?

Is it clear how this work

Is related work adequately

Does it provide a unique theoretical or

The idea0 is: [idea0]
The idea1 is: [idea1]
The topic is: [topic]
Which idea do you think is better?
to explain your choice.
Here are your evaluation criteria:
1.
combination of familiar techniques?
differs from previous contributions?
referenced?
2.
(practitioners or researchers) likely to use these ideas or build
on them?
than previous research?
pragmatic approach?
3.
Feasibility:
or methods? Are there any technical difficulties or bottlenecks?
Is the idea clear and logical?
unreasonable part in the idea, and can the experiment be designed
normally according to this idea.
4.
Does it adequately inform the reader?
5.
well (e.g., better than existing baselines).
Note:
Avoid any position biases and ensure that the order in which
the responses were presented does not influence your decision.
DO NOT allow the LENGTH of the responses to influence your
evaluation, choose the one that is straight-to-the-point instead
of unnecessarily verbose.
important!!!)
If you think idea0 is better than idea1, you should output 0. If
you think idea1 is better than idea0, you should output 1.
think idea0 and idea1 are equally good, you should output 2.

How likely the proposed idea is going to work

Clarity: Is the paper clearly written?

Be as objective as possible. (very

Is there any obvious error or

Is it well-organized?

Effectiveness:

If you

Your output should be strictly in following format:
Your thinking process:
...
Your choice:
Novelty:
Significance:
Feasibility: 0/1/2
Clarity:
Effectiveness:

0/1/2

0/1/2

0/1/2

0/1/2

1512
1513

1514
1515
1516
1517
1518
1519

1520
1521
1522
1523
1524
1525

1526
1527
1528
1529
1530

1531
1532
1533
1534
1535
1536

1537
1538
1539
1540
1541
1542

1543
1544
1545
1546
1547
1548

1549
1550
1551
1552
1553

1554
1555
1556
1557
1558
1559

1560
1561
1562
1563
1564
1565

29

Under review as a conference paper at ICLR 2025

Table 25: Prompt used to compare two experiments

You are a judge in a competition.
experiment is better.

You have to decide which

Are

Quality:

Feasibility:

Please write a short

Can the experiment be realized with existing

Is the experimental plan detailed and feasible?

Is there a clear rationale for each step of the
Are the baseline and evaluation metrics
Has the design taken into account the

The idea of experiment0 is: [idea0]
The experiment0 is: [experiment0]
The idea of experiment1 is: [idea1]
The experiment1 is: [experiment1]
Which experiment do you think is better?
paragraph to explain your choice.
Here are your evaluation criteria:
1.
technology or methods? Are there any technical difficulties or
bottlenecks?
the experimental steps clear and logical? Is there any obvious
error or unreasonable part in the experiment. Consider the
rationality of its steps and the possibility that the idea can be
successfully implemented.
2.
experimental design?
chosen appropriately?
potential advantages and limitations of the methods used?
this experimental design effectively support the claims made in the
idea.
3.
provide enough information for the expert reader to understand the
experiment?
reader?
Note:
the responses were presented does not influence your decision.
DO NOT allow the LENGTH of the responses to influence your
evaluation, choose the one that is straight-to-the-point instead
of unnecessarily verbose.
important!!!)
If you think experiment0 is better than experiment1, you should
output 0.
If you think experiment1 is better than experiment0,
you should output 1.
equally good, you should output 2.

Avoid any position biases and ensure that the order in which

If you think experiment0 and experiment1 are

Is the experimental plan clearly written?

Does it adequately inform the

Be as objective as possible.

Is it well organized?

Clarity:

Dose it

(very

Can

Your output should be strictly in following format:
Your thinking process: ...
Your choice:
Feasibility:
Quality:
Clarity:

0/1/2
0/1/2

0/1/2

30

1566
1567

1568
1569
1570
1571
1572
1573

1574
1575
1576
1577
1578
1579

1580
1581
1582
1583
1584

1585
1586
1587
1588
1589
1590

1591
1592
1593
1594
1595
1596

1597
1598
1599
1600
1601
1602

1603
1604
1605
1606
1607

1608
1609
1610
1611
1612
1613

1614
1615
1616
1617
1618
1619

Under review as a conference paper at ICLR 2025

1620
1621

1622
1623
1624
1625
1626
1627

1628
1629
1630
1631
1632
1633

1634
1635
1636
1637
1638

1639
1640
1641
1642
1643
1644

1645
1646
1647
1648
1649
1650

1651
1652
1653
1654
1655
1656

1657
1658
1659
1660
1661

1662
1663
1664
1665
1666
1667

1668
1669
1670
1671
1672
1673

Table 26: Evaluation results of idea generation for both model-based evaluation and human-based
evaluation.

Novelty

Significance Clarity

Feasibility Effectiveness Average Rank

Real Paper

CoI Agent (ours)

RAG

GPT-Researcher

ResearchAgent

AI-Scientist

Real Paper

CoI Agent (ours)

GPT-Researcher

ResearchAgent

RAG

AI-Scientist

Real Paper

CoI Agent (ours)

GPT-Researcher

ResearchAgent

RAG

AI-Scientist

Real Paper

CoI Agent (Ours)

GPT-Researcher

ResearchAgent

RAG

AI-Scientist

n
a
m
u
H

o
4
-
T
P
G

7
2
8
0
p
x
E
-
o
r
P
-
5
.
1
i
n
i
m
e
G

t
e
n
n
o
S
-
5
.
3
-
e
d
u
a
l
C

1075

1100

1021

988

982

835

1063

1144

995

1005

914

878

1102

1124

1002

986

914

873

1099

1165

986

1008

886

855

1071

1103

1038

993

975

820

1089

1138

1007

1016

918

831

1101

1119

997

986

929

868

1123

1154

977

1023

907

815

1127

1065

1030

990

975

813

1165

1021

1010

946

1023

836

1120

1082

1014

975

958

851

1149

953

1039

926

1038

895

1109

1078

1035

999

970

809

1123

1152

989

1004

918

814

1102

1113

998

986

932

869

1179

1162

977

997

884

800

1100

1085

1029

992

980

812

1115

1107

999

995

950

833

1110

1107

1003

983

936

860

1145

1094

1000

998

938

825

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1118

1081

1022

993

1001

784

1137

1080

995

1005

978

806

1125

1098

1005

984

948

840

1174

1033

1022

1034

977

760

31

