hallucination_check = """ Your job is to check if a review is hallucinating details not found in the paper. Additionally, the review has gone through multiple iterations of revision -- make sure this final review does not make reference to those intermediate steps. For example, statements like "I have re-examined the manuscript together with the newly supplied materials" would be an example of an extra reference to remove. The review should read like a direct response to the paper, without any extra information or mention of any intermediate steps. If you see any, just remove those references completely. Here is the review: ```{review}``` First, within the tags ..., go through each point and make sure it is grounded in the paper. Focus on specific numbers, tables, figures, etc. what are referenced and make sure that the review is not making up details. If it is, bring it up and plan a fix by either correcting the error or removing the point entirely. Then, within the tags ..., write the review again, with the problematic points removed or corrected. You should otherwise keep the review the same (verbatim). """ acceptance_criteria = """# Acceptance Criteria Acceptance of a submission to TMLR should be based on positive answers to the following two questions. **Are the claims made in the submission supported by accurate, convincing and clear evidence?** This is the most important criterion. This implies assessing the technical soundness as well as the clarity of the narrative and arguments presented. Any gap between claims and evidence should be addressed by the authors. Often, this will lead reviewers to ask the authors to provide more evidence by running more experiments. However, this is not the only way to address such concerns. Another is simply for the authors to adjust (reduce) their claims. **Would some individuals in TMLR's audience be interested in the findings of this paper?** This is arguably the most subjective criterion, and therefore needs to be treated carefully. Generally, a reviewer that is unsure as to whether a submission satisfies this criterion should assume that it does. Crucially, it should not be used as a reason to reject work that isn't considered “significant” or “impactful” because it isn't achieving a new state-of-the-art on some benchmark. Nor should it form the basis for rejecting work on a method considered not “novel enough”, as novelty of the studied method is not a necessary criteria for acceptance. We explicitly avoid these terms (“significant”, “impactful”, “novel”), and focus instead on the notion of “interest”. If the authors make it clear that there is something to be learned by some researchers in their area from their work, then the criterion of interest is considered satisfied. TMLR instead relies on certifications (such as “Featured” and “Outstanding”) to provide annotations on submissions that pertain to (more speculative) assertions on significance or potential for impact. Here's an example on how to use the criteria above. A machine learning class report that re-runs the experiments of a published paper has educational value to the students involved. But if it doesn't surface generalizable insights, it is unlikely to be of interest to (even a subset of) the TMLR audience, and so could be rejected based on this criterion. On the other hand, a proper reproducibility report that systematically studies the robustness or generalizability of a published method and lays out actionable lessons for its audience could satisfy this criterion.""" review_format = """# Review Format A review should have the following content. **Summary of contributions** Brief description, in the reviewer's words, of the contributions and new knowledge presented by the submission. **Strengths and weaknesses** List of the strong aspects of the submission as well as weaker elements (if any) that you think require attention from the authors. **Requested changes** List of proposed adjustments to the submission, specifying for each whether they are critical to securing your recommendation for acceptance or would simply strengthen the work in your view. **Broader impact concerns** Brief description of any concerns on the ethical implications of the work that would require adding a Broader Impact Statement (if one is not present) or that are not sufficiently addressed in the Broader Impact Statement section (if one is present). This part should be very brief (one sentence maximum). **Claims and evidence** Are the claims made in the submission supported by accurate, convincing and clear evidence? Just provide a yes or no answer. **Audience interest** Would some individuals in TMLR's audience be interested in the findings of this paper? Just provide a yes or no answer. """ review_prompt = """Your job is to provide a high-quality review of a paper for TMLR. Ensure that your review is constructive, actionable, and aligns with the standards of TMLR. Use the following guidelines: 1. Here is the acceptance criteria of TMLR: ```{acceptance_criteria}``` 2. Adhere to the following review format: ```{review_format}``` Additionally, you are given a literature search report on the most relevant articles to the paper. If you use any of the articles in your review, make sure to cite them. This is how you should cite them: "[Author, et al. (Year)](https://arxiv.org/abs/2003.01469)" Here is the literature search report: ```{literature_search_report}``` In each of the review points, make sure to be specific and detailed. Make specific references to Figures and Tables in the paper to support your points. Provide the review in the review format, and make sure to follow the acceptance criteria. If appropriate, perform a web search for relevant academic literature to include in your review. First, within the tags ..., think about what to write in the review, focusing on potential positive and negative points to make. Reason through which ones are the most reasonable and most important to include. Then, within the tags ..., write the review in the review format. """ literature_search = """Do a web search for academic papers related to this paper that are useful to reference in a peer review. Your job is to create a detailed report on the most relevant articles, so that a reviewer can reference it and inform their review. For example, you can include how the methodology is similar or different, or point out in more detail how it differs, etc. Write each summary as follows: Write the citation in the format of "Author, et al. (Year)", and also include the link to the paper. Then provide the summary of the paper. Only take articles from academic sources, such as arxiv, neurips, journals, etc. Do not take from non-academic sources, such as news sites, blogs, etc. Make sure you don't include the paper you are reviewing in the search! """ combine_prompt = """Your job is to act as a meta-reviewer for a TMLR paper review. For the given PDF, you are given several reviews of the paper. Your job is to combine the following reviews into one cohesive review that is better than the individual reviews. Do not shorten the original review. Make sure each review point is a full sentence that explains relevant details. Think about which points are the strongest and fairest, and how to best present them in the review with clarity. The combined review should be a full, standalone review that does not reference the original reviews. Group together the points from each subsection according to the review format: ```{review_format}``` Here are all the reviews: ```{all_reviews_string}``` First, within the tags ..., think through which points are the strongest and fairest, and how to best present them in the review with clarity. Then, within the tags ..., write the review in the review format. """ defend_prompt = """ You are given the PDF of a paper and the review for that paper for TMLR. Your job is to take the perspective of the author and defend the author's paper against the reviewer's criticisms in a fair and accurate manner. First, think through which points can be clarified (eg. by asking for more details). For example, if the reviewer says "The authors do not provide enough experimental validation", you can ask for clarification on what the reviewer means by "enough experimental validation". If the reviewer makes a criticism that is not valid, point out why it is not valid. If the reviewer asks for additional experimentation that is not clear (eg. "The authors should provide more experiments"), ask for more details on what the experiments should be. Second, defend the paper against the reviewer's criticisms. For example, if the reviewer says "The authors do not provide enough experimental validation", you can defend the paper by saying "The authors provide ablation studies and empirical evaluations on several datasets, showing that the method is effective and generalizable". However, do not be combative, as the goal is to improve the review and ultimately the paper. Third, do not actually make changes to the original paper, or talk about doing experiments in response to the reviewer's criticisms. This stage is purely to improve the review, not for the authors to make changes to the paper or add experiments. The paper should be defended AS-IS. Provide clear reasons for your defense, citing specific parts of the paper. Here is the review: ```{combined_review}``` """ revise_prompt = """Improve a paper review by incorporating one round of feedback from another reviewer. Based on the initial review, another reviewer has provided feedback to the initial review by defending the authors claims against the reviewer's criticisms. Reason through each of the criticisms and corresponding responses of the original review point by point, and modify the review accordingly. Remove criticisms where the author's defense points out legitimate flaws in the original criticism. Take this opportunity to reduce the burden on the paper authors by removing experiments which may be out of scope and too burdensome without being absolutely necessary. Respond to requests for clarifications, and provide more details to strengthen the review. Use this defense to strengthen the review by being more specific and detailed that go beyond the original criticism. For example, if the original criticism was "The authors do not provide enough experimental validation", the defense might be "The authors provide ablation studies and empirical evaluations on several datasets, showing that the method is effective and generalizable". Use this time to provide necessary context for each criticism made. Provide at least one extra sentence of context for each criticism, motivating it properly. If there is a requested change, try to give an example of what that might entail rather than just stating what the reviewer asked for. Important: Do not reference anything in the rebuttal of the review or anything written in the process of review. For example, do not say "In the rebuttal, the author defends themselves by saying..." or "In the rebuttal, the author provides more details on the experimental validation...". The revision should be a standalone review that only needs the original paper to be read. Produce the revised review in the same format as the original review. Here is the review format that was used: ```{review_format}``` Ensure that your listed weaknesses / recommended changes pertain to the criteria of (1) ensuring claims are met with accurate, convincing and clear evidence and (2) whether there will be individuals in TMLR's audience interested in this topic. Here is the original review: ```{combined_review}``` Here is the rebuttal of the review: ```{defended_paper}``` Before producing the modified review, think step by step within the tags .... Then, produce the modified review in the tags, with all the sections filled out and necessary context: .... Remember to keep the citations in the same format as the original review. """ human_style = """ Here is a review of a paper for TMLR: ```{review}``` Help re-style this review to be more human-like while maintaining a professional and objective tone. (Don't make it sound casual!) Keep all the core content the same and only change the style. You can use the following techniques to make the review more human-like: 1. Use of first person singular, like "I wonder" or "I think". 2. Use of variability in sentence structure, like short and long sentences. 3. Sprinkle in 3-5 typos or minor errors. It should not be gramatically or syntactically perfect. 4. Don't use too many bullet points when the same idea can be expressed in a single sentence. 5. Do not use double dashes too often in the review as a way to express a pause or break (eg. "The paper is well written -- there is evidence of thorough experimentation"). Remember, this should still sound professional and expert-level. Be objective and do not reflect any personal feelings about the paper in the review (eg. do not say "I was amazed by the results"). Finally, do not use the following words in the review: commendable innovative meticulous intricate notable versatile noteworthy invaluable pivotal potent fresh ingenious cogent ongoing tangible profound methodical laudable lucid appreciable fascinating adaptable admirable refreshing proficient intriguing thoughtful credible exceptional digestible prevalent interpretative remarkable seamless economical proactive interdisciplinary sustainable optimizable comprehensive vital pragmatic comprehensible unique fuller authentic foundational distinctive pertinent valuable invasive speedy inherent considerable holistic insightful operational substantial compelling technological beneficial excellent keen cultural unauthorized strategic expansive prospective vivid consequential manageable unprecedented inclusive asymmetrical cohesive replicable quicker defensive wider imaginative traditional competent contentious widespread environmental instrumental substantive creative academic sizeable extant demonstrable prudent practicable signatory continental unnoticed automotive minimalistic intelligent meticulously reportedly lucidly innovatively aptly methodically excellently compellingly impressively undoubtedly scholarly strategically intriguingly competently intelligently hitherto thoughtfully profoundly undeniably admirably creatively logically markedly thereby contextually distinctly judiciously cleverly invariably successfully chiefly refreshingly constructively inadvertently effectively intellectually rightly convincingly comprehensively seamlessly predominantly coherently evidently notably professionally subtly synergistically productively purportedly remarkably traditionally starkly promptly richly nonetheless elegantly smartly solidly inadequately effortlessly forth firmly autonomously duly critically immensely beautifully maliciously finely succinctly further robustly decidedly conclusively diversely exceptionally concurrently appreciably methodologically universally thoroughly soundly particularly elaborately uniquely neatly definitively substantively usefully adversely primarily principally discriminatively efficiently scientifically alike herein additionally subsequently potentially ``` In your response, only provide the re-styled review. Do not include any other text. Remember to keep the citations in the same format as the original review. """ formatting_prompt = """Parse the review into the provided review format. Do not change any of the wording of the content, only reformat the bullet points, line breaks, etc. Review: ```{review}``` Please keep the original Markdown tags, like bold (two asterisks) or linebreak. After parsing the importance, remove the original importance tag. Shorten the Impact Concerns to a single sentence only. Remember, do not change any actual wording of the content!! """