# It’s Not Easy Being Wrong:
Evaluating Process of Elimination Reasoning in Large Language Models

Nishant Balepur Shramay Palta Rachel Rudinger
University of Maryland, College Park, USA
{nbalepur, spalta<EMAIL_ADDRESS>
###### Abstract

Chain-of-thought (COT) prompting can help large language models (LLMs) reason
toward _correct_ answers, but its efficacy in reasoning toward _incorrect_
answers is unexplored. This strategy of process of elimination (PoE), when
used with COT, has the potential to enhance interpretability in tasks like
medical diagnoses of exclusion. Thus, we propose PoE with COT, a new task
where LLMs must reason toward incorrect options on multiple-choice questions.
We evaluate the ability of GPT-3.5, LLaMA-2, and Falcon to perform PoE with
COT on 2-choice commonsense and scientific reasoning datasets. We show that
PoE consistently underperforms directly choosing the correct answer. The
agreement of these strategies is also lower than the self-consistency of each
strategy. To study these issues further, we conduct an error analysis and give
suggestions for future work.111Code at https://github.com/nbalepur/PoE

##  1 Introduction

Recent research has explored methods to unlock the reasoning capabilities of
large language models (LLMs) Nye et al. (2022); Zhou et al. (2023). As part of
this effort, researchers have introduced techniques such as chain-of-thought
(COT) prompting to enable LLMs to verbally reason toward correct answers Wei
et al. (2022b); Kojima et al. (2022). Such verbal reasoning can improve both
the accuracy and interpretability of LLM decision-making Creswell et al.
(2023); Huang and Chang (2023).

While previous works use COT to select _correct_ answers (Figure 1 blue), they
do not explore if COT can identify _incorrect_ answers (Figure 1 red). Thus,
drawing from student test-taking strategies for multiple-choice (MC) exams
Tversky (1972), we propose a new task for LLM reasoning: process of
elimination (PoE) with COT. Score-based PoE methods have previously been
applied to LMs Ma and Du (2023), but the ability of LLMs to perform PoE with
COT has not been investigated.

Figure 1: ChatGPT using direct answer and process of elimination strategies
via chain-of-thought prompting.

PoE with COT poses several benefits. First, the task can give insights into
the consistency of LLM reasoning Wang et al. (2023b). For 2-choice MC
questions, we can study if the decisions from PoE with COT and directly
answering the question with COT agree, as both should arrive at the same
answer (section 4.2). Second, PoE with COT can enhance interpretability. COT
reasoning is often viewed as a rationale as to why the model thinks an option
is correct, but users may also want to know why the model thinks other options
are incorrect. Third, there are many applications that benefit from PoE, such
as reaching medical diagnoses through exclusion Fred (2013); Kline (2018),
troubleshooting by eliminating root causes Gutoff and Cohen (2006); Gugerty
(2007), and ruling out scientific hypotheses in light of new evidence Norton
(1995); Forber (2011). Thus, if we wish to enable interpretable LLMs in
domains like medicine, customer service, and scientific research, we must
study: _Do LLMs have the ability to perform PoE with COT?_

To answer this question, we conduct a comparative analysis on 2-choice MC
commonsense Sap et al. (2019); Talmor et al. (2019) and scientific Clark et
al. (2018); Mihaylov et al. (2018) reasoning datasets. We independently prompt
the GPT-3.5 Brown et al. (2020), LLaMA-2 Touvron et al. (2023), and Falcon
Penedo et al. (2023) LLMs to directly choose the correct option and to
eliminate the incorrect one, both with and without COT. Through this approach,
we can juxtapose LLM reasoning abilities in direct answer selection against
PoE.

We find that PoE always underperforms direct answer selection for our tested
LLMs, but this gap narrows as model size scales, implying that PoE with COT
may only be attainable for larger models (section 4.1). Further, these two
strategies do not reliably arrive at the same answer, revealing a fundamental
inconsistency in LLM reasoning (section 4.2). To study these issues further,
we conduct an error analysis of the rationales generated from PoE with COT
(section 4.3). Based on our results, we suggest directions for future work.
Our contributions are as follows:

1) We introduce the new task of PoE with COT, a reasoning strategy that can
benefit interpretability, self-consistency, and downstream applications.
2) We benchmark the abilities of GPT, LLaMA-2, and Falcon to perform PoE with
and without COT on scientific and commonsense reasoning datasets.
3) We evaluate the consistency and errors of PoE with COT to suggest
directions for future research.

##  2 Problem Definition

We aim to study LLM abilities when selecting correct and incorrect choices in
multiple-choice (MC) settings, where we are given a question $q$ and a list of
$n$ choices $\mathcal{C}$, exactly one of which is correct. To simplify our
discussion, we define two strategies:
1) Direct Answer (DA) Strategy: The LLM aims to select the correct answer
choice $c_{t}\in\mathcal{C}$. This is how most users ask LLMs to answer MC
questions.
2) Process of Elimination (PoE) Strategy: The LLM aims to select an incorrect
choice $c_{f}\in\mathcal{C}$.

Humans apply PoE iteratively, removing up to $n-1$ incorrect options before
choosing $c_{t}$. However, this leads to error propagation, as earlier
mistakes prevent the selection of $c_{t}$ later, shown by our experiments in
Appendix B.4. Thus, to fairly compare DA and PoE and isolate LLM reasoning, we
focus on 2-choice questions (i.e. $\mathcal{C}=\\{c_{t},c_{f}\\}$), where DA
and PoE _should_ be logically consistent: predicting that one choice is
correct entails that the other is incorrect. This setup allows us to study LLM
reasoning and decision-making without the confounding factors of multiple
iterations.

##  3 Experimental Setup

###  3.1 Strategy Implementation

Following the work on COT Wei et al. (2022b), we use few-shot (10) prompts to
implement the strategies, leaving the analysis of 0-shot PoE with COT for
future work. We create the following prompts:
1) DA Base and PoE Base: These prompts task the LLM with performing the DA and
PoE strategies without reasoning. The in-context examples in these prompts
follow a similar format to Figure 1, but only the correct/incorrect answer
choice ((A) or (B)) follow the “Correct/Incorrect Answer:” labels.
2) DA COT and PoE COT: These prompts add COT reasoning to the DA and PoE Base
prompts. The in-context examples in these prompts follow the exact format as
Figure 1, requiring the LLM to generate a step-by-step rationale before
answering.

To study if COT reasoning improves the accuracy of each strategy (section
4.1), we can compare the accuracy of the Base prompts with their COT
counterparts (e.g. PoE Base vs PoE COT). Further, comparing the outcomes of
the DA and PoE prompts (e.g. DA COT vs PoE COT) lets us measure the LLMs’
logical consistency when choosing correct answers and eliminating incorrect
answers (section 4.2).

The COT rationales are written by one Ph.D. student and verified by a second
(the authors). Each rationale reasons toward the correct or incorrect answer
choice without mentioning the other choice. In this format, we aim to make
each LLM give a concrete explanation as to why certain choices are valid or
invalid, avoiding unhelpful PoE rationales like “I know the answer is (A). So
the incorrect answer is (B).” For each prompt, we also prepend a natural
language instruction explaining the strategy (e.g. _Your goal is to identify
the incorrect answer to the multiple choice question_). More prompting details
and examples can be found in Appendix A.2.

Figure 2: Accuracy of Direct Answer and Process of Elimination, with and
without chain-of-thought, on commonsense (CQA, SIQA) and scientific (ARC,
OpenBookQA) reasoning datasets. Numerical results are in Appendix B.1.

###  3.2 Datasets

We examine multiple-choice commonsense and scientific reasoning questions,
given that the DA strategy has shown to perform better on these questions
using COT reasoning Wei et al. (2022b); Lu et al. (2022); Zhang et al.
(2023c); Lanham et al. (2023).222It is also straightforward to give
explanations as to why incorrect choices are incorrect on these datasets. In
other COT settings like arithmetic Cobbe et al. (2021), it is difficult to
perform PoE without first arriving at the correct answer.

Commonsense reasoning questions ask models to reason about ordinary
interactions based on knowledge understood by most humans. To represent this
setting, we choose CommonsenseQA (CQA) Talmor et al. (2019), which concerns
commonsense knowledge encoded in ConceptNet, and Social IQa (SIQA) Sap et al.
(2019), which contains questions about everyday social interactions.

Scientific reasoning questions require models to make logical inferences
grounded in scientific facts. For this setting, we choose the ARC Clark et al.
(2018) and OpenBookQA Mihaylov et al. (2018) (OBQA) datasets, both of which
ask questions that necessitate scientific knowledge, multi-step reasoning, and
some commonsense reasoning to answer.

We sample 500 random questions from the test set of each dataset for
evaluation, and sample 10 random training examples for the 10-shot prompts. To
leave each question with only two choices, we randomly eliminate incorrect
choices until only the correct answer and one incorrect answer remain. Dataset
statistics can be found in Appendix A.1.

###  3.3 Models

We study three families of LLMs. First, we use GPT-3.5 (gpt-3.5-turbo-0613)
Ouyang et al. (2022) with the OpenAI API. We also use two open-source base
LLMs: LLaMA-2 (7B, 13B, 70B) Touvron et al. (2023) and Falcon (7B, 40B,
180B333Due to GPU constraints, Falcon-180B is 4-bit quantized.) Penedo et al.
(2023), implemented via huggingface. Each model decodes with a temperature of
0.3 and a minimum generation length of 5.

##  4 Results

###  4.1 Can LLMs Perform PoE?

In Figure 2, we find that both the Base and COT variants of the DA strategy
outperform their PoE counterparts for every model and dataset. Notably, many
of the smaller LLMs (7B, 13B) display PoE accuracies that fall near or below
the accuracy of a 50/50 guessing model, suggesting that the smaller models are
fully misunderstanding the PoE instruction, despite being given 10 examples.
In contrast, the performance discrepancy between the PoE and DA strategies is
less pronounced for larger LLMs, implying that PoE may only be an attainable
ability for larger LLMs Wei et al. (2022a). Our findings indicate that LLMs,
especially smaller models, may have a training bias toward identifying correct
answers, and thus struggle to perform PoE accurately.

Further, while COT tends to slightly improve the accuracy of the DA strategy
across LLMs, the same cannot be said for PoE. Specifically, DA COT has equal
or better accuracy than DA Base in 19/24 cases (8/12 for LLaMA, 7/12 for
Falcon, 4/4 for GPT). However, while PoE COT surpasses PoE Base in 10/12 cases
for LLaMA and 2/4 cases for GPT, this improvement is never seen for Falcon.
This discrepancy across models further motivates PoE as a promising research
area. While COT tends to improve or maintain accuracy when choosing correct
answers, its accuracy wavers by LLM when selecting incorrect choices. Thus, we
believe that the accuracy of PoE with COT could serve as an evaluation metric
for LLM reasoning capabilities.

###  4.2 Are PoE and DA Consistent?

In Table 1, we study the logical consistency of DA and PoE, i.e., if the
strategies converge to the same answer. For every LLM and dataset, we find
that DA more often agrees with itself upon a repeated inference (i.e. self-
consistency) than agrees with its PoE counterpart. This suggests that LLMs
have an innate logical inconsistency when asked to execute these two
strategies, which cannot be ascribed to sampling variation from our
temperature selection.

| _Falcon 180B_ | _GPT-3.5_
---|---|---
Dataset | DA Base | DA COT | DA Base | DA COT
PoE | Self | PoE | Self | PoE | Self | PoE | Self
CQA | 91.4 | 97.6 | 78.2 | 92.9 | 81.3 | 97.3 | 86.0 | 97.6
SIQA | 86.6 | 95.0 | 69.4 | 91.4 | 80.7 | 97.0 | 76.2 | 95.2
ARC | 89.3 | 98.5 | 79.1 | 90.8 | 86.6 | 97.5 | 85.5 | 97.0
OBQA | 89.7 | 96.2 | 77.2 | 95.1 | 72.3 | 98.2 | 81.4 | 96.8
Table 1: Agreement of DA strategies (Base/COT) with PoE counterparts vs. Self-
consistency. More consistent methods are underlined. Full results in Appendix
B.2.

We suggest three research directions based on this lack of consistency. First,
the agreement of DA and PoE can offer a way to assess the logical robustness
of LLMs. Future LLMs could strive to achieve DA/PoE consistency levels closer
to DA self-consistency. Second, measuring when DA and PoE agree could be used
for confidence calibration Kadavath et al. (2022); Cheng et al. (2023), as we
might expect an LLM that can arrive at the same solution with two different
strategies to be more confident in its answer. Lastly, since LLMs have been
shown to produce better rationales by combining DA COT reasoning chains Wang
et al. (2023b); Yoran et al. (2023), future works could similarly try to
synthesize DA and PoE COT reasoning chains.

###  4.3 Why Does PoE COT Fail?

We conduct an error analysis to study the issues of PoE. We take 50 rationales
from CQA and ARC where PoE COT failed and study the largest LLM from each
family, deemed the best at PoE. Our error types are as follows: 1) Misaligned
Rationale: Justifies a choice as correct instead of incorrect or fails to
justify why its selection is incorrect; 2) Logical Error: Inaccurate
premise/conclusion; 3) No Breakdown: Restates choice without breakdown into
reasoning steps; 4) Hallucination: References non-existent parts of the
question; 5) PoE N/A: Question is unsuitable for PoE; and 6) Dataset: Dataset
quality issue. Examples of PoE rationales with each error type are in Appendix
Table 14.

Figure 3: Error distribution of PoE COT on ARC/CQA.

In Figure 3, we find that the majority of errors are misaligned rationales and
logical errors. The prevalence of these errors suggests that our PoE COT
setting reveals the weaknesses of our tested LLMs to reason and follow in-
context instructions, making PoE with COT a suitable testbed for these
abilities. Notably, we find that many misaligned rationales occur with negated
questions (e.g. “What would Grace _not_ do?”). LLMs tend to struggle with
negation Truong et al. (2023) and since PoE is a “negated” reasoning
technique, we may expect LLMs to perform noticeably worse on such questions.
One solution to these issues is to fine-tune LLMs on PoE rationales. This may
enhance PoE abilities, but it would be interesting to see if this could also
bolster overall reasoning capabilities.

##  5 Related Work

LLM Reasoning: Recent research has studied the ability of LLMs to reason with
chain-of-thought (COT), defined as intermediate reasoning steps and rationales
before a final answer Wei et al. (2022b); Huang and Chang (2023). Recent works
build upon chain-of-thought with problem decomposition Dua et al. (2022); Zhou
et al. (2023), self-consistency Wang et al. (2023a, b), and rationale
refinement Zhang et al. (2023b); Fu et al. (2023). Several works also analyze
different facets of COT reasoning, such as faithfulness Lyu et al. (2023);
Lanham et al. (2023), planning Saparov and He (2023), and self-refinement
Huang et al. (2023). While these works all use COT to arrive at the correct
answer, we study it through the lens of process of elimination, i.e., arriving
at the incorrect answer.

Process of Elimination (PoE): PoE has been studied in various settings. Parikh
et al. (2018) train a neural model to perform PoE for reading comprehension,
using evidence from the input text. Zhang et al. (2023a) ask LLMs to estimate
the probability that candidate responses should be excluded to improve prompt
ensembling. Ma and Du (2023) execute a score-based PoE strategy for MCQA,
primarily with smaller LMs. Different from these works, we evaluate the innate
ability of LLMs to perform PoE with COT. While PoE with COT shares
similarities with experiments using flipped classification labels for in-
context learning Wei et al. (2023), our task does not override the semantics
of labels.

##  6 Conclusion

We study COT with process of elimination (PoE) and direct answer (DA)
strategies in 2-choice commonsense and scientific reasoning datasets. We find
that PoE underperforms DA in accuracy across all tested LLMs, but this gap
narrows as model size scales. We also explore the logical inconsistencies
between DA and PoE and categories of errors present in PoE rationales. Based
on our results, we suggest several research directions: using PoE with COT as
LLM reasoning/consistency benchmarks, combining DA and PoE with COT for
calibration and refined rationales, and fine-tuning LLMs for PoE to enhance
overall reasoning capabilities.

##  7 Limitations

LLMs are sensitive to prompt formats Min et al. (2022), so the accuracy of PoE
COT could vary depending on the number of in-context examples and the human-
written rationales in these prompts Wei et al. (2022b). To mitigate this, we
use an ample number (10) of in-context examples and have each rationale
written by one Ph.D. student and verified by a second Ph.D. student, both of
whom work on NLP and LLM research, to ensure the rationales are high-quality.
Rationales were also edited after running inference and identifying problems
on a small validation set. Future works could explore rationale refinement or
self-correction techniques Zhang et al. (2023b); Fu et al. (2023); Huang et
al. (2023) to have LLMs create their own PoE prompts.

Further, while PoE should ideally be able to give a rationale as to why one
choice is correct without mentioning the other choice, this may be infeasible
in certain scenarios. For example, given a mathematics question like “What is
2+2?” with the choices “4” and “6”, it is very difficult to justify why “6” is
incorrect without knowing that “2+2=4”. To try to avoid these types of
questions, we only look at scientific and commonsense reasoning questions, as
we find it to be straightforward to explain why incorrect choices are
incorrect. PoE is one of many test-taking strategies, so similar to students,
future works could try teaching LLMs to choose the best strategies when
answering multiple-choice questions, which would overcome this limitation.

Lastly, we note that due to resource constraints, we have not tested the
ability of GPT-4 to perform PoE. In future iterations of this work, we may
evaluate GPT-4 to see if the LLM outperforms GPT-3.5 when executing PoE. If
the gap between DA and PoE is smaller for GPT-4 than GPT-3.5, it would further
support the idea that larger and more capable LLMs are more adept at
performing PoE.

##  8 Ethical Considerations

Process of elimination is a strategy that aims to help LLMs eliminate
incorrect options, with the goal of promoting interpretability and self-
consistency, as well as enhancing downstream applications. However, our
current findings suggest that PoE with COT may be an unreliable technique.
Thus, we advise researchers and LLM practitioners to take caution before
deploying PoE in any application.

Further, as with all reasoning techniques, researchers should ensure that PoE
rationales are not based on stereotypes or biases. This is especially
important in some of the downstream applications we mention in this work, like
medical diagnoses of exclusion, where the backgrounds of certain individuals
could introduce bias into the LLM’s reasoning. We hope that future research
endeavors will jointly attempt to address these biases and improve the
robustness of LLM reasoning, fostering more equitable decision-making
frameworks.

## References

  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , NIPS’20, Red Hook, NY, USA. Curran Associates Inc.
  * Cheng et al. (2023) Silei Cheng, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang. 2023.  Prompting gpt-3 to be reliable.  In _International Conference on Learning Representations (ICLR 23)_.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021.  Training verifiers to solve math word problems.  _ArXiv_ , abs/2110.14168.
  * Creswell et al. (2023) Antonia Creswell, Murray Shanahan, and Irina Higgins. 2023.  Selection-inference: Exploiting large language models for interpretable logical reasoning.  In _The Eleventh International Conference on Learning Representations_.
  * Dua et al. (2022) Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner. 2022.  Successive prompting for decomposing complex questions.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 1251–1265, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Forber (2011) Patrick Forber. 2011.  Reconceiving eliminative inference.  _Philosophy of Science_ , 78(2):185–208.
  * Fred (2013) Herbert L Fred. 2013.  The diagnosis of exclusion: an ongoing uncertainty.  _Texas Heart Institute Journal_ , 40(4):379.
  * Fu et al. (2023) Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023.  Complexity-based prompting for multi-step reasoning.  In _The Eleventh International Conference on Learning Representations_.
  * Gugerty (2007) Leo Gugerty. 2007.  Cognitive components of troubleshooting strategies.  _Thinking & reasoning_, 13(2):134–163.
  * Gutoff and Cohen (2006) Edgar B Gutoff and Edward D Cohen. 2006.  _Coating and drying defects: troubleshooting operating problems_.  John Wiley & Sons.
  * Huang and Chang (2023) Jie Huang and Kevin Chen-Chuan Chang. 2023.  Towards reasoning in large language models: A survey.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 1049–1065, Toronto, Canada. Association for Computational Linguistics.
  * Huang et al. (2023) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023.  Large language models cannot self-correct reasoning yet.  _arXiv preprint arXiv:2310.01798_.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, T. J. Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Christopher Olah, and Jared Kaplan. 2022.  Language models (mostly) know what they know.  _ArXiv_ , abs/2207.05221.
  * Kline (2018) Jeffrey A Kline. 2018.  Diagnosis and exclusion of pulmonary embolism.  _Thrombosis research_ , 163:207–220.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _ArXiv_ , abs/2205.11916.
  * Lanham et al. (2023) Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson E. Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, John Kernion, Kamil.e Lukovsiut.e, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Samuel McCandlish, Sandipan Kundu, Saurav Kadavath, Shannon Yang, T. J. Henighan, Timothy D. Maxwell, Timothy Telleen-Lawton, Tristan Hume, Zac Hatfield-Dodds, Jared Kaplan, Janina Brauner, Sam Bowman, and Ethan Perez. 2023.  Measuring faithfulness in chain-of-thought reasoning.  _ArXiv_ , abs/2307.13702.
  * Lu et al. (2022) Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022.  Learn to explain: Multimodal reasoning via thought chains for science question answering.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 2507–2521. Curran Associates, Inc.
  * Lyu et al. (2023) Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023.  Faithful chain-of-thought reasoning.  _ArXiv_ , abs/2301.13379.
  * Ma and Du (2023) Chenkai Ma and Xinya Du. 2023.  Poe: Process of elimination for multiple choice reasoning.  _arXiv preprint arXiv:2310.15575_.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Rethinking the role of demonstrations: What makes in-context learning work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11048–11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Norton (1995) John D Norton. 1995.  Eliminative induction as a method of discovery: How einstein discovered general relativity.  In _The creation of ideas in physics: Studies for a methodology of theory construction_ , pages 29–69. Springer.
  * Nye et al. (2022) Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2022.  Show your work: Scratchpads for intermediate computation with language models.  In _Deep Learning for Code Workshop_.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 27730–27744. Curran Associates, Inc.
  * Parikh et al. (2018) Soham Parikh, Ananya B. Sai, Preksha Nema, and Mitesh M. Khapra. 2018.  Eliminet: A model for eliminating options for reading comprehension with multiple choice questions.  In _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ , IJCAI’18, page 4272–4278. AAAI Press.
  * Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023.  The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only.  _arXiv preprint arXiv:2306.01116_.
  * Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019.  Social IQa: Commonsense reasoning about social interactions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4463–4473, Hong Kong, China. Association for Computational Linguistics.
  * Saparov and He (2023) Abulhair Saparov and He He. 2023.  Language models are greedy reasoners: A systematic formal analysis of chain-of-thought.  In _The Eleventh International Conference on Learning Representations_.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Truong et al. (2023) Thinh Hung Truong, Timothy Baldwin, Karin Verspoor, and Trevor Cohn. 2023.  Language models are not naysayers: an analysis of language models on negation benchmarks.  In _Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)_ , pages 101–114, Toronto, Canada. Association for Computational Linguistics.
  * Tversky (1972) Amos Tversky. 1972.  Choice by elimination.  _Journal of mathematical psychology_ , 9(4):341–367.
  * Wang et al. (2023a) Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren. 2023a.  SCOTT: Self-consistent chain-of-thought distillation.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5546–5558, Toronto, Canada. Association for Computational Linguistics.
  * Wang et al. (2023b) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023b.  Self-consistency improves chain of thought reasoning in language models.  In _The Eleventh International Conference on Learning Representations_.
  * Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a.  Emergent abilities of large language models.  _Transactions on Machine Learning Research_.  Survey Certification.
  * Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022b.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Wei et al. (2023) Jerry W. Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. 2023.  Larger language models do in-context learning differently.  _ArXiv_ , abs/2303.03846.
  * Yoran et al. (2023) Ori Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel Deutch, and Jonathan Berant. 2023.  Answering questions by meta-reasoning over multiple chains of thought.  _ArXiv_ , abs/2304.13007.
  * Zhang et al. (2023a) Chenrui Zhang, Lina Liu, Jinpeng Wang, Chuyuan Wang, Xiaodi Sun, Hongyu Wang, and Mingchen Cai. 2023a.  Prefer: Prompt ensemble learning via feedback-reflect-refine.  _ArXiv_ , abs/2308.12033.
  * Zhang et al. (2023b) Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023b.  Automatic chain of thought prompting in large language models.  In _The Eleventh International Conference on Learning Representations_.
  * Zhang et al. (2023c) Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alexander J. Smola. 2023c.  Multimodal chain-of-thought reasoning in language models.  _ArXiv_ , abs/2302.00923.
  * Zhou et al. (2023) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023.  Least-to-most prompting enables complex reasoning in large language models.  In _The Eleventh International Conference on Learning Representations_.

##  Appendix A Experimental Setup Details

###  A.1 Dataset Statistics

In Table 2, we display summary statistics for the datasets used in our
experiments.

###  A.2 Prompt Details

Examples of the prompts used for DA COT and PoE COT on each dataset can be
found in Figures 5, 6, 7, 8, 9, 10, 11, and 12. The Base versions of these
prompts follow the same format, except only the answer choice follows the
“Correct Answer:” and “Incorrect Answer:” labels. For example, if the question
was “Where is Chicago?” with choices “Illinois” and “the Moon,” the DA Base
prompt would be:

Question: Where is Chicago?
Choices:
(A) Illinois
(B) The Moon
Correct Answer: (A)

and the PoE Base prompt would be:

Question: Where is Chicago?
Choices:
(A) Illinois
(B) The Moon
Incorrect Answer: (B)

Before the in-context learning examples, we prepend the instruction “ _Your
goal is to identify the correct answer to the multiple choice question_ ” to
the DA prompts, and the instruction “ _Your goal is to identify the incorrect
answer to the multiple choice question_ ” to the PoE prompts. For the 10 total
in-context examples, five of the examples select choice (A) and five of the
examples select choice (B), which are shuffled. We release the full prompts
along with our code.

##  Appendix B Detailed Results

###  B.1 Detailed Quantitative Evaluation

In Tables 3, 4, 5, and 6, we provide the numerical results from Figure 2. We
also calculate the difference in accuracies of the DA and POE strategies,
along with their associated $p$-values. We find that a majority of the
differences in accuracy between DA and PoE are statistically significant.

###  B.2 Full Consistency Experiments

We report the consistency results from section 4.2 for all LLMs and datasets.
In addition to the self-consistency of DA, we include the self-consistency of
PoE to ensure that the lack of consistency between DA and PoE is not due to
higher sampling variance for PoE. In Tables 7, 8, and 9, we report the results
for LLaMA-2. In Tables 10, 11, and 12, we report the results for Falcon. In
Table 13, we report the results for GPT-3.5.

We find that the agreement of DA and PoE is the lowest across all datasets and
LLMs, except for LLaMA-2 70B on Commonsense QA. We also note that PoE self-
consistency is typically lower than DA self-consistency, meaning that PoE has
a higher variance which further points to its unreliability.

###  B.3 Qualitative Results

Examples of PoE rationales that fall into the categories of our error analysis
can be found in Table 14. Examples of sound rationales generated by PoE COT
can be found in Table 15. In Figure 4, we compare the distribution of errors
of PoE COT and DA COT. For DA COT, we use the same error categories defined in
section 4.3, except “PoE N/A” is replaced with “DA N/A”. Further, we only
evaluate 20 DA COT rationales, since there were fewer instances of errors from
DA COT to choose from.

The majority of the DA COT errors were logical errors (and not misaligned
rationales), suggesting that the LLMs better understand the DA task compared
to PoE. This may further indicate that LLMs have a training bias toward
identifying correct answers. Further, we find that many of the misaligned
rationale errors that occur with DA COT are also due to negated questions
(e.g. “What would Grace not do?”), reinforcing the difficulty of LLMs to
properly address questions with negation.

###  B.4 Iterative PoE

Humans typically apply PoE iteratively, but we focus on the two-choice setting
to reduce error propagation. To confirm that iterative PoE leads to error
propagation, we conduct a preliminary experiment using LLaMA-2, asking the
model to perform PoE iteratively on the Commonsense QA and ARC datasets. If a
question has $n$ choices, we independently prompt the LLM using PoE Base and
PoE COT to eliminate an answer $n-1$ times, so the model is unaware of its
previous decisions. At each step when there are $i\in[2,n]$ choices, we also
run the DA Base and DA COT prompts, corresponding to every possible use case
of PoE (i.e. _eliminate $i$ choices and then choose the correct answer_). PoE
Base only is used with DA Base, and PoE COT is only used with DA COT.

In Tables 16 and 17, we display the results for iterative PoE on Commonsense
QA and ARC, respectively. Overall, we find that the raw accuracy decreases
significantly as more choices get eliminated, confirming our intuition that
error propagation is an issue in this setting. Although PoE has higher
accuracy than DA when the number of choices remaining is higher, this finding
is not that significant when you consider the probability of performing each
strategy accurately by chance. For example, when there are 5 choices, there is
a probability of $\frac{4}{5}=0.8$ of guessing an incorrect answer correctly,
while there is only a probability of $\frac{1}{5}=0.2$ of guessing the correct
answer correctly. If future research eventually leads to LLMs that can perform
the DA and PoE strategies at similar abilities, it would be interesting to
revisit the reliability of iterative PoE.

Dataset | Category | # Questions | # Choices |  | Proportion where
---
Gold Answer
is Choice (A)
| Avg Question
---
Length
| Avg Choice
---
Length
Commonsense QA | Commonsense Reasoning | 500 | 2 | 0.514 | 15.60 | 2.06
Social IQa | Commonsense Reasoning | 500 | 2 | 0.490 | 24.21 | 4.09
ARC | Scientific Reasoning | 500 | 2 | 0.486 | 25.78 | 6.31
OpenBook QA | Scientific Reasoning | 500 | 2 | 0.495 | 11.72 | 3.72
Table 2: Summary statistics for the datasets used in our experiments. Average length is calculated with the GPT-4 tokenizer, implemented through the tiktoken library. Model | Without COT | With COT
---|---|---
DA Base | POE Base | Base Diff | Base $p$-val | DA COT | POE COT | COT Diff | COT $p$-val
LLaMA 7B | 0.816 | 0.320 | 0.496 | 0.000 | 0.860 | 0.458 | 0.402 | 0.000
LLaMA 13B | 0.858 | 0.266 | 0.592 | 0.000 | 0.868 | 0.420 | 0.448 | 0.000
LLaMA 70B | 0.930 | 0.846 | 0.084 | 0.000 | 0.914 | 0.808 | 0.106 | 0.000
Falcon 7B | 0.516 | 0.472 | 0.044 | 0.164 | 0.690 | 0.466 | 0.224 | 0.000
Falcon 40B | 0.892 | 0.740 | 0.152 | 0.000 | 0.876 | 0.530 | 0.346 | 0.000
Falcon 180B | 0.924 | 0.912 | 0.012 | 0.490 | 0.922 | 0.786 | 0.136 | 0.000
GPT 3.5 | 0.888 | 0.778 | 0.110 | 0.000 | 0.926 | 0.848 | 0.078 | 0.000
Table 3: Evaluation of Direct Answer (DA) and Process of Elimination (POE) strategies on Commonsense QA, with and without chain-of-thought reasoning. Diff denotes the difference between the accuracies of the two strategies. $p$-val corresponds to a difference in means t-test. Underlined values denote $p$-val $\leq$ 0.05. Model | Without COT | With COT
---|---|---
DA Base | POE Base | Base Diff | Base $p$-val | DA COT | POE COT | COT Diff | COT $p$-val
LLaMA 7B | 0.762 | 0.440 | 0.322 | 0.000 | 0.758 | 0.548 | 0.210 | 0.000
LLaMA 13B | 0.772 | 0.408 | 0.364 | 0.000 | 0.806 | 0.578 | 0.228 | 0.000
LLaMA 70B | 0.876 | 0.814 | 0.062 | 0.007 | 0.844 | 0.796 | 0.048 | 0.048
Falcon 7B | 0.518 | 0.500 | 0.018 | 0.570 | 0.650 | 0.482 | 0.168 | 0.000
Falcon 40B | 0.808 | 0.718 | 0.090 | 0.001 | 0.818 | 0.628 | 0.190 | 0.000
Falcon 180B | 0.858 | 0.802 | 0.056 | 0.018 | 0.840 | 0.716 | 0.124 | 0.000
GPT 3.5 | 0.838 | 0.820 | 0.018 | 0.450 | 0.858 | 0.748 | 0.110 | 0.000
Table 4: Evaluation of Direct Answer (DA) and Process of Elimination (POE) strategies on Social IQa, with and without chain-of-thought reasoning. Diff denotes the difference between the accuracies of the two strategies. $p$-val corresponds to a difference in means t-test. Underlined values denote $p$-val $\leq$ 0.05. Model | Without COT | With COT
---|---|---
DA Base | POE Base | Base Diff | Base $p$-val | DA COT | POE COT | COT Diff | COT $p$-val
LLaMA 7B | 0.770 | 0.478 | 0.292 | 0.000 | 0.784 | 0.494 | 0.290 | 0.000
LLaMA 13B | 0.816 | 0.266 | 0.550 | 0.000 | 0.830 | 0.578 | 0.252 | 0.000
LLaMA 70B | 0.930 | 0.744 | 0.186 | 0.000 | 0.930 | 0.874 | 0.056 | 0.003
Falcon 7B | 0.544 | 0.470 | 0.074 | 0.019 | 0.544 | 0.420 | 0.124 | 0.000
Falcon 40B | 0.836 | 0.656 | 0.180 | 0.000 | 0.804 | 0.542 | 0.262 | 0.000
Falcon 180B | 0.938 | 0.874 | 0.064 | 0.001 | 0.888 | 0.814 | 0.074 | 0.001
GPT 3.5 | 0.914 | 0.856 | 0.058 | 0.004 | 0.946 | 0.852 | 0.094 | 0.000
Table 5: Evaluation of Direct Answer (DA) and Process of Elimination (POE) strategies on ARC, with and without chain-of-thought reasoning. Diff denotes the difference between the accuracies of the two strategies. $p$-val corresponds to a difference in means t-test. Underlined values denote $p$-val $\leq$ 0.05. Model | Without COT | With COT
---|---|---
DA Base | POE Base | Base Diff | Base $p$-val | DA COT | POE COT | COT Diff | COT $p$-val
LLaMA 7B | 0.790 | 0.399 | 0.391 | 0.000 | 0.850 | 0.593 | 0.257 | 0.000
LLaMA 13B | 0.818 | 0.277 | 0.541 | 0.000 | 0.856 | 0.529 | 0.327 | 0.000
LLaMA 70B | 0.938 | 0.677 | 0.261 | 0.000 | 0.922 | 0.824 | 0.098 | 0.000
Falcon 7B | 0.523 | 0.483 | 0.040 | 0.206 | 0.629 | 0.453 | 0.176 | 0.000
Falcon 40B | 0.848 | 0.747 | 0.100 | 0.000 | 0.868 | 0.683 | 0.184 | 0.000
Falcon 180B | 0.924 | 0.888 | 0.036 | 0.051 | 0.928 | 0.812 | 0.116 | 0.000
GPT 3.5 | 0.892 | 0.745 | 0.146 | 0.000 | 0.908 | 0.850 | 0.058 | 0.005
Table 6: Evaluation of Direct Answer (DA) and Process of Elimination (POE) strategies on OpenBookQA, with and without chain-of-thought reasoning. Diff denotes the difference between the accuracies of the two strategies. $p$-val corresponds to a difference in means t-test. Underlined values denote $p$-val $\leq$ 0.05. | _LLaMA-2 7B_
---|---
Dataset | Base | COT
DA Self | PoE Self | DA vs. PoE | DA Self | PoE Self | DA vs. PoE
Commonsense QA | 90.6 | 78.8 | 37.4 | 88.2 | 68.1 | 47.9
Social IQa | 90.0 | 74.6 | 44.0 | 83.2 | 66.2 | 52.8
ARC | 89.0 | 74.1 | 51.0 | 84.6 | 71.2 | 45.3
OpenBookQA | 86.3 | 77.9 | 35.1 | 84.7 | 70.5 | 52.7
Table 7: Self-Consistency of the DA and PoE strategies versus the consistency of DA and PoE for LLaMA-2 7B. Least consistent methods are underlined. | _LLaMA-2 13B_
---|---
Dataset | Base | COT
DA Self | PoE Self | DA vs. PoE | DA Self | PoE Self | DA vs. PoE
Commonsense QA | 92.6 | 74.4 | 25.3 | 91.2 | 69.1 | 42.3
Social IQa | 92.7 | 64.6 | 42.4 | 82.8 | 71.0 | 55.3
ARC | 92.8 | 75.9 | 21.7 | 87.5 | 71.4 | 55.4
OpenBookQA | 86.5 | 76.7 | 23.7 | 87.9 | 69.8 | 50.2
Table 8: Self-Consistency of the DA and PoE strategies versus the consistency of DA and PoE for LLaMA-2 13B. Least consistent methods are underlined. | _LLaMA-2 70B_
---|---
Dataset | Base | COT
DA Self | PoE Self | DA vs. PoE | DA Self | PoE Self | DA vs. PoE
Commonsense QA | 97.3 | 83.3 | 86.0 | 94.4 | 83.0 | 80.8
Social IQa | 95.2 | 90.9 | 86.9 | 90.0 | 85.9 | 77.6
ARC | 97.8 | 89.5 | 77.8 | 93.5 | 89.6 | 85.3
OpenBookQA | 96.8 | 72.9 | 67.6 | 93.1 | 84.8 | 81.4
Table 9: Self-Consistency of the DA and PoE strategies versus the consistency of DA and PoE for LLaMA-2 70B. Least consistent methods are underlined. | _Falcon 7B_
---|---
Dataset | Base | COT
DA Self | PoE Self | DA vs. PoE | DA Self | PoE Self | DA vs. PoE
Commonsense QA | 52.6 | 56.3 | 50.6 | 81.8 | 85.0 | 25.6
Social IQa | 54.2 | 63.9 | 50.8 | 84.1 | 82.6 | 27.2
ARC | 58.7 | 50.3 | 49.2 | 78.1 | 74.6 | 30.1
OpenBookQA | 55.5 | 59.1 | 52.1 | 82.2 | 76.9 | 31.5
Table 10: Self-Consistency of the DA and PoE strategies versus the consistency of DA and PoE for Falcon 7B. Least consistent methods are underlined. | _Falcon 40B_
---|---
Dataset | Base | COT
DA Self | PoE Self | DA vs. PoE | DA Self | PoE Self | DA vs. PoE
Commonsense QA | 97.3 | 84.8 | 76.1 | 91.4 | 67.6 | 54.8
Social IQa | 94.5 | 88.0 | 76.0 | 86.2 | 75.0 | 61.5
ARC | 95.0 | 87.5 | 68.1 | 84.8 | 70.1 | 50.3
OpenBookQA | 94.1 | 90.7 | 73.6 | 88.4 | 71.7 | 61.9
Table 11: Self-Consistency of the DA and PoE strategies versus the consistency of DA and PoE for Falcon 40B. Least consistent methods are underlined. | _Falcon 180B_
---|---
Dataset | Base | COT
DA Self | PoE Self | DA vs. PoE | DA Self | PoE Self | DA vs. PoE
Commonsense QA | 97.6 | 94.1 | 91.4 | 92.9 | 83.1 | 78.2
Social IQa | 95.0 | 92.8 | 86.6 | 91.4 | 81.2 | 69.4
ARC | 98.5 | 94.1 | 89.3 | 90.8 | 88.3 | 79.1
OpenBookQA | 96.2 | 94.4 | 89.7 | 95.1 | 83.1 | 77.2
Table 12: Self-Consistency of the DA and PoE strategies versus the consistency of DA and PoE for Falcon 180B. Least consistent methods are underlined. | _GPT-3.5_
---|---
Dataset | Base | COT
DA Self | PoE Self | DA vs. PoE | DA Self | PoE Self | DA vs. PoE
Commonsense QA | 97.3 | 89.2 | 81.3 | 97.6 | 93.3 | 86.0
Social IQa | 97.0 | 91.6 | 80.7 | 95.2 | 89.1 | 76.2
ARC | 97.5 | 94.0 | 86.6 | 97.0 | 93.5 | 85.5
OpenBookQA | 98.2 | 82.2 | 72.3 | 96.8 | 93.1 | 81.4
Table 13: Self-Consistency of the DA and PoE strategies versus the consistency
of DA and PoE for GPT-3.5. Least consistent methods are underlined.

Error Type | Question | Output
---|---|---
Misaligned Rationale |  Question: Which biomolecule does not have a carbon-nitrogen bond? Choices: (A) nucleic acid (B) carbohydrate Correct Answer: (B) carbohydrate |  Carbohydrates are made up of carbon, hydrogen, and oxygen. There is no nitrogen in carbohydrates. So the incorrect answer is "carbohydrate" which is choice (B)
Misaligned Rationale |  Question: What is someone who isnt clever, bright, or competent called? Choices: (A) ineffectual (B) stupid Correct Answer: (B) stupid |  The term "stupid" is used to describe someone who is not clever, bright, or competent. So the incorrect answer is "stupid" which is choice (B)
Logical Error |  Question: Which substance will retain the largest amount of energy from the Sun? Choices: (A) sand (B) grass Correct Answer: (A) sand |  Sand is a poor conductor of heat, so it will not retain much energy from the Sun. So the incorrect answer is "sand" which is choice (A)
Logical Error |  Question: Miss Grady took a stick from Bob because he was playing with it in class. She wanted to make sure that he couldn’t get to it so she put it where? Choices: (A) desk drawer (B) office Correct Answer: (A) desk drawer |  Miss Grady wanted to make sure that Bob couldn’t get to the stick, so she would not have put it in the desk drawer where he could easily access it. So the incorrect answer is "desk drawer" which is choice (A)
No Breakdown |  Question: Which object in the solar system is orbited by a belt of asteroids? Choices: (A) Saturn (B) the Sun Correct Answer: (B) the Sun |  The Sun is not orbited by a belt of asteroids. So the incorrect answer is "the Sun" which is choice (B)
No Breakdown |  Question: Where would you get some maps that you own? Choices: (A) cabinet (B) important when traveling Correct Answer: (A) cabinet |  Maps are important when traveling, but you would not get them from a cabinet. So the incorrect answer is "cabinet" which is choice (A)
Hallucination |  Question: If a hyrdrogen atom gives up 1 electron, what is its overall charge? Choices: (A) +1 (B) +2 Correct Answer: (A) +1 |  Atoms have a neutral charge, so if an atom gives up one electron, it will have a positive charge. So the incorrect answer is "neutral" which is choice (A)
Hallucination |  Question: What do the feathers look like on birds found in the rainforest? Choices: (A) singing (B) very colorful Correct Answer: (B) very colorful |  Feathers on birds found in the rainforest can be very colorful, but not all birds found in the rainforest have colorful feathers. So the incorrect answer is "very colorful" which is choice (B)
PoE N/A |  Question: Arturo ran a 3,000-meter race. His running time from start to finish was 10 minutes. What was Arturo’s average speed? Choices: (A) 5 m/s (B) 100 m/s Correct Answer: (A) 5 m/s |  The question states that Arturo ran the race in 10 minutes, but the question does not state the number of seconds in 10 minutes. So the incorrect answer is "5 m/s" which is choice (A)
Dataset Issue |  Question: When someone is physically competing what does their body do? Choices: (A) releases heat (B) sweat Correct Answer: (B) sweat |  When someone is physically competing, their body releases heat. Sweat is a result of the body releasing heat. So the incorrect answer is "sweat" which is choice (B)
Table 14: Examples of errors in PoE COT rationales uncovered during the error analysis. Model | Question | Output
---|---|---
LLaMA-2 70B |  Question: Which of these is a response of cats to external stimuli? Choices: (A) hairs on the back stand up when scared (B) movement of waste into the bladder Correct Answer: (A) hairs on the back stand up when scared |  The movement of waste into the bladder is an internal process, so it is not a response to an external stimulus. So the incorrect answer is "movement of waste into the bladder" which is choice (B)
LLaMA-2 70B |  Question: Where would someone keep their nylon leggings? Choices: (A) rope (B) drawer Correct Answer: (B) drawer |  Nylon leggings are a type of clothing and are not used to tie things together. So the incorrect answer is "rope" which is choice (A)
Falcon 180B |  Question: Which is a renewable natural resource that can be replenished over a period of time? Choices: (A) coal (B) trees Correct Answer: (B) trees |  Coal is a nonrenewable natural resource because it takes millions of years to form. So the incorrect answer is "coal" which is choice (A)
Falcon 180B |  Question: What do you do in order to fly in airplane? Choices: (A) buy tickets (B) add gas Correct Answer: (A) buy tickets |  Airplanes are large and require a lot of gas. However, passengers do not add gas to the airplane. So the incorrect answer is "add gas" which is choice (B)
GPT 3.5 |  Question: Which object occupies the greatest amount of space? Choices: (A) a galaxy (B) a neutron star Correct Answer: (A) a galaxy |  A neutron star is much smaller and denser than a galaxy, so it does not occupy the greatest amount of space. So the incorrect answer is "a neutron star" which is choice (B)
GPT 3.5 |  Question: Anybody could be hired in the kitchen, what was needed of them? Choices: (A) forget (B) wash dishes Correct Answer: (B) wash dishes |  If someone is hired in the kitchen, they are expected to work and perform tasks. Forgetting is not a desirable trait or skill in a kitchen. So the incorrect answer is "forget" which is choice (A)
Table 15: Examples of sound rationales elicited from PoE COT. Figure 4: Error
distribution of PoE COT and DA COT on ARC and Commonsense QA

Metrics LLaMA 7b LLaMA 13b LLaMA 70b Base COT Base COT Base COT 5 Choices Raw
Accuracy 0.530 0.664 0.646 0.69 0.814 0.782 DA Accuracy 0.530 0.665 0.646
0.691 0.814 0.784 PoE Accuracy 0.594 0.680 0.456 0.782 0.942 0.940 4 Choices
Raw Accuracy 0.322 0.450 0.262 0.580 0.782 0.774 DA Accuracy 0.542 0.662 0.575
0.742 0.830 0.823 PoE Accuracy 0.362 0.486 0.230 0.526 0.902 0.864 3 Choices
Raw Accuracy 0.232 0.350 0.158 0.416 0.778 0.710 DA Accuracy 0.641 0.720 0.687
0.791 0.863 0.826 PoE Accuracy 0.224 0.348 0.128 0.376 0.788 0.760 2 Choices
Raw Accuracy 0.168 0.270 0.092 0.330 0.706 0.662 DA Accuracy 0.750 0.776 0.719
0.887 0.896 0.873 PoE Accuracy 0.084 0.194 0.056 0.210 0.636 0.598

Table 16: Iterative process of elimination results on Commonsense QA. Raw
accuracy denotes the proportion of all questions that are answered correctly.
DA Accuracy and PoE Accuracy are the accuracies of the two strategies,
accounting for the errors that could have previously been made by PoE. DA and
PoE Accuracy are computed as the accuracy conditioned on if the remaining
choices still contain the gold answer.

Metrics LLaMA 7b LLaMA 13b LLaMA 70b Normal COT Normal COT Normal COT 4
Choices Raw Accuracy 0.526 0.596 0.642 0.664 0.834 0.814 DA Accuracy 0.530
0.601 0.647 0.671 0.844 0.821 PoE Accuracy 0.572 0.642 0.458 0.634 0.926 0.892
3 Choices Raw Accuracy 0.322 0.422 0.278 0.426 0.812 0.750 DA Accuracy 0.563
0.659 0.61 0.681 0.877 0.854 PoE Accuracy 0.334 0.380 0.238 0.392 0.832 0.808
2 Choices Raw Accuracy 0.232 0.288 0.176 0.306 0.768 0.744 DA Accuracy 0.699
0.766 0.752 0.797 0.923 0.937 PoE Accuracy 0.150 0.216 0.106 0.222 0.624 0.708

Table 17: Iterative process of elimination results on ARC. Raw accuracy
denotes the proportion of all questions that are answered correctly. DA
Accuracy and PoE Accuracy are the accuracies of the two strategies, accounting
for the errors that could have previously been made by PoE. DA and PoE
Accuracy are computed as the accuracy conditioned on if the remaining choices
still contain the gold answer.

Figure 5:  Five in-context learning examples for DA COT on Commonsense QA

Figure 6:  Five in-context learning examples for PoE COT on Commonsense QA

Figure 7:  Five in-context learning examples for DA COT on Social IQa

Figure 8:  Five in-context learning examples for PoE COT on Social IQa

Figure 9:  Five in-context learning examples for DA COT on ARC

Figure 10:  Five in-context learning examples for PoE COT on ARC

Figure 11:  Five in-context learning examples for DA COT on OpenBookQA

Figure 12:  Five in-context learning examples for PoE COT on OpenBookQA


# Superkernel Neural Architecture Search for Image Denoising

Marcin Możejko
<EMAIL_ADDRESS>Tomasz Latkowski
<EMAIL_ADDRESS>Łukasz Treszczotko
<EMAIL_ADDRESS>Michał Szafraniuk
<EMAIL_ADDRESS>Krzysztof Trojanowski
<EMAIL_ADDRESS>
###### Abstract

Recent advancements in Neural Architecture Search (NAS) resulted in finding
new state-of-the-art Artificial Neural Network (ANN) solutions for tasks like
image classification, object detection, or semantic segmentation without
substantial human supervision. In this paper, we focus on exploring NAS for a
dense prediction task that is image denoising. Due to a costly training
procedure, most NAS solutions for image enhancement rely on reinforcement
learning or evolutionary algorithm exploration, which usually take weeks (or
even months) to train. Therefore, we introduce a new efficient implementation
of various superkernel techniques that enable fast (6-8 RTX2080 GPU hours)
single-shot training of models for dense predictions. We demonstrate the
effectiveness of our method on the SIDD+ benchmark for image denoising [11].

##  1 Introduction

Neural architecture search (NAS) seeks to automate the process of choosing an
optimal neural network architecture. Thanks to the trainable formulation of a
model structure selection, it enables optimization not only for the underlying
task but also for additional execution properties, _e.g_. memory constraints,
or inference time. A number of search techniques have been proposed, including
ones based on reinforcement learning [41], evolutionary algorithms [29] or
gradient descent [22, 36, 20, 5].

The greatest challenge posed by most NAS algorithms is a trade-off between
search time and memory requirements of the search procedure. In classical
approaches, namely reinforcement learning or evolutionary algorithms, the
statistics collected during parallel training of multiple samples from the
architecture space drive the exploration process. As the exploration algorithm
usually needs to train these samples from scratch, the search procedure
requires hundreds [21] or thousands [42, 28, 6] of GPU hours to be performed.
In order to alleviate this problem, multiple techniques were proposed. One of
the most popular ones either narrows down the search to find a local cell
structure which is later replicated throughout the network [42] or sharing
weights across trained model samples [27]. It is crucial to notice that the
memory requirements of a given training procedure are proportional to the most
memory consuming model from the explored search space. This property makes
these methods a good first shot at performing neural architecture search for
image enhancement tasks, where the size of individual models often fills the
entire GPU memory.

An alternative approach, which is usually called single-shot [3, 4, 15]
search, seeks a different solution to the aforementioned issues. It models the
search space in the form of a supernetwork, which is also a neural network
itself. Then the search process is performed using gradient descent. These
methods frequently reduce the overall search time from a couple of weeks to
even a few hours [32]. Unfortunately, a huge downside of this approach is the
fact that since supernetwork contains the whole search space, its memory
requirements are significantly larger than those of typical network from the
search space. This issue is even more important for the case of dense
prediction tasks, where even a single model might be highly demanding in terms
of memory use.

An efficient midway point is the superkernel approach. It reduces the
structural supernetwork part to a single convolutional kernel, which cuts its
memory requirements by order of magnitude. This improvement is achieved at the
cost of narrowing down the search space selection to kernel sizes and the
number of filters of a standard convolution operator.

A common solution to both prolonged training time and memory usage is using a
proxy dataset. This dataset is usually smaller, both in terms of the number of
examples and image size (_e.g_. using CIFAR10 instead of ImageNet for image
classification). After initial training on a smaller dataset, the search
procedure transfers the final architecture to the target task. Several
approaches try to perform an architecture search procedure in a proxyless
manner, directly on target data. This is achieved by either using a simplified
search space [36] or by compressing / pruning supernet during training [5].
The proxy-less approach may be particularly appealing for image enhancement
tasks (like image denoising). For these tasks, using a proxy dataset is hard
as proper training procedure requires high-resolution images.

In this work, we propose a new relaxed superkernel solution for image
denoising, which is fast in training (6-8 GPU hours), memory-efficient (a
supernetwork fits a single GPU with a batch size of 4) and might be trained in
a proxyless manner using input image resolution of 128x128. We evaluate our
models on a SIDD+ [11], (part of NTIRE 2020 Challenge on Real Image Denoising
[2, 11, 1]) dataset for image denoising achieving state-of-the-art results.

##  2 Related work

###  2.1 Reinforcement Learning and Evolutionary algorithms for Neural
architecture search

Initial approaches tackled Neural Architecture Search by using reinforcement
learning and evolutionary algorithms. A seminal paper [41] used a
reinforcement learning controller in order to produce network structures which
were then trained from scratch and evaluated. Because of that, the whole
training procedure took thousands of GPU hours to train a state-of-the-art
model for the CIFAR10 dataset. In order to speed up computations, in the
following work [42], the search space was narrowed down to only a small number
of subnetwork structures, later repeated to form the final architecture. This
approach was later improved by sharing weights across different subnetworks in
order to transfer knowledge [27]. Similarly, [28] used a genetic algorithm
instead of reinforcement learning to drive the optimization process, whereas
[6] combined both of these approaches.

As memory requirements of these methods are proportional to the memory
requirements of the biggest model in search space, this technique was used for
memory demanding image enhancement tasks like super-resolution [14, 31, 7],
medical image denoising [23], image restoration [37, 34, 16] and image
inpainting [19].

###  2.2 Single-shot approaches

Another approach is to model the search space itself in the form of a neural
network. Usually, it is done by a continuous sampling relaxation procedure,
which approximates the discrete process of architecture selection in a
differentiable manner. _e.g_. in [22] authors used softmax weights to
approximate operation selection. More precisely, the approximation was
achieved by combining outputs from each operation using softmax weights with
learnable logits. The supernetwork was trained using a two-level gradient-
based optimization process to separate training of convolutional weights and
structural parameters. The final architecture was obtained by choosing
operations by the highest logit value.

Authors of [20] pushed this approach even further, and used softmax weights as
a relaxation of both global CNN architecture (channels, strides, depth,
connectivity) and local cell structure. The algorithm finds a network for a
semantic image segmentation task that adjusts its architecture to a provided
dataset. A search space forms a grid built of a local layer structure called a
cell. Each cell is a general acyclic graph with learnable connections between
$k$ individually chosen and optimized operations. A grid consists of a
replicated cell and a connectivity structure, which enables modeling multiple
popular CNN designs like DeepLabv3, AutoEncoder, or Stacked Hourglass. The
architecture search phase lasts several days on a P100 GPU.

Given that softmax weights can be an inaccurate approximation of discrete
sampling when the entropy of modeled distribution is high, [36] introduced
Gumbel Softmax approximation to neural architecture search [17]. In this
technique, stochastic weights, used for combining different operations, seem
to resemble one-hot coefficients better than softmax. The proposed solution is
a computationally light-weight algorithm that finds a device-aware CNN
architecture for an image classification task. The algorithm works on a
predefined global CNN structure, /ie, channels, strides, and depth. The search
space is spanned by a set of tensor operations, including different
convolution setups, pooling, and skip connections. Optimal operation is found
via gradient descent for each layer individually, taking into account
classification metrics as well as FLOPs. The algorithm finds different
architectures for different hardware setups, i.e., Samsung S10, and iPhone X.
Again, the architecture search phase is time-consuming at hundreds of GPU
hours.

The massive downside of the single-shot methods presented above is that they
require computations of all possible operations in order to perform a single
iteration of the search. In [5], authors try to circumvent this issue by
introducing search space subsampling, which decreases memory cost. At each
iteration, a training update is performed only on a randomly selected subset
of possible operations. Still, these models usually require approximately
twice as much memory as the most demanding model from the search space.

###  2.3 Superkernels

In order to alleviate memory issues, [32] introduces a new technique called a
superkernel. In this method, the authors switch the search procedure from
operation selection to kernel selection. It uses a concrete distribution in
order to zero out parts of the maximal kernel, which is trained and shared
across all models from the search space. As the size of the output of the
convolution operation is usually of a few orders of magnitude greater than the
size of its parameters, the additional cost of a search performed in the
kernel space is negligible. In 4, we introduce a few new variations of this
technique based on the Gumbel Softmax relaxation [17].

###  2.4 Deep Learning for Image denoising

Several deep learning approaches were applied to image denoising task since
the first successful MLP approach [30] achieved performance comparable to then
state-of-the-art BM3D [9] approach. In [38], the authors used stacked
convolutional blocks in order to approximate the residuum between noisy image
and its cleaned version. Similarly, [25] introduced a residual connection not
only between the input and the output of a network but also between model
chained convolutional blocks. In Memnet [33], the authors introduced memory
mechanisms based on recurrent and gating units in order to combine multi-level
representations of an input image.

All of the methods presented above do not use any form of pooling/downsampling
that introduces a high computational cost. In order to alleviate this issue
[24] used U-NET architecture. By applying the pooling / upsampling operations,
they decreased the resolution of the inner convolutional volumes. It
significantly cut the memory and computational burden of computations. In SGN
[13], the authors noticed that using traditional pooling / upsample operations
significantly decreased the quality of reconstruction as a lot of low-level
image information is lost. Because of that, they introduced a shuffling
(subpixel) rescaling designed to keep more details without significant growth
in a computational cost. Moreover - thanks to a top-down self-guidance
algorithm enables a light-weight combination of the multiscale image features.

##  3 Models for image denoising

In this section, we will introduce the base architectures that we used in our
experiments. We transformed each of them into Neural Architecture Search
supernetworks by introducing superkernels instead of standard convolutional
filters.

###  3.1 Superkernel-based Multi Attentional Residual U-Net

The proposed architecture of the Superkernel-based Multi Attentional Residual
U-Net network is shown in Figure 1. The network comprises multiple subnetworks
named SK-A-RES-UNET trained simultaneously. The outcome of each subnetwork is
passed to the channel attention block [39]. As the final layer, we used
convolution with a kernel size of $k=3$ to reduce the number of channels. The
final output is additionally summed up with the input image.

Figure 1: The architecture of Superkernel-based Multi Attentional Residual
U-Net network Figure 2: The architecture of Superkernel-based Attentional
Residual U-Net subnetwork

The architecture of Superkernel-based Attentional Residual U-Net subnetwork is
shown in Figure 2. This network is a modification of U-Net architecture,
additionally equipped with attention mechanism and superkernel-based densely
connected residual blocks (SK-DCRB). Each encoder layer has the same
structure. Specifically, it consists of a convolutional layer with kernel size
$k=3$ followed by the ReLU activation function and SK-DCR block. As in vanilla
U-Net architecture, the output of the encoder layer is passed to the decoder
layer at the same spatial level. Each decoder layer takes the output from skip
connection and the result from the previous layer. Unlike the common U-Net
architectures, after the concatenation operation in each decoder layer, the
channel attention block (CAB) is applied. We used the convolutions with a
stride of 2 for down-sampling and shuffle / subpixel layers [13] for up-
sampling.

Figure 3: The outline of Superkernel Densely Connected Residual Block

Figure 3 presents the schema of Superkernel Densely Connected Residual Block.
This block has a similar structure as proposed in [26]. Each SK-DCR block
comprises three convolutional layers followed by ReLU activation functions. We
substituted the first two convolutional operations with superkernel. This
approach allows the network to learn the most appropriate kernel size and
growth rate. The output convolutional layer restores the number of filters.

###  3.2 Superkernel SkipInit Residual U-net

The architecture of the network is similar to the one delineated in Figure 2.
The part that differs is the actual convolutional blocks. In Superkernel-based
SkipInit Residual U-net there are multiple (2 in the chosen network) Densely
Connected Residual Blocks (see for instance [40]) with a SkipInit scalar
multiplication to stabilize learning as proposed in [10]. The outline of this
block is presented in Figure 4. Briefly, the output of each residual branch is
multiplied by a trainable scalar, which is initialized to zero, thus making
the block working like identity transformation at the beginning. If the
optimization chooses to use the residual branch, then this scalar (denoted by
alfa in Figure 4) will alter. The number of filters at a particular level is
double the number of filters at the level directly above it, starting with 64
at level $0$. The training of convolutional hyperparameters follows the Joint
Superkernel regime described in Section 4.1.

Figure 4: The architecture of Densely Connected Residual Block with SkipInit
Connection subnetwork

##  4 Superkernels

In each of these architectures, we applied a neural architecture search in
order to find the optimal number of filters and kernel sizes for each
convolution block. The search procedure was based on multiple modifications of
a superkernel technique, namely:

  * •

Factorized Gumbel Superkernel,

  * •

Joint Gumbel Superkernel,

  * •

Filterwise Gumbel Superkernel,

  * •

Filterwise Attention-based Superkernel.

In each of these methods, the neural architecture search explores different
possibilities of kernel sizes and the number of filters. Each selection is
obtained by appropriate slicing of a maximal kernel, called superkernel, and
the way how this slicing is performed differs across different methods. In
each, the algorithm aims to optimize the structural distribution over a set of
possible slices. We will later refer to the procedure of optimization of these
structural parameters as search. Below one may find a detailed description of
each NAS method.

###  4.1 Joint Superkernel

Using this technique, search is performed over a set of slices $S_{k,f}$ where
$k\in\\{k_{1},\dots k_{n_{k}}\\}$ is a size of a kernel and
$f\in\\{f_{1},\dots,f_{n_{f}}\\}$ is a number of filters. Both values $f_{i}$
and $k_{i}$ differ across different models and architectures. For each of
these superkernels a maximal superkernel with a kernel size of
$\max_{i=1,\dots n_{k}}k_{i}$ and $\max_{i=1,\dots n_{f}}f_{i}$ filter numbers
is trained and shared across every subkernel. A slice $S_{k,f}$ consists of a
centered subkernel with a kernel size $k$ and first $f$ filters of the maximal
superkernel (see Figure 5).

The structural distribution is modeled as a softmax distribution over a set of
all possible tuples
$\left\\{(k,f):k\in\\{k_{1},\dots,k_{n_{k}}\\},f\in\\{f_{1},\dots,f_{n_{f}}\\}\right\\},$
with a single logit parameter $\theta_{(k,f)}$ per tuple.

###  4.2 Factorized Superkernel

In this technique the set of subkernels is the same as for Joint Superkernel
in Sec 4.1, but the structural distribution is factorized into two independent
distributions $p_{k,f}=p_{k}p_{f},$ where $p_{k}$ and $p_{f}$ are softmax
distributions over a set of possible kernels and filter sizes. In this
scenario the optimized parameters form two sets of logit parameters:
$\left\\{\theta_{k_{i}}:k_{i}\in\\{1,\dots n_{k}\\}\right\\}$ for kernel
slices and $\left\\{\theta_{f_{i}}:f_{i}\in\\{1,\dots n_{f}\\}\right\\}$ for
filter slices. In comparison to Joint Superkernel, such factorization
significantly reduces number of parameters of structural distributions (from
$n_{f}n_{k}$ to $n_{f}+n_{k}$) at the cost of smaller modeling flexibility.

Figure 5: The schema of Joint Superkernel and Factorized Superkernel slice
generation. A slice generated from choosing kernel size 1 from possible
$k\in\\{1,3\\}$ and filter size 32 from possible $f\in\\{32,48\\}$.

###  4.3 Filterwise Superkernel

In this technique the search is performed over a set of slices $S_{k,M}$ where
$k\in\\{k_{1},\dots,k_{n}\\}$ \- is a size of a kernel and $M\in\\{0,1\\}^{F}$
is the set of all possible subsets of $F$ potential filters. For each
subkernel a maximal superkernel with kernel size of $\max k_{i}$ and $F$
filters is trained and shared across every subkernel. A slice $S_{k,M}$
consists of a centered subkernel with kernel size $k$ and and those filters
for which $M_{i}=1$ (see Figure 6).

The structural distribution is factorized into two independent distributions
$p_{k,M}=p_{k}p_{M}$ where $p_{k}$ is a softmax distribution over the set of
possible kernels and $p_{M}$ is modelled by $F$ $i.i.d.$ Bernoulli
distributions. $p_{k}$ distribution is parametrized by a set of logit
parameters $\left\\{\theta_{k_{i}}:i\in\\{1,\dots,k_{n}\\}\right\\}$, where
each of $F$ independent Bernoulli filterwise distribution over $M_{i}$ is
controlled by a single logit parameter $\theta_{f_{i}}.$ .

###  4.4 Filterwise Attention-based Superkernel

In this technique, the set of subkernels is the same as in Filterwise
Superkernel (Sec. 4.3). Similarly, the distribution over kernel size $k$ is
also a softmax distribution over possible kernels set. A substantial
difference is in the way the distributions over masks
$M\in\left\\{0,1\right\\}^{F}$ are governed. Once again, each distribution
over mask $M_{i}$ is independent Bernoulli. This time, however, each of these
distributions is parametrized by a single base logit parameter
$\theta^{b}_{f_{i}}$ and an attention key vector $v^{A}_{i}\in\mathbb{R}^{l}$.
The final logit of Bernoulli distribution $\theta_{f_{i}}$ is computed
according to attention mechanism [35]:

$\theta_{f_{i}}=softmax\left(v^{A}(v^{A})^{T}\right)\theta^{b}_{f_{i}}.$ (1)

Figure 6: The schema of Filterwise Superkernel and Filterwise Attention-based
Superkernel slice generation. A slice generated by choosing kernel size 1 from
possible $k\in\\{1,3\\}$ and with selected filters $f\in\\{2,3,7,9\\}$ from 11
possible.

###  4.5 Details of superkernel implementation

####  4.5.1 Sampling Relaxation

As slicing is a non-continuous operation, structural parameters of networks
cannot be directly optimized using gradient descent. In order to enable this
type of training, we applied a continuous approximation of sampling, namely
Gumbel Softmax and Relaxed Bernoulli distributions [17]. We present an outline
of this relaxation for the case of Factorized Superkernel.

We can rewrite a convolution on a volume $x$ with a sampled subkernel
$S_{k,f}$ in the following manner:

$Conv\left(S_{k,f},x\right)=\sum_{S_{k^{\prime},f^{\prime}}}Conv\left(S_{k^{\prime},f^{\prime}},x\right)\mathbb{I}_{(k,f)=(k^{\prime},f^{\prime})},$
(2)

where $S_{k^{\prime},f^{\prime}}$ goes through the set of all possible
superkernels and $\mathbb{I}$ is an indicator function. The Gumbel relaxation
is equivalent to the following approximation:

$\sum_{S_{k^{\prime},f^{\prime}}}Conv\left(S_{k^{\prime},f^{\prime}},x\right)\mathbb{I}_{(k,f)=(k^{\prime},f^{\prime})}\approx\\\
\approx\sum_{S_{k^{\prime},f^{\prime}}}Conv\left(S_{k^{\prime},f^{\prime}},x\right)GS(k^{\prime},f^{\prime}),$
(3)

where $GS$ is a sample from a Gumbel Softmax distribution with
$\theta_{(k,f)}$ logit parametrization of a softmax distribution over a set of
possible slices tuples (see Section 4.1).

####  4.5.2 Mask sampling reparametrization

In the approximation presented above, one still needs to compute $n_{k}n_{f}$
operations in order to perform the final summation. We decided to decrease the
computational and memory burden by using a mask sampling trick. A superkernel
slice might be reparametrized as $S_{k,f}=S*I_{k,f}$ where $S$ is a
superkernel shared across every subkernel, $I_{k,f}$ is the appropriate
subkernel slice applied to all-ones kernel of the same shape as $S$ and $*$ is
the Hadamard product of two tensors. This reparametrization enables the
following reformulation of the sampling operation:

$\sum_{S_{k^{\prime},f^{\prime}}}Conv\left(S_{k^{\prime},f^{\prime}},x\right)GS(k^{\prime},f^{\prime})=\\\
=Conv\left(S*\sum_{S_{k^{\prime},f^{\prime}}}I_{k^{\prime},f^{\prime}}GS(k^{\prime},f^{\prime}),x\right).$
(4)

The RHS of the above equation needs only a single run of convolution operation
with the maximal superkernel $S$ masked with an average subkernel mask. This
substantially decreases computational burden, both in terms of FLOPs and
memory requirement, and enables full NAS treatment for image denoising.

####  4.5.3 Mask Sampling reparametrization - issue with a non-linear
activation function

One may notice that the derivation above is not accurate for a search
procedure which involves not only a $Conv$ operator but also a non-linear
activation function. This is because of the relation:

$\sum_{S_{k^{\prime},f^{\prime}}}f\left(Conv\left(S_{k^{\prime},f^{\prime}},x\right)GS(k^{\prime},f^{\prime})\right)\\\
=f\left(Conv\left(S*\sum_{S_{k^{\prime},f^{\prime}}}I_{k^{\prime},f^{\prime}}GS(k^{\prime},f^{\prime}),x\right)\right)$
(5)

holds only for additive functions $f$. Both ReLU and PReLU, which were used in
our experiments, do not have this property. Because of that, we tested two
possibilities:

  * •

full \- where in order to keep computational feasibility we decided to ignore
this fact and treat the RHS of the equation above as an approximation of the
LHS,

  * •

separate \- where all components of the LHS were computed separately.

We have tested both approaches on a set of smaller architectures and did not
notice any significant difference. If a model has a separate component in its
name, it indicates it was trained using a separate approach. Otherwise it was
trained using full treatment.

Table 1: Superkernel-based SkipInit Residual U-Net model with two different architectures (with self-ensemble). Superkernel Type | Network architecture
---|---
Model 1 | Model 2
PSNR | SSIM | PSNR | SSIM
no superkernel | $52.6738$ | $0.99170$ | $52.6883$ | $0.99170$
full | $52.7330$ | $0.99183$ | $\mathbf{52.7344}$ | 0.99183
separate | $\mathbf{52.74498}$ | $\mathbf{0.99184}$ |  |
filterwise | $52.6444$ | $0.99179$ | $52.6817$ | 0.99183
filterwise with attention | $52.7059$ | $0.99173$ | $52.7053$ | $0.99191$
Table 2: Superkernel-based SkipInit Residual U-Net model with two different architectures (without self-ensemble). Superkernel Type | Network architecture
---|---
Model 1 | Model 2
PSNR | SSIM | PSNR | SSIM
no superkernel | $52.4937$ | 0.99157 | $52.4490$ | $0.99140$
full | $52.4638$ | $0.99140$ | 52.4752 | $0.99144$
separate | 52.4957 | $0.99148$ |  |
filterwise | $52.4561$ | $0.99147$ | $52.4151$ | 0.99145
filterwise with attention | $52.4239$ | $99138$ | $52.4278$ | $0.99152$

####  4.5.4 Bias sampling

We have applied a similar sampling technique as in the case of convolutional
kernels to the biases. As the biases vector size is equal to the number of
filters, the appropriate mask for bias slice is obtained by averaging kernel
mask in all but filter dimensions.

####  4.5.5 Final model distillation

Once structural parameters and weights of a model are trained, we need to
distill the model with the best kernel and filter sizes. We applied the
following distillation strategy:

  * •

for structural parameters governed by a softmax distribution, an option with a
maximal probability was chosen,

  * •

for structural masks governed by a Bernoulli distribution, a filter was
selected if its logit was greater than $0.5$.

This distillation procedure was done independently for each superkernel. We
plan to use more sophisticated distillation strategies as part of our future
work.

##  5 Experimental results

We performed several experiments in raw RGB image denoising task in order to
test whether our approach to the search for the best kernel size/number of
output channels combination provides noticeable benefits to the baseline (no-
NAS) solution developed for the same task. In this section, we briefly
describe the training procedure and the models used. We also provide the final
results in terms of achieved PSNR and SSIM scores.

Table 3: Superkernel-based Multi Attentional Residual U-Net model for different number of subnetworks and superkernel types (without self-ensemble). Superkernel Type | Number of subnetworks $n$
---|---
$n$=1 |  $n$=3 |  $n$=6
PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
no superkernel | $52.3517$ | $\mathbf{0.99156}$ | $\mathbf{52.4523}$ | $0.99126$ | $\mathbf{52.4833}$ | $0.99136$
factorized | $52.2690$ | $0.99124$ | $52.3799$ | $0.99134$ | $52.4032$ | $\mathbf{0.99150}$
joint | $\mathbf{52.3925}$ | $0.99138$ | $52.4153$ | $0.99140$ | $52.4118$ | $0.99147$
filterwise | $52.2887$ | $0.99121$ | $52.3503$ | $\mathbf{0.99145}$ | $52.3516$ | $0.99131$
filterwise with attention | $52.2932$ | $0.99122$ | $52.4365$ | $0.99144$ | $52.4195$ | $0.99119$
$\mu$ $\pm$ $\sigma$ | $52.3190$ | $0.99132$ | $52.4069$ | $\mathbf{0.99138}$ | $\mathbf{52.4139}$ | $0.99137$
$\pm 0.0459$ |  $\pm 13.39e$-$5$ | $\pm 0.0373$ |  $\pm 7.05e$-$5$ | $\pm 0.0421$ |  $\pm 11.22e$-$5$
Table 4: Superkernel-based Multi Attentional Residual U-Net model for different number of subnetworks and superkernel types (with self-ensemble). Superkernel Type | Number of subnetworks $n$
---|---
$n$=1 |  $n$=3 |  $n$=6
PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
no superkernel | $52.4470$ | $0.99157$ | $52.5969$ | $0.99150$ | $52.6642$ | $0.99159$
factorized | $52.4998$ | $0.99157$ | $52.6184$ | $0.99167$ | $\mathbf{52.6981}$ | $\mathbf{0.99176}$
joint | $\mathbf{52.5668}$ | $\mathbf{0.99163}$ | $52.6016$ | $0.99164$ | $52.6189$ | $0.99159$
filterwise | $52.5144$ | $0.99150$ | $52.5982$ | $0.99180$ | $52.6654$ | $0.99163$
filterwise with attention | $52.5438$ | $0.99157$ | $\mathbf{52.6880}$ | $\mathbf{0.99186}$ | $52.5943$ | $0.99150$
$\mu$ $\pm$ $\sigma$ | $52.5144$ | $0.99157$ | $52.6206$ | $\mathbf{0.99169}$ | $\mathbf{52.6482}$ | $0.99161$
$\pm 0.0409$ |  $\pm 4.12e$-$5$ | $\pm 0.0346$ |  $\pm 12.64e$-$5$ | $\pm 0.0369$ |  $\pm 8.45e$-$5$

###  5.1 Training details

We trained all our models on the SSID+ [11] training data provided by
NTIRE2020 Real Image Denoising rawRGB competition organizers. In our training
procedure, we split the challenge images into two parts: 90% of images for a
training set, and the rest 10% for validation. The split is stratified with
respect to dataset mobile phone types. We used the validation data provided by
the organizers of the competition as the final test set. We cut the training
and validation images into 128x128 patches. We use Adam optimizer [18] with a
learning rate of $2*10^{-5}$ for model training. The learning rate was kept
constant during the learning process. As our pipeline comprised of a large set
of models of different architectures and capacities, we decided to use a
popular early stopping technique to constrain the training time. During the
learning process, we monitored the PSNR value on a validation dataset. We
evaluated our model after every 1K steps. In the case of no improvement in the
PSNR within 30 validation evaluations, we stopped the training process.
Depending on the hardware used (GeForce GTX 1080Ti or GeForce RTX 2080Ti), the
training of our models lasted from a few hours up to 3 days.

###  5.2 Superkernel SkipInit Residual U-net

The overall architecture of the models in this section is based on the U-net
presented in [26]. Model 1 in this subsection is the one described in Section
3.2 with U-net depth $3$, three searchable growth rates $0.2$, $0.4$ and $0.6$
in the DCR block. There are $2$ DCR blocks of depth $2$ at each U-net level
(both for the encoder and decoder parts). At level $0$, the maximum number of
possible output channels is $32$, and it increases with arithmetic progression
at each layer. Model 2 differs from Model 1 only in the U-net depth ($2$
instead of $3$) and the maximum number of channels at level $0$ ($64$ instead
of $32$). The performance of theses models can be seen in Tables 1 and 2.

####  5.2.1 Chosen architecture

The search space for both models included the choice of kernel size ($3$ or
$5$) and growth rate ($0.2$, $0.4$, $0.6$) for the DCR block, see Figure 4 and
Section 3.2. There where $28$ DCR blocks in Model 1 and $20$ in Model 2.

Model 1 in the full version of the superkernel joint framework, consecutively
chose kernel size $5$, with only a couple of exceptions. As regards growth
rates, all but three DCR blocks chose the largest growth rate possible. In the
separate version, the story was similar, with kernel size $3$ chosen only
once.

In Model 2 with full version of superkernel joint, which had a higher baseline
number of filters, but was shallower, as far as U-net architecture is
concerned, chose kernel size $5$ virtually everywhere. Growth rates were a bit
different, with approximately 40% DCR blocks choosing a growth rate of $0.4$
and the rest choosing $0.6$.

###  5.3 Superkernel-based Multi Attentional Residual U-Net

Tables 3 and 4 contain PSNR and SSIM scores for architecture described in
Section 3.1. We trained several models with a different number of subnetworks
and superkernel types. Table 3 lists results without self-ensemble whereas
models presented in Table 4 utilize self-ensemble technique. We use
architectures with 1, 3, and 6 subnetworks. Each subnetwork has the same
structure, as shown in Figure 2. As illustrated in Table 4, the application of
the self-ensemble technique significantly improved the average PSNR and SSIM
metrics. The best PSNR results were obtained for models containing six
subnetworks. The mean PSNR value of models equipped with six subnetworks (with
self-ensemble) outperformed the 1-subnetwork and 3-subnetwork models by 0.13
and 0.03, respectively.

####  5.3.1 Chosen architecture

The search space for the model consisted of kernel size and growth rate of
densely connected residual blocks. For every first two convolutional layers in
DCR block, the superkernel had to select a filter size in $k\in\\{3,5,7\\}$
and a growth rate in $1/r\in\\{1,0.5,0.25\\}$. The final superkernel choices
differed depending on the part of the network. In the encoder part, the
superkernel selected the kernel sizes of 5 and 7 and the growth rate of 1 and
0.5. In the decoder part, NAS selected the kernel sizes of 7 and the maximum
number of filters ($r=1$) with only some small exceptions.

##  6 Discussion

In all architectures introduced, at least one of Neural Architecture Search
methods significantly improved over no-NAS baseline. The best single model -
with $52.74$ PSNR used the separate joint superkernel - and was achieved by
SkipInit Residual U-net for Model 1. The difference between baseline
architectures and searched ones is the most significant (approx. $0.12$ of
PSNR) for small models (with single subnetwork), which is promising from the
perspective of model deployment on a mobile device.

The ablation study showed the importance of a self-ensembling technique. On
average, it provided an improvement in both PSNR ($0.21$) and SSIM ($0.0002$)
metrics. Interestingly, without self-ensembling, NAS models perform
significantly worse as compared to their no-NAS baselines.

##  7 Potential improvements and future work

Although our search procedure shows the most significant PSNR improvement for
small models, this comes at the cost of the selection of convolutional kernels
with more spacious kernel sizes and higher the number of filters. However, a
differentiable representation of kernel size enables the introduction of
additional computational constraints to an optimization process. We expect
that training equipped with additional computational regularization should
result in finding models deployable on portable devices.

Our search procedure resembles variational inference methods for Bayesian Deep
Learning; thus, our method might suffer from an insufficient exploration of a
search space as reported in [12]. This issue might arise as our methods
explore only points which are close to gradient descent trajectory, which
narrows the explored region of the search space and makes it potentially
sensitive to weight initialization. We are planning to explore this
sensitivity as our future work. Moreover, in our training procedure, both
weights and structural parameters are optimized jointly. We are planning to
explore a two-level optimization (similar to [22]) in order to inspect how the
coupling of these parameters affects the results.

As sampling within different superkernels is performed independently, there
may arise issues connected to fairness [8]. Because of that, the network might
quickly converge to a random sub-optimal solution, according to Matthew’s law.
We plan to test more advanced probabilistic models (_e.g_. based on Markov
Random Fields, recurrent neural networks, or attention) in order to model more
advanced search spaces and perform broader exploration.

Additionally, we observed that the training procedure could have a significant
impact on the final results. Specifically, models equipped with superkernel
might need longer training time (higher patience) in comparison to non-
superkernel models. We suspect that this might be a reason behind performance
drop for NAS models without self-ensembling. We plan to investigate this topic
in future work.

##  8 Conclusion

In this work, we introduced a fast and light-weight algorithm for neural
architecture search for image denoising. We proved that it could achieve
state-of-the-art results exceeding no-NAS solution by a significant margin. We
have shown that proposed superkernel techniques can achieve results comparable
to the state-of-the-art architectures for image denoising within few GPU
hours.

## References

  * [1] Abdelrahman Abdelhamed, Stephen Lin, and Michael S. Brown.  A high-quality denoising dataset for smartphone cameras.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [2] Abdelrahman Abdelhamed, Radu Timofte, Michael S. Brown, et al.  Ntire 2019 challenge on real image denoising: Methods and results.  In The IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), June 2019.
  * [3] Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le.  Understanding and simplifying one-shot architecture search.  In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 550–559, Stockholmsmässan, Stockholm Sweden, 10–15 Jul 2018. PMLR.  http://proceedings.mlr.press/v80/bender18a.html.
  * [4] Andrew Brock, Theo Lim, J.M. Ritchie, and Nick Weston.  SMASH: One-shot model architecture search through hypernetworks.  In International Conference on Learning Representations, 2018.  https://openreview.net/forum?id=rydeCEhs-.
  * [5] Han Cai, Ligeng Zhu, and Song Han.  ProxylessNAS: Direct neural architecture search on target task and hardware.  In International Conference on Learning Representations, 2019.  https://openreview.net/forum?id=HylVB3AqYm.
  * [6] Liang-Chieh Chen, Maxwell Collins, Yukun Zhu, George Papandreou, Barret Zoph, Florian Schroff, Hartwig Adam, and Jon Shlens.  Searching for efficient multi-scale architectures for dense image prediction.  In Advances in neural information processing systems, pages 8699–8710, 2018.
  * [7] Xiangxiang Chu, Bo Zhang, Hailong Ma, Ruijun Xu, Jixiang Li, and Qingyuan Li.  Fast, accurate and lightweight super-resolution with neural architecture search.  CoRR, abs/1901.07261, 2019.
  * [8] Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li.  Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search.  CoRR, abs/1907.01845, 2019.
  * [9] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian.  Image denoising by sparse 3-d transform-domain collaborative filtering.  IEEE transactions on image processing : a publication of the IEEE Signal Processing Society, 16:2080–95, 09 2007.
  * [10] Soham De and Samuel L Smith.  Batch normalization biases deep residual networks towards shallow paths.  arXiv preprint arXiv:2002.10444, 2020.
  * [11] Abdelrahman Abdelhamed et al.  Ntire 2020 challenge on real image denoising: Dataset, methods and results.  In CVPR Workshops, 2020.
  * [12] Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan.  Deep ensembles: A loss landscape perspective.  arXiv preprint arXiv:1912.02757, 2019.
  * [13] Shuhang Gu, Yawei Li, Luc Van Gool, and Radu Timofte.  Self-guided network for fast image denoising.  In The IEEE International Conference on Computer Vision (ICCV), October 2019.
  * [14] Yong Guo, Yongsheng Luo, Zhenhao He, Jin Huang, and Jian Chen.  Hierarchical neural architecture search for single image super-resolution.  arXiv preprint arXiv:2003.04619, 2020.
  * [15] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun.  Single path one-shot neural architecture search with uniform sampling.  2019\.
  * [16] Kary Ho, Andrew Gilbert, Hailin Jin, and John Collomosse.  Neural architecture search for deep image prior, 2020.
  * [17] Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  arXiv e-prints, 2017.  https://arxiv.org/abs/1611.01144.
  * [18] Diederik P. Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization.  arXiv e-prints, Dec. 2014.  https://arxiv.org/pdf/1412.6980.pdf.
  * [19] Yaoman Li and Irwin King.  Architecture search for image inpainting.  In ISNN, 2019.
  * [20] Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan Yuille, and Li Fei-Fei.  Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation.  2019\.
  * [21] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy.  Progressive neural architecture search.  2017\.
  * [22] Hanxiao Liu, Karen Simonyan, and Yiming Yang.  DARTS: Differentiable architecture search.  In International Conference on Learning Representations, 2019.  https://openreview.net/forum?id=S1eYHoC5FX.
  * [23] Peng Liu, Mohammad D [El Basha], Yangjunyi Li, Yao Xiao, Pina C. Sanelli, and Ruogu Fang.  Deep evolutionary networks with expedited genetic algorithms for medical image denoising.  Medical Image Analysis, 54:306 – 315, 2019.
  * [24] Pengju Liu, Hongzhi Zhang, Kai Zhang, Liang Lin, and Wangmeng Zuo.  Multi-level wavelet-cnn for image restoration.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 773–782, 2018.
  * [25] Xiaojiao Mao, Chunhua Shen, and Yu-Bin Yang.  Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections.  In Advances in neural information processing systems, pages 2802–2810, 2016.
  * [26] Bumjun Park, Songhyun Yu, and Jechang Jeong.  Densely connected hierarchical network for image denoising.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.  http://openaccess.thecvf.com/content_CVPRW_2019/papers/NTIRE/Park_Densely_Connected_Hierarchical_Network_for_Image_Denoising_CVPRW_2019_paper.pdf.
  * [27] Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean.  Efficient neural architecture search via parameter sharing.  CoRR, abs/1802.03268, 2018.
  * [28] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le.  Regularized evolution for image classifier architecture search.  2018\.
  * [29] Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin.  Large-scale evolution of image classifiers.  In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 2902–2911. JMLR.org, 2017.  https://dl.acm.org/doi/10.5555/3305890.3305981.
  * [30] Uwe Schmidt and Stefan Roth.  Shrinkage fields for effective image restoration.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.
  * [31] Dehua Song, Chang Xu, Xu Jia, Yiyi Chen, Chunjing Xu, and Yunhe Wang.  Efficient residual dense block search for image super-resolution, 2019\.
  * [32] Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu.  Single-path NAS: designing hardware-efficient convnets in less than 4 hours.  2019\.  http://arxiv.org/abs/1904.02877.
  * [33] Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu.  Memnet: A persistent memory network for image restoration.  In Proceedings of the IEEE international conference on computer vision, pages 4539–4547, 2017.
  * [34] Gerard Jacques van Wyk and Anna Sergeevna Bosman.  Evolutionary neural architecture search for image restoration.  CoRR, abs/1812.05866, 2018.
  * [35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  arXiv e-prints, 2017.  https://arxiv.org/abs/1706.03762.
  * [36] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer.  Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search.  arXiv e-prints, 2018.  http://arxiv.org/abs/1812.03443.
  * [37] Haokui Zhang, Ying Li, Hao Chen, and Chunhua Shen.  Memory-efficient hierarchical neural architecture search for image denoising, 2019.
  * [38] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang.  Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising.  IEEE Transactions on Image Processing, 26(7):3142–3155, 2017.
  * [39] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu.  Image super-resolution using very deep residual channel attention networks.  07 2018.  https://arxiv.org/abs/1807.02758.
  * [40] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu.  Residual dense network for image super-resolution.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2472–2481, 2018.  http://openaccess.thecvf.com/content_cvpr_2018/papers/Zhang_Residual_Dense_Network_CVPR_2018_paper.pdf.
  * [41] Barret Zoph and Quoc V. Le.  Neural architecture search with reinforcement learning.  2017\.  https://arxiv.org/abs/1611.01578.
  * [42] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le.  Learning transferable architectures for scalable image recognition.  2017\.