My main concern is that it is not completely clear to me how the authors suggest using the dataset for developing AI that is more ethical. I can clearly understand that one can use it to train an auxiliary model that will test/verify/give value for RL etc. I can also see that using it to fine tune language models and test them as done in the paper, can give an idea of how the language representation is aligned with or represents well ethical concepts. But it seems that the authors are trying to claim something broader when they say ““By defining and benchmarking a model’s understanding of basic concepts in ETHICS…” and “To do well on the ETHICS dataset, models must know about the morally relevant factors emphasized by each of these ethical systems”. It sounds as if they claim that given a model one can benchmark it on the dataset. If that is the case, they should explain how (for example say I develop a model that filters CVs and I want to see if it is fair, how can I use the dataset to test that model?). If not, I would suggest being clearer about the way the dataset can be used.
In addition, I personally do not like using language such as “With the ETHICS dataset, we find that current language models have a promising but incomplete understanding of basic ethical knowledge.” Or “By defining and benchmarking a model’s understanding of basic concepts in ETHICS, we enable future research necessary for ethical AI”. I think that even if a model can perform well on the ETHICS dataset, it is far from clear that it has understanding of ethical concepts. It is a leap of faith in my mind to conclude from what is essentially learning a classification task to ethical understanding. I would like to see the authors make more precise claims in that respect.
Recommendation: I vote for accepting this paper, at its current state marginally above threshold but provided some clarifications, I find this a clear accept. I think the area of ethical AI is important, releasing a well-constructed dataset is an important step forward and overall this paper should be of interest to the ICLR community.
Questions and minor comments: 1.There are missing details about division to train and test sets, numbers as well as how the division was made (simply random? Any other considerations?). These details should be added. 2. In the Impartiality section there is missing reference to Fig 2 – it is given only later so one does not see the relevant examples.
Post-rebuttal comments: My concerns are resolved. I have changed my vote to acceptance. (7).
1.There are missing details about division to train and test sets, numbers as well as how the division was made (simply random? Any other considerations?). These details should be added.