Moving away from platforms, we can see something similar with, for example, RSs that recommend job candidates. The algorithm succeeds when the top recommended candidates get hired. But this is simply circular. The entire purpose of the system is to recommend 'good' candidates; however, 'good' is simply a measure of whether or not the candidate gets hired. The company wants to hire good candidates – but whomever they hire will be considered 'good'. The reader here may ask how the algorithm determines the top ranked candidates. This may be based on past hiring decisions by the employer (see problems with this in the preceding section) or even on video analysis of candidates asking questions – which has raised ethical

issues surrounding cultural, racial, and gender differences biasing the results. In the next sect. I get more into the opacity of the considerations that lead to an algorithm's output – and why that is a problem. Here it is important to note that because of this opacity we are forced to determine the success of the algorithm based on the subject's engagement with the top recommendations – rather than an evaluation of the recommendation itself.

With recommender systems in evaluative contexts, there is a danger that simplistic goals like 'engagement' or 'clicks' will drive us away from what is good for us in those contexts. Platforms may claim that the goals of their algorithms have nothing to do with the good – they are neutrally trying to get you to engage more with their platform so that they can sell ads. However, determining which job candidates are rated highest, which convicted criminals are rated the riskiest, which news stories are prominently displayed, and social media post is at the top of your feed is a huge responsibility with normative implications.

## *8.2.4 Black Boxed*

If someone were to tell you to that you should quit your job, they are making an evaluative statement. Something to the effect of "it would be good for you to quit your job." I imagine that if you had yet to think about the prospect of quitting your job, then you would not just follow this person's advice without further inquiry. Instead, it would be appropriate to ask why they think that. What are their *reasons* for thinking that you should quit your job. Without those reasons the judgment is worth little. For it is the reasons that need scrutiny. For example, their reasoning might be that academia doesn't pay enough and you could make much more in the private sector. This reason may or may not be a good reason *for you*.

With AI powered recommender systems, we have judgements without the reasons. They are black boxed because, as it stands, it is not possible to understand the reasoning for an output of machine learning systems. While this is often portrayed as ML's biggest problem, it is also the source of its power. ML is not restricted to reasons that are articulable to humans – they have access to patterns and considerations that would be impossible for us to comprehend (Robbins 2019, 2020). If we restricted machines to human articulable reasons, like we did with expert systems or good old-fashioned AI (GOFAI), then we would not have had the breakthroughs in AI we have had today.

However, a decision about whether to quit your job seems to require human articulable reasons. In fact, it is diffcult to fnd ethical or aesthetic decisions that do not require justifying reasons (which are human articulable reasons that justify a particular decision). This presents a problem for ML powered recommender systems used for such outputs. For using these recommender systems implies that reasons are not important for the outputs. To take another example, ML systems used for recommending people to be hired for positions implies that the reasons for hiring a particular person are not important. It does this because whatever internal

logic that the recommender system is using is opaque to us. There may be reasons (human inarticulable) that could justify the output. However, without access to these reasons we cannot question their ability to justify the output. When we take such recommendations without the ability to question the reasons which lead to that recommendation, we are implicitly disregarding the importance of those reasons. This is a problem because the reasons justifying why you hire one person over another are of the utmost importance. Amazon's ill-fated hiring tool mentioned earlier in this chapter highlights this point. A reason for hiring one person rather than another was gender. Knowing that gender is a reason precludes someone that wants to make ethical decisions from accepting the output of the system.

While using gender as a reason to hire one person over another is almost always unethical there are other reasons that are seemingly irrelevant that also have ethical import. If the number of letters in someone's name or the number of lines on someone's CV were to be used to hire one person over another it would also be unethical. These are not good reasons to hire or not to hire someone. A hiring committee could decide to use such considerations because they were going with a random approach – which would not be unethical; however, a machine doing this without our knowledge that these were considerations would cause us to place higher evaluative weight on a candidate based on reasons that are irrelevant to their candidacy. If a human being recommended someone for a job based on the number of letters in their name without telling you that was a consideration – then it would be deceptive at best. Knowing that this situation is possible with algorithms should give us pause. When we do not know what the reasons are for a particular ML recommendation, then we are left without a way to evaluate whether the output is acceptable. In cases like hiring, not having the reasons for why a particular person was hired is unacceptable.

This has led some to argue that what is needed is so-called 'explainable AI' (XAI) (Floridi et al. 2018; Wachter et al. 2017). Many researchers are now working on trying to make ML explainable (see e.g. Adadi and Berrada 2018; Linardatos et al. 2021). While there has been some success, we are a long way from knowing the reasons that justify a particular output – which is what is needed in situations described above.

This all goes to show that ML based recommender systems should probably not be used for evaluative outputs like hiring and anywhere reasons are important. What I want to argue for now is that it is worse than it looks. These systems also have the capacity to infuence our evaluative reasoning. To highlight this let us look at news recommender systems.

What is going on in the world is important. We form opinions about where resources should be allocated (frequent stories about traffc jams may make one conclude that we need to fund a light rail), who to vote for (a candidate may be involved in a scandal reported by the news causing you to vote for someone else) and gives you an overall picture of what is currently going on in the world. We can easily see the ethical import of this in the U.S. right now (July 2022). The hearings are going on in in Washington D.C. regarding the pro-Trump violence in the capital

on January 6, 2020. All but one news network is broadcasting the hearings live. Fox News has decided it is not important enough to broadcast (Peters and Koblin 2022).

## **8.3 Changing Human Values**

The ultimate concern that this chapter is focusing in on is that recommendation algorithms are not simply trying to fgure out what is good for you. They are, intentionally or not, infuencing what you think is good. They are changing your behavior to realize their goals. When I say that a recommender algorithm has a goal, I do not mean that the algorithm is an agent with its own goals. These goals are given to the algorithm. They are 'surrogate agents.' Despite not being conscious or intelligent, they are able to act as surrogate agents and act on behalf of those human agents that gave them their goals (Johnson and Powers 2008).

Information released by former employees at YouTube has shown that the designers of these recommender systems understand that their algorithms change user behavior – and that this is the point. Platforms like YouTube make money by selling advertisements. They have a fnancial incentive to keep you engaged on their platform for as long as possible so that you see a maximum number of ads. This incentive has driven changes in the recommender algorithm. For example, in response to users getting bored of watching recommended videos that were simply very similar to things they had already watched, Google built an RS called ReinForce. This algorithm was designed to "maximize users' engagement over time by predicting which recommendations would expand their tastes" (Roose 2019).

The idea that algorithms could have the power to 'expand our tastes' should be of the utmost concern. Remember – the algorithm is not driving you towards some agreed upon 'good'. The algorithm is simply maximizing user engagement. So, in sum, the algorithm is designed to change what you value so that you spend more time on YouTube. The entire project is premised upon the idea that an algorithm can take some control over what a person values. In other words, recommendations can impact how an individual values. This is an extremely important point. While ReinForce was designed to change values, other recommendation systems may change values without such malicious intent.

The concern is that algorithms could infuence how we come to view what a good X is – in light of its recommendations. For example, if you were on a hiring committee and were given 20 CVs to review and you picked your top 5 which had zero overlap with the 5 that were selected by a hiring algorithm, you could either believe that the algorithm was wrong, or you might adjust the standards you use considering the recommendations (though you may not consciously do this). Though many of us are sure to claim that we would never simply take the algorithm's recommendations as truth that would override our own intuitions and standards, the situation is far from clear.

Humans suffer from various biases which may cast doubt as to whether we, as humans, will be able to prevent recommender algorithms from infuencing our evaluative standards. Automation bias, for example, "occurs when a human decision maker disregards or does not search for contradictory information in light of a computer-generated solution which is accepted as correct" (Cummings 2012). This has resulted in disasters like the 1983 Korean Air fight which was shot down in Soviet airspace after the automated system was incorrectly setup and not monitored by the fight crew (Skitka et al. 1999). We humans, tend to be biased towards machine solutions and do not tend to verify every solution offered by these machines. Piling onto that bias is confrmation bias, which biases us towards looking for confrming data rather than disconfrming data (Cummings 2012).

This gets much more complicated with ML powered recommender systems – as we have very little information to go on to confrm, disconfrm, or in any way check the output of such a system. In evaluative cases, there may be simply no way to check. When YouTube recommends you a video to watch next – there is little by which we can check to see if that is indeed the best video to follow the video that we previously watched. Having an algorithm recommend to us the fve best candidates in a pool of 1000 is near impossible to verify. This is in part because the real evaluation – as discussed earlier – should be made regarding the evaluative standards. Are these the right standards? With ML we will not know what those standards are. I am not trying to say that there is some objective set of standards that we have that are perfect. What I mean to say is that what we need to do is evaluate the standards themselves. Whatever process we use to, for example, hire someone – we must do it in a way that we can critique and question the evaluative standards used.

This does not stop us from being infuenced by those standards. It will be diffcult to understand the effect that these multitude of recommendations will have on us – especially children. The news, music, videos, people, products, etc. that are recommended to us will no doubt inform our understanding of what 'good' is in whatever context. The hypothesis is that this will affect our behavior – which will feedback into the algorithms that make recommendations. This vicious cycle would wrest control over the evaluative from human beings and give it to machines. It is necessary to study how people are infuenced by recommender systems, and how we can mitigate those infuences before we have lost control over the evaluative.

## **8.4 Same Problem with Humans?**

Here, there may rise an objection due to the idea that our access to humans' reasons for recommendations are also not accessible to us. This is what Jocelyn Maclure calls the argument from "limitations of the human mind":

Decision-making, either by human beings or machines, lacks transparency. As was abundantly shown by researchers in felds such cognitive science, social psychology, and behavioural economics, real world human agents are much less rational than imagined by either some rationalist philosophers or by rational choice theorists in the social sciences. (Maclure 2021)

Top AI researchers, including Geoffery Hinton have made a point like this:

When you hire somebody, the decision is based on all sorts of things you can quantify, and then all sorts of gut feelings. People have no idea how they do that. If you ask them to explain their decision, you are forcing them to make up a story. Neural nets have a similar problem (Simonite 2018)

It is an intuitive point – especially in light of some of the research that has been done by Daniel Kahneman (2013; Kahneman et al. 2021) and Jonathan Haidt (2001). They have made the empirical case that humans do not simply act in light of the reasons that they claim to have used. Rather, it is the other way around. They make up the reasons for their actions after the fact. Their actions were infuenced by several situational factors that were outside of their control and unknown to them. This is supported by a number of studies that, for example, show that judges give harsher sentences before lunch than afterwards (Danziger et al. 2011). The implication is that how hungry a judge is affects sentencing decisions – though no judge has used that as a reason to hand down harsh sentences. So, though it feels like we have good reasons for human decisions, it is simply a myth. Therefore, there is no reason to not use machines simply because the process for reaching outputs is opaque.

However, this misses some crucial points. First, if expensive machines are simply re-creating the problems that we have with humans – then there seems to be no reason to use the machines. The burden is on those pushing for the adoption of these systems to show a good reason to use them. This is especially true considering the wealth of ethical issues associated with these systems. Most importantly are the environmental costs of these systems (Crawford 2021; van Wynsberghe 2021; Robbins and van Wynsberghe 2022).

More to the heart of the matter, Maclure makes the point that institutions and processes are designed to account for human biases and defciencies: "in non-ideal normative theory, none of these institutions are seen as perfectly capable of neutralizing human foibles, but they can be criticized and continuously improved" (Maclure 2021). The fact that, for example, judges hand down harsher sentences when they are hungry can be mitigated with better scheduling. Institutions and people can be criticized for their failures and can change due to public scrutiny. Also, individuals – especially professionals like judges – must take moral and legal responsibility for their decisions. This is something that RSs – and machines in general – cannot accept. So, an argument would have to be made that decisions like these can be taken by agents that cannot accept moral or legal responsibility before we delegate such decisions to them.

## **8.5 Conclusion**

Recommender systems are increasingly being used for many purposes. With this chapter I have shown that this is creating a deeply problematic situation. First, recommender systems are likely to be wrong when used for these purposes because there are distorting forces working against them. RS's are based on past evaluative standards which will often not align with current evaluative standards. RS's algorithms must reduce everything to computable information – which will often, in these cases, be incorrect and will leave out information that we normally consider to be important for such evaluations. The algorithms powering these RSs also must use proxies for the evaluative 'good'. These proxies are not equal to the 'good' and therefore will often go off track. Finally, these algorithms are opaque. We do not have access to the considerations that lead to a particular recommendation. I have argued that it is precisely these considerations that are used to evaluate whether or not a particular recommendation is good. Without these considerations we are taking the machine's output on faith.

Second, I have shown that these algorithms can modify how we evaluate. YouTube has modifed its algorithm explicitly to 'expand our tastes'. This is an extraordinary amount of power – and one that if my frst argument goes through, shows that these algorithms will be expanding our tastes in a manner that is likely to take us away from the good. This infuences our behavior which feeds back into the algorithms that make recommendations. It is important that we establish some meaningful human control over this process before we lose control over the evaluative.

Finally, I have anticipated that readers may say that the way we receive recommendations without RSs is also problematic. There is no way to verify that someone's recommendation of a job candidate, movie, prison sentence, etc. is 'good'. To this I have replied in two ways. First, that if all things were equal, and we can't verify either, why would we use a machine which has environmental and economic costs over a human being? Second, that things are not equal – machines cannot accept the moral responsibility required to make evaluative choices that cannot be verifed by human beings. Giving up this control is giving up control over the evaluative – something that requires good reasons which have yet to be offered.

## **References**

van Wynsberghe, Aimee. 2021. Sustainable AI: AI for Sustainability and the Sustainability of AI. *AI and Ethics* 1 (3): 213–218. https://doi.org/10.1007/s43681-021-00043-6.

Wachter, Sandra, Brent Mittelstadt, and Luciano Floridi. 2017. Transparent, Explainable, and Accountable AI for Robotics. *Science Robotics* 2 (6): eaan6080. https://doi.org/10.1126/scirobotics.aan6080.

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Part III Designing and Evaluating Recommender Systems**

# **Chapter 9 Ethical and Legal Analysis of Machine Learning Based Systems: A Scenario Analysis of a Food Recommender System**

**Olga Levina and Saskia Mattern**

**Abstract** Laws are the refection of the ethical and moral principles of the society. While the use of technology infuences users' behavior in a pace that is affected by the technology introduction to the market, legal activities can be driven by the society as the results of such interactions. This scenario analysis- based research focuses on a classic but fctional food recommender system and the ethical issues that might occur from its usage. The recommender system is taken here as an example of machine learning-based systems (MLS) that can often be found in the individual, business and administrative applications. The research compares the existing legal solutions, with the focus on the GDPR legislation, and the discovered ethical issues. The ethical analysis is led along the ALTAI principles suggested by the European Commission, the common good approach as well as the general principles constituted in human rights. While the GDPR-based analysis showed that this data- and privacy-based legislation addressed most of the identifed ethical issues, questions related to the common good approach in the context of environment and mobility that arise due to the wide spectrum of the MLS usage require further legal discussion. The application of the two approaches shows that conducting the ethical and legal analysis is benefcial for both the designers of such MLS as well as the legal actors. The fndings can enhance the design and functions of a user-facing MLS as well as infuence or validate legal activities.

**Keywords** Ethical analysis · Legal analysis · GDPR · IT impact · Affected user

O. Levina

S. Mattern (\*) FZI Forschungszentrum Informatik, Berlin, Germany e-mail<EMAIL_ADDRESS>
© The Author(s) 2023

Brandenburg University of Applied Sciences, Brandenburg, USA e-mail<EMAIL_ADDRESS>
S. Genovesi et al. (eds.), *Recommender Systems: Legal and Ethical Issues*, The International Library of Ethics, Law and Technology 40, https://doi.org/10.1007/978-3-031-34804-4\_9

## **9.1 Introduction**

The presence and application of digital services has become an integral part of the personal daily routine as well as within business processes. In the last two decades an increasing number of companies have adopted a business model based on digital services attained from user data. Accompanying these developments are not only the changes in the business processes but also societal attention to the effects and the constitution of these services.

Moral and ethical demands for the design and application of digital technologies, especially in the machine learning context, are steadily rising and are resulting in regulatory activities in some countries. In this paper, we use a scenario case study of a recommender system to identify potential ethical issues within the composition and usage of this digital artifact, i.e., software or an information system, as well as the correspondent legislation in force. We explain what ethical aspects are addressed by legal requirements. We show that without social awareness and responsibility, legal regulations alone cannot guarantee socially compliant information technology (IT). Here, we use the scenario of a food recommender system, the FoodApp, that demonstrates the multi-dimensional effects that such a data-based information system can have. The widespread use of smartphones for purchasing goods and services has led to a debate in research and public about the effects of the recommender systems on users' behavior and thus the potential for ethical considerations. The food recommender systems encapsulated in a digital application delude the relation between the consumer and the market by providing a physical good (food) in a short time after a virtual interaction (app-based selection and purchase). The effects of this interaction on the environment, the business and on the stakeholders involved is largely invisible for the consumer making this scenario helpful for the analysis of ethical issues. Since legal measures incorporate ethical values and norms of a society, this paper uses ethical analysis to identify issues that might occur when using a food recommendation system. The results of the legal analysis of this scenario are then used to identify the ethical issues that lack legal equivalents. These results can be used for public discussion about the scope and necessity of future legal regulations.

Hence, the contribution of the research is twofold. First, the analysis method using a combination of ethical and legal assessment is described as a part of the development process of an IT system. Second, the application of the combined ethical and legal analysis of a Machine Learning-based System (MLS) is presented with the focus on recommender systems and its implications are outlined. For the legal analysis, legislation of the European Union (EU), especially the General Data Protection Regulation (GDPR), is considered. The ethical aspects are analyzed using the data-oriented ethical analysis (Levina 2020) and relying on the ALTAI requirements as suggested by the European Commission (European Commission 2020) and subsequently restated as the suggestion for legislation in April 2021 (European Commission 2021). The ethical analysis looks at the data process within the system's design and identifes some of the relevant points in the development

process, where ethical questions arise and infuence the design of the system. The legal analysis reveals where the emerging ethical questions are already legally regulated. While the presented results cannot be considered exhaustive as to the incorporation of ethical and legal issues during MLS design and use, it provides an exploration framework for designers, developers and users of such systems as well as for Legal and Information Systems Researchers (ISR) by offering an addition to the ISR research methods.

The paper is structured as follows: A brief overview of the state-of- the-art research on ethical issues related to recommender systems follows the scenario introduction of the FoodApp in Sect. 9.3. The ethical analysis and the legal analysis are conducted in Sects. 9.4 and 9.5 respectively, before the combination of their results is presented and discussed in Sect. 9.6. The paper concludes with an outlook on research perspectives.

## **9.2 An Example Application: FoodApp- the Application for Meal Delivery**

For demonstration purposes of the application of the ethical and legal analysis, a scenario of a fctional food recommendation application, the FoodApp, is used here. The fctional app description is adopted from Levina (2022).

The FoodApp is a fctional application based on a three-sided digital platform that is implemented as a mobile app. It is a branch of the fctional large company Acima that offers on demand individual transportation provided by freelancing drivers. To further explore the transportation market, Acima started the FoodApp, a fast-growing food delivery platform connecting the customer, restaurant owner and the delivery partner. It allows the customer to choose from a large database of participating restaurants and order a menu to be delivered to the customer's address via delivery partners. The eater can choose a specifc delivery partner based on the ratings of the currently available partners. The payment process is integrated into the platform as is the real-time tracing of the order delivery.

The platform business goal is "fast and easy food delivery whenever, wherever". To achieve this goal, a MLS, a recommender system, is used to provide the best food suggestions for the user in accordance to the indicated preferences and the order history. The business performance indicators for the FoodApp include the return and re-order customer rates, as well as customer number growth rates. The implemented ML-model is thus optimized to drive user's re-ordering on the platform.

To use the FoodApp, the customer downloads it on the mobile device granting permissions for it to access the location of the device. Further, a profle including information on delivery address, name, e-mail and phone number is required. Payment methods and logging in to the payment provider is further required. No manual modifcations concerning data collection by the app are possible. Then, the meal preferences such as preferred cuisine or menu item need to be indicated or a meal can be chosen from the provided suggestions. The frst suggestions are based on the historical frequency of the orders made within the community in the area of eater's location. A rating system for restaurant and delivery partner performance is implemented.

The platform gains revenue from the customer via convenience charge, fxed commissions and marketing feeds from the restaurants, while providing the assignments and the payment to the delivery partners, as well as the technical infrastructure for the platform participants. The application is a key driver of Acima's revenue and is a fast- growing meal delivery service with over 15 million users worldwide. Additionally, the platform includes an app for delivery partners that provides the possibility to accept or decline a specifc delivery job, monitor the revenues, rate the restaurant's delivery process, as well as provide directions to the restaurant and to the eater.

## **9.3 Current Approaches to Ethical Analysis of Recommender Systems**

Ethical issues in the context of IT-artifacts have gained increasing attention in research over the last decade. Paraschakis explores e-commerce recommender applications and identifes fve ethically problematic areas: user profling, data publishing, algorithms design, user interface design and online experimentations, i.e. exposing selected groups of users to specifc features before making them available for everybody (Paraschakis 2016, 2017).

Milano, Taddeo, and Floridi conduct an exhaustive literature review of the research on recommender systems and their ethical aspects and identify six areas of ethical concern: ethical content, i.e. content that is or can be fltered according to societal norms; privacy as one of the primary challenges of a recommender system; autonomy and personal identity, opacity, i.e. lack of explaining how the recommendations are generated; fairness, i.e. the ability to not refect social biases; polarization and social manipulability by insulating users from different viewpoints or specifcally promoting one-sided content (Milano et al. 2019).

Milano, Taddeo and Floridi also show that the recommender systems are designed with the user in mind, neglecting the interests of the variety of other stakeholders, i.e., interest groups that are being directly or indirectly affected by the recommendation (Milano et al. 2019). Polonioli presents an analysis of the most pressing ethical challenges posed by recommender systems in the context of scientifc research (Polonioli 2020). He identifes the potential of these systems to isolate and insulate scholars in information bubbles. Also, popularity biases are identifed as an ethical challenge potentially leading to a winner-takes-all scenario and reinforcing discrepancies in recognition.

Karpati, Najjar and Ambrosio analyze food recommendation systems and identify several ethically questionable practices (Karpati et al. 2020). They name the commitment to already given preferences and thus to the values of the designers as a contradiction to the potential for ethical content. Privacy, autonomy and personal identity that the authors identify as potentially vulnerable and hence suggest need to be realized via an informed concern and a disclosure about the business model used. Opacity about the origin of the recommendations as well as of the criteria and algorithms used to generate the recommendations. Fairness, polarization and social manipulability as well as robustness of the system complete the list of identifed ethical issues for a food recommender.

These approaches discuss ethical impacts of recommender systems from the perspective of the receivers of the recommendations. Milano, Taddeo and Floridi argue that the social effects such as manipulability and personal autonomy of the user are hard to address, as their defnitions are qualitative and require the implementation of the recommender system in the context they operate (Milano et al. 2019), while Karpati, Najjar and Ambrosio offer a multi-stakeholder approach to address these issues (Karpati et al. 2020).

The data processing-centered approach to analyzing ethical issues suggested by Levina (2020) identifes the decision points during the MLS development, while advocating the inclusion of a laboratory phase into the system design to assess the potential consequences (see also Coravos et al. 2019). This research applies a data process-centered combination of ethical and legal analysis in the attempt to identify how or whether the identifed threads to ethical values that can be realized via an MLS are being already covered by the legislation of the EU and where in the artifact design these can be addressed.

## **9.4 Ethical Analysis**

Here the ethical analysis following the steps of data processing in MLS is applied. To understand ethical implication of a process or technology, specifc values that are being affected by the technology deployed need to be taken into account. There are a number of values that can be considered, but here, we look at the values represented as requirements in the "Assessment List for Trustworthy AI (ALTAI)" (European Commission 2020) by the High-Level Expert Group on Artifcial Intelligence of the European Commission and that also are the fundament for the regulation proposal for governing artifcial intelligence technologies (European Commission 2021) being: Human agency and oversight; technical robustness and safety; privacy and data governance; transparency; diversity, non-discrimination and fairness; societal well-being; accountability. Those have been chosen specifcally to assess ethics aspects of AI technology, and are therefore appropriate to our goal.

The data processing-centered ethical analysis puts privacy and data governance in its focus and thus allows a stringent analysis of the legal aspects along the General Data Protection Regulation (GDPR), EU legislation. The specifc stages of the data processing considered here are: *sense*, including: collect data, detect data, and select (problem-related) data; *transform*, including: clean data, chose machine learning model, train model, integrate model into the software system; *act*, including generate recommendations or more general, a result based on the chosen ML-model; and the *apply* stage that considers effects on stakeholders by the results and the effects introduced by the ML-model defnition and application. These data processing stages are based on the data management process by Shutt and O'Neill (2013). While the apply stage is not an integral part of the data process, the effects of the application of the MLS on user behavior are important for its design. Furthermore, it is differentiated here between the MLS-user, i.e., a person using the MLS directly, and an MLS-*affected* user, i.e., a person or a stakeholder that is affected by the effects of the MLS.

In the *sense* phase of data processing, it needs to be assured that the data have been collected with the *informed consent and voluntariness* of the data subject. The legal analysis presented in Sect. 9.5 elaborates further on these considerations. Informed consent also includes the statement of the *purpose* of the data collection implying an opt-in function for data collection. What data are being collected is normally described in the *terms and conditions document of the* recommender system. Their legal suffciency is further discussed in the legal analysis. Often, under the claim of providing more personalized suggestions, profle data provided by the user as well as e.g., the location data automatically provided by the mobile device are also collected. Nevertheless, it is not clear to the user, *when* the location *data are collected* challenging the value of *privacy*, nor is it clear what role these data paly for recommendation creation, addressing the value of transparency. While *opt-out functions* are often cited by the enterprises producing data-based services as user empowerment (Abdelaziz et al. 2019), they are legally insuffcient (see legal analysis) and shift too much responsibility towards the user (Milano et al. 2019).

FoodApp's business goal is to engage the user in the re-ordering of food via its digital platform. The user interacts with the app aiming for a comfortable provision of the favorite food in an effcient way. Therefore, the user is inclined to give up some autonomy within this process. Nevertheless, in the digital realm the user is often not aware of what elements of his/her *autonomy* are jeopardized when the digital service, here food selection and ordering via a digital platform, is performed. While the user can still change the app's settings, s/he is often unaware of the default access requirements. Data are the fundament for the further model building for the recommendation algorithm and their amount and sources are strongly defned by the business model that this MLS is supposed to support.

As FoodApp would like their users to return to the app, it will need among other factors, very good recommendation results as well as a frictionless ordering process together with a reliable problem handling mechanisms to fulfll basic customer expectations (Karpati et al. 2020). The frst requirement, i.e., very good recommendation results in terms of user's preferences, can be realized using a recommendation algorithm based on the collected data from the user as well as from the users with similar preferences or history on the platform (Karpati et al. 2020). Since the user activity data might provide additional patterns for the recommendation, it also provides a potential reason to keep the user engaged on the app for the longest possible time, which might involve the use of dark patterns in the app design (Gray et al. 2018). FoodApp's user profle provides the information that is, among others, needed for the algorithms in the MLS to derive food recommendations. The user does not have any information about the exact *purpose* of the provided datasets, the *data lifecycle*, nor about who has *access* to the (possibly) un-anonymized profle or historical data and about the *data state timeline*, i.e., when the data are transferred or deleted. These aspects can be categorized as "*transparency* issues", since the user does not have the information about FoodApp's processes s/he might need or would like to have.

Aside from the user data, FoodApp's database would include data on the restaurants available for ordering and delivery through the platform. Addressing the restaurants is part of Acima's business model and might also be part of the business focus of the restaurants, as they can be included on the platform according to specifc *criteria*, e.g., reviews on other platforms, personal preferences, number of years in business, etc. leading to a potential pre-selection of available food choice on the platform. Additionally, the delivery network of partners that will pick up food at the restaurants and deliver it to the customer's door need to be established and equipped with the means to be contacted, paid and managed by the platform. Hence, FoodApp needs to establish an *ecosystem*, similar to a classic supply chain, to be able to fulfll its business goal or even to be able to operate according to its business model. Building up such an ecosystem as well as the potential to manage the orders for delivery, provides Acima as a digital platform with a specifc power over the delivery partners as well as the restaurants that can have extensional effects on the partners involved in the ecosystem as well as the bigger area of stakeholders (Levina 2019).

The user can flter the suggestions within the FoodApp using the provided *flter categories*. These categories, defned by the MLS-engineers and designers, include cuisine and menu item names, as well as the ratings of the accordant restaurants. In future interactions with the FoodApp its home screen offers the meals and food items that are most frequently ordered by the eater or users that were identifed to have a similar ordering behavior, thus nudging the eater to order the same or similar kind of food (Zhou et al. 2010).

Also, how the data are analyzed, i.e., anonymously or connected to the profle might raise further *privacy* and *transparency* questions. In cases when the recommender system dictates what data are collected, the mode of data collection and their importance within the algorithm are also decided without user participation. Hence, the question about the *diversity* and *quality* of data and consequently the ones of the recommendations arise. The lack of diversity can be manifested in other aspects that are indirectly related to the business goal, such as feedback from the stakeholders, the effects on the ecosystem the MLS is acting in as well as the potential impact of its recommendations on the environment. In a case of a food recommender as in (Karpati et al. 2020), these can be a considerable environmental impact

in terms of air pollution (Saner 2020) besides the generated plastic waste (Zhong and Zhang 2019).

The *transformation* phase is the last phase within the data process where the fundamental decision to apply machine learning instead of a less resource-intensive solution to realize a business goal can be revised. Training and using an MLS requires increased power usage. Parameters within the MLS that lead to recommendations are designed and confgured by data analysts and software engineers, thus refecting their values and reducing the *diversity* of the recommendations as well as promoting decisional de-skilling (Floridi 2016) and increasing *trust* into the algorithmic suggestions (Gille et al. 2020; Krügel et al. 2021; Fritz et al. 2020) by the user.

To reduce energy and time *resources* to train the particular model in use, an existing pre-trained model form the application domain can be used. Although, the choice of the model often does not consider its *explainability* (Kamiran and Calders 2012) or its requirements for the resources. Its accuracy is often the main criterion, which might neglect user's preferences. Also, observations on the effects of MLS application on user's *behavior* rarely precede the going live of the MLS.

The FoodApp has based its business model on the data-based provision of food recommendations and the forwarding of the recommendations to the restaurants and delivery partners. Thus, being data-based, these business questions would require the use of data analysis tools, although the added value of the neuronal networks for the recommendations depends on the quality of data and the accuracy thresholds defned by the product designers.

The *model quality* is in the center of the ethical inquiry in the transformation phase. The set thresholds defne mathematical methods, e.g., neural networks vs., e.g., support vector machines, and thus the resources needed to train the model as well as to generate the recommendation. The transform phase does not only include the training and optimization of the models used for the recommendation, but it also considers the inclusion of the ML-models into the information systems context.

While defnition of food categories as well as the selection of the included cuisines and restaurants is part of the *sense* phase and especially the *select* sub-phase, questions in the transform phase focus on the mathematical transformation of these selected details. Inclusion of, e.g., nudging techniques is also part of the sense phase and the collect sub-phase, but it is strongly defned by the *business model*.

The *act* phase comprises the defnite initiating of an action based on the provided MLS recommendation. The interaction with the recommender is often implemented as a "human-in-the-loop" interaction pattern that puts the human in *control* only over the recommendation-based action itself. It often leaves the user oblivious about the following process that was initiated by this action, leaving the user oblivious about the consequences that were induced by the action, as no feedback from outside the user-focused process is taken into account.

For Acima, the value is created when the food delivery order is completed in the FoodApp. Hence, the ordering process is organized in a way that no extended explications or additional information are given so that the user does not have to choose, decide or react during the interaction process. This design allows a *fast phase-out*

between opening the FoodApp and ordering the food. This effect can be expected to contribute to user satisfaction and thus re-visiting the platform for the next order.

The process effciency offered by the FoodApp is also built on the lack of decision possibilities and a limited items selection that is based on the historic and profle preferences for the user. Additionally, the gained comfort for the user in terms of food selection and delivery has implications on the *ecosystem* of the FoodApp. The restaurant partners will be faced with an increased amount of reviews from delivery customers, potentially forcing them to concentrate on robust packaging to ensure the sound condition of the meal for delivery. More robust packaging means more damage to the environment but potentially better ratings from the FoodApp users (Zhong and Zhang 2019).

Furthermore, the food recommendations based on historic and similar orders might lead to homogenization of the food offered in the participating restaurants, as menu items that are ordered less often might not be prepared by the restaurants anymore, potentially leading to the decrease or shift of skills of the cooking staff. The individual delivery of the food orders requires reliable and effcient delivery partners. Acima relies here on its network of drivers for personal transportation that are also incentivized to transport food orders via reward programs. This effcient and effortless process of ordering food for individual consumption can and does cause signifcant environmental damage in terms of air pollution through traffc and waste (Zhong and Zhang 2019).

Further effects on the *social* environment can also occur. The eater rates the restaurant on the food quality and the delivery partner on the quality of the delivery. The rating is based on eaters' satisfaction with the end result, whereas the traffc situation and other external effects of the recommendation process are not considered. This relationship pattern causes societal effects that are visible in the traffc situation, environmental damages as well as reduction of labor costs and conditions and also affect user's behavior (De-Arteaga et al. 2020).

While the analyzed aspects in previous phases were based on the data process as an enabler of the business value creation, the *apply* phase leaves the realm of the software system as an artifact and enters into the realm of its usage that can directly or indirectly infuence behavior of individuals. One potential effect might be the reduced diversity in the choices that survive in such ecosystems due to the focus of the recommender system to provide suggestions based on e.g. item's popularity, leading to the extinction of less favored choices as well as impacts on the collateral effects, e.g. effects on air quality, quantity of waste and quality of life via increased delivery traffc are commonly observed in the cities where food recommender are active (Karpati et al. 2020; Saner 2020; Zhong and Zhang 2019). Thus, even if the MLS is not directly critical to the human life, rights or *well-being*, there is a necessity to consider the societal effects of its implementation as outlined in the criticality assessment of algorithmic systems by the German Ethics Commission (Germany. Datenethikkommission 2018) that takes the effects of the application of an MLS on critical goods such as human life and wellbeing into consideration.

In the FoodApp scenario, rating of the delivery partners results in an increasing number of orders for high ranked drivers and in a reduction of delivery orders for the worse ranked drivers. Hence promoting the reviews into the main factor for job acquisition, and thus income, for the drivers. This type of job market is known as the *gig economy* (Friedman 2014). It provides income potential for the workers while creating an interdependency between the platform customer and the gig worker. This relation seems to remain unclear for the platform customer and is often debated by the platform owner (Susser and Grimaldi 2021). Consequently, the OECD stated in 2016 that digital platforms need social values to be refected in the platform governance (OECD 2016).

## **9.5 Legal Considerations**

Using MLS can cause novel challenges for the legal compliance. Ideally, there is a common goal between legal experts, ethicists and technicians: an optimized and legally compliant design and implementation of machine learning-based artifacts. To further this, some main aspects are presented using the food recommender system as an exemplary scenario.

## *9.5.1 Data Protection Law*

The availability of large amounts of data are essential for machine learning. But when *personal data are* involved, the *European General Data Protection Regulation (GDPR)* comes into play. There are actually very few data, such as pure machine data, which no longer have any personal reference. Personal data refers to all information relating to an identifed or identifable natural person.1 This broad defnition covers all information that can somehow be attributed to a specifc person. Even *pseudonymous data,* like an IP address, is classifed as personal data, because, although not directly, it can be attributed to a natural person by means of a transfer.2

In the FoodApp, user, restaurant and delivery data are collected, selected, processed and stored. All this constitutes *data processing* in the sense of data protection law. It is clear – not only because of the tremendous possible *penalties and fnes*<sup>3</sup> imposed by the GDPR that it is absolutely necessary to plan MLS such as FoodApp in compliance with data protection regulations. Albeit, the issues of big data, Machine Learning (ML) and algorithms were not explicitly addressed by the GDPR. Here very few references can be found, e.g., Art. 22 GDPR, which regulates automated individual decision-making, including profling. Even though Art. 22 GDPR appears to be extremely relevant to ML- processes at frst sight, its legal

<sup>1</sup>Defnition in Art. 4 No. 1 GDPR.

<sup>2</sup>Recital 50, Kühling/Buchner/Klar, GDPR Comment, Art. 4 No. 1 marginal no. 28.

<sup>3</sup>See Art. 83, 84 GDPR.

scope is actually rather limited. According to the prevailing and convincing view,4 Art. 22 GDPR applies only to those automated processes, which are intended to evaluate certain characteristics and features of a person, despite its rather open wording.5 Hence, the data process itself is subject to the general provisions of the GDPR, which were not specifcally designed for ML-processes. Subsequently, some points of relevance are presented.

## *9.5.2 General Principles and Lawfulness of Processing Personal Data*

First, Art. 5 GDPR is to be mentioned, which lays down the general principles of the data processing such as lawfulness, fairness and transparency. Art. 5 No. 1 a GDPR reads: "Personal data shall be processed lawfully, fairly and in a transparent manner in relation to the data subject". Equally relevant is Art. 6 GDPR, which regulates the lawfulness of processing. Art. 5 No. 1 a GDPR reads: "Personal data shall be processed lawfully, fairly and in a transparent manner in relation to the data subject". The main scope of the transparency requirements is defned by the information obligations in Art. 12 and Art. 13 GDPR, for the exercise of the rights6 of the affected *data subject*. Art. 12 GDPR initially states that all information and communications must be provided in a precise, transparent, comprehensible and easily accessible form in clear and simple language. When data for the usage in ML are collected, affected persons need to be explicitly informed about this fact. Thus, the FoodApp should provide explicit mentions of the data being collected and the purpose the data is being used for.

However, *transparency* seems to be relatively diffcult to fully accomplish in this context, because this also requires explainability of the decision process in algorithmic decision procedures. It is essential that users of algorithmic systems are at least able to understand, explain and control their operation, and that those affected receive suffcient information to enable them to exercise their rights in Art 12–23 GDPR properly. It appears to be excluded, that the process can be reproduced in all details afterwards.7 So it is to be assumed that only the principle underlying the algorithm, i.e. the basic assumptions of the logic underlying the algorithm, but not the concrete computation formula or the algorithm itself must actually be presented

<sup>4</sup> If you consider Art. 4 No. 4 GDPR and recital 71 of GDPR.

<sup>5</sup>Recital 71 of GDPR; Art. 4 No. 4 GDPR; Abel: Automatisierte Entscheidungen im Einzelfall gem. Art. 22 DS-GVO (Automated decisions in individual cases according to Art. 22 GDP9", ZD 2018, p. 304.

<sup>6</sup>See Art. 12–23 GDPR.

<sup>7</sup>Rosenthal: GDPR vs. AI, LR 2018, 173; Bitkom, 2018: Machine Learning and the Transparency requirements of the GDPR.

to fulfl the information obligations in Art. 12 and 13 GDPR.8 The biggest diffculty of transparent presentation here probably lies in describing the complexity of the analytical procedures used in such a clear and easy way that their scope is somehow tangible for the person affected.9 On the other hand, companies also may claim that the disclosure of the algorithm and data categories would affect their business model by hinting to their business secrets.10 In order to balance the interests of the MLSuser and the MLS- affected user, the provisions of Art. 42 Paragraph 1 GDPR, as well as data protection certifcation procedures and data protection seals and test marks, might represent a suitable instrument for proving compliance with the GDPR by the person responsible or their contract processors.11

## *9.5.3 Lawfulness*

In addition to the transparency and fairness of the data processing, it must also be lawful under Art. 5 and Art. 6 GDPR. Processing personal data are generally prohibited, unless the data subject has consented to the processing or one of the other legal bases stated in Art. 6 GDPR apply. It seems to be tempting to regulate data processing for ML-Systems in general terms of use or general terms and conditions. If customers want to use a certain service or app, which uses ML, he or she simply has to agree to the terms and thus also to the processing of data for ML-purposes.

Nevertheless, an opt-in or even opt-out in terms of use is not equivalent to *informed consent*. The basic requirements for the effectiveness of valid legal informed consent are defned in Art. 4 and Art. 7 GDPR and specifed further in recital 32 of the GDPR. In order to obtain freely given consent, it must be given on a voluntary (free) basis, which implies a real choice by the data subject. Therefore, the tying prohibition12 applies: the fulflment of a contract may not be made dependent on the consent to the processing of further personal data which are not necessary for the fulflment of this contract. Though one can already doubt the voluntary aspect here, there is again a major problem in particular with regard to the transparency of information. For consent to be informed and specifc, the data subject must

<sup>8</sup>Art. 29 Data Protection Working Party (WP 251 rev.01) – Guidelines on the Automated individual decision-making and Profling for the purposes of Regulation 2016/679; Rosenthal: GDPR vs. AI, LR 2018, 173, Ehmann/Selmayr: GDPR Comment, Art. 15 Rn. 12; Paal/Hennemann, GDPR Comment, Art. 13 Rn. 31; Kamlah, in: Plath, BDSG GDPR Comment, Art. 13 No. 28.

<sup>9</sup>Bitkom/DFKI, 2017: KI – Wirtschaftliche Bedeutung, gesellschaftliche Herausforderungen, menschliche Verantwortung (Artifcial intelligence: economic signifcance, social challenges, human responsibility) p. 134; Gausling: Artifcial Intelligence and GDPR, DSRITB 2018, p. 519.

<sup>10</sup> In German legislation it ist he "Gesetz zum Schutz von Geschäftsgeheimnissen (GeschGehG)"that is meant to protect the business secrets in legal sense.

<sup>11</sup>Bitkom/DFKI, 2017: "Artifcial intelligence: economic signifcance, social challenges, human responsibility", p. 134; Gausling: Artifcial Intelligence and GDPR, DSRITB 2018, p. 519.

<sup>12</sup>Recital 43 of GDPR, Gola GDPR Comment/Schulz, Art. 7, No. 21–33.

at least be notifed what kind of data will be processed, how it will be used and the purpose of the processing operations.13 If all this is applicable, an opt-in to general terms of use/terms and conditions could be suffcient. An opt-out, on the other hand, is never suffcient, as it must be an unambiguous action.14

Secondly, it is questionable whether another legal basis would apply unless consent can be assumed. It would be conceivable in particular that the use of ML is necessary within the meaning of Art. 6 No. 1 b GDPR for the performance of a contract to which the data subject is party. That means, the ML algorithm must be used and also be necessary to enable the service provision to a customer, e.g., food recommender. However, ML will probably be used more often to optimize processes, but not to make them possible in the frst place. Another commonly used reason for lawful data processing can be fnd in Article 6 paragraph 1 lit. f: "processing is necessary for the purposes of the legitimate interests pursued by the controller […]." If a company argues that it has a legitimate interest in using ML, because its services would otherwise not be competitive, for example, this might actually be suffcient in an individual case. As the legal assessment depends on the individual case and interpretation, many uncertainties remain.

## *9.5.4 Purpose Limitation and Access to Data*

Art. 5 No. 1 b GDPR requires a *purpose* limitation, which means that data may only be processed for specifed, explicit and legitimate purposes and must not be processed beyond those purposes. Thus, the purpose legitimates the processing of the data in terms of necessity, adequacy, completeness and duration of the processing.15 Once the purpose is fulflled, the collected data must be deleted, since the purpose cannot be modifed. This is only exceptionally possible if the present and initial purposes are compatible (Art. 6 paragraph 4 GDPR). In order to ascertain whether a purpose of further processing is compatible with the initial purpose, the controller should take several factors into account, e.g., the context in which the personal data have been collected, the nature of the personal data and the consequences of the intended further processing for data subjects.16 This is interpreted rather restrictively. In this context, it is important to also point out, that the GDPR does not take a stance on the issue of "data ownership". Therefore, the question arises, who actually "owns" the original and generated data, has access to them and can use or even sell them. According to the prevailing opinion "data ownership" does not exist.17

<sup>13</sup>Recital 32 of GDPR.

<sup>14</sup>Recital 32 of GDPR.

<sup>15</sup>Paal/Pauly/Frenzel DS-GVO Art. 5 Rn. 23–25.

<sup>16</sup>Recital 50 of GDPR.

<sup>17</sup>Among others: Czychowski/Siesmayer, Computer Law Manual, para. 20.5, margin no. 42; Zech, GRUR 2015, 1151, margin no. 1157; Hoeren, MMR 2019, p. 5; Determann, ZD 2018, p. 503; Working Group "Digital Reboot" of the Conference of the Ministers of Justice of Germany, report

Rather, it seems to be the current consensus to realize data sovereignty in each individual case through contractual agreements between the parties involved in a data exchange. This further underlines and demonstrates the absolute importance of purpose limitations as it restricts the power of disposal and the data subject retains his or her sovereignty.

## *9.5.5 Data Minimization and Storage Limitation*

In order to derive accurate results using MLS, considerable amounts of data are required. These data need to be collected, selected and also stored to be usable. Therefore, the data collection in the FoodApp is already *conceptually opposed* to the principles of data minimization and *storage limitation* in Art. 5 No. 1 c, e GDPR as the data is being collected without the naming of the purpose and without its clear attachment to the business model. According to the principles mentioned above, only as much data as absolutely necessary may be collected and stored for as long as absolutely necessary. So, how can a balance be struck between the conficting goals of large data collection as exercised by some data-based information systems and data protection? The principle of *data minimization* obliges data controllers in Art. 25 GDPR to design their systems technically in such a way that the risks for data subjects are minimized (privacy by design) and that default settings ensure that only personal data that are necessary for the purpose are processed (privacy by default). As stated in Art. 25 (2) GDPR privacy by default should ensure that personal data is processed with the highest privacy protection. Hence, personal data is made accessible to a defnite number of persons and only personal data that is necessary for a specifc reason shall be obtained. The principles of data minimization and purpose limitation relate to the concept (Ježová 2020).

As a consequence, the FoodApp would have to provide a data-protection-friendly default setting and offer the user a detailed selection option. In addition, an individual setting by the user, which can be made at any time, must be possible.

Yet here another divergence is evident – manufacturers or software developers de facto have infuence on the data protection conformity of data processing processes and are able to implement "Privacy by design". However, manufacturers or software developers, are actually not responsible in the sense of the GDPR, but only possibly commissioned data processor (Art. 24 GDPR). The Controller, the one that uses MLS, stays responsible. In this context Art. 28 GDPR sates that the controller shall use only processors providing suffcient guarantees to implement appropriate technical and organizational measures in such a manner that processing will meet the

of 15.5.2017 https:// jm.rlp.de/fleadmin/mjv/Jumiko/Fruehjahrskonferenz\_neu/Bericht\_der\_AG\_ Digitaler\_Neustart\_vom\_15.\_Mai\_2017.pdf; German Federal Ministry of Transport and Digital Infrastructure, "Ownership Regulations" for Mobility Data? – A study from a technical, economic and legal perspective", August 2017, https://www.bmvi.de/SharedDocs/DE/Publikationen/DG/ eigentumsordnung-mobilitaetsdaten.html

requirements of the GDPR and ensure the protection of the rights of the data subject. But this actually does not seem to be suffcient as no real obligation is imposed. The manufacturer or software developers should thus be somehow included in the scope of this provision.18

Art. 25 paragraph 1 GDPR describes that *pseudonymization* might help to effectively implement these data protection principles and thus to protect the rights of the data subjects. The exact way in which the legislator envisages this is not explained in any detail, not even in the recitals. Art. 25 GDPR does not provide a solution for the principle of storage limitation, but the legislator has written specifc requirements in the recitals of the GDPR: The personal data should be limited to what is necessary for the purposes for which they are processed. In order to ensure that the personal data are not kept longer than necessary, time limits should be established by the controller for erasure or for a periodic review.19 So, the principle requires erasure routines that take effect at regular intervals and prevent endless storage.

## *9.5.6 Accuracy, Security and Impact Assessment*

Data allow a reconstruction of an individual's characteristics or information and must therefore be accurate in order to permit such a reconstruction. The principle of accuracy is concretized in Art. 5 No.1 d. Accordingly, every reasonable step must be taken to ensure that personal data that are inaccurate, with regard to the purposes for which they are processed, are erased or rectifed without delay. Therefore, there should be an obligation to evaluate the systems on a regular basis, as is the case for the security measures under Article 32 GDPR.20 In its recitals, the legislation makes clear at various points that strict security measures are important, because personal data processing can lead to severe physical, material or non-material damage.21 In order to maintain security and to prevent processing in infringement of this regulation, the controller or processor should evaluate the risks inherent in the processing and implement measures to mitigate those risks, such as encryption.22 In Art. 35 GDPR, the legislation provides an instrument for the evaluation of risks, the so called "*data protection impact assessment*". A data protection impact assessment must be established for sensitive processes before they are introduced. The legislator has hereby provided for a risk-based "technology impact assessment", which serves to take a closer look at the effects of the use of new technology on society and

<sup>18</sup>See also Conrad: Künstliche Intelligenz – Die Risiken für den Datenschutz (Artifcial intelligence – The risks for data protection), DuD, 12/2017, p.743.

<sup>19</sup>Recital 39 of GDPR.

<sup>20</sup>Conrad: Künstliche Intelligenz – Die Risiken für den Datenschutz (Artifcial intelligence – The risks for data protection), DuD, 12/2017, p.743.

<sup>21</sup>Recitals 75, 76, 77, 78, 79, 83 of GDPR.

<sup>22</sup>Recital 83 of GDPR.

the environment.23 The outcome of the assessment should then be taken into account when determining the appropriate measures to be taken in order to demonstrate that the processing of personal data complies with the GDPR.24 However, MLS that are trained using external data, might be very easily manipulated. Since the MLS decision criteria are often unknown, it is diffcult to foresee and block all conceivable manipulation attempts. Accordingly, it is to be expected that MLS will be subject to special security requirements. Here, special certifcations and audits could be helpful in the future.

## **9.6 Results of the Combined Ethical and Legal Analysis Approach**

The ethical analysis was conducted along the data processing steps of an MLS and under the consideration of the ethical values suggested in ALTAI. Ethical aspects discussed here accorded to the ones previously identifed by Paraschakis (2016, 2017) for recommender systems in e-commerce, such as user profling, privacy and online experimentation and Karpati, Najjar and Ambrosio for a food recommender system (Karpati et al. 2020).

Also, some of the identifed ethical issues can be associated with the ethical areas of concern found by Milano et al. (2019) such as social manipulability. Additionally, using the FoodApp scenario some aspects were found that cannot be easily added to one of the categories in the previous work, such as environmental concerns induced by the increasing traffc due to the individual food delivery and packaging waste. Questions related to the use of digital applications for individual services resulting in the appearance and stimulation of the gig economy were also evident in the FoodApp scenario. Here, the delivery workers were a major part of the business model, while also facing uncertain labor conditions and minimal autonomy within their employment (Rosenblat and Stark 2016; Doteveryone 2019; Aguiléra et al. 2018). The last issue is being addressed by the European Commission and resulted in a directive that defnes platform work as well as the workers' rights, especially in the context of algorithmic decisions (European Commission 2021).25

Using the FoodApp scenario several ethical issues were identifed. The legal analysis showed that most of these issues have corresponding legal requirements that need to be implemented into the digital product for compliance. On the other hand, the combined ethical and legal analysis identifed some ethical issues that are not yet included in the considered legislation.

<sup>23</sup>Conrad: Artifcial intelligence – The risks for data protection, DuD, 12/2017, p.743.

<sup>24</sup>Recital 84 GDPR.

<sup>25</sup>Directive Of The European Parliament And Of The Council, 9.12.2021, https://ec.europa.eu/ commission/presscorner/detail/en/ip\_21\_6605

The FoodApp scenario demonstrated the lack of transparency in the collection and use of user data. The purpose and the lifecycle are not visible for the user, depriving him or her from autonomous decision making about the data collection. An informed choice about the impact on user's privacy is thus not possible. These ethical issues are addressed by the GDPR in the Art. 5 that requires specifed, explicit and legitimate purpose for data collection. Art. 12 GDPR requires transparent information, communication and modalities for the exercise of the rights of the data subject. The lack of possibility for opting out of specifc data collection and therefore the ethical issue of autonomous control over the algorithm parameters of a recommender system. Articles 25 and 32 GDPR require privacy by default and opt-in for the specifc data aspects to be collected, making the opt-out option not suffcient for the compliance. Art. 17 of GDPR states the right to erasure by the data subject, thus allowing the user to end the use of her or his data by the recommender system algorithm. Another transparency and privacy aspect identifed in the FoodApp scenario is that the FoodApp is part of the Acima enterprise. Since the user does not have any information about the data lifecycle, it is fair to assume here that Acima stakeholders or products can have access to FoodApp user data. Legal analysis showed that this ethical issue of accessibility is addressed by the GDPR Art. 29 and 5 that require contractual agreement between data subject and controller or controller and processor.

The FoodApp scenario demonstrated the lack of a feedback loop from the app's stakeholders. The ordering and delivery processes are not refected upon with the restaurants involved. The waste and traffc issues are not included into the recommender algorithms as well as are not monitored together with the local authorities. On the other hand, FoodApp's algorithm is optimized for re-ordering, hence for the increase of individual deliveries and waste associated with these deliveries. Hence, the values of social and environmental well-being and the associated ethical issues such as waste reduction, reduction of traffc and fuel consumption are not considered. Also, no legal consideration was identifed that considers these issues in the digital consumer context. While FoodApp's business model heavily relies on the network of the delivery partners, the job assignment algorithm is not known for the delivery persons and the effect of the user reviews on the job assignments is not transparent neither for the users nor the delivery partners. These ethical issues of fairness and job accessibility are also not addressed by a legal requirement.

Autonomous decision making as well as the tendency for ethical consumption is endangered in the context of the FoodApp by the optimization goal of re-ordering and the given choice of restaurants and cuisines. The user is not enabled to make suggestions about restaurants or cuisines or forage for new options, engraving the skills and tastes within the app. The working conditions of the delivery persons are not visible or known to the user as well as the rewarding mechanism, e.g., tips, are not made available. In this process, the user is detached from the physical part of the service.

Combining legal and ethical analysis shows that some of the identifed ethical issues are already covered by existing legislation. Nevertheless, bigger negative effects such as the effects on the environment or the society are part of the social awareness and responsibility that are not (and maybe should not be) regulated, but can be supported by socially acceptable IT artefacts.

As the legal norms can be implemented differently by those responsible for business and design decisions, as they do not provide specifc processes for the implementation into the digital technologies. Ethical issues and views are more complex and disperse than legal norms and as such their implementation into the digital products can be less explicit. Therefore, we introduce and use the terms of socially aware IT, as such a system would consider and integrate the legal and ethical requirements into the design of the information system. The added effort could lead to a socially acceptable IT product. To ensure the remaining and homogeneous quality adherence, inter-company assessment mechanisms could be put in place.

Ethical issues that occur due to the use and implementation of digital products have been identifed in research over the years. The FoodApp scenario analysis has demonstrated some of the complex effects that MLS can introduce as well as the resulting ethical issues. While some of these ethical issues were converted into legislation in some countries, e.g., within the European Union, others are actively debated in terms of the regulatory needs. ALTAI provides the set of ethical values that need to be considered when an MLS is being designed or used. While ethical values are not as binding as legal requirements, some of them are already incorporated into the legislation, i.e., the ones that are provided by the European Commission. The ethical analysis would uncover the ethical issues that might not yet have a legal equivalent. It is a matter of democratic discussion to decide whether and which ethical issues will need legal regulation and which ones can rely on the social contract.

## **9.7 Conclusion and Outlook**

In this paper a combination of legal and ethical analyses was presented and applied to use case of a food recommender system, the FoodApp. This analysis approach showed aspects for ethical concern, such as decisional deskilling, emergence of structures of economic dependencies as well as additional effects on the environment induced by the individualization of the recommended service. The data-based approach chosen here presents an actionable radius for the system engineer and data analyst to include ethical and legal compliance during the design process of the ML-component by identifying operationalized ethical issues within software development.

Legal analysis of the FoodApp scenario showed that many of the ethical concerns are already addressed by the European GDPR legislation. However, targeting a broad area of data processing applications, the GDPR depends on interpretation, future jurisprudence or even new, more detailed legislation. Especially, the values of social and environmental well-being and even some of the aspects addressing human autonomy and oversight might need a legal fundament.

Although, the study showed that users of digital services may have expectations that are in part already covered by legal regulations but some of the identifed ethical issues also rely on the ethical awareness of the company and thus go beyond legal compliance. While the ethical analysis revealed issues that go beyond the existing regulations on data protection, legal analysis showed the range for the interpretation of the legal regulation. Also, issues that are a matter of business ethics rather than legal regulation such as the awareness of the environmental impact and on the labor market originating from the broad usage of the digital platform were identifed.

The combined analysis showed that MLS, specifcally the examined food recommendation systems, have effects that do not only concern data processing, but that are rather beyond the direct interaction between the user and the system. While the GDPR addresses the data processing aspects such as user privacy and transparency, the effects of the usage of a food recommender system require an interdisciplinary discussion about the need of further regulation.

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Chapter 10 Factors Infuencing Trust and Use of Recommendation AI: A Case Study of Diet Improvement AI in Japan**

**Arisa Ema and Takashi Suyama**

**Abstract** To use AI systems that are trustworthy, it is necessary to consider not only AI technologies, but also a model that takes into account factors such as guidelines, assurance through audits and standards and user interface design. In this paper, we conducted a questionnaire survey focusing on (1) AI intervention, (2) data management, and (3) purpose of use. The survey was conducted on a case study of an AI service for dietary habit improvement recommendations among Japanese people. The results suggest that how the form of communication between humans and AI is designed may affect whether users trust and use AI.

**Keywords** Trustworthy AI model · Data management · Health recommendation · HCI · AI ethics

## **10.1 Society 5.0 and Recommendation AI in Japan**

In the wake of social issues where knowledge and information are not suffciently shared in a society, Japan's Cabinet Offce has proposed a vision of a future society called Society 5.0. As a follow-up to the hunting and gathering (Society 1.0), agricultural (Society 2.0), industrial (Society 3.0) and information societies (Society 4.0), Society 5.0 aims to develop the economy and solve social issues by integrating cyberspace and physical space (Cabinet Offce 2016).

Society 5.0 was announced as part of Japan's Fifth Science and Technology Basic Plan in 2016 and continues to be referred to as the vision of the Japanese government and business entities. For example, in 2018, 2020, and 2022, Keidanren (Japan Business Federation) issued its 'Healthcare in the Age of Society 5.0' proposal, presenting a vision of a society that offers new options for diverse needs in health management and medical treatment (Keidanren 2022). It proposes that

A. Ema (\*) · T. Suyama

The University of Tokyo, Tokyo, Japan e-mail<EMAIL_ADDRESS>
<sup>©</sup> The Author(s) 2023

S. Genovesi et al. (eds.), *Recommender Systems: Legal and Ethical Issues*, The International Library of Ethics, Law and Technology 40, https://doi.org/10.1007/978-3-031-34804-4\_10

'health management can be achieved by making appropriate recommendations suited to the situation at the time, without having to make a diffcult decision', and it is expected that artifcial intelligence (AI) will be customized to recommend public actions.

While recommendation AI is expected to be used in many felds to realize Society 5.0, there are also many challenges. Some people may psychologically resist recommendations made by AI. It has been pointed out that that especially in Japan, elderly respondents showed negative attitudes towards recommendation AI services (Ikkatai et al. 2022). Data management is also a challenge for AI to provide information customized to individuals. For example, there was a controversy in Japan surrounding the use of data of job-seeking students by companies without the students' consent in 2019 (Kudo et al. 2020). Whether data is appropriately managed or used for purposes for which it was not intended can infuence people's decisions to use AI services. In this paper, we present the results of a survey that examined the perspectives of users in deciding whether they would like to use the recommendation AI services.

## **10.2 Model for Ensuring Trustworthiness of AI Services**

Ensuring trustworthiness of AI services has been an important topic in recent years. In Europe, a report on Trustworthy AI was released by the European Commission in March 2019. Simultaneously, Japan's Cabinet Offce released 'Social Principles of Human-Centric AI', which include building a mechanism to ensure the trustworthiness of AI and the data or algorithms that support it (Cabinet Secretariat 2019).

Incorporating various AI principles into practical processes has been gaining increasing attention in recent years (Jobin et al. 2019). Trusted AI is achieved not only through technology but also through an 'AI governance ecosystem' that includes various elements such as guidelines, auditing, standardization, user interface design, literacy and education (Japan Deep Learning Association 2021), and various models for ensuring trustworthiness have been proposed.

The DaRe4TAI framework by Thiebes et al. examines whether privacy protection and discriminatory judgements occur at all stages of data input and output and AI model building (Thiebes et al. 2020). In Japan, several frameworks have been published to promote trusted AI in society. For example, the Ministry of Economy, Trade and Industry (METI) released the 'Governance Guidelines for Implementation of AI Principles, ver.1.1' in 2022, which presents action goals for AI providers to implement (METI 2022). An increasing number of companies are establishing their own AI principles and guidelines; Fujitsu Limited, for example, has developed a method to disseminate trustworthy AI to society and published its procedures and application examples (Fujitsu Limited 2022). Research is also being conducted at universities, and the Risk Chain Model (RCModel) is structured into three layers: AI model (e.g., accuracy and robustness), system (e.g., data and system infrastructure), service provider (e.g., behavioral norms and communication) and user (e.g., understanding, utilization and usage environment) (Matsumoto and Ema 2020). The framework guide and case studies1 are available for considering ways to ensure the reliability and transparency of AI services by structuring them into these three layers.

By establishing such ethics policies and guidelines, AI service providers are trying to ensure social trust, and publicize that they are developing and using technology in an appropriate manner. However, having policies and guidelines is not the same as being a truly trustworthy organization. The issue of 'ethics washing' has been pointed out: an organization may claim to behave ethically, but it may be just a façade or a 'smokescreen to hide their transgressions' (Dand 2021). Therefore, while it is important to have these principles in place, it is also important to communicate appropriately with users.

## **10.3 Components of a Trustworthy AI Model**

While there are several models and frameworks for ensuring trustworthiness of AI services, this paper proposes a model that places particular emphasis on the interface between the service provider and the user. In particular, we investigate not from the perspective of what technical requirements are necessary for AI technology to be trustworthy, but rather from the hypothesis that how the form of communication between humans and AI is designed may affect whether users trust and use it. For example, even if the AI services offered by companies are the same, user preferences will vary depending on how purpose of use is explained and how much freedom of choice users are given.

There are many studies on AI-human interface design, especially on the general bases for human trust in automated machines such as AI (Madhavan and Wiegmann 2007). Research points to the performance of the machine (e.g., abilities), the process of how the machine works, and whether its design achieves the designer's purposes as the bases for trust (Lee and See 2004). To gain the trust of users as well as designers, it is important that AI is in accordance with their preferences. Trust in machines is not only related to the machine, but also to the governance of the company, such as whether data is appropriately managed. Therefore, we conducted a survey based on the hypothesis that three aspects of interaction are important: (1) AI intervention, (2) data management and (3) purpose of use. Although a variety of AI services are currently available, this paper specifcally investigates recommendation AI as a case study.

<sup>1</sup>Case studies are published at The University of Tokyo website: https://if.u-tokyo.ac.jp/en/projects/ai-service-and-risk-coordination/

## *10.3.1 AI Intervention*

Service providers need to understand users' preferences for AI services. For example, some users may not want AI to make decisions and prefer human judgement, while others may prefer to use AI services only as a reference but only if a human makes the fnal decision. Conversely, others may prefer to have AI alone make decisions without any human intervention.

Thus, the degree of AI intervention varies widely, and it may be diffcult to tell the difference between a pattern in which decisions are made by humans alone and a pattern in which AI is used but fnal decisions are made by humans unless the process is clearly explained. Therefore, from the viewpoint of transparency, service providers are expected to log the process by which decisions are made and to establish organizational governance to provide AI services. Users are also expected to use AI based on information and explanations provided by developers and service providers in some cases, as considered in the principles of proper utilization in the 'AI Utilization Guidelines' proposed by the Japanese government (Ministry of Internal Affairs and Communications 2019).

## *10.3.2 Data Management*

To ensure the reliability of AI services, it is imperative for users and service providers to discuss whether data management is appropriate. Some users may ask service providers to explain the security measures they have employed while learning how they manage data, while others may trust the company and refrain from checking the security measures.

Therefore, service providers are expected to indicate what data management they adopt. While there are various certifcations and standards for data management, some companies have adopted their own standards, and management methods are diversifying.

## *10.3.3 Purpose of Use*

When users are deciding whether or not to use an AI service, it is important for them to know what information will be taken from the service and for what purpose it will be used. The general means of communicating the purpose of use is the terms of service agreements.

To avoid too-long and too-complex terms of service agreements that go unread (Maronick 2014), service providers should provide agreements that are easy to understand and of appropriate length. Currently, there are formats allowing users to choose what they agree or disagree with, rather than just agreeing or disagreeing with all of them at once. In addition, some terms are designed so that there are no negative consequences, such as services being provided even if users do not agree with all of them. The way the terms of use are written and presented is also becoming more diverse.

## **10.4 Verifcation of Trustworthy AI Model: A Case Study of AI for Dietary Habit Improvement Recommendations**

There are various types of recommendation AI services, but this paper takes recommendation AI for dietary habit improvement utilizing users' dietary data as the case study. As Japan's Keidanren (Japan Business Federation) has proposed a vision of what health management should be in Society 5.0, there is a high affnity and need for AI and healthcare services. Applications in which AI provides menu and nutrition management support and interactive AI that analyses meal menus in real time and guides people towards appropriate eating habits are available in Japan.

A survey was conducted from January to March 2021 to examine what kind of AI services, data management methods and purpose of use would make users willing to use AI for dietary habit improvement recommendations.

## *10.4.1 Subjects*

A research frm was commissioned to select respondents to ensure that Japanese men and women and their ages (20–60s) were equally represented, and in-depth interviews were frst conducted with nine of them. For the subsequent survey, the same group of respondents was commissioned to a research frm, and 500 respondents were included. Since the purpose of the survey was to target general users, those who responded in the pre-survey that they were 'familiar with AI' and 'not at all reluctant to use AI services' were excluded from the survey.

## *10.4.2 Verifcation 1: AI Intervention*

In-depth interviews were conducted to determine the degree of AI intervention in recommending services. It was found that respondents tend to prefer services in which 'AI can be used, but the fnal recommendation is made by a human' rather than 'AI making the recommendation' for dietary improvement advice. Two trends were obtained as reasons for this: AI performance is considered to be not as good as that of humans, and the ability to ask questions and have conversations is considered important in healthcare.

Therefore, four patterns were displayed and tested in the quantitative survey as demonstrated in Table 10.1: a dietitian with AI (No. 1) and AI (No. 2); a dietitian with AI (No. 1) and a high-performance AI that makes suggestions equivalent to those of a dietitian (No. 3); a dietitian with no questions or conversations (No. 4) and AI (No. 2); and a dietitian with no questions or conversations (No. 4) and a high-performance AI (No. 3). Respondents were asked to indicate which of these four patterns of food improvement advice they would prefer to use.

**Table 10.1** AI intervention descriptions

## *10.4.3 Verifcation 2: Data Management*

When we conducted an in-depth interview to determine how users evaluate whether data management is properly implemented, we received several comments indicating that there is an emphasis on whether some kind of certifcation is obtained rather than a detailed explanation of what kind of technology is used to ensure security measures.

Therefore, we displayed three patterns of questions in the survey, as Table 10.2 shows: a comparison between ISO2 and a company's own standards as a representative example of certifcation, and an explanation of those certifcations and specifc security technologies to verify which service users would prefer.

## *10.4.4 Verifcation 3: Purpose of Use*

In-depth interviews were conducted to investigate what type of explanation of the purpose of use is preferred in the terms of service agreements for AI services and how best to obtain consent. From the interviews, 'the items for obtaining consent are explained in detail' and 'users can customize what they agree or do not agree to' tended to be preferred.

Therefore, in the survey, we created terms of service agreements (Table 10.3) and three patterns of questions were displayed: users agree collectively by checking one

**Table 10.2** Data management option descriptions

<sup>2</sup> ISO is a private organization headquartered in Switzerland and stands for International Organization for Standardization. Since differences in product size, quality, safety, and functionality from country to country can hinder international trade, the ISO is intended to create standards. Currently, ISO/IEC JTC1 is the forum for discussing international de jure standards for AI. There are several working groups and trustworthiness of AI is also discussed.

**Table 10.3** Description of purpose of use in terms of service agreements

item; users check fve items per category, and users check all (15 items), to verify whether users' preferences change with the number of checked items. In the cases of checking fve and 15 items, we also created a pattern with optional check items and compared it with a pattern in which check items are required.

## *10.4.5 Method*

The quantitative survey format was created using the maze web tool,3 and the two patterns were displayed simultaneously on the left and right sides of the screen, with the different patterns highlighted. The respondents were asked to choose which service they would prefer to use (Figs. 10.1 and 10.2).

<sup>3</sup>maze, https://maze.co/

195

**Fig. 10.1** Example of screens with different 'AI intervention'

**Fig. 10.2** Example of screens with different 'terms of service agreements'. (A has fve check items with customization, B has 15 check items with no customization)

## *10.4.6 Results*

#### **<IP_ADDRESS> AI Intervention**

Among the recommended services, Fig. 10.3 shows the results of user preferences for the pattern in which a person or AI ultimately recommends the service, the pattern in which AI is as high performing as a person, and conversely, the pattern in

**Fig. 10.3** Preference for AI intervention (n = 500)

which a user cannot talk to a dietitian (human). The fgure shows that the strongest preference is for services that are ultimately recommended by a dietitian (human) using AI. In addition, when a dietitian (human) is compared with a high-performance AI that can make judgements similar to a human, the human recommendation nevertheless prevails. Conversely, the ratio of those who prefer to use the AI recommendation increases, reversing the result for the recommendation by a dietitian (human) who cannot ask questions. These results suggest that users emphasize the interactivity of the recommendation service, such as the ability to have a conversation, rather than the high-performance AI. This tendency was more pronounced among the group that responded that they are 'resistant to AI services' (Fig. 10.4).

### **<IP_ADDRESS> Data Management**

What do users tend to look for in a recommendation AI to determine if the data is being appropriately managed? Figure 10.5 shows the results of user preference for patterns in which ISO or a company's in-house standards are clearly stated and in which there is an explanation of specifc security technology. The fgure suggests that some kind of certifcation, such as ISO or acquisition of a company's own standards, is more important for gaining user trust than an explanation of specifc security technologies.

Compared to in-house standards, users tend to prefer ISO. This suggests that socially recognized certifcation is more reliable. In this survey, it was set as 'major well-known Japanese companies' that established their own in-house standards. Contrary to the assumption that the standard set by a major well-known company

**Fig. 10.4** AI service preference of those with AI service resistance (n = 500)

would be considered more reliable, users preferred ISO, suggesting that it is less signifcant to display one's own standards, regardless of company size or name recognition. Contrarily, the results of the study suggest that companies would be more likely to earn socially recognized certifcation such as ISO, which would contribute to user preference.

#### **<IP_ADDRESS> Purpose of Use in Terms of Service Agreements**

Which consent formats are preferred by users? Figure 10.6 shows the results of user preferences, which are classifed into patterns based on the number of consent check items and customizability. The fgure shows that users prefer the one with a higher number of consent check items. This suggests that a large number of items is preferred, even though it is troublesome to read through them.

Meanwhile, users preferred the customizable pattern with 'optional' check items in addition to the 'required' ones, regardless of the consent items. This suggests that users prefer patterns that allow them to choose the contents of consent by themselves. However, in this survey, users do not actually click the mouse to check the item. The results of this survey should be verifed further, as the cumbersomeness of the operation may prevail if users are actually required to click to check the boxes.

**Fig. 10.5** Preferences of data management (n = 500)

## **10.5 Necessary Elements for Trusted AI**

Technical developments such as explainable and fair AI are considered important for trusted AI services. This paper focuses on communication between users and service providers among non-technical aspects. To facilitate communication, we investigated what kind of AI intervention, data management and purpose of use would be considered acceptable by users for dietary habit improvement recommendation AI.

**Fig. 10.6** Preferences of terms of use (n = 500)

Reliability of services in general depends on users understanding an accurate description of how they work. However, AI is subject to changes in the environment in which it is used and in the algorithms that result from the learning process. Considering that predictability of behavior is diffcult not only for users but also for service developers, transparency and accountability are important for users to trust AI services. Specifcally, communication is required so that there is no discrepancy in understanding between users and service providers regarding AI intervention and data management.

The results of this survey suggest that people tend to trust recommendation AI that is communicative, allowing users to ask questions and engage in dialogue when it is recommended. For those who are resistant to AI, it is also expected that rather than promoting the technical perspective of 'being a high-performance AI', it would be more acceptable to develop a service that allows users to talk and ask questions to people while using AI in the background for analysis. Advertising highperformance AI that can provide advice equivalent to that of a human being or providing detailed information about the technology used for data management is not likely to have a signifcant effect on user preference.

In addition, the pattern of ISO for data management was highly rated. However, when the respondents who preferred the pattern with ISO in the in-depth interview were asked if they could explain what ISO is, few had extensive knowledge except that it is a standard of some type. Furthermore, the results showed that the more detailed the items for obtaining agreement to the terms of service agreements, the better, but other studies show that people don't read terms of agreements (F-secure 2014; Japan Fair Trade Commission 2021). Therefore, this suggests that the detailed content itself is a sign of trust. These results also indicate that non-technical aspects, such as socially formed understandings, have a greater infuence on user preferences than the actual existence of technical guarantees and the fact that they are explained. This study suggests that service providers should not ignore these nontechnical perspectives when it comes to trusted AI.

This is a case study of AI recommendation for dietary improvement services. However, AI recommendation services are employed in various felds, and not all recommendation AIs are similar to those in this survey. It can be assumed that health management and medical-related felds are areas where interaction with people is particularly important. For example, users may prefer recommendation AI rather than people for complex route guidance because AI can better perform complex processing. Therefore, an international trustworthiness survey focusing on AI intervention, data management, and purpose of use in various service cases is needed to consider the requirements for trustworthy AI.

**Acknowledgements** This paper was supported by the Toyota Foundation D18-ST-0008 'Building platform on ethics and governance of artifcial intelligence'.

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Chapter 11 Ethics of E-Learning Recommender Systems: Epistemic Positioning and Ideological Orientation**

#### **Lisa Roux and Thierry Nodenot**

**Abstract** Recommender systems are increasingly used in e-learning to provide users with personalized services and advice. Depending on the specifc context for which the system is implemented (e.g., homework on a specifc subject for university students, new training courses for life-long learners), the objectives and proposed items, the chosen recommendation techniques, the features that are considered, the way the recommendations are presented to the users are closely related to the designers' perception of learners and knowledge. The various approaches refect different epistemic and ethical viewpoints; for example, representing people using fxed models is easier to process, diagnose, predict and explain, but presents a partial view of reality and obscures the fact that they are complex and evolving individuals. Similarly, some fltering methods can restrict the view of available courses to items considered similar to those that the learner has already followed, thus promoting specialization rather than diversifcation and openness. This aspect is closely related to fundamental issues involved in the theory of knowledge, questioning the notions of utility and purposes of science, as well as a key issue for academic change and, more fundamentally, that of modern societies. Indeed, these issues should be seen in a broader context of refection about the economic changes and ideological transformations of a society grounded on neoliberal capitalism. The main goal of this study is to explain how the design of recommender systems in e-learning has both ethical and practical implications since it refects an ideological conception of science and techniques, thus requiring a previous examination of these issues in order to defne the theoretical model of knowledge in which it takes place. For that purpose, we study the certain visions of teaching and learning that can be brought about by algorithms and models used by existing recommender systems in e-learning.

**Keywords** Recommender system · e-learning · Ethics · Implications of design choices

L. Roux (\*) · T. Nodenot

Université de Pau et des Pays de l'Adour, E2S UPPA, LIUPPA, Anglet, France e-mail<EMAIL_ADDRESS>
<sup>©</sup> The Author(s) 2023

S. Genovesi et al. (eds.), *Recommender Systems: Legal and Ethical Issues*, The International Library of Ethics, Law and Technology 40, https://doi.org/10.1007/978-3-031-34804-4\_11

## **11.1 Introduction**

Recommender systems are widely used in e-commerce, social networks, Video On Demand, etc. They are based on collected information about user preferences, which can be acquired implicitly (e.g., collecting the data describing the users' behavior) or explicitly (e.g., collecting the users' ratings, using social and demographic information). They aim at guiding users within the wide range of products offered by e-platforms so that they can fnd the items that they are most likely to engage with. Similarly, recommender systems for e-learning are used to help learners to deal with the abundant learning resources and activities available on e-learning platforms. They are used to support individual learning and provide the learner with the appearance of personalized tutoring in environments of large classes but limited human contact (i.e., reduced teacher to student ratio). In the context of the ongoing COVID-19 pandemic, the use of such recommender systems appeared crucial for the student's involvement and engagement, and to help teachers optimize the assistance they provide. They are also expected to effciently help students to choose their learning paths, effcient pedagogical activities, and suitable course material. Indeed, many papers present good results concerning the effectiveness of their recommendation approach, whether it be based on task diffculty ranking (Segal et al. 2014), learning material or path recommendation (Mbipom et al. 2018; Ye et al. 2015; Shi et al. 2020), friend recommendation (Rafaeli et al. 2005), learning object recommendation (Gallego et al. 2012; Fraihat and Shambour 2015), or performance evaluation (Amasha et al. 2020; Moore et al. 2019). Furthermore, from the institution's perspective, they can be used to increase their economic benefts, since they enable larger volumes of students with fewer teachers.

However, if lucrative economic interests come frst, this may raise a number of ethical concerns, both theoretically and practically (Beach and Dovemark 2009). Recommender systems are widely used for e-commerce and entertainment (i.e. music, video on demand (VOD), video game platforms, etc.). In early 2000, the frst notable applications appeared in the domain of education (Manouselis et al. 2012). The development of the Internet has stimulated the research in recommender systems in order to improve the fltering and assessment methods, in particular through the involvement of economic actors who encourage them to increase their benefts (e.g. the Netfix prize that challenged machine learning and data mining). Such practices not only promote algorithmic development but also affect our relationship to education, culture, etc. (Hallinan and Striphas 2016).

First of all, as Catherine Hayles (1999) shows, new technological contexts (thus by inclusion new algorithmic contexts) change our relationship to our environment and, more fundamentally, alter our meaning of humanity, culture, sociality, etc. Moreover, the methods developed for VOD, e-commerce, e-learning etc. are the same; thus, the methods used to sell products and increase the benefts of an enterprise are also used to recommend learning items to students, which may cause problems. Indeed, recommendations in e-commerce can be accused of being more meant to help sellers than customers, just as advertising is. In e-commerce, the sellers use recommender systems to increase the number of sales and their income, sometimes with very little concern about what their customers really need. From this standpoint, they can be considered as the real customers of recommender systems, instead of the sellers' customers. Some studies have shown that recommender systems in e-commerce not only help the sellers' customers to get what they prefer: they also contribute to shaping their preferences (Adomavicius et al. 2019). Yet, even though this very practice is already highly questionable in e-commerce, it is even more critical in the case of e-learning. In particular, recommender systems in e-learning cannot pretend to stand apart from the academic debates about university, notably concerning its aims and values. For example, should university train future workers with competencies consistent to the job market, or whole citizens who will be able to take an active part in the social and political life of the society? (Florian 2018; Brighouse and McPherson 2015) Is university threatened by a commodifcation of knowledge process? (Jacob 2003).

As part of the refection about education trends and policies, and besides all its potential benefts, the development of recommender systems in e-learning gives rise to ethical and social issues, including privacy problems, lack of control, etc. Just as the other personalized e-learning functions, the recommender systems use personal information, generally using algorithms poorly understood by the users, in order to offer them services. Consequently, the opacity of such systems raises a main issue because the users are recommended items without being provided with the knowledge to understand these recommendations, thus knowing when to choose to follow them or not. In order to understand why it can be problematic, we will explain how recommender systems can promote particular visions about education, pointing out that these assumptions need to be disclosed.

The paper is organized as follows: Section 11.2 presents the main methods used in recommender systems; Section 11.3 exposes the potential problems posed by applying them in the feld of pedagogical recommender systems; the problem statement is exposed in Sect. 11.4, which is followed by an outline of proposals to overcome some harmful consequences of the use of recommender systems for e-learning (Sect. 11.5).

## **11.2 Methods of Recommender Systems**

In the literature, recommender systems are generally classifed into two types, depending on how the recommendations are made: content-based and collaborative fltering. Also other methods, such as knowledge-based fltering, more complex and diffcult to implement, receive attention in specifc felds, such as e-learning.

Content-based fltering (Van Meteren and Van Someren 2000; Pazzani and Billsus 2007) is a very common method for recommendations in e-commerce, VOD, etc. because it is both effcient and easy to implement. The principle is very simple, and entirely centered on the comparison between the user's interests and the item's features. The user is recommended items that most resemble the items that they have already liked, or just consulted. For that purpose, the system analyzes the user's favorite items and the features of these items to identify the user's preferences. It only requires building a model of the user preferences, using the item features and a history of the user's consultations, likes and dislikes, etc. But other information can be taken into account to build the user model, such as demographic characteristics (e.g. age, nationality, gender).

Collaborative fltering (Schafer et al. 2007; Afoudi et al. 2018) is another common method, in which the user is recommended the favorite items of the users with similar tastes. For that purpose, the system identifes the active user's closest users and the items that they have liked but that the active user has never consulted yet, in order to recommend them. The recommendation can be based on the values of recorded interactions only (memory-based approaches) or on a generative model that explains the user-item interactions in order to make predictions (model-based approaches). This method has been popularized by Amazon, considering that whether a user has consulted a product x, they should be also interested in a product z, but it is mostly used by social networks to recommend new social connections (e.g. friends, groups).

Knowledge-based fltering (Aggarwal 2016; Bouraga et al. 2014) is far more diffcult to implement. It consists in recommending items based on explicit recommendation criteria, information about the user preferences, and all the characteristics of every available item. The system resorts to knowledge representation, for example in the form of rules about how an item meets a particular user. In e-commerce, they are generally used for items that are not purchased very often, such as a car, tourist destination, etc.

Since every method has drawbacks and limits (e.g. cold-start problem of collaborative fltering: in case of new users, the system does not have information about their preferences in order to make recommendations (Lika et al. 2014)), hybridization (Burke 2007; Isinkaye et al. 2015) is increasingly used to overcome them. It consists in coupling different fltering methods to make better recommendations. Many hybrid methods can be used, such as cascade (i.e. one recommender produces recommendations that will be refned by the other) or switching (i.e. the system uses a criterion dependent on the situation to switch between recommendation techniques).

Different evaluation metrics can be used to assess how a recommender system performs (Isinkaye et al. 2015). Among the most commonly used are the methods based on precision (i.e. number of selected items that are relevant) and recall (i.e. number of relevant items that are selected). In such methods, the assessment uses the user's ratings to know which items are relevant. For example, the system tries to predict the score that a user would grant to an item, then calculate how much the predicted score is from the actual score.

## **11.3 Recommender Systems in e-Learning**

E-learning recommender systems are based on the same methods as recommender systems for e-commerce, VOD, etc. (i.e. content-based, collaborative, and knowledge-based fltering, or hybrid methods). Due to the numerous and various courses, programs, pedagogical resources, and activities available online, more and more research is conducted to propose recommender systems able to support learners, either students in school or long-life learners, in deciding which courses to follow, what resources to consult, depending on their preferences, their needs, their expectations, their skills, at a given time.

We consider that a recommender system in e-learning can be described by the following dimensions:

Some recommender systems are described by an additional dimension:

4. The teaching model that describes how the recommendations are provided to the student (e.g. the rules of item selection, the granularity of recommendations, the communication acts).

To a certain extent, recommender systems in e-learning have similarities with recommender systems in e-commerce. For example, they use the same fltering techniques (i.e. content-based fltering, collaborative fltering, knowledge-based fltering, or hybridization). But we notice that they also have some specifcities:

There is not one good way to model a recommender system in e-learning. The effciency and the relevance of the methods and features used may vary depending on the problems (i.e. the learning objectives, the intended users, the variety and the degree of specialization of course material, etc.). But, beyond their purely practical aspects (i.e. effciency and accuracy of the recommendations, system speed and responsiveness, adaptability, etc.), some ethical issues intrinsic to the feld of learning are specifc to recommender systems in e-learning and have some direct practical implications. In particular, the choices of fltering techniques, learner model, and assessment methods have direct consequences on social and epistemic openmindedness, the diversity of thinking, and the conception of knowledge and learning. As we will see, these issues are closely related to the aforementioned debates about the function and the utility of knowledge.

## *11.3.1 Filtering Techniques: What Implications on Social and Epistemic Open-Mindedness?*

Recommender systems in e-learning use the same methods and adapt them to the specifc educational feld; but some of them are more likely to offer the user a choice of tools directly relevant to the current job market, thus promoting a utilitarian vision of science, while others tend to encourage and support the idealist perception of holistic knowledge by proposing more diversifed courses, maybe dealing with rarely studied subjects. For example, one shortcoming of content-based methods is that they induce a lack of serendipity, that is, very few encounters with the unexpected when seeking something else. The lack of serendipitous exposure has been denounced as harmful in social media and online newspapers since it causes the apparition of "flter bubbles", seen as huge threats to democracy and social links, polarizing and fragmenting the social space (Pariser 2011). This method is sometimes used in e-learning recommender systems, such as labelled items (Li et al. 2008) and distance between concept vectors (i.e. measurement of the difference between two concepts that are described through mathematical variables) (Ye et al. 2015),

In e-learning, these methods tend to propose courses directly related to the one that the learner has just fnished, encouraging specialization. For example, in the case of content-similarity based on the distance between concept vectors (Ye et al. 2015), only the semantic relatedness is taken into account, using an automatically calculated relatedness matrix. Therefore, in this case, whether a student has just fnished a course about Python programming essentials for data analysis, which deals with Python programming language, data science, and object-oriented programming, they could be proposed courses about the basics of Python programming language, dealing with Python programming language and object-oriented programming, and courses about graphics for data visualization, dealing with Python programming language, data science, and graphic tools, rather than a course about the ethical issues of AI. Still, these ethical issues could be very relevant for their whole training. Thus, some learning materials will never be proposed to the students and will even be overshadowed by the propositions of the algorithm. On the one hand, it allows scientifc expertise and a good and deep comprehension of the non-trivial scientifc objects that are specifc to one discipline, thus increasing the learner's understanding of structures and processes. On the other hand, a certain amount of openness may be essential to remind learners of the multiplicity of the reality levels and the necessity to look beyond the disciplines, because knowledge is in essence an opening to the external world: learning is always a coming out from oneself and an act of inhabiting the world, through insights, perception, and intentionality.

That is why it must be an informed designer choice, for example when an online platform is explicitly designed to improve the learners' expertise in one targeted feld. Indeed, such platforms can be very useful for learners (e.g. when they need to master a specifc tool) and teachers (e.g. in the frame of one specifc subject taught by a teacher, with defned resources and activities, and for which the possibility of distance-learning is offered), but they are not appropriate for general-interest e-learning platforms, where learners want to enrich their global knowledge. Since they facilitate in-depth study of one feld or subject, they can hinder openness. Not only does serendipity allow one to compare and further up one's knowledge with other forms of knowledge, models, and paradigms, but it also brings interest to the discovery. More generally, the models on which the recommendations are based are important too.

## *11.3.2 Model Selection: A Risk of Thinking Homogenization?*

As mentioned, recommender systems generally use two different information sources: the features of users and the features of items. A user model (i.e. the system's internal representation of the user's preferences, needs, expectations, etc.) can be built mainly based on users' ratings on items, users' previous navigation patterns, or the content features of purchased items. Similarly, in e-learning, a learner model, which corresponds to the user model, is mostly used to make personalized recommendations, also based on the features of the learning material, activities, paths, etc. Although some e-learning recommender systems do not use any (Ye et al. 2015), even most content-based recommender systems are based on a learner model, which expresses the learner's interests (Shu et al. 2018), internet history (Khribi et al. 2008), learning styles (Dwivedi and Bharadwaj 2013; Severac et al. 2012), etc. The choice of what is represented through this model is crucial since it refers to a specifc conception of learners and strongly infuences the recommendation. For example, in the Fuzzy Tree Matching-Based Personalized E-Learning Recommender System developed by Wu et al. (2015), the learner profle contains the learner's background, learning goals, prior knowledge, and learner characteristics, specifed by the learner themselves when they registered; while in the Hybrid Attribute-based Recommender System for E-learning Material Recommendation proposed by Salehi and Kmalabadi (2012), the learner profle refers to their preferences obtained from their ratings. Sometimes, but rarely, educational recommender systems use a teaching model. For example, the adaptive neuro-fuzzy pedagogical recommender designed by Sevarac et al. (2012) integrate a set of modal rules based on the student's knowledge, the course sequencing, etc. in order to determine what the system should recommend to the student in a specifc situation. They show that this model is very benefcial for the quality of the recommendations because, although more complex, the learner model is mostly different from the models used, either consciously or unconsciously, by the teachers to make recommendations to their students. Some learner models can encourage overspecialization and pattern reproduction, they can cause a split between different ways of thinking and ignorance of other beliefs, approaches, etc. Since they help learners to flter out information, they prevent them from being exposed to new learning perspectives.

For example, the recommender agent for e-learning systems developed by Zaiane (2002) uses data mining techniques such as association rules mining in order to build a model that represents on-line user behaviors, to suggest activities. In this case, the prediction is based on the current sequence of activities or pages visited by the learner and the other users' frequent sequences of visited pages: basically, the system "learns" from the past activities of one user or a group of users and predicts activities or pages that a given user might be interested in before suggesting them to the user. Thus, learners do not have personalized activity recommendations since these lead to the same learning patterns reproduction. This recommender agent is also based on the course sequencing: if a learner has studied the A course, then they should study the B course. Here again, the recommendations are called "personalized" while it is in fact only an appearance of tailored assistance: the learner model is just based on the history of the followed courses in order to recommend the next one. Other works use a student modelling based on the learning style (Dwivedi and Bharadwaj 2013; Severac et al. 2012). The learning style approach has been widely called into question by recent researches in education (Rohrer and Pashler 2012; Kirschner, 2017), in which it was pointed out it that this approach can have prejudicial effects, such as freezing the students in one single way of learning thus being counterproductive to the development of varied skills. However, since these models remain easy to implement, they are still often used by recommender system's designers. Such systems can reduce the learner's exposition to new ways of learning, new kinds of informational sources. Here again, this can be a pedagogical choice. Indeed, even though differentiated learning is now widely identifed as a crucial approach to improve student academic engagement and success and has become a standard requirement in educative policy led in various countries (e.g. England (Mills et al. 2014), France (Kahn 2017)) notably with the aim to promote social justice, there are also strong critics, notably conceptual (Needham 2011), practical (Mahony and Hextall 2009), and ideological (Pykett 2010; Beach and Dovemark (2009)). The problem is that recommendations provided by recommender systems in e-learning are generally misleadingly presented as personalized. This lack of variety can prevent students from getting used to seeking and exploiting new information vehicles, which can be problematic if teachers expect the recommender system to take over students' exposition to multiple kinds of information. Due to their opacity, the models and data on which recommendations are based remain unknown by teachers and students, who do not have information necessary to understand what "personalized" really means in the case of the systems they use. There is a lack of information and transparency that can prevent teachers from choosing a system that really suits their pedagogical strategy and appropriately use it.

## *11.3.3 Assessment Methods: What Do They Value?*

In some papers (Khribi et al. 2008; Wu et al. 2015; Salehi 2013), the limitations of content-based and collaborative fltering are reported in order to propose better recommendation methods, such as hybridization, coupling content-based and collaborative fltering, but they do not all result in more serendipity. For example, when designers choose the cascade method to couple the content-based and collaborative fltering, the cold-start problem is overcome, thus the recommendation results can be better, but the system still encourages overspecialization. Indeed, the collaborative fltering step only refnes the results obtained by the content-based fltering step. The main problem comes from the metrics used to evaluate the system and what is actually valued.

Several methods are used to evaluate recommender systems in e-learning, such as surveys (Sevarac et al. 2012), the evolution of the number of requests after cleansing the data (Khribi et al. 2008), MAE (Bobadilla et al. 2009). Depending on the method used, various characteristics can be as assessed: surveys assess the user's satisfaction, the evolution of the number of requests assesses the utility, MAE assesses the accuracy, etc. The choice of the evaluation method and what needs to be evaluated is crucial because it both refects the purchased main goals of the system and contributes to defning the fnal system design. Indeed, for example, the assessment methods can be applied to different versions of the system to decide which one to choose. It can also be applied to a single version of the system in order to know whether it can be deployed or whether the design has to be modifed.