Data sharing is the core aspect of open banking. Data sharing drives innovation and in most cases increases the effectiveness and efficiency of existing services, reduces costs and strengthens the relationships of businesses with their clients. However, in the case of open banking as it also involves sharing of personal data, strong requirements related to data protection exist, as explained above. While being two distinct regulations, the bridge between PSD2 and GDPR lays on the consent of the customer of the bank to share his/her data. According to GDPR [5], consent is one of the six legal grounds of lawful processing of personal data, and it needs to be given by a statement or by a clear affirmative action (Fig. 8.1).

Hence, the need for robust and efficient consent management becomes evident. Consent management is the act or process of managing consents from the data subject (or customers in the case of banks) for processing and/or sharing their personal data. And as the PSD2 is built around digital innovation and integration, digital consent management is considered the proper way to tackle the challenges faced for banks towards the compliance with PSD2 and GDPR. The Open Banking Working Group of Euro Banking Association (EBA) reports that digital consent management lies at the heart of any possible solution for the challenges ahead [7]. Digital consent is enabling the banks and/or third-party providers to leverage the opportunities offered by PSD2 in a GDPR-compliant manner. However, the consent management constitutes a process with multiple aspects which shall be taken into consideration during the design and implementation phase.

#### **2 Consent Management for Financial Services**

Consent is defined in Article 4 of GDPR as "any freely given, specific, informed and unambiguous indication of a data subject's wishes by which he or she, by a statement or by clear affirmative action, signifies agreement to the processing of personal data relating to him or her". Furthermore, GDPR requires the explicit consent of the customer in order to process and/or share his/her data. Additionally, the conditions of a consent are strictly defined under Article 7 of GDPR and include, among others, the exact definition of the context of the given consent, the explicit and distinguishable way of the consent gathering as well as the right of the customer to withdraw his/her consent at any time. PSD2 also clearly states that TPPs shall access, process and retain only the personal data that is necessary for the provision of their payment services and only with the "explicit consent" of the payment service user.

Hence, the main characteristics of consent can be grouped as follows [8]:

In addition to this, a consent must contain at least the following information [8] (Fig. 8.2):

Consent management constitutes the process of managing consents from the customers for processing their personal data. The effective consent management enables tracking, monitoring and management of the personal data lifecycle from the moment of opt-in to the data erase in GDPR-compliant manner. The core aim of this process is to facilitate and improve the customer's control over their personal

**Fig. 8.2** Core content of a valid consent

data enabling their right to provide (or withdraw) consent which allows authorised parties to access their personal and business information and its immediate effect. As it enables granular permission consent, it constitutes a key enabler of trust which is vital to maximise data sharing and assure that customers are comfortable with sharing data.

Consent management is composed of the following distinct phases (Fig. 8.3):

A consent management system is the software that facilitates the execution of digital consent management by effectively handling all the operations performed during the complete consent management lifecycle. It acts as the mediator between the involved parties, enabling the formulation, as well as the update or a withdrawal, of a consent and finally its storage into a secure data storage modality.

#### **3 Related Work**

The importance of consent management is evident from the large research efforts and the broad number of diverse approaches and solutions that have been proposed in various domains. All the proposed approaches have the safeguarding of individual's privacy as their core requirement; however, the proposed solutions are handling the specific requirement in many different ways. It is also acknowledgeable that although some generic and cross-domain solutions exist, the majority of them are tailored to the needs of a specific domain. It is also evident from literature that IoT and health domains are the most widely explored research areas on this topic.

Taking into consideration the requirement for the existence of valid consent, a web standard protocol called User-Managed Access (UMA) is proposed [9] in an attempt to enable digital applications to offer the required consent management functionalities in a domain-agnostic manner. UMA enables the individuals to perform authorisation control over their personal data access and usage in a unified manner. Nevertheless, this generic solution is lacking in terms of data access auditing, access revocation and the right to be forgotten. Leveraging blockchain technology and smart contract is an additional proposed solution [10] for efficient and effective consent management. Another domain-agnostic approach is based on blockchain technology and smart contracts [11]; however, it is more focused on data accountability and data provenance. The solution encapsulates the user's consent into a smart contract but is focused on the data subject and data controller aspects while lacking support of the data processor aspects in case where data controller and data processors are different entities.

When it comes to the health domain, multiple approaches and solutions have been proposed, as expected due to the nature of health data which are characterised as sensitive data per the GDPR. The generic consent management architecture (CMA) is proposed [12] as an extension of UMA in combination with the MyData [13] approach. It enables secure data transactions with a context-driven authorisation of multisourced data for secure access of health services maintaining personal data ownership and control. Nevertheless, the proposed architecture is still defined at a general level, and further work is still needed, while some of the utilised technologies require further development. Another approach for consent-based data sharing is also proposed via a web-based solution and video files [14]. In this approach for genomic data sharing, a single one page consent form is introduced on a web application, supplemented with online video that informs the user for the key risks and benefits of data sharing. Nevertheless, this approach does not effectively cover the complete consent management lifecycle. For mobile applications, another initiative is the participant-centred consent (PCC) toolkit [15] whose purpose is to facilitate the collection of informed consents from research participants via mobile. Researchers have also proposed several consent management solutions which are based on the blockchain technology and enable health data sharing. A domainspecific solution is proposed with a blockchain-based data sharing consent model that enables control of the individuals' health data [16]. The proposed solution offers – with the help of smart contracts – dynamic consent definition over health data with the Data Use Ontology (DUO), as well as their search and query via the Automatable Discovery and Access Matrix (ADA-M) ontology. An additional blockchain-based solution for consent management utilises smart contracts for interorganisational health data sharing [17]. It supports searching for patients in the framework that match certain criteria; however, it is based on open and public blockchain networks. Furthermore, several blockchain-based solutions have been proposed in literature [18–20] for the effective data sharing of EHRs and health data in a secured manner.

In addition to the health domain, for the IoT domain, a large number of solutions for consent management is also offered. ADVOCATE [21] platform is providing a user-centric solution that effectively implements the consent management lifecycle for secure access of personal data originating from IoT devices. The solution offers consent management and data disposal policy definition; however, it is domainspecific to the IoT ecosystem. Another solution offers the ability to set their privacy preferences of their IoT devices and safeguards that their data are only transmitted based on these preferences [22]. In this solution, blockchain is used to store and protect these preferences; however, this solution can be considered as a GDPR-compliant solution. A lightweight privacy-preserving solution for consent management is also proposed [23] with cryptographic consents being issued by the individuals. The specific proposal supports multiple security features, such as untraceability and pseudonymity, exploiting the hierarchical identity-based signature (HIBS). Nevertheless, the proposed approach partially addresses the requirements of GDPR. Additionally, several solutions are proposed for specific IoT device-type only, such as Cooperative Intelligent Transport Systems [24], smart homes [25], medical devices [26, 27] and smart watches [28].

In the finance domain, despite the fact that the requirement for consent management is crucial for financial institutions, the list of available solutions is still limited. A solution built around the double-blind data sharing on blockchain has been proposed [29]. The specific solution is focused on establishing a Know Your Customer (KYC) application through which the consents are formulated supporting the dynamic definition of the consent terms with respect to the data usage. Nevertheless, the specific solution is not supporting the dynamic definition of the purpose of the data usage in the formulated consents. A new blockchain-based data privacy framework that is combined with the nudge theory is also proposed [30]. The specific solution offers a data privacy classification method according to the characteristics of financial data, as well as a new collaborative filtering-based model and a confirmation data disclosure scheme for customer strategies based on the nudge theory. However, the specific approach requires a layered architecture for the designed application that is based on hybrid blockchain, a combination of public and private blockchain which is not usually embraced by financial institutions that operate under strict regulations.

#### **4 Methodology**

Within the INFINITECH project, the consortium partners seek to exploit the open banking opportunity, aspiring to examine how open banking and internal banking data sources can be effectively combined in order to gain useful insights and extract knowledge on the customer's patterns of behaviour from a range of data analysis methods.

To this end, the consortium partners were engaged into a series of internal roundtable discussions and brainstorming sessions, bridging the expertise within the consortium in the banking sector services and the existing regulatory framework in the banking sector with the expertise in the design and delivery in technological solutions in order to formulate a novel solution.

The scope of these interactive sessions can be grouped into the following axes:

• To define the proper framework which supports customer-centric data services that involve both data acquisition and collaborative data sharing where customers maintain complete control over the management, usage and sharing of their own banking data

As mentioned before, there is a clear need for a solution that bridges the gap between the two distinct regulations, namely, the PSD2 and GDPR regulations. The consent of the bank's customers to share their financial information is considered a hard requirement imposed by these regulations. Hence, it is imperative that banks seeking to leverage the opportunities of open banking design a data-sharing framework which is built around their customer's consent to provide access to their data to any interested party outside their organisation. Within this context, banks shall evaluate how these data are shared and how this proposition is aligned with the requirements and needs of the fintech ecosystem. The data-sharing framework shall take into consideration how these parameters are directly depicted into each consent agreement provided by their customers. As it is clearly stated in GDPR article 7 [5], the consent of the data subject (in this case, the customer of the bank) must contain, in clearly and in easily understandable terms, the information about what they are consenting and, above all, the use of data cannot go beyond what is specified in the consent agreement. Hence, the consent and its terms shall define not only which data can be shared but also when and under which conditions.

In addition to this, a crucial aspect in every data-sharing approach is the security measures that are employed in order to ensure the proper access control besides the security of data at rest. Security is the heart of any data-sharing framework. It constitutes one of the major concerns of any data provider and one of the core aspects that can lower the barriers of adoption of any data-sharing framework. In our case, the proper data access control should be based on the terms of the consent as provided by the customer of the bank. The terms of the consents should be translated into the proper data access policies which prevent unauthorised disclosure to personal data and clearly define what personal data can be accessed and by whom, as well as when and for how long can this personal data be accessed.

Finally, banks need a framework that is, on the one hand, capable of offering custodian services and, on the other hand, ensuring that customers maintain the control over the management, usage and sharing of their banking data. By placing the consent of a customer in the centre of the data-sharing framework, customers will remain in control of their data. This supports secure and controlled accessibility of the data, as the explicit consent of the customers will formulate the decision what data are accessed, who can access them and when this access is revoked. Through the use of consent, the data-sharing framework is enabling both data acquisition and collaborative data sharing to deliver added-value services, both being basic principles of PSD2, in a GDPR-compliant manner.

Towards the design of a novel consent management system that will empower the banks to leverage the PSD2 opportunities in a GDPR-compliant manner, the consortium analysed:

From the analysis of these requirements and characteristics, it became clear that with regard to the consents and their validity period, two different approaches should be supported by the consent management system. The first approach supports the consents for which the validity period is a predefined period as set in the terms of the consent. This consent type is referred as "once off" consent. The expiration of the validity is translated as automatic revocation of the consent, and a new consent is required in order for the recipient to be able to access the customer's data. An example of the cases considered for this consent type is the customer's consent for sharing of KYC data between two financial institutions (banks) for the scenario of a loan origination/application or an account opening. The second approach supports the consents that have an infinite validity period, referred to as permanent (or "regular") consents. This consent type is only invalidated in the case where the consent is withdrawn. An example of the cases considered for this consent type is the peer-to-peer (including person-to-person, person-to-organisation, organisationto-organisation cases) customer data-sharing consent in which a customer utilises an interface of a mobile application to select a set of its specific customer data (such as specific accounts, specific transactions or alerts) that they would like to share with an individual person or a group of persons.

In addition to the different consent types, the analysis of these requirements and characteristics has also driven the design of the corresponding use cases that should be implemented in order to ensure that all requirements are effectively addressed and that the designed consent management system will bring the added value to the banking sector's stakeholders. The defined use cases cover the complete lifecycle of consent management system, starting from the registration of the relevant users, the formulation of the consent and its storage to the retrieval of the stored consent in order to formulate the data access control decision.

The key stakeholders which are involved and are interacting with the proposed solution are:

The consortium analysed also the latest state-of-the-art technologies which can be leveraged for the implementation of the aspired solution. The core aspect of any data-sharing framework is the employed security. However, traditional technologies have failed to become a key enabler of trust, due to multiple security/data tampering incidents as it is evidenced by the recent large-scale personal data breaches around the world [31]. Nevertheless, blockchain technology and its latest advancements appear as a compelling technology to overcome the underlying challenges around trusted data sharing by exploiting its attractive features such as immutability and decentralisation.

In the following paragraphs, the different use cases which are supported by the aspired solution are presented in detail.

#### *4.1 User's Registration*

The first step includes the registration of the different users in the consent management system. Different roles are assigned to the users based on the role in the consent formulation process. In order for a customer to be able to receive consent requests, it is mandatory that they are registered in the consent management system, creating their profile that will be used in order to receive consent requests. Their profile includes the minimum customer discovery and communication details required in order to receive a consent request, while it ensured that this information and any consequent private information are not disclosed to any interested party. In the same manner, the third-party provider registers in the consent management system in order to be able to initiate consent requests to a customer of the bank. The users of the bank, who are considered as the administrators of the consent management system, are already preregistered so as to facilitate the whole execution (Fig. 8.4).

#### *4.2 Customer Receives a Request to Provide New Consent for Sharing His/Her Customer Data*

The consent formulation is initiated from the third-party provider. In particular, the third-party provider issues a new consent request to the consent management system indicating the specific customer that accesses to his/her is requested along with the details of the requested customer data (i.e. specific data, period of usage). As a first step, the initial request is received and reviewed by the bank before it is preapproved and received by the customer. Upon this approval, the customer receives a notification with all the required information for the consent request in a proper way that it allows them to review the request (Fig. 8.5).

**Fig. 8.5** New consent request

#### *4.3 Definition of the Consent*

In this step of the process, the customer is able to review, and possibly alter, the details and conditions of the consent request before he/she formulates his/her decision to provide the consent or deny the access to his/her personal data. In the case of approval, the final terms of the consent are defined by the customer, and it is submitted to the consent management system. The third-party provider is then informed and can approve the final terms of the consent, as set by the customer, or abandon the consent request. In the case of denial from the customer on the initial request, the request is blocked, and the third-party provider is informed for the completion of the process (Fig. 8.6).

**Fig. 8.6** Definition of the consent

**Fig. 8.7** Digital signing of the consent

#### *4.4 Signing of the Consent by the Interested Parties*

Once the terms of the consent have been formulated, the consent management system provides the formulated consent form to both parties (the customer and the third-party provider) in order to be digitally signed. The consent management system collects the digitally signed consent forms from both parties. At this point, the consent status is now set as "active" (Fig. 8.7).

#### *4.5 Consent Form Is Stored in the Consent Management System*

Once the digitally signed consent form is available, the consent management system is able to store this information in order to be used in the access control formulation process. In the case of the "once off" consent, where a specific validity period is defined, the consent management system is internally handling the validation of consent time period by creating and monitoring the specific timer in order to perform the validation of consent time period.

#### *4.6 Consent Update or Withdrawal*

In accordance with the GDPR regulation, the formulated consent form can be updated or withdrawn at any time by the customer side. On the one hand, the terms of the consent can be updated following the previously described steps (from the definition of the consent till the consent storage) in order to formulate the updated consent and maintain the consent status "active". In the case of the "once off" consent, the associated timer is restarted when the consent is updated. On the other hand, the consent can be withdrawn, which is internally translated into the update of the consent status to "withdrawn". In this case, if the withdrawn consent is a "once off" consent, the associated timer is stopped. For both cases, the consent management system ensures that the complete history of each consent is maintained for later usage (Figs. 8.8 and 8.9).

**Fig. 8.8** Consent update

**Fig. 8.9** Consent withdrawal

**Fig. 8.10** Consent expiration

#### *4.7 Expiration of the Validity Period*

In the case of the "once off" consent, the validity period is set to a predefined time period. When consent is formulated, the consent management system creates and monitors the specific timer in order to perform the validation of consent time period. The consent management system informs the customer before the validity period of a "once off" consent expires in case he/she would like to initiate an update before the validity period expires. Once this timer is expired, the consent management system sets the status of the specific consent to "expired" (Fig. 8.10).

#### *4.8 Access Control Based on the Consent Forms*

The key aspect of the aspired data-sharing framework is the efficient and robust access control mechanism with the consent management system at its core. During each data access request to underlying data management system, the consent management system is consulted in order to validate the consent status between the requesting party and the customer whose data are requested. The data access control decision is formulated by the existing consents and their status, as well as the underlying terms of each consent (Fig. 8.11).

#### *4.9 Retrieve Complete History of Consents*

Another key aspect of the proposed solution is that the customer is able to be constantly informed of all the consents that are given to each specific recipient, as well as of the complete history of these consents. Furthermore, for each consent, all the different versions of the provided consent can be retrieved besides the latest

**Fig. 8.11** Access control based on consents

**Fig. 8.12** Complete history of consents retrieval

one. In this sense, the customer shall be able to retrieve at any time their consent history per specific stakeholder or for all stakeholders, while the bank shall be to retrieve all the consents given by the customers for a stakeholder (Fig. 8.12).

#### **5 The INFINITECH Consent Management System**

The designed solution aims to enable the collaborative data sharing between customers, banks and other organisations, in order to facilitate the application of advanced analytics over particular datasets and intelligent support tools for better understanding of customers and their financial relationships among others, which is considered critically important in today's financial markets.

As explained, the development of such intelligence support tools is highly dependent on the customer's permission to share data. Hence, the requirement for a trusted and secure sharing mechanism of customer consent arises in order enable the development of new customer services that solve business problems, such as improved KYC processes and consequently AML, credit scoring and fraud detection services. A robust and solid consent management system that will support the granular permission consent is considered as a key enabler of trust which is vital to maximise data sharing and ensure customers are comfortable with sharing data.

Blockchain is a continuously growing, distributed, shared ledger of uniquely identified, linked transaction records organised in blocks that are sealed cryptographically with a digital fingerprint generated by a hashing function and are sequentially chained through a reference to their hash value [32]. In general, the blockchain technology is composed of multiple technologies related to cryptography, peer-to-peer networks, identity management, network security, transaction processing, (distributed) algorithms and more, which are all leveraged in order to formulate an immutable transaction ledger which is maintained by a distributed network of peer nodes formulating the blockchain network. The key characteristics of the blockchain technology are that it is decentralised, immutable, transparent, autonomous and open-sourced [33].

The blockchain technology has a set of key concepts that includes (a) the distributed ledger that is composed by blocks containing the transaction records, (b) the consensus model that is utilised in order to validate a transaction and to keep the ledger transactions synchronised across the blockchain network and (c) the smart contracts or chaincodes which are the trusted distributed applications that are deployed within the nodes of the blockchain network and encapsulate the business logic of the blockchain applications. The blockchain implementations can be characterised and grouped into two major high-level categories based on the permission model applied on the blockchain network, the *permissionless blockchain networks* which are open and publicly and anonymously accessible blockchain networks and the *permissioned blockchain networks* where only authorised users are able to maintain, read and access the underlying blockchain.

Our solution exploits the benefits of blockchain technology and specifically the permissioned blockchain that is considered as the appropriate candidate solution due to its security and trust characteristics. It is built around a blockchain application that implements a decentralised and robust consent management mechanism which facilitates the sharing of the customers' consent to exchange and utilise their customer data across different banking institutions. Built on top of the core offerings of the blockchain technology, namely, its decentralised nature and immutability, as well as the impossibility of ledger falsification, our approach ensures the integrity of customer data processing consents and their immutable versioning control through the use of the appropriate blockchain infrastructure. Hence, the blockchain-enabled consent management mechanism enables the financial institutions to effectively manage and share their customers' consents in a transparent and unambiguous manner.

Its key differentiating points, from the financial institutions' perspective, is the ability to inform the customer at any time:

On the other hand, its key differentiating points, from the customer's perspective, are enables them to:

Our solution exploits the benefits of the blockchain technology in order to assure the integrity of the formulated consents with the utilisation of its cryptographic techniques in combination with the usage of digital signatures. In addition to this, the formulated consent and their complete update history are stored in a secure and trusted manner within the blockchain infrastructure. Hence, the blockchain technology is leveraged to store this sensitive information, to apply immutable versioning control and to enable the retrieval of this information in an indisputable manner. With the use of blockchain technology, both the financial institutions and their customers are able to retrieve the latest untampered consent information, as well as any previous versions of the consent.

The solution is designed to hold the role of the middleware application between the data management applications of the financial institution, in which the secured APIs are available, and the third-party providers. Hence, the designed consent management system does not save or distribute any customer data. Being a middleware application, it provides the means to formulate a consent agreement and utilise the existing consent agreements to formulate an access control decision that should be propagated to the underlying data management applications through which the customer data are actually shared to the third-party provider.

The high-level architecture of our solution is depicted in Fig. 8.13. Our solution is composed by a set of core components, namely, the *consent management system*, the *file storage* and the *blockchain infrastructure*, which are effectively interacting via well-defined RESTful APIs. Following the analysis performed in the previous section, the main stakeholders that are involved are the internal financial institution (namely, the bank), the customer of internal financial institution (namely, customer of the bank) and the external financial institution or peer (namely, third-party provider).

The consent management system constitutes the mediator between the stakeholders and the underlying blockchain infrastructure. It receives and effectively handles the requests for the consent formulation from the external financial institution or peer to the internal financial institution and consequently the customer of the internal institution. The consent management system implements the complete consent

**Fig. 8.13** High-level architecture of the blockchain-enabled consent management system

management lifecycle by leveraging the functionalities offered by the underlying blockchain infrastructure. To facilitate the interactions of the stakeholders with the consent management system, it offers both a Web application and a mobile application. The Web application is offered to all the involved stakeholders, in order to interact with the system on each step of the process. The Web application is composed of the core backend component that provides all the backend operations of the system, such as the processing of the input data, the retrieval of the requested information and all the interactions with the underlying blockchain infrastructure. The core backend component is supported by a local database instance in which only the operational data is maintained, such as the user management data or the data of a consent that is still under formulation. The graphical interface of the consent management system which provides all the functionalities of the Web application to the stakeholders is offered by the frontend component. The frontend component interacts with the backend via a set of APIs provided by the backend in order to perform all the requested activities by the stakeholders. In addition to the Web application, the customers of the bank are offered with the mobile application through which they are able to perform all the required operations in the same manner as they are offered by the web application.

The file storage component is providing the file repository that stores the finalised consent forms in digital format once all the steps of the process have been completed upon the acceptance of the details and terms of the consent by all the involved parties. It provides the required interface to the consent management system in order to store or retrieve the formulated consent forms upon need.

The blockchain infrastructure is providing the permissioned blockchain network of the solution. The designed blockchain network constitutes the cornerstone of the proposed solution and is based on Hyperledger Fabric1. In detail, the designed blockchain network is formulated by two peers (P1 and P2), which are owned by

<sup>1</sup> Hyperledger Fabric, https://www.hyperledger.org/use/fabric

**Fig. 8.14** Blockchain network of the solution

one single organisation (FO1) and are hosting their own copy of the distributed ledger (L1). Both peers are operating in one channel (C1) in which the smart contract/chaincode (S1) is deployed. In the designed blockchain network, a single order (O), which formulates the transactions into blocks and ensures the delivery of the blocks to the peers of the channel, is deployed. In this blockchain network, the consent management system is deployed on an external node which also holds the single certificate authority (CA1) that is utilised for the secure interaction with channel C1. Channel C1 is regulated by the underlying channel policy CP1 that ensures the isolation of the channel from external peers and other channels. Finally, the complete blockchain network is regulated by the network policy NP1 that is applied across the whole network with permissions that are determined prior to the network creation by organisation FO1. The interaction between the blockchain infrastructure and the backend of the consent management system is realised through the well-defined APIs provided by the smart contract/chaincode S1 (Fig. 8.14).

As explained before, the smart contract/chaincode is the trusted distributed application that encapsulates the business logic of solution. It contains the definition of the business objects for which the current and historical state will be maintained and updated through a set of functions which are also included in the chaincode. To facilitate the desired operations, the data schema that defines the core business objects has been defined, and it is depicted in Fig. 8.15. The definition was based on the consent receipt specification that is proposed by the Kantara Initiative [34] which has been adapted in terms of terminology in order to be aligned with the EU GDPR

**Fig. 8.15** Data schema

legislation. The designed data schema covers all the aspects of a consent, spanning from the generic information, such as the version number, the unique identifier and the timestamp of creation or modification, to more specific information such as the data subject information, the data controllers' information, the applied privacy policy, the service or group of services for which personal data are collected and the relevant purposes and of course the exact personal and/or sensitive data that are collected. Additionally, key information for the consent management lifecycle execution is included, such as the latest status of the consent, the validity type (permanent or once off) and the validity period in the case of once off consent**.**

The chaincode is directly built on top of the presented data schema with a set of functions which are conceptually organised into the following groups of functions:

#### *5.1 Implemented Methods*

In the following paragraphs, the implemented methods of the designed solution are presented. The basis of the implementation was the initial use cases, as well as the collected requirements, which were described in the methodology section. The implemented methods effectively cover all the aspects of the consent management lifecycle. On each method, the different interactions of the various components of the solution, as well as the interactions of the stakeholders with these components, are highlighted.

#### **5.1.1 Definition of Consent**

The formulation of the consent involves several steps from the initial consent request, as initiated by the third-party provider to the consent management system, to the actual consent formulation with the exact terms and validity period and its storage within the blockchain infrastructure. At first, the received consent request is pre-approved by the bank through the consent management system via the Web application. Once it is pre-approved by the bank again via the Web application, the request is received by the customer through the consent management system either via the Web application or via the mobile application. The customer reviews and defines the final terms of the consent and submits the digitally signed consent in the consent management system. The formulated consent is received by the thirdparty provider and is also digitally signed. Once the consent is digitally signed by both parties, the consent management system interacts with the blockchain infrastructure via the provided API, and a new transaction is created and inserted into the blockchain via the deployed chaincode with status "active". In the case of the "once off" consent, where a specific validity period is defined, the consent management system is internally handling the validation of consent time period

**Fig. 8.16** The definition of consent method

by creating and monitoring the specific timer in order to perform the validation of consent time period (Fig. 8.16).

#### **5.1.2 Consent Update or Withdrawal**

The customer is able to retrieve and update or withdraw a consent at any time. At first, the selected consent is retrieved from the consent management system by interacting with the API of the blockchain infrastructure. At this point, the customer is presented with the existing terms of the consent and can modify its terms or withdraw it. In the consent update case, the updated terms are digitally signed by the customer and received by the consent management system. The involved thirdparty provider is notified and digitally signs the updated consent. Once the updated consent is digitally signed by both parties, the consent management system interacts again with the blockchain infrastructure via the provided API in order to initiate a new transaction and insert it into the blockchain via the deployed chaincode while the status remains "active". In the case of the "once off" consent, the associated timer is restarted when the consent is updated.

In the case of withdrawal, the consent management system interacts with the blockchain infrastructure via the provided API in order to "invalidate" the existing consent with the new transaction which sets the new status to "withdrawn". In the case of the "once off" consent, the consent management system also stops the respective timer. Every update or withdrawal of an existing consent introduces in

**Fig. 8.17** The consent update method

**Fig. 8.18** The consent withdrawal method

a new version of the specific consent is appended into the complete history that is maintained within the blockchain infrastructure (Figs. 8.17 and 8.18).

#### **5.1.3 Consent Expiration**

The "once off" consent has a validity period as defined in the terms of the consent. Once a "once off" consent has been formulated, the consent management system creates and monitors the specific timer. At the point where the timer is about to expire, the consent management system informs the respective customer to initiate an update of the consent that will result in the timer renewal based on the terms of the updated consent. In the case where the timer expires, the consent management

**Fig. 8.19** The consent expiration method

system interacts with the blockchain infrastructure via the provided API in order to "invalidate" the existing consent with the new transaction which sets the new status to "expired" (Fig. 8.19).

#### **5.1.4 Access Control**

Once a data access request is received by the bank on their data management system from a third-party provider, the bank consults the consent management system via the Web application in order to verify and approve the specific data access request based on the terms and current status of the consent between the third party and the customer whose data are requested. To this end, the consent management system receives the information of the involved parties from the bank and initiates a query into the blockchain infrastructure via the provided API in order to retrieve the latest consent between them.

In the case where the consent status is "active" and the terms are met, the data access request can be approved, while in the case where the status is set to "withdrawn" and "expired" or a consent between the involved parties does not exist, the data access request should be denied (Fig. 8.20).

#### **5.1.5 Complete History of Consents**

The consent management system offers the retrieval and display of the latest consent information, as well as the complete history of the existing consents, at any point. The customer can retrieve the list of consents in a user-friendly way via the Web application or the mobile application. Depending on his/her selection, the customer can select and view a specific consent that has been provided to a third-party provider, as well as the complete of consents that he/she has provided to any third party provider. On the other hand, the bank is also able to retrieve a specific consent between a specific customer and a third-party provider, all the consents that a third

**Fig. 8.20** The access control based on consent method

**Fig. 8.21** The retrieval of the complete history of consent method

party has received by any of its customers and finally the complete list of consents provided by all his/her customer to any third-party provider.

To achieve this, the consent management system translates the request initiated by either the customer or the bank into a query with specific parameters which is executed into the blockchain infrastructure via the provided API in order to retrieve the requested information and display it to the requestor (Fig. 8.21).

#### **6 Conclusions**

The designed and implemented solution of the blockchain-empowered consent management system enables the sharing of customers' consent, thus facilitating the exchange and the utilisation of customer data, across different banking institutions. It enables the exploitation of the open banking opportunities towards the realisation of novel financial services through the collaborative data sharing in a PSD2- and GDPR-compliant manner.

Building directly on top of the key offering of the blockchain technologies that address the underlying challenges around trusted data sharing, the proposed consent management system effectively supports the employment of a trusted and secure sharing mechanism that is based on the customer consent to share his/her data. The provided solution constitutes a robust and solid consent management system that can act as a key enabler of trust of an ecosystem of innovative financial services in which customers are comfortable with sharing data since they are maintaining complete control over the management, usage and sharing of their own banking data.

**Acknowledgments** The research leading to the results presented in this chapter has received funding from the European Union's funded Project INFINITECH under Grant Agreement No. 856632.

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Part III Applications of Big Data and AI in Digital Finance**

## **Chapter 9 Addressing Risk Assessments in Real-Time for Forex Trading**

**Georgios Fatouros, Georgios Makridis, John Soldatos, Petra Ristau, and Vittorio Monferrino**

#### **1 Introduction**

Risk assessment refers to the processes required to quantify the likelihood and the scale of loss on an asset, loan, or investment. In investment banking, risk estimation and monitoring are of great importance as invalid models and assumptions could cause substantial capital losses. This was the case for several financial institutions in the 2008 financial crisis and the COVID-19 pandemic as well.

Despite the recent hype for digitization in the financial industry, many companies lack accurate real-time risk assessment capabilities. This is due to the current practice of updating portfolios' risk estimation only once in a day, usually overnight in a batch mode, which is insufficient for trading in higher frequency markets like Forex (FX) [1]. Surprisingly, this is still an open challenge for various banks

G. Fatouros (-)

INNOV-ACTS Limited, Nicosia, Cyprus e-mail<EMAIL_ADDRESS>
G. Makridis University of Piraeus, Piraeus, Greece e-mail<EMAIL_ADDRESS>
J. Soldatos INNOV-ACTS LIMITED, Nicosia, Cyprus

University of Glasgow, Glasgow, UK e-mail<EMAIL_ADDRESS>
P. Ristau JRC Capital Management Consultancy & Research GmbH, Berlin, Germany e-mail<EMAIL_ADDRESS>
V. Monferrino GFT Technologies, Genova, Liguria, Italy e-mail<EMAIL_ADDRESS>
**Fig. 9.1** Data pipeline

and asset management firms, where the risk evaluation is outsourced to third party consulting companies, which are in charge of risk monitoring and regulatory compliance with for an agreed investment strategy. Furthermore, in cases where a real-time overview of the trading positions might be provided, the data needed for computing the risk measures and other relevant indicators are updated only once a day. As a result, the risk exposure from intra-day price fluctuations is not monitored properly, which may lead to both trading and investment inefficiencies [2].

When it comes to HFT, the speed of financial risk calculations (derived from algorithm complexity) is the main barrier towards real-time risk monitoring [3]. Traditionally, there is a trade-off between speed and accuracy of the financial risk calculations, as the less computationally intensive risk models, which might be able to yield their assessments near-instantly, are not considered sufficiently reliable in the financial domain [4]. In addition, real-time risk monitoring is also a regulatory requirement [5] that can be met if risk information is updated timely.

This Chapter introduces a new tool, named *AI-Risk-Assessment*, which represents a potential key offering within this space. More specifically it addresses the challenge of real-time risk estimation for Forex Trading, in the scope of the European Union's funded INFINITECH project, under grant agreement no 856632. The primary added-value provided by this tool can be divided into three pillars (components): (1) risk models; (2) real-time management; and (3) pre-trade analysis, which are each described in the following sections along with the tool's core architecture that allows seamless interaction between these components. The proposed tool follows a containerized micro-service design, consisting of different components that communicate with each other in an asynchronous manner, forming a real-time data processing pipeline, illustrated in Fig. 9.1.

#### **2 Portfolio Risk**

Portfolio risk is the likelihood that the combination of assets that compromise a portfolio fails to meet financial objectives. Each investment within a portfolio carries its own risk, with higher potential return typically meaning higher risk. Towards quantifying this risk, each asset's future performance (i.e., returns) should be predicted. To this end, various models and theories regarding the nature of the financial time-series have been proposed. Most of these models assume that the underlying assets' returns follow a known distribution (e.g., Gaussian distribution) and try to estimate its parameters (e.g., mean and variance of asset's returns) based on their historical performance. Moreover, the portfolio risk depends on the weighted combination of the constituent assets and their correlation as well. For instance, according to Modern Portfolio Theory (MPT) [6], which assumes Normal distribution for the financial returns, one could obtain a portfolio's risk performance under a known probability portfolio risk calculating the distribution parameters (*μp, σ*<sup>2</sup> *<sup>p</sup>*) from Eqs. 9.1 and 9.2.

$$
\mu\_p = E(R\_p) = \sum\_i w\_i E(R\_i) \tag{9.1}
$$

where *Rp* is the return on the portfolio, *Ri* is the return on asset *i*, and *wi* is the weighting of component asset *i* (that is, the proportion of asset "*i*" in the portfolio).

$$
\sigma\_p^2 = \sum\_l w\_l^2 \sigma\_l^2 + \sum\_l \sum\_{i \neq j} w\_i w\_j \sigma\_l \sigma\_j \rho\_{lj} \tag{9.2}
$$

where *σ* is the (sample) standard deviation of the periodic returns on an asset, and *ρij* is the correlation coefficient between the returns on assets *i* and *j* .

Furthermore, the Efficient Market Hypothesis implies that in highly efficient markets, such as the foreign exchange market [7], a financial instrument's value illustrates all the available information about that instrument [8]. As a result, the utilized risk models should be fed with the latest market data to provide valid portfolio risk estimations. The need for continuous updates in the risk estimates is more evident in HFT where traders usually manage numerous portfolios with different strategies (i.e., holding periods, composition, risk profile, and volume) that are mainly based on algorithmic trading [9]. Thus, there is a need for a risk assessment tool that will handle and process a large volume of market and trading data and, based on them to provide risk estimates. The latter should be updated in (near) real-time capturing intra-day volatility in financial markets that could cause both loss of liquidity and capital.

#### **3 Risk Models**

The proposed application leverages two standard risk metrics which are broadly adopted not only in risk management but also in financial control, financial reporting and in computing the regulatory capital of financial institutions, namely: Value at Risk (VaR); and Expected Shortfall (ES), which we discuss in more detail below.

#### *3.1 Value at Risk*

The first underlying risk metric is the Value at Risk (VaR), which measures the maximum potential loss of an investment under a confidence probability within a specific time period (typically a single day). There are three fundamentally different methods for calculating VaR; parametric, non-parametric, and semi-parametric [10]. *AI-Risk-Assessment* offers four VaR models based on these methods to enhance the risk monitoring process.

**Parametric** The most popular parametric VaR method (and what is implemented in the tool) is based on the Variance-Covariance (VC) approach [10]. The key assumption here is that the portfolio returns follow the normal distribution, hence the variance of the market data is considered known. Also, a proper history-window (e.g., 250 samples) should be defined, and the variance-covariance matrix of returns should be calculated. The VaR is then obtained by Eq. 9.3.

$$VaR^{\alpha} = z\_{1-\alpha} \sqrt{w^T \Sigma w} \tag{9.3}$$

where *z*, *α*, *w*, are the standard score, the confidence probability of VaR prediction, the portfolio assets weights, and the covariance matrix of the portfolio assets, respectively.

**Non-parametric** From the non-parametric category, the well-known Historical Simulation (HS) VaR method [11] was selected. In this approach, the historical portfolio returns are taken into account in the VaR calculation along with a selected confidence level. Under the Historical Simulation approach, the first step is to sort the historical portfolio returns for a given time-window. Then the VaR can be derived by selecting the n-worst portfolio performance that corresponds to the required confidence probability ((100-n)%) of the VaR estimation. For instance, to calculate the 1-day 95% VaR for an portfolio using 100 days of data, the 95% VaR corresponds to the best performing of the worst 5% of portfolio returns. Consequently, HS is a computationally effective risk model with acceptable returns when large historical windows are taken into account. However, the latter results in high VaR estimations that restrict the accepted investment strategies.

**Semi-parametric** As for the semi-parametric VaR method, a Monte Carlo (MC) [12] model was developed. In the context of the proposed solution, the mean and the standard deviation of returns are calculated from the available historical data and then these values are used to produce MC random samples from the Gaussian distribution. To calculate VaR, this process draws the distribution of portfolio returns for the next time step (see Eq. 9.4).

$$VaR^{\alpha} = q\_{\mathbb{I}-\alpha} \tag{9.4}$$

where *qα* is the *α*-th percentile of the continuously compounded return.

**Semi-parametric with RNNs** In addition, this tool introduces a novel semiparametric VaR model [13] combining deep neural networks with MC simulations. In this approach, the parameters of the returns distribution are initially estimated by a Recurrent Neural Network (RNN) [14] based model, with the network output being used to predict all possible future returns in a MC fashion. VaR based on the input time-series can then be obtained by (Eq. 9.4). The RNN-model in question, named DeepAR, follows the methodology proposed by Salinas et al. [15] and is provided by GluonTS Python library<sup>1</sup> for deep-learning-based time-series modeling [16]. This model is able to capture nonlinear dependencies of the input time-series, such as seasonality, resulting in consistent quantile estimates. Moreover, DeepAR can be fed with several input time-series simultaneously, enabling cross-learning from their historical behavior jointly. As a result, changes in the dynamics of the one time-series may affect the predicted distributions of the other time-series.

#### *3.2 Expected Shortfall*

Alternatively, risk assessment can also be performed via Expected Shortfall (ES), also known as Conditional Value at Risk (C-VaR) [17], which is a conservative risk measure calculating portfolio loss with the assumption that this loss is higher than the estimated VaR as illustrated in Fig. 9.2. ES was originally suggested as a practicable and sound alternative to VaR, featuring various useful properties such as sub-additivity that are missing from VaR [18]. This has led many financial institutions to use it as a risk measure internally [19]. Expected Shortfall can be calculated via a parametric equation shown below (see Eq. 9.5) [20].

$$ES\_p^\alpha = -\mu\_p + \sigma\_p \frac{\phi\left(\Phi^{-1}\left(\alpha\right)\right)}{a} \tag{9.5}$$

where *φ (x)* = <sup>√</sup> 1 <sup>2</sup>*<sup>π</sup> <sup>e</sup>* <sup>−</sup>*x*<sup>2</sup> <sup>2</sup> , *(x)* is the standard normal c.d.f., so −<sup>1</sup> *(a)* is the standard normal quantile, and *μp*, *σp* the mean and standard deviation of portfolio returns, respectively.

The user interface of the application offers a page for analysis and back-testing of the provided risk assessments (Fig. 9.3). In this way the user is able to compare the true performance of various VAR/ES models in both 95 and 99% confidence level.

<sup>1</sup> https://ts.gluon.ai/.

**Fig. 9.2** Comparison between value at risk and expected shortfall

**Fig. 9.3** Analysis and Back-Testing application page: The green line is the actual portfolio returns. The dense lines are for 95% confidence probability, while the dash lines for 99%

#### **4 Real-Time Management**

Delivering the aforementioned risk assessments while leveraging the latest available data is a challenging task, as FX market prices are updated at inconsistent high frequency time intervals (e.g., 1–8 seconds). Moreover, even small price fluctuations can have a significant impact on portfolios' value, particularly in cases where an high-risk/return investment strategy is being employed. Thus, additional technologies are required that can provide seamless data management with online analytical processing capabilities.

Within Infinitech, this is accomplished via a new data management platform referred to as the InfiniSTORE. The InfiniSTORE extends a state-of-the-art database platform2 and incorporates two important innovations that enable real-time analytical processing over operational data. Firstly, its hybrid transactional and analytical processing engine allows ingesting data at very high rates and perform analytics on the same dataset in parallel, without having to migrate historical data into a data temporary warehouse (which is both slow and by its nature batch-orientated). Secondly, its *online aggregates* function enables efficient execution of aggregate operations, which improves the response time for queries by an order of magnitude. Additional details can be found in Chap. 2.

The *AI-Risk-Assessment* requires the datastore to firstly store the raw input ticker data along with the risk estimations and secondly to enable online data aggregation and integrated query processing for data in-flight and at rest. These requirements are enabled by the InfiniSTORE and underlying LeanXcale database via its dual interface, allowing it to ingest operational data at any rate and also to perform analytical query processing on the live data that have been added in parallel.

In order for the application to achieve its main objective of measuring intra-day risk in a timely fashion (e.g., updating VaR/ES approximately every 15-minutes), the input tick data (with rate 1–8 second per instrument) should be resampled for the required frequency. LeanXcale provides online aggregates that enables realtime data analysis in a declarative way with standard SQL statements. In this way, only the definition of the required aggregate operations, such as average price per FX instrument per quarter-hour, is required and the result of the execution is precalculated on the fly, ensuring consistent transactional semantics. As a result, the typically long-lasting query can be transformed into a very light operation that requires the read-access to a single value, thus removing the need to scan the whole dataset.

Additionally, an interface between the data provider (e.g., a bank's data API) and the datastore is required to handle the real-time data streams ingested. As these microservices need to communicate asynchronously ensuring high availability and fault-tolerance, the most popular intermediate is the use of data queues [21]. Apache Kafka [22] is the most dominant solutions when it comes to data queues [23]. Using Kafka, external components can send data feeds to a specified queue, by subscribing to a specific topic and then sending the data in a common format. Infinitech provides a pre-built container image that contains a Kafka data queue, while the datastore has a built-in Kafka connector which enables interconnection between the LeanXcale datastore and Kafka. The data injection process along with the online data aggregation are illustrated in Fig. 9.4.

<sup>2</sup> https://www.leanxcale.com/.

**Fig. 9.4** Data injection and online aggregation

#### **5 Pre-trade Analysis**

Besides risk monitoring, traders, asset, and risk managers are seeking tools that enable pre-trade analysis. Such a tool could serve as a trader's digital twin, allowing the filtering of potential trading positions based on their risk. In addition, this feature provides direct information to the trader about the change in the risk of his portfolio in case the trader approves the simulated position.

The *AI-Risk-Assessment* incorporates another important feature which is based on what-if-analysis focusing on pre-trade risk assessment. In this way, if a trader sees an opportunity for a trade, they may enter this trade into the provided application (platform) and request a calculation of the risk measures with this new trade added to the portfolio as if it were done already. The pre-trade analysis is a useful tool for traders in order to understand how a potential new trade position may affect the portfolio and its risk Fig. 9.5.

However, in order to enable this feature, the developed risk models should be optimized to yield results instantly using the latest input data (i.e., asset prices). To this end, portfolio risk estimation follows a two-step procedure. Initially, both VaR and ES are calculated for each portfolio instrument separately as a univariate timeseries based on the latest market prices available from the real-time management tools described in Sect. 4. It is noted that these calculations are performed under the hood and are updated as new data is made available. The second step is related to the user inputs. For instance, when the user is experimenting with new trading positions, the weight of each asset is obtained, with univariate risk assessments being used to calculate the portfolio VaR/ES. This procedure can be completed very quickly as it requires only simple matrix algebra, as illustrated in Eq. 9.6 [10].

#### 9 Addressing Risk Assessments in Real-Time for Forex Trading 167

**Fig. 9.5** Pre-Trade analysis application page: The user enters (left) the desirable risk parameters (i.e. confidence level and time horizon) along with the positions on the financial instruments of the portfolio to be simulated. The application updates its risk estimations (right) on the fly

$$VaR\_p = \sqrt{VRV^T} \tag{9.6}$$

where *V* is the vector of VaR estimates per instrument,

$$V = [w\_1 VaR\_1, w\_2 VaR\_2, \dots, w\_n VaR\_n] \tag{9.7}$$

and *R* is the correlation matrix,

$$\mathcal{R} = \begin{bmatrix} 1 & \rho\_{1,2} & \dots \rho\_{1,j} \\ \rho\_{2,1} & 1 & \dots \rho\_{2,j} \\ \vdots & \vdots & \ddots & \vdots \\ \rho\_{l,1} & \rho\_{l,2} & \dots & 1 \end{bmatrix} \tag{9.8}$$

where *ρ*1*,*<sup>2</sup> is the correlation estimate between instruments' 1 and 2 returns. This process, described by Algorithm 1 also applies to portfolio ES calculation.

#### **6 Architecture**

This section focuses on technical details regarding the deployment of the proposed application in a production environment following the reference architecture described in Chap. 1. Each component is deployed as a docker container in an automated manner, using the Kubernetes container orchestration system. In this way, all software artefacts of the integrated solution are given a common namespace, and they can interact with each other inside the namespace as a sandbox. This deployment method allows for portability of the sandbox, as it can be deployed in different infrastructures.

#### **Algorithm 1** Portfolio risk estimation

	-

**Fig. 9.6** Application deployment scheme

A high-level overview of the Infinitech implementation is illustrated in Fig. 9.6. Each component runs within a Pod, which is the smallest compute unit within Kubernetes. A Pod encapsulates one (or more) containers, has its own storage resources, a unique network IP, access port and options related to how the container should run. This setup enables the auto-scaling of the underlying services according to the volume of external requests. For instance, in case of influx of clients (i.e., traders) requiring risk assessment of their portfolios, the corresponding (red) Pod will automatically scale-out as needed.

It is also noted that as each service is accessed via a REST API, they are able to communicate as required by client requests, while additional features (e.g., financial news sentiment analysis) could be added as a separate component without requiring modification to the existing services.

#### **7 Summary**

This chapter addresses one of the major challenges in the financial sector which is real-time risk assessment. In particular, the chapter covers risk models implemented within Infinitech and their integration with associated technological tooling to enable processing of large amounts of input data. The provided Infinitech application leverages several well-established risk models, in addition to introducing a new one based on deep learning techniques. These assessments can be updated in (near) real-time using streaming financial data, and used to drive what-if analysis for portfolio modification. The Infinitech solution can be a valuable asset for practitioners in high frequency trading where consistent risk assessment is required in real-time. It is worth mentioning that the underlying architecture can also be used with different risk models, optimized for different types of financial portfolios, while additional features could be integrated as microservices following Infinitech reference architecture.

**Acknowledgments** The research leading to the results presented in this Chapter has received funding from the European Union's funded Project INFINITECH under grant agreement no 856632.

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Chapter 10 Next-Generation Personalized Investment Recommendations**

**Richard McCreadie, Konstantinos Perakis, Maanasa Srikrishna, Nikolaos Droukas, Stamatis Pitsios, Georgia Prokopaki, Eleni Perdikouri, Craig Macdonald, and Iadh Ounis**

#### **1 Introduction to Investment Recommendation**

In developed nations, there is growing concern that not enough people are saving for their later life and/or retirement. For instance, 35% (18.4 million) of the UK adult population reported they do not have a pension, and 43% admit they do not know how much they will need for retirement according to a study by Finder.1 One of the most stable ways to save effectively is through long-term investment in a broad portfolio of financial assets [13]. However, for the average citizen, investing is a difficult, time consuming and risky proposition, leading to few doing so. Hence, there is significant interest in technologies that can help lower the barriers to entry to investment for the average citizen.

Meanwhile, recent advances in Big Data and AI technologies have made the idea of an automated personal assistant that can analyse financial markets and recommend sound investments seem possible. Indeed, there are already a range of online services that will provide financial investment recommendations for a fee.

R. McCreadie (-) · M. Srikrishna · C. Macdonald · I. Ounis University of Glasgow, Glasgow, Scotland e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
K. Perakis · S. Pitsios UBITECH, Chalandri, Greece e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
N. Droukas · G. Prokopaki · E. Perdikouri National Bank of Greece, Athens, Greece e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<sup>1</sup> https://www.finder.com/uk/pension-statistics.

Manual financial investment advice services such as Seeking Alpha2 and Zacks Investment Research3 have existed for over a decade, while new automated 'Robo-Advisors' like SoFi Automated Investing4 and Ellevest5 have been gaining traction over the last few years. The primary advantages of such automatic systems are that they are more accessible and scalable than other forms of investment advice, while also being flexible with regard to how 'hands-on' the investor wants to be with their portfolio [35]. Hence, such systems are seen as one viable solution to providing personalized financial recommendations for the public.

The European Commission's H2020 INFINITECH project is investigating these technologies within its 'Automated, Personalized, Investment Recommendations System for Retail Customers' pilot, led by the National Bank of Greece. The primary use-case within this pilot is producing automated financial asset recommendations for use by the bank's financial advisors, either during a physical meeting between advisor and customer within a bank branch or remotely. The goals here are threefold: (i) enhanced productivity (i.e. improved productivity of investment consultants of the bank, through faster access to recommendations tailored to their retail customer accounts), (ii) better advice for investments based on a better understanding of customer preferences and behaviour and (iii) increased trading volumes, by widening the pool of investors the bank services.

In the remainder of this chapter, we will discuss the whys and hows of developing an automated financial asset recommender for this use-case. In particular, the chapter is structured as follows. In Sect. 2, we begin by describing the regulatory environment for such systems in Europe, to provide some context of where regulatory boundaries exist that need to be considered. Section 3 then formally introduces the definition of what a financial asset recommender needs to perform in terms of its inputs and outputs, while Sect. 4 discusses how to prepare and curate the financial data used by such systems. In Sect. 5, we provide a review of the scientific literature that is relevant to creating different types of financial asset recommendation systems. Additionally, in Sect. 6, we present experimental results produced within the scope of the INFINITECH project, demonstrating the performance of different recommendation systems using past customers from the National Bank of Greece and over the Greek stock market. A summary of our conclusions is provided in Sect. 7.

<sup>2</sup> https://seekingalpha.com/.

<sup>3</sup> https://www.zacks.com/.

<sup>4</sup> https://www.sofi.com.

<sup>5</sup> https://www.ellevest.com/.

#### **2 Understanding the Regulatory Environment**

Before diving into the practicalities of developing a financial asset recommendation system, it is important to consider the regulatory environment that such a system needs to exist within. Within Europe, there are two main pieces of legislation of interest, which we discuss below:

**General Data Protection Regulation (GDPR)** GDPR is an EU regulation that was introduced in 2018 and governs the use of personal data regarding EU citizens and those living within the European Economic Area (EEA), and however it has subsequently been used as the basis for similar laws in other countries and so is more widely applicable. GDPR is relevant to the development of financial recommendation systems, as such systems by their nature will need to process the personal data of each customer to function. The key provisions in GDPR to consider therefore are:

**Markets in Financial Instruments Directive (MiFID)** MiFID has been applicable across the European Union since November 2007 and forms the cornerstone of the EU's regulation of financial markets, seeking to improve their competitiveness by creating a single market for investment services and activities and to ensure a high degree of harmonized protection for investors in financial instruments. Later in 2011, the European Commission also adopted a legislative proposal for the revision of MiFID, resulting in the Directive on Markets in Financial Instruments repealing Directive 2004/39/EC and the Regulation on Markets in Financial Instruments, commonly referred to as MiFID II and MiFIR, which are currently in effect. These regulations are quite wide-ranging, as they cover the conduct of business and organizational requirements for investment firms, authorization requirements for regulated markets, regulatory reporting to avoid market abuse, trade transparency obligations for shares and rules on the admission of financial instruments to trading.

The MiFID Directive regulates all investment products, including Equities, Bonds, Mutual Funds, Derivatives, Structured Deposits (but not other products such as Loans or Insurance). From the perspective of these regulations, there are three key provisions to account for when developing financial asset recommendation systems (note that if the system is also performing buy/sell orders, there are other provisions in addition to the following to consider):

#### **3 Formalizing Financial Asset Recommendation**

From a customer's perspective, the goal of financial asset recommendation is to provide a ranked list of assets that would be suitable for the customer to invest in, given the state of the financial market and the constraints and desires of that customer. In this case, 'suitability' is a complex concept that needs to capture both the profitability of the asset with respect to the market and the likelihood that the customer would actually choose to invest in that asset (requiring solutions to model factors such as the perceived risk of an asset in combination with the customer's risk appetite). The notion of suitability also has a temporal component, i.e. how long the customer wants to invest their capital for (known as the investment horizon). With regard to what precisely is recommended, these might be individual financial assets, but more likely are a group of assets (since it is critical to mitigate risk exposure by spreading investments over a broad portfolio). Despite this, from a system perspective, it is often more convenient to initially treat financial asset recommendation as a scoring problem for individual assets, i.e. we identify how suitable an asset is in isolation first, and then tackle aggregation of individual assets into suitable asset groups as a second step. Hence, individual asset scoring can be formulated as follows:

**Definition 1 (Individual Asset Scoring)** Given a financial asset '*a*' (e.g. a stock or currency that can be traded) from a set of assets within a financial market *a* ∈ M, a customer '*c*' with a profile P*<sup>c</sup>* and an investment time horizon T produce a suitability score *s(*H*a,* P*c,*M*,*T*)*, where 0 ≤ *s* ≤ 1 and a higher score indicates that *a* is more suitable for investment. Here H*<sup>a</sup>* represents the historical properties of the asset *a*, P*<sup>c</sup>* represents the customer profile, M comprises contextual data about the market as a whole and T is the investment time horizon (e.g. 3 years).

**Asset History** H*<sup>a</sup>* When considering an asset's history H*a*, we typically consider the past pricing data for that asset on a daily basis (such as open/close prices, as well as highs/lows for that asset each day). Such time series data can be converted into multiple numerical features describing the asset across different time periods (e.g. the last [1,3,6,12,24,36] months). For instance, we might calculate the Sharpe Ratio over each period to provide an overall measure of the quality of the asset or use volatility to measure how stable the gains or losses from an asset are (representing risk).

**Customer Profile** P*<sup>c</sup>* To enable personalization of the recommendations for a particular customer, it is important to factor in the financial position and preferences defined by their profile P*c*. The most important factors to consider here are the liquidity of the customer (i.e. how much they can invest safely) and the customer's risk appetite (which should influence the weighting of associated high/low risk assets). We may also have other historical data about a customer in their profile as well, such as past financial investments, although this is rare as most customers will not have invested in the past.

**Factoring in the Market** M Markets will behave differently depending on the types of assets that they trade, which is important since we should consider profitability in the context of the market. In particular, there are two ways that we can consider profitability: (1) raw profitability, typically represented by annualized return on investment (where we want returns upward of 7% under normal conditions) and (2) relative profitability, which instead measures to what extent an investment is 'beating' the average of the market (e.g. by comparing asset return to the market average). Relative profitability is important to consider as markets tend to follow boom/bust cycles, and hence a drop in raw profitability might reflect the market as a whole rather than just the asset currently being evaluated.

**Investment Time Horizons** T Lastly, we need to consider the amount of time a customer has to invest in the market, as this can strongly impact the suitability of an asset. For example, if working with a short horizon, then it may be advantageous to avoid very volatile assets that could lose the customer significant value in the short term unless they are particularly risk seeking.

By developing an asset scoring function *s(*H*a,* P*c,*M*,*T*)*, we can thereby use the resultant scores to identify particularly good assets to either recommend to the customer directly or include within a broader portfolio of assets. The same scoring function can similarly be used to update asset suitability (and hence portfolio quality) over time as more data on each asset H*<sup>a</sup>* and the market as a whole M is collected over time. Having formulated the task and introduced the types of data that can be used as input, in the next section, we discuss how to prepare and curate that data.

#### **4 Data Preparation and Curation**

#### *4.1 Why Is Data Quality Important?*

Data quality issues can have a significant impact on business operations, especially when it comes to the decision-making processes within organizations [5]. As a matter of fact, efficient, accurate business decisions can only be made with clean, high-quality data. One of the key principles of data analytics is that the quality of the analysis strongly depends upon the quality of the information analysed. However, according to Gartner,<sup>6</sup> it is estimated that more than 25% of critical data in the world's top companies are flawed, at the same time when data scientists worldwide spend around 80% of their time on preparing and managing data for analysis, with approximately 60% of their time being spent on cleaning and organizing data and with an additional 19% of their time being spent on data collection.<sup>7</sup> It is thus obvious that in our big data era, actions that can improve the quality of diverse high volume (financial) datasets are critical and that these actions need to be facilitated by (automated, to the maximum extent possible) tools, optimizing them in terms of (time) efficiency and effectiveness.

The complete set of methodological and technological data management actions for rectifying data quality issues and maximizing the usability of the data are referred to as *data curation* [7]. Data curation is the active and on-going management of data through its lifecycle of interest and usefulness, from creation and initial storage to the time when it is archived for future research and analysis, or becomes obsolete and is deleted [4]. Curation activities enable data discovery and retrieval, maintaining quality, adding value and providing for re-use over time. Data curation has emerged as a key data management activity, as the number of data sources and platforms for data generation has grown. *Data preparation* is a sub-domain of data curation that focuses on data pre-processing steps, such as aggregation, cleaning and often anonymization. Data preparation is the process of cleaning and transforming raw data prior to processing and analysis. It often involves reformatting data, making corrections to that data and combining datasets to add value. The goal of data preparation is the same as other data hygiene processes: to ensure that data is consistent and of high quality. Inconsistent low-quality data can contribute to incorrect or misleading business intelligence. Indeed, it can create errors and make analytics and data mining slow and unreliable. By preparing data for analysis up front, organizations can be sure they are maximizing the intelligence potential of that information. When data is of excellent quality, it can be easily processed and analysed, leading to insights that help the organization make better decisions. High-

<sup>6</sup> https://www.reutersevents.com/pharma/uncategorised/gartner-says-more-25-percent-criticaldata-large-corporations-flawed.

<sup>7</sup> https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consumingleast-enjoyable-data-science-task-survey-says/#34d3b5ad6f63.

quality data is essential to business intelligence efforts and other types of data analytics, as well as better overall operational efficiency.

#### *4.2 Data Preparation Principles*

To make financial data useable, that data must be cleansed, formatted and transformed into information digestible by the analytics tools that follow in a business intelligence pipeline. The actual data preparation process can include a wide range of steps, such as consolidating/separating fields and columns, changing formats, deleting unnecessary or junk data and making corrections to that data.<sup>8</sup> We summarize each of the key data preparation steps below:

**Discovering and Accessing Data** Data discovery is associated with finding the data that are best-suited for a specific purpose. Data discovery can be a very painful, frustrating and time-consuming exercise. An essential enabler of efficient discovery is the creation and maintenance of a comprehensive, well-documented data catalogue (i.e. a metadata repository). In this context, key data sources include asset pricing data, company and customer profiles, as well as investment transaction data.

**Profiling and Assessing Data** Profiling data is associated with getting to know the data and understanding what has to be done before the data becomes useful in a particular context and is thus key to unlocking a better understanding of the data. It provides high-level statistics about the data's quality (such as row counts, column data types, min, max and median column values and null counts), and visualization tools are very often exploited by users during this phase, enabling them to profile and browse their data. This is particularly important in the finance context to identify and understand unusual periods in financial time series data, e.g. caused by significant real-world events like COVID-19.

**Cleaning and Validating Data** Cleaning up the data is traditionally the most timeconsuming part of the data preparation process, but it is crucial for removing faulty data and filling in any data gaps found. Some of the key data cleansing activities include:

<sup>8</sup> https://blogs.oracle.com/analytics/what-is-data-preparation-and-why-is-it-important.

Once data has been cleansed, it must be validated by testing for errors in the data preparation process up to this point. Often times, an error in the system will become apparent during this step and will need to be resolved before moving forward. Outlier removal is particularly important here, as individual data points such as very high or low prices (e.g. caused by an erroneous trade or a malicious actor) can confuse machine learning systems.

**Transforming Data** Data transformation is the process of changing the format, structure or values of data. Data transformation may be constructive (adding, copying and replicating data), destructive (deleting fields and records), aesthetic (standardizing salutations or street names) or structural (renaming, moving and combining columns in a database). Proper data transformation facilitates compatibility between applications, systems and types of data.

**Anonymizing Data** Data anonymization is the process of protecting private or sensitive information by masking, erasing or encrypting identifiers (e.g. names, social security numbers and addresses) that connect an individual to stored data. Several data anonymization techniques exist, including data masking, pseudonymization, generalization, data swapping, data perturbation and more, each one serving different purposes, each coming with its pros and cons and each being better suited than the others depending upon the dataset at hand, but all serving the common purpose of protecting the private or confidential information included within the raw data, as required by law (see Sect. 2).

**Enriching Data** Data enrichment is a general term that applies to the process of enhancing, refining and improving raw data. It is the process of combining firstparty data from internal sources with disparate data from other internal systems or third-party data from external sources. As such, data enrichment could be interpreted both as a sub-step of the data cleansing process, in terms of filling in missing values, and as an independent step of the process. This may require the association (or linking) of the raw data with data from external sources or the enrichment of the raw data with meaningful metadata, thus facilitating their future discovery and association and thus empowering the extraction of deeper insights. A common example here is connecting asset pricing data with profiles for associated companies that influence those assets.

**Storing Data** Once prepared, the data can be stored or channelled into a third-party application—such as a business intelligence tool—clearing the way for processing and analysis to take place.

#### *4.3 The INFINITECH Way Towards Data Preparation*

Within the context of the INFINITECH project, the consortium partners have designed and developed a complete data ingestion/data preparation pipeline, aiming to provide an abstract and holistic mechanism for the data providers, addressing the various connectivity and communication challenges with the variety of data sources that are exploited in the finance and insurance sector, as well as the unique features that need to be considered when utilizing a range of heterogeneous data sources from different sectors. Hence, the scope of the Data Ingestion mechanism is fourfold:

The INFINITECH complete data ingestion/data preparation pipeline is composed of four main modules, as also illustrated in Fig. 10.1, depicting the high-level architecture of the pipeline. We describe each of these modules in more detail below:

**Data Retrieval** This module undertakes the responsibility to retrieve or receive the new datasets from a data source or data provider either periodically or ondemand. The scope of the Data Retrieval module is to facilitate the data acquisition from any relational database, HDFS deployments, FTP or HTTP servers, MinIO storage servers, as well as from any API of the data source. The prerequisite is that the appropriate information is provided by the data provider. Additionally, the Data Retrieval module enables the ingestion of new datasets that are pushed from the data provider to its exposed RESTful APIs. Hence, the Data Retrieval module supports all the aforementioned data source types which are considered the most commonly used data sources in current Big Data ecosystems. Nevertheless, the modular architecture of the described solution facilitates the future expansion of the list of supported data source types in an effortless and effective manner.

**Data Mapper** This module is responsible for the generation of the mapping between the entities of a retrieved dataset and the ones of an underlying data model (derived from the data provider's input). The scope of the Data Mapper module is to enable the mapping of the data entities from a new dataset to the data model that is given by the data provider. In this sense, the data provider is able to create the mappings for each entity of the new dataset to a specific data entity of the data model. To achieve this, at first the Data Mapper module offers the means to

**Fig. 10.1** INFINITECH data preparation pipeline

integrate a data model during its initial configuration. Then, during processing, the data entities of the provided dataset are extracted and displayed to the data provider via its user friendly and easy-to-use user interface. Through this user interface, the data provider is able to select the corresponding entities of the integrated data model that will be mapped to the entities of the dataset. The generated mappings are stored for later reuse in a JSON format.

**Data Cleaner** This module undertakes the responsibility to perform the data cleaning operations on the retrieved dataset, based on the data provider's input. The scope of the Data Cleaner is to provide the data cleaning operations that will ensure that the provided input datasets, which originate from a variety of heterogeneous data sources, are clean and complete to the maximum extent possible. The specific functionalities of this module enable the detection and correction (or removal) of inaccurate, corrupted or incomplete values in the data entities, with the aim of increasing data quality and value. To this end, the data cleaning process is based on a set of data cleansing rules that are defined by the data provider on a data entity level via the Data Cleaner's user friendly and easy-to-use user interface and is a four-step process that includes:

4. The maintenance of complete history records containing the history of errors identified and the data cleaning operations that were performed to address them

**Data Anonymizer** This module undertakes the responsibility of addressing the various privacy concerns and legal limitations imposed on a new dataset as instructed by the data provider. To this end, the anonymizer provides the advanced privacy and anonymization toolset with various data anonymization techniques that can be tailored by the data provider in order to filter or eliminate the sensitive information based on his/her needs. To achieve this, the anonymizer employs a generic data anonymization process which is highly customizable through the definition of anonymization rules which are set by the data provider leveraging his/her expertise.

All four processes can be performed as background processes, provided that the data providers have created in advance the corresponding data retrieval/mapping/cleaning/anonymization profiles for a specific dataset, which will in turn be used in an automated way for the execution of the processes. In the next section, we will discuss approaches to use our curated data to perform financial asset recommendation.

#### **5 Approaches to Investment Recommendation**

With the underlying data needed to perform asset scoring prepared, we next need to understand how our asset scoring function can be built. In this section, we will provide a brief background into one class of methods for developing such a scoring function, namely using recommendation models, with a particular focus on recommendation as it applies to financial products and services.

**What Is a Recommender?** Recommendation systems are a popular class of algorithms that aim to produce personalized item suggestions to a user. Popular examples of such algorithms are movie recommendation (e.g. on Netflix) [3] or product recommendation (e.g. Amazon recommended products) [36]. For our later reported results, we will only be considering supervised machine-learned recommenders, i.e. approaches that leverage machine learning to analyse the past history of items, user interactions with those items and/or the user's relationships to the items, with the goal of producing a recommendation model [15]. However, we will discuss a couple of unsupervised approaches later in this section. The process of creating a supervised machine learned model is known as training. A trained recommendation model can be considered as a function that (at minimum) takes a user and item as input and produces a score representing the strength of relationship between that user and item pair. An effective recommendation model should find a right balance between relevance, diversity and serendipity in the recommended items, using the different types of evidence discussed in Sect. 3. Hence, for financial asset recommendation, we want to train a model to represent the function *s(*H*a,* P*c,*M*,*T*)* (see Sect. 3), where H*<sup>a</sup>* represents the item (a financial asset) and P*<sup>c</sup>* represents the user (a customer).

**Types of Recommender** Recommendation algorithms can be divided along three main dimensions:

In terms of evidence, a model might utilize explicit interactions between users and items (e.g. when a user watches a movie) [18], intrinsic information about an item (e.g. what actors star in a movie) [41], time series data about an item (e.g. popularity of a movie over time) [37], intrinsic information about a user (e.g. the user says they like action movies) [24] or time series data about the user (e.g. the user has been watching a lot of dramas recently) [43]. When considering recommendation context, most approaches consider each item in isolation, but some approaches examine items within the context of a set (e.g. recommending an item to add to a customer's basket that already contains items [27]) or when recommending a sequence of items (e.g. recommending a series of places to visit when on holiday [19]). Finally, depending on how the model is trained, simple approaches consider all prior interactions as an unordered set, while more advanced techniques will factor in that recent interactions are likely more relevant than older interactions [40].

Within the context of the financial asset recommendation, explicit interactions between users and items are somewhat rare and difficult to obtain, as they represent past investments made by customers (which is not public data). Hence, approaches that can more effectively leverage available intrinsic and historical data about the customers and assets for recommendation are more desirable here. With regard to recommendation context, we will be focusing on asset recommendations in isolation here, and however we would direct the reader to the literature on basket recommenders if interested in recommendation of items to add to an existing portfolio [1]. Meanwhile, capturing the temporal dynamics of our historical data and interactions is intuitively important within financial markets, as past asset performance is not always reflective of future performance, and hence models that can capture temporal trends will be more effective in this domain.

In the remainder of this section, we will discuss a range of recommendation approaches that can be applied for financial asset recommendation, as well as highlight their advantages and disadvantages. To structure our discussion, we will group models into six broad types. We will start with four supervised approaches that we will experiment with later in Sect. 6, namely *Collaborative filtering*, *User Similarity*, *Key Performance Indicator Predictors*, and *Hybrid*. We then discuss two unsupervised approaches, namely *Knowledge Based* and *Association Rule Mining*.

#### *5.1 Collaborative Filtering Recommenders*

Collaborative filtering recommenders focus on providing item recommendations to users based on an assessment of what other users, judged to be similar to them, had positively interacted with in the past. These models work with a matrix of user–item interactions, which is typically sparse, owing to the fact that most customers would not have interacted with (invested in) more than a small subset of items (financial assets) previously. The intuition behind these models is that missing ratings for items can be predicted, or imputed, as they are likely correlated across the user and item space. By learning the similarities between users and items, inferences can be made about the missing values. As this approach leverages past interactions between the user and items in order to provide recommendations, it typically works best with large quantities of interactions such that re-occurring patterns can be found. These interactions can be in the form of explicit feedback, wherein a user provides an explicit, quantitative rating for the item, or as implicit feedback, where user interactions are gauged from their behaviour using system, such as clicking a link, or spending a significant amount of time browsing a specific product [1].

**Advantages and Disadvantages** Collaborative filtering models can be advantageous since they tend to generate diverse recommendations, since they are looking for items interacted with (invested in) by customers that are similar to you, but you have not interacted with. On the other hand, particularly when neighbourhoodbased, they suffer from the problem of sparsity, where it is difficult to provide recommendations for a user who has little to no interaction data available, as correlations cannot easily be drawn. More advanced model-based techniques alleviate the sparsity problem, but often still require large amounts of existing interaction data, which presents a particular problem in the financial domain due to the difficulty of curating and obtaining sufficient anonymized financial transactions [14]. Also of note is that many of these models employ dimensionality reduction (to speed up computation and remove noise) and learn *latent* factors between users and items in order to perform recommendation. This makes explaining why certain recommendations were made challenging, as the latent factors learned may not be interpretable by humans.

**Applications in Finance** Collaborative filtering techniques are often applied for financial recommendation. Some notable approaches include the implementation of a fairness-aware recommender system for microfinance, by Lee et al. [16], which uses item-based regularization and matrix factorization, as well as risk-hedged venture capital investment recommendation using probabilistic matrix factorization, as proposed by Zhao et al. [45]. Luef et al. [17] used user-based collaborative filtering for the recommendation of early stage enterprises to angel investors, as a method of leveraging the investors' circle of trust. Meanwhile, Swezey and Charron [38] use collaborative filtering recommendations for stocks and portfolios, reranked by Modern Portfolio Theory (MPT) scores, which incorporate metrics of risk, returns and user risk aversion. From these approaches, it can be gleaned that while there is value in utilizing user–product interaction data to provide financial recommendation, this is not sufficient as a representation of the suitability of the recommended financial products, and augmentation with asset intrinsic information is often necessary. Furthermore, gathering the required interaction data for collaborative filtering recommendation can present a significant challenge due to the rarity of investment activity among the general populace.

#### *5.2 User Similarity Models*

Like collaborative filtering models, user similarity models are based on the intuition that if two users are similar, then similar items should be relevant to both. However, instead of directly learning latent similarity patterns between users and items from past interaction data, user similarity models generate a feature vector for each user and directly calculate similarity between all pairs of users (e.g. via cosine similarity). Once a top-n list of most similar users is found, a ranked list of items can be constructed based on items those similar users have previously interacted with (e.g. invested in), where the more users interact with an item, the higher its rank.

**Advantages and Disadvantages** The primary advantage of user similarity models is that they are highly explainable (e.g. 'x users like you invested in this asset'). However, in a similar way to collaborative filtering models, they are reliant on sufficient prior interaction data to identify 'good' items for groups of similar users.

**Applications in Finance** Luef et al. [17] implement a social-based recommender model for investing in early stage enterprises, which asks users to specify an inner circle of trusted investors and then provide recommendations on the basis of what those users invested in. Yujun et al. [44] propose a fuzzy-based stock recommender algorithm, which recommends instruments that were chosen by other similar users. Such models that focus on similar users tend to be seen as more convincing by end users, as the results can be presented in not only an explainable but also a personable manner by alluding to other experts in the field who made similar decisions.

#### *5.3 Key Performance Indicator Predictors*

KPI predictor-based recommenders utilize descriptive features of the users and items in order to produce recommendations, rather than relying on the existence of explicit or implicit interactions between those users and items. These features are typically expressed as a single numerical vector representation per user–item pair. Using these vectors as a base, content-based recommenders will typically train a model to predict a rating, score or label (KPI) that can act as a proxy for item suitability to the user within the context of the target domain. For example, in the financial asset recommendation context, such a model might aim to predict the profitability of the asset given the user's investment horizon.

**Advantages and Disadvantages** The advantage that KPI predictors hold over collaborative filtering is that they do not rely on past interaction data, and hence new items that have not been interacted with in the past can be recommended based on their descriptive features. Such models can also be applied to cold-start users (those with no investment history). The drawback of these models is that they rely on a proxy KPI rather than a true measure of suitability, meaning that performance will be highly dependent on how correlated the KPI is with actual suitability. Additionally, these models tend to only have a limited capacity to personalize the recommendations for each individual user, i.e. the item features tend to dominate the user features in such models.

**Applications in Finance** Seo et al. [33] produced a multi-agent stock recommender that conducts textual analysis on financial news to recommend stocks. This is one of several models that utilize natural language processing techniques to extract stock predictors from textual data [31, 32]. Musto et al. also develop a case-based portfolio selection mechanism that relies on user metadata for personalization [22], while Ginevicius et al. attempt to characterize the utility score of real estate using complex proportional evaluation of multiple criteria [8].

#### *5.4 Hybrid Recommenders*

In practical applications, multiple recommender models are often combined or used in tandem to improve overall performance. Such combined approaches are known as hybrid recommenders. There are two broad strategies for combining recommenders: unification and voting. Unification refers to the combination of recommender models to produce a singular algorithm that returns a result. This might involve feeding the output of multiple existing models into a weighted function that combines the scores from each (where that function may itself be a trained model). Meanwhile, voting approaches consider the output of the different learned models as votes for each item, where the items can be ranked by the number of votes they receive. Recommenders can also be cascaded, wherein multiple recommenders are given strict orders of priority. The absence of relevant information for one recommender will then trigger recommendations from the model of lower priority. This can present a useful solution to the problem of datasets being sparse or the unavailability of consistent feedback from users across various methods.

**Advantages and Disadvantages** The main advantage of hybrid approaches is that they alleviate the disadvantages of each individual approach and are able to recommend a diverse range of items. On the other hand, they add significant complexity to the recommendation task (particularly when additional model(s) need to be trained to perform the combination) and make outcome explainability more challenging [14].

**Applications in Finance** Taghavi et al. [39] proposed the development of an agent-based recommender system that combines the results of collaborative filtering together with a content-based model that incorporates investor preferences and socioeconomic conditions. Luef et al. [17] also employed a hybrid recommendation approach that compares the rankings produced by their previously described social user-based recommender and knowledge-based recommender using Kendall's correlation. Matsatsinis and Manarolis [20] conduct the equity fund recommendation task using a combination of collaborative filtering and multi-criteria decision analysis and the associated generation of a utility score. Mitra et al. [21] combine a collaborative filtering approach along with an attribute-based recommendation model in the insurance domain, in order to account for the sparsity of user ratings. Hybrid techniques can provide recommendations that are representative of the different variables that influence the suitability of financial assets for investment, such as investor preferences, textual analysis, expertly defined constraints, prior investment history and asset historical performance and perhaps present the best theoretical solution for the current problem setting. However, discovering the optimal combination of these models can be challenging, especially as their numbers increase.

#### *5.5 Knowledge-Based Recommenders*

Having discussed supervised approaches to the problem of financial asset recommendation, to complete our review, we also highlight a couple of unsupervised approaches, starting with knowledge-based recommenders.

Knowledge-based recommendation is based on the idea of an expert encoding a set of 'rules' for selecting items for a user. In this case, the system designer defines a set of filtering and/or scoring rules. At run-time, a user fills in a questionnaire on their constraints and preferences (this information might also be derived from an existing customer profile), needed to evaluate those rules. The rules are then applied to first filter the item set, then score and hence rank the remaining items, forming the recommended set [6].

**Advantages and Disadvantages** In terms of advantages, they allow for highly customizable and explainable recommendations, as they are based on humandefined rules. They are also usable for cold-start users with no prior history (as long as they correctly fill in the questionnaire). On the other hand, they are reliant on humans defining good generalizable rules, which is both time consuming and error-prone for diverse domains like finance and needs to be continually updated manually [6].

**Applications in Finance** Gonzalez-Carrasco et al. [9] utilized a fuzzy approach to classify investors and investment portfolios, which they then match along with one another. Set membership for investments is calculated using a set of predefined rules. For instance, a product in prime real estate with normal volatility might be classified as socially moderate and psychologically conservative, but an investment in solar farms with a low long-term return is classified as socially and psychologically aggressive. Luef et al. [17] evaluated the performance of multiple recommender models to recommend early stage enterprises to investors, and one such model was knowledge-based, where investor profiles are used to determine constraints on recommendation, with profile features acting as hard filters and investor preferences acting as soft constraints on enterprise profiles. These recommenders allow for the fine-grained specification of requirements and constraints upon the recommendation and are helpful in providing context to support investment decisions by presenting an integrated picture of how multiple variables influence investment instruments. These recommenders can provide highly customized results that can be tweaked as the user desires alongside their changing preferences; however, they cannot glean latent patterns representative of the userbase and their investment trends, which might prove useful in providing recommendations that leverage a user's circle of trust.

#### *5.6 Association Rule Mining*

Association rule mining is an unsupervised, non-personalized method of recommendation which seeks to identify frequent itemsets—which are groups of items that typically co-occur in baskets. In doing so, it is often used as a method of recommending items that users are likely to interact with given their prior interactions with other items identified within those itemsets. These relationships between the items, or variables, in a dataset are termed association rules and may contain two or more items [2]. This technique is typically applied to shopping settings where customers place multi-item orders in baskets and therefore is also termed market basket analysis.

**Advantages and Disadvantages** Association rules are straightforward to generate in that they do not require significant amounts of data or processing time, and as an unsupervised technique, the input data does not need to be labelled. However, as an exhaustive algorithm, it discovers all possible rules for the itemsets that satisfy the required thresholds of co-occurrence. Many of these rules may not be useful or actionable; however, this requires expert knowledge to evaluate and provide insight upon the meaningfulness of the generated rules.

**Applications in Finance** One such approach by Nair et al. [23] uses temporal association rule mining to account for the general lack of consideration of time information in typical association rule mining approaches. They leverage a genetic algorithm called Symbolic Aggregate approXimation (SAX) to represent time series data in a symbolic manner and then feed these representations into the model to obtain association rules and provide recommendations on the basis of these rules. Other approaches by Paranjape-Voditel and Deshpande [25], as well as Ho et al. [11], look at the usage of fuzzy association rule mining to investigate hidden rules and relationships that connect one or more stock indices. The former recommends stocks, by using the principle of fuzzy membership in order to calculate relevance of various items in the portfolio, and also introduces a time lag to the inclusion of realtime stock price information, so that any emerging patterns in price movement are aptly captured. This becomes especially applicable in the financial domain, wherein quantitative data about instruments can be classified better using a fuzzy approach as opposed to one using crisp boundaries, so as to identify the ranges of the parameters.

#### **6 Investment Recommendation within INFINITECH**

Having summarized how we can generate recommendation scores for assets using different types of recommendation algorithms, in this section, we illustrate how these models perform in practice. In particular, we will first describe the setup for evaluating the quality of financial asset recommendation systems and then report performances on a real dataset of financial transactions data from the Greek stock market.

#### *6.1 Experimental Setup*

**Dataset** To evaluate the quality of a financial asset recommendation system, we need a dataset comprising financial assets and customers, as well as their interactions. To this end, we will be using a private dataset provided by the National Bank of Greece, which contains investment transactions spanning 2018– 2021, in addition to supplementary information on demographic and non-investment behaviour data from a wider subset of the bank's retail customers, and historical asset pricing data. From this dataset, we can extract the following main types of information:

**Table 10.1** Greek stock market sample dataset statistics by year

spends across multiple sectors and deposit account balances with aggregations spanning different time windows and demographic features on the customers. These KPIs are recorded in entirety across 2018, 2019 and 2020, with only incomplete aggregates available across 2021. Only a limited subset of users possess investment transactions.

• **Asset Intrinsic and Historical Features**: This data contains the historical prices of the assets spanning 2018–2021, alongside other descriptive information on the assets such as their name, ISIN and financial sector. The price data points are available for almost all days (although there are a few gaps due to data collection failures, see Table 10.1).

**Models** For our experiments, we will experiment with eight representative supervised recommendation models from those described earlier, starting with two collaborative filtering models:

To represent a user similarity-based approach, we implement a user encoding strategy inspired by the proposed hybrid approach of Taghavi et al. [39], albeit not relying on the use of agents, and using a different content-based model:

• **Customer Profile Similarity (CPS) [User Similarity]**: This strategy represents users in terms of a subset of the non-investment indicators concerning their retail transactions and balances with the bank. These indicators are weighted by age where possible, i.e. customer purchase features from two years ago have a lower weight than those from one year ago. Given these vector representations of each customer, we use these customer vectors to represent the financial assets. Specifically, for each financial asset, we assign it a vector representation that is the mean of the vectors of the customers that invested in that asset previously. The intuition here is that these aggregate vectors will encode the key indicators that make assets suitable for each customer. Finally, to score each asset for a new customer, we can simply calculate the cosine similarity between that customer and the aggregate asset representation, as both exist in the same dimensional space, where a higher similarity indicates the asset is more suitable for the customer.

Meanwhile, since it makes intuitive sense to use profitability as a surrogate for suitability in this context, we also include a range of key performance indicator predictors, which rank financial assets by their predicted profitability:

	- **Linear Regression (LR)** [29]: Linear regression attempts to determine a linear relationship between one or more independent variables and a dependent variable by assigning these independent variables coefficients. It seeks to minimize the sum of squares between the observations in the dataset and the values predicted by the linear model.
	- **Support Vector Regression (SVR)** [34]: Support vector regression is a generalization of support vector machines for continuous values, wherein the aim is to identify the tube which best approximates the continuous-valued function. The tube is represented as a region around the function bounded on either side by support vectors around the identified hyperplane.
	- **Random Forest Regression (RFR)** [30]: Random forest regression is an ensemble method that utilizes a diverse array of decision tree estimators and conducts averaging upon their results in a process that significantly reduces the variance conferred by individual estimators by introducing randomness. Decision trees, in turn, are models that infer decision rules in order to predict a target variable.

In addition, we also implement a range of hybrid models that combine the aforementioned collaborative filtering approaches with customer and item intrinsic and historical features, in a similar manner to the experiments conducted by Taghavi et al. [39], Luef et al. [17] and others who use the intuition of combining intrinsic attributes of customers and items alongside collaborative filtering inputs:

• **MF+CI [Hybrid]**: This model combines the ranked recommendation lists of matrix factorization and the customer similarity model using rank aggregation. The specific rank aggregation method used is score voting.

• **LightGCN+CI [Hybrid]**: This model combines the ranked recommendation lists of LightGCN and the customer similarity model using rank aggregation. The specific rank aggregation method used is score voting.

**Training Setup** To train our supervised models, we need to divide the dataset into separate training and test subsets. We use a temporal splitting strategy here, where we define a time point to represent the current moment where investment recommendations are being requested. All data prior to that point can be used for training, while the following 9 months of data can be used to evaluate success. We create three test scenarios in this manner, where the three time points are 1st of October 2019 (denoted 2019), 1st of April 2020 (denoted 2020A) and the 1st of July 2020 (denoted 2020B). The collaborative filtering models are each trained for 200 epochs. Due to the large size of the dataset, a batch size of 100000 is chosen for training in order to speed up training time. Each model is run 5 times, and performance numbers are averaged out across the 5 runs.

**Metrics** To quantitatively evaluate how effective financial asset recommendation is, we want to measure how satisfied a customer would be if they followed the investment recommendations produced by each approach. Of course, this type of evaluation is not possible in practice, since we would need to put each system into production for an extended period of time (e.g. a year). Instead, we rely on surrogate metrics that are more practically available and capture different aspects of how satisfied a user might be if they followed the recommendations produced. There are two main aspects of our recommendations that we examine here:

type of metric is it ignores any personalized aspects regarding the customer's situation and can be highly volatile if there are assets who's price experiences significant growth during the test period (as we will show later).

For both of these aspects, we calculate the associated metrics when analysing the top '*k*' assets recommended, where '*k*' is either 1, 5 or 10.

#### *6.2 Investment Recommendation Suitability*

In this section, we will report and analyse the performance of the aforementioned financial asset recommendation models when applied to the Greek Stock market for customers of the National Bank of Greece. The goal of this section is not to provide an absolute view of asset recommendation approach performance, since there are many factors that can affect the quality of recommendations, most notably the characteristics of the market in question for the time period investigated. Instead, this analysis should be used as an illustration of how such systems can be evaluated and challenges that need to be considered when analysing evaluation output.

**Investment Prediction Capability** We will start by evaluating the extent to which the recommendation approaches recommend the assets that the banking customers actually invested in. Table 10.2 reports the investment prediction capability of each of the recommendation models under NDCG@[1,5,10] for each of the three scenarios (time periods tested). The higher NDCG values indicate that the model

**Table 10.2** Financial asset recommendation model performances when evaluating for investment prediction capability. The best performing model per scenario (time period) is highlighted in bold

is returning more of the assets that the customers invested within in the top ranks. The highest performing model per scenario is highlighted in bold. Note that we are omitting the KPI prediction models from this analysis, as they can (and often do) recommend items that no user has invested in, meaning that their NDCG performance is not truly comparable to the other models under this type of evaluation.

As we can see from Table 10.2, of all the models tested, the type of model that produces the most similar assets to what the customers actually choose to invest in is the collaborative filtering type models, where the more advanced graph-based neural model (LightGCN) is better than the classical matrix factorization (MF) model. Given that collaborative filtering models aim to extract patterns regarding past investments across customers, it indicates the customers in our dataset largely invest in similar items over time (hence there are patterns that the collaborative filtering approaches are learning). On the other hand, the user similarity-based model recommends very different assets to what the customers invested within, indicating that the customers do not cluster easily based on the features we extracted about them. Finally, we observe that the hybrid models that integrate more features about the customers and items result less similar assets being recommended than the stock collaborative filtering only approaches. This likely means that the dataset is too small (i.e. has too few examples) to confidently extract generalizable patterns from the large number of new features being given to the model here.

However, just because the collaborative filtering models have the highest scores here does not necessarily mean that they will be the best. Recall that this metric is capturing whether the model is producing the same assets as what the customer actually invested in, not whether those were effective investments. Hence, we will next examine the profitability of the assets recommended.

**Investment Profitability** We next evaluate the profitability of the recommended assets in order to assess the effectiveness of the recommendation models. Profitability is evaluated as the expected return over each 9-month testing period and then annualized. Table 10.3 reports annualized profitability when considering the top [1,5,10] recommended financial assets by each recommendation model. Profitability is calculated as a percentage return on investment over the year period.

From Table 10.3, we can observe some interesting patterns of behaviour between the different types of recommendation models. Starting with the collaborative filtering-based approaches, we can see that for the 2019 test scenario, these models would have lost the customer a significant amount of value. However, for the 2020A/B scenarios, these models would have returned a good profit of around 30%. This indicates that there was a marked shift in the types of financial asset that was profitable between 2018 and 2020. It also highlights a weakness of these collaborative filtering models, in that they assume that past investments were profitable and hence similar investments now will also be profitable, which was clearly not the case between 2018 (initial training) and the 2019 test scenario. Furthermore, we also see the hybrid models that are based on the collaborative filtering strategies suffering from the same issue, although it is notable that when

**Table 10.3** Financial asset recommendation model performances when evaluating for asset profitability. The best performing model per scenario (time period) is highlighted in bold

they are working as intended, they recommend more profitable assets than the models that use collaborative filtering alone, showing that the extra features that were added are helping identify profitable asset groups. In terms of the user similarity-based model, we again see poorer performance in the 2019 scenario than the 2020A/B scenarios, which is expected as this type of model is based on analysis of the customers investment history. However, the returns provided by this model are reasonably consistent (around 20–23%) in 2020.

Next, if we compare to the KPI Predictor models that are attempting to explicitly predict assets that will remain profitable, we see that these models can be quite volatile. For instance, for the 2019 test scenario and the LR model, the top recommended asset exhibited a return of 114%, but that same model in 2020B placed an asset in the top rank that lost 20% of its value. This volatility is caused since these models are relying on (recent) past performance of an asset remaining a strong indicator of its future performance. For example, if we consider the 2020B scenario with the 20% loss for the top-ranked asset, that is a case where the asset's price had been recently inflated, but dropped rapidly during the test period. However, if we consider the KPI predictors holistically, as a general method for building a broader portfolio (e.g. if we consider the returns for the top 10 assets), these approaches seem to be consistently recommending highly profitable assets in aggregate and generally recommend more profitable assets than the collaborative filtering-based approaches.

Overall, from these results, we can see some of the challenges when evaluating models automatically without a production deployment. Measuring investment prediction capacity of a model can provide a measure of how similar the recommendations produced are to how the customers choose to invest. However, our analysis of profitability indicates that there is no guarantee that these are good investments from the perspective of making a profit. On the other hand, we can clearly see the value that these automatic recommender systems can bring for constructing portfolios of assets, with some quite impressive returns on investment when averaged over the top-10 recommended assets.

#### **7 Summary and Recommendations**