Collaborations, in particular with other PPPs, European and international standardisation bodies, industrial technology platforms, data-driven research and innovation initiatives, user organisations and policymakers, had been identified and developed at national, European and international level since the launch of the PPP and the creation of the Association, influencing the level of maturity of these collaborations.

A key part of ensuring the sustainability of the BDV ecosystem was to develop collaborations with complementary ecosystems with an impact on technology integration and the digitisation of industry challenges. These collaborations, detailed in Fig. 2, include the ETP4HPC (European Technology Platform for HPC) (for HPC),

Fig. 2 Map of collaboration for BDV ecosystem

ECSO (for cybersecurity), AIOTI (for IoT), 5G (through 5G PPP), the European Open Science Cloud (EOSC) (for the Cloud) and the European Factories of the Future Research Association (EFFRA) (for factories of the future).

## 7 Summary

Creating a productive ecosystem for big data and driving accelerated adoption requires an interdisciplinary approach addressing a wide range of challenges from access to data and infrastructure, to technical barriers, skills, and policy and regulation. To overcome these challenges, collective action is needed from all stakeholders working together in an effective, holistic and coherent manner. To this end, the Big Data Value Public-Private Partnership was established to develop the European data ecosystem and enable data-driven digital transformation, delivering maximum economic and societal benefit, and achieving and sustaining Europe's leadership in the fields of big data value creation and Artificial Intelligence. The BDV PPP follows a phased roadmap with the use of five strategic mechanisms to drive the adoption of big data value and to encourage cooperation and coordination in the data ecosystem. The PPP proactively engaged with the key communities, which helped to enhance the development of the European Data Value Ecosystem.

Acknowledgements We greatly acknowledge the collective effort of the SRIA teams: Carlos A. Iglesias, Antonio Alfaro, Jesus Angel, Sören Auer, Paolo Bellavista, Arne Berre, Freek Bomhof, Stuart Campbell, Geraud Canet, Giuseppa Caruso, Paul Czech, Stefano de Panfilis, Thomas Delavallade, Marija Despenic, Wolfgang Gerteis, Aris Gkoulalas-Divanis, Nuria Gomez, Paolo Gonzales, Thomas Hahn, Souleiman Hasan, Bjarne Kjær Ersbøll, Bas Kotterink, Yannick Legré, Yves Mabiala, Julie Marguerite, Ernestina Menasalves, Andreas Metzger, Elisa Molino, Thierry Nagellen, Dalit Naor, Maria Perez, Milan Petkovic, Roberta Piscitelli, Klaus-Dieter Platte, Pierre Pleven, Dumitru Roman, Titi Roman, Alexandra Rosén, Nikos Sarris, Stefano Scamuzzo, Simon Scerri, Corinna Schulze, Bjørn Skjellaug, Francois Troussier, Colin Upstill, Josef Urban, Meilof Veeningen, Tonny Velin, Ray Walshe, Walter Waterfeld and Stefan Wrobel.

## References

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Achievements and Impact of the Big Data Value Public-Private Partnership: The Story so Far

Ana García Robles, Sonja Zillner, Wolfgang Gerteis, Gabriella Cattaneo, Andreas Metzger, Daniel Alonso, Martina Barbero, Ernestina Menasalvas, and Edward Curry

Abstract The European contractual Public-Private Partnership on Big Data Value (BDV PPP) has played a central role in the implementation of the revised Digital Single Market strategy, contributing to multiple pillars, including "Digitising European Industry", "Digital Skills", "Building the European Data Economy" and "Developing a European Data Infrastructure". The BDV PPP and the Big Data Value Association have also played a pivotal role in the European Artificial Intelligence and Data Strategies launched by the European Commission in 2018. This chapter provides an overview and an in-depth analysis of the impact of the PPP by mid-2019, with a focus on the achievements and the overall impact since the launch of the PPP.

Keywords Public-private partnership · Data impact · Data PPP · Big data value

A. García Robles (\*) · M. Barbero Big Data Value Association, Bruxelles, Belgium e-mail<EMAIL_ADDRESS>
S. Zillner Siemens AG, Munich, Germany

W. Gerteis SAP, Walldorf, Germany

G. Cattaneo IDC, Milan, Italy

A. Metzger paluno, University of Duisburg-Essen, Duisburg, Germany

D. Alonso ITI, Valencia, Spain

E. Menasalvas Universidad Politécnica de Madrid, Madrid, Spain

E. Curry Insight SFI Research Centre for Data Analytics, NUI Galway, Galway, Ireland

© The Author(s) 2021 E. Curry et al. (eds.), The Elements of Big Data Value, https://doi.org/10.1007/978-3-030-68176-0\_4

## 1 Introduction

The European contractual Public-Private Partnership on Big Data Value (BDV PPP) was signed on 13 October 2014. It marked the commitment of the European Commission, industry and partners from research to build a data-driven economy across Europe, mastering the generation of value from Big Data and creating a significant competitive advantage for European industry, thus boosting economic growth and employment. The BDV PPP started in 2015 and was operationalised with the launch of the Leadership in Enabling and Industrial Technologies (LEIT) work programme 2016/2017 of Horizon 2020 (H2020) with the first PPP projects (Call 1) starting in January 2017. With 57 projects,1 an allocated investment of public funding of €301 million by the end of 2019 and around 300 organisations as part of the private association2 (Big Data Value Association, BDVA) over the years, the Big Data Value PPP has played a central role in the implementation of the revised Digital Single Market (DSM) strategy, contributing to multiple pillars including "Digitising European Industry", "Digital Skills", "Building the European Data Economy" and "Developing a European Data Infrastructure". The BDV PPP and the BDVA have also played an important role in the European AI and Data Strategies launched by the European Commission in 2018. This chapter provides an overview and an in-depth analysis of the impact of the PPP by mid-2019, with a focus on the achievements and the overall impact since the launch of the PPP.

This chapter details the achievements and the impact of the Big Data Value PPP. After explaining the key elements of the Big Data Value PPP in Sect. 2, and presenting a summary of the achievements and impact created during 2018 discussed in Chap. "A Roadmap to Drive Adoption of Data Ecosystems", an in-depth analysis of the overall progress towards the mains goals of the partnership by mid-20193 is given in Sect. 4. Finally, the Sect. 5 concludes with a summary and perspectives on the future.

## 2 The Big Data Value PPP

The vision, overall goals, main technical and non-technical priorities and a research and innovation roadmap for the European Public-Private Partnership (PPP) on Big Data Value are defined in the Big Data Value Strategic Research and Innovation Agenda (BDV SRIA) (Zillner et al. 2017).

1 Considering projects selected for funding by end of December 2019.

2 Includes all BDVA members, including active and terminated/resigned (source: BDVA).

3 The BDVA is responsible for providing a full monitoring report on its activities. Since 2019 and in accordance with the European Commission, the full monitoring report of the Partnership will only be submitted every 2 years. The most recent version was delivered in 2019 covering the period from beginning 2018 to beginning 2019 (https://bdva.eu/MonitoringReport2018).

### The BDV PPP SRIA defined the roadmap and methodology by describing three different phases:

The BDV SRIA has been regularly updated incorporating the multi-annual roadmap of the BDV PPP. BDV SRIA v4 (delivered at the end of 2017) provides direct input to the LEIT WP 2018–2020 as defined in its updated Phases II and III.

The BDV PPP projects cover Big Data technology, including Artificial Intelligence methods, and application research and innovation, new data-driven business models, data ecosystem support, data skills, regulatory and IPR requirements, and societal aspects. The value generated by Big Data technologies empowers Artificial Intelligence to foster linking, cross-cutting and vertical dimensions of value creation at the technical, business and societal level across many different sectors.

#### 2.1 BDV PPP Vision and Objectives for European Big Data Value

The Big Data Value Association (BDVA) and the BDV PPP have pursued a common shared vision of positioning Europe as the world leader in the creation of big data value. The BDV PPP vision for Europe in 2020 has concerned the following aspects:

research and innovation efforts will have led to advanced technologies that make it significantly easier to use Big Data across sectors, borders and languages.

The above-addressed aspects were planned to impact the European Union's priority areas as follows:

These three factors were designed to support the major EU pillars as stated in the Rome Declaration of March 2017 (European Council 2017): a safe and secure Europe; a prosperous and sustainable Europe; a social Europe; and a stronger Europe on the world stage.

#### 2.2 Big Data Value Association (BDVA)

The BDVA is an industry-driven and fully self-financed international not-for-profit organisation under Belgian law. The BDVA has over 220 members all over Europe with a well-balanced composition of large, small and medium-sized industries (over 30% of SMEs), as well as research and user organisations. The Big Data Value Association is the private counterpart to the European Commission in implementing the BDV PPP.

BDVA members come together to collaborate on a joint mission: developing the European Big Data Value Ecosystem (BDVe) that will enable the data-driven digital transformation in Europe, delivering maximum economic and societal benefit, and achieving and sustaining Europe's leadership on big data value creation and Artificial Intelligence (Zillner et al. 2019). To achieve this mission, in 2017, the BDVA defined four strategic priorities:

Since 2017 the cross-technological nature of the data value chains, flowing across different technologies (IoT, Cloud, 5G, Cybersecurity, infrastructures, HPC, etc.), has triggered and accelerated the development of stronger collaborations between the BDV PPP/BDVA and other technological (cross-sectorial) sectorial communities and, in particular, other partnerships.

#### 2.3 BDV PPP Objectives

As laid out in the Contractual Arrangement (CA) of the BDV PPP (BDVPPP Contractual Arrangement n.d.), the overarching general objectives are as follows:

Fig. 1 BDV PPP governance structure

#### 2.4 BDV PPP Governance

The main governance structure of the BDV PPP (Fig. 1) was prepared and delivered at the beginning of the PPP to provide the framework for collaboration and alignment among all members of the PPP (EC, funded projects, the Association and its members).

The Cooperation Charter4 was created by the Association as one of the key governance mechanisms to facilitate cooperation among the BDV PPP actions and the BDVA and has been updated every year accordingly.

4 The Cooperation Charter was produced by the BDVA during 2016 and it has been integrated in the CAs or GAs of the Call 1 and Call 2 actions, thereby formalising the actions' commitment to supporting the cooperation within the BDV ecosystem. Latest version: http://www.bdva.eu/sites/ default/files/BDV%20PPP%20COOPERATION%20CHARTER%20January%202019\_approved. pdf

The BDVe project (CSA of the BDV PPP) has supported the implementation of the PPP projects governance structure by establishing the BDV PPP Steering Committee (SC) and the Technical Committee (TC). The Steering Committee (SC) provides executive-level steering and advice to ensure effective and efficient coordination and communication between the BDV PPP actions. The Technical Committee (TC) facilitates knowledge exchange and cooperation on the technical aspects, methodology and implementation of the BDV PPP programme. A non-formal Communication Committee was also established to support cooperation in Marketing and Communications.

The Board of Directors5 (BoD) of the BDVA is selected by the General Assembly of the Association (2-year mandate) and is in charge of achieving the objectives of the association. It follows the resolutions, instructions and recommendations adopted by the General Assembly.

The Partnership Board (PB) is the monitoring body of the PPP formed by selected directors of the Board of the BDVA, and representatives of the European Commission. The PB meets approximately 1–2 times per year and complements this with regular bi-weekly exchanges of information. The European Commission is represented by DG Connect Directorate G (Unit G1 in particular).

#### 2.5 BDV PPP Monitoring Framework

The BDVA leads the production of the Monitoring Report of the Big Data Value PPP as part of its contractual obligations in the PPP. The work is developed by the BDVA TF2 (impact). Since 2019 and in accordance with the European Commission, the full monitoring report of the partnership will only be submitted every 2 years. The most recent version was delivered in 2019 covering the period from beginning 2018 to beginning 2019. The list of key performance indicators (KPIs) for this PPP, description and target values are defined by the following documents:

To produce the monitoring reports the association gathers input from all the running and selected Big Data PPP projects, all for-profit project partners from the projects, the members of the BDVA, the BDVA Task Forces and the BDVA Office, the EC DG CNECT G1 Unit and the European Data Market Monitoring Tool.7

5 List of BoD members: http://www.bdva.eu/board-members

6 http://www.bdva.eu/sites/default/files/BDVPPP\_Contractual\_Arrangement\_.pdf

7 SMART 2016/0063 – Study "Update of the European data marketMOnitoring Tool", IDC and Lisbon Councils.

## 3 Main Activities and Achievements During 2018

## The main achievements of the Big Data Value PPP during 2018 can be

8 http://marketplace.big-data-value.eu/

9 https://landscape.big-data-value.eu/

10All Information about the i-Spaces labelling can be found on the BDVA website. General information: http://bdva.eu/I-Spaces. Labelling process: Information about labelled i-Spaces 2018: http://www.bdva.eu/node/1172

11https://eurohpc-ju.europa.eu/

including on essential topics such as data protection in the era of Artificial Intelligence and use of data in Smart Manufacturing.

### 3.1 Mobilisation of Stakeholders, Outreach, Success Stories

The year 2018 was one of remarkable progress and advancements for the Big Data Value PPP and the BDVA. In its second year of operations, the PPP showed a great quantity and variety of success stories from projects and the association. The main success stories from the projects related to:

12https://ai-data-robotics-partnership.eu/home/

The European Data Incubators/accelerators DataPitch and EDI gave support and new opportunities to 47 start-ups and entrepreneurs, helping them to grow their business in the new Data Economy offering skills development, access to resources, data, infrastructure, ecosystem and additional private funding. This has generated a significant impact on revenues, jobs created and competitiveness.

It is important to highlight the positive effect that participation in a more extensive programme has brought to individual projects. Eighty per cent of the projects reported value created for their Research and Innovation projects by being part of the BDV PPP, e.g. facilitating collaboration and exchanges between projects, such as complementary functionalities (e.g. SLIPO and QROWD), reuse of projects outcomes (functionality, solutions or ontologies, data sharing13 and specific knowhow sharing). Additionally, the PPP is seeking to be effective in coordinating communication activities, providing new opportunities for start-ups, and providing a common framework and vocabulary to develop effective end-to-end ecosystems.

It is also quite remarkable to note the overall impact in communication and engagement of the PPP, with the estimated number of people outreached in dissemination activities around 7.8 million in 2018 with the objective of raising awareness about their different activities, to engage new stakeholders, and communicating the result. Additionally, the BDV PPP organised 181 training activities involving over 18,300 participants during 2018. The range and diversity of actors and stakeholders outreached is very broad, in alignment with the overall objectives of the PPP.

## 4 Monitored Achievements and Impact of the PPP

Enabled by the monitoring framework, as described above, the progress of the BDV PPP is continuously monitored. Below we report the key achievements and impacts in alignment with the development phases described in the SRIA that are backed by the monitoring data.

#### 4.1 Achievement of the Goals of the PPP

According to the Big Data Value PPP SRIA v4,14 the programme would develop the European data ecosystem in three distinct phases of development, each with a primary theme:

13Discussions going on between projects working in same sector.

14And Multi-Annual roadmap version 2017.

The PPP goals achieved are analysed based on the defined roadmap. The year 2018 lies between Phase I and Phase II, and thus the progress of the PPP is assessed considering the objectives of both phases.

Phase I: Establish an Innovation Ecosystem (WP 2016–17) focused on laying the foundations needed to establish a sustainable European data innovation ecosystem (Table 1).

Phase II: Pioneer disruptive new forms of Big Data Value solutions (Lighthouses, technical projects) in high-impact domains of importance for EU industry, addressing emerging challenges of the data economy (WP 18–19). According to the SRIA, this second phase is meant to build on the foundations established in Phase I and will have a primary focus on Research and Innovation (R&I) activities to deliver the next generation of Big Data Value solutions. Although the projects implementing Phase II started in 2019 (or 2020), there are some activities in 2018 supporting the implementation of this stage, in particular those listed in Table 2.

Phase III15: Develop long-term ecosystem enablers to maximise sustainability for economic and societal benefit (WP 19–20). This phase started in late 2019 and will continue until the end of the PPP. As this phase has only just started, the analysis can only be incomplete. Some ideas about possible achievements are provided in Table 3.

#### 4.2 Progress Achieved on KPIs

## 4.2.1 Private Investments

Through this KPI, we attempt to understand and capture/show the level of industrial engagement within the BDV PPP. This KPI includes both direct and indirect leverage, as described in Fig. 2.

Two hundred and ninety-six companies representing all for-profit organisations participating in Big Data Value PPP projects active during 2018 (including not only project partners but also third parties engaged through cascade funding) and all

15Reported as part of the BDVA annual report 2019: https://bdva.eu/sites/default/files/BDVA%20- %20BDVA%20PPP%20Annual%20Report%202019\_v1.1%20for%20publication.pdf

Table 1 Summary achievements of the goals of the BDVA PPP: Phase I of the roadmap

(continued)

### Table 1 (continued)

for-profit organisation members of the BDVA were outreached to provide input to this KPI with an overall response rate of 40.9%.

Table 4 shows the evolution of the reported numbers in private investments from 2015 to 2018, as well as the EU contributions.

Aggregated to the numbers reported in 2015 (€280.9 million), 2016 (€338.5 million) and 2017 (€482.25 million), the amount of mobilised private investments since the launch of the PPP until the end of 2018 was 1569.1M€ (€1.57 billion). Considering the amount of EU funding allocated to the PPP by that time (€201.3 million), the BDV PPP ended 2018 with a leverage factor of 7.8, much higher than the leverage factor of 4 committed contractually.

## 4.2.2 Job Creation, New Skills and Job Profiles

Seventy-seven per cent of the BDV PPP projects indicated that their project would contribute to job creation by 2023, with an estimation in accumulated numbers of thousands. The estimated numbers surpass 7500 new jobs created by 2023 linked to project activities and many more considering indirect effect.

BDV PPP projects contribute to job creation in Europe by (1) increasing the market share of Big Data Technology providers in Europe; (2) developing new job profiles that generate new jobs... the creation; (3) developing new opportunities for entrepreneurs and start-ups in the new Data Economy; (4) generating job opportunities by increasing data sharing; (5) creating new jobs already during the lifetime of the project; and (6) forecasting jobs created as a follow-up of project results.

Table 2 Summary of achievements of the goals of the BDVA PPP: Phase II of the roadmap

Table 3 Summary of achievements of the goals of the BDVA PPP: Phase III of the roadmap

(continued)

### Table 3 (continued)

Fig. 2 Methodology and KPI structure proposed by EC for MR2018 (PPPs) (by European Commission licensed under CC BY 4.0)

Table 4 Evolution of private investments in BDV PPP over time

Input to PPP project investment was 0 before 2017 as no projects had started. The number €12.4 million is calculated based on real input extrapolated from the percentage of responses and expected annual private investment

On the other hand, 40% of the BDVA members stated that their participation in the BDVA/BDV PPP had already contributed directly or indirectly to job creation, mainly because of the hiring of new experts to develop H2020 projects, start-ups created...), and new profiles hired to develop operations.

Projects reported that 48 job profiles were created or identified in 2018, and 106 new job profiles were reported as expected to be created from 2019 onwards and by the end of the project linked to the project activities.

Sixty-seven per cent of the projects running in 2018 reported contribution to the generation of new skills by the end of the project. In addition to the skills linked to the new job profiles, new skills are expected to be developed in cross-sectorial domains (e.g. in the form of "privacy-aware data processing" and "privacy-aware big data innovation" as reported by the SPECIAL project) and in specific sectors (e.g. analysis techniques using weather data, reported by the EW-SHOPP project). The BDV PPP incubators help start-ups to develop both the technical and non-technical skills needed to develop business in the Data Economy.

Among BDVA members, 51% of organisations reported contribution to the creation of new job profiles, and almost 60% contribute to the creation of new skills linked to the Big Data Value PPP. Finally, 60% of the projects and 51% of BDVA members have reported contributions to the Skills Agenda for Europe.

The BDV PPP organised 181 training activities involving over 18,300 participants during 2018. Projects contributed to this with 85 training activities during 2018 involving over 9700 participants. BDVA members reported 96 training activities involving over 8500 participants. Projects developed 16 interdisciplinary programmes during 2018 outreaching 250 participants.

During 2018, 396 equivalent FTEs masters and PhD students "(260 masters and 136 PhD) were involved in PPP projects, thereby collaborating with industrial players in developing industry-driven solutions and deploying experimentation testing scenarios. Contributing to raising awareness in professionals, users and the general public, the BDV PPP organised 323 events outreaching around 630,000 participants during 2018 contributing to raising awareness in professionals, users and the general public.

## 4.2.3 Impact of the BDV PPP on SMEs

Results of the Monitoring Report 2018 showed that a wide range of SMEs in Europe benefit from the Big Data Value PPP, considering the size (12% medium-size companies, 41% small companies and 48% micro-companies16), age (20% of the SMEs are 0 to 4 years old, 36% are 5 to 10 years old and 42% are 10 years old or older) and wide geographical distribution. SMEs play a variety of roles in the data value chain. SMEs participating in PPP projects clearly show a trend of an increase in turnover and in the number of employees. It is also important to mention that not all the SMEs involved in BDV PPP projects are technology companies but are also data users or providers, and the overall results and trend indicate an ongoing growth of turnover along the whole value chain.

Total turnover reported for SMEs in 2017 was €260.4 million.17 In terms of turnover evolution, there is an increase in turnover in the SME companies that are part of the PPP with reported numbers of 60% increase in turnover with respect to 2014 and 17.7% in the last year. This number is in full alignment with the macroeconomic numbers of data companies in Europe, and higher for some specific categories. In particular, young SMEs (5 and 10 years old) show on average the highest growth in turnover in relation to 2014 (up to 284%). The youngest companies (<5 years) show on average the highest growth in the last year (54.8%).

16Criteria for classification following EC rules: http://ec.europa.eu/growth/smes/business-friendlyenvironment/sme-definition\_en

17Aggregated total of the companies.

In terms of employment evolution, the trend is also very positive in all companies that are part of the PPP, with an average increase in employment for the SMEs that are part of the PPP of 75% with respect to 2014 and a growth of 11.83% in the last year (2018 compared with 2017).

Special emphasis should be given to PPP instruments focused on supporting SMEs, in particular the Data Incubators and i-Spaces. The average age of the companies receiving cascade funding from the Data Incubators (DataPitch and EDI) is 4.9 years; 41% of those SMEs are younger than 5 years, 50% are between 5 and 10 years, and only 9% are older than 10 years old. Companies reported an increase in turnover of 315% for 2014 and 48.8% for 2017, and an 118.5% increase in employment for 2014 with a 22.4% increase in the last year.

## 4.2.4 Innovations Emerging from Projects

Innovations arising from the BDV PPP include:

In its second year of operation, the BDV PPP's 32 running projects reported 106 innovations of exploitable value as delivered in 2018: 63% have a medium impact and 37% are considered innovations of significant impact. Fifty per cent of the innovations delivered in 2018 are incremental innovations, 6% are architectural, 36% are disruptive and 1% are radical innovations.18

Ninety-three per cent of the innovations delivered in 2018 have an economic impact, and 48% have a societal impact. 19 Forty-one per cent are technologies

18Eight per cent are not included in any of these categories.

19Note that many innovations have both economic and societal impact.

(including platforms), 32% are services, 7% are products, 8% are methods, 8% are systems, 1% are software, 4% are components and/or modules and 11% are others, including frameworks/architectures, processes, tools and toolkits, spin-offs, datasets, ontologies, patents and knowledge.

Sixteen per cent of the innovations delivered in 2018 are fully cross-sectorial. Sevety-five per cent provide solutions to the transport, mobility and logistics sector (the one with the best coverage in the PPP by the end of 2018); 20% of the innovations related to public services and smart cities; 19% to industry and manufacturing; 14% to bio-economy; 13% to the Telco sector; 12% marketing activities; 8% relate to health and healthcare; 8% to the ICT market; 7% to geospatial market; 5% to commerce; and 3% to others (including fashion, retail, business services, energy, media, compliance, etc.).

In relation to the maturity levels and TRLs, 7% of the innovations delivered are TRL 3 (experimental proof of concept), 10% are TRL 4 (technology validated in lab), 36% are TRL 5 or TRL 6 (technology validated in relevant environment, industrially relevant environment in the case of key enabling technologies), 32% are TRL 7 (system prototype demonstration in operational environment), 8% are TRL 8 (system complete and qualified) and 1% are TRL 9 (actual system proven in

Fig. 3 BDV PPP innovations to market 2018

an operational environment—competitive manufacturing in the case of key enabling technologies—or in space).

Figure 3 provides a full overview of the innovations delivered by the BDV PPP during 2018, combining level of significance, type of innovation (incremental, disruptive, architectural or radical) and the TRLs. Although a large number of innovations are classified as incremental innovations of medium impact, it is remarkable to note the high percentage of significant innovations (and expected growth in the upcoming years), the high number of disruptive innovations and the high TRLs in some cases close to deployment. Although at a lower level, the BDV PPP is also delivering some architectural and radical innovations.

Sixty-three new economically viable services of high societal value were developed during 2018 as a result of the projects. Forty-seven per cent (over 30 projects) contributed to this KPI.

Projects reported 204 new systems and technologies developed during 2018. Many of them are already reported as part of the KPI "Significant Innovations to Market". Systems and technologies developed are not limited to one sector, and, in fact, the majority of the new systems and technologies can be utilised in different sectors/markets, thus stimulating the use of Big Data technologies in many areas.

Finally, many solutions and innovations arising from the Big Data PPP have been promoted in the BDV PPP Marketplace20 developed by the BDVe CSA project to spread knowledge about the outcomes of the PPP.

### 4.2.5 Supporting Major Sectors and Major Domains with Big Data Technologies and Applications

The BDV PPP Lighthouse projects21 active in 2018 focused on the bio-economy (agriculture, fisheries and forestry) (DataBio project), transport, mobility and logistics (transforming transport project), health and healthcare (BigMedilytics project) and manufacturing (BOOST4.0), with a total of four major sectors supported by Lighthouse projects and therefore widely supported by multiple use cases, scenarios and solutions.

Twenty per cent of the projects are fully cross-sectorial (their outcomes can be used in any sector or application domain) and 80% of the projects are working in more than one sector or application domain (this explains why the total is higher than 100% in Table 5). In particular, the BDV PPP projects address a wide variety of sectors22, as shown in Table 5.

20http://marketplace.big-data-value.eu/

21Large-scale data-driven innovation and demonstration projects that aim at creating superior visibility, awareness and impact in specific relevant economic sectors.

22Grouped with a good level of alignment with the NACE registry. These categories are part of the information in the BDV PPP Marketplace that will be used for promoting all exploitable solutions coming out of the PPP projects (if needed, new categories can be added).

Table 5 Support to major sectors and domains

Others (43% of the projects) includes sectors such as insurance, public safety, personal security, public tenders, e-commerce, marketing, fashion industry, citizen engagement, ICT/Cloud services, social networks, procurement and legal domain.

Considering the whole project portfolio, the number of sectors supported is higher than 15, with a solid distribution of use cases, experiments, solutions and outreach activities among different sectors.

## 4.2.6 Experimentation

Projects reported 224 use cases and/or experiments conducted during 2018 with contributions from 18 different projects. This is an increase of 48.3% with respect to 2017 (151 experiments). The BDVA i-Spaces reported an additional 165 experiments with 6 i-Spaces contributing to this KPI.

Projects reported 82 large-scale experiments developed during 2018, 64 involving closed (private) data (78% of the total). Large-scale experiments either involve a large number of users with high TRLs or are developed in large geographical areas, in many cases involving a large number of users and actors or a combination of data volume, complexity and velocity; a large number of data sources; or integrated complex datasets flowing across borders. The BDVA i-Spaces also contributed to this KPI, reporting in total 38 large-scale experiments performed during 2018, 28 of them involving private data.

In relation to the amount of data made available for experimentation, reported information from projects and i-Spaces (members of BDVA) shows that the amount of data made available by the BDV PPP for experimentation in 2018 is 0.10696 Exabytes (106.96 Petabytes). A total of 0.08625 Exabytes (86.25 Petabytes) was reported by the projects.23 It is important to note that some of the projects are not only providing internal access to diverse data sets from different sources but are also improving and creating new valuable datasets (e.g. of DataBio project). BDVA i-Spaces contributed to this KPI, reporting an additional 20.71 Petabytes of data for experimentation.

## 4.2.7 SRIA Implementation and Update

Concerning SRIA coverage, measured as "% of research priorities covered compared to the overall scope of research priorities defined in SRIA", projects have delivered contributions during 2018 already covering 100% of all the SRIA technical priorities. The major focus of technical contributions was "Data Analytics", followed at some distance by "Data Processing Architectures" and "Data Management". This is a significant change from the 2017 coverage, where "Data Management" was the top priority. A clear trend to focus on technical contributions in the areas of "Data Analytics" and "Data Processing Architectures" was anticipated in the BDV PPP Annual Monitoring Report 2017,24 thus supporting our explanation that a solid base of "Data Management" solutions will enable analytics and processing innovations.

In relation to the BDV SRIA update, at the end of 2017 the BDVA released the BDV PPP SRIA v4.0 (detailed process and results reported in the 2017 Monitoring Report). This version was the basis to support the H2020 LEIT ICT WP2018–20. During 2018 a minor update, towards a version 4.1, was released in the community, crystallising in a series of individual deliverables in the format of vision, position or discussion papers that supported the transition towards the next framework programme and the creation of a new strategic agenda and roadmap.

In total, there were at least 12 events organised during 2018 that contributed to input in the BDVA strategic papers – multiple online meetings with a total of 2085 participants/contributions.

In total, since the launch of the BDV PPP, we can count 6422 potential contributions to the strategic roadmapping activities.

23Thirteen projects provided data for this KPI (Aegis, BigDataOcean, DataBio, euBusinessGraph, EW-Shopp, TT, QROWD, BigDataStack, BigMedilytics, Boost 4.0, CLASS, EDI, TheBuyForYou).

24http://www.bdva.eu/sites/default/files/MR2017\_BDV\_PPP\_Main%20Report\_September% 202018\_1.pdf

## 4.2.8 Technical Projects

The BDV PPP contributes to enabling advanced privacy- and security-respecting solutions for data access, processing and analysis. For 2018, 97 contributions were reported (2 patents,25 61 publications and 24 OSS/SW/Products).

Fifty per cent of the projects confirmed that they are assessing quality, diversity and value of data assets. These results show the intense usage of metrics to measure quality, diversity and value of data assets in projects, and some projects have developed specific metrics and methods to ensure quality, diversity and value in the data (e.g. I-BiDaaS has developed a Data Quality Assurance Process (DQAP) aiming at ensuring the high quality of the data generated/collected during the lifetime of the project). However, we cannot talk yet (2018) about the "PPP"-developed metric expected for 2019+.

Concerning the speed of data throughput, 40% of the projects reported that they expect the project to improve data throughput. Some projects, such as BigDataOcean and FashionBrain, measured improvements over 1000%. Others such as I-BiDaaS have specific objectives to develop data processing tools and techniques applicable in real-world settings and to demonstrate a significant increase in speed of data throughput and access.

## 4.2.9 Macro-economic KPIs

The monitoring of macro-economic KPIs is based on input from the European Data Market Monitoring Tool26 as they are presented in the most recent report by IDC (https://www.idc.com/).27

Development of the market share of the European Union in the global Big Data Market. As an indicator, we compare the total revenues of EU Data Companies with other economies, i.e. the US, Japan and Brazil, as they are used as a benchmark in the IDC report.28 The EU share of the total revenues in these economies the 2013 baseline was 27.7%. This share increased slightly to 27.9% in 2018, which is remarkable because the international indicators grew very fast in this period, but the EU kept pace with them. In absolute terms, the total revenue of US data companies in 2018 was approximately twice that of EU28 data companies in the same year (€162 billion vs. €77 billion). Seventy per cent of PPP projects active in 201829 reported contribution to increasing the revenue share of EU companies. Projects contributed by:

25Filled patents.

26SMART 2016/0063 – Study "Update of the European Data Market Monitoring Tool", IDC and Lisbon Councils.

27Gabriella Cattaneo, Giorgio Micheletti et al. "Update of the European Data Market Tool - Second report on Facts and Figures" April 2019 www.datalandscape.eu

28Gabriella Cattaneo et al., ibid. Chap. 10, pp. 129–142.

29Based on number of respondents.

According to the most recent report,30 the number of data companies increased to 283,100 by 2018, compared to 271,700 in 2017, with a growth rate of 4.2%. It should be noted that almost half of them are based in the UK, due to the high concentration of the ICT industry there. BDVA i-Spaces and Data Incubators (ICT 14-b projects, i.e. DataPitchand EDI projects) are in particular designed to contribute to this KPI as they support start-ups and entrepreneurs from early ideas to technical and business development until the go-to-market stage.31 Seventy-seven per cent of the BDV PPP projects active in 201832 reported contribution towards increasing the number of European companies offering data technology and applications. The projects contributed in different ways, such as:

In addition, 25% of BDVA members reported that their organisation ran or supported a programme that is specifically targeted at supporting start-ups or entrepreneurs in the field of Big Data.

The revenue of data companies in the European Union, according to the IDC report,33 reached €77 billion in 2018 compared to €69 billion the previous year, with a growth rate of 12%. The revenue share of SMEs in 2018 amounts to €55.5 billion (72% of the total revenue), an absolute growth of €5.7 billion on the year before. The growth rate of revenue increases in proportion to company size, with the revenue of large companies with over 500 employees growing at 16% in 2018 over 2017. Seventy-seven per cent of the PPP projects active in 201834 reported contribution (or plan to contribute) to the revenue generated by European data companies. Project contribution to this KPI is mainly by:

30Gabriella Cattaneo et al., ibid.

31Further information can be found in Sect. 2.1 of this report.

32Based on number of respondents.

33Gabriella Cattaneo et al., ibid. pp. 89–97.

34Based on number of respondents.

The baseline for data professionals in the European Union in 2013 amounts to 5.77 million. The number of data professionals increased to a total of 7.2 million by 2018, resulting in an absolute growth rate of 1453 million professionals since 2013. The rate of growth of data professionals is increasing, with approximately 559,000 positions added in 2018 and an increase of 8.4% on the year before.35 Eighty-seven per cent of the PPP projects active in 201836 reported contribution from their project to increase the number of data workers in Europe. Projects contribute to this KPI in different ways:

## 4.2.10 Contributions to Environmental Challenges

Over 20% of the projects running in 2018 reported that they contribute to the reduction of energy, and 30% contribute to reduction in CO2 emission. Quantitative results are provided by some projects, such as the Transforming Transport (TT) project that shows that in some specific monitored items improvements in efficiency range between 25% and 51% in energy reduction, and improvements concerning CO2 emissions reach up to 29% and emission reductions in general (including PM and NOx) up to 23%.

The three Lighthouse projects running in 2018 (DataBio, Transforming Transport and Boost4.0) have reported contribution to reduction in waste. For example, in DataBio and in particular in forestry, although still with early data and experiments, the experience from customer cases shows a reduction in waste of up to 10%. Some pilot TT projects show approximately 25% improvement in the management of

35Ibid.

36Based on number of respondents.

assets, which can adequately demonstrate a relative high-level achievement in waste reduction at this final stage of the project.

Seventeen per cent of the projects running in 2018 have reported contribution to reduction in the use of material resources; e.g. BigMedilytics provides quantitative data in a particular scenario, reporting that the Asset Management pilot aims to reduce the number of unused mobile assets in hospitals by up to 20%.

Finally, in relation to energy reduction in big data analytics, there is no quantitative input in results provided by any project but, e.g., the E2Data project develops a framework that optimises calculations, leading to decreased use of energy.

### 4.2.11 Standardisation Activities with European Standardisation Bodies

During 2017, the BDVA and the BDV PPP set up some foundations defining priorities for the PPP in Big Data standardisation implemented during 2018 as follows:

Thirty per cent of the projects running in 2018 reported that they perform activities leading to data/Big data standardisation. Three projects reported contribution to European standardisation bodies (ESBs) activities and reported 11 working items in ESBs. Twenty per cent of BDVA members reported that their organisations perform activities leading to data/Big data standardisation. In particular, BDVA members have reported contributions to IEC, DIN DKE and other consensusbased standardisation bodies; OPC foundation and other consortia-based standardisation bodies; OASIS; W3C committees and community group discussions; open data harmonisation national activities; ISO/IEC JTC1; and defining standards in georeferenced data for geoscience (Open Geospatial Consortium (OGC) and Commission for the Management and Application of Geoscience Information (IUGS/CGI)) and ETSI.

## 5 Summary and Outlook

The year 2018 was a transition year and an important inflexion point between the so-called Phase I (establishment of the ecosystem) and Phase II of the BDV PPP (pioneer disruptive new forms of big data value solutions). New calls for proposals were in place during 2018 and 2019 as part of the H2020 WP 2018–2020 (calls closing in April 2018, November 2018, April 2019 and November 2019) that brought new projects that enriched the BDV PPP portfolio, also increasing challenges of coordination, communication and cooperation. The year 2018 was also a transition year in defining the strategy and direction of the next partnership framework programme (2021–2028).

The increase in the quality and quantity of the data available for experimentation and the launch of the cross-border Industrial Data Platforms and Personal Data Platforms at the beginning of 2020, supported by other ecosystem enablers, have directed the final transition period towards Phase III as defined in SRIA v4. The BDV PPP projects starting in 2020 (e.g. EUHubs4Data project) are establishing a strong foundation for the next framework programme (deployment of data platforms, the federation of Big Data Innovation Hubs/data experimentation facilities, and advances in data and data-driven AI capabilities).

On 25 April 2018, the European Commission outlined a European strategy for AI to boost investment and set ethical guidelines. In its communication, the European Commission put forward a European approach to Artificial Intelligence based on three pillars: (i) "boosting financial support and encouraging uptake by public and private sectors", (ii) "preparing for socio-economic changes brought about by AI", and (iii) "ensuring an appropriate ethical and legal framework". The strategy acknowledged that member states had existing research and innovation objectives that focused on AI and encouraged alignment of individual roadmaps towards a European partnership. Also on 25 April the European Commission proposed a package of measures as a key step towards a common data space in the EU – a seamless digital area with a scale that will enable the development of new products and services based on data.

On 6 June 2018, the European Commission announced its proposal to create the first ever Digital Europe programme and invest €9.2 billion to align the next longterm EU budget 2021–2027 with increasing digital challenges. The Commission's proposal focused on five areas: supercomputers, Artificial Intelligence (AI) (including Data/European Data Space), cybersecurity and trust, digital skills, and ensuring a wide use of digital technologies across the economy and society.

On 7 June 2018, the European Commission announced Horizon Europe (research and innovation programme for the next long-term EU budget 2021–2027) with plans to bring a new generation of European Partnerships and increase collaboration with other EU programmes.

Towards the end of 2018, the BDVA committed its official participation as a private member of the EuroHPC Joint Undertaking aiming at bringing synergy between HPC, Big Data and Artificial Intelligence, and providing industry perspective.

Additionally, the BDVA and euRobotics officially joined forces at the end of 2018 and announced their intentions of working together in a future AI, Data and Robotics Partnership. On 7 December 2018, the European Commission presented a coordinated plan prepared with the members states to foster the development and use of AI in Europe. The plan proposes the development of a European AI public-private partnership building on the BDV PPP and SPARC PPPs.

During 2019 the BDVA and euRobotics developed a common vision paper and the first version of a common AI-PPP Strategic, Research Innovation and Deployment Agenda with strong involvement of ongoing PPP projects, members and many external communities. At the end of 2019, CLAIRE, ELLIS and EurAI joined forces with the BDVA and euRobotics, and the five organisations submitted a joint Partnership Proposal (Zillner et al. 2020). This document lays down the context, vision and objective, and suggests the impact of a possible Partnership of Data, AI and Robotics, building upon the strong assets developed by the BDV PPP and the SPARC PPP. During the first months of 2020, the member states and the European Commission carefully considered the Partnership Proposal and provided feedback for its improvement, which resulted in several updates of the document. On 22 September 2020, the joint release of the Strategic Research and Deployment Agenda (SRIDA v3.0) was published, paving the way towards the new Partnership for Horizon Europe and the Digital Europe Programme, bringing investments and new instruments to scale up the assets and impact of the current Big Data Value PPP.

## References

BDVPPP Contractual Arrangement. (n.d.).

European Council. (2017). The Rome Declaration.

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## Part II Research and Innovation Elements of Big Data Value

## Technical Research Priorities for Big Data

Edward Curry, Sonja Zillner, Andreas Metzger, Arne J. Berre, Sören Auer, Ray Walshe, Marija Despenic, Milan Petkovic, Dumitru Roman, Walter Waterfeld, Robert Seidl, Souleiman Hasan, Umair ul Hassan, and Adegboyega Ojo

Abstract To drive innovation and competitiveness, organisations need to foster the development and broad adoption of data technologies, value-adding use cases and sustainable business models. Enabling an effective data ecosystem requires overcoming several technical challenges associated with the cost and complexity of management, processing, analysis and utilisation of data. This chapter details a community-driven initiative to identify and characterise the key technical research priorities for research and development in data technologies. The chapter examines the systemic and structured methodology used to gather inputs from over 200 stakeholder organisations. The result of the process identified five key technical research priorities in the areas of data management, data processing, data analytics, data

S. Zillner Siemens AG, Munich, Germany

A. Metzger paluno, University of Duisburg-Essen, Duisburg, Germany

A. J. Berre · D. Roman SINTEF Digital, Oslo, Norway

S. Auer Leibniz Universität Hannover, Hannover, Germany

R. Walshe ADAPT SFI Centre for Digital Content, Dublin City University, Dublin, Ireland

M. Despenic ABN AMRO Bank, Amsterdam, the Netherlands

M. Petkovic Philips and Eindhoven University of Technology, Eindhoven, the Netherlands

W. Waterfeld Saarbrücken, Germany

R. Seidl Nokia Bell Labs, Munich, Germany

© The Author(s) 2021 E. Curry et al. (eds.), The Elements of Big Data Value, https://doi.org/10.1007/978-3-030-68176-0\_5

E. Curry (\*) · S. Hasan · U. ul Hassan · A. Ojo

Insight SFI Research Centre for Data Analytics, NUI Galway, Galway, Ireland e-mail<EMAIL_ADDRESS>
visualisation and user interactions, and data protection, together with 28 sub-level challenges. The process also highlighted the important role of data standardisation, data engineering and DevOps for Big Data.

Keywords Research challenges · Data management · Data processing · Data analytics · Data visualisation · User interactions · Data protection · Data standardisation · Data ecosystem

## 1 Introduction

The expectations in refining data as the new oil of the twenty-first century are currently so high that virtually no business can afford not to have a big data project that 'unlocks' the value in their data (Chen et al. 2012). There is a noticeable increase in the adoption of data-driven business scenarios in sectors other than the web-based 'traditional' big data companies such as Google, Yahoo, Facebook and Twitter (Lavalle et al. 2011). However, many sectors still struggle with the adoption of data technologies, often due to a lack of expertise, regulatory barriers and unclear business value. This is especially true in non-IT-focused sectors, such as the energy sector that struggles with the adoption of data technologies (Rusitschka and Curry 2016). The benefits of sharing and linking data across domains and industry are apparent. Initiatives such as Smart Cities are showing how different sectors (i.e. energy and transport) can collaborate to maximise the potential for optimisation and value return (Communication: A European strategy for data 2020). The crossfertilisation of stakeholders and datasets from different sectors is a key element for advancing the data economy.

To support the emergence of a data ecosystem, it was important that the different actors within the ecosystem 'define a shared vision and jointly identify gaps in the current data landscape' (DG Connect 2013). Data ecosystems face several problems such as data discovery, curation, linking, synchronisation, distribution, business modelling, sales and marketing (José María Cavanillas et al. 2016). To address these issues, the Big Data Value contractual Public-Private Partnership (BDV PPP) between the European Commission and the Big Data Value Association aimed to strengthen the data value chain (Curry 2016), foster cooperation in data research and innovation, enhance community building around data and set the groundwork for a thriving data-driven economy in Europe. The BDV PPP was driven by the conviction that research and innovation focusing on a combination of business and usage needs is the best long-term strategy to deliver value from big data and create jobs and prosperity. An essential requirement was to identify and characterise the key technical research challenges that need to be tackled to enable a data ecosystem.

This chapter identifies the key technical research priorities for research and development in data technologies. It presents the results of an investigation and consultation process that was conducted to capture the priorities for big data in public and private organisations across Europe. The chapter starts with an introduction to the methodology for the identification and prioritisation of the technical challenges for the adoption of data technologies. The chapter details the key challenges and outcomes needed in terms of data management, data processing, data analytics, data visualisation and user interaction, and data protection. It highlights the role of standardisation to further the development of data technology and the key role of data standards. Challenges with data engineering and DevOps for big data systems ensure productivity and quality are detailed. Finally, the chapter presents a scenario from the healthcare sector to emphasise the importance of adopting better big data strategies.

## 2 Methodology

In order to correctly identify the technical research priorities a systemic and structured methodology was needed to gather inputs from over 200 stakeholder organisations. The methodology built on and extended an established roadmapping methodology to gather consensus from a range of stakeholders (Curry et al. 2016). The key phases in the methodology, as illustrated in Fig. 1, are (a) technology state of the art and sector analyses, (b) subject matter expert interviews, (c) stakeholder workshops, (d) requirements consolidation and (e) community survey.

#### 2.1 Technology State of the Art and Sector Analysis

The goal of the first phase was to identify the sectorial needs and requirements gathered from different stakeholders and the state of the art of data technologies, as well as identifying research challenges. As part of the investigation, application sectors expressed their need for the technology as well as possible limitations and expectations regarding its current and future deployment. The first step was to perform a systematic literature review based on the following activities:

Fig. 1 The workflow of research methodology

• Synthesisation of the key message of each data source into state-of-the-art descriptions for each identified topic

The following types of data sources were used: scientific papers published in workshops, symposia, conferences, journals and magazines, company white papers, technology vendor websites, open-source projects, online magazines, analysts' data, web blogs other online sources and interviews. The groups focused on sources that mention concrete technologies and analysed them concerning their values and benefits. The synthesis step compared the key messages and extracted agreed views. Topics were prioritised based on the degree to which they can address business needs.

#### 2.2 Subject Matter Expert Interviews

The literature survey was complemented by a series of interviews with subject matter experts for relevant topic areas. Subject matter expert interviews are a technique well suited to data collection and particularly for exploratory research because it allows extensive discussions that illuminate factors of importance (Oppenheim 1992; Yin 2013). The information gathered is likely to be more accurate than information collected by other methods since the interviewer can avoid inaccurate or incomplete answers by explaining the questions to the interviewee (Oppenheim 1992). The interviews followed a semi-structured protocol. The topics of the interview covered different aspects of big data:

Interviewees were selected to be representative of the different stakeholders within the data ecosystem. The selection of interviewees covered (1) established providers of big data technology (typically MNCs), (2) innovative sectorial players who are successful at leveraging big data, (3) new and emerging SMEs in the big data space and (4) world-leading academic authorities in technical areas related to the big data value chain.

The data collection and the analysis strategy were inspired by the triangulation approach (Flick 2004). Reviewing and quantitatively assessing the high-level application scenarios derived a reliable analysis of user needs. Examinations of the likely constraints of big data applications helped to identify the relevant requirements that needed to be addressed.

#### 2.3 Stakeholder Workshops

The third step involved a cross-check and validation of the initial results of the first two steps by involving stakeholders from multiple domains in dedicated workshops and webinars to discuss and review the outcomes. Multiple workshops and consultations took place to ensure the most comprehensive representation of views and positions, including the full range of public and private sector entities not only from technology provision but also technology adoption. Sectoral workshops were conducted in various fields: geospatial/environment, energy, media, mobility, manufacturing, retail, health and the public sector. The purpose was to identify the main priorities with approximately 200 organisations and other relevant stakeholders physically participating and contributing. A wide range of stakeholders contributed to the process with inputs and analysis from SMEs and large enterprises, public organisations, and research and academic institutions. They included suppliers and service providers, data owners and early adopters of big data in many sectors. Extensive analysis reports were then produced, which helped both formulate and reformulate the identified requirements. From the analysis of the results, it was clear that addressing the technical needs of these vertical application markets required a set of cross-sector technologies.

#### 2.4 Requirement Consolidation

Comparison among the different sectors enabled the identification of commonalities and differences at multiple levels. The analysis was used to define integrated crosssectorial priorities that provide a coherent, holistic view of the big data domain and establish a common understanding of requirements, as well as technology descriptions and terms used across domains. A consolidated description was established to align the sector-specific labelling of requirements. In doing so, each sector provided its requirements with the associated user needs. Thus, the initial list of 13 high-level requirements and 28 sub-level requirements could be reduced to 5 high-level requirements and 20 sub-level requirements.

#### 2.5 Community Survey

The objective of the community survey was to engage with the broader community to ensure a comprehensive perspective concerning the technical and business impact of the identified technical priorities, as well as to identify emerging priorities with high impact for the European big data economy. An inclusive approach was taken to ensure stakeholder engagement, with inputs actively solicited from the wider community composed of experts in technical domains as well as in business sectors. The

Fig. 2 Distribution of participants in terms of the type of organisation

Fig. 3 Number of organisations associated with different sectors

survey received participation from a wide range of organisations. In total, 135 organisations responded to the survey through their representatives.

Figure 2 shows the distribution of participants in terms of the type of organisation. The majority of participant organisations (almost 95%) were either private companies or research and academic institutions. The response indicates a broader interest and contribution from stakeholders in shaping the future of the European big data community.

Figure 3 shows the number of organisations working in various sectors. In general, the organisations identified themselves as being active in multiple sectors, which underlines the cross-sectoral perspectives on the technical and non-technical priorities of big data as identified by the survey. Figure 4 shows that more than 70% of the participants chose two or more sectors. On average, more than three different sectors were chosen by participants to indicate the diversity of their portfolio. This

Fig. 4 Histogram of the number of sectors per organisation

Fig. 5 Composition of participating organisations in terms of number of employees (left) and annual revenue (right)

also highlights the need to consider the multidisciplinary nature of the big data economy.

To quantify the size of the organisation, the survey participants were asked to indicate the number of employees (full-time equivalent) and annual revenue. Figure 5 summarises the composition of participating organisations in terms of employees and revenue. Primarily due to participation from the public sector and large corporates, the majority of organisations have more than 200 employees and revenue higher than 10 million. It should be noted that big data challenges for companies with more than 1000 employees are not only limited to their specific sectors but also in their day-to-day operations, such as human resource management and finance. The following section discusses the technical priorities for data technologies, in addition to their ranking based on the community survey.

## 3 Research Priorities for Big Data Value

The first three steps of the methodology produced a set of consolidated crosssectorial technical research requirements. The result of this process was the identification of five key technical research priorities as illustrated in Fig. 6 (data management, data processing architectures, deep analytics, data protection and pseudonymisation, advanced visualisation and user experience), together with 28 sub-level challenges to delivering big data value. In this section, we report on the results of the survey to identify a prioritisation of the cross-sectorial requirements. As far as possible, the roadmaps were quantified using the results of the survey to allow for well-founded prioritisation and action plans, as illustrated in Fig. 7. The remainder of this chapter summaries the technical priorities as defined in the Strategic Research and Innovation Agenda (SRIA) of the BDVA (Zillner et al. 2017).

#### 3.1 Priority 'Data Management'

More and more data are becoming available. This data explosion, often called a 'data tsunami', has been triggered by the growing volumes of sensor data and social data,

Fig. 6 High-level technical priorities for data technologies

Fig. 7 Distribution of high-level technical priorities across participants

born out of Cyber-Physical Systems (CPS) and Internet of Things (IoT) applications. Traditional means for data storage and data management are no longer able to cope with the size and speed of data delivered in heterogeneous formats and at distributed locations.

Large amounts of data are being made available in a variety of formats ranging from unstructured to semi-structured to structured formats, such as reports, Web 2.0 data, images, sensor data, mobile data, geospatial data and multimedia data. For instance, important data types include numeric types, arrays and matrices, geospatial data, multimedia data and text. A great deal of this data is created or converted and further processed as text. Algorithms or machines are not able to process the data sources due to the lack of explicit semantics. In Europe, text-based data resources occur in many different languages, since customers and citizens create content in their local language. This multilingualism of data sources means that it is often impossible to align them using existing tools because they are generally available only in the English language. Thus, the seamless aligning of data sources for data analysis or business intelligence applications is hindered by the lack of language support and gaps in the availability of appropriate resources.

Isolated and fragmented data pools are found in almost all industrial sectors. Due to the prevalence of data silos, it is challenging to accomplish seamless integration with and smart access to the various heterogeneous data sources. And still today, data producers and consumers, even in the same sector, rely on different storage, communication and thus different access mechanisms for their data. Due to a lack of commonly agreed standards and frameworks, the migration and federation of data between pools impose high levels of additional costs. Without a semantic interoperability layer being imposed upon all these different systems, the seamless alignment of data sources cannot be realised.

To ensure a valuable big data analytics outcome, the incoming data has to be high quality; or, at least, the quality of the data should be known to enable appropriate judgements to be made. This requires differentiating between noise and valuable data, and thereby being able to decide which data sources to include and which to exclude to achieve the desired results.

Over many years, several different application sectors have tried to develop vertical processes for data management, including specific data format standards and domain models. However, consistent data lifecycle management – that is, the ability to clearly define, interoperate, openly share, access, transform, link, syndicate and manage data – is still missing. In addition, data, information and content need to be syndicated from data providers to data consumers while maintaining provenance, control and source information, including IPR considerations (data provenance). Moreover, to ensure transparent and flexible data usage, the aggregation and management of respective datasets enhanced by a controlled access mechanism through APIs should be enabled (Data-as-a-Service, or DaaS).

## 3.1.1 Challenges

As of today, collected data is rapidly increasing; however, the methods and tools for data management are not evolving at the same pace. From this perspective, it becomes crucial to have – at a minimum – good metadata, Natural Language Processing (NLP), and semantic techniques to structure the datasets and content, annotate them, document the associated processes, and deliver or syndicate information to recipients. The following research challenges have been identified:

management structures are based on microservices with the possibility of integrating data transformations, data analysis and data anonymisation, in a decentralised manner.

## 3.1.2 Outcomes

The main expected advances in data management are as follows:

#### 3.2 Priority 'Data Processing Architectures'

The Internet of Things (IoT) is one of the key drivers of the big data phenomenon. Initially, this phenomenon started by applying the existing architectures and technologies of big data that we categorise as data-at-rest, which is data kept in persistent storage. In the meantime, the need for processing immense amounts of sensor data streams has increased. This type of data-in-motion (i.e. non-persistent data processed on the fly) has extreme requirements for low-latency and real-time processing. What has hardly been addressed is the concept of complete processing for the combination of data-in-motion and data-at-rest.

For the IoT domain, these capabilities are essential. They are also required for other domains like social networks or manufacturing, where huge amounts of streaming data are produced in addition to the available big datasets of actual and historical data.

These capabilities affect all layers of future big data infrastructures, ranging from the specifications of low-level data, to flows with the continuous processing of micro-messages, to sophisticated analytics algorithms. The parallel need for realtime and large data volume capabilities is a key challenge for big data processing architectures. Architectures to handle streams of data, such as the lambda and kappa architectures, will be considered as a baseline for achieving a tighter integration of data-in-motion with data-at-rest.

Developing the integrated processing of data-at-rest and data-in-motion in an ad hoc fashion is, of course, possible, but only the design of generic, decentralised and scalable architectural solutions leverages their true potential. Optimised frameworks and toolboxes to enable the best use of both data-in-motion (e.g. data streams from sensors) and data-at-rest leverage the dissemination of reference solutions which are ready and easy to deploy in any economic sector. For example, proper integration of data-in-motion with the predictive models based on data-at-rest enable efficient, proactive processing (detection ahead of time). Architectures that can handle heterogeneous and unstructured data are also important. When such solutions become available to service providers, in a straightforward manner, they can focus on the development of business models.

The capability of existing systems to process such data-in-motion and answer queries in real time and for thousands of concurrent users is limited. Special-purpose approaches based on solutions like Complex Event Processing (CEP) are not sufficient for the challenges posed by the IoT in big data scenarios. The problem of achieving effective and efficient processing of data streams (data-in-motion) in a big data context is far from being solved, especially when considering the integration with data-at-rest and breakthroughs in NoSQL databases and parallel processing (e.g. Hadoop, Apache Spark, Apache Flink, Apache Kafka). Applications, for instance of Artificial Intelligence, are also required to fully exploit all the capabilities of modern and heterogeneous hardware, including parallelism and distribution to boost performance.

To achieve the agility demanded by real-time business and next-generation applications, a new set of interconnected data management capabilities is required.

## 3.2.1 Challenges

There have been several advances in big data analytics to support the dimension of big data volume. In a separate development, stream processing has been enhanced in terms of analytics on the fly to cover the velocity aspect of big data. This is especially important as business needs to know what is happening now. The main challenges to be addressed are:

element for effective stream processing. Especially important is efficient distribution of the processing to the Edge (i.e. local data Edge processing and analytics), as a part of the ever-increasing trend of Fog computing.

## 3.2.2 Outcomes

The main expected advances in data processing architectures are:

reacting to dynamic data) and analysing sizable amounts of data to update the analysis results as the information content changes. It is important to access only relevant and suitable data, thereby avoiding accessing and processing irrelevant data. Research should provide new techniques that can speed up training on large amounts of data, for example by exploiting parallelisation, distribution and flexible Cloud computing platforms, and by moving computation to Edge computing.

#### 3.3 Priority 'Data Analytics'

The progress of data analytics is key not only for turning big data into value but also for making it accessible to the wider public. Data analytics have a positive influence on all parts of the data value chain, and increase business opportunities through business intelligence and analytics while bringing benefits to both society and citizens.

Data analytics is an open, emerging field, in which Europe has substantial competitive advantages and a promising business development potential. It has been estimated that governments in Europe could save \$149 billion (Manyika et al. 2011) by using big data analytics to improve operational efficiency. Big data analytics can provide additional value in every sector where it is applied, leading to more efficient and accurate processes. A recent study by the McKinsey Global Institute placed a strong emphasis on analytics, ranking it as the main future driver for US economic growth, ahead of shale oil and gas production (Lund et al. 2013).

The next generation of analytics needs to deal with a vast amount of information from different types of sources, with differentiated characteristics, levels of trust and frequency of updating. Data analytics have to provide insights into the data in a costeffective and economically sustainable way. On the one hand, there is a need to create complex and fine-grained predictive models for heterogeneous and massive datasets such as time series or graph data. On the other hand, such models must be applied in real time to large amounts of streaming data. This ranges from structured to unstructured data, from numerical data to micro-blogs and streams of data. The latter is exceptionally challenging because data streams, aside from their volume, are very heterogeneous and highly dynamic, which also calls for scalability and high throughput. For instance, data collection related to a disaster area can easily occupy terabytes in binary GIS formats, and real-time data streams can show bursts of gigabytes per minute.

In addition, an increasing number of big data applications are based on complex models of real-world objects and systems, which are used in computation-intensive simulations to generate new massive datasets. These can be used for iterative refinements of the models, but also for providing new data analytics services which can process massive datasets.

## 3.3.1 Challenges

Understanding data, whether it is numbers, text or multimedia content, has always been one of the most significant challenges for data analytics. Entering the era of big data, this challenge has expanded to a degree that makes the development of new methods necessary. The following list details the research areas identified for data analytics:

empowering enterprises and other organisations to make accurate and instant decisions to shape their markets. The simplification and automation of these techniques are necessary, especially for SMEs.

## 3.3.2 Outcomes

The main expected advanced analytics innovations are as follows:

#### 3.4 Priority 'Data Visualisation and User Interaction'

Data visualisation plays a key role in effectively exploring and understanding big data. Visual analytics is the science of analytical reasoning assisted by interactive user interfaces. Data generated from data analytics processes need to be presented to end-users via (traditional or innovative) multi-device reports and dashboards which contain varying forms of media for the end-user, ranging from text and charts to dynamic 3D and possibly augmented-reality visualisations. For users to quickly and correctly interpret data in multi-device reports and dashboards, carefully designed presentations and digital visualisations are required. Interaction techniques fuse user input and output to provide a better way for a user to perform a task. Common tasks that allow users to gain a better understanding of big data include scalable zooms, dynamic filtering and annotation.

When representing complex information on multi-device screens, design issues multiply rapidly. Complex information interfaces need to be responsive to human needs and capacity (Raskin 2000). Knowledge workers need to be supplied with relevant information according to the just-in-time approach. Too much information, which cannot be efficiently searched and explored, can obscure the most relevant information. In fast-moving, time-constrained environments, knowledge workers need to be able to quickly understand the relevance and relatedness of information.

## 3.4.1 Challenges

In the data visualisation and user interaction domain, the tools that are currently used to communicate information need to be improved due to the significant changes brought about by the expanding volume and variety of big data. Advanced visualisation techniques must therefore consider the range of data available from diverse domains (e.g. graphs or geospatial, sensor and mobile data). Tools need to support user interaction for the exploration of unknown and unpredictable data within the visualisation layer. The following list briefly outlines the research areas identified for visualisation and user interaction:

## 3.4.2 Outcomes

The main expected advances in visualisation and user experience are as follows:

• Scalable data visualisation approaches and tools: To handle extremely large volumes of data, the interaction must focus on aggregated data at different scales of abstraction rather than on individual objects. Techniques for summarising data in different contexts are highly relevant. There is a need to develop novel interaction techniques that can enable easy transitions from one scale or form of aggregation to another (e.g. from neighbourhood level to city level) while supporting aggregation and comparisons between different scales. It is necessary to address the uncertainty of the data and its propagation through aggregation and analysis operations.

#### 3.5 Priority 'Data Protection'

Data protection and anonymisation is a significant issue in the areas of big data and data analytics. With more than 90% of today's data having been produced in the last 2 years, a huge amount of person-specific and sensitive information from disparate data sources, such as social networking sites, mobile phone applications and electronic medical record systems, is increasingly being collected. Analysing this wealth and volume of data offers remarkable opportunities for data owners, but, at the same time, requires the use of state-of-the-art data privacy solutions, as well as the application of legal privacy regulations, to guarantee the confidentiality of individuals who are represented in the data. Data protection, while essential in the development of any modern information system, becomes crucial in the context of large-scale sensitive data processing.

Recent studies on mechanisms for protecting privacy have demonstrated that simple approaches, such as the removal or masking of the direct identifiers in a dataset (e.g. names, social security numbers), are insufficient to guarantee privacy. Indeed, such simple protection strategies can be easily circumvented by attackers who possess little background knowledge about specific data subjects. Due to the critical importance of addressing privacy issues in many business domains, the employment of privacy-protection techniques that offer formal privacy guarantees has become a necessity. This has paved the way for the development of privacy models and techniques such as differential privacy, private information retrieval, syntactic anonymity, homomorphic encryption, secure search encryption and secure multiparty computation, among others. The maturity of these technologies varies, with some, such as k-anonymity, more established than others. However, none of these technologies has so far been applied to large-scale commercial data processing tasks involving big data.

In addition to the privacy guarantees that can be offered by state-of-the-art privacy-enhancing technologies, another important consideration concerns the ability of the data protection approaches to maintain the utility of the datasets to which they are applied, to support different types of data analysis. Privacy solutions that offer guarantees while maintaining high data utility will make privacy technology a key enabler for the application of analytics to proprietary and potentially sensitive data.

There is a need for a truly modern and harmonised legal framework on data protection which has teeth and can be enforced appropriately to ensure that stakeholders pay attention to the importance of data protection. At the same time, it should enable the uptake of big data and incentivise privacy-enhancing technologies, which could be an asset for Europe as this is currently an underdeveloped market. In addition, users are beginning to pay more attention to how their data are processed. Hence, firms operating in the digital economy may realise that investing in privacyenhancing technologies could give them a competitive advantage.

## 3.5.1 Challenges

In this perspective, the following main challenges have been identified:

• A more generic, easy-to-use and enforceable data protection approach suitable for large-scale commercial processing is needed. Data usage should conform to current legislation and policies. On the technical side, mechanisms are needed to provide data owners with the means to define the purpose of information gathering and sharing and to control the granularity at which their data will be shared with authorised third parties throughout the lifecycle of the data (data-in-motion and data-at-rest). Moreover, citizens should be able, for example, to have a say over the destruction of their personal data (the right to be forgotten). Data protection mechanisms also need to be 'easy', or at least capable of being used and understood with a reasonable level of effort by the various stakeholders, especially the end-users. Technical measures are also needed to enable and enforce the auditability of the principle that the data is only used for the defined purpose and nothing else – in particular, in relation to controlling the usage of personal information. In distributed settings such as supply chains, distributed trust technologies such as blockchains can be part of the solution.

## 3.5.2 Outcomes

The main expected advances in data protection are as follows:

• Complete data protection framework: A good mechanism for data protection includes protecting the Cloud infrastructure, analytics applications and the data from leakage and threats, but also provides easy-to-use privacy mechanisms. Apart from the specification of the intended use of data, usage control mechanisms should also be covered.

## 4 Big Data Standardisation

Standardisation is a fundamental pillar in the construction of a Digital Single Market and Data Economy. It is only through the use of standards that the requirements of interconnectivity and interoperability can be ensured in an ICT-centric economy. Further development of technology and data standards for big data is needed by:

Standards are the essential building blocks for product and service development as they define clear protocols that can be easily understood and adopted internationally. They are a prime source of compatibility and interoperability and simplify product and service development as well as speeding the time-to-market. Standards are globally adopted; they make it easier to understand and compare competing products, and thus drive international trade.

In the data ecosystem, standardisation applies to both the technology and the data.

Technology Standardisation Most technology standards for big data processing are de facto standards that are not prescribed (but are at best described after the fact) by a standards organisation. However, the lack of standards is a major obstacle. One example is the NoSQL databases. The history of NoSQL is based on solving specific technology challenges that lead to a range of different storage technologies. The broad range of choices, coupled with the lack of standards for querying the data, makes it harder to exchange data stores, as this may tie application-specific code to a specific storage solution. A pragmatic approach to standardisation is needed by influencing, in addition to NoSQL databases, the standardisation of technologies such as complex event processing for real-time data applications, languages to encode the extracted knowledge bases, Artificial Intelligence, computation infrastructure, data curation infrastructure, query interfaces and data storage technologies.

Data Standardisation The 'variety' of big data makes it very difficult to standardise. Nevertheless, there is a great deal of potential for data standardisation in the areas of data exchange and data interoperability. The exchange and use of data assets are essential for functioning ecosystems and the data economy. Enabling the seamless flow of data between participants (i.e. companies, institutions and individuals) is a necessary cornerstone of the ecosystem. Collaborative efforts are needed to support, where possible and pragmatic, the definition of semantic standardised data representation, ranging from domain (industry sector)-specific solutions, like domain ontologies, to general concepts such as Linked Open Data, to simplify and reduce the costs of data exchange.

## 5 Engineering and DevOps for Big Data

Big data technologies have gained significant momentum in research and innovation. However, mature, proven and empirically sound engineering methodologies for building next-generation big data value systems are not yet available. Also, we lack proven approaches for continuous development and operations (DevOps) of big data value systems. The availability of engineering methodologies and DevOps approaches – combined with adequate toolchains and big data platforms – will be essential for fostering productivity and quality. As a result, these methodologies and approaches will empower the new wave of data professionals to deliver high-quality next-generation big data value systems.

#### 5.1 Challenges

Engineering and DevOps toolchains for big data value systems need to look at and systematically integrate a diverse set of aspects for: (1) system/software engineering, (2) development and operations and (3) quality assurance.

The main challenges to be addressed include:

#### 5.2 Outcomes

The expected primary outcomes for engineering and DevOps are:

## 6 Illustrative Scenario in Healthcare

This section illustrates how the technical priorities may help in delivering big data solutions for specific industry sectors. To this end, we present a scenario from the healthcare sector. A BDVA white paper collected and analysed the needs, opportunities and challenges for big data technologies in healthcare (TF7 Healthcare subgroup 2016).

There is a clear opportunity to transform healthcare by applying data technologies. To improve the productivity of the healthcare sector, it is necessary to reduce costs while maintaining or improving the quality of the care provided. The fastest, least costly and most effective way to achieve this is to use the knowledge that is hiding within the already existing large amounts of generated medical data. According to current estimates, medical data is already at the zettabyte scale and will soon reach the yottabyte (e.g. 1000 zettabytes, a billion petabytes) scale. While most of this data was previously stored in hard copy format, the current trend is towards digitisation of these large amounts of information, thus making them amenable to analysis, resulting in what is known as big data.

The challenges and needs for research and innovation in this illustrative scenario are quite evident for each of the technical priorities listed above. Let's consider them one by one, starting with data management.

• Data management: Access to high-quality, large healthcare datasets to optimise care processes, disease diagnosis, personalised care and the healthcare system in general. Furthermore, a real transformation of the healthcare sector can only be achieved if all stakeholders and verticals in the healthcare sector (the HealthTech industry, healthcare providers, pharma, and insurance) share data and allow free data flow. Topics such as data quality, semantic interoperability and data management lifecycles are of the utmost importance in breaking down data silos in healthcare.

## 7 Summary

Enabling an effective data ecosystem requires overcoming several technical challenges associated with the cost and complexity of extracting value from data. This chapter identifies and characterises the key research areas. A systemic and structured methodology was used to gather inputs from over 200 stakeholder organisations. The results of this process, as illustrated in Fig. 8, identify the five technical research priorities together with 28 sub-challenges of big data. The requirement analysis was done in consultation with a community of stakeholders that included organisations for industry, research and government.

The results presented in this chapter provide a prioritised list of cross-sectorial business needs of data technologies and their impact in industry, research and government. These findings serve as a guide for directing the research and development efforts towards fostering a data ecosystem. The findings indicate that deep analytics and data management are viewed as the top two technical challenges for big data, with more than 60% of organisations prioritising them as having a high

Fig. 8 High-level technical priorities and sub-challenges for big data value

impact on the data ecosystem. Although data privacy was considered a significant challenge, it was ranked lowest compared to other key challenges. This may be because not all data applications and domains have privacy implications and may focus on industrial/machine data.

Finally, these data research priorities have laid the foundations for a joint Strategic Research, Innovation and Deployment Agenda for an AI, Data and Robotics Partnership in Europe (Zillner et al. 2020) with the goal to unify the strategic focus of each of the three disciplines engaged in creating the Partnership.

Acknowledgements We greatly acknowledge the collective effort of the SRIA teams: Carlos A. Iglesias, Antonio Alfaro, Jesus Angel, Sören Auer, Paolo Bellavista, Arne Berre, Freek Bomhof, Stuart Campbell, Geraud Canet, Giuseppa Caruso, Edward Curry, Paul Czech, Davide Dalle Carbonare, Nuria de Lama, Stefano de Panfilis, Thomas Delavallade, Marija Despenic, Ana Garcia Robles, Wolfgang Gerteis, Aris Gkoulalas-Divanis, Nuria Gomez, Paolo Gonzales, Thomas Hahn, Souleiman Hasan, Jim Keneally, Bjarne Kjær Ersbøll, Bas Kotterink, Yannick Legré, Yves Mabiala, Julie Marguerite, Dirk Mayer, Ernestina Menasalves, Andreas Metzger, Elisa Molino, Thierry Nagellen, Dalit Naor, Maria Perez, Milan Petkovic, Roberta Piscitelli, Klaus-Dieter Platte, Pierre Pleven, Dumitru Roman, Titi Roman, Alexandra Rosén, Nikos Sarris, Stefano Scamuzzo, Simon Scerri, Corinna Schulze, Robert Seidl, Bjørn Skjellaug, Caj Södergård, Claire Tonna, Francois Troussier, Colin Upstill, Josef Urban, Meilof Veeningen, Tonny Velin, Ray Walshe, Walter Waterfeld, Stefan Wrobel, and Sonja Zillner.

## References

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## A Reference Model for Big Data Technologies

Edward Curry, Andreas Metzger, Arne J. Berre, Andrés Monzón, and Alessandra Boggio-Marzet

Abstract The Big Data Value (BDV) Reference Model has been developed with input from technical experts and stakeholders along the whole big data value chain. The BDV Reference Model may serve as a common reference framework to locate big data technologies on the overall IT stack. It addresses the main technical concerns and aspects to be considered for big data value systems. The BDV Reference Model enables the mapping of existing and future data technologies within a common framework. Within this chapter, we detail the reference model in more detail and show how it can be used to manage a portfolio of research and innovation projects.

Keywords Reference model · Big data technologies · Data management · Data processing · Data analysis · Data visualisation · Data protection

## 1 Introduction

The Big Data Value (BDV) Reference Model has been developed with input from technical experts and stakeholders along the whole big data value chain. The BDV Reference Model may serve as a common reference framework to locate big data technologies on the overall IT stack. It addresses the main concerns and aspects to be considered for big data value systems. Within this chapter, we detail the reference

E. Curry (\*)

A. Metzger paluno, University of Duisburg-Essen, Duisburg, Germany

A. J. Berre SINTEF Digital, Oslo, Norway

A. Monzón · A. Boggio-Marzet Universidad Politécnica de Madrid, Madrid, Spain

Insight SFI Research Centre for Data Analytics, NUI Galway, Galway, Ireland e-mail<EMAIL_ADDRESS>
Fig. 1 Big Data Value Reference Model

model in more detail and show how it can be used to manage a portfolio of research and innovation projects. Section 2 details the Reference Model with its horizontal and concerns. Section 3 describes the use of the Reference Model within large-scale data projects to map projects' technical outcomes. Finally, Sect. 4 concludes the chapter.

## 2 Reference Model

An overview of the BDV Reference Model is shown in Fig. 1. It distinguishes between two different elements. On the one hand, it describes the elements that are at the core of the BDVA (also see Chap. "The European Big Data Value Ecosystem"); on the other, it outlines the features that are developed in strong collaboration with related European activities.

The BDV Reference Model has been developed by the Big Data Value Association (BDVA), taking into account input from technical experts and stakeholders along the whole big data value chain, as well as interactions with other related public-private partnerships (PPPs) ( Zillner et al. 2017). The BDV Reference Model may serve as a common reference framework to locate big data technologies on the overall IT stack. It addresses the main concerns and aspects to be considered for big data value systems.

The BDV Reference Model is structured into horizontal and vertical concerns.

It should be noted that the BDV Reference Model has no ambition to serve as a technical reference architecture. However, it is compatible with such reference architectures, most notably the emerging ISO JTC1 WG9 Big Data Reference Architecture.

The following elements as expressed in the BDV Reference Model are elaborated in the remainder of this section.

#### 2.1 Horizontal Concerns

Horizontal concerns cover specific aspects of a big data system. On the one hand, they cover the different elements of the data processing chain, starting from data collection and ingestion up to data visualisation and user interaction. On the other hand, they cover elements that facilitate deploying and operating big data systems, including Cloud and HPC, as well as Edge and IoT.

## 2.1.1 Data Visualisation and User Interaction

This concern covers advanced visualisation approaches for improved user experience. Data visualisation plays a key role in effectively exploring and understanding big data. Visual analytics is the science of analytical reasoning assisted by interactive user interfaces. Data generated from data analytics processes need to be presented to end-users via (traditional or innovative) multi-device reports and dashboards which contain varying forms of media for the end-user, ranging from text and charts to dynamic, 3D and possibly augmented-reality visualisations. In order for users to quickly and correctly interpret data in multi-device reports and dashboards, carefully designed presentations and digital visualisations are required. Interaction techniques fuse user input and output to provide a better way for a user to perform a task. Common tasks that allow users to gain a better understanding of big data include scalable zooms, dynamic filtering and annotation.

When representing complex information on multi-device screens, the design issues multiply rapidly. Complex information interfaces need to be responsive to human needs and capacity (Raskin 2000). Knowledge workers need to be supplied with relevant information according to the just-in-time approach. Too much information, which cannot be efficiently searched and explored, can obscure the information that is most relevant. In fast-moving time-constrained environments, knowledge workers need to be able to quickly understand the relevance and relatedness of information.

## 2.1.2 Data Analytics

This concern covers data analytics, which ranges from descriptive analytics ("What happened and why?") through predictive analytics ("What will happen and when?") to prescriptive analytics ("What is the best course of action to take?"). The progress of data analytics is key not only for turning big data into value but also for making it accessible to the wider public. Data analytics will have a positive influence on all parts of the data value chain (Cavanillas et al. 2016) and increase business opportunities through business intelligence and analytics while bringing benefits to both society and citizens.

Data analytics is an open, emerging field, in which Europe has strong competitive advantages and a promising business development potential. It has been estimated that governments in Europe could save \$149 billion (Manyika et al. 2011) by using big data analytics to improve operational efficiency. Big data analytics can provide additional value in every sector where it is applied, leading to more efficient and accurate processes. A study by the McKinsey Global Institute placed a strong emphasis on analytics, ranking it as the main future driver for US economic growth, ahead of shale oil and gas productions (Lund et al. 2013).

The next generation of analytics will be required to deal with a vast amount of information from different types of sources, with differentiated characteristics, levels of trust and frequency of updating. Data analytics will have to provide insights into the data in a cost-effective and economically sustainable way. On the one hand, there is a need to create complex and fine-grained predictive models for heterogeneous and massive datasets such as time series or graph data. On the other hand, such models must be applied in real time to large amounts of streaming data. This ranges from structured to unstructured data, from numerical data to micro-blogs and streams of data. The latter is exceptionally challenging because data streams, in addition to their volume, are very heterogeneous and highly dynamic, which also calls for scalability and high throughput. For instance, data collection related to a disaster area can easily occupy terabytes in binary GIS formats, and real-time data streams can show bursts of gigabytes per minute.

In addition, an increasing number of big data applications are based on complex models of real-world objects and systems, which are used in computation-intensive simulations to generate new huge datasets. These can be used for iterative refinements of the models, but also for providing new data analytics services which can process extremely large datasets.

## 2.1.3 Data Processing Architectures

This concern covers optimised and scalable architectures for analytics of both dataat-rest and data-in-motion, thereby delivering low-latency real-time analytics.

The Internet of Things (IoT) is one of the key drivers of the big data phenomenon. Initially, this phenomenon started by applying the existing architectures and technologies of big data that we categorise as data-at-rest, which is data kept in persistent storage. In the meantime, the need for processing immense amounts of sensor data streams has increased. This type of data-in-motion (i.e. non-persistent data processed on the fly) has extreme requirements for low-latency and real-time processing. What has hardly been addressed is the concept of complete processing for the combination of data-in-motion and data-at-rest.

For the IoT domain, these capabilities are essential. They are also required for other domains like social networks or manufacturing, where huge amounts of streaming data are produced in addition to the available big datasets of actual and historical data.

These capabilities will affect all layers of future big data infrastructures, ranging from the specifications of low-level data flows with the continuous processing of micro-messages, to sophisticated analytics algorithms. The parallel need for realtime and large data volume capabilities is a key challenge for big data processing architectures. Architectures to handle streams of data such as the lambda and kappa architectures will be considered as a baseline for achieving a tighter integration of data-in-motion with data-at-rest.

Developing the integrated processing of data-at-rest and data-in-motion in an ad hoc fashion is of course possible, but only the design of generic, decentralised and scalable architectural solutions will leverage their true potential. Optimised frameworks and toolboxes allowing the best use of both data-in-motion (e.g. data streams from sensors) and data-at-rest will leverage the dissemination of reference solutions which are ready and easy to deploy in any economic sector. For example, proper integration of data-in-motion with predictive models based on data-at-rest will enable efficient, proactive processing (detection ahead of time). Architectures that can handle heterogeneous and unstructured data are also important. When such solutions become available to service providers, in a straightforward manner, they will then be free to focus on the development of business models.

The capabilities of existing systems to process such data-in-motion and answer queries in real time and for thousands of concurrent users are limited. Specialpurpose approaches based on solutions like Complex Event Processing (CEP) are not sufficient for the challenges posed by the IoT in big data scenarios. The problem of achieving effective and efficient processing of data streams (data-in-motion) in a big data context is far from being solved, especially when considering the integration with data-at-rest and breakthroughs in NoSQL databases and parallel processing (e.g. Hadoop, Apache Spark, Apache Flink, Apache Kafka). Applications, for instance of Artificial Intelligence, are also required to fully exploit all the capabilities of modern and heterogeneous hardware, including parallelism and distribution to boost performance.

To achieve the agility demanded by real-time business and next-generation applications, a new set of interconnected data management capabilities is required.

## 2.1.4 Data Protection

This concern covers privacy and anonymisation mechanisms to facilitate data protection. This is shown related to data management and processing as there is a strong link here, but it can also be associated with the area of cybersecurity.

Data protection and anonymisation is a major issue in the areas of big data and data analytics. With more than 90% of today's data having been produced in the last 2 years, a huge amount of person-specific and sensitive information from disparate data sources, such as social networking sites, mobile phone applications and electronic medical record systems, is increasingly being collected. Analysing this wealth and volume of data offers remarkable opportunities for data owners, but, at the same time, requires the use of state-of-the-art data privacy solutions, as well as the application of legal privacy regulations, to guarantee the confidentiality of individuals who are represented in the data. Data protection, while essential in the development of any modern information system, becomes crucial in the context of large-scale sensitive data processing.

Recent studies on mechanisms for protecting privacy have demonstrated that simple approaches, such as the removal or masking of the direct identifiers in a dataset (e.g. names, social security numbers), are insufficient to guarantee privacy. Indeed, such simple protection strategies can be easily circumvented by attackers who possess little background knowledge about specific data subjects. Due to the critical importance of addressing privacy issues in many business domains, the employment of privacy-protection techniques that offer formal privacy guarantees has become a necessity. This has paved the way for the development of privacy models and techniques such as differential privacy, private information retrieval, syntactic anonymity, homomorphic encryption, secure search encryption and secure multiparty computation, among others. The maturity of these technologies varies, with some, such as k-anonymity, more established than others. However, none of these technologies has so far been applied to large-scale commercial data processing tasks involving big data.

In addition to the privacy guarantees that can be offered by state-of-the-art privacy-enhancing technologies, another important consideration concerns the ability of the data protection approaches to maintain the utility of the datasets to which they are applied, with the goal of supporting different types of data analysis. Privacy solutions that offer guarantees while maintaining high data utility will make privacy technology a key enabler for the application of analytics to proprietary and potentially sensitive data.

A truly modern and harmonised legal framework on data protection which has teeth and can be enforced appropriately will ensure that stakeholders pay attention to the importance of data protection. At the same time, it should enable the uptake of big data and incentivise privacy-enhancing technologies, which could be an asset for Europe as this is currently an underdeveloped market. In addition, users are beginning to pay more attention to how their data are processed. Hence, firms operating in the digital economy may realise that investing in privacy-enhancing technologies could give them a competitive advantage.

## 2.1.5 Data Management

This concern covers principles and techniques for data management, including data ingestion, sharing, integration, cleansing and storage. More and more data are becoming available. This data explosion, often called a "data tsunami", has been triggered by the growing volumes of sensor data and social data, born out of Cyber-Physical Systems (CPS) and Internet of Things (IoT) applications. Traditional means for data storage and data management are no longer able to cope with the size and speed of data delivered in heterogeneous formats and at distributed locations.

Large amounts of data are being made available in a variety of formats – ranging from unstructured to semi-structured to structured – such as reports, Web 2.0 data, images, sensor data, mobile data, geospatial data and multimedia data. Important data types include numeric types, arrays and matrices, geospatial data, multimedia data and text. A great deal of this data is created or converted and further processed as text. Algorithms or machines are not able to process the data sources due to the lack of explicit semantics. In Europe, text-based data resources occur in many different languages, since customers and citizens create content in their local language. This multilingualism of data sources means that it is often impossible to align them using existing tools because they are generally available only in the English language. Thus, the seamless aligning of data sources for data analysis or business intelligence applications is hindered by the lack of language support and gaps in the availability of appropriate resources.

Isolated and fragmented data pools are found in almost all industrial sectors. Due to the prevalence of data silos, it is challenging to accomplish seamless integration with and smart access to the various heterogeneous data sources. And still today, data producers and consumers, even in the same sector, rely on different storage, communication and thus different access mechanisms for their data. Due to the lack of commonly agreed standards and frameworks, the migration and federation of data between pools impose high levels of additional costs. Without a semantic interoperability layer being imposed upon all these different systems, the seamless alignment of data sources cannot be realised.

In order to ensure a valuable big data analytics outcome, the incoming data has to be of high quality, or, at least, the quality of the data should be known to enable appropriate judgements to be made. This requires differentiating between noise and valuable data, and thereby being able to decide which data sources to include and which to exclude to achieve the desired results.

Over many years, several different application sectors have tried to develop vertical processes for data management, including specific data format standards and domain models. However, consistent data lifecycle management – that is, the ability to clearly define, interoperate, openly share, access, transform, link, syndicate and manage data – is still missing. In addition, data, information and content need to be syndicated from data providers to data consumers while maintaining provenance, control and source information, including IPR considerations (data provenance). Moreover, to ensure transparent and flexible data usage, the aggregation and management of respective datasets enhanced by a controlled access mechanism through APIs should be enabled (Data-as-a-Service).

## 2.1.6 Cloud and High-Performance Computing (HPC)

Efficient big data processing, data analytics and data management require the effective use of Cloud and High-Performance Computing infrastructures to address the computational resource and storage needs of big data systems.

Cloud Data ecosystems, promoted by the BDVA, should include strong links to scientific research that is becoming predominantly data driven. The BDVA is in a strong position to nurture such links as it has established strong relationships with European big data academia. However, a lack of access, trust and reusability prevents European researchers in academia and industry from gaining the full benefits of data-driven science. Most datasets from publicly funded research are still inaccessible to the majority of scientists in the same discipline, not to mention other potential users of the data, such as company R&D departments. Approximately 80% of research data is not in a trusted repository. However, even if the data openly appears in repositories, this is not always enough. As a current example, only 18% of the data in open repositories is reusable.1 This leads to inefficiencies and delays; in recent surveys, the time reportedly spent by data scientists in collecting and cleaning data sources made up 80% of their work (G. Press 2016).

In response to these challenges, the Commission has launched a large effort to create "a European Open Science Cloud to make science more efficient and productive and let millions of researchers share and analyse research data in a trusted environment across technologies, disciplines and borders" 1 . The initial outline for the European Open Science Cloud (EOSC) was laid out in the report from the High-Level Expert Group.2 The report advised the Commission on several measures needed to implement the governance and the financial scheme of the European Open Science Cloud, such as being based on a federated system of existing and emerging research (e-)infrastructures operating under light international governance with well-defined Rules of Engagement for participation. Machine understanding of

1 "Are FAIR data principles FAIR?" LIBER Webinar by Alastair Dunning, 10.03.2017.

2 Realising the European Open Science Cloud, 2016, https://ec.europa.eu/research/openscience/pdf/ realising\_the\_european\_open\_science\_cloud\_2016.pdf

data – based on common or widely used data standards – is required to handle the exponential growth in publications. Attractive career paths for data experts should be created through proper training and by applying modern reward and recognition practices. This should help to satisfy the growing demand for data scientists working together with substance scientists. Turning science into innovation is emphasised, and alongside this there is a need for industry, especially SMEs and start-ups, to be able to access the appropriate data resources.

A first phase aims at establishing a governance and business model that sets the rules for the use of the EOSC, creating a cross-border and multi-disciplinary open innovation environment for research data, knowledge and services, and ultimately establishing global standards for the interoperability of scientific data.

The EU has already initiated and will go on to launch several more infrastructure projects, such as EOSC-hub, within H2020 for implementing and piloting the EOSC. In addition to these projects, Germany and the Netherlands, among other countries, are promoting the GO FAIR initiative (Germany and the Netherlands 2017). The FAIR principles aim to ensure that Data and Digital Research Objects are Findable, Accessible, Interoperable and Reusable (FAIR) (Wilkinson et al. 2016). As science becomes increasingly data driven, making data FAIR will create real added value since it allows for combining datasets across disciplines and across borders to address pressing societal challenges that are mostly interdisciplinary.

The GO FAIR initiative is a bottom-up, open-to-all, cross-border and crossdisciplinary approach aiming to contribute to a broad involvement of the European science community as a whole, including the "long tail" of science.

The EOSC initiative is aligned with the BDVA agenda, as both promote data accessibility, trustworthiness and reproducibility over domains and borders. In the BDVA, this mainly applies to the i-Spaces and Lighthouse instruments, where the interoperability of datasets is central. Data standardisation is a self-evident topic for cooperation, but there are also common concerns in non-technical priorities – most notably skills development (relating to data-intensive engineers and data scientists). Both industry and academia benefit from findable, accessible, interoperable and reproducible data.

High-Performance Computing In some sectors, big data applications are expected to move towards more computation-intensive algorithms to reap deeper insights across descriptive (explaining what is happening), diagnostic (exploring why it happens), prognostic (predicting what can happen) and prescriptive (proactive handling) analysis. The adoption of specific HPC-type capabilities by the big data analytics stack is likely to be of assistance where big data insights will be of the utmost value. Faster decision-making is crucial and extremely complex datasets are involved – i.e. extreme data analytics.

The Big Data and HPC communities (through BDVA and ETP4HPC collaboration1 ) have recognised their shared interests in strengthening Europe's position regarding extreme data analytics. Recent engagements between PPPs have focused on the relevant issues of looking at how HPC and Big Data platforms are implemented, understanding the platform requirements for HPC and Big Data workloads, and exploring how the cross-transfer of certain technical capabilities belonging to either HPC or big data could benefit each other. For example, the application of deep learning is one such workload that readily stands to benefit from certain HPC-type capabilities regarding optimising and parallelising difficult optimisation problems.

Major technical requirements include highly scalable performance, high memory bandwidth, low power consumption and excellent short arithmetic performance. Additionally, more flexible end-user education paths, utilisation and business models will be required to capitalise on the rapidly evolving technologies underpinning extreme data analytics, as well as continued support for collaboration across the communities of both big data and HPC to jointly define the way forward for Europe.

## 2.1.7 IoT, CPS, Edge and Fog Computing

The main source of big data is sensor data from an IoT context and actuator interaction in Cyber-Physical Systems. To meet real-time needs, it will often be necessary to handle big data aspects at the edge of the system. This area is separately elaborated further in collaboration with the IoT (Alliance for Internet of Things Innovation (AIOTI)) and CPS communities.

Internet of Things (IoT) technology, which enables the connection of any type of smart device or object, will have a profound impact on many sectors in the European economy. Fostering this future market growth requires the seamless integration of IoT technology (such as sensor integration, field data collection, Cloud, Edge and Fog computing) and big data technology (such as data management, analytics, deep analytics, edge analytics and processing architectures).

The mission of the Alliance of Internet of Things Innovation (AIOTI) is to foster the European IoT market uptake and position by developing ecosystems across vertical silos, contributing to the direction of H2020 large-scale pilots, gathering evidence on market obstacles for IoT deployment in the Digital Single Market context, championing the EU in spearheading IoT initiatives, and mapping and bridging global, EU and Members States' IoT innovation and standardisation activities. AIOTI working groups cover various vertical markets from smart farming to smart manufacturing and smart cities, and specific horizontal topics on standardisation, policy, research and innovation ecosystems. The AIOTI was launched by the European Commission in 2015 as an informal group and established as a legal entity in 2016. It is a major cross-domain European IoT innovation activity.

Close cooperation between the AIOTI and the BDVA is seen as being very beneficial for the BDVA. The following areas of collaboration are of particular interest to the BDVA:

• Alignment of high-level reference architectures: A common understanding of how the AIOTI High-Level Architecture (HLA) and the BDVA Reference Model are related to each other enables well-grounded decisions and prioritisations related to the future impact of technologies.

Aligning Security Efforts The efforts to strengthen security in the IoT domain will have a huge impact on the integrity of data in the big data domain. When IoT security is compromised, so too is the generated data. By developing a mutual understanding on security issues in both domains, trust in both technologies and their applications will be increased.

#### 2.2 Vertical Concerns

Vertical concerns address cross-cutting issues, which are relevant and may affect more than one of the horizontal concerns. They may not be purely technical and also involve some non-technical aspects.

## 2.2.1 Big Data Types and Semantics

One specific vertical concern defined by the BDV Reference Model is data types. Different data types may require the use of different techniques and mechanisms in the horizontal concerns, for instance for data analytics and data storage.

The following six big data types have been identified as the main relevant data types used in big data systems: (1) structured data, (2) time series data, (3) geospatial data, (4) media data (image, video, audio, etc.), (5) text data (including natural language data and genomics representations) and (6) graph or network data. In addition, it is important to support both the syntactical and semantic aspects of data for all big data types, in particular, considering metadata.

## 2.2.2 Standards

This concern covers the standardisation of big data technology areas to facilitate data integration, sharing and interoperability.

Standardisation is a fundamental pillar in the construction of a Digital Single Market and Data Economy. It is only through the use of standards that the requirements of interconnectivity and interoperability can be ensured in an ICT-centric economy. The PPP will continue to lead the way in the development of technology and data standards for big data by:

Standards are the essential building blocks for product and service development as they define clear protocols that can be easily understood and adopted internationally. They are a prime source of compatibility and interoperability and simplify product and service development as well as speeding the time-to-market. Standards are globally adopted; they make it easier to understand and compare competing products, and thus drive international trade.

In the data ecosystem, standardisation applies to both the technology and the data.

Technology Standardisation Most technology standards for big data processing are de facto standards that are not prescribed (but are at best described after the fact) by a standards organisation. However, the lack of standards is a significant obstacle. One example is the NoSQL databases. The history of NoSQL is based on solving specific technology challenges that lead to a range of different storage technologies. The broad range of choices, coupled with the lack of standards for querying the data, makes it harder to exchange data stores, as this may tie application-specific code to a specific storage solution. The PPP is likely to take a pragmatic approach to standardisation and look to influence, in addition to NoSQL databases, the standardisation of technologies such as complex event processing for real-time big data applications, languages to encode the extracted knowledge bases, Artificial Intelligence, computation infrastructure, data curation infrastructure, query interfaces and data storage technologies.

Data Standardisation The "variety" of big data makes it very difficult to standardise. Nevertheless, there is a great deal of potential for data standardisation in the areas of data exchange and data interoperability. The exchange and use of data assets are essential for functioning ecosystems and the data economy. Enabling the seamless flow of data between participants (i.e. companies, institutions and individuals) is a necessary cornerstone of the ecosystem.

To this end, the PPP is likely to undertake collaborative efforts to support, where possible and pragmatic, the definition of semantic standardised data representation, ranging from the domain (industry sector)-specific solutions, like domain ontologies, to general concepts, such as Linked Open Data, to simplify and reduce the costs of data exchange.

In line with JTC1 Directives Clause <IP_ADDRESS>, the Big Data Value Association (BDVA) requested the establishment of a Category C liaison with the ISO/IEC JTC1/WG9 Big Data Reference Architecture. This request was processed at the August Plenary meeting of ISO IEC JTC1 WG9, and the recommendation was unanimously approved by the working group. This liaison moves the BDVA work forward from a technology standardisation viewpoint, and now the BDVA Big Data Reference Model is closely aligned with the ISO Big Data Reference Architecture, as described in ISO IEC JTC1 WG9 20547-3. The BDVA TF6SG6 Standardisation Group is now also in the process of using the WG9 Use Case Template to extract data from the PPP Projects to extend the European use case influence on the ISO big data standards.

As the data ecosystem overlaps with many other ecosystems, such as Cloud computing, IoT, smart cities and Artificial Intelligence, the PPP will continue to be a forum for bringing together industry stakeholders from across these other domains to collaborate. These fora will continue to drive interoperability within the big data domain but will also extend this activity across the other technological ecosystems.

## 2.2.3 Communication and Connectivity

This concern covers effective communication and connectivity mechanisms, which are necessary for providing support for big data. This area is separately further elaborated, along with various communication communities, such as the 5G community.