The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Chapter 9 Big Data and Cloud Computing**

**Yun Li, Manzhu Yu, Mengchao Xu, Jingchao Yang, Dexuan Sha, Qian Liu and Chaowei Yang**

**Abstract** Big data emerged as a new paradigm to provide unprecedented content and value for Digital Earth. Big Earth data are increasing tremendously with growing heterogeneity, posing grand challenges for the data management lifecycle of storage, processing, analytics, visualization, sharing, and applications. During the same time frame, cloud computing emerged to provide crucial computing support to address these challenges. This chapter introduces Digital Earth data sources, analytical methods, and architecture for data analysis and describes how cloud computing supports big data processing in the context of Digital Earth.

**Keywords** Geoscience · Spatial data infrastructure · Digital transformation · Big data architecture

#### **9.1 Introduction**

Digital Earth refers to the virtual representation of the Earth we live in. It represents the Earth in the digital world from data to model. Data are collected and models are abstracted to build the digital reality. Massive amounts of data are generated from various sensors deployed to observe our home planet while building Digital Earth. The term "big data" was first presented by NASA researchers to describe the massive amount of information that exceeds the capacities of main memory, local disk, and even remote disk (Friedman 2012). According to the National Institute of Standards and Technology (NIST), "*Big Data is a term used to describe the large amount of data in the networked, digitized, sensor*-*laden, information*-*driven world*" (Chang and Grady 2015). This definition refers to the bounty of digital data from various data sources in the context of Digital Earth, which focus on big data's geographical aspects of social information, Earth observation (EO), sensor observation service (SOS), cyber infrastructure (CI), social media and business information (Guo 2017; Guo et al. 2017; Yang et al. 2017a, b). Digital Earth data are collected from satellites,

Y. Li · M. Yu · M. Xu · J. Yang · D. Sha · Q. Liu · C. Yang (B)

Department of Geography and GeoInformation Science, NSF Spatiotemporal Innovation Center, George Mason University, Fairfax, VA, USA e-mail<EMAIL_ADDRESS>
<sup>©</sup> The Editor(s) (if applicable) and The Author(s) and European Union 2020 H. Guo et al. (eds.), *Manual of Digital Earth*, https://doi.org/10.1007/978-981-32-9915-3\_9

**Table 9.1** Definition of the "9Vs" of big data

sensors, simulation models, mobile phones, utilities, vehicles, and social networks in different formats, e.g., imagery, text, video, sound, geometries and combinations of them (Yang et al. 2017a, b). Digital Earth data are naturally big data because of the variety of data sources and enormous data volume.

The increasing availability of big Earth data has provided unprecedented opportunities to understand the Earth in the Digital Earth context. In recent research, big data have been characterized by 5 Vs (volume, variety, velocity, veracity, and value) (Gantz and Reinsel 2011; Zikopoulos and Barbas 2012; Marr 2015). Firican (2017) extended the 5 Vs into big data characteristics including variability, validity, vulnerability, volatility and visualization (as defined in Table 9.1 and further elaborated below).

#### Volume

The volume of remote sensing imagery collected by satellites and drones easily reaches the TB and PB levels. For example, the Integrated Multi-satellite Retrievals for GPM (IMERG) data product records global precipitation information every half hour, producing up to 3.45 TB data yearly (Huffman et al. 2015). Other locationbased data such as social media (e.g., Twitter) and VGI (e.g., OpenStreetMap) are constantly growing.

#### Variety

Data sources include sensors, digitizers, scanners, numerical models, mobile phones, the Internet, videos, emails, and social networks in the context of Digital Earth. All types of geospatial data require a more effective data structure, framework, index, model, management methodology, and tactics. In addition, these geospatial data are formatted in various data models, e.g., vector and raster, structured and unstructured.

#### Velocity

The speed of Earth data collection and generation has increased with the development of advanced techniques such as drone observation for disaster monitoring. For example, with the massive number of object-based sensors in the IoT, the data generation of IoT nodes is fast since most sensors continuously generate data in real time.

#### Veracity

The accuracy of geospatial data varies by data source (Li et al. 2016). Taking precipitation as an example, the quality of remote sensing images such as TRMM and IMERG depends on the sensor configuration, calibration methods, and retrieval algorithms. Precipitation information in MERRA (Modern Era Retrospective-analysis for Research and Applications) data relies on the sophistication of meteorological models. Stationary data collected by rain gauges are more accurate even though they are sparse.

#### Validity

Similar to veracity, validity concerns the accuracy and correctness of Earth data for the intended usage. In addition to data selection in which data are chosen with appropriate spatial and temporal resolutions and variables for a specific application, data preprocessing, e.g., data augmentation, interpolation, outlier detection, also play an important role in uncovering information from big Earth data. Consistent data quality, common definitions and metadata can benefit the community, resulting in Earth data of high validity.

#### Variability

Variability refers to the continuous change in the meaning of data in the context of big Earth data, particularly for data that relies on natural language processing. For example, Twitter data emerged as an additional source for natural disaster management (Yu et al. 2018), as tweets posted during disasters can be collected to aid situational awareness. The meaning of words constantly changes over time, for example, the word "Irma" may be a name but started to represent the strongest observed hurricane in the Atlantic in most tweets around October 2017.

#### Vulnerability

Security is a challenging aspect because some geospatial data contain identifiable information or are sensitive. For example, cellular data have been widely utilized to analyze human activities in smart city applications, however, showing phone numbers may divulge people's private affairs.

#### Volatility

Volatility refers to the timeliness and freshness of Earth data, i.e., how long the Earth data stay useful and relevant to applications and how long the data should be kept. Due to the velocity and volume of big Earth data, it is impossible to store all the data in a live database without any performance issues. A series of rules should be established for data currency, availability and rapid retrieval (Firican 2017), e.g., historical and less frequently visited Earth data could be archived on a lower-cost tier of storage.

#### Visualization

Visualization of Earth data is a challenging task with limited memory due to poorly scalable, low-functionality, and high-velocity datasets. Traditional methods may fail to render billions of points, polylines and polygons when visualizing geospatial vector data, therefore graphical methods, e.g., data clustering, parallel coordinates, cone tree or circular network diagrams, should be used to represent Earth data (Firican 2017).

#### Value

Value presents a low-density pattern in the current big data ecosystem where only a small portion of data is utilized in practice. Earth data occupies 80%+ of our data assets (Dempsey 2012), but most datasets are not excavated and are under-utilized. With appropriate spatiotemporal resolution and analysis methods, the 9Vs have been addressed to obtain actionable knowledge to increase the value of big data.

Data collection strategies, data storage facilities, data analysis methods, and data access services facilitate the transformation from the other 9Vs to the 10th V of value. With the continuing increases in the volume and complexity of data, there are challenges in the life cycle of data management, including data storage, data query, data analysis, data sharing, and many other aspects. Managing big data requires an extensible, interoperable and scalable architecture that supports data storage and analysis. Fortunately, recent years have witnessed the evolution of cloud computing, which brings potential solutions to support the life cycle of big data management.

Cloud computing is a new computing paradigm for delivering computation as a fifth utility, which became popular earlier than big data (Yang et al. 2011a). It has the features of elasticity, pooled resources, on-demand access, self-service and pay-as-you-go characteristics (Mell and Grance 2011) and was termed spatial cloud computing in the context of Digital Earth (Yang et al. 2011a). Big data technologies, e.g., big data storage and big data analytics, evolve and benefit significantly from their integration with cloud computing.

To provide a comprehensive overview of how cloud computing supports big data in the context of Digital Earth, this chapter introduces Digital Earth data sources (Sect. 9.2), data analysis methods (Sect. 9.3), architecture for big data analysis (Sect. 9.4), and cloud computing and its support of big data management (Sect. 9.5). Two examples of EarthCube and Data Cube are introduced in Sect. 9.6 to exemplify cloud-based big data frameworks in the Digital Earth context.

#### **9.2 Big Data Sources**

With the advanced developments in Earth observation systems, various Earth data have been gathered at a high velocity from five major sources: (1) remote sensing, (2) in situ sensing, (3) simulation, (4) social media, and (5) infrastructure management (Fig. 9.1). Each covers more than one characteristic of big data. This section discusses the five data sources.

Remote sensing data

Referring to the USGS's definition, remote sensing is the process of detecting and monitoring the physical characteristics of an area by measuring the reflected and

**Fig. 9.1** Big Earth data sources: **a** remote sensing data (JPL 2001); **b** in situ data (NOAA 2017); **c** simulation data (Lipponen 2017); **d** social media data (Gundersen 2013); and **e** infrastructure data (Canada Line Vancouver Transit Map 2019; Robert 2000)

emitted radiation at a distance from the targeted area (USGS 2019). Such remotely observed data serve as a vital source for tracking natural phenomena, the growth of a city, changes in farmland or forest, and discovery of the rugged topography of the ocean floor. According to the Earth Observing System Data and Information System (EOSDIS) 2014 statistics, EOSDIS manages over 9 PB of data and adds 6.4 TB of data to its archives every day (NASA 2016). As data precision and density increase over time, data volume increases exponentially. In addition, ongoing international initiatives monitor the Earth in near-real time using satellites to support rapid data collection for quick and effective emergency response (Zhang and Kerle 2008). Remote sensing data are big data due to the big volume, variety, veracity and volatility.

#### In situ data

According to NOAA (National Oceanic and Atmospheric Administration), in situ data are measurements made at the actual location of the object. In contrast to remote sensing, in situ sensing harvests data directly at the observation location, and often provides continuous data streams to reflect the actual situation with very low latency. Examples of such measurements are (1) tall tower networks (NOAA ESRL/GMD) that provide regionally representative measurements of carbon dioxide (CO2) and related gases and (2) moored and drifting buoys for marine/ocean data collection. In situ data are big data considering the volume, velocity, and veracity.

#### Simulation data

Simulation datasets or reanalysis datasets refer to the outputs of Earth models (e.g., climate) based on geophysical principles. By assimilating observations with models, better initial conditions can be leveraged and simulation results can be significantly improved, especially for short-term predictions. Simulation datasets can be used in various applications. For example, the precipitation, evaporation, and runoff from MERRA datasets can drive river flow models and enhance the study of sensitive ecosystems such as estuaries (Rienecker et al. 2011). In addition, the reanalysis winds used in transport models support the evaluation of aerosols. Simulation data are big data due to its volume, variety and validity.

#### Social media data

In recent years, social media has become one of the most popular sources of big data and provides valuable insights on event trends and people's references. Social networks such as Twitter and Facebook generate a vast amount of geo-tagged data every second and are transforming social sciences (Yang et al. 2017a). Scientists from economics, political science, social science, and geoscience domains utilize big data mining methods to detect social interactions and analyze health records, phone logs, and government records (Balakrishna 2012). For example, in Digital Earth, social media and crowdsourcing data can provide trends of the urban flooding events or wildfire spread, as well as support near-real time situational awareness when other types of data are limited or hard to obtain. However, social media data have high uncertainty and vary in format and quality. Tweet content analysis highly relies on natural language processing, but word meaning constantly changes. Social media data are big data due to its volume, velocity, variety, veracity, validity, variability and volatility.

#### Infrastructure data

Infrastructure data serve as a vital data source of Digital Earth information, especially for developing smart cities. For example, basic infrastructure data (e.g., utility, transportation, and energy), healthcare data and governance data (e.g., environmental and construction management) should be proposed, planned and provided by local official departments and business agencies for a smart city (Hashem et al. 2016). Some infrastructure data may contain sensitive information. Taking water distribution management systems as an example, a synthetic data methodology was proposed to reproduce water consumption data according to privacy constraints (Kofinas et al. 2018). With the upgrades in infrastructure, Internet of Things (IoT) data, geo-tagged or geo-referenced data are continuously produced by various devices, sensors, systems and services (Boulos and Al-Shorbaji 2014). In the near future, various applications based on IoT data will benefit individuals and society. For example, near-real time data including temperature and wind information gathered by IoT sensors could support real-time urban microclimate analysis (Rathore et al. 2017). Infrastructure data are big data due to its volume, velocity, variety, veracity, vulnerability, validity and volatility.

Earth data are continuing to grow in volume and complexity. Big data analytical methods are utilized to mine actionable knowledge from big Earth data to convert the 9Vs of Earth data to the 10th V, which is discussed in the next section.

#### **9.3 Big Data Analysis Methods**

The advancements in remote sensing, social networking, high-performance simulation modeling and in situ monitoring provide unprecedented big data about our planet. The large volume and variety of data offer an opportunity to better understand the Earth by extracting pieces of knowledge from these data. This section discusses data analysis methods from the three aspects of data preprocessing, statistical analysis and nonstatistical analysis. The characteristics, applications, and challenges of these methods are introduced below.

#### *9.3.1 Data Preprocessing*

Real-world data are usually incomplete, noisy and inconsistent due to data collection limitations and sensor issues. Raw data may contain errors or outliers, lack specific attributes or have discrepancies in the descriptions. Therefore, data preprocessing (e.g., data cleaning, fusion, transformation, and reduction) are required to remove noise, correct data, or reduce data size.

Low-quality values (missing values, outliers, noises, inconsistent values) in raw data are often removed or replaced with user-generated values, e.g., interpolation values. The missing value is usually represented with a symbol (e.g., N/A) in raw data and easily recognize. Outliers and inconsistent values are hidden in the raw data and can be detected through statistical analysis. Taking water usage behavior analysis as an example, data preprocessing is necessary to turn smart water meter data into useful water consumption patterns because IoT sensors may fail to record data (Söderberg and Dahlström 2017).

Data transformation also plays an essential role in data preprocessing. Multiple Digital Earth data, e.g., climate data, soil moisture data, crop data, are converted to monthly z-score data before analysis to eliminate the seasonal trends that usually make the patterns of interest undiscoverable. Aggregation, another important data transformation method, groups data based on numerical attributes (Heuvelink and Pebesma 1999). In the Earth science domain, aggregating raw data to the county or state levels could uncover essential patterns for decision making, urban planning, and regional development.

Another trend in Digital Earth data analysis is multisource data fusion, which provides comprehensive data retrieved from several data sources. Generally, vector and raster data store Earth information with different spatial-temporal resolutions; thus, data must be converted to the same resolution by interpolating the lower resolution data or aggregating the higher resolution data for intelligent analysis to investigate scientific questions at a specific scale. Sharifzadeh and Shahabi (2004) introduced a spatial aggregation method that takes the sensor data distribution into account. Spatial interpolation is interpolation of point and areal data. Point interpolation is applied to contour mapping and areal interpolation is used in isopleth mapping (Lam 1983). In addition to spatial interpolation, temporal interpolation predicts values between timestamps (Lepot et al. 2017).

#### *9.3.2 Statistical Analysis*

In the era of big data, statistical analysis is a common mathematical method of information extraction and discovery. Statistical methods are mathematical formulas, models, and techniques used to find patterns and rules from raw data (Schabenberger and Gotway 2017). Data mining is the process of discovering patterns from large datasets involving statistical analysis. Through data mining, historical data can be transformed into knowledge to predict relevant phenomena. Both traditional statistics and data mining methods are discussed in this section. These methods include but are not limited to regression analysis, spatiotemporal analysis, association rules, classification, clustering, and deep learning.

#### **Regression analysis**

Regression models the relationships between a dependent variable and one or more explanatory variables (Yoo et al. 2014; Anderson 2015) by estimating the values of a dependent variable when the values of the independent variables are known and the relationships exist. Regression models describe the strength or weakness of the relationship between several variables. For example, Blachowski (2016) proposed a weighted spatial regression method that identified four significant factors inducing land subsidence: thickness, inclination, the depth of coal panels, and the slope of the surface. There are challenges in spatial data analysis using regression methods, especially for situations that are complicated enough to result in serious residuals in the regression models.

#### **Spatiotemporal analysis**

Spatiotemporal data analysis investigates the trajectories and trends of spatiotemporal data. Li et al. (2016) investigated the spatiotemporal trends in the fluctuations of housing price data. Spatial data analytics and modeling techniques were used to identify the spatial distribution of housing prices at the micro level and explore the space-time dynamics of residential properties in the market, as well as the detected geographic disparity in terms of housing prices. Rahman and Lateh (2017) analyzed the temperature and rainfall time series data from 34 meteorological stations distributed throughout Bangladesh over 40 years (1971–2010) to statistically evaluate the magnitude of temperature and rainfall changes across space and time. Spatiotemporal analysis is still in its initial stage of development. Challenging questions remain, such as what kinds of patterns can be extracted from time series data and which methods and algorithms should be applied.

#### **Association rule**

Association rule learning is the process of discovering strong relationships between variables, i.e., rules, in a large database using measurements of support and confidence (Agrawal et al. 1993). In Digital Earth, Yang (2011b, 2016) applied association rules to mine the variables of Atlantic hurricanes from 1980 to 2003 and discovered a combination of factors related to rapid intensification probability, the low vertical shear of the horizontal wind (SHRD = L), high humidity in the 850–700 hPa range (RHLO = H), and tropical cyclones in an intensification phase (PD12 = H). Compared with traditional statistical methods, the rule-based mining method can find combinations of factors instead of a single factor related to an event.

#### **Classification**

Classification learning is the task of mapping input variables to discrete output variables called labels or categories, for example, 'building' or 'road.' It is the process of recognizing, differentiating and understanding objects. Support Vector Machine (SVM) is a classical classification algorithm in which a kernel-based metric is used to differentiate objects. Jiang et al. (2018b) integrated the ranking support vector machine (RankSVM) model from the computer science community with ocean data attributes to support data ranking in ocean portals. An SVM model is also used to predict geological lithofacies from wireline logs.

#### **Clustering**

Clustering is the process of splitting a set of objects into closely related groups, and each group is regarded as a cluster. Objects falling in the same cluster are more similar to each other than those in other clusters. In Digital Earth, clustering plays an important role in pattern analysis. Hong and O'Sullivan (2012) clustered empirical datasets in Auckland, New Zealand for ethnic residential cluster detection, which is useful to understand contemporary immigrants and ethnic minorities in urban areas. Zhao et al. (2017) proposed a method to generate urban road intersection models from low-frequency GPS trajectory data. These patterns identified from empirical data are crucial for urban transportation planning and management.

#### **Deep learning**

As a new paradigm of machine learning, deep learning has achieved remarkable success in discovery of implicit knowledge (LeCun et al. 2015). In Digital Earth, deep learning algorithms have been adopted to solve domain problems. For example, Guo and Feng (2018) used multiscale and hierarchical deep convolutional features to assign meaningful semantic labels to the points in a three-dimensional (3D) point cloud, which is essential for generating 3D models. Li and Hsu (2018) proposed a deep learning approach to automatically identify terrain features (i.e., sand dunes, craters) from remote sensing imagery. Compared with traditional induction-based approaches, the deep learning approach could detect diverse and complex terrain features more accurately and process massive available geospatial data more efficiently.

#### *9.3.3 Nonstatistical Analysis*

In addition to statistical analysis, nonstatistical analysis methods also play an essential role in helping us descriptively understand Earth phenomena. This section introduces two representative models in Digital Earth, linked data and 3D city modeling.

#### **Linked data**

Linked data are structured data in which datasets are interlinked in the collection, which is useful for semantic queries and reasoning (Bizer et al. 2011). With linked data, data are sharable, and the relationships among the data are recorded. Standard web technologies such as RDF (Resource Description Framework) provide a way to build shareable linked data. In Digital Earth, heterogeneous Earth data (multidisciplinary, multitemporal, multiresolution, and multilingual) can be integrated based on linked data principles for decision making and knowledge discovery (Vilches-Blázquez et al. 2014). For example, Mc Cutchan (2017) proposed a structure of embedding geographic data into linked data and forecasted spatial phenomena with associated rules extracted from the linked data.

#### **3D city modeling**

A trend in Digital Earth analysis is to build a real 3D model with the aid of a computer, especially for cities where most human activities occur. 3D models provide real three-dimensional information for analysis, going beyond simple visualization of 3D objects. 3D informatics has become a cornerstone for a series of city-related applications such as urban planning, skyline analysis, crisis and disaster management, route selection and navigation (El-Mekawy 2010). For example, Amirebrahimi et al. (2016) assessed and visualized flood damage using 3D urban modeling and a building information model (BIM), improving the resilience of the community to floods using detailed 3D information.

Knowledge distillation from Earth data has demonstrated excellent improvements in our understanding of the planet we live. As Earth data increase faster than ever, state-of-the-art analysis methods should be developed to handle the increasingly complicated spatiotemporal data. In addition, an extensible, interoperable and scalable architecture is a prerequisite for massive geographic data analysis, and we present a big data analysis architecture in the next section.

#### **9.4 Architecture for Big Data Analysis**

To support Earth data access/query/analysis in a reasonable response time, it is crucial to build a sophisticated analytical platform with robust architecture to reveal insights from the data (Yang et al. 2017a, b). Generally, the architecture of analytical platforms consists of a data storage layer, a data query layer, a data processing layer, and a visualization layer (Fig. 9.2).

**Fig. 9.2** Architecture for big data analyses

#### *9.4.1 Data Storage Layer*

Digital Earth is heavily involved in processing big data from Earth observations and model simulations, and these vast amounts of high-dimensional array-based spatiotemporal data pose challenges for data storage (Li et al. 2017b). Customizations are indispensable in integrating advanced data storage technologies with big Earth data storage, as general distributed file systems such as Hadoop are not designed to store spatiotemporal data.

A robust and stable data storage framework is the foundation of the data analysis architecture. A series of research efforts focused on optimizing spatiotemporal data storage in distributed file systems or databases. For example, Hu et al. (2018a) reorganized NetCDF (Rew and Davis 1990), a standard data format for array-based raster data, into CSV files and deployed them within SciDB (Cudre-Mauroux et al. 2009), a scalable multidimensional array clustering database. Zhao et al. (2010) converted NetCDF data into CDL (network Common data form Description Language) files and distributed them on HDFS (Hadoop Distributed File System). MERRA data, which store reanalysis Earth climatic variables in NetCDF format, were transformed into Hadoop Sequence Files to be processed by standard MapReduce functions (Duffy et al. 2012). Li et al. (2015) decomposed array-based raster data and stored them with HBase, a NoSQL database built upon HDFS in a cloud computing environment for efficient data access and query.

To enable efficient big data query, logical query capabilities have been proposed to support spatiotemporal query of array-based models such as SciHadoop (Buck et al. 2011). A spatiotemporal index was designed to efficiently retrieve and process big array-based raster data using MapReduce and a grid partition algorithm atop the index to optimize the MapReduce performance (Li et al. 2017a). SciHive was developed as an extension of Hadoop Hive, mapping arrays in NetCDF files to a table and calculating the value range for each HDFS to build a distributed adaptive index (Geng et al. 2013, 2014). Malik (2014) introduced a Z-region index into GeoBase to facilitate array-based data storage.

#### *9.4.2 Data Query Layer*

To help data consumers efficiently discover data from the massive available Earth data, the Digital Earth communities have built various data portals to improve the discovery, access, and usability of Earth data. The portals are normally supported by text search and spatiotemporal search and include the GeoPortal,<sup>1</sup> GeoNetwork2 Spatial Web Portal (Xu et al. 2011), Global Earth Observation System of Systems GEOSS Clearinghouse (Liu et al. 2011; Nativi et al. 2015; Giuliani et al. 2017),

<sup>1</sup>https://www.geoportal.gov.pl/.

<sup>2</sup>https://geonetwork-opensource.org/.

GeoSearch (Lui et al. 2013) and many others. For example, GEOSS is a cloudbased framework for global and multidisciplinary Earth observation data sharing, discovery, and access (Nativi et al. 2015). In the framework, datasets or workflows are registered into shared collections or global catalogs, allowing for end users to search for workflows and datasets across multiple granularity levels and disciplines.

In addition, open-source information retrieval frameworks, e.g., Apache Lucene or its variants such as Solr and Elasticsearch (McCandless et al. 2010), were adopted to establish an Earth data portal instead of implementing a search engine for Earth data from scratch. Lucene uses the Boolean model and the practical scoring function to match documents to a query. Solr and Elasticsearch improve the Lucene index to enable big data search capabilities. The Physical Oceanography Distributed Active Archive Center (PO. DAAC) serves the oceanographic community with 514 collection level datasets and massive granule level data atop Solr (Jiang et al. 2018a). Elasticsearch is the fundamental component of NOAA's OneStop portal in which data providers manage data and metadata with increased discoverability and accessibility.

However, solely relying on open source solutions is insufficient for Earth data discovery because these solutions only rely on a keyword-based relevance score for ranking and ignore other user preferences, e.g., data processing level, sensor type. A few related research efforts have been conducted in the Earth science domain to make data search engines smarter and more intelligent. For example, an algorithm combing Latent Semantic Analysis (LSA) and a two-tier ranking was reported to build a semantic-enabled data search engine (Li et al. 2014a, b). Jiang et al. (2018a) developed a smart web-based data discovery engine that mines and utilizes data relevancy from metadata and user behavior data. The engine enables machine-learned ranking based on several features that can reflect users' search preferences.

#### *9.4.3 Data Processing Layer*

Data processing layer is a core component of the data analytics architecture. To analyze terabyte and petabyte datasets with low time latency, even in a real-time manner, sophisticated parallel computing algorithms and scalable computing resources are required in the big data processing framework (Yang et al. 2015a). Advanced opensource parallel computing solutions, e.g., Hadoop MapReduce, Spark, and their variants in the Earth data domain have been leveraged to support data analysis and mining tasks with better performance.

Hadoop MapReduce is a high-performance batch processing parallel framework that solves large computational problems on distributed storage systems (White 2012). It transfers the algorithm code to data nodes rather than moving data blocks to a compute node to avoid I/O bottlenecks. Spark enables high-performance data analysis with in-memory computing. An in-memory data structure called the Resilient Distributed Dataset (RDD) manages datasets distributed in a Spark cluster (Zaharia et al. 2012). However, the original distributed frameworks have limitations on big spatiotemporal data processing. Yu et al. (2015) noted that the system scalability for spatiotemporal data and interactive performance are the two main challenges for big Earth data processing. To solve these problems, scientists and engineers have customized open-source solutions for spatiotemporal data analysis.

SpatialHadoop is a MapReduce-based framework with native support for spatial data including a simple spatial high-level language, a two-level spatial index structure, a fundamental spatial component built on the MapReduce layer and three basic spatial operations (range query, k-NN query, and spatial link) (Eldawy and Mokbel 2015). GeoSpark provides operational tools for spatial big data processing based on Spark (Yu et al. 2015). The Spatial Resilient Distributed Datasets (SRDDs) structure represents spatial data blocks in memory and index objects using quad-tree and r-tree in each RDD partition (Lenka et al. 2016). ClimateSpark integrates Spark SQL and Apache Zeppelin to develop a web portal that facilitates interaction among climatologists, climate data, analytical operations and computing resources (Hu et al. 2018b). As an extension of Scala, GeoTrellis supports high-performance raster data analysis. GeoMesa provides spatiotemporal data persistence on Hadoop and column-family databases (e.g., Accumulo, HBase), as well as a suite of geospatial analytical tools for massive vector and raster data (Hughes et al. 2015).

As described in this section, a service-oriented, scalable architecture usually contains three major layers to provide desirable functionalities and capabilities: (1) the bottom data storage layer provides physical data storage, (2) the data query layer enables data discovery capabilities with proper functionality and interoperability, and (3) the data processing layer supports extensible, interoperable and scalable analytical functionalities based on open source solutions and their variants from the geoscience communities. With the architecture, big Earth data could be accessed and analyzed with low time latency or even in real time. However, it is challenging to set up such architecture and share data stored inside them due to the requirements of storage resources, computing resources, complicated configurations, and domain knowledge. Fortunately, the paradigm of cloud computing, discussed in the next section, brings potential solutions to ease the process of analytical framework setup and data sharing.

#### **9.5 Cloud Computing for Big Data**

#### *9.5.1 Cloud Computing and Other Related Computing Paradigms*

Grid computing and High Performance Computing (HPC) have been utilized for big data analytics. Grid computing, a distributed system of computer resources, performs large tasks using loosely coupled computers in a distributed system (Hamscher et al. 2000). The European Data Grid project utilizes grid computing to support exploration of multi-petabyte datasets (Segal et al. 2000) and the TeraGrid GIScience gateway utilized grid computing to perform computationally intensive geographical analytics (Wang and Liu 2009). HPC uses supercomputers to run applications in parallel, efficiently and quickly, and is used in the PRACE project to serve European scientists with high-performance computing capabilities to conduct research (Hacker et al. 2010).

Cloud computing virtualizes computer resources as a resource pool to provide computing resources over the network by optimizing resource usage in terms of the CPU, RAM, network, and storage. Cloud computing has intrinsic connection to the Grid Computing paradigm (Foster et al. 2008) in that both are distributed computing systems. Cloud computing relies on remote servers whereas grid computing connects servers or personal computers over a common network using a Wide Area Network (WAN) to perform parallel tasks (Foster et al. 2008). Compared with HPC, cloud computing is cost effective and easy to use. Although cloud computing can provide high performance computing capability, HPC is irreplaceable for some applications since supercomputers are required to process very complicated processes such as climate simulations. In addition, resources in cloud computing are controlled by the service providers and users have limited controls.

In addition to cloud computing, other new computing paradigms have emerged to build a comprehensive and economic computing framework. For example, edge computing can process data at the edge of network due to the advancement of the IoTs. IoT applications usually produce a massive amount of streaming data and require near-real time response; thus, cloud computing alone is not an optimal solution for data collection and analysis for such real-time applications. In edge computing, edge nodes serve as data providers and consumers to protect data privacy and make full use of the computing capacity of edge nodes. Less data is transferred to the cloud computing platform after data preprocessing in edge nodes, reducing the response time and bandwidth cost (Shi et al. 2016).

Mobile computing with portable computing nodes has become an important computing paradigm with the improvements in the computing and storage capacity of smart devices such as smartphones and tablets (Qi and Gani 2012). Although services provided by mobile computing are not as reliable as edge computing and cloud computing due to the restrictions in battery volume and network connection, mobile computing can collect data and reach end users where cloud computing and edge computing are inaccessible.

These computing paradigms have advantages and disadvantages, and can be integrated to complement each other and provide reliable and effective data storage and processing frameworks according to the data characteristics and computing requirements. Cloud computing and big data are the two most important technologies in Digital Earth. The following section discusses the utilization of cloud computing to support big data management in Digital Earth.

#### *9.5.2 Introduction to Cloud Computing*

As a new computing paradigm, cloud computing delivers scalable, on-demand, payas-you-go access to a pool of computing resources (Mell and Grance 2011; Yang et al. 2011a). Practically, cloud computing aims to maximize the utilization rate of physical resources and provide virtual resources to aid applications and services. Cloud computing relies on several technologies including virtualization, network security, and high availability to provide services over the network. These technologies make it easier, more efficient, and more economical to set up architecture for big data analysis.

Virtualization is the fundamental technology for cloud computing, which abstracts an application, operating system, or data store from the underlying hardware or software. Virtualization creates a "layer of abstraction" between the physical systems and a virtual environment in the virtualization process (Big Data Virtualization). Server virtualization optimizes the use of redundant computing and storage resources by virtualizing distributed computer resources (e.g., CPU, RAM, Network, and Disk) and managing them in the same resource pool. With virtualization, cloud computing can provide on-demand big data services and support big data technologies including big data storage, process, analysis, visualization, and remote collaboration (Fig. 9.3). Virtualizing big data resources as a pool serves as a user-friendly interface and makes big data analytics accessible to end users.

As one of the cloud solutions, public clouds are the most accessible cloud computing services offered by third-party providers (e.g., Amazon Web Services (AWS), Microsoft Azure, Alibaba Cloud) over the Internet. Public clouds are available to the public and may be offered on a pay-per-usage model (Li et al. 2010). In contrast to public clouds, private clouds are dedicated for use inside an organization. Private cloud resources can be managed by an organization or by a third-party vendor,

**Fig. 9.3** Cloud computing for big data analysis

regardless of the physical location of the resources (Dillon et al. 2010). The computing resources in a private cloud are isolated and delivered via a secure private network.

The advanced features of auto scaling and load balancing through resource monitoring further maximize the capability of cloud computing resources. Based on the individual performance of a machine, autoscaling can be applied to allow for some servers to rest during times of low load to save electricity costs and automatically add more instances during times of high demand. In addition, load balancing improves the distribution of workloads across multiple instances to optimize resource use, minimize response time, and avoid overloading any single instance.

#### *9.5.3 Cloud Computing to Support Big Data*

Cloud computing combines distributed computing resources into one virtual environment, providing big data analytics and solutions during the life cycles of big data. Three main categories of cloud computing services are (1) Infrastructure as a Service (IaaS), (2) Software as a Service (SaaS), and (3) Platform as a Service (PaaS). Together with Data as a Service (DaaS), Model as a Service (MaaS; Li et al. 2014a, b) and workflow as a service (WaaS; Krämer and Senner 2015), cloud computing offers big data researchers the opportunity of anything as a service (XaaS; Yang et al. 2017b).

#### **Cloud Storage for Big Data Storage**

The characteristics of big data in high volume lead to challenges for data storage. Cloud computing's potential for unlimited storage support helps solve the volume challenge of big data, as the cloud provides virtually customizable storage with elastically expandable and reducible size. An alternative solution is Data Storage as a Service (DSaaS) enabled by block storage, which is the capability of adding external storages as "blocks". With block storage, it is possible to enlarge the storage size without physically loading hard drives. Virtually unlimited scalable storage offered by cloud computing grants users the capability of dynamic adjustment to satisfy the storage requirements of data with high volume and velocity. The modularized virtual resource offers effortless data sharing within production environments by allowing for an external data block to be detached and remounted from one machine to another. External data storage can be automatically backed up to prevent users from losing data, and backups that are securely saved at the back-end server can be easily transferred and restored. In addition, information security is guaranteed because the physical location cannot be obtained from the disk drive (Mayama et al. 2011).

#### **Cloud Computing for Big Data Processing**

Processing large volumes of data requires dedicated computing resources, e.g., faster CPUs and networks and larger disks and RAMs (Yang et al. 2017b). Cloud computing provides on-demand resources and delivers configurable resources including mountable external storage spaces, computing resources (CPU, RAM), and network services. Traditionally, a computer uses approximately two-thirds of the power of a busy computer (JoSEP et al. 2010), and cloud computing has the potential to provide on-demand computing resources. Isolated virtual structures have been created for big data systems to enhance system stabilities, which can be easily managed in different file systems and replicated through backup images to provide fast configuration recovery. The ability to replicate environments automates the expansion of compute nodes in virtual machine clusters, thereby efficiently utilizing resource pools to support big data analytics. With the foundational support of storage for big data, data processing inherits the advantages of fast data acquisition and relocation.

Although cloud computing could serve as an excellent infrastructure option for big data processing, several aspects should be considered to minimize the bottleneck effect for the general processing speed, such as the choice of cloud volume type according to I/O demand and cloud bandwidth selection according to application requirements.

#### **Cloud Computing for Big Data Analytics**

Popular big data analytical platforms such as Apache Hadoop are traditionally installed on physical machine clusters, resulting in a waste of computing resources due to hardware redundancy (CPU and RAM). With the virtual clusters provided by cloud computing through virtualization technology, distributed analytical platforms can be migrated to the virtual clusters from physical machine clusters, optimizing the usage of computing resources in an efficient manner.

With the aid of autoscaling and load balancing, deploying on-demand and scalable big data analytical platforms could easily provide resilient analytical frameworks and minimize waste of computing resources. Autoscaling supports parallel algorithms on distributed systems and architectures for scalability. It allows for the expanded resources to function when the algorithms or programs are enabled with parallel computing capability. Without it, public cloud providers such as AWS could not offer automatic scalability (JoSEP et al. 2010). The load balancer distributes workloads among virtual clusters and triggers autoscaling functions when analytics require higher computing configurations. The virtual system as a whole could dynamically fit higher computing requirements by launching more virtual duplications as needed. The load balancer acts as a virtual network traffic distributor and can be optimized to better allocate overall resources.

#### **Cloud Computing for Big Data Sharing and Remote Collaboration**

Traditional deployment of big data systems requires complicated settings and efforts to share data assets. It lacks access control and often leads to data security and data privacy issues. Cloud computing enhances the sharing of information by applying modern analytical tools and managing controlled access and security (Radke and Tseng 2015). Virtualization enables different parties to share data assets to achieve various goals and objectives under a centralized management system. With the support of cloud computing, it is possible to flexibly share data and remotely collaborate, which involve interdisciplinary collaborations and advanced workflows. Though data sharing, computational resource sharing, and production environment sharing, cloud computing can potentially be used to build a perceptual environment to support various businesses and applications (Li et al. 2015). Unfortunately, workflow sharing remains challenging due to domain boundaries (Yang et al. 2017b).

#### **9.6 Case Study: EarthCube/DataCube**

Big data and cloud computing enable Earth scientists and application developers to create web-accessible frameworks and platforms to efficiently store, retrieve and analyze big Earth data. In the Earth science domain, scientists have proposed a series of data models, frameworks, and initiatives to ensure the success of heterogeneous data sharing and analysis. For example, a 10-year framework initiative on sustainable consumption and production from 2013 to 2023 was launched by the United Nations Environmental Program (UNEP). The Future Earth framework, an international research program in the environmental science community, serves as an evolving platform to support transitions toward sustainability (Lahsen 2016). Microsoft's Eye-On-Earth platform aids climate change research in several European countries by collecting and sharing water and air quality data (Microsoft 2011). As part of the European program to monitor the Earth, the Copernicus Data and Information Access Service (DIAS) platform collects and processes data from remote and in situ sensors and provides reliable information covering six thematic areas including land, ocean, atmosphere, climate, emergency, and security (Bereta et al. 2019). Through its Exploitation Platforms (EP) initiative, the European Space Agency (ESA) built several cloud-based Thematic Exploitation Platforms (TEPs) in a preoperational phase for geo-hazard monitoring and prevention (Esch et al. 2017). The CASEarth Poles comprise a comprehensive big data platform of the three poles including the Arctic, Antarctic and the Tibetan plateau within the framework of the "Big Earth Data Science and Engineering" program of the Chinese Academy of Science (Guo et al. 2017). One of the current initiatives is the NSF EarthCube originated from the NSF GEO Vision report (NSF Advisory Committee for Geosciences 2009). In this section, we introduce the EarthCube project and a big data infrastructure, Data Cube, as two cases of big data and cloud computing in the context of Digital Earth.

#### *9.6.1 EarthCube*

NSF EarthCube involves (1) Building Blocks (BBs), to develop novel infrastructure capabilities and demonstrate their value in a science context; (2) Research Coordination Networks (RCNs), to engage the science community around joint goals; (3) Conceptual Designs (CDs), to develop broad architecture design and explore integrative systems; (4) Integrative Activities (IAs), to explore concepts for the design of an enterprise architecture, and (5) Data Infrastructures (DIs) to lay the groundwork for shared data. The EarthCube concept originated from the NSF GEO Vision report (NSF Advisory Committee for Geosciences 2009), which was issued by the Advisory Committee for NSF's Geosciences Directorate (GEO) and identified the future focus of the Earth science community as '*fostering a sustainable future through a better understanding of our complex and changing planet*.' To achieve the GEO vision, the GEO and Office of Cyberinfrastructure (OCI) jointly launched the EarthCube (NSF 2011) initiative as a driving engine to build a geospatial cyberinfrastructure (similar to a Digital Earth infrastructure, Yang et al. 2010) to (1) understand and forecast the behavior of a complex and evolving Earth system; (2) reduce vulnerability and sustain life; and (3) train the workforce of the future.

EarthCube (2012) is targeted at (1) transforming the conduct of data-enabled geoscience-related research, (2) creating effective community-driven cyberinfrastructure, (3) allowing for interoperable resource discovery and knowledge management, and (4) achieving interoperability and data integration across disciplines (Fig. 9.4).

In addition, EarthCube is evolving within a rapidly growing, diverse, and wideranging global environment. In addition to the collaboration within EarthCube, there are other contributing entities ranging from individual data sets and software applications to national and international cyberinfrastructure systems. The NSF has funded the development of EarthCube through individual EarthCube awards since 2013. In 2016, the NSF awarded 11 new EarthCube activities, for a total of 51 awards. A sampling of efforts in EarthCube that benefit from big data and cloud computing are introduced below.

**Fig. 9.4** Examples of related projects (derived from EarthCube goals, EarthCube Office 2016)

#### **Conceptual design DASHER**

Yang et al. (2015b) proposed a conceptual EarthCube Architecture, DASHER (Developing a Data-Oriented Human-Centric Enterprise Architecture for EarthCube), to support EarthCube and facilitate data communication and social collaboration in pursuit of collaborative Earth sciences research. The final product is a four-volume report containing different viewpoints that describe EarthCube architecture from different conceptual perspectives such as capabilities, operations, services, and projects. It provides a comprehensive conceptual reference for developing a detailed and practical architecture to address the requirements of the EarthCube community. DASHER was one of the first projects funded by EarthCube to design the conceptual framework integrating computational resources and big data sources.

#### **Building Block GeoSciCloud**

GeoSciCloud (Deploying Multi-Facility Cyberinfrastructure in Commercial and Private Cloud-based Systems) investigated two medium-size NSF funded data centers to deploy data collections with cloud-based services in different environments to assess feasibility and impact (EarthCube 2019). These environments include (1) commercial cloud environments offered by Amazon, Google, and Microsoft and (2) NSF-supported extensive computing facilities that are just beginning to offer services with characteristics of cloud computing.

GeoSciCloud helps EarthCube compare and contrast these three environments (the Extreme Science and Engineering Discovery Environment (XSEDE), commercial cloud, and current infrastructure) in the massive data ingestion to the cloud, data processing time, elasticity, the speed of data egress from multiple environments, overall costs of operation, interoperability, and reliability of real-time data streaming.

#### **Integrated Activity ECITE**

The EarthCube Integration and Test Environment (ECITE) is an outgrowth of activities of the EarthCube Testbed Working Group. The ECITE approach focuses on integrating existing effective technologies and resources as well as capabilities built by the EarthCube community using a cloud platform to provide a federated and interoperable test environment (EarthCube 2016). ECITE engages scientists and technologists from multiple disciplines and geographic regions across the Earth science community to develop requirements, prototype, design, build, and test an integration test-bed that will support cross-disciplinary research. The hybrid federated system will provide a robust set of distributed resources including both public and private cloud capabilities. This research addresses timely issues of integration, testing and evaluation methodologies and best practices with a strong interoperability theme to advance disciplinary research through the integration of diverse and heterogeneous data, algorithms, systems, and sciences.

#### **Integrated Activity Pangeo**

Pangeo3 (an open-source big data climate science platform) integrates a suite of open-source software tools that can tackle petabyte-scale Atmosphere/Ocean/Land/Climate (AOC) datasets. Pangeo aims to cultivate an ecosystem in which the next generation of open-source analysis tools for ocean, atmosphere and climate science can be developed, distributed, and sustained. These tools must be scalable to meet the current and future challenges of big data, and the solutions should leverage the existing expertise outside of the AOC community. The resulting software improvements contribute to upstream open source projects, ensuring the long-term sustainability of the platform. The result is a robust new software toolkit for climate science and beyond. This toolkit will enhance the Data Science aspect of EarthCube. Implementation of these tools on the cloud was tested, taking advantage of an agreement between commercial cloud service providers and the NSF for big data solicitation.

#### *9.6.2 Data Cube*

The term 'data cube' was originally used in Online Analytical Processing (OLAP) of business and statistical data but has more recently been used in Earth domains as an approach to manage and analyze large and rapidly growing datasets. In Digital Earth, a data cube represents a multidimensional (n-D) array that stores gridded data or array-based data produced by remote sensing and simulation (Zhang et al. 2005). A data cube can be based on regular or irregular gridded, spatial and/or temporal data with multiple parameters. To support the management, sharing, and serving of Digital Earth data, tools and models, different cyberinfrastructures have been developed based on data cubes. Examples include the EarthServer that provides data cube services for Earth observations based on the RASDAMAN array database (Baumann et al. 2016). Another example is the Earth Observation Data and Processing Platform developed by the European Commission to integrate and analyze the combination of satellite and in situ Earth observations for sustainable development goals (Soille et al. 2016). The Committee on Earth Observation Satellites (CEOS) provides a data processing infrastructure based on data cubes to support Earth science objectives in developing countries, with a focus on remote sensing data (Nativi et al. 2017). The platform automatically ingests different remote sensing data into an N-dimensional data array.

Challenging issues in providing data services in data cube infrastructure include interoperability, rapid data access and transfer, and real-time processing and analysis (Strobl et al. 2017). Interoperability issues occur because datasets from various sources can have distinct parameterizations, spectral band definitions, projections, file formats, and database structures. One solution is to standardize the preprocessing procedure before storage and sharing with the community. The Open Geospatial

<sup>3</sup>http://pangeo.io/.

Consortium (OGC) Sensor Web Enablement (SWE) Common Data Model (CDM) defines important element parameterization (Robin 2011). Datasets must be represented along different predefined dimensions of the data cube, including space, time, and parameter properties. For projection difference issues, OGC recently developed the Discrete Global Grid System (DGGS) to optimize the loss of geospatial data during the reprojection process and seamlessly integrate GIS data from various sources (Stefanakis 2016). Data cube infrastructures also require rapid data access and transfer and real-time processing and analysis. Functionalities for user interactions must be built for various user demands, including file manipulation, data preprocessing, and analysis. These functionalities should also meet the standards of geographical information processing in OGC Web Coverage Services (WCS), geographic information analysis in the OGC Web Coverage Processing Service (WCPS), and format-independent data cube exchange in the OGC Coverage Implementation Schema (CIS).

Cloud computing and big data frameworks could enhance the data cube archive, visualization, and analysis in many ways to meet the needs of big Earth data knowledge mining. In cloud computing, storage is a virtual resource that can be attached and scaled on demand. By leveraging cloud computing and big data frameworks, visualizing data cubes and performing complicated spatiotemporal queries are more effortless than ever before (Zhizhin et al. 2011). One example of data cube visualization in the Earth science domain is the EOD4 (Earth Observation Data Cube), which enables advanced data access and retrieval capabilities for the European coverage of Landsat-8 and the global coverage of Sentinel2 data. It aims to improve the accessibility of Big Earth data and offers more than 100 TB of Atmosphere, Land and Ocean EO products, demonstrating satellite data in the context of a virtual globe. The ESDC5 (Earth System Data Cube) is another example of climate data cube visualization and analysis that aims to develop an environment to tap into the full potential of the ESA's Earth observations and integrate with the Biosphere-Atmosphere Virtual Laboratory (BAVL) analysis environment. The use of cloud computing technologies in big data visualization enables a massive amount of end users to explore data online at the same time with very low latency.

Data cube partition and parallel query could be achieved by utilizing distributed big data frameworks, which are faster and easier than traditional noncluster methods. Pagani et al. combined the data cube concept with cloud computing to manage and analyze large Earth datasets and observed better outcomes than traditional file-based approach (2018). Open Data Cube6 is another example of the utilization of advances in cloud computing, providing free and open technologies to end users without local infrastructure. Thus, developing countries can access data and computing resources to build applications that aid decision making. The Australian Geoscience Data Cube (AGDA) solves similar problems of data sharing. It makes more than three decades of satellite imagery available for the first time, spanning Australia's total land area at

<sup>4</sup>https://eodatacube.eu/.

<sup>5</sup>https://cablab.readthedocs.io.

<sup>6</sup>https://www.opendatacube.org/.

a resolution of 25 square meters with more than 240,000 images that show how Australia's vegetation, land use, water movements, and urban expansion have changed over the past 30 years (NCI).

#### **9.7 Conclusion**

The advancement of Digital Earth drives collection of massive data such as transportation data, utility data, hazard data and forest data to monitor the Earth and support decision making. Valuable information extracted from the collected big data can further speed the development of Digital Earth. Digital Earth data continue to grow at a faster speed and with more heterogeneous types, leading to challenges to the lifecycle of data management including storage, processing, analytics, visualization, sharing, and integration. Fortunately, the emerging paradigm of cloud computing brings potential solutions to address these challenges. Compared with traditional computing mechanisms, cloud computing has the advantages of better data processing computing supports. The customizable configuration saves computing resources elastically, and data manipulation with higher security and flexibility offers secure data storage, transfer and sharing. Analytics enabled by cloud computing advance the process by allowing for automatic resource expansion when there are higher requirements.

To manage and analyze big Earth data, a service-oriented, scalable architecture based on cloud computing was introduced in a three-layer architecture: (1) the bottom data storage layer provides physical infrastructure, storage, and file systems; (2) the data query layer supplies data discovery capabilities with proper functionality and interoperability; and (3) the data processing layer supports extensibility, interoperability and scalability based on open source solutions and their variants from Earth science communities. With this architecture, big Earth data can be accessed and analyzed with low time latency or even in real time. The analysis results could be published by a web-based map server (e.g., GeoServer) or web-based notebook (e.g., Zeppelin) for visualization, public access, and collaboration, contributing to advancements in handling big data in Digital Earth to fulfill the requirements of scalability, extensibility and flexibility.

#### **References**

Anderson A (2015) Statistics for big data for dummies. John Wiley & Sons, Hoboken, NJ

**Yun Li** is a Ph.D. candidate majoring in Earth Systems and Geoinformation Sciences at George Mason University. Her research mainly focuses on improving Earth data discovery using machine learning based methods, high performance computing, and spatiotemporal analytics for environmental and climate data.

**Manzhu Yu** is an Assistant Professor of GIScience at Pennsylvania State University. Her research focuses on spatiotemporal theories and applications, atmospheric modeling, environmental analytics, big data and cloud computing, and the ability of using the above to solve pressing issues in natural hazards and sustainability.

**Mengchao Xu** is a Ph.D. candidate in Earth Systems and Geoinformation Sciences program from George Mason University. His research mainly focuses on the cloud computing, high performance computing networks, distributed database systems, and precipitation downscaling. He manages a 500-nodes cluster specialized for cloud computing (supported by NASA, NSF and GMU).

**Jingchao Yang** is a Ph.D. student majoring in Earth Systems and Geoinformation Sciences at George Mason University. He has been contributing to several NSF and NASA funded projects at the NSF Spatiotemporal Innovation Center. His research mainly focuses on the smart city problems with machine- and deep-learning enabled spatiotemporal pattern analysis using multi-sensor (IoT) datasets.

**Dexuan Sha** is a Ph.D. student in Earth Systems and Geoinformation Sciences at the George Mason University. He researches arctic sea ice and melt pond from high spatial resolution aerial imagery with big spatial–temporal data mining and analysis methodology.

**Qian Liu** is a Ph.D. student in Geography and Geoinformation Science at George Mason University. Her research mainly focuses on the core areas of precipitation detection and analysis using AI-based methods, and outreaches to environmental and climate science and spatial temporal analytics.

**Chaowei Yang** is Professor of GIScience at George Mason University, where he established and directs the NSF spatiotemporal innovation center. His academic activities include spatiotemporal computing research, future GIScience leader education, and Geoinformatics outreach.

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Chapter 10 Artificial Intelligence**

**Eric Guérin, Orhun Aydin and Ali Mahdavi-Amiri**

**Abstract** In this chapter, we provide an overview of different artificial intelligence (AI) and machine learning (ML) techniques and discuss how these techniques have been employed in managing geospatial data sets as they pertain to Digital Earth. We introduce statistical ML methods that are frequently used in spatial problems and their applications. We discuss generative models, one of the hottest topics in ML, to illustrate the possibility of generating new data sets that can be used to train data analysis methods or to create new possibilities for Digital Earth such as virtual reality or augmented reality. We finish the chapter with a discussion of deep learning methods that have high predictive power and have shown great promise in data analysis of geospatial data sets provided by Digital Earth.

**Keywords** Artificial intelligence · Machine learning · Generative models · Statistical data analysis

#### **10.1 Introduction**

Earth and its associated data sets are massive. Various forms of geospatial data sets are constantly accumulated and captured by different forms of sensors and devices (Mahdavi-Amiri et al. 2015). Managing such an immense data set is a challenge. As a result, many automated techniques have been designed to process geospatial data sets with minimal human interference. Since manual involvement should be minimal, the machines should be capable of processing data and delivering meaningful information to the users. With advancements in machine learning, processing

E. Guérin

LIRIS, Université de Lyon, Lyon, France

O. Aydin ESRI, Los Angeles, Los Angeles, USA

University of Southern California, Los Angeles, USA

A. Mahdavi-Amiri (B) Department of Computing Science, Simon Fraser University, Burnaby, Canada e-mail<EMAIL_ADDRESS>
<sup>©</sup> The Editor(s) (if applicable) and The Author(s) and European Union 2020 H. Guo et al. (eds.), *Manual of Digital Earth*, https://doi.org/10.1007/978-981-32-9915-3\_10

geospatial data sets has significantly improved. In this chapter, we discuss artificial intelligence and machine learning techniques that have been useful to manage and process geospatial data sets. Because the processing of geospatial data can also be a source of knowledge, some methods use existing data to generate and synthesize new data.

We start by discussing some traditional and statistical approaches in machine learning and then present more recent learning techniques employed for geospatial data sets. Traditional methods include predefined models such as linear regression, PCA, SVD, active contour, and SVM, in which the model is fixed and the learning is based on an optimization. We also briefly discuss evolutionary and agent-based methods and autoencoders as traditional methods that can be deep or shallow. We then discuss more recent deep learning techniques, including reinforcement learning, deep convolutional networks and generative models such as variational autoencoders and generative adversarial networks. In this chapter, we describe some applications of these machine learning techniques to handle geospatial data sets that are the main content of Digital Earth. In the future, a dynamic Digital Earth that can use such techniques to work with geospatial datasets is extremely practical. Currently, such methods are sparsely used on very specific Digital Earth data sets. We imagine that a more advanced Digital Earth will use state-of-the-art machine learning techniques much more than they are currently used.

#### **10.2 Traditional and Statistical Machine Learning**

Inferring patterns and forming relationships using artificial intelligence require knowledge of some characteristics of the phenomena/system of interest. One of the early approaches to enabling artificial intelligence for complex problems was to create knowledge bases that contain explicit sets of rules and associations, also known as ontology (Gruber 1993). For data pertaining to Earth system modeling, different niche knowledge bases were designed by various authors (McCarthy 1988; Rizzoli and Young 1997). The knowledge base approach to artificial intelligence required expert input to define the rules and associations. In addition, the expert knowledge had to be represented in a "computable form" (Sowa 2000), posing a bottleneck for these approaches. For spatially varying, complex phenomena, ontology representations were defined for Earth's subsystems such as in environmental modeling and planning (Cortés et al. 2001), and ecological reasoning (Rykiel 1989). General spatial and GIS knowledge bases were proposed by various authors (Kuipers 1996; Egenhofer and Mark 1995; Fonseca et al. 2002).

Despite the plethora of niche knowledge bases, knowledge base artificial intelligence requires assertions and ground truths (Lenat 1995), which can conflict with observations (Goodfellow et al. 2016). Numerous attempts to address this limitation have been presented by various authors, such as defining hierarchical (Kuipers 1996), or location/problem-tailored knowledge bases (Rizzoli and Young 1997).

Statistical machine learning alleviates the limitations of the knowledge-based approach to artificial intelligence and discovers rules and patterns from the data directly without explicit supervision (Goodfellow et al. 2016). In the case of statistical learning, patterns and rules from an unknown underlying process are defined for descriptive, predictive and prescriptive analytics.

Applications of statistical learning to understand and forecast natural and human phenomena are evaluated with respect to the components of the general definition of machine learning (Mitchell 1997). Mitchell's (1997) definition is as follows:

A computer program is said to learn from experience [D] with respect to some class of tasks T and performance measure [Q], if its performance at tasks in T, as measured by [Q], improves with experience [D].

Machine learning methods are broadly grouped into supervised and unsupervised methods. Supervised machine learning methods experience modeled phenomena through so-called labeled training data. Labels in the training data correspond to the target variable to be predicted, either quantitative (regression) or qualitative (classification). Training data consists of predictors and their corresponding predictand. Thus, supervised machine learning methods learn relationships in the data through experiencing input/output pairs.

Unsupervised machine learning methods discover patterns in the data without supervision or explicit rules. Clustering is one of the most common unsupervised machine learning methods for geospatial datasets.

#### *10.2.1 Supervised Learning*

Supervised learning aims to define a relationship between *r* predictor variables, denoted by *X* = (*X*1, *X*2,..., *Xr*), and *e* predictands, *Y* = (*Y*1, *Y*2,..., *Ye*). Supervised learning can be posed as a density estimation problem (Hastie et al. 2001):

$$P(Y|X) = P(Y,X)/P(X) \tag{10.1}$$

where *P*(*Y* |*X*) is the conditional probability density of observing the predictand given the predictors, *P*(*Y*, *X*) is the joint probability distribution of the predictand and predictors, and *P*(*X*) is the marginal probability distribution of the predictors. Using Mitchell's (1997) description, the performance Q can be quantified using a loss function *L* where, for a given method and set of parameters -, a location function, μ(*x*), is minimized (Hastie et al. 2001) in Eq. 10.2.

$$\mu(\mathbf{x}) = \arg\min\_{\Theta} E\_{Y|X} \mathcal{L}(Y, \Theta) \tag{10.2}$$

For a given -, a supervised machine learning method predicts the values at X as *y*ˆ. The loss function, *L*, quantifies the error between *y*ˆ and the training data *y*. Some examples of supervised machine learning methods as they pertain to geospatial analysis are given in the following subsection.

#### **<IP_ADDRESS> Random Forest**

Random forest is a framework for nonparametric estimation in which both classification and regression can be performed (Breiman 2001). It has gained popularity in numerous geospatial applications due to its flexibility in accommodating different types of inputs (categorical or continuous) and its ability to model complex relationships in the data.

Random forest addresses the overfitting limitation of **c**lassification **a**nd **r**egression **t**rees (CART). Random forest uses bootstrap aggregating, also known as bagging, to create subsets of the training data by sampling with replacement to build different CARTs (Breiman 1996). Each of the CARTs that make up the forest predict, or vote, for a given data point of *x* and the forest returns the majority vote in a classification or the average forest prediction for a regression. The voting scheme of random forest allows for complex relationships to be captured in the data that might not be possible otherwise. A pictorial summary of a random forest classifier for classifying a successful retail store (one) or an unsuccessful one (zero) with respect to its distance to the nearest highway exit and the number of brands it carries is given in Fig. 10.1.

**Fig. 10.1** Cartoon representation of a random forest classifier

**Fig. 10.2 a** Satellite image over southern California, with training data marked with black polygons **b** classified land coverage map using random forest

Note that every tree experiences different subsets of training data and their structures are different from one another. The voting scheme allows for capturing underlying patterns in the data by defining complex relationships captured in a large ensemble of trees rather than a single tree.

In geospatial problems, various random forest classifiers are used in a wide range of problems, including land cover classification (Gislason et al. 2006) and ecological modeling (Cutler et al. 2007). In land cover classification, random forest speeds up classification of land use by forming a relationship between the satellite image RGB value and the type of land it corresponds to. In this case, the training data consists of tagged locations at which the land cover and RGB values are known. An example of the random forest classifier output for land use classification is given in Fig. 10.2.

In Fig. 10.2, a small number of farms and areas around them were used as training data (marked with black polygons). The training set that consists of 300 farms was used within the random forest classifier to define land use in southern California.

#### **<IP_ADDRESS> Geographically Weighted Regression**

**G**eographically **w**eighted **r**egression (GWR) provides a statistical framework for incorporating spatial dependency within a linear regression system (Fotheringham

et al. 2003). GWR provides spatial extensions to ordinary least squares and generalized linear models (Nelder and Wedderburn 1972) such as geographically weighted logistic regression. GWR is depicted conceptually in Fig. 10.3.

Figure 10.3 illustrates a regression system solved within the neighborhood (red circle) for the location indicated in orange. First, GWR defines a weighting scheme to determine spatial weights for the neighbors, and the predictors *X* at every location (blue) are weighted with respect to their distance to the location for which the regression is performed (orange). The geographically weighted linear system of equations solved at a point *i* can be expressed as follows:

$$\hat{\boldsymbol{\beta}}(\boldsymbol{u}\_{i},\boldsymbol{v}\_{i}) = \left(\mathbf{X}^{T}\mathbf{W}(\boldsymbol{u}\_{i},\boldsymbol{v}\_{i})\mathbf{X}\right)^{-1}\mathbf{X}^{T}\mathbf{W}(\boldsymbol{u}\_{i},\boldsymbol{v}\_{i})\mathbf{Y} \tag{10.3}$$

where **β**ˆ(*ui*, *vi*) is the coefficient matrix for the predictors **X** at location *i*. **W**(*ui*, v*i*) is a diagonal weighting matrix that contains geographic weights on its diagonal elements for neighbors inside the neighborhood window (red circle in Fig. 10.3), and **Y** contains the variable being predicted. Note that the linear system above is similar to the general linear regression system given in Eq. 10.4.

$$
\hat{\boldsymbol{\beta}} = \left(\mathbf{X}^{\mathsf{T}}\mathbf{X}\right)^{-1}\mathbf{X}^{\mathsf{T}}\mathbf{Y} \tag{10.4}
$$

where **β**ˆ is defined globally for the entire dataset. The geographic weights are inversely weighted with respect to the distance. Thus, the weights have large values for neighbors close to the regression location *i*. Different weighting schemes and neighborhood definitions are possible; the reader is encouraged to explore seminal work on this topic (Fotheringham et al. 2003).

Spatial representation via a weighting scheme can give GWR high predictive power for geospatial datasets in which a strong spatial autocorrelation is observed. The impact of incorporating spatial relationships in the regression model is demonstrated by comparing GWR with a nonspatial supervised machine learning method. In this example, GWR is juxtaposed against a random forest predictor for a problem with strong spatial autocorrelation in the data. Statistical climate downscaling (Wilby and Wigley 1997) was performed with GWR and a random forest regressor. Statistical downscaling calibrates the output of a global circulation model (GCM) to observed climate data such as temperature or precipitation. In this example, climate downscaling for the lower 48 US states; a regression model can be defined between 19 predictors (from GCM) and the observed average temperature. The regression model can be used to predict the average temperature for the entire lower 48 states. A random forest predictor can be trained using the observed average temperature and simulated GCM variables. The GWR model is formed using only 3 of the independent predictors due to the collinearity restriction of GWR. Below are the predicted average temperature profiles.

Note that the average temperature profile estimated in Fig. 10.4a depicts the patterns of temperature change captured in Fig. 10.4b. Even though fewer predictors are used in the GWR than in the random forest regressor, large-scale patterns in the temperature profile changes are captured. The GWR model in Fig. 10.4a was also compared to a random forest regressor model trained using the same three predictors. In that case, the GWR returned a mean-squared error that was 60% of that of the random forest regressor.

#### **<IP_ADDRESS> SVM**

Support vector machine (SVM) is a supervised nonparametric statistical learning method (Corinna and Vapnik 1995). In its original form, the method comprises a set of labeled data instances and the SVM attempts to find a hyperplane that separates the dataset into a discrete predefined number of classes as consistently as possible for the training data (see Fig. 10.5) (Vapnik 1979). It is possible to generalize SVM to nonlinear kernels such as radial basis functions to learn and classify data sets with higher complexity (Schölkopf and Smola 2002).

As studied and discussed by Mountrakis et al. (2011), SVMs have been extensively employed in remote sensing and geospatial data analysis due to their ability to use small training data sets, often resulting a higher classification accuracy than the traditional methods (Mantero et al. 2005). For instance, SVM has been used in road extraction from IKONOS imagery by (Huang and Zhang 2009) assessing the influence of the slope/aspect of the terrain on the forest classification accuracy (Huang et al. 2008), a crop classification task (Wilson et al. 2004), and many more factors.

**Fig. 10.4 a** Downscaled temperature profile using GWR **b** downscaled temperature profile using a random forest regressor

**Fig. 10.5** SVM attempts to distinguish two categories of data by a hyperplane. Image from Mountrakis et al. (2011)

#### **<IP_ADDRESS> Active Contours and Active Shapes**

Active contours or snakes have been developed with the aim of finding important features in an image by fitting a curve to the edges and lines of an image (Kass et al. 1988). Active contours are a set of energy-minimizing splines that are guided by external forces from the image. Snakes have been used extensively in geospatial image processing to detect features such as roads and buildings.

Active contours were later extended to active shapes to accommodate specific patterns in a set of objects and identify only those that are present in the training data (Cootes et al. 1995). In essence, they are very similar to active contours, but active shapes can only deform and fit the data that is consistent with the training set. Both active shapes and active contours have been extensively used in different applications of remote sensing and geoscience, such as object extraction (Liu et al. 2013), lane detection (Heij et al. 2004), and road extraction (see Fig. 10.6) (Kumar et al. 2017; Laptev 1997).

#### *10.2.2 Unsupervised Learning*

Unsupervised learning aims to infer the distribution of *P*(*X*) in Eq. 10.1. Unlike supervised learning, *P*(*Y* |*X*) or *P*(*X*, *Y* ) is not employed (Hastie et al. 2001). Thus,

**Fig. 10.6** Active contours used to extract roads. Image taken form Laptev (1997)

unsupervised learning does not utilize any training dataset that contains information on *P*(*X*, *Y* ). One of the most common uses of unsupervised learning in geospatial analysis is in defining clusters and regions. These two terms differ, as clustering refers to defining groups based on value similarity in the data whereas regionalization performs clustering under spatial constraints (Duque et al. 2007). Both of these unsupervised learning approaches have wide applications (Duque et al. 2007; Hastie et al. 2001; Mitchell 1997; von Luxburg 2010). Most clustering and regionalization methods require definition of *k*, the number of clusters to divide *X* into. There are extensive surveys of clustering and regionalization in the literature for readers to refer to (Duque et al. 2007; Jain et al. 1999).

#### **<IP_ADDRESS> SKATER Algorithm**

As discussed in Chap. 8, the K-means algorithm (Macqueen 1967) aims to partition *X* into *k* groups and minimize the intergroup dissimilarity with the assumption that minimal intergroup dissimilarity corresponds to distinct groups. K-means seeks to create groups that consist of similar elements, ensuring that dissimilar elements are assigned to different groups. Mathematically:

$$\mu(\mathbf{x}) = \arg\min\_{\mathbf{C}} \sum\_{i=1}^{k} \sum\_{\mathbf{x} \in \mathbf{c}\_{i}} \left\| \mathbf{x} - \bar{\mathbf{C}}\_{i} \right\|^{2} \tag{10.4}$$

where *C* = {*C*1, *C*2,..., *Ck*} is the group of clusters, with a cluster *cm* consisting of a subset of X and *c*<sup>1</sup> ∪ *c*<sup>2</sup> ∪···∪ *ck* = *X*. K-means has various uses in geospatial analysis, including detecting patterns in traffic accidents (Anderson 2009), analyzing landslides (Keefer 2000) and creating labels by clustering topo-climatic data (Burrough et al. 2001).

The SKATER algorithm is a regionalization algorithm that imposes graph-based spatial constraints on the k-means algorithm (Assunção et al. 2006). Unlike Lloyd's algorithm (Lloyd 1982), SKATER only assigns spatially contiguous and similar objects to the same cluster. Regionalization has vast uses in geospatial analysis, including analysis of gerrymandering, healthcare services (Church and Barker 1998) and resource allocation (Or and Pierskalla 1979).

Clustering and regionalization were applied to the same dataset to juxtapose the types of patterns they expose in the data and the resulting understanding gained using these two methods. The average temperature in the United States in June 2012 was used. The resulting clusters and regions are displayed below.

The regionalization and clustering results in Fig. 10.7 show similarities in the overall temperature patterns, which change N-S in the eastern portion of the US and W-E in the western portion. Notably, the k-means result in Fig. 10.7b displays isolated patches whereas the regionalization result has spatially contiguous regions. Due to the constrained optimization scheme to satisfy the spatial constraints, the regions defined by regionalization have a higher variance than those in the k-means result. However, both maps display similarities in the temperature and the extent to which these similarities can be aggregated into homogeneous zones.

#### **<IP_ADDRESS> Autoencoders**

Another very useful and common machine learning technique is autoencoders (Rumelhart et al. 1985). In an autoencoder, the data passes through a bottleneck, where the bottleneck is a lower representation of the same data. Autoencoders are made of two neural networks called the encoder and decoder (Fig. 10.8). The encoder receives data D, maps it to a lower space and obtains L; a decoder receives L, maps it back to the same dimension of D and obtains D'. The distance between D and D', which is called the reconstruction loss, should be minimized. A direct application of autoencoders is in compression, in which one can reduce the dimension of D to L and work with L and the decoder instead of the data D in its native resolution. Autoencoders have also been used in geospatial applications to find water bodies (Zhiyin et al. 2015) or denoise satellite images (Liang et al. 2017).

Machine learning techniques are not limited to the list of applications and methods provided here. Several variations of these methods as well as many other standalone techniques have been successfully employed in the Digital Earth, geoscience and remote sensing fields. For a more in-depth and comprehensive study, refer to the work of Lary et al. (2016).

#### *10.2.3 Dimension Reduction*

There have been extensive efforts to learn the patterns and forms that data sets contain. It is possible to predict the behavior of a data set and/or compress the data set into a more compact form for transmission, storage, and retrieval. In addition to autoencoders that can be used for dimensionality reduction, one of the easiest methods for compression and dimensionality reduction for a given data set and subsequent prediction of its behavior for unknown data points is *linear regression*.

**Fig. 10.7 a** Temperature regions defined by SKATER **b** temperature regions defined by k-means

**Fig. 10.8** The autoencoder passes the data (yellow neurons) through an encoder to learn a lower dimension (hidden/latent space; gray neurons) representation of the data. The decoder attempts to reconstruct the data (red neurons) as closely as possible to the given data

In the 2D case of this method, the data points attain two coordinates, and the line that best represents these data sets is considered as the model representative of the data. The best representation can have different meanings, including the line that has the smallest least square distance with all the data points. Regression, linear or nonlinear, has been a great tool to analyze spatial data. Belae et al. (2010) provided a survey of regression techniques used to represent and analyze spatial datasets. For Digital Earth platforms, Mahdavi-Amiri et al. (2018) combined regression with a wavelet to transmit quantitative datasets on a discrete global grid system (DGGS).

#### **<IP_ADDRESS> PCA**

Another form of linear representation of a data set is principal component analysis (PCA). In this representation, the covariance matrix of the data is initially formed by applying the inner product of a data matrix A in its transpose - *Co*v = *A<sup>T</sup> A* . The eigenvectors of the covariance matrix, λ*<sup>i</sup>* , represent the main trends of the data. If we have a data set forming an ellipsoid in 2D, the eigenvectors are the two main axes of the ellipsoid. Figure 10.9 represents PCA in 2D. PCA has been extensively used in

many applications including computer graphics, computer vision, and data science. PCA has been used in different applications related to geospatial data representation and geospatial data analysis (Demšar et al. 2013). For instance, PCA has been successfully used to study drought areas (Gocic and Trajkovic 2014), evaluate water quality (Parinet et al. 2004), and distinguish vegetation (Panda et al. 2009).

#### **<IP_ADDRESS> SVD**

Singular value decomposition (SVD) is a decomposition that reveals important information about a matrix. In SVD, a matrix A is decomposed into the form *USV <sup>T</sup>* , in which U and V are two rotation matrices and S is a diagonal scale matrix with values called the singular values, σ*<sup>i</sup>* , of matrix A. There is a direct connection between PCA and SVD because the singular values of the singular value decomposition of data matrix A are the square root of the eigenvalues of the covariance matrix that is found in PCA - <sup>σ</sup>*<sup>i</sup>* <sup>=</sup> <sup>√</sup>λ*<sup>i</sup>* . To compress or denoise data, it is possible to zero out small eigenvalues obtained by SVD and keep important portions of the data. SVD has been extensively employed in image processing applications (Sadek 2012). It has also been used in geospatial applications. For instance, Wieland and Dalchow (2009) used SVD to detect landscape forms, and Dvorsky et al. (2009) used SVD to determine the similarity between maps.

#### **<IP_ADDRESS> Evolutionary and Agent-Based Techniques**

Evolutionary and Agent-based techniques have also been extensively used to perform analyses of geospatial data sets. Two important algorithms are genetic algorithms (GAs) and ant colony optimization (ACO).

In GAs, a set of random solutions is initially produced and these solutions are considered parents to make a new generation of solutions based on three rules: **Selection** rules that select parents based on their fitness,**Crossover** rules that combine two parents to generate children for the next generation, and **Mutation** rules that apply random changes to parents to form children (Mitchel 1998). GAs have been used in many applications in geospatial data analysis such as road detection (Jeon et al. 2002) and satellite image segmentation (Mohanta and Binapani 2011).

ACO is an optimization technique that works based in an agent-based environment. In this stochastic environment, the ants are agents that walk over a certain solution path and leave a track called a pheromone. Paths with more pheromone are usually more optimal (shortest) than others, and they attract more agents. A classic problem that can be solved by ACO is the travelling salesman problem. ACO has been successfully employed to solve other types of hard problems including those involving geospatial data analysis. For instance, ACO has been used for path planning considering traffic (Hsiao et al. 2004) and road extraction from raster data sets (Maboudi et al. 2017).

#### **10.3 Deep Learning**

When a large amount of data is involved and/or a complex model for representing the data is used, it is common to employ deep learning methods (Goodfellow et al. 2016). Digital earth data represents a massive amount of data, for example, high-precision digital elevation models or aerial photography. Because the rules that produce this kind of data are very complex and involve many natural or human processes, it can be difficult to apply standard learning models or algorithms and retain this complexity. Thus, the deep models described in this section are relevant.

#### *10.3.1 Convolutional Networks*

Deep learning has been popularized by image processing applications. In this context, the processed data is arranged into a regular grid and is adapted to so-called convolutional layers. Data extracted from Digital Earth can be of this nature by construction. For example, raster data such as digital elevation models or aerial photography images are already arranged into regular grids and can be processed out of the box with convolutional layers. Convolutional neural networks rely on the fact that the same processing can be applied to different parts of the image. Traditional

**Fig. 10.10** Convolutional layers use fewer coefficients and are spatialized

fully connected schemes for neural network layers use many coefficients that can be spared with convolutional layers and used in other features. Figure 10.10 compares the principle of a convolutional layer to that of a traditionally fully connected layer. Both examples show an input of size 9. While a fully connected layer uses 27 coefficients to produce an output of size 3, the convolutional layer can produce 9 outputs from only 3 different coefficients. This means that the same feature extraction is performed but at different locations, which is relatively close to traditional convolution in the discrete domain.

Recently, a convolutional network was used to infer the super-resolution of a digital elevation model by using aerial photography (Argudo et al. 2018). Figure 10.11 shows the architecture of this network. This work comes from the observation that publicly available high-resolution DEMs (resolution less than 2 m) do not cover the full Earth whereas it is possible to find high-resolution imagery (orthophotos) with good coverage of the Earth. Many applications require a fine resolution for the DEM, and Argudo et al. proposed inserting details into a coarse DEM using inferred information drawn from the high-resolution orthophoto of the same footprint (Fig. 10.12).  To produce this result, a fully convolutional network was used.

In the literature, a full system to automatically infer street addresses from satellite imagery was proposed (Demir et al. 2018a). One step that must be performed is the extraction of roads from the satellite images. This was done using a modified

**Fig. 10.11** A fully convolutional network was used to infer the high-resolution DEM from its coarse version and the high-resolution orthophoto (courtesy of O. Argudo et al.)

**Fig. 10.12** Super-resolution of a 15 m precision DEM (top right) using an orthophoto (top left). Result (bottom left) and the ground truth reference (bottom right) (courtesy of O. Argudo et al.)

version of SegNet, a convolutional network primarily used for image segmentation. In this architecture, the input and output resolutions are identical, and the network consists of several encoder layers that decrease the resolution followed by decoders that increase the resolution. The network is trained using manually labeled 192 × 192 pixel images, in which a binary road mask is associated with each pixel of the image to indicate if the pixel belongs to a road or not. Figure 10.13 shows an example of the results obtained in automatic extraction of the road information compared with the ground truth.

More generally, automatic processing of satellite images with a deep learning approach appears to be very efficient in segmentation and feature extraction. The DeepGlobe project (http://deepglobe.org) aims to challenge authors to use deep learning for three applications: road extraction, building detection and land cover classification (Demir et al. 2018b).

**Fig. 10.13** Automatic extraction of the road mask (right) from the satellite image (left), compared with the ground truth road network (center) (courtesy of I. Demir et al.)

#### *10.3.2 Recurrent Neural Networks*

While convolutional neural networks and dense neural networks work well for static data in which there is no sense of time, a **r**ecurrent **n**eural **n**etwork (RNN) (Jain and Medsker 1999) processes data by iterating through the input elements and maintaining a state that contains information relative to what it has seen until then. An RNN is a neural network with an internal loop (see Fig. 10.14). The state of the RNN is updated between processing independent sequences; therefore, we still consider one data sequence as a single data point in the network. The difference is that this data point is not processed in a single step as opposed to those in dense or convolutional neural networks. In an RNN, the network internally loops over sequence elements until it learns the flow of the data. An RNN is helpful when dealing with a temporal data set. In geospatial data analysis, an RNN has been recently applied in interesting applications such as correction of satellite image classification (Maggiori et al. 2017) and land cover classification (Ienco et al. 2017). Since many types of geospatial data sets such as weather, satellite images, or seasonal animal behavior have timing attached to them, we expect that RNNs will be widely used in the analysis of geospatial data sets in the near future and that Digital Earth will benefit from such networks.

#### *10.3.3 Variational Autoencoder*

Deep neural networks are useful to analyze data sets and are also helpful in generating new data sets. It is possible to consider two deep neural networks as the encoder and decoder of an autoencoder and produce a latent space that represents the data. Using only L and an encoder, we can reproduce a lossy representation of D. However, it is not possible to pick a vector in L and expect to reproduce a meaningful result by feeding it to the encoder because the distribution of L is unknown if autoencoders are used. In **v**ariational **a**uto**e**ncoders (VAEs) (Kingma and Welling 2014), in addition to the compression loss, another loss is minimized that forces the L to be a Gaussian distribution. Thus, VAEs can be used as a generative neural network in which one can sample the Gaussian distribution and feed it to the encoder to generate a new shape that does not necessarily belong to the training data set. Although VAEs have potential to generate data and learn low-dimensional data for geospatial data sets, VAEs have not been extensively tested for geospatial data analysis and generation.

#### *10.3.4 Generative Adversarial Networks (GANs)*

Similar to VAEs, **g**enerative **a**dversarial **n**etworks (GANs) (Goodfellow et al. 2014) are also generative models. GANs consist of a pair of networks that have two different and adversarial roles. These networks have a convolutional architecture and are often complex to retain the complexity of the underlying models. The first network is a generator that we denote as G, which attempts to generate the best result, for example, an image. Then, the second network takes the image as input and tries to infer if it is a generated image or not. This second network is called a discriminator and we denote it as D. Both G and D are trained alternatively. The objective of G is to fool D whereas D aims to avoid being fooled by G. The strength of this kind of adversarial formalism is that it is equivalent to use of a very complex function to train the generator G (encoded into the discriminator), far more complex than traditional distance would be.

**C**onditional **G**ANs (cGANs) are GANs with a particular setup in which the discriminator is trained to recognize the matching between an input image A and an output B whereas a traditional GAN only tests the plausibility of the output without any knowledge of the input. The training principle of a cGAN is explained in Fig. 10.15.

**Fig. 10.15** cGAN principle: a training pair (A, B) is used to learn positive examples. For negative examples, only A is used together with the generator to form the pair (A, G(A))

Conditional GANs have recently been used to automatically generate digital elevation models from user sketches (Guérin et al. 2017). The user sketches the river network, the crests and some altitude cues and obtains a plausible terrain that matches the given constraints, based on a training dataset made of sketch/terrain pairs. The method consists of building such a dataset by extracting the sketch from a real-world terrain. The difficulty of this kind of setup is to automatically build a sketch that is compatible with user sketches, i.e., similar to what a user would draw. Building a sketch that is too close to the terrain features will force the user to draw very precisely, which is not relevant in a sketching context but would be useful in a reconstruction process. The digital elevation model must be simplified to produce simpler features. In their work, Guérin et al. propose initially downsampling the digital elevation model and then smoothing it. This coarse digital elevation model is then processed by a flow simulation, from which the skeleton is extracted. The same process is applied to extract ridges. This feature extraction is illustrated in Fig. 10.16.

The training dataset is formed of pairs that describe the matching between the sketch and the terrain. Figure 10.17 gives examples of such pairs. To create a more pliable terrain synthesizer, the sketches randomly include one, two or the three features among the river lines, crest lines and altitude cues.

Figure 10.18 shows examples of outputs produced by the DEM generator from sketches. The results were obtained by using training from a DEM extracted from the NASA SRTM dataset at 1 arc-second from different locations in the United

**Fig. 10.16** Training database examples

Initial DEM Downsampled DEM Flow simulation Skeleton

**Fig. 10.17** Training database examples.  Sketches can feature river lines (blue), crests (red) and altitude cues (green)

**Fig. 10.18** Examples of generated digital elevation models from simple sketches. A canyon generated using river and crest lines (left). A volcanic island generated using only crest lines (right)

States. In the same article, the authors proposed the use of the same principle to automatically generate digital elevation models from a single level set sketch. They also described examples of automatic void filling in digital elevation models. Finally, because cGANs can embed very complex models, they used it to mimic an erosion process.

#### *10.3.5 Dictionary-Based Approaches*

Approaches based on base function decompositions have intrinsic limitations. Base functions are usually used because they have orthogonality properties that lead to an efficient decomposition. Selecting the base can be difficult because it heavily depends on the nature of the signal. Thus, it can be a viable option to use dictionary-based descriptions. A signal is represented as a linear combination of atoms from a dictionary. Atoms do not need to have special properties such as orthogonality. They are typically chosen directly from the data by picking the most representative signals or by using an optimization. A survey of dictionary-based methods for 3D modeling was conducted by Lescoat et al. (2018). One of the applications of dictionary-based modeling is called sparse modeling, which adds an additional constraint on the number of atoms used to represent the final signal, called *sparsity*.

#### **<IP_ADDRESS> Dictionary Decomposition**

Given a dictionary, the decomposition of a signal consists of finding the best atom, i.e., the atom that maximizes the projection. Then, the same process is applied iteratively to the residual until reaching the target sparsity. This process is called matching pursuit and was introduced by Mallat and Zhang (1993). This decomposition algorithm was further improved by Cai and Wang (2011) by introducing the *Orthogonal Matching Pursuit* (OMP) algorithm. The main difference is that the best decomposition of the already-found atoms is recomputed after each new atom is found.

**Fig. 10.19** An example of terrain amplification that adds plausible details from a given exemplar using a dictionary-based approach. The original terrain had a precision of 1 km, and successive amplifications by a factor of 4 increase the precision to 4 m

#### **<IP_ADDRESS> Dictionary Optimization**

One aim of dictionary based approaches is to find a dictionary that is adapted to a given context or set of signals. This can be done by an optimization process. One goal of this optimization is to minimize the reconstruction error, for example, by computing an L2 distance between the reconstructed signal and the original. It is common to add a constraint on the type of decomposition, for example, by setting a maximum sparsity. Unfortunately, the optimization problem under this type of constraint is too difficult to solve in an optimal way. Heuristics have been proposed that lead to good results with a relatively low cost. K-SVD is one of these algorithms (Aharon et al. 2006), which consists of iterating between two steps. The first step consists of optimizing the decomposition, which can be done using a standard OMP algorithm. The second step optimizes the dictionary with respect to the previously computed decomposition. The two steps are repeated until a number of iterations is reached or a given error is obtained.

Several applications of sparse modeling with terrains have been proposed by Guérin et al. (2016) and Argudo et al. (2018). The terrain is decomposed into patches that compose input signals. A so-called amplification process is used to introduce plausible details into the terrain by mapping between low-resolution and hi-resolution atoms. The dictionary is drawn from an exemplar terrain at high resolution and automatically transformed into low resolution by a trivial downsampling process. The amplification algorithm simply decomposes the patches from a given terrain in the low-resolution dictionary and uses the corresponding high-resolution atoms to reconstruct it. Because the dictionary has been extracted from real terrain, the added details are plausible and realistic, as shown in Fig. 10.19.

#### *10.3.6 Reinforcement Learning*

**R**einforcement **l**earning (RL) is a powerful learning method in dynamic environments (Sutton and Barto 1998). In RL, there is usually an agent in an environment and the agent receives rewards based on its actions. The final goal is to learn how to take actions to maximize the rewards. At any time *t*, an environment is defined by states *St* in which an agent can take action *At* and change the environment state to *St*+1. When

**Fig. 10.20** An agent receives state s\_t, performs an action and receives reward r\_t from the environment. The state of the environment changes to s\_(t+1). This process continues until a terminal state is achieved

the agent takes action *At* , the environment receives a reward *rt* . These iterations continue until the environment reaches a terminal state (Fig. 10.20). Examples of applications that RL can be extremely useful for are games or robot locomotion in which more points and more stable states are the rewards of the game and locomotion environments, respectively.

RL has also been used in applications in GIS and geospatial data analysis. For instance, RL has been used to model land cover changes (Bone and Dragicevic 2009). With recent advances in RL and the growth of computational power, we expect that RL will receive more attention from the GIS and Digital Earth communities. For instance, one application of RL can be to simulate the behavior of endangered species in different simulated environments.

#### **10.4 Discussion**

In the past, machine learning has seen hypes and winter seasons. It started with symbolic AI in the 1960s, which claimed the ability to make machines with intelligence comparable to an average human being in less than a decade. However, people soon realized that they were far from reaching that point. In the 1980s, with the rise of *expert systems*, similar hype was seen in the area of machine learning, followed by a winter season due to the lack of generality of expert systems and their high maintenance costs (Chollet 2017). Recently, deep learning methods became popular again and showed great success in different areas of computer science including geospatial analysis, which is an important portion of Digital Earth platforms. Deep learning will likely continue to grow and be applied more in this field, especially because of the availability of computational power and big data sets that help create more powerful models. However, deep learning cannot solve all problems. For instance, current deep learning models are unable to solve problems that require reasoning or long-term planning (Chollet 2017). Deep learning models work extremely well in mapping an input to a desired output with very little human-level knowledge about the input or output and their effect on industry and science will probably remain for a very long time. There is plenty of discussion about the future of deep learning and AI, notably by its great pioneers such as Lecun et al. (2015) and in the European perspective on AI (Craglia et al. 2018).

Artificial intelligence and particularly machine learning and deep learning have great potential to contribute to the generation, analysis, and management of geospatial data sets. Digital Earth should benefit from such opportunities, as a place holder to represent such data sets and a platform to analyze them. Since Digital Earth is constantly receiving geospatial data sets, a successful Digital Earth should use reliable, fast, and comprehensive techniques to manage and make use of such data. Deep Learning techniques show promise in these directions. However, there are still issues in their use in Digital Earth platforms that must be addressed. In the following sections, we discuss some of these issues.

#### *10.4.1 Reproducibility*

If a technique such as a deep neural network produces particular results, such results should be reproduceable by others. Placing the code on GitHub and providing free access to data sets have been helpful for this issue. However, there are still some issues, especially when the data are owned by a company or the network was designed by an industrial team. In particular neural network architectures, randomness can be included, usually to improve the training. When this randomness is also present in the operational network, it can disrupt the reproducibility of results.

#### *10.4.2 Ownership and Fairness*

Ownership of artifacts provided by machine learning techniques is also heavily under question. If a person with almost no knowledge about a network takes information from available sources, modifies a few parameters, takes data from an available source and produces something unique or obtains a certain analysis, who is the owner of such results? The data owner, developer of the network, or the person who combined these ingredients? In more serious scenarios, who is at fault when a system that works based on machine learning techniques makes a catastrophic mistake or performs a discriminatory action that may involve racism or sexism? Another question is whether data sets and computation power are available to everyone, i.e., do we have "data democratization"? Fortunately, the wealth of free access data sets and code bases along with cheap computational power such as Amazon Web Services (AWS) have resolved some of these issues but we are still far from perfect.

#### *10.4.3 Accountability*

Due to the nature of some algorithms involved in machine learning, it usually cannot be used in contexts where accountability is a strong constraint. This is especially the case with deep neural networks where a lot of information is hidden in the layers, which can lead to unexpected and unwanted results. Conversely, traditional machine learning methods such as linear regressions or PCA are very reliable even if they are limited in terms of applications. Reasonably, one could consider using deep learning methods only when traditional methods fail or are lacking.

#### **10.5 Conclusion**

In conclusion, we provided a sampling of artificial intelligence techniques and their applications in geospatial data generation, analysis, and management. We discussed how AI can be beneficial for generating new terrain data sets, identifying roads and analyzing various geospatial data sets such as satellite imagery. AI techniques and deep learning methods appear very promising. Extensive research on these topics will likely make them even more suitable for use in different domains including geospatial analysis and Digital Earth. However, these techniques are unfortunately standalone and have not been integrated into a Digital Earth platform that makes use of such techniques. Appropriate artificial intelligence techniques should be meticulously included in Digital Earth, considering their pros and cons including fairness and bias to provide interactive, comprehensive and meaningful analysis to users.

#### **References**

McCarthy J (1988) Mathematical logic in artificial intelligence. Daedalus 117(1):297–311

Medsker L, Jain LC (1999) Recurrent neural networks: design and applications. CRC Press, Boca Raton

**Eric Guérin** is Assistant Professor at INSA de Lyon, France. He received an engineering degree from INSA de Lyon in 1998 and a Ph.D. in Computer Science from Université Claude Bernard Lyon 1 in 2002. His research interests include procedural modeling of virtual worlds, natural phenomena and machine learning. He is head of the LIRIS/Geomod team since 2015.

**Orhun Aydin** is a researcher in Environmental Systems Research Institute and assistant professor in Spatial Sciences Institute at U.S.C. Prior to joining ESRI and USC, he obtained a Ph.D. and M.Sc. in Geostatistics from Stanford University. Orhun's research pertains to modelling, exploring and forecasting spatial and spatio-temporal phenomena and their applications for Earth Systems.

**Ali Mahdavi-Amiri** is a University Research Associate at Computing Science department in Simon Fraser University. His research interests are in Computer Graphics with specialty in Geometric Modeling, Geospatial Modeling and Visualization, Digital Fabrication, and Machine Learning. Ali has received several awards including NSERC Postdoctoral Fellowship, Graduate Achievement Award, Michael A. J. Sweeney Award, and Teaching Excellence Award.

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Chapter 11 Internet of Things**

**Carlos Granell, Andreas Kamilaris, Alexander Kotsev, Frank O. Ostermann and Sergio Trilles**

**Abstract** Digital Earth was born with the aim of replicating the real world within the digital world. Many efforts have been made to observe and sense the Earth, both from space (remote sensing) and by using in situ sensors. Focusing on the latter, advances in Digital Earth have established vital bridges to exploit these sensors and their networks by taking location as a key element. The current era of connectivity envisions that everything is connected to everything. The concept of the Internet of Things (IoT) emerged as a holistic proposal to enable an ecosystem of varied, heterogeneous networked objects and devices to speak to and interact with each other. To make the IoT ecosystem a reality, it is necessary to understand the electronic components, communication protocols, real-time analysis techniques, and the location of the objects and devices. The IoT ecosystem and the Digital Earth (DE) jointly form interrelated infrastructures for addressing today's pressing issues and complex challenges. In this chapter, we explore the synergies and frictions in establishing an efficient and permanent collaboration between the two infrastructures, in order to adequately address multidisciplinary and increasingly complex real-world problems. Although there are still some pending issues, the identified synergies generate optimism for a true collaboration between the Internet of Things and the Digital Earth.

**Keywords** Internet of Things · Geospatial standards · Smart scenarios

The views expressed are purely those of the authors and may not in any circumstances be regarded as stating an official position of the European Commission.

C. Granell · S. Trilles (B)

Institute of New Imaging Technologies, Universitat Jaume I, Castelló de la Plana, Spain e-mail<EMAIL_ADDRESS>
A. Kamilaris Department of Computer Science, University of Twente, Enschede, The Netherlands

A. Kotsev European Commission, Joint Research Centre, Ispra, Italy

F. O. Ostermann

Faculty of Geographic Information Science and Earth Observation (ITC), Department of Geo-Information Processing, University of Twente, Enschede, The Netherlands

© The Editor(s) (if applicable) and The Author(s) and European Union 2020 H. Guo et al. (eds.), *Manual of Digital Earth*, https://doi.org/10.1007/978-981-32-9915-3\_11

#### **11.1 Introduction**

According to Jayavardhana (Gubbi et al. 2013), the term Internet of Things (IoT) was first coined by Kevin Ashton in 1999 in the context of supply chain management. Empowered by the latest advances in Information and Communication Technology (ICT), the IoT is revolutionizing the world, opening new possibilities and offering solutions that were unthinkable even only a few years ago. The concept of the IoT is highly multidisciplinary because it brings together a wide variety of technologies, protocols, applications, scenarios, and disciplines (Atzori et al. 2010; Gubbi et al. 2013). The International Telecommunication Union (ITU) Standardisation Sector defines it as '*a global infrastructure for the information society, enabling advanced services by interconnecting* (*physical and virtual*) *Things based on existing and evolving interoperable information and communication technologies*' (International Telecommunication Union 2018). As an infrastructure, the IoT can be seen as a broader system involving data, resources, standards and communication protocols as well as theoretical studies.

The pace of IoT development seems quite fast, with continuous proposals of new approaches, applications, and use case scenarios, increasing the presence of IoT in multiple and varied applications, and aspects of daily life. To date, smart devices constitute the IoT's most visible form, applied in a wide range of scenarios and sectors such as cities, industry, commerce, agriculture, home, and mobility. Although we are far from the 200 trillion smart devices as predicted by 2020 (Intel, n.d.), significant progress has been made in this direction. Estimates suggest that there will be 26 smart devices per person in 2020, 40.2% of which will be located in the business environment (termed Industry 4.0).

According to the Forbes analyst Daniel Newman (Newman 2017), the IoT is one of the most rapidly evolving trends today, especially in three development lines: the analytics arena, the development of edge computing, and the deployment of 5G networks. As 5G technology is progressively implemented and deployed (Shafi et al. 2017), the current analysis platforms will need adaptation in order to analyze effectively the large amount of data flows acquired, produced by IoT devices with increasingly more powerful built-in sensors and emerging real-time analysis functions, empowered even more by the rapid emergence and (parallel) development of edge computing (Shi et al. 2016).

Edge computing is a recent paradigm motivated by bandwidth limitations between the producer (smart objects) and consumer parts (cloud server), as well as the need for improved performance in computing and consumer smart objects. The main feature of edge computing is that data can be processed locally in smart devices rather than being sent to the cloud for further processing.

Like the IoT, Digital Earth (DE) also entails an **infrastructure**. Al Gore, at his famous speech in 1998 (Gore 1998), introduced the concept of a DE with the vision of extending the real Earth with a digital/virtual replica or counterpart. Over the last two decades, many geographic phenomena and observations have been converted to digital data to be used, analyzed, and visualized using digital tools such as virtual globes (Butler 2006). In this chapter, we use the term DE to refer to a network infrastructure that allows for the discovery, access, analysis, and processing of spatially referenced data. For more details on DE, we refer the reader to Schade et al. (2013). In particular, Schade et al. describe the origins and evolving concepts of terms such as DE, Geographic Information Infrastructures and Spatial Data Infrastructures, together with their theoretical and technical features.

This chapter takes a technological perspective focusing on the description of the current relationships between DE and the IoT, identifying ongoing efforts, potential synergies and bridges, as well as existing limitations and barriers that prevent both infrastructures from collaborating and communicating in practical terms. Instead of operating in parallel, scientists and researchers need the IoT and DE to work jointly by establishing an efficient and permanent collaboration to adequately address the multi-disciplinary nature and growing complexity of the pressing problems that characterize modern science.

The rest of the chapter is divided into five sections. In Sect. 11.2, we provide an overview of the most frequent definitions of the IoT, describe our working definitions throughout this chapter, and briefly review related work in the interplay of the IoT and the DE. In Sect. 11.3, we analyze the existing interplay between both infrastructures in the context of the main, high-level functions of DE. Then, an overview of relevant case studies across several smart scenarios in which the symbiosis of the IoT and DE could lead to beneficial results is provided in Sect. 11.4. Afterwards, Sect. 11.5 analyses the frictions and possible synergies today and in the future. Finally, concluding remarks and emerging trends for the immediate future are provided in Sect. 11.6.

#### **11.2 Definitions and status quo of the IoT**

This section defines the current state of the IoT with respect to the concept of the DE. The first subsection examines the different definitions of a 'Thing', adopted by standardization organizations, followed by our working definition for this chapter. The last subsection describes related works in which interaction between IoT and DE is the main goal.

#### *11.2.1 One Concept, Many Definitions*

The concept of a 'Thing' may seem generic. A 'Thing' can be characterized as a network object or entity that can connect to the Internet directly or through a network gateway. This exemplifies a network-centric perspective of the IoT in which a variety of interrelated 'Things' are able to communicate with each other to deliver new applications and services (Atzori et al. 2010). In contrast to the network-centric vision focusing on the communication technologies being used, the IoT can be seen from a purely Thing-centric perspective in which the services associated with Things are pivotal. These services are expected to manage large amounts of data captured by smart objects or 'Things' as a result of interacting with the environment.

Regardless of the vision, the definition of the term 'Thing' is extensive and includes a wide variety of physical elements. Examples of these elements include: (i) personal objects such as smartphones, smart watches or bands; (ii) ordinary objects and appliances in our daily lives such as refrigerators, lights, cars, and windows; (iii) other identifiable objects equipped with Radio-frequency identification (RFID) tags, Near-field communication (NFC), or Quick Response (QR) codes; and (iv) objects equipped with small microcontrollers.

Because of the heterogeneity of the technology and hardware, there is no single, unified definition of the term 'Thing'. Different international standardization bodies and organizations have suggested a definition, resulting in multiple interpretations of the concepts of Things and the IoT, which sometimes differ only slightly. Consequently, each stakeholder group may have a particular view of what the IoT and Things are, as demonstrated below by the definitions of some internationally renowned organizations.