Industry partners from both countries offer high-tech products and services. They have the combined potential to validate many platform aspects and offer selective digital services. Having generally superior products regarding efficiency and cost effectiveness automatically open new sales channels through the platform since the selection process would always suggest the superior product according to customer needs. This leads to challenges when different companies offer similar components, where the differentiation is only represented in smaller but potentially significant details. Those details should be adequately represented in the selection process but are sometimes impractical or difficult to assess e.g. previous experiences with the product, availability and quality of services like maintenance etc.

As with any platform solution, platform growth and quality is generated and therefore highly dependent on both growth of the demand as well as the supply side. Demand side has to create a sufficient pull effect to create enough traction for the supply side to increase their offerings. One of the biggest challenges of the platform is to enable the collaboration between different suppliers, but also making them comparable with little effort. Oftentimes comparable information is not readily available or needs to be homogenized therefore posing hurdles to platform entry. In our risk analysis we analyzed, how potential upsides of products might be overstated, while downsides of products might also be described in less detail or not be stated at all.

Companies would have big interests in offering services on the platform that would accompany sales processes in the form of product service systems. We therefore integrate easily accessible application programming interfaces (APIs) to connect existing services into the platform. E.g. one of them being the planning and configuration of delta robots according to product material, piece number and weight as well as a robotic gripper selection dependent on the product geometries and weight. Furthermore, we connected preexisting shopfloor monitoring platforms with their virtual representation on the platform, to have an integrated view for the customer, on the one side handling the planning phase of the production system but also the execution phase, for which further integrated planning and reconfiguration scenarios can easily be validated virtually.

## **5 Capacity Building**

This project showcases a trend that's currently happening in all of industry. Connecting suppliers, integrating different services for products into platforms generates new and different sales channels, accelerates decision processes and helps reducing the time to market of new products. As a byproduct new jobs are being generated due to new market needs always adjusting to customer needs over time. New online sales channels offer opportunities to higher capacity utilization of manufacturing companies, thus enabling lower price offerings, optimizing workloads over all connected companies. Customers of the platform services who want to set up production for new products have fast, cost transparent and, in comparison likely inexpensive opportunities to set up their own production facilities or reconfigure their preexisting facilities. This creates opportunities for new and relevant jobs in the meantime. The platform itself offers a lot of potential for new work, but also connects existing domains of work. Due to its open framework, partners wanting to offer services related to machine tools, automation, consulting, AI and other micro services, data analytics, maintenance, marketing, etc. are all welcomed and increase platform value and traction with each new contribution. As a result, creating a new marketplace of services and products related to customized production system configurations and reconfigurations.

In the new digital era, 5G-based Plug and Play hardware modules not only give full use of advanced technologies, but also meet the needs and trends of the manufacturing industry's current development. However, the use of new technologies is also changing the associated requirements for professional skills, which poses a challenge for talent development. In order to provide application scenarios for the rapid establishment and operation of production systems, and effectively cultivate the knowledge and skills of young talents and professionals in IIOT, CPS, AI in the field of advanced manufacturing, as well as provide resources for the retraining and upskilling of employees, it is necessary to create an industry-oriented education and research environment, i.e. to establish student-oriented intelligent manufacturing centers. Therefore, a 5G-based turnkey manufacturing facility will be established in China at AMTC and in Germany at KIT for relevant research and education purposes, so that students can personally experience the practical application of 5G, turnkey systems combined with artificial intelligence methods, learn relevant knowledge and independently design and build modular production lines, gain technical experience, train analytical thinking and innovation abilities, and fully convert the learned knowledge into skills. We now shortly describe the design and progress of the case implementation. Currently, the intelligent factory design is being prepared at AMTC. Construction of modular units has begun, with initial results in application development of 5G modules and software for production line configuration and layout. For example, the 5G module is applied for remote access to a Siemens PLC, which can be used to perform both program download, maintenance and troubleshooting. Then, the modules are installed in the modular units and the master control system for real-time communication between these units to support the transfer of operational data, processing programs and operating instructions, and to solve the docking and coordination problems between units. These application scenarios and tools can provide a foundation for building a complete plant. By learning and using 5G modules and software systems, and comparing the construction and efficiency of this generated solution with the traditional manually developed solutions, students can more intuitively appreciate the advantages and application prospects of 5G and plug and play, and master the relevant knowledge and skills.

#### **6 Summary**

In this article we present a Sino-German approach for a factory automation platform in the context of Industry 4.0 and the Internet of Things. We describe technical details of the platform, while also discussing the effects a platform like this will have on the human workforce.

A prototype of a technical solution of a factory automation platform has been implemented during the course of this Sino-German project including basic framework concepts as well as exemplary services for product feature extraction, processing information recommendation, machine tool configuration selection, simulation and validation services, as well as after sales monitoring services.

Through open accessibility, the platform generates new business opportunities for many different stakeholders, enabling new sales channels for production system and component manufacturers, new as well as established companies with new product ideas. Services surrounding and accompanying the product development and production system configuration and -reconfiguration processes as well as monitoring and prognostic elements are manifold. Thus, the factory automation platform creates new digital opportunities for engineering service providers, production system integrators, consulting businesses e.g. in the field of sustainability and circular economy, simulation experts, automation and artificial intelligence experts, creating new and exciting jobs, shifting expertise to the digital realm while collecting data and knowledge for new and more integrated business opportunities.

**Acknowledgement.** The Sino-German project was sponsored by the German BMBF (Federal Ministry of Education and Research) and the Chinese MOST (Ministry of Science and Technology of the People's Republic of China).

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Quantification of Uncertainties in Neural Networks**

Xinyang Wu1, Philipp Wagner1, and Marco F. Huber1,2(B)

1 Fraunhofer Institute for Manufacturing Engineering and Automation IPA, Stuttgart, Germany<EMAIL_ADDRESS>2 Institute of Industrial Manufacturing and Management IFF, University of Stuttgart, Stuttgart, Germany https://www.ipa.fraunhofer.de/

**Abstract.** Artificial neural networks only compute point estimates and thus, do not provide the user with a proper decision space. In high-risk use cases, the confidence of the neural network is an important support for decision-making. Bayesian neural networks extend classical deep neural networks with a probability component and allow the user to assess the probability distribution over the prediction. Due to the large number of parameters to be learned, the calculation of the predictive probability can only be performed approximately in practice. In recent years, many methods have been developed to efficiently learn the parameter distributions for Bayesian neural networks. Each of these has different advantages and disadvantages, and thus can be used for different applications. Quantifying uncertainty in the context of neural networks allows the user to interpret the results more comprehensively as well as to assess the risk and therefore makes an important contribution to the user's digital sovereignty.

**Keywords:** Artificial intelligence · Machine learning · Uncertainty quantification · Neural networks

## **1 Introduction**

In classical machine learning, predictions are usually expressed as point estimates. In a point estimate, the learned model simply returns a single value to the user without informing how confident it is about that prediction. In many use cases, this is acceptable. For example, in the case of movie recommendations on streaming platforms or book recommendations based on books already read, single wrong or bad decisions often have no severe consequences. However, the situation is different in critical application areas of artificial intelligence (AI). In medicine, autonomous driving, or quality testing in industrial production, the financial risks but also, and especially, the impact on humans is significantly greater. Here, a desirable behavior of the AI would be that it provides a confidence to its decision or, in case of very uncertain decisions, signals to the user: "I don't know" or "I am uncertain".

c The Author(s) 2023 A. Shajek and E. A. Hartmann (Eds.): *New Digital Work*, pp. 276–287, 2023. https://doi.org/10.1007/978-3-031-26490-0\_16

Developing new methods with this functionality and extending already existing methods with a probability component is the endeavor of the research field *probabilistic machine learning*. In recent years, the focus here has been particularly on deep neural networks (NNs), as they have become the gold standard for many problems, especially in the area of supervised learning. The quantification of uncertainty by the machine learning model allows the user to assess the quality of the prediction and whether the confidence of the prediction is sufficient for the particular use case. Thus, the user is not left completely alone with the algorithm's decision, but is provided with additional information to help him evaluating this decision and take additional actions if necessary. These can include, for example, retraining with new data if the learning algorithm has a high uncertainty in general, or transferring a specific decision to a human review if the algorithm's decision seems too risky. Uncertainty quantification of machine learning algorithms and in particular NNs can thus make an important contribution to digital sovereignty.

The goal of this work is to motivate the quantification of uncertainty for NNs and to present different methods that make this possible in practice. To this end, the different types of probability and the need for approximate methods are first discussed. Afterwards, the Bayesian NN (BNN) is introduced, which is a principled way to realize probabilistic machine learning. The Sects. 3.1–3.6 deal with methods to approximately calculating the posterior probabilities for BNNs. Finally, we discuss the future of the research field and how it can contribute to digital sovereignty.

#### **2 Probabilistic Machine Learning**

#### **2.1 Basic Principles**

In supervised learning, we always consider a dataset D = {(**x**i, **y**i)}N i=1 with N inputs **x** and outputs **y**. While in classical machine learning a function f with parameters θ is to be learned with fθ(**x**i) = **y**i, in probabilistic machine learning the *output* or *predictive probability distribution* p(**y**|**x**, D) is to be learned. p(**y**|**x**, D) is a conditional probability distribution that gives a probability of output **y** based on the training data D and the current data point **x**. In order to calculate p(**y**|**x**, D), the parameters of the machine learning model used are also assumed to be probabilistic. By integrating over the probability distribution of the network parameters, p(**y**|**x**, D) = - p(**y**|**x**, θ) · p(θ|D) d θ can be used to compute the probability distribution of the output **y**.

Illustratively, by means of the parameter θ, all possible realizations of an NN are considered here in terms of the probability distribution p(**y**|**x**, θ). The output of all possible NNs is averaged, with each NN being weighted by p(θ|D), which represents the probability of different parameter realizations given the training data. Thus, the flexibility and variability in the choice of model parameters is taken into account when calculating the output probability distribution.

In principle, two types of uncertainties must be distinguished in this context. The *aleatoric* uncertainty describes the intrinsic uncertainty within the data used, which is often also referred to as measurement noise. The *epistemic* uncertainty describes the lack of model knowledge, which in our case is reflected in the probability distribution over the model parameters. Based on the available data, for example, different parameter configurations can lead to very similar prediction results. The problem, however, is that the distribution p(θ|D) can mostly only be calculated approximately. For a large number of parameters θ a high-dimensional integration arises, which can be solved exactly only in exceptional cases. Here, it is often assumed that all relevant quantities are normally distributed in order to simplify the problem.

#### **2.2 Bayesian Neural Networks**

If NNs are extended by a probability distribution over their weights, they are called BNNs (see Fig. 1). The concept of BNN has existed for several decades [1], but gained renewed attention in recent years due to the popularity of deep NN. In general, a BNN is characterized not only by model parameters θ, i.e., the weights, but also by a probability distribution p(θ) over all weights. After successful training, this depends on the learned data D and becomes the *posterior distribution* p(θ|D). During the training process, the famous *Bayes' rule* is used to process the information about the training data in order to adjust the distribution p(θ). The Bayesian rule can be formulated for a BNN as

$$p(\theta|\mathcal{D}) = \frac{p(\mathbf{Y}|\mathbf{X}, \theta) \cdot p(\theta)}{p(\mathbf{Y}|\mathbf{X})} \tag{1}$$

with the training data **X** - [**x**1 ... **x**N ], **Y** - [**y**1 ... **y**N ]. Here p(θ) denotes the *prior distribution*, which is assumed to be the initial distribution of weights. This can either already contain information and prior knowledge about the problem or be as general and uninformative as possible. p(**Y**|**X**, θ) is called *likelihood*. It describes how well the training data can be modeled with the available model parameters θ. p(**Y**|**X**) is the so-called *evidence*. It serves as a normalization factor and describes the principal probability over the training data independent of the parameter choice. The distribution over the model parameters is adjusted using Eq. (1) as new data becomes available. p(**Y**|**X**) cannot be calculated exactly in general, which is why Eq. (1) can only be solved approximately in practice. The following section deals with different methods for the approximate calculation of the posterior distribution of the parameters of a BNN based on the available data.

#### **3 Overview of Methods**

#### **3.1 The Dropout Method**

Despite the high performance of deep NN, overfitting to the training data is a major challenge in many use cases. Hinton et al. [2] and Srivastava et al. [3] introduced the dropout method to reduce the negative impact of overfitting for deep

**Fig. 1.** Illustration of a BNN with a single hidden layer (green) compared to a classical NN. The weights are represented as connections between neurons. The weights of the BNN are associated with a probability distribution to model the uncertainty in the parameter choice, while the weights of the NN are deterministic quantities (Source: own illustration).

NN. Subsequently, many papers have been published dealing with the functionality and theoretical understanding of the dropout method. For example, Baldi and Sadowski [4] proposed to interpret dropout as an l2 regularizer in the training process of deep NN. Damianou and Lawrence [5], among others, have proved that a deep NN with dropout layers in front of each hidden layer is mathematically approximately equivalent to a Gaussian process. Based on this, Gal and Ghahramani [6] took one step further and prove that by using dropout, training the deep NN can be viewed as minimizing the Kullback-Leibler divergence (KL divergence) between the approximate probability distribution of the deep NN and the posterior distribution of the underlying Gaussian process. And by using dropout before each hidden layer, regardless of the type of hidden layers (fully connected, convolutional, or recurrent), the deep NN can be considered as BNN with uncertainty quantification.

The application of the dropout method is very simple: First, a dropout layer with an appropriate dropout rate must be added before each hidden layer, regardless of whether it is the first layer after the input layer or the last layer before the output layer. In addition, a regularizer must be chosen for the dropout layer. The authors recommend l2 regularization if the goal is to have uncertainty increase far from the data.

For normal deep NNs, the dropout layers are only active during training and are switched off in the inference phase. In the case of BNNs, the dropout layers remain active during the inference phase to provide an estimate of the probability distribution instead of a point estimate for a prediction. This means that even though the deep NN has exactly the same input, it can make different predictions because the structure of the network is slightly different for each inference due to the dropout layer. Therefore, for each input, we can get n different output values for n times inference. Gal and Ghahramani [6] proved that the mean and standard deviation of these n output values are approximately equal to the mean and standard deviation of the posterior Gaussian distribution of the underlying Gaussian process with the given input. In practice, the mean is considered as the final prediction for the given input and the standard deviation quantifies the uncertainty for this prediction. Considering that the estimation of the probability distribution is based on the multiple repetition of the inference process, the method is also called *Monte Carlo Dropout* (MC Dropout).

A disadvantage of MC Dropout is that this method introduces new hyperparameters, for example, the dropout rate for the dropout layers. To address this issue, Gal et al. published a new method called Concrete Dropout in [7] that allows automatic exploration of the dropout rate and allows deep NNs to dynamically adjust their uncertainty quantification as more data are observed. This variant saves the user time for fine-tuning the dropout rate. However, the training and inference of this method requires more resources than normal dropout-based BNN.

#### **3.2 Ensembles**

As mentioned in the previous section, in MC Dropout the prediction is summarized from the multiple repetitions of the network's inference processes with the same input, and in each inference the network structure is slightly changed due to the active dropout layers. In this regard, MC dropout could also be interpreted as *ensemble* of multiple deep NNs [3], where the individual NNs differ due to the dropout layers but still share most of the parameters. Lakshminarayanan et al. propose in [8] to use an ensemble of several differently initialized NNs with the same structure directly as an approximation to the Bayesian inference model instead of MC dropout. The advantage is that additional hyperparameters, such as dropout rates for each dropout layer, are avoided. This interpretation motivated the authors to investigate ensembles under the name *Deep Ensemble* as an alternative approach for uncertainty quantification of deep NNs.

In Fig. 2 we show the difference between MC Dropout and Deep Ensemble. In the case of Deep Ensemble, all the networks in the ensemble have the same structure, but have been randomly initialized differently for training, and the data points in the dataset are randomly shuffled for training each network.

Based on the Deep Ensemble method of Lakshminarayanan et al., Pearce et al. proposed a new method called *Anchored Ensembling* in [9]. Compared to Deep

**Fig. 2.** Difference between MC Dropout and Deep Ensemble (Source: own illustration).

Ensemble, Anchored Ensembling regularizes the parameters of the deep NN with assumed prior probability distributions. The authors report better performance and more accurate probability estimation. However, the prior probability distribution must be carefully chosen. In our experiments, the performance of this method has been shown to be sensitive to the selected hyperparameters.

#### **3.3 Variational Inference**

In addition to the previously presented simple methods for approximate Bayesian inference, *Variational Inference (VI)* attempts to formulate and solve the Bayesian inference problem as an optimization problem. In this section, we will first explain the mathematical principles behind this method and then introduce some useful tools for the application of VI.

As explained in Sect. 2.2, a common problem with BNN is that the evidence p(**Y**|**X**) is difficult or even impossible to calculate exactly. To avoid the intractability of the evidence in Eq. (1), VI adopts a simpler surrogate function q(θ) to approximate the true posterior probability distribution p(θ|D). The measurement of the similarity between two probability distributions is the KL divergence

$$\text{KL}(q(\theta) \| p(\theta|\mathcal{D})) = \int q(\theta) \left[ \log \frac{q(\theta)}{p(\theta|\mathcal{D})} \right] d\theta \,\,. \tag{2}$$

Here, the optimal surrogate function for the approximation is exactly the function that minimizes the KL divergence, i.e.,

$$q^\*(\theta) = \arg\min \text{KL}(q(\theta) || p(\theta | \mathcal{D})) \,. \tag{3}$$

Bishop et al. [29] proved that minimizing the KL divergence is equivalent to maximizing the *evidence lower bound loss function (ELBO)*

$$\begin{split} \text{ELBO}(q(\theta)) &= \int q(\theta) [\log p(\mathbf{Y}, \theta | \mathbf{X}) - \log q(\theta)] \, \mathrm{d}\theta \\ &= \int q(\theta) \log p(\mathbf{Y}|\mathbf{X}, \theta) \, \mathrm{d}\theta - \mathrm{KL}(q(\theta) \| p(\theta)) \, . \end{split} \tag{4}$$

Here, (4) is equal to the term log p(**Y**|**X**) − KL(q(θ)p(θ|D)). Given that the KL divergence is always non-negative, it can be seen that ELBO(q(θ)) is always less than or equal to the *log-evidence* log p(**Y**|**X**), which explains the name of this loss function. Since the evidence is constant and independent of θ, maximizing the ELBO automatically leads to minimizing the KL divergence. Consequently, the optimization problem in Eq. (3) can be rewritten into

$$\begin{split} q^\*(\theta) &= \arg\min \text{KL}(q(\theta) \| p(\theta | \mathcal{D})) \\ &= \arg\max \int q(\theta) \log p(\mathbf{Y} | \mathbf{X}, \theta) \, \text{d}\, \theta - \text{KL}(q(\theta) \| p(\theta)) \, . \end{split} \tag{5}$$

In Eq. (5), it can be seen that the intractable evidence p(**Y**|**X**) is not required as the posterior distribution p(θ|D) is avoided. Instead, we only need to process the known priori distribution p(θ) and the likelihood p(**Y**|**X**, θ), which are easier to handle. In this way, we simplify the inference problem into a solvable and tractable optimization problem.

If we examine Eq. (5) again, we notice that the first part of the equation corresponds to a maximum likelihood estimator and the second part of the equation adds a prior part to the ELBO loss function to make the estimation based on the prior knowledge, hence it resembles the maximum a posteriori estimator in this aspect.

Several open-source libraries for Bayesian inference and probabilistic modeling for machine learning have already been developed. Table 1 includes a comparison and summary of the most popular probabilistic programming language tools. In our work and research, we usually use Pyro and Tensorflow Probability (TFP), depending on the deep learning frameworks we choose. The two libraries are preferred because they are under active development and are associated with well-known frameworks for deep NNs.

#### **3.4 Laplace Approximation**

The *Laplace Approximation* is based on a relatively simple idea [10]. Often, only the region around the maximum is of interest in the posterior distribution. Provided that this maximum is known, the distribution in its neighborhood can be approximated with a Taylor-series expansion. The maximum θMAP can be determined by classical training via gradient descent. For the Taylor-series expansion up to the second order we can obtain

$$\log p(\theta|\mathcal{D}) \approx \log p(\theta\_{\text{MAP}}|\mathcal{D}) - \frac{1}{2}(\theta - \theta\_{\text{MAP}})^\top \mathbf{H} \left(\theta - \theta\_{\text{MAP}}\right) \tag{6}$$

**Table 1.** Current open-source libraries for Bayesian inference and probabilistic modeling. All libraries support VI and MCMC methods.

with the Hessian matrix **H**. The first term of the Taylor-series expansion vanishes as we expand around a maximum. The Taylor-series expansion around θMAP thus results in a normal distribution of the form

$$p(\boldsymbol{\theta}|\mathcal{D}) \approx p(\boldsymbol{\theta}\_{\text{MAP}}|\mathcal{D}) \cdot \exp\left(-\frac{1}{2}(\boldsymbol{\theta} - \boldsymbol{\theta}\_{\text{MAP}})^{\top} \mathbf{H} \left(\boldsymbol{\theta} - \boldsymbol{\theta}\_{\text{MAP}}\right)\right) \tag{7}$$

with Hessian matrix being the inverse of the covariance matrix. However, this approximation of p(θ|D) is not normalized and thus not yet a valid probability density. With the normalization factor of a multivariate normal distribution we obtain

$$p(\boldsymbol{\theta}|\mathcal{D}) \approx \frac{1}{\sqrt{|2\pi\mathbf{H}|}} \exp\left(-\frac{1}{2}(\boldsymbol{\theta} - \boldsymbol{\theta}\_{\text{MAP}})^\top \mathbf{H} \left(\boldsymbol{\theta} - \boldsymbol{\theta}\_{\text{MAP}}\right)\right) \,. \tag{8}$$

A limiting factor is the calculation of the Hessian matrix, since it can quickly become very large for a huge number of parameters. In practice, therefore, further approximation methods are often used to calculate **H**. This allows the method to be used even for large NNs. An advantage of the Laplace approximation is that it can also be applied to already trained NNs to add an uncertainty component in a post-hoc fashion. Since the method approximates the true distribution only locally, the approximated distribution can in principle deviate strongly from the true one. In most cases, however, especially for large data sets, satisfactory results can be achieved with the Laplace approximation.

#### **3.5 Kalman Filter-Based Approaches**

While VI treats the Bayesian inference problem as an optimization problem using the ELBO loss function, many other researchers attempt to treat this problem as a filtering problem using Kalman filters. Classical Kalman filters are applicable only to linear systems; however, many variants extend them quite well to nonlinear systems. Singhal and Wu presented the first algorithm in 1989 that uses the extended Kalman filter to train BNN [11]. Compared to the normal gradient-based and batch-based methods, such Kalman filter-based methods proved to be much more effective than the standard backpropagation in terms of

**Fig. 3.** Comparison between Gaussian Process (GP), Stochastic VI, MCMC method and KBNN on a synthetic classification dataset. The first row shows the predictions for the binary classes. The second row shows the uncertainty quantification of the predictions in the data space. The three BNN variants have similar higher uncertainty in the transition region between classes (Source: own illustration).

the number of training epochs [12]. Watanabe and Tzafesta proposed a different approach in [13], in which the weights of the networks are assumed to be Gaussian distributed and the mean and variance of each weight are updated using an extended Kalman filter, but which requires local linearization for updating the neurons in hidden layers. This method was extended by Puskorius and Feldkamp [14] to allow for layer-wise correlated or even network-wide correlated neurons.

To avoid linearization, Huber proposes in [15] the so-called *Bayesian Perceptron*. Even though it is restricted to a single neuron, this work proves that a closed-form computation of the mean and covariance matrix of the posterior weight distribution is possible, provided that the weights are assumed to be Gaussian distributed. Based on the Bayesian Perceptron, Wagner et al. in [16] extended this method from a single neuron to a multilayer perceptron (MLP) and called it *Kalman Bayesian Neural Network (KBNN)*. In this work, a closed-form forward and backward propagation of the weight distributions in each layer is introduced. This method shows its advantages in terms of online learning capacity and learning efficiency compared to other popular BNN methods such as VI and MCMC. Figure 3 shows a comparison between different BNN methods. Among them, KBNN has the best learning efficiency. Chen et al. introduced another method [17] in 2021, which uses the ensemble Kalman filter to handle the measurement noise in the data and to account for it in the uncertainty quantification.

#### **3.6 Markov Chain Monte Carlo**

Another popular and well-researched approach for Bayesian inference to learn BNNs is the *Markov Chain Monte Carlo (MCMC)* method. A Markov chain describes a process in which the current state depends only on the last state. Under certain assumptions and starting from an initial state, a Markov chain always converges to a stationary distribution. Once this distribution is reached, all further states will correspond to this distribution. However, it is in general not known a priori how many steps along the Markov chain are necessary until the stationary distribution is reached.

In the context of Bayesian inference, Monte Carlo methods without Markov chains form a broad class of sampling algorithms that use repeated random sampling to generate samples from complex posterior distributions. Rejection sampling [23] is the basic Monte Carlo method to generate samples from a given distribution. However, rejection sampling is very inefficient because the samples are uncorrelated. Combining these sampling methods with various algorithms to construct Markov chains for the desired probability distribution (e.g., the posterior distribution of weights in BNNs) yields MCMC methods whose stationary distribution are proportionate to the desired posterior distribution. Hence, the samples of the stationary distribution represent an approximation of the posterior distribution and thus, characteristic parameters of this distribution such as the mean or the variance can be calculated.

One of the earliest MCMC methods is the Metropolis-Hastings algorithm [24,25]. Many improvements have been proposed for that, such as Gibbs sampling [26], hybrid Monte-Carlo or Hamiltonian-Monte-Carlo (HMC) [27]. A major extension of HMC is the No-U-Turn Sampler (NUTS) [28], which usually works much more efficiently. The libraries listed in Table 1 support MCMC methods as well as VI.

## **4 Conclusion**

In this paper, we have presented BNNs as an approach to achieve uncertainty quantification in the field of deep learning. Similar to other Bayesian inference problems, BNNs assume a prior probability distribution of the weights of the NN and attempt to learn a posterior distribution of these weights using the available data.

Several methods have been proposed in the literature to realize the training and inference of BNNs as efficiently as possible. Dropout and ensemble methods are among the simple variants, where the learning and inference of BNNs is considered as aggregation through an ensemble of different networks. However, the weights are considered to be deterministic. In the other, "real" Bayesian methods, the weights of the NN are modeled as random variables. Variational inference formulates the learning task as an optimization problem, which is approximately solved by using the ELBO loss function. In comparison, MCMC algorithms use Markov chains to draw samples from the posterior distribution of weights. While the Laplace approximation utilizes the local Taylorseries expansion to obtain an additional uncertainty quantification on trained NNs, Kalman filter-based methods propose to formulate the calculation of the posterior distribution of the weights as a filtering problem in which the weights are recursively updated.

Compared to conventional NNs, BNNs offer advantages in active learning, causal inference, out-of-distribution detection, and security-related use cases thanks to the capability of uncertainty quantification. In the future, we focus on the scalability of different methods and the applications of BNNs in novel fields, such as reinforcement learning and verification of AI. We believe that uncertainty quantification will be crucial for increasing the reliability and explainability of AI systems, which are key elements for digital sovereignty at the level of robustness and trustworthiness.

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **A Final Word**

Alexandra Shajek(B) and Ernst Andreas Hartmann

Institut für Innovation und Technik, Berlin, Germany<EMAIL_ADDRESS>
Having reached the end of this book, we will now draw a provisional conclusion and give an outlook on further projects. As already mentioned in the introduction (Hartmann and Shajek 2022, this volume), the Institute for Innovation and Technology (iit) has been working on the future of work for many years. In the publications of the 2010– 2020 decade (in particular the volumes by Botthof and Hartmann 2015; Wischmann and Hartmann, 2018), the main aim was to present fundamental theories on the then still new topic of Industry 4.0, to initiate a socio-political discourse, and subsequently to compile and focus on practical examples from the field of research and development.

In the early 2020s, a new substantive focus was then set on digital sovereignty and outlined in its complexity with two edited volumes (Hartmann 2021, 2022). In the present publication, we now put the workplace in the center and focus on the opportunities, possibilities, and limits of digital sovereignty from a wide range of academic disciplines, for different industries and in different parts of the world. What becomes clear is: Digitalization now permeates almost every aspect of our workplace, it has complex and multi-layered effects, and it can only be analyzed here by way of example.

At the same time, our work is characterized by a fundamentally optimistic view: Digitalization can enable sovereignty in the first place or promote sovereignty and open up the scope for action, for example, with the help of digital assistance systems (see the contribution by Bächler and Behrendt 2022, this volume) or through the possibilities of digitally supported education (see the contributions by Kanyane 2022, this volume and Windelband 2022, this volume). In addition, approaches are presented to positively shape the change towards digital work. The underlying assumption here is that there is also freedom of choice in the digital workplace and no technological determinism (see also Botthof and Hartmann 2015). All facets resulting from the dimensions transparency and explainability, confidence and freedom of action for people, and technology and organization (Hartmann and Shajek 2022, this volume) can be positively influenced.

This focus inevitably leaves out some aspects that will, however, also be of central importance for the (digital) future of work: From a work psychology perspective, these include questions of psychological stress resulting from the changed working conditions and corresponding occupational health management issues (some individual aspects are, however, dealt with in the article by Mayer et al. 2022, in this volume). But there are also many aspects arising from the digitalization of workplaces for the interactions of employees, e.g., for the cooperation between supervisors and their employees. In this context, these are common conditions for success in digital work (see, e.g., Busch-Heizmann et al. 2021). Digital work platforms are also given far less consideration (cf. Hartmann and Shajek 2022, in this volume), which might be a consequence of the editors' one-sided German perspective. Questions of the cognitive-enhancement debate also remain largely untouched.

However, the topic of the future of work has not yet come to a close for us: We are already planning another volume with which we want to deepen the understanding of digital sovereignty at the corporate level (or any other type of organization). In particular, the publication intends to support management-level actors in designing strategies, products, structures, and processes to improve digital sovereignty. Topics include external challenges to organizational autonomy, such as the nature of digital products and markets, legal frameworks, and industry policies related to digital markets and products. Managing digital sovereignty at the enterprise level shall be addressed, with topics including, e.g., skills development or risk management as well as product and service design in support of digital sovereignty. Also planned are case studies to illustrate conditions for success.

We hope therefore to be able to provide one or two impulses in the future that will at least maintain and, if possible, promote the ability of companies and employees to act in the face of the constantly expanding capabilities of digital technologies.

 Thank you for giving us a profound insight into the aspects of digital work to which you devote your research. Without such constructive and smooth cooperation, the publication of this work would not have been possible. Our thanks also goes to our colleagues at the Institute for Innovation and Technology (iit), in particular Désirée Tillack, Alexandra Lescher, and Pierre Dombrowski, for their excellent support.

Alexandra Shajek and Ernst Hartmann in November 2022.

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Author Index**

#### **B**

Bächler, Liane 118 Behrendt, Hauke 118 Bönsch, Jakob 68 Brzoska, Stefanie 135 Bulander, Rebecca 68

#### **C**

Chardonnet, Jean-Rémy 87

#### **D**

Deuse, Jochen 31, 161 Dumitrescu, Roman 51

**E** Elliott, Stuart 16

**G** Guertler, Matthias 161

#### **H**

Häfner, Polina 87 Häfner, Victor 68 Hahn, Peter 135 Hartmann, Ernst Andreas 1, 135, 171, 288 Huber, Marco F. 276

#### **K**

Kanyane, Modimowabarwa 149 Kimmig, Andreas 68, 260 Ködding, Patrick 51 Koldewey, Christian 51 Kreuzwieser, Simon 68 Krieger, Benedikt 135 Künzel, Matthias 135

**L** Lammers, Thorsten 161

#### **M**

Mayer, Anjela 87 Michels, Felix 68 Mühlbradt, Thomas 214

**O** Ovtcharova, Jivka 68, 87, 260

#### **P**

Peng, Jieyang 260 Pentenrieder, Annelie 135 Peters, Robert 135

#### **R**

Rieger, Christoph 31

#### **S**

Schaffrath, Scarlet 135 Senderek, Roman 239 Shajek, Alexandra 1, 171, 288 Sick, Nathalie 161 Staneva, Mila 16

#### **W**

Wagner, Philipp 276 Wagstyl, David 31 Weßkamp, Vanessa 31 Windelband, Lars 195 Wöstmann, René 31 Wu, Xinyang 276

#### **Y**

Yan, Min-Ren 171


*3.7. Secondary Outcomes: E*ff*ects of Psilocybin on Systolic Blood Pressure, Diastolic Blood Pressure and Heart Rate*

Systolic blood pressure (SBP), diastolic blood pressure (DBP) and heart rate were evaluated in this meta-analysis as secondary outcomes. The obtained results are summarized in Table 4. As noted, psilocybin produced significant increases in SBP and DBP, up to 6 (*p*-value < 0.017) and 5 h (*p*-value < 0.001) following administration, respectively. The DBP verified increases between 1.194 and 11.381 mmHg, being the average of the increase 7.741 mmHg. Both SBD and DBP tended to stabilize to normal values after 6 or more hours after administration. Psilocybin also significantly increases the heart rate, this increase being highest on the 3rd and 4th hours following administration. Similarly to what was verified for SBP and DBP, heart rate tended to stabilize after 6 or more hours after the administration.

**Table 4.** Meta-analysis for secondary outcomes.

WMD—weighted mean differences; CI—confidence interval; \* Indicates a significant result.


*1.3.5 Coupling agents*

KH-550 silane coupling agent chemically known as 3-(2,3-epoxypropoxy) propyl trimethoxy silane is selected for silica nanoparticle surface modification. It contains an organic functional group, a linker, a silicon, and a hydroxyl group. The organic functional group can bond to the organic aromatic polyimide ring. The general chemical formula for the silane group and the hydrolyzable functional group, typically alkoxy, amine, or chlorine, involved in the reaction with the inorganic silica substrate. The silanol groups of the nanosilica surface can react with the hydroxyl groups of the silane after hydroxylation through hydrogen bonding. The covalent bond improves interfacial adhesion between inorganic silica particles and organic monomer molecules.


**6. Conclusion**

Development of the analytical methods for identification, purity evaluation and quantification of statin drugs has received a great deal of attention in the field of pharmaceutical analysis in recent years. This review includes trends and advances in separation methods developed for the analysis of statin molecules with different physical and chemical properties. The chapter surveys the application of chromatographic techniques for the determination of statins in pharmaceutical dosage forms and biological samples.

A Review of Current Trends and Advances in Analytical Methods for Determination of Statins: Chromatography and Capillary Electrophoresis 419

[4] Bu D, Griffin G, Lichtman AH (2011) Mechanisms for the anti-inflammatory effects of

[5] A.W. Alberts, J. Chen, G. Kuron, V. Hunt, C. Hoffman, J. Rothrock, M. Lopez, H. Joshua, E. Harris, A. Patchett, R. Monaghan, S. Currie, E. Stapley, G. Alberts-Schonberg, O. Hensens, J. Hirshchfield, K. Hoogsteen, J. Liesch, J. Springer (1980) Mevinolin: A highly potent competitive inhibitor of hydroxymethyl glutaryl-coenzyme A reductase and a

[6] Endo A (1979) Monacolin K, a new hypocholesterolemic agent produced by a *Monascus*

[7] Schachter M (2005) Chemical, pharmacokinetic and pharmacodynamic properties of

[8] Corsini A, Maggi FM and Catapano AL (1995) Pharmacology of competitive inhibitors of

[9] Jemal M, Xia YQ (2000) Bioanalytical method validation design for the simultaneous quantitation of analytes that may undergo interconversion during analysis. J. Pharm.

[10] Nirogi R, Mudigonda K, Kandikere V (2007) Chromatography-Mass Spectrometry Methods for the Quantitation of Statins in Biological Samples. J. Pharm. Biomed. Anal.

[11] Yang DY, Hwang LS (2006) Study on the Conversion of Three Natural Statins from Lactone Forms to their Corresponding Hydroxy Acid Forms and their Determination in

[12] Huang Z, Xu Y, Li Y, Wang Y (2010) Conversion Investigation for Lovastatin and Its

[13] Li Y, Zhang F, Wang Z, Hu Z (2004) Identification and Chemical Profiling of Monacolins in Red Yeast Rice Using High-Performance Liquid Chromatography with Photodiode Array Detector and Mass Spectrometry. J. Pharm. Biomed. Anal. 3: 1101-

[14] Huang HN, Hua YY, Bao GR, Xie LH (2006) The Quantification of Monacolin K in Some Red Yeast Rice from Fujian Province and the Comparison of the Other Product, Chem

[15] Li YG, Liu H, Wang ZT (2005) A Validated Stability Indicating HPLC with Photodiode Array Detector (PDA) Method for the Stress Tests of *Monascus purpureus* Fermented

[16] Song F, El-Demerdash, Lee SSH, Smith RE (2012) Fast Screening of Lovastatin in Red Yeast Rice Products by Flow Injection Tandem Mass Spectrometry. J. Pharm. Biomed.

[17] Novakova L, Šatinsky D, Solich P (2008) HPLC Methods for the Determination of

[18] Ortega Markman BE, Pires Rosa PC, Walter Koschtchak MR (2010) Assessment of the Quality of Simvastatin Capsules from Compounding Pharmacies. Rev. Saude Publica

cholesterol-lowering agent. Proc. Natl. Acad. Sci. USA 77: 3957–3961.

statins: an update. Fundam. Clin. Pharmacol. 19: 117-125.

HMG-CoA reductase. Pharmacol. Res. 34: 9-27.

Pu-Erh Tea. J. Chromatogr. A. 119: 277-284.

Derivates by HPLC. J. Chromatogr. Sci. 48: 631-636.

Rice, Red Yeast Rice, J. Pharm. Biomed. Anal. 39: 82-90.

Simvastatin and Atorvastatin, Trends Anal. Chem., 27(4):352-367.

statins. Curr. Opin. Lipidol. 22: 165-170.

sp. J. Antibiot. 32: 852–854.

Biomed. Anal. 23: 813-827.

Pharm Bull 54: 687-689.

Anal. 57: 76-81.

44: 1055-1062.

44: 379-387.

1112.

Stability indicating methods and papers describing impurity profiling are discussed in this review. Special emphasis is given to sample preparation as unavoidable and delicate step in bioanalitical methods for quantification of statins and their metabolites. The hyphenated technique that incorporates the efficient separation using liquid chromatography and sensitive detection by mass spectrometry has become an indispensable tool for quantification of statins in biological fluids and pharmacokinetic studies. Methods describing simultaneous analysis of different statins as well as drugs in combined pharmaceutical products and other co-administered drugs in therapy of cardiovascular disease are also described. The application of capillary electrophoresis as alternative separation technique for statins is considered and compared with chromatographic methods.

The use of statin drugs has augmented in recent years and is expected to increase further in the years ahead because high cholesterol and cardiovascular diseases are being diagnosed more frequently. Therefore, the development of new analytical methods for commercially available statins as well as novel upcoming statin drugs will be a future challenging task for many analysts.


**Reliable Road Scene Interpretation Based on ITOM with the Integrated Fusion of Vehicle and Lane Tracker in Dense Tra**ffi**c Situation**

#### **Jinhan Jeong, Yook Hyun Yoon and Jahng Hyon Park \***

Department of Automotive Engineering, Hanyang University, Seoul 04763, Korea<EMAIL_ADDRESS>(J.J<EMAIL_ADDRESS>(Y.H.Y.)

**\*** Correspondence<EMAIL_ADDRESS>
Received: 26 March 2020; Accepted: 24 April 2020; Published: 26 April 2020

**Abstract:** Lane detection and tracking in a complex road environment is one of the most important research areas in highly automated driving systems. Studies on lane detection cover a variety of difficulties, such as shadowy situations, dimmed lane painting, and obstacles that prohibit lane feature detection. There are several hard cases in which lane candidate features are not easily extracted from image frames captured by a driving vehicle. We have carefully selected typical scenarios in which the extraction of lane candidate features can be easily corrupted by road vehicles and road markers that lead to degradations in the understanding of road scenes, resulting in difficult decision making. We have introduced two main contributions to the interpretation of road scenes in dense traffic environments. First, to obtain robust road scene understanding, we have designed a novel framework combining a lane tracker method integrated with a camera and a radar forward vehicle tracker system, which is especially useful in dense traffic situations. We have introduced an image template occupancy matching method with the integrated vehicle tracker that makes it possible to avoid extracting irrelevant lane features caused by forward target vehicles and road markers. Second, we present a robust multi-lane detection by a tracking algorithm that incudes adjacent lanes as well as ego lanes. We verify a comprehensive experimental evaluation with a real dataset comprised of problematic road scenarios. Experimental result shows that the proposed method is very reliable for multi-lane detection at the presented difficult situations.

**Keywords:** automated driving system (ADS); sensor fusion; multi-lane detection

#### **1. Introduction**

The Euro NCAP, NHTSA and ISO have published assessment protocols to meet the proper criteria of the well-known Advance Driver Assistance System (ADAS). The assessment protocols have been renewed almost every year and have become more sophisticated [1]. These documents contain the main fundamental functions of road environment perception to perform ADAS functionalities, like Adaptive Cruise Control, Automatic Emergency Braking and Lane Keeping System. However, each level of the automated driving system (ADS) has an Operational Design Domain (ODD) which means the system is not capable of constant full driving automation since these protocols are subject to the current production line for automakers [2]. To move towards advanced ADS, enormous surges of autonomous driving technologies have focused on the perception of the road environment as well as vehicle control.

Among the autonomous driving technologies, road environment perception requires the ability to use onboard sensors to extract lane markers, road users (like vehicles and pedestrians), and infrastructure (like traffic signs and traffic structures). Especially, the most important but basic research fields are lane and vehicle detection and tracking. We can currently utilize these two states of ego

lane and vehicles to adjust the spacing/speed control and the lane keeping system which corresponds to level-2 driving automation out of the 6 levels of driving automation [3]. To advance to level-3 or ultimately full AD, the system requires the ability to overcome issues with reliable and robust multi-lane detection and tracking in complex environments. Especially, the adjacent lanes deliver more specific information to the ADS module to reliably be able to perform accurate and comfortable driving. For example, if the system can predict paths of the surrounding vehicles through the road geometry from lanes, including adjacent lanes, the system can take more safety actions in advance depending on the road scene interpretation. Therefore, to realize highly-advanced ADS technologies, such as planning driving strategies in a complex environment, the perception module must provide key driving information about the accurate lane information, including adjacent lanes, to situational awareness processes.

Most existing lane detection research has concentrated on securing robustness to environmental factors, such as various challenging road types, and shadow and light conditions. Several important studies have explored to multi-lane detection [4–11]. However, there is still a lack of research on adjacent lane detection in the cases that lane features are disturbed by heavy traffic situations or poor visibility. These factors are a critical problem for the previous lane tracker frameworks since most tracking processes rely on the given sequence of image frames. In this paper, we focus on resolving the problems in the extraction of lane candidate features due to occlusion and disturbing features caused by the presence of traffic, road markers, and shadows. We describe the poor extraction of lane candidate features originating from traffic, especially, when the adjacent lane of the road and road markers pose bad influential factors on the process of lane model fitting.

The proposed framework for multi-lane tracking is divided into two main parts: one is an image template occupancy matching (ITOM)-based lane tracker, the other is a frame-level detection approach [6]. Each part was applied with our practical data fusion techniques that combine the vehicle tracker and lane tracker to obtain improvement in managing the adjacent lane as well as the ego lane with an experimental dataset acquired from onboard sensors. Our approach has main contributions of a novel framework of multi-sensor data fusion for a maximum four-lane tracker in the following aspects:

This specialized road scene interpretation method will be discussed throughout the paper. The rest of this paper is organized as follows. Section 2 will explore and summarize related works. Section 3 describes the system overview, including the problem definition, algorithm overview, and problem formulation. The main algorithm is divided into two parts, with the method described in detail in Sections 3 and 4. The experimental results are presented in Section 5. Finally, the conclusions can be found in Section 6.

#### **2. Related Work**

Recently, as the ADS techniques have advanced, the need for reliable road scene interpretation, including multi-object detection and tracking methods, has steadily increased. Among these methods, vision-based lane detection and tracking have been stated as one fundamental component of ADS functionalities [2]. In brief, previous research has mainly focused on robust and accurate lane detection. Regarding the specific details in the related works, the lane detection scope can be divided into two parts; single-lane detection and multi-lane detection, as shown in Table 1. According to the detection scope, various approaches for lane detection have been conducted considering road types and conditions with different strategies and characteristics. In single-lane detection, there are various driving situations including from simple road condition [12] to urban areas [13,14], and road environmental difficulties with more complex scenarios [15–18]. For robustness in lane detection and tracking, parameter estimation based on probabilistic approaches in the form of tracking, using the well-known Kalman Filter (KF) and particle filter, should be conducted. The use of the KF, which is a well-known clothoid lane model-based approach that uses lane features to fit lane models, offers prediction and the smoothing of outliers. However, it is clear that the single model-based approach is not the best choice due to the frequently poor quality of lane features. Therefore, a combination with a multi-model tracking algorithm [12,19] or a feature-based approach, like the ones presented using particle filters [19,20], integrated detection and tracking in the usage of particle filters in [13,16], provides a way to handle the discontinuous road conditions. In [16], Kim introduced a method to find robust lane-boundary hypotheses combining the Random sample consensus (RANSAC) algorithm for detection and a particle filter for tracking. Some other approaches have also proposed a robust lane detection based on the M-estimator Sample Consensus (MSAC) [17], which has a better performance than the RANSAC method for lane model fitting [15]. Recently, several methods based on deep learning for single lane detection have been proposed in [18,21]. Multi-lane detection needs more sophisticated algorithm due to the unknown number of lanes and increase complexities. For a multi-lane feature clustering and association, a method based on conditional random field (CRF) [4,5], lane stability optimization [6], and homgraphy matrix estimation [7,10] are presented. For a multi-lane tracking, particle filter is used considering the dependencies of multiple lane geometric constraints in [8,11], and spline model based KF is also presented [9]. On the other hand, the previous study deal with a robustness on the occlusion and shadow for a short time. For a consideration of complex road environment in practical dense traffic, multi-sensor data fusion to combine data that is not accessible from an individual sensor is inevitable [22–25]. Over the past few decades, researchers have made substantial progress in multiple research fields studying object tracking. In this paper, we propose a novel framework of multi-sensor data fusion for a multi-lane tracker that works well in a dense road scene interpretation.

#### **3. System Overview**

#### *3.1. Multi-Sensor Data Fusion Framework*

The main goal of this research is to detect and track a maximum of four-lanes through an integration of road scene information including forward vehicle tracking. As related works have shown in Section 2, it is difficult to handle an algorithm for multi-lane detection that covers all of the various scenarios. Most studies solve the difficulties to combine detection and tracking or apply methods with known robust estimation methods under the tracking process [12–16]. In this paper, the proposed multi-lane tracker framework has key processes for the robust extraction of lane candidate features, including the preprocessing of multi-sensor data, with the integration and fusion of the vehicle tracker and lane tracker. The integrated fusion system with the vehicle tracker compensate for the regions of unwanted feature points that make multi-lane detection difficult in the top-view image frame of the flowchart in Figure 1, which will be explained in detail in Section 4.

**Figure 1.** System overview of the multi-sensor data fusion framework.

• Preprocessing

The coordinate systems are aligned with the point of origin of the front-centered-ground of the ego vehicle which follows a standard SAE coordinate system, but differentiates the center of the rear axis from the front-centered-ground to handle all of the sensor data intuitively in Figure 2a. The *X*-axis is positive toward the front and the *Y*-axis is positive toward the left side on the vehicle. The coordinate matching for each sensor should be conducted since they are installed at different mounting positions. For a radar sensor in Figure 2b, it is easy to employ the coordinate by a simple transition from polar coordinates to vehicle coordinates. However, for the camera in Figure 2c, the image frame should be rectified into the top-view image, which can be aligned with the same vehicle coordinates by Inverse Perspective Mapping (IPM) [15–17]. Through the image transformation, we intuitively interpret a road scene situation with the integrated vehicles and lanes information at the same time. To obtain a top-view image or bird-eye view (BEV), we need to find a homography matrix by the camera calibration process in Figure 2d. For clarity, each sensor transformation is constructed as follows:

• The transformation from each sensor frame to the vehicle local frame:

$$\text{webubic local } frame: \text{ : } \langle \mathbf{X}\_{\text{local}} \rangle \mathbf{y}\_{\text{local}} \text{)}\,, \tag{1}$$

• Typical installation positions of the camera and radar sensor:

$$\mathbf{Radar} : \begin{bmatrix} X\_{r\text{\textquotedblleft local}} \\ Y\_{r\text{\textquotedblleft local}} \end{bmatrix} = \begin{bmatrix} R\_r \cos(\theta\_r) \\ R\_r \sin(\theta\_r) \end{bmatrix} \tag{2}$$

$$\mathbf{Camera} : \begin{bmatrix} X\_{v\\_local} \\ Y\_{v\\_local} \end{bmatrix} = \begin{bmatrix} x\_{BEV} + l\_{offset} \\ y\_{BEV} \end{bmatrix} \tag{3}$$

• Image transformation

$$\text{A camera matrix : } H\_{3 \times 4} = \begin{bmatrix} fx & 0 & cx \\ 0 & fy & cy \\ 0 & 0 & 1 \end{bmatrix} [R|T]\_{3 \times 4'} \tag{4}$$

A homography transformation matrix : *Trans* = *H*3×4(:, [124]), (5)

The camera intrinsic parameters (*f x*, *f y*, *cx*, *and cy*) and extrinsic parameters (*R*: rotation, *T*: Translation matrix) are obtained from the process of the camera calibration in Figure 2d.

**Figure 2.** Multi-sensor coordination alignment process: (**a**) multi-sensor coordinate matching, (**b**) radar coordinates, (**c**) transformation of image frame to top-view frame, (**d**) camera calibration, (**e**) transformation into vehicle coordinates.

Once we obtain the homography matrix through the Equations (4) and (5), we can find a point (*u*, *v*) in the image frame corresponding to the point (*xBEV*, *yBEV*) which indicates Figure 2c,e as follows: Vehicle to image transformation:

$$\left(\left(\mathbf{u}',\mathbf{v}',\mathbf{a}\right)^{\mathrm{T}} = \mathrm{Trans}\,\ast\left(\mathbf{x}\_{\mathrm{BEV}},\mathbf{y}\_{\mathrm{BEV}},\mathbf{1}\right)^{\mathrm{T}}\left(\mathbf{u},\mathbf{v}\right)^{\mathrm{T}} = \left(\mathbf{u}'/\alpha,\mathbf{v}'/\alpha\right)^{\mathrm{T}},\tag{6}$$

Image to vehicle transformation:

$$\left(\mathbf{x}', \mathbf{y}', \mathbf{a}\right)^{T} = \mathbf{T} \mathbf{x} \mathbf{a} \mathbf{s}^{-1} \ast \left(\mathbf{u}, \mathbf{v}, \mathbf{1}\right)^{T} \left(\mathbf{x}\_{\text{BEV}}, \mathbf{y}\_{\text{BEV}}\right)^{T} = \left(\mathbf{x}'/a, \mathbf{y}'/a, \mathbf{1}\right)^{T},\tag{7}$$

#### *3.2. Experimental Setup*

The test vehicle is equipped with a radar, vision module, and a camera. The self-collected datasets are acquired for evaluating the proposed multi-lane tracker in a variety of complex road conditions and the validation process was conducted in post-process. The onboard perception module obtains the radar object tracks, vision lane/object tracks, and ego vehicle data from the CAN bus and the current image frame at different cycle times. As shown in Figure 3, the multi-rate and multi-sensor fusion flow chart depicts the test sensor setup and how the system works with a different sensor set.

• A 76-77GHz (174 m, +/−10◦) and mid-range (60 m, +/−45◦) delphi-ESR1.1 radar provides simultaneous multimode electronic scanning simultaneously. However, we are only concerned with the object's range (*Rr*), azimuth (θ*r*), and speed (*vr*) data up to 64 targets within the region of interest of the extent of the image frame (35 m).

**Figure 3.** The multi-rate and multi-sensor fusion flow chart.

#### **4. Integrated Fusion of the Vehicle and Lane Tracker**

In the case of a presence of vehicles to the front or side, or road markers and shadows, it is impossible to consistently track multiple lanes consistently without any complementary solutions. We present an integrated fusion method for the multi-lane tracker and vehicle tracker that are complementary to the process of updating each track state. The robust multi-lane detection and tracking algorithm consists of four main steps: a vehicle tracker with vision and radar, robustness under complex road conditions with ITOM, frame level multi-lane detection, and multi-lane track management as shown in Figure 4.

**Figure 4.** The robust multi-lane detection and tracking algorithm.

#### *4.1. Vehicle Tracker*

In this research, the vehicle tracker is responsible for combining the multi-rate multi-sensor track data fusion. The issues in the vehicle tracker process can be categorized into three steps: validating the track data, data association, and lowered delay of track fusion, which is part of the track-to-track fusion with the radar and vision system [24]. In other words, these datasets have already tracked the object states. One major characteristic is that the radar track includes clutter and noisy radar data, from guardrails, bridges, tunnels, etc., besides road users within the radar sensor range while the vision track only indicates certain vehicles in front of the ego vehicle. Therefore, we should consider the two issues of validating the track data and data association. First, a region of interest (ROI) is set up for validating the track data. The ROI is based on a lane boundary, which is determined by a lane tracker, as shown in Figure 5. In the case of Figure 5, where the lane tracker has four active lanes, the region is restricted to along the adjacent lane boundaries for the lateral direction and a distance limitation for longitudinal direction. The plot describes that the irrelevant radar data, such as guardrails and clutter, can be filtered out with the ROI. Secondly, the track data from each sensor decides whether fused track state and the new incoming track data is admissible inside gating using the Mahalanobis distance which is derived from χ2-distribution [25]. Finally, the track update process follows the work well accomplished by the use of the Kalman Filter [26]. In the track update process, there are three states which are track creation, standby, and confirmed. We take the confirmed track when the vision track comes at the first time of track update states, while we take the standby track when the radar track arrives first due to the uncertainty. That is, the track update rules give priority to the vision track for lowered delay time while updating vehicle track states.

**Figure 5.** Multi-object data association within region of interest.

#### *4.2. Lane Tracker*

As mentioned in the related works, many unfortunate influential factors make lane detection surprisingly difficult, even if the extraction of lane candidate features from a road may seem straightforward. Since most tracking processes depend on the lane candidate features, the previous lane tracker frameworks are easily destroyed by consistently wrong or absent lane candidate features that occur from occlusion, poor visibility, or shadows in dense traffic conditions as shown in Figure 6. While these road conditions can be handled by previous research methods for a short time. ADS must catch adjacent lanes to be able to aware of the current traffic situation in the case of complex road conditions. To accomplish this special case, we present a robust lane tracker that secures adjacent lanes as well as ego lanes while integrating the vehicle tracker. In brief, the proposed lane tracker consists of two main parts: frame-level multi-lane detection and tracking, and securing robustness in complex road condition.

**Figure 6.** Three bad influential factors for lane detection: (**a**) a challenging scenario for road adjacent lane detection due to a vehicle in the side lane, (**b**) a challenging scenario for road lane detection due to road markers, (**c**) a shadowy road condition, with (a1,b1,c1) lane candidate points from a bird's-eye view and (a2, b2, c2) lane candidate points in the vehicle coordinates.

4.2.1. Frame Level Multi-Lane Detection and Tracking

• Lane Candidate Feature Extraction

This method requires initial lane candidate feature extraction. The features are extracted using a designed kernel edge filter based on the dark-light-dark (DLD) intensity transition characteristic [14]. For the kernel design, we first transform the image frame to a bird's-eye view image with IPM. From the rectified image frame, we analyze the lane width in the bird's-eye view image and design the specified kernel function, which consists of emphasizing the lane feature function (*fE*) and shrinking the road region feature function *fS*. Finally, we can extract the lane candidate features from the specified kernel combination function *fc*, as shown in Figure 7a–c and Equations (8)–(10).

$$f\_E = conv(I, \begin{bmatrix} -1 \ 0\_{1 \times n} \ 2 \ 0\_{1 \times n} - 1 \end{bmatrix}),\tag{8}$$

$$f\_S = \text{conv}(I, \begin{bmatrix} 1 \ 0\_{1 \times n} \ 0 \ 0\_{1 \times n} - 1 \end{bmatrix}) - \text{conv}(I, \begin{bmatrix} 1 \ 0\_{1 \times n} \ 0 \ 0\_{1 \times n} 1 \end{bmatrix}),\tag{9}$$

$$f\_{\mathbb{C}} = af\_{\mathbb{E}} + f\_{\mathbb{S}} \tag{10}$$

where *conv*(*I*, *kernel*) means a convolution with image frame *I* (a 640 × 480 pixel image) and *kernel*. The α scale factor indicates the amplification of the brightness. However, this extraction method is very unstable under shadows because of the gradient changes over different road environments. To handle the shadow problem, an adaptive threshold has been employed in this work. The process of reconstruction in the shadow region is conducted as shown in Figure 7d–g. Figure 7d represents the multi-lane ROI derived with the information from the front vehicle states and previous lane track states. Within the multi-lane ROI, we take an adaptive threshold for each lane to reconstruct lane candidate features as shown in Figure 7e. Finally, we can obtain well-resolved features of the current frame and we can compare the results in Figure 7f,g.

**Figure 7.** A robust lane feature extraction method, (**a**) emphasizing lane features (indicated by brightness), (**b**) shrinking the road region (darkness), (**c**) combining the two kernels, (**d**) frame level multi-lane region of interest (ROI), (**e**) reconstruction of the shadowy region features with a lane-level adaptive threshold, (**f**) lane feature extraction results without an adaptive threshold, (**g**) lane feature extraction results with an adaptive threshold.

#### • Robust Lane Detection and Tracking

The lane model described in [27] is a third order polynomial, clothoid model. In this work, we reduce the order of model to a parabolic or linear model according to feature quality because these models are more stable in complex traffics, and the model can be formularized as follows;

$$\text{Clothoidal model}: \; f\_{\text{indcx}}^{\text{3nd}}(l) = y\_0 + \psi l + \frac{\mathcal{C}\_0}{2}l^2 + \frac{\mathcal{C}\_1}{6}l^3 \tag{11}$$

$$\text{Parabolic module}: \ f\_{\text{index}}^{2 \text{nd}}(l) = y\_0 + \psi l + \frac{\mathbb{C}\_0}{2}l^2 \tag{12}$$

$$\text{Linear model}: \; f\_{\text{index}}^{1st}(l) = \psi l + y\_0 \tag{13}$$

where *index* indicates the *le f tego*, *rightego*, *le f tadjacent*, *rightadjacent* lanes, respectively, and the lane parameters consist of *y*0, ψ, C0, C1 which are the lateral offset, lane heading, lane curvature, curvature rate, and *l* is a longitudinal distance.

For multi-lane detection, the proposed approach has been designed to handle more sophisticated individual lane detection based on the MSAC [28,29] under strictly controlled feature selection methods using the frame-level track coherence. Taking advantage of this characteristic, we applied an individual lane feature selection strategy to find the feature cluster more quickly and efficiently as shown in Figure 4. The individual four lanes have their own constraints based on a validation boundary region of the lane model parameters in Equation (12). Since the lane features cannot be changed abruptly, we take advantage of the validation boundary regions defined by the estimated lane parameters in the previous frame where the candidate points are located in the current frame. Through multi-lane track management, we group the subject multi-lane parameters (*le f tego*, *rightego*, *le f tadjacent*, *rightadjacent*) and manage them separately, as shown in Figure 8a. Every lane tracker cycle time increment, we check for adjacent lane validations, as shown in Figure 8b,c. We make sure of two check points (lane types and *Lifetime*) for clarity to validate the adjacent lane: a check on whether each ego lane is a dashed or solid lane, and the life time, which represents the number of track update after track creation, of the adjacent lane (*Lifetime* > 4). The reason for finding lane types is to determine whether the ego vehicle is in a driving edge lane, which means the lane no longer exists over the lane boundary. We also double-check any vehicle that has the same direction and velocity with the ego vehicle for third check.

**Figure 8.** Lane track management, (**a**) subject multi-lane (*le f tego*, *rightego*, *le f tadjacent*, *rightadjacent*), (**b**,**c**) two cases of adjacent left lane with lane validation logic.

#### 4.2.2. Robustness under Complex Road Conditions

In feature-based methods, Otsu thresholding segmentation has often been used and proven in preprocessing [23]. This thresholding method is also useless when the whole image frame is full of bright objects, when most of the image frame region is occupied by road markers, or in the presence of bright cars on the road. Therefore, we intend to mask all these factors to extract brightness intensities only corresponding to the lane features. In order to eliminate these factors, we introduce a novel concept of image template occupancy matching (ITOM). This concept finds a literally occupied region that matches with vehicles or road markers using predefined templates, as shown in Figure 9a. These templates consist of base templates and vehicle templates based on each lane geometry. The base template is responsible for searching hypotheses for road markers or vehicles, as shown in Figure 9b. After finding a hypothesis location, the occupancy-matching procedure is conducted. Each base template determines the occupancy of the vehicle position on the image from the front vehicle states from the vehicle tracker. If there are no matching results with a vehicle, it regards the candidate as a road marker, as shown in Figure 1 (results). The ITOM-based multi-lane tracker results are shown in Figures 10 and 11. In Figure 10a depicts the results of multi-lane detection on an image frame, Figure 10b displays the lane candidate features, Figure 10c depicts noisy lane candidate features in the vehicle coordinates, Figure 10d shows base templates occupied with road markers and the lane geometry, Figure 10e shows the elimination of road makers for each lanes, Figure 10f shows the lane candidate feature with removed road markers, and Figure 10g shows the multi-lane detection results for current frame. In Figure 11, when vehicles and road markers appear at the same time, the ITOM-based multi-lane tracker has successfully eliminated the disturbance factors and parameterized the driving environment.

**Figure 9.** Image template occupancy matching (ITOM) concept plot, (**a**) template design, (**b**) search base template and match on image frame.

**Figure 10.**

**Figure 11.** Complex road conditions: (**a**) template-based detection, (**b**) ITOM remapping on the origin image frame, (**c**) robustness in complex road conditions.

#### **5. Experimental Results**

#### *5.1. Experimental Methodology*

The performance of the reliable road scene interpretation based on ITOM with the integrated fusion of the vehicle and lane tracker in a dense traffic situation is evaluated through mainly self-collected dataset from real driving data in dense traffic conditions. The purpose of this experiment mainly focuses on the specific driving conditions which fall under the consistent traffic occlusion from forward vehicles and the existence of road markers and shadows that yield incorrect lane parameter estimations. The video with a resolution of 640 × 480 from the evaluation dataset are manually labeled using the Matlab ground truth labeler. If the differences between the tracked multi-lanes and the ground truth are within a pre-defined threshold, a true positive detection is counted, otherwise a false negative [9]. The performance metric is as follows: A true positive (TPR) = (the number of detected lanes)/(the number of target lanes), a false positive rate(FPR) = (the number of false positives)/(the number of target lanes) [5,7].

#### *5.2. Experimental Results*

We test and verify the multi-lane tracker performance on the normal PC, Intel(R) Core i9-9900KF CPU @ 3.60GHz and 32.00GB RAM. The processing speed is around 25 frame per second (fps). It is a little slow, but it has enough possibility to optimize the proposed algorithm which is reasonable speed for ADs in a real time processor. A total of 3350 frames were tested and the target four-lanes are considered within 35m. The performance metric shows that the overall TPR of the algorithm for identifying lanes, including adjacent lanes, is 99.5 percent and the adjacent lane TPR is 96.8 percent, as shown in Table 2. This is a significant improvement compared to the Mobileye 630 lane tracker results, which has a TPR of only 90 percent and 88.6 percent respectively. We show a variety of the experimental results and validation in Figure 12. The figure describes presence of road marker and multiple vehicle ahead of ego vehicle. We picked specific difficult scenarios in Figure 12 which depict a road marking scenario in first row of (a), vehicle appearance on right or left lane in the rest rows of (a), mixed scenarios of complex road marking and vehicle ahead of ego vehicle in (b), and finally validation with the tracked multi-lanes and ground truth in (c). It is shown that the proposed multi-lane tracker outperformed in the ego lanes and adjacent lanes through disturbing features elimination.

**Table 2.** Various approaches for lane detection.

(a) (b) (c)

**Figure 12.** Experimental results and validation (**a**) scenario I: successful masking of the road marker, side vehicle emergence, front, and side vehicles. (**b**) Scenario II: successful masking of a complex scenario, (**c**) ground truth and results from the top view for validation.

#### **6. Conclusions**

This work proposed and evaluated a road scene interpretation algorithm based on ITOM with the integrated fusion of the vehicle and lane tracker in a dense traffic situation. The system mainly focused on resolving the problems in the extraction of lane candidate features due to three bad influential factors in complex road conditions under the process of a multi-lane tracker. To overcome this problem, the framework for a multi-lane tracker presents two key processes for the robust extraction of lane candidate features. The first process introduces an elimination process of bad influential factors including the presence of vehicles and road markers in the current top-view image frame through the proposed ITOM method. In this process, we consider validating track data, data association, and the radar track-to-vision track fusion method which gives priority to vision track for a lower delay time while updating vehicle track states. In the second process, the proposed robust methods for frame-level multi-lane detection and tracking are presented, including the extraction of lane candidate features under shadows, a frame-level lane hypothesis region, and managing the multi-lane tracker using the MSAC algorithm. In the experiment, we have carefully selected dataset to verify specific bothersome scenarios in which the study is interested. The test results showed that the proposed method can significantly improve the detection rates when it comes to the adjacent lanes under complex road conditions. Through this framework, we ensure multi-lane detection and tracking that make further driving strategies with situational awareness in ADs. However, this work needs to check the reliability in much more road scenarios and a variety of road types as well since this work mainly focus on the methodology of robust lane feature extraction and managing in the presence of road marking, vehicles on the road and shadows.

**Author Contributions:** Conceptualization, J.J.; Methodology, J.J.; Software, J.J.; Supervision, J.H.P.; Validation, J.J. and Y.H.Y.; Visualization, J.J.; Writing – original draft, J.J.; Writing – review & editing, J.J. and J.H.P. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by the Technology Innovation Program (Fault Detection and Diagnosis for ADAS-Sensors and Hazardous Analysis and Development of Fault Management Strategy, 10076338) funded by the Ministry of Trade, Industry & Energy (MOTIE, Korea).

**Acknowledgments:** The experiment sensors and equipment were provided and supported by the Hyundai motor company.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).