Therefore, in addition to the supervised learning model an approach is presented, which does not involve training and manual data labeling. This section discusses the influencing effects and proposes methods to process non-labeled data as well as labeled data.

### **4.2.2 Influencing effects**

First, the principal influencing effects of the vehicle body vibration is discussed followed by the determination of important influencing effects of the sound pressure level of tire cavity sound (SPL(TCS)).

Figure 4.6 shows the standard deviation of the vertical acceleration and the roll rate of the vehicle body driving over various road features with different velocities. The standard deviation is calculated for a frame length of 5 m. The experiment was conducted four times for each velocity and road feature. The marker in the error bar plot represents the mean of the measurement values and the error bars show the standard deviation of the four observations. The standard deviation for the figure is multiplied by 10 for a better visualization of the error bars. The error bars remain small except for observations for pothole with velocities 40 and 50 km/h.

The trend of the observations indicate, that the vibration of the vehicle is dependent on the velocity. The vibration due to a pothole remains constant from 50 to 60 km/h since the wheel does not fully rebound. In contrast, a manhole cover above surface level increases the vibration of the vehicle body with higher velocities. Furthermore, the vibration from the smooth surface slightly increases with the velocity. Further influencing effects, such as load or type of vehicle, are investigated with the extended simulation approach in Section 3.5.

Figure 4.6: Vehicle vibration (standard deviation (SD)) due to road features for various velocities.

To determine and quantify the influence of the various variables on the SPL(TCS), data from an experimental study [159] are analyzed. The experiments were performed under controlled conditions with the internal drum test bench from the Institute of Vehicle System Technology at Karlsruhe Institute of Technology, which is described in various dissertations and publications, recently in [160]. Table 4.1 shows the full factorial variation of the parameters of the considered variables for the experimental study.

The mean profile depth (MPD) of the selected asphalt is 0.56 mm and of concrete 1.38 mm. The sound pressure level (SPL) of the tire cavity sound ranges from 100 to 200 dB. The tire temperature was measured with a sensor placed in the tire torus.

Regression analysis is performed to estimate the relationship of the independent variable (Table 4.1) and the dependent variable, SPL(TCS). Firstly, simple regressions are conducted and afterwards a multiple regression with all considered independent variables.

Table 4.1: Considered variables and variation of their parameters.

Since the data is from experiments and subject to outliers, a robust regression instead of a standard least squares regression is applied. An overview of robust estimators can be found in [161].

Here, a bisquare weight function with default tuning constant of 4.685 is performed [162]. A *t*-test statistics is used to test that the coefficient is not equal to zero but has significant influence on the dependent variable.

Figure 4.7: Exemplary results from the investigation of the influence of various variables on the SPL of the tire cavity sound. Y¯0 represents the average regression line. (\* p < 0.05, \*\* p < 0.01, \*\*\* p < 0.001)

Figure 4.7 shows the influence of various variables on the SPL of the tire cavity sound and Figure 4.7a indicates that speed has the major impact on the SPL. The regression lines indicate the influence of the variables, whereas the data points in Figure 4.7b, c, and d are colored corresponding to the velocity. The captions represent the average regression line. The figure suggests that the surface in this experiment has the second strongest impact.

Table 4.2 shows the results from a multiple robust regression. For the regression the values of the features are scaled into the range of 0 to 1 for a better comparison of the regression coefficients.

Table 4.2: Variable and corresponding regression coefficient.

A closer investigation of the influence of speed controlled for all other described variables show, that it has a non-linear incluence on SPL(TCS). Figure 4.8 shows the observations of the experiment [159] as well as the lines of a robust linear and robust non-linear regression.

The non linear regression model better fits the observations, which confirms the mean square error (MSE) of 2.04 for the residuals of the non linear and 2.85 of the linear model.

Figure 4.8: Observations and lines of a linear and non linear regression model for SPL as dependent and velocity as independent variable with all other variables constant.

This finding corresponds to the results of many experimental investigations of the dependence of the rolling noise on the rolling speed and on the temperature. These have even found their way into the standard describing the legal noise release testing of tires (ISO 13325:2003).

The best fit of the measured values results from a linear regression of the logarithm of the speed. The non linear dependency needs to be kept in mind for the development of a method to process the tire cavity sound data (Section 4.2).

## **4.2.3 Road unevenness and roughness estimation with non-labeled data**

This section presents the developed and applied methods to process data of inertial and acoustic sensor without a training process. The advantage is that no labeling process is needed, which is time consuming and expensive. So far, no approach has been presented, which does not involve a training process or calibration of the vehicle. However, an absolute value for the road condition, such as the international roughness index (IRI), can not be estimated, and specific attributes, such as potholes, can not be detected without a training process.

The aim of the methods is to estimate a road condition score based on the intensity of the vehicle vibration and tire vibration. Moreover, influencing effects on inertial sensor and acoustic sensor measurements, identified in subsection 4.2.2, should be incorporated. The score of various vehicles for identical road segments should be merged to reduce outliers of single vehicles. Lastly, the results, namely the road condition score, is to be mapped and potential road hazards are to be automatically labeled. Herefore, the following data processing, summarized in Figure 4.9, is developed.

After preprocessing of data (Section 4.1), features are extracted for specific framelengths of the vehicle trajectory from the space series, which represent the vibration of the vehicle or tire mainly due to road unevenness or roughness. Based on these features, a road condition score is calculated for each frame of all vehicles and all trajectories are divided into road segments. The scores from various vehicles in each road segment are merged to one score. Afterwards, a double thresholding algorithm of the road segments is applied to suppress road segments in good road condition and to combine road segments in bad road condition. In the next, the connected-component labeling algorithm automatically detects and labels areas of the map in bad road condition or with obstacles. Lastly, these areas are exported with various information, such as the severity and the size of the obstacle, and the overall calculated scores, representing the road condition, are visualized on a map.

Figure 4.9: Overview of methods to process non-labeled data.

#### **Inertial sensor data**

To describe and evaluate the vibration of the vehicle due to road obstacles, the weighted root mean square (RMS) acceleration is extracted from the inertial sensor data. The RMS acceleration is described in [163]. It is expressed in m/s 2 for translational vibration and rad/s 2 for rotational vibration. The running weighted RMS takes occasional shocks and transient vibration into account, e.g. through potholes, and is in space domain defined by

$$a\_{WRMS}(\mathbf{s}\_0) = \sqrt{\frac{1}{\tau} \int\_{s\_0-\tau}^{s\_0} a\_W(\mathbf{s})^2 \mathbf{d} \mathbf{s}},$$

where *a*W (*s*) is the instantaneous frequency-weighted acceleration and τ is the integration space for sliding average. Since the aim of this study is to identify defects in the road surface, the acceleration is not weighted to keep all information of the signal. However, for the identification of subjectively disturbing defects, the signal can be multiplied with a weighting function. In discrete form, the RMS is calculated with

$$a\_{WRMS}(\mathbf{s}\_0) = \sqrt{\frac{1}{k} \sum\_{i=0}^{k} a\_W \mathbf{2}[(\mathbf{s}\_0 - i)]},$$

with *k* as the numbers of data points in one frame.

To consider vibration due to overall road unevenness and single lane obstacles like potholes, the sliding weighted RMS acceleration is iteratively calculated from the vertical acceleration and the roll acceleration of the vehicle body.

To reduce the influence of the velocity on the vibration of the vehicle, a statistical data binning is applied and the running weighted RMS acceleration is grouped into bins of velocity intervals. The velocity bins can be freely defined. However, a good option is to define velocity bins representing velocity intervals for 30 km/h zones, inner-city roads, rural roads, and freeways.

Firstly, the maximum and minimum of both the RMS of the vertical and roll acceleration are determined for each velocity bin *i* and vehicle *j* (minj,i(*a*zRMS),maxj,i(*a*zRMS),minj,i(ω˙ xRMS),maxj,i(ω˙ xRMS)). After having acquired a sufficient amount of data, reasonable values for the maxima and minima are identified. However, the values of the extrema are updated with new acquired data.

Secondly, the scaled RMS for vertical and roll acceleration for small segments of the trajectory and for each vehicle *j* is calculated with

$$\begin{split} \text{scaled } a\_{\mathbb{z}\_{j}\bar{\jmath}\_{RMSS}} &= \frac{a\_{\mathbb{z}\_{r}RMSS} - \min\_{j,i}(a\_{\mathbb{z}\_{r}RMSS})}{\max\_{j,i}(a\_{\mathbb{z}\_{r}RMSS}) - \min\_{j,i}(a\_{\mathbb{z}\_{r}RMSS})},\\ \text{scaled } \dot{\omega}\_{\boldsymbol{x}j\_{RMSS}} &= \frac{\dot{\omega}\_{\boldsymbol{x}\_{r}RMS} - \min\_{j,i}(\dot{\omega}\_{\boldsymbol{x}\_{r}RMSS})}{\max\_{j,i}(\dot{\omega}\_{\boldsymbol{x}\_{r}RMSS}) - \min\_{j,i}(\dot{\omega}\_{\boldsymbol{x}\_{r}RMSS})}. \end{split}$$

Both scaled values are in the range of 0 to 1. To include the strongest shock or vibration of the vehicle due to road obstacles, the maximum of the scaled RMS for vertical and the RMS roll acceleration is calculated, which is referred to as road unevenness index (RUI):

$$RUI\_{f,i} = \max\left(\frac{a\_{\mathbb{Z}RMS} - \min\_{f,i}(a\_{\mathbb{Z}RMS})}{\max\_{f,i}(a\_{\mathbb{Z}RMS}) - \min\_{f,i}(a\_{\mathbb{Z}RMS})}, \\ \begin{aligned} &\frac{\dot{\omega}\_{\times RMS} - \min\_{f,i}(\dot{\omega}\_{\times RMS})}{\max\_{f,i}(\dot{\omega}\_{\times RMS}) - \min\_{f,i}(\dot{\omega}\_{\times RMS})} \end{aligned} \right).$$

The final score RUI for vehicle *j* ranges from 0 to 1 and represents the road condition, whereas 1 is bad condition.

To reduce outliers of single vehicles, the RUIs of various vehicles for the same road segment are merged. For this purpose, the road infrastructure area to be analyzed is divided into regular quadrilaterals, as exemplary illustrated in Figure 4.10, and the frames of the vehicle trajectories are assigned to the corresponding quadrilaterals. The resolution of the quadrilaterals, referred to as road segments, can be individually defined. Lastly, the RUIs of various vehicles for each road segment are combined by the arithmetic mean or maximum. Hereby, the arithmetic mean reduces outliers and the maximum takes the worst calculated RUI of one vehicle into account. Furthermore, the standard deviation of the RUIs can be calculated to evaluate the variation of the score among vehicles or drives.

Figure 4.10: Road infrastructure divided into segments.

To highlight road segments with a high excitement on the vehicle, which indicates a road hazard, a double threshold algorithm and a connected component labeling algorithm are applied.

The double threshold algorithm accounts for noise and variation of the score and merges road segments with similar RUIs. The algorithm suppresses road segments with a score below a low threshold *T*1. Road segments with a score above the high threshold *T*2 are marked as relevant segments. If the score of road segments with a threshold above the low threshold and below the high threshold is close to relevant segments, it is connected to those segments, and otherwise suppressed.

The procedure is illustrated in Figure 4.11 and described with the following assignment procedure (4.1) for RUI of road segment *S*(v,*u*), where (v,*u*) is the location expressed by the longitude and latitude.

*S*(v,*u*) is assigned to a region *R*0, which is of no interest, or a region *R*2, which is of interest and contains only road segments with a high RUI. *S* 0 (v 0 ,*u* 0 ) is any connected road segment to *S*(v,*u*). The assignment procedure (4.1) ends if no more road segments can be assigned. *A* is the the identifier of the attribute.

$$\begin{aligned} A\_{S(\boldsymbol{\nu},\boldsymbol{\mu})} &:= R\_0, \text{ if } \mathrm{RUI}\_{S(\boldsymbol{\nu},\boldsymbol{\mu})} \le T\_1, \\ A\_{S(\boldsymbol{\nu},\boldsymbol{\mu})} &:= R\_2, \text{ if } T\_1 < \mathrm{RUI}\_{S(\boldsymbol{\nu},\boldsymbol{\mu})} \le T\_2 \text{ and } \mathrm{RUI}\_{S'(\boldsymbol{\nu}',\boldsymbol{\mu}')} > T\_2, \\ A\_{S(\boldsymbol{\nu},\boldsymbol{\mu})} &:= R\_2, \text{ if } \mathrm{RUI}\_{S(\boldsymbol{\nu},\boldsymbol{\mu})} > T\_2. \end{aligned} \tag{4.1}$$

Figure 4.11: Double thresholding method. Black segments ≤ T1, T1 < red segments ≤ T2, blue segments > T2.

The connected component labeling algorithm (Algorithm 4.1), firstly presented by Rosenfeld and Pfaltz [164], clusters and labels the connected road segments with a high score obtained from double thresholding. Finally, the scores of the road segments can be visualized and the clusters from connected component labeling represent the road hazards.

The identified areas with road hazards can be summarized in a list with the position of the center of the cluster, the radius of the cluster and additional statistics, such as the minimum, average, maximum and standard deviation of the score over the trajectories, the number of drives through the road hazard, and the mean and standard deviation of the velocity.

#### **Acoustic sensor data**

The data from the acoustic sensor in the tire cavity is processed with the same method as described for the inertial sensor except for the feature extraction. Hereby, the SPL of the tire cavity sound is calculated to estimate the roughness of the road. The estimation is referred to as road roughness index (RRI).

The SPL feature is more sensitive to the velocity and therefore finer velocity intervals with 7.2 km/h steps are used. Furthermore, the data set to be investigated is reduced by data, for which the tire temperature, tire pressure and load is out of a specific defined range to have comparable data and to avoid data acquired under extreme conditions. For example, during parking of the vehicle, the tire cavity adopts a similar temperature of the environment, which can range from -10 ◦C in the winter to 40 ◦C in the summer. However, various drives in different environment conditions have shown that the temperature stays approximately constant at 35 to 40 ◦C after 10 min drive. Consequently, the beginning of a drive and longer stops when tire temperature decreases again are filtered out analogous to the tire pressure, which approximately follows the tire temperature (Figure 3.9). The remaining data is then processed according to the flow chart in Figure 4.9.

#### **Trend identification**

A trend over time of the estimated RUI or RRI can be identified by applying a regression over time. A robust regression is applied with the same parameters as described in subsection 4.2.2 for the RUI or RRI merged over all vehicles for each road segment. Figure 4.12 shows two scatter plots of the score of RUI over time for two road segments from the same vehicle. On the x-axis, day 0 corresponds to 8th December 2016. Figure 4.12a indicates, that the score for the road segment increases, which suggests a degradation the road condition.

This indication is underlined by the estimated regression function with a highly significant positive regressor. Figure 4.12b shows a slightly decrease of the score. This method presented here can be applied to a large data set. The trend can be visualized on a map and the road segments can be colored accordingly to the value of the calculates regressor.

Figure 4.12: Examples for trend identification. (\* p < 0.05, \*\* p < 0.01, \*\*\* p < 0.001)

## **4.2.4 Road attribute and surface estimation with labeled data**

Labeled data allow the detection of specific objects or road surfaces with specific characteristics via supervised classification. The road roughness is estimated with supervised regression. The most important part of the processing of labeled data is to find features, which can separate the desired classes in the feature space. Firstly, candidates of features are extracted based on theoretical and physical considerations or the analysis of data from pretests, which is referred to as feature engineering. Another option is to use feature learning techniques, to automatically find representations of the raw data for classification [165, 166]. However, the latter approach is not considered in this work.

The feature candidates are automatically evaluated with multivariate analysis of variances (MANOVA) (subsection 2.5.4) and the best features to separate the classes are selected to reduce the dimension of the feature vector and to avoid overfitting. Overfitted models have a poor generalization with too many parameters compared to the number of observations. Then, the model rather describes random effects than the underlying relationship.

It is difficult to illustrate the high dimensional feature space. Therefore, the dimensions can be further reduced by feature aggregation, for example, with discriminant function analysis (subsection 2.5.5). Therewith, the aggregated feature space with for example two dimensions, can be visualized. The disadvantage of this method is a decrease of accuracy of the classification and the difficult interpretation of the aggregated features. Finally, the regression or classification is performed with a support vector machine (SVM) (subsection 2.5.2 and 2.5.3).

This work differentiates between the classification of road attributes, such as potholes, and the estimation of the road roughness within the macro texture. The object detection is based on the vibration of the vehicle body measured by the inertial sensor while the road roughness estimation is based on the tire vibration captured by the tire cavity sound (Section 3.4). Therefore, the first data set consists of the acceleration and angular rates of the vehicle in three directions and the second of sound pressure in the tire cavity. Both data sets are extended by influencing parameters, such as the vehicle velocity (subsection 4.2.2).

The methods to process labeled data are summarized in Figure 4.13. The raw data is preprocessed and transformed into space domain to extract suitable features.

Figure 4.13: Overview of methods to process labeled data.

Afterwards, the best features are selected with MANOVA and aggregated with discriminant function analysis. To finally estimate the road type or attributes, a SVM classification is applied and a regression to estimate the road roughness.

#### **Inertial sensor data**

For the feature extraction of inertial sensor data the following data streams are considered:

Furthermore, the roll and pitch acceleration as well as the jerk are deviated in time domain of the above data streams. The jerk *a*Ûz for step *t* is calculated with

$$\dot{a}\_{\varepsilon}(t) = \frac{a\_{\varepsilon}(t) - a\_{\varepsilon}(t-1)}{t}.$$

The time series is transformed into space domain, as discussed in subsection 4.2.2. Furthermore, the data is transformed into spatial frequency domain, which contains the short-term, space-localized frequency content of the signal [167]. The spectrogram for a signal *x* with window function w(*s*), spatial frequency ω and space index τ is expressed in the discrete path case as

$$\operatorname{spec}\{\mathfrak{x}(s)\}(\tau,\omega) = \sum\_{k=-\infty}^{\infty} \mathfrak{x}[k]\mathfrak{w}[k-\tau]e^{-j\omega k}.$$

Hereby, features based on specific frequency bands can be investigated. In addition, the spectral centroid from the spectrograms of signal *x* for each time index τ is calculated with

$$\text{centr}\{\boldsymbol{x}(\boldsymbol{s})\}(\boldsymbol{\tau}) = \frac{\sum\_{k=1}^{N} m(k)\omega(k)}{\sum\_{k=1}^{K} m(k)},$$

where *m*(*k*) is the magnitude of frequency bin number *k*, ω(*n*) the representing frequency, and *K* the number of frequency bins [168]. The spectral centroid is a quantification of the power distribution of the signal [169]. Overall, the three data streams are extended by the following data streams, which leads us to 10 data streams in total:

The features are calculated for frames with a specific length in spatial domain and a specific overlap. If a longer frame length is chosen, short peaks for example due to potholes have a weaker impact on the value of features, which incorporate the overall signal, such as standard deviation. These short amplitudes can be captured by shortening the frame length or using features, which calculate extrema.

From the data streams (1) - (4), the following features are calculated

with

$$\begin{aligned} \bar{\boldsymbol{x}} &= \frac{1}{N} \sum\_{i=1}^{N} \boldsymbol{x}[i], \\ \sigma(\boldsymbol{x}) &= \sqrt{\frac{\sum\_{i=1}^{N} \left(\boldsymbol{x}[i] - \bar{\boldsymbol{x}}\right)^{2}}{N-1}}, \\ p(\boldsymbol{x}) &= \max\left(\boldsymbol{x}\right) - \min\left(\boldsymbol{x}\right), \end{aligned}$$

where *x* is a data frame of the corresponding data stream (1) - (4) and *N* the framelength.

The effective value or RMS (*x*¯RMS(spec)) is calculated for the short-time Fourier transformed data streams (5) - (7) for the following spatial frequency bands (1/m)

[0.1, 0.5] [0.5, 15] [15, 20] [0.1, 25] [0.1, 50].

Afterwards, the following features are extracted for each frame of the vehicle trajectory from *x*¯RMS(spec) of each frequency band and from the spectral centroids (8) - (10).

Subsection 4.2.2 discusses the strong influence of velocity on the vehicle vibration. Previous research suggests to perform a linear regression with each feature as the dependent variable and the velocity as the independent variable [94].

The velocity dependency is then reduced by subtracting the estimated linear equation from the corresponding feature. However, the vehicle vibration and the extracted features are not linear dependent on the velocity (subsection 4.2.2).

In this work, the dependent parameters are incorporated in the classification and therefore, mean velocity is calculated for each window as additional feature. To allow non-linear relationships a polynomial kernel function of higher order is applied on the classification (subsection 2.5.2), which is defined as

$$k(\mathbf{x}, \mathbf{y}) = (\mathbf{x}^T \mathbf{y})^d,\tag{4.2}$$

where *x* and y are features and *d* the degree of the polynomial.

#### **Acoustic sensor data**

The described data processing method for the inertial sensor data is analogous applied for the tire cavity sound data. Road surfaces with various characteristics especially with different grain size distributions can be classified based on power features of different frequency bands, as we have already shown in [170, 171].

This work pursues to estimate the road roughness based on the tire vibration represented by the SPL of the tire cavity sound and applies a SVM regression. Subsection 4.2.2 shows that the velocity has also a strong influence on the tire cavity sound pressure and is incorporated as a feature for this regression.

Furthermore, the tire pressure, tire temperature, and vehicle load is included although the influence is weaker compared to the velocity.

Overall, the RRI can be estimated for each window with the following regression function

$$RRI = \beta\_1 \cdot SPL\_{(TCS)} + \beta\_2 \cdot \vec{\nu} + \beta\_3 \cdot \vec{p} + \beta\_4 \cdot \vec{T} + \beta\_5 \cdot \vec{P} + \beta\_0,$$

with the following features:

The features are scaled to the range of 0 to 1 with

$$
\alpha' = \frac{x - \min(\alpha)}{\max(x) - \min(\alpha)},
$$

where *x* is the feature and *x* 0 the scaled feature. Furthermore, a polynomial kernel function of higher order as presented in (4.2) accounts for the non-linear relationship of velocity and SPL(TCS). Further features can be integrated, such as the type of tire, temperature of the environment, if water or snow is on the road, among others.

# **4.3 Exemplarily application of methods on benchmark data set**

A benchmark data set is used to exemplarily show the data processing with the introduced supervised learning methods (Section 4.2) with inertial sensor data of the vehicle body. Moreover, the influence of classification parameters, such as the kernel trick or feature aggregation, and the performance measures are explained with this data set.

The benchmark data set is designed for good illustration and less complexity. Therefore, only three features are extracted to separate three classes, which are *potholes*, *surface with light damages*, and *smooth surface*. The extracted features are standard deviation of the vertical acceleration, roll and pitch rate, which represent the principle vibration of the vehicle.

The data was acquired on different road segments consisting of the mentioned road features with a BMW 116d. To further reduce the complexity of the benchmark data set, the road features were overrun with specific velocities,

Figure 4.14: Distribution of feature and velocity values of benchmark data set.

which are 30, 40, 50, and 60 km/h. The distribution of the values of these features and the velocity for the acquired data is shown in Figure 4.14.

Figure 4.15: Single feature space of benchmark data set.

The histograms indicate, that the number of data points per velocity is approximately uniformly distributed (Figure 4.14a). The values for data points corresponding to potholes are mainly larger over all features compared to other data points (Figure 4.14b, c, and d).

The standard deviations for damaged and smooth surface overlap for some data points. Therefore, we can expect wrong predictions from the classification of these features.

Figure 4.15 illustrates the classes in features space. Potholes are separable from the other classes, whereas damaged and smooth surface overlap.

Table 4.3 shows the confusion matrix of the classifier, a standard linear SVM with penalty term *C* = 1000 and = 1. Wrong predictions occur only between the classes damaged and smooth surface.

Table 4.3: Confusion matrix from single feature classification.

Performance measures of the training process and 5-fold cross-validation, derived from the confusion matrix as explained in subsection 2.5.6 are listed in Table 4.4. Overall the accuracy of the cross-validation is 99.0 %.

Table 4.4: Performance measures of single feature classification.

The high dimensional feature space is reduced to two dimensions with two features aggregated with linear discriminant analysis (LDA) function (subsection 2.5.5). Figure 4.16 shows the two-dimensional feature space. The solid lines represent the hyperplane to separate the classes and the dashed line shows exemplarily the maximum margin of the support vectors and the hyperplane. Figure 4.16a shows the hyperplane for the linear SVM and Figure 4.16b for the nonlinear SVM with polynomial kernel for degree-2 polynomials. For this example, the nonlinear SVM creates no big advantage. However, for a larger data set with various classes and features, it might be necessary to increase the accuracy of the classifier.

Figure 4.16: Aggregated feature space of benchmark data set.

Table 4.5: Confusion matrix from aggregated feature classification.

Table 4.6: Performance measures of aggregated feature classification.

Table 4.5 and 4.6 indicate that the values of the performance measures decrease with the aggregation of the features. Therefore, if computation time of classification is not critical, a feature aggregation should be avoided.

## **4.4 Multiple vehicle fusion**

#### **4.4.1 Overview**

The vibration behavior of various vehicles differ greatly due to the structure, suspension system characteristics of the vehicle, and the position of the measuring system. Each vehicle might have an individual classifier based on different feature spaces. This section addresses two problems, which are published in [172, 173].

Firstly, trajectories or trajectory frames from various vehicles through the identical road segments are found based on distance algorithms. Four distance algorithms are presented in subsection 4.4.2 and the results and computing time of the algorithms are compared. In contrast to other methods, such as *R* 2 , the distance algorithms find similar trajectory frames more precisely. Hereby, trajectories pointing in other directions are not considered as similar. This is essential to ensure accurate training data for new vehicles, which can be acquired automatically by labeling the data based on the label of the road segment from ground truth data of other vehicles.

Secondly, the classification results from different vehicles for one identical road segment can be merged with different methods to achieve a combination with the highest confidence. For this purpose, different combination strategies are presented and investigated in various scenarios, which are outlined in subsection 4.4.3.

### **4.4.2 Identification of road segments with trajectories from different vehicles**

To find identical road segments, which are overrun by various vehicles, the trajectories from those vehicles are compared. Firstly, a range search algorithm finds the trajectories from vehicles, which are close to a reference trajectory. Trajectories *A*T and *B*T of the same length are time series of global position system (GPS) coordinates in the form [174]

$$\begin{aligned} A\_T &= ((a\_1^\chi, a\_1^\circ), (a\_2^\chi, a\_2^\circ), \dots, (a\_n^\chi, a\_n^\circ)), \\ B\_T &= ((b\_1^\chi, b\_1^\circ), (b\_2^\chi, b\_2^\circ), \dots, (b\_n^\chi, b\_n^\circ)), \end{aligned}$$

where *a* x and *b* x represents the longitude coordinate, *a* y and *b* y the latitude coordinate and *n* the number of GPS coordinates representing the trajectory.

To find trajectories, which are close to each other, a range search algorithm is applied to reduce the computing time of the overall procedure. The range search algorithm, which is based on a k-nearest-neighbor (knn) algorithm, is to find all points *P*q within a radius around a point *P*p [175]. A k-dimensional (kd) tree is constructed out of all points with Algorithm 4.2, which assigns each point to a node to have an equal distribution of number of points in each node, d represents the depth of the tree. Figure 4.17 exemplary shows a 2d tree with 100 points and 8 nodes, each consists of 12 to 13 points on average and the application on trajectories, where the points are represented by GPS coordinates.

After the construction of the tree, the points can be identified, that lie within a circle of radius *r* around a new point *P*p. The corresponding algorithms are explained in more detail in our paper [172]. The basic idea is to find all nodes, which share an area within the circle around point *P*p. In the second step, only these nodes are searched for points, that have an Euclidean distance smaller

Figure 4.17: Illustration and application of the range search algorithm.

than *r* to the point *P*p. With this method the time complexity in big *O* notation can be reduced from *O*(*n*) for common search algorithms to *O*(log*n*).

However, as Figure 4.17b indicates, also coordinates from trajectories that run in a different direction as the reference trajectory are recognized as close points. The transferring of labels or fusion of classification outputs lead to wrong results. Therefore, further algorithms must process the trajectories to narrow the candidates of GPS coordinates belonging to the identical road segment and road lane.

Methods, which can follow the range search algorithm and eliminate wrong points are the Euclidean distance, principal component analysis (PCA), hausdorff distance, and dynamic time warping (DTW) distance.

The Euclidean distance is defined as

$$D\_1 = \frac{1}{N} \sum\_{n=1}^{N} ((a\_n^\ge - b\_n^\ge)^2 + (a\_n^\ge - b\_n^\ge)^2)^{\frac{1}{2}},$$

where the length of the compared segments of trajectories *A*T and *B*T needs to be equal. With the Euclidean distance, only segments of trajectories with the same driving direction and therefore on the same driving lane are determined, as shown in Figure 4.18a.

The PCA converts the points into a set of points of linearly uncorrelated principal components *a* c k and *b* c k of trajectory *A*T and *B*T respectively. Afterwards, the Euclidean distance of the coefficients is calculated, whereas a smaller distance indicates a greater similarity of the two trajectories [176]:

$$D\_2 = \left(\sum\_{k=1}^2 (a\_k^c - b\_k^c)^2\right)^{\frac{1}{2}}$$

Figure 4.18b shows the green colored trajectory segments, which were identified by the *D*2 function as close to the red trajectory.

Figure 4.18: Application of range search algorithm with Euclidean distance and PCA algorithm.

After the application of the PCA algorithm, the trajectory segments that turn on the interaction and point into the orthogonal direction of the red trajectory are not detected as close. However, the trajectory segments with an opposite direction are incorrectly detected.

To address this problem Algorithm 4.3 is applied to consider the driving direction of trajectory *A*T and trajectory *B*T and to get same results as for function *D*1.

The Hausdorff distance calculates the spatial distance between two trajectories as [177]

$$D\_3 = \max\{d(A\_T, B\_T), d(B\_T, A\_T)\},$$

where *d*(*A*T,*B*T ) = maxa∈AT minb∈BT ||*a*− *b*||.

The application of the Hausdorff distance shows similar results to the PCA distance, where trajectory segments with an other direction could not be filtered (Figure 4.18b).

The DTW algorithm finds the minimum comprehensive path between two trajectories, which minimizes the cost of the warping. The distance is defined as [178]

$$D\_4 = \min\{\frac{1}{N}(\sum\_{n=1}^N w\_n)^{\frac{1}{2}}\},$$

where wn is the *n*th element of the warping path. The DTW algorithm returns similar results as the Euclidean distance (Figure 4.18b).

#### **Calculation time of distance algorithms**

Since all distance algorithms, partly by applying the direction Algorithm 4.3, produce similar results, the algorithm is selected based on runtime. Table 4.7 shows a comparison of the algorithms with respect to the measured time per 1,000 calculations. The Euclidean distance is outperforming the other algorithms due to its simple computation.

Table 4.7: Average run time of the algorithms for 1,000 trajectories with a length of 50 or 200 m.

#### **Calculation times with and without range search algorithm**

The range search algorithm function is intended to accelerate the algorithm, since all nearby points can also be found using the algorithms discussed in subsection 4.4.2. A test is performed with a data set of 215 km travel distance and 251,177 data points that are up to 70 km apart from each other. The method to find close trajectories is applied with and without the range search algorithm. The results in Table 4.8 show, that the use of the range search algorithm is indispensable. The calculation time with this algorithm is almost 500 times faster than the Euclidean distance algorithm alone.

Table 4.8: Run times with and without range search algorithm (RSA) for a traveled distance of 215 km.

The performance of the Euclidean distance algorithm with and without range search algorithm is also tested with a high density of trajectories in a small area. Therefore, the intersection data, which has been used in Subsection 4.4.2, is multiplied 50 times. Consequently, there are 1,050 trajectories with a total of 48,350 data points in a small area. A test with many vehicles can therefore be simulated. If each car passes an intersection twice a day, the data set corresponds to approximately 18 vessels, which navigate the crossing in 30 days. The results in Table 4.9 show that the Euclidean distance algorithm performs well even at a high density of trajectories. The calculation time is even smaller including range search algorithm in the pre-processing.

#### **Accuracy of the transfer of ground truth data**

The goal is to communicate locations together with the corresponding road attribute, where an other vehicle has recognized road attributes. Thus, the second vehicle has training data to learn a correct classification of the characteristics based on its own measurements. To test the method to transfer ground truth data to train a new vehicle, an intersection with trajectories from the BMW 116d test vehicle is used. A single road infrastructure attribute, namely a *railway crossing*, is approximately at position (8.4465, 49.0365). The corresponding data from the BMW test vehicle are annotated with *railway crossing*. To show the functionality and accuracy of the method to transfer ground truth data, *railway crossing* is overrun multiple times with a new test vehicle, a Smart.

Figure 4.19 shows the result of this test. The green sections are the actual *railway crossings* from the ride with the Smart and the blue sections represent data with the automatically transferred label *railway crossings* from already existing BMW 116d data. The figure shows that the label *railway crossing* is successfully transferred except for one case. The unsuccessful transfer is due to bad GPS signal and consequently the route is too far away from the previously traveled routes. However, this underlines the motivation to only transfer ground truth data, if the conditions, e.g. the GPS signal, are well and if there actually was an anomaly in vibration of the vehicle.

Figure 4.19: The result of the test to transfer ground truth data to new vehicles to train a new classifier.

With the described method in subsection 4.4.2, the label of already existing training data can be transferred to data from a new vehicle with a different suspension system to develop a new individual classifier. Labels are only transferred if a certain amount of labels from different vehicles and classifiers exist. Furthermore, labels of road hazards, e.g. potholes, are only transferred if the signal of the new vehicle actually shows an anomaly. Therefore, the amount of labels, which are wrongly transferred, decreases. Without the mentioned conditions, the transferred label might not be correct in the following situations:

#### **4.4.3 Multiple vehicle combination**

Since functions *D*1 to *D*4 find similar trajectories on the same road lane, the output for a specific road segment form various vehicles can be combined with the following method. The precision presents the fraction of retrieved instances that are relevant and can be seen as the probability πi j of the classifier to predict class *i* as class *j* for *i*, *j* = 1,. . .,*l*.

A precision matrix Π can be derived from the confusion matrix for each vehicle *m* with *k* overruns as shown in Table 4.10.

The precision vector for class *K*j for vehicle *m* is then defined as

$$
\pi\_f^{(k)} = (\pi\_{1j}, \dots, \pi\_{lj})^{(m)} \cdot
$$

The approach, illustrated in Figure 4.20, is to combine the prediction from various vehicles or overruns with prediction probability πj for a specific road

Table 4.10: Precision matrix derived from the confusion matrix.

segment with the median position (long., lat.). The goal is to get a robust and more precise estimation of the corresponding ground truth, in contrast to consider the output of only one vehicle or a simple maximum vote.

Figure 4.20: Approach to combine the predictions of the condition of road segments from various vehicles. The final prediction has the highest confidence by incorporating the confidence π of the predictions of each vehicle.

Vehicles might have different classifiers with different precision vectors. Also, the actual precision vector of a vehicle might be unknown but can be approximately estimated through the testing process of the classifier.

#### **Combination strategies**

Since the prediction of a vehicle usually has an uncertainty, multiple vehicles can be combined to increase the confidence of the prediction compared to only one vehicle, which is regarded as multiple expert problem. The final decision for each road segment can then be made by the combination of the individual predictions [179]. The classifier predicts one out of *l* possible classes and from the precision matrix π, the confidence of the corresponding classification be be derived.

The collected predictions for one road segment, named observation *o* ∈ R l , are distributed according to a multinomial distribution. The observation *o* ∈ R l counts the number of predictions per class.

The following methods provide a final estimation of the condition of one road segment. One possibility is to count how often each attribute is predicted, which is referred to as *majority vote*. The one with most votes is assumed to be the ground truth

$$V\_{\text{majority}} = \arg\max\_{i \in \{1, \dots, l\}} o\_i \dots$$

An extension to the *majority vote* approach is the vote with the diagonal vector from the precision matrix, which is referred to as *vote with precision diagonal matrix*. Here, the counts for the predictions are weighted with the precision vector:

$$
\omega\_{\pi} = (\pi\_{11}, \pi\_{22}, \dots, \pi\_{ll}).
$$

That leads to an adapted observation *o* 0 :

$$o' = o w\_{\pi}{}^T \, .$$

As before, the final decision is class with the most votes:

$$V\_{\text{diagonal}} = \arg\max\_{i \in \{1, \dots, l\}} o'.$$

Each of the *n* vehicles might use an individually optimized classifier. This leads to a precision matrix Π (k) and observation *o* (k) for vehicle *k*. Due to the assumption of independent predictions, the probability that class *i* is the ground truth, is calculated as

$$p\_{O|i} = \frac{\prod\_{k=1}^{n} \prod\_{j=1}^{l} (\pi\_{ij}^{(k)})^{o\_j^{(k)}}}{\sum\_{q=1}^{l} \prod\_{k=1}^{n} \prod\_{j=1}^{l} (\pi\_{qj}^{(k)})^{o\_j^{(k)}}},$$

with *O* = {*o* (k) } for *k* = 1,. . .,*n* [179].

As final estimate, the class with highest probability is selected with

$$V\_{\text{matrix}} = \arg\max\_{i \in \{1, \dots, l\}} p\_{O \mid i \sim l}$$

The latter vote is referred to as *vote with precision matrix*.

#### **Scenarios**

The presented methods to estimate the condition of a road segment are evaluated by simulating different scenarios. The proposed scenarios considers various situations, e.g. that the road condition changes after maintenance or that a classifier of one vehicle is over-fitted and outputs random predictions. The point of interest is how many vehicles or overruns are necessary to reach a very small proportion of misclassifications for the prediction of the road segment and how to react to latter described scenarios.

**Scenario 1: Classification with one global precision matrix** In Scenario 1, one global precision matrix Πb for all vehicles is considered. For simulation, a road with a specific number of segments, each segment having a different attribute, is assumed. The outputs of various vehicles lead to the evaluation of the different fusion strategies. Additionally, the prediction quality of the different classes is analyzed.

**Scenario 2: Classification with unknown vehicle precision matrices** In reality, not all vehicles use the same distribution for prediction and the real precision matrix is unknown, since it might be different to the precision matrix calculated during training or testing. Therefore, in the second scenario each vehicle classifies the road segments by its own classifier while the final decision is made based on the global precision matrix πb. Hereby the vehicle precision matrix π (k) v is an slightly adapted version of the global one. In the simulation, vehicle *k* generates a random number to classify the road segments. For segment having property *i*, it follows the multinomial distribution described by the entries πi j(*k*), *j* = 1,. . .,*l* from the adapted precision matrix π (k) v .

**Scenario 3: Classification with equally distributed precision matrix** Another scenario under consideration is the classification by a vehicle totally independent of the ground truth. An example in real life is a vehicle with a classifier, which is over-fitted, a change of the vehicle vibration behavior or a broken sensor. For this scenario, the effect on the fusion strategies, the possibilities to identify the vehicle making trouble, and to react with adjustments of the combination method are analyzed.

**Scenario 4: Change of the road condition** As last scenario the change of the ground truth from one time to another is simulated. It characterizes situations like a pothole that gets repaired, or a hard winter that destructs the road. It is investigated if rules of forgetting or less weighting the past can improve the outcomes.

The calculations and investigation of multiple vehicle fusion with 10 different vehicles are implemented in Matlab. In the calculation, the road consists of *s* segments and each segment represents one of *l* classes as ground truth with *l* = 5 for all scenarios. In reality, theses classes could correspond to types of damages, such as potholes. Therefore, the number of wrong predictions ranges from 0 to *s* for the created road. The values of the global precision matrix are set as shown in Table 4.11. This precision matrix is calculated from a confusion matrix based on a testing process from a real vehicle.

Each vehicle drives over the road and classifies the segments with its individual precision matrix, which is unknown to the back-end.

Table 4.11: Global precision matrix for simulation.

For **Scenario 1 to 3**, 5 segments are simulated and 10,000 iterations of the drives are performed to reduce stochastic effects. In **Scenario 1** the precision matrix of the vehicles correspond to the global precision matrix. The total number of wrong predictions are counted and the proportion of wrong classifications of all classes are analyzed as well as the wrong predictions of each class and number of vehicles. The wrong predictions in scenario 1 are calculated with the combination method based on *simple vote*, *vote with precision diagonal matrix*, and *vote with precision matrix* (subsection 4.4.3).

In **Scenario 2**, each vehicle has an unknown precision matrix, which simulates e.g. different vibration characteristics. However, the combination of the predictions made by all vehicles is still performed with precision matrix used for scenario 1. The difference of the vehicle and global precision matrix could correspond to differences in the vehicle mass or various environment conditions and therefore a slightly different vibration behavior of the vehicles.

For the simulation of the further scenarios the output fusion is performed only with the method *vote with precision matrix* since the results of the previous scenarios have demonstrated the highest accuracy with this method.

In **Scenario 3**, one vehicle with a problem of the classifier is simulated, which is referred to vehicle *with broken sensor*. It classifies the segments totally random and the question is, how much impact it has on the final prediction. To decrease the negative impact of such a scenario, the following method is applied.

After 5,000 overrun segments and at least 3 vehicles, the simulation compares the output of one vehicle with the output of the other vehicles, which are fused with the system precision matrix as explained in earlier scenarios. Lines 8 to 16 in Algorithm 4.4 show the pseudo-code of this method, which is embedded in the simulation algorithm. Based on these comparisons, the precision matrix of the one vehicle is estimated and checked if the values differ greatly from the back-end precision matrix.

If the difference exceeds a value, which is set to 0.15 in this simulation, the vehicle with the equally distributed precision matrix and wrong predictions is identified. Furthermore, the estimated vehicle precision matrix is used for further calculations to reduce the confidence of the output of this vehicle and therefore reduce the overall number of misclassifications.

For the simulation of this method and for **Scenario 4**, the number of segments is set to *s* = 10,000 and one run of the simulation per vehicle is performed. The properties of the segments, representing one of the five classes, are nearly uniformly distributed.

In the simulation, the ground truth changes after 5 runs for every third road segment. It depicts situations like the fix of road damages or the formation of new ones over time. The scenario should clarify that the back-end needs a forgetfulness of previous classifications to guarantee a fast reaction to the change. As benchmark it is counted how many of the *s* = 10,000 segments are wrongly classified when using the *vote with precision matrix* without forgetting. Additionally, a modified version is implemented, which has the ability to forget. Therefore, the vehicle observations *o* (k) , with *k* = 1,. . .,3, are scaled down. E.g. the classification of vehicle *k* from time step *t* −*i* has only the weight *f* i in time step *t* with *f* ∈ [0,1]. A forgetting factor of *f* = 0.5 is used for the simulation. A higher factor means less forgetting because the observations are weighted stronger.

## **4.5 Conclusion of novel methods for road condition estimation**

This chapters describes the developed and applied methods to process inertial and acoustic sensor data for road condition estimation. Firstly, preprocessing is applied to align the labels, the desired output, with the raw data. Furthermore, the data in time domain is transformed in space domain by velocity and time vector. The data is linearly resampled to a uniform sample rate of 100 m−1 . The rate sensor data is smoothed with a median filter and the vehicle velocity and acceleration data with a Savitzky-Golay filter.

Before presenting methods to process the data, influencing effects on the sensor data or the output variable are experimentally identified. Besides the road condition, the velocity is the main influencing effect of vehicle body vibration. The main parameter influencing the sound pressure in the tire cavity is also velocity besides the road roughness or obstacles. The type of tire, surface temperature, tire temperature, tire pressure, and load have minor impact on the sound pressure.

Incorporating these effects, methods to estimate the road unevenness and roughness with non-labeled data are presented. The output of the road condition estimation is a colored map. Moreover, obstacles are determined, highlighted, and listed in a table with information, such as number of drivers, size, average velocity. Furthermore, a trend identification is developed based on regression of the road condition over time. For data, which are labeled with the desired output variable, methods are presented to predict road attributes based on inertial sensor data. Moreover, a regression of sound pressure level in the tire cavity and influencing factors, such as velocity, is introduced to estimate the road roughness. The classification process, possible parameter variations, and performance measures are exemplarily demonstrated and discussed with a benchmark data set, which incorporates three classes and three features, representing the vehicle vibration.

To combine the prediction from the classification of various vehicles, the following procedure is developed. Firstly, similar trajectories are identified with range search and distance algorithms to combine the classification results of only those trajectories. Secondly, methods to combine these classification results are presented, which are referred to as *majority vote*, *vote with precision diagonal*, and *vote with precision matrix*.

Overall, the presented methods cover the whole data processing chain to estimate the road condition with vehicles as sensor platform, from preprocessing to visualization and multiple vehicle combination.

# **5 Implementation**

*Outline:* This chapter describes the developed measuring devices as well as the software to acquire, transmit, and centrally store the data (Section 5.2). The presented methods to estimate the road condition are implemented in a Matlab toolbox, which is introduced in Section 5.3. Overall, the measuring devices and the software to acquire and analyze the data are developed and implemented under the following aspects, also to allow the application of a large number of measuring devices: automation, low-cost, no manual intervention, only few different computing environments and programming languages to allow a continuous data processing flow. The software for the measuring device to acquire the data is written in Python with bash scripts for routines, the data processing methods are implemented in Matlab.

## **5.1 Overview**

The vertical vehicle dynamics simulation to vary parameters and to investigate their influence on road condition estimation with data mining, described in Section 3.5 is implemented in Matlab. The corresponding code and description to perform the analysis can be found in [147].

The methods to combine the classifications derived by various vehicles of an identical road segment are also implemented in Matlab. The developed toolbox is published in [180].

The following sections focus on the implementation of data acquisition and a Matlab toolbox to process the acquired sensor data. The complexity of the selection and combination of hardware, especially the sensors and data logger in the vehicle and the acoustic sensor in the tire cavity, as well as the requirements of the measuring device are outlined in Section 3.4. This chapter mainly presents the final result and application of the implementations and describes the most important steps of the set-up and operation of the measuring devices. The detailed instructions for setting up and operating the measuring device to acquire data can be found in [181]. The instruction includes the self-developed software and installation file, files to 3D-print the cases, files for the self-developed printed circuit board (PCB) and software for the microcontroller. The developed toolbox in Matlab, named Vehicle Learner Toolbox, consisting of the implementation of the data processing methods, presented in Section 4.1 and 4.2 is published in [182].

Overall, the measuring devices and the toolbox are developed to enable an automatic data acquisition as well as processing. For the set-up and installation of the measuring devices, only few manual interventions are needed. The measurements run automatically and the toolbox only needs the desired parameters as input.

# **5.2 Data acquisition**

The developed methods and described sensors for acquiring sensor data automatically (Section 3.4.3) are the basis for the measuring devices developed in this work. The measurement system for acquiring inertial sensor data, vehicle position, and velocity and to transmit the data to a central data base is shown in Figure 5.1. To annotate the raw data with the label of the ground truth, two buttons are installed. For every measurement drive, data describing a certain type of surface (e.g. if asphalt 1, otherwise 0) and an attribute (e.g. if pothole 1, otherwise 0) are recorded. After the measurement drives, binary coding of the data of the respective files is transformed into the coding given in Table 5.1. The unix time *t*, *ID* for the sensor, speed vV , position and time stamp of the GPS *lat*, *lon*, and *t*GPS, accelerations *a* and rotation rates ω along all three axes, and the two labels for the attribute *e* and surface type *m* are recorded and stored in a csv-table on the measurement system (Figure 5.1).

The sound pressure in the tire cavity is acquired with a self-developed telemetry system incorporating the novel methods described in Section 3.4.3, which is attached to the wheel (Figure 5.2). The audio data are transmitted via Bluetooth to the measurement system in the vehicle, along with the state of charge (SOC) of the battery, and saved as a wav-file. The tire pressure monitoring system

Figure 5.1: Measurement unit, consisting of Raspberry Pi, inertial sensor, Adafruit ultimate GPS Hat, UPS Hat, and two buttons for labeling the data. In service, the inertial sensor is fixed under the Raspberry Pi on the bottom of the case. The case is cut open for visualization purposes.

Table 5.1: Label and name of the classes for road surface and road attribute.

(TPMS) data is transmitted on the radio frequency 433 MHz and fused with the inertial sensor data among others in the csv-table.

To enable an automatic installation of the self-developed software, the SD card for the Raspberry Pi needs to be written with a self-developed image analogous to a normal Raspian OS image. The basis as Raspian OS of the self-developed image is Jessies. Alternatively, a new self-developed image can be created with the script *kernel\_img\_push-to-SDCard.sh* on Linux. To download and install the software on the Raspberry Pi, it needs to be connected to the internet

(a) telemetry system without wheel (b) telemetry system attached to the wheel Figure 5.2: Telemetry system, consisting of an acoustic sensor and TPMS in the tire cavity, a self-developed y-valve for cable entry and air, and the wireless transmitter in a case (without cover for visualization purposes).

via LAN cable to automatically download the self-developed software from a Git-repository on a web-based hosting service. For the installation process, there is no manual interaction needed. If the Raspberry Pi is to be coupled with the telemetry system and the software is to be installed on the Bluetooth module of the board, the Bluetooth antenna needs to be plugged in and the Raspberry Pi needs to be connected to the telemetry via UART. The credentials for the access point and the remote data storage to transmit the data can be adjusted in the repository in the file *rpi\_config.cfg* in the sections 'WLAN' and 'Samba' or directly on the Raspberry Pi.

The micro-controller of the telemetry system, which is responsible for activating the PCB by rotating vehicle wheels, needs to be flashed with the self-developed software. Furthermore, the firmware of the Bluetooth module on the board needs to be updated. The software of the Bluetooth module is automatically installed during the installation of the Raspbeery Pi.

The software for the measurement system in the vehicle is mainly developed in Python, routines for automation are implemented as bash scripts. The *sensorrecorder.py* is the main measurement collection service of the software. The collection service runs the processing task of every sensor in a separate thread, as illustrated in Figure 5.3. The threads are responsible to acquire data from global position system (GPS), inertial sensor referred to as IMU, TPMS, and on-board diagnostics (OBD) and are divided into initializing and running. On program start, the threads initialize the required libraries for the sensors, which are initialized and set up. For GPS the gps3 and for IMU the rtimu library are imported. Furthermore, the settings, such as orientation, and the poll interval are set for IMU. For TPMS and OBD, the serial connection to the receivers are started. Moreover, the pin configuration for the buttons is initialized. Afterwards, the sensor threads get data. In addition, the GPS thread generates data dictionary and the IMU thread log the sample rate. OBD and TPMS thread decode the received data. In this work, only the data for vehicle velocity and wheel revolutions per minute are acquired. However, it is possible to sample other sensor data from OBD. The button thread checks the states of the pins. While running, the service collects the data from all sensors, combines and tags them with a timestamp and queues the data for writing them to an output file. The write-out task then will write the measurement data from all tasks into a file, creates a new one after five minutes and compresses the old one. If a sensor or receiver is not connected to the measurement system, the data from the others are still acquired.

The *audiorecorder.py* to acquire the audio data from the telemetry system at wheel runs separately from the main collection service. It starts automatically triggered by userspace /dev (udev), a device manager for the Linux kernel, as soon as a bluetooth connection between the Raspberry Pi and the telemetry system is established. The telemetry system is configured as master and actively tries to connect to the Raspberry Pi. If the connection is successful, the recorder starts the audio transmission via a remote console on the bluetooth module of the telemetry system and the audio data is acquired and saved as wav-file.

New files are created every 5 minutes to avoid large data files for the upload to a remote file storage. The file names contain an ascending ID and the timestamp to allow merging the audio data with data from the other sensors.

The *wifiupload.py* script is triggered by the uninterruptible power supply (UPS) module, which manages the power outages and shutdown of the measurement system. The script assumes, that the vehicle's engine is powered off and a

Figure 5.3: Structure of the main da ta collection service *sensorrecorder.py* with processing tasks for each sensor as separate, independent threads. The data is combined and tagged with a timestamp in a csv-table.

properWIFI link is available. If these conditions are valid, the script will update the Git repository and transmit the files to a remote file storage. Afterward, it will switch off the measurement system or restart the recording software, if power is available again.

During the data recording, one can connect to the measurement system via LAN to monitor the recording and, if necessary, to save the data files. On a Windows PC, the programs 'Putty' and 'WinSCP' are suitable for this purpose. The IP address of the measurement system is set to <IP_ADDRESS> and the data is located in the folder /home/pi/logging\_data.

Overall, the measurement device runs mainly autonomously. The measurements of the sensors in the vehicle body start and stop with the ignition of the vehicle and the measurements of the sensors at the wheel with its rotation. After the measurement is finished and the data logger is connected to a known access point, all data is automatically transferred to server. From there, the data can be processed with the toolbox, which is presented in the following section. The only manual intervention needed is to charge the battery of the telemetry system at the wheel via mini USB with output voltage of 5V after more than 150 h of measurement time..

# **5.3 Data processing**

## **5.3.1 Overview and import**

To facilitate operation by non-experts, the methods presented in this work to preprocess and process the data (Section 4.1 and 4.2) are implemented in a graphical user interface called *Vehicle Learner Toolbox*. It is based on Matlab and implements several machine learning operations of the freely available toolbox SciXMiner [183] (formerly, Gait-CAD [184]). The *Vehicle Learner Toolbox* provides the possibility to

A project folder can be selected and sensor data can be imported in the corresponding menu element *Data* (Figure 5.4). There is the option to assign the sensor data to specific vehicles, since they vary in suspensions, damping, and other parameters, which has an impact on the vibration behavior. Therefore, *supervised learning* data processing, described in subsection 5.3.3, can be performed for data dependent on the vehicle. The data from various vehicles can be merged for the *unsupervised learning* data processing. The import allows .csv and .xlsx file format with the following column headers:

#### 5 Implementation

Figure 5.4: The import data frame of the *Vehicle Learner Toolbox*. Different vehicles can be chosen and the data type can be set.

Furthermore, csv files along with wav files can be imported, if tire cavity sound pressure was acquired. The sensor data from the csv and wav files are aligned and merged based on the unix timestamp.

The data is imported and preprocessed with the methods presented in Section 4.1. Moreover, the imported data set can be categorized as training, testing, or unlabeled data.

Furthermore, the parameters for the length of road segments and overlapping of these segments as well as the resampling frequency can be determined. The standard window profiles are *surface type* with a window length of 50 m and *attribute* with a window length of 5 m.

#### **5.3.2 Non-labeled data processing**

The sections of the toolbox for the non-labeled data processing for inertial sensor and acoustic sensor data automatically apply the methods presented in subsection 4.2.3. Figure 5.5 shows the frame of the toolbox for the processing of inertial sensor data. The frame for the process of acoustic data is similar except for the possibility to filter the data for the range of the tire pressure and tire temperature. The data of various vehicles can be combined and the time interval as well as the area of interest can be chosen. Furthermore, the lower threshold and upper threshold as well as the resolution for the algorithms needs to be selected. Furthermore, the maximum, mean or the standard deviation of the score for each road segment can be calculated. The output of this data processing can be a map based on open street map (OSM) format with the road state scores, a csv file with the most damaged road segments and an OSM colored with the trend of the road state score.

#### 5 Implementation

Figure 5.5: The non-labeled data processing frame of the toolbox.

### **5.3.3 Labeled data processing**

After the import and preprocess of the data, new data streams are calculated and features are automatically extracted, as proposed in subsection 4.2.4. Any code to calculate new data series or features can be easily added in the corresponding Matlab function. The in Section 4.2 proposed data mining methods are implemented in the toolbox. The user can select and use the methods under the menu element *Supervised Learning* (Figure 5.6). Furthermore, the toolbox contains additional data mining methods from the SciXMiner toolbox, introduced in subsection 5.3.1, which the user can select to process data. However, this work concentrates on the data mining methods described in Section 4.2. These methods showed the best performance compared to other methods from SciXMiner, as investigated in [203].

Figure 5.6: The train classifier frame of the *Vehicle Learner Toolbox*. Various classifiers can be trained with a wide range of options.

From the training or testing data set, labels are encoded according to Table 5.1 and data annotated with specific labels can be excluded from the classification. Another option is to thin out classes with significant more data points than other classes to allow an approximately uniform distribution of data points among the classes to prevent over-fitting of specific classes. Furthermore, systematic errors during labeling the data can be removed, e.g. if the button to annotate the data live was activated too early or too late the annotation can be shifted or data points with the wrong annotation can be excluded. After the generation of the data set to be processed, the settings for the classifier can be determined under the tab *Train Classifier*, shown inFigure 5.6.

A new classifier model can be created with a training data set or an existing model can be selected to test on new data. The next part contains the settings of feature selection (e.g. multivariate analysis of variances (MANOVA)) and aggregation (e.g. discriminant analysis), as proposed in subsection 4.2.4. Reducing the amount of features highly influences the classification result by reducing the chances for over-fitting. It is possible to cross-validate the training process by setting the k-fold-cross-validation value to higher than 1. The last section offers a variety of settings for the classifier, e.g. for a support vector machine (SVM), including the kernel function and penalty term. Afterwards, the classifier can be trained and data can be plotted on OSMs. Furthermore, the confusion matrix and the total loss is shown in the Matlab console.

Figure 5.7: The train regression frame of the *Vehicle Learner Toolbox*. Various regressions can be trained with a wide range of options.

Similar to training the classifier with features from inertial sensor data, a new regression model can be trained with various variations under the menu *Supervised Learning* and *Train Regression* (Figure 5.7). For example, the vehicle can be chosen and the time interval and area to be investigated can be set. Furthermore, the independent variables of the regression as well as the dependent variable have to be determined. Lastly, the size of the dataset can be automatically reduced to avoid long calculation time and the number of folders for the cross-validation can be entered. The output is a plot with the regression function and the measured observations as well as performance measures. Analogous to the classification, new data can be tested as well as plotted on OSMs.

For testing new data, a data set with modifications in time range and area to be analyzed can be generated as described for training, and a trained classifier needs to be selected. If the test data set is labeled, the output of the prediction is again a confusion matrix and the classification error. Moreover, the results can be visualized and plotted on OSMs. The trajectories will be cut into segments of different color referring to the corresponding classes, which are predicted.

# **5.4 Conclusion of implementation**

This chapter demonstrates the developed measuring device to acquire data from inertial sensors, acoustic sensors in the tire cavity, GPS, OBD, and TPMS. The instructions for setting up and operating the measuring devices, including the software, technical drawings of the cases, the developed PCB, and a list of the required hardware, is available in [181]. The measuring device is designed for an automated installation and autonomous operation and therefore for mass production and comprehensive data acquisition with various vehicles.

The methods to process the acquired data are implemented in a user friendly Matlab toolbox, which is available in [182]. With this toolbox, the data processing is automated, from importing and storing data to combination and visualization of the road condition estimates. The user only has to set the parameters for the data processing methods.

# **6 Results and discussion**

*Outline:* This chapter presents the results of the validation of the developed measurement device (Section 6.2) and the estimation of the road condition with non-labeled and labeled data from inertial and acoustic sensor (Section 6.3). The visualization of the road condition estimation is implemented in the developed Matlab toolbox. Matlab provides maps based on open street map (OSM), on which the estimations can be visualized. These maps are only available colored and the investigation of the results works on monitors. However, for this work, the estimations are mapped with the open-source application QGIS, which provides Stamen's toner maps as basemap layer that are black and white and more suitable to read. The results of the investigation of the influence of different parameters and vehicles on the classification accuracy in simulation are presented in Section 6.4. Furthermore, the methods to combine the prediction of various vehicles are evaluated in Section 6.5. Each presentation of results is followed by a discussion. In particular, the developed measuring device is discussed and contrasted with vehicle on-board sensors. Moreover, the results of the road condition estimation (subsection 6.3.3) and the implementation of the presented methods on an electronic control unit (ECU) of the vehicle to run online (subsection 6.3.4) are discussed. Furthermore, the extended simulation approach is concluded and the application and results are discussed (subsection 6.4.3). Lastly, each section in this chapter includes open problems and suggestions for succeeding works with the topic of road condition estimation using vehicles as sensor platforms.

### **6.1 Overview**

For the validation of the development measurement device, which is presented in Section 3.4, data were acquired with a BMW 116d driving specific maneuvers. In addition to data from the developed measurement device, reference data from established measurement devices, which are too expensive for fleet use, were simultaneously acquired. Based on the comparisons of the data, the developed measurement device is evaluated in Section 6.2.

Data for road condition estimation were acquired over a period of three years with different vehicles, different drives and under different conditions, such as temperature and weather. The measurement device in the vehicle body with the GPS and inertial sensor was always placed between the driver and co-driver seat, glued on the armrest behind the hand brake. The acoustic sensor was attached to a free-rolling wheel.

For the estimation of road unevenness and roughness with the methods presented in subsection 4.2.3, the time consuming process of data labeling is not necessary. Therefore, data were constantly acquired over the period of three years, mainly with a Ford Galaxy. In addition, a BMW 116d, VW Golf, Smart and Mercedes-Benz S-Class were used to acquire data to test the methods for different vehicles. Overall, over 300 h of data were gathered for the estimation of road unevenness and roughness with non-labeled data. Subsection 6.3.1 shows results of the application of the methods from subsection 4.2.3 on the mentioned data set. Furthermore, the estimation accuracy is compared for two different vehicles on the same track and the influence of the combination of multiple estimation results is identified.

To apply the supervised learning methods, explained in subsection 4.2.4, the data need to be manually annotated with the desired output variable, which is time consuming. Therefore, less data were acquired compared to the data set for non-labeled data processing. To ensure a generalization of the presented methods, measurement data were recorded on randomly selected roads in the Karlsruhe area. The velocity, the road conditions, and the environmental conditions of the data set vary in a wide range. In total, reference data are recorded over a distance of 250 km over a period of 6 months. The data were recorded on 14 days by three different drivers. The acquisition of reference data is a lengthy process, as individual attributes, such as potholes, need to be overrun several times with varying approach angles, vehicle lanes and velocities. The results of applying the methods from subsection 4.2.3 on the described labeled data set are shown in subsection 6.3.2.

The variation of vehicle parameters and the influence of this variation is investigated in the extended simulation approach, presented in Section 3.5. Vehicles used for real measurement data acquisition, are replicated in simulation to generate a larger data set. The results and the robustness of the application of the methods from subsection 4.2.3 on the data set generated with simulation and varied paramaters are presented in Section 6.4.

The basis for the multiple vehicle combination, such as the precision matrix, is derived from real measurement data. However, the calculations to combine the classification results are performed offline and results and evaluation of the combination strategies are shown in Section 6.5.

## **6.2 Verification of the developed measurement device**

#### **Inertial sensor**

The performance of the microelectromechanical systems (MEMS) inertial sensor is compared to a dedicated inertial measuring unit (IMU) (OXTS-RT2502 from Oxford Technologies Soluations Ltd.), which is accurate but too expensive for fleet use. Figure 6.1 shows the comparison of the signal of the MEMS inertial sensor and from OXTS-RT2502, which is referred to as reference, for various maneuvers, including slalom, acceleration and deceleration, and a cleat test. The filtered signals are smoothed with a Savitzky-Golay filter with parameters *k* = 7 and *f* = 21 for the vertical acceleration and with *k* = 3 and *f* = 51 for others. Although the vertical acceleration show higher noise (0.1921 m/s2 ) compared to the reference sensor (0.1428 m/s2 ), Figure 6.1 indicates that the MEMS sensor is suitable to measure the vehicle motion with a similar accuracy like the dedicated data logger. Table 6.1 shows performance measures for signals of the maneuvers and proves this suggestion.

#### **Acoustic sensor**

Figure 6.2a exemplarily shows the power spectrum from the CMA-4544pf-4w std. from the THD test, which is the basis to calculate the THD value of the sensor. Figure 6.2b shows the SPL for the microphone INMP411 from the excitation of the tire with an impulse hammer. The maximum SPL in the tire cavity was 165 dB. However, the microphone INMP411 shows a plateau with (a) unfiltered signals from longitudinal acceleration of a start and braking maneuver

(b) filtered signals of start and braking maneuver

(c) filteres signals from vertical acceleration of a cleat test maneuver

(d) filtered signals from yaw rate of a slalom maneuver

Figure 6.1: Comparison of signals from MEMS and reference inertial sensor.

Table 6.1: Summary of statistics for reference and MEMS inertial sensor signal.

a SPL of 132 dB, which corresponds to the acoustic overload point of 131 dB specified by the manufacturer.

Figure 6.2: Results from THD and SPL test.

Figure 6.3 shows the frequency response of all microphones during vehicle operation. The figure indicates that the frequency response of the modified conventional electret microphone (CMA-4544pf-w mod.) approximates the response from the reference sensor (PCB 103B02) best.

Figure 6.3: SPL with a 1/12 octave band smoothing of all microphones during vehicle operation with constant speed of 60 km/h.

Table 6.2 lists quantitative criteria of all sensors and tests. The table underlines, that the modified electret microphone measures almost the same sound pressure level (SPL) as the reference microphone. Furthermore, it has a similar total harmonic distortion (THD) value and power spectral density (PSD) values for the test runs with 30 and 60 km/h compared to the reference microphone. Lastly, *R* 2 between the reference and each microphone is highest for the modified electret microphone.

The results of the experiments suggest, that the microphone CMA4544PF-w a modified configuration is a good alternative to the expensive PCB 103B02 to perform acoustic measurements of the tire cavity sound.

Table 6.2: Summary of acoustic and statistics of the experimental examinations.

#### **GPS**

The accuracy of the GPS was determined by stopping a test vehicle five times at the same position. The orthodromic distance, the shortest distance between two points in the surface of a sphere, from the five acquired positions is illustrated in Figure 6.4a and the standard deviation of the distances is 1.12 m. Figure 6.4b shows the comparison of the vehicle velocity obtained from onboard diagnostics (OBD) and global position system (GPS). The signals have a similar graph, the *R* 2 is 1.00 and NRMSE is 1.90. Overall, the GPS sensor has a good accuracy in velocity as well as in position.

#### **Discussion**

A novel measurement device for acquiring GPS, inertial sensor, and acoustic sensor data is presented in this work. It works autonomously and any vehicle can be equipped with this device.

However, inertial sensors are even partly integrated in the standard equipment of new vehicles for fusion with GPS for a more precise position determination. The data from the measuring device used in this work are compared with

Figure 6.4: Orthodromic distance from five GPS signals at the same position and velocity from GPS and OBD .

inertial sensor data from an Audi A7. The sliding standard deviation of the rolling rate, pitch rate and vertical acceleration of both sensors proved to be almost identical with coefficient of determination of 0.9577 on average and a normalized root mean square error (NRMSE) of 0.0416 on average, which we have showed in [185]. Therefore, the developed methods for the inertial sensor can be applied on modern vehicles without the employment of an external measuring device. The advantage of the inertial sensor is that it is inexpensive and already on-board in many new vehicles. In comparison to a camera, inertial sensors can only be used to evaluate the directly traversed lane area. However, data of the inertial sensor already contain information about minor unevenness which causes the vehicle to vibrate, which might be mistaken by the camera. Moreover, road condition estimation with cameras is problematic under varying conditions, such as darkness and moist on the road surface. All in all, a system consisting of inertial sensor is a cost-effective and computationally efficient alternative to cameras to monitor the condition of the road infrastructure. The fusion of the two systems represents a possibility to optimize mobile condition estimation.

# **6.3 Road condition estimation**

## **6.3.1 Road unevenness estimation and trend identification with non-labeled data**

Figure 6.5 shows the estimation of the road unevenness index (RUI) based on the data processing methods presented in subsection 4.2.3. The frame length for the road unevenness index (RUI) estimation is 5 m. The road condition estimation for the highway *B10* in Niefern, Germany, in the figure are based on inertial sensor data acquired with a VW Golf. The estimation shows areas with severe unevenness, colored red. These locations are automatically detected and tagged with a circle with the double thresholding and connected-component labeling algorithms (subsection 4.2.3). The tags are shown in Figure 6.5 and labeled with number 1 to 4. The estimations are compared to the officially determined condition assessment and evaluation 2015 (ZEB).

Figure 6.5: Continous road uneveness estimation in Niefern, Germany.

Label 1 and 2 are bridge gaps, which have a strong excitation of vibration of the vehicle, and Label 3 and 4 are severe road surface defects. Overall, the coefficient of determination for the estimated RUI compared to the ground truth for the whole test drive around Niefern is 54 % and for the unweighted aggregation of various drives 63 %, which we have showed in [185]. The coefficient of determination for a different vehicle, a BMW 116d, for the same test track is 50 % [185]. The coefficient of determinations show, that the estimation can explain a large portion of the ground truth and that the aggregation of various drives improves the estimation result. Furthermore, the estimation seems to be stable for different types of vehicles. The *R* 2 of RUI estimation of the autobahn (A6) close to Niefern with the VW Golf yields 45 %, which corresponds to the estimation accuracies for highway B10. The estimation accuracy based on a camera is higher for B10 with 76 % and similar for A6 with 47 %[185].

Table 6.3: Automatically detected hotspots with severe unevenness.

Table 6.3 exemplarily shows the output of the toolbox with the automatically detected severe unevenness hotspots. In addition to maximum and on average estimated RUI, max(RUI) and µ(RUI), of the corresponding hotspot, the coordinates of the midpoint of the hotspot, long. and lat., are displayed. Furthermore, the radius of the hotspots*r* and the average and standard deviation of the velocity of vehicles driven over the hotspots, µ(velocity) and σ(velocity), and the amount of drives are automatically calculated.

Figure 6.6 exemplarily shows the estimation of the RUI for a greater road network based on data acquired with Ford Galaxy between 2016 and 2017. The Labels 1 to 3 show road segments with severe unevenness, which are confirmed by the driver.

The former examples show the advantage of the developed methods and toolbox. The data are automatically acquired, processed, and visualized. The maps with the colored estimation of the road condition and the table with hotspots give a comprehensive overview of the road condition. However, the output is only an unevenness index, without knowledge of the actual attribute on the road. The results of the prediction of such attributes are shown in subsection 6.3.2.

Figure 6.6: Continous road uneveness estimation in Karlsruhe, Germany.

Figure 6.7 shows the results of the trend identification method for RUI for Karlsruhe, Germany, and surrounding area. The trend identification method, which is presented in Subsection 11, estimates if the condition of a road segment decreased or increased over time. The figure indicates, that the quality of the road infrastructure remained constant except for small road segments in the city and larger areas marked with Label 1 and 2. With such an output, road infrastructure engineers can identify road segments, where the condition greatly decreases in short time intervals to identify the reason and to take countermeasures. The reasons for such decreases in road condition might be a high portion of heavy goods vehicles or mistakes during the construction of the road.

Figure 6.7: Trend identification in and around Karlsruhe, Germany.

## **6.3.2 Road attributes and surface estimation with labeled data**

This work distinguishes between the classification of attributes and road surfaces. The classification of road surfaces aims to give a rough overview of the condition of the road network. Hereby, the length of each road segment to be classified is longer, 50 to 100 m, compared to the segment length for attribute classification, where it is 5 to 10 m long.

Road surface classification separates the classes *smooth asphalt and concrete*, *damaged asphalt*, *damaged concrete*, *cobblestones/off-road*. Road attribute classification incorporates the classes *potholes*, *manhole covers*, *railways crossings*, *speed bumps*, *light damages*, *good condition*.

Figure 6.8 illustrates the classification results of both approaches in aggregated features space. Table 6.4 shows the performance measures (subsection 2.5.6) of the classification.

Figure 6.8: Classification results with two aggregated features and borders of the classifier in black.

The results, including Figure 6.8, and visualized examples for both examples are explained in more detail in Subsection 6.3.2, as well as the regression and visualization of the roughness estimation of road surfaces based on sound data of tire cavity.

#### **Attribute classification**

The cross-validated accuracy of classifying attributes yields 81 % on average without feature aggregation. The most important features to separate the classes, determined with multivariate analysis of variances (MANOVA) are:

Table 6.4: Results of single features classification.

Previous analysis have shown that the vehicle velocity has a high influence on the sensor data and is therefore selected as an important feature for the classification. The aggregated feature space, aggregated with linear discriminant analysis, and the lines of the function to classify the attributes are shown in Figure 6.8b.

The illustration of the classification shows, that road segments in *good condition*, with *light damages*, *speed bumps*, and *potholes* can be separated well. This indication is proofed by the quantitative results, listed in Table 6.4. The precision and recall for *good condition*, *light damages*, *speed bumps*, and *potholes* is above 70 %, whereas the performance measures for *manhole cover* and *railway crossing* is below 62 % on average.

By comparing each class with each other, it emerges that the peak-to-peak values of pitch and roll acceleration are mainly responsible to separate attributes, which occur on

In addition, the average RMS of the vertical acceleration is important to separate light damages and road segments in good condition. Furthermore, *potholes* and *manhole covers* are dividable by the maximum RMS of the roll acceleration in the frequency range 15 to 25 m−1 . However, latter attributes are often misclassified as segments in *good condition* or *light damages*. *Speed bumps* and *railways crossings* are separable by the value of the peek-to-peek of the pitch rate, whereas *railways crossings* are also often misclassified as light damages.

Figure 6.9: Attribute classification results of two high speed roads in the south west of Karlsruhe, Germany.

To test the classifier, a data set of more than 200 km of real data is classified and plotted on maps based on open street map (OSM). The results are promising and represent the actual street condition in many occasions. A few examples of classified areas are shown below.

The first example shows the attribute classification results on two different high speed roads (Figure 6.9). The upper one with Label 1 is a freshly renovated asphalt highway with close to no damages and the lower one with Label 2 is a poorly patched asphalt road with a lot of medium and severe damages. The classification successfully predicted the upper roadway as *good street*. Most parts of the lower street were predicted as *light damage* and some points even as *potholes*. The results conform to the actual road condition.

Figure 6.10: Attribute classification results for road segments in the city of Karlsruhe, Germany.

The second example presents data acquired in an urban area in Karlsruhe, the predictions are shown in Figure 6.10. The roads in this area are poorly preserved and there is a *speed bump* at a pedestrian crossing (Label 1). The classification model correctly predicts the *speed bump* (Label 1) for all overruns and a *pothole* (Label 2).

The third interesting sector is shown in Figure 6.11. *Potholes* (Label 2 and 3), which were at the edge of the driving line, were overrun multiple times and the classifier predicts the severe damage accordingly. Sometimes the output at the road segments is not *pothole* but *light damages* or even *good road condition*. The reason might be that the pothole was avoided by the driver.

Figure 6.11: Attribute classification results of road segments outside of Karlsruhe, Germany, for the events *potholes* and *railways crossing*.

The *railway crossing* (Label 1) is more elevated than other crossings and misclassified as *speed bump* in few cases.

#### **Road surface classification**

When classifying road surfaces, cross-validation yields 96.1 % accuracy on average without aggregation of features. The three best individual features for the classification of road surface according to MANOVA are:

The aggregated feature space is shown in Figure 6.8a. The figure indicates, that the biggest portion of misclassifications results from asphalt classified as *damaged asphalt* or *damaged concrete* and vice versa. The illustrated results are underlined by Table 6.4, where the precision and recall for *cobblestone* is above 99.0 %, whereas the performance measures for *asphalt*, *damaged asphalt*, and *damaged concrete* are between 92.0 and 97.6 %.

The values of RMS of the roll acceleration and vertical acceleration separate the classes *smooth surfaces*, *damaged asphalt* and *cobblestone*. The values are highest for *cobblestone* and low for *smooth surface*. Standard deviation of pitch rate separates the classes *damaged concrete* from all other classes. The reason is probably poor and aged concrete joints.

The surface classifier was applied on the same data set described for attribute classification. The classifier was able to reflect the road surface precisely. The following figures display the performance on different surfaces. Analogous to the attribute classification shown in Figure 6.9, the surface classifier could distinguish between both pavements and correctly classified them as *smooth surface* and *damaged asphalt*, respectively.

Figure 6.12: Road surface classification of road segments in urban area.

The classification results from data acquired in the urban area of Karlsruhe (Figure 6.12) show two correctly predicted areas of cobblestone (Label 1 and 2). The remaining road segments are correctly classified as segments with *light damages* or in *good condition*. Especially the latter class was correctly predicted for a road segment, which was recently renewed (Label 3). One misclassification of *cobblestone* can be found close to Label 1. However, this road segment is highly damaged with various potholes, which have a high impact on the vehicle vibration similar to cobblestone.

Figure 6.13 shows highway segments with aged concrete and distinctive concrete joints (Label 1 and 2), which have to be maintained shortly. Except for one short segment, which was classified as *smooth surface*, the road state was correctly predicted.

Figure 6.13: Road surface classification of Karlsruhe, Germany, and highways with different surface types.

#### **Road roughness estimation**

Figure 6.14: Estimated regression function for road roughness index (RRI).

Figure 6.14 shows the observations and estimated regression function for a portion of the acquired data. The regression error, expressed as normalized mean square error, of a five fold cross-validation of all acquired data is 4.54h on average and the standard deviation is 0.15h.The figure and the calculated error indicate that the regression function is able to reproduce the actual observations very well and it can also be used for the estimation of road surface roughness for future measurements.

For example, Figure 6.15 and 6.16 show the estimated roughness based on acoustic data acquired with BMW 116d with varied velocities. Figure 6.15 shows the roughness on a highway from Karlsruhe to Niefern, Germany. Changes of road roughness (Label 1 and 2) are visible along the highway, which correspond to the actual roughness condition.

Figure 6.16 indicates, that the roughness at the crossover in Niefern (Label 1), at which traffic is very heavy, is coarser compared to other road segments in Niefern, Germany. Furthermore, the highway leading to Niefern (Label 2) is new, which is represented by the results of this analysis.

Figure 6.15: Road roughness estimation on highway from Karlsruhe to Niefern, Germany.

Figure 6.16: Road roughness estimation in Niefern, Germany.

### **6.3.3 Discussion of road condition estimation**

The results (Section 6.3) of the non-labeled data processing of the vehicle body vibration and the tire vibration give a good estimation of the degree of the road unevenness and road roughness. The achieved coefficient of determination for two different vehicles is even comparable to results from recent studies, but those involve a training process, e.g. [90]. The output of the data processing, e.g. the maps or tables with hotspots, provide a comprehensive overview of the condition of the road infrastructure. Road inspectors can identify road segments in poor conditions and can structure and prioritize the maintenance of the infrastructure.

In contrast to supervised learning methods, a costly labeling process for this approach is not needed. However, the method can not differentiate, if the vibration of the vehicle or the tire originate from road defects or from structural obstacles. But if the vibration of the vehicle is high and the RUI accordingly close to 1 due to a striking structural obstacle, a maintenance might be necessary even it is not a defect. Examples are railway crossings or bridge expansion joints, which are corrupted.

Moreover, the thresholds for the algorithm to identify road segments with severe excitation on the vehicle are adjustable based on the purpose of the road inspector. The estimator of the RUI is dependent on the vehicle, since different vehicles have various suspension systems and vibration characteristics. However, the estimates can be merged across different vehicles and trajectories.

An extension or specification dependent on the vehicle of the feature set might improve the estimator. For example, the features based on the pitch rate can be considered. Moreover, the lower and higher frequency thresholds for the calculation of the RMS with can be specified for vehicles to decrease the portion of frequencies not relevant of road unevenness. However, this requires an extensive study of the vehicle vibration characteristic due to obstacles and a defined test track with various defects.

The results of the data processing for the inertial sensor data show that both, road materials and attributes can be classified. The features automatically selected by MANOVA are in agreement with the theory of vehicle excitation. Surface type classification performs well according to the results of the crossvalidation and the test data.

There are various misclassifications among the prediction of attributes, especially for structural obstacles, such as *manhole covers* and *railway crossing*. However, these attributes might be marked on a map and excluded from classification and investigation, as the main objective is to detect road damage.

One reason for misclassifications of the attributes *good condition*, *light damages*, and *potholes* might be false manual annotating, since there is sometimes only a fine line between the degree of damages, or the events were not fully overrun, especially for *potholes*.

The inertial sensor and the developed data processing flow can be adapted to detect other road features, which might be interesting for transportation authorities. Road features, which might be classified with a good accuracy, are wavelike deformations in longitudinal direction.

Other road infrastructure information, such as road slope or radius of curves, can also be estimated solely based on inertial sensor data, as we showed in [112, 186]. For this purpose, features from other time series, such as yaw rate, longitudinal and lateral acceleration of the vehicle, are more important than time series used for the estimation of road condition.

The data processing of acoustic sensor data can be extended to estimate the tire road noise. In general, the degree of road roughness is positive correlated with tire road noise. However, there are exceptions, such as a open-pored road surface, which absorbs sound and reduces the tire road noise. Moreover, the microphone inside the tire cavity might also be useful to detect extended water residues on the road surface.

Overall, the methods can be tuned and the results can be improved with novel demonstration, examination, and reference areas, such as *duraBASt*. Ground truth data for different surface textures for the estimation of the RRI and of different longitudinal unevenness for the estimation of RUI can be obtained with lower effort under controlled conditions. Moreover, ground truth data for defined road defects, which can be modified over time, can be acquired under defined environment conditions, such as rain and temperature.

### **6.3.4 Discussion of online estimation**

The data processing presented in this work can probably run on electronic control units (ECUs) of modern vehicles. However, the ECU has limits in computational resources, such as working memory. The results of MANOVA of the labeled inertial sensor data processing show that the most important features are peak-to-peak or standard deviation of specific space series data. The major task is to avoid the calculation of features for windows, since it is costly to keep all values. There exist online algorithms, especially for the calculation of mean and variances [187, 188], which can be applied to run the data processing on ECUs. Algorithms for the online calculation of other features such as peaks, can be adjusted from ECG signal analysis, e.g. [189]. Furthermore, there exist approaches for real-time frequency and harmonic evaluation, e.g. [190, 191].

## **6.4 Influences on classification performance**

### **6.4.1 Variation of vehicle parameters**

The classifier is trained with data, which is simulated from the BMW 116d passive full car model driving on a 4 *km*-long road with fixed size of attributes. Figure 6.17 shows the classes in feature space, described by two aggregated features from 15 single features. The class *pothole* has two clusters because two different sizes of potholes, each with smaller variations in dimensions, are simulated, which have different impacts on the vehicle vibration. The figure indicates, that the road features *pothole* and *cobblestone* are clearly separable to the other classes. *Unevenness* is close to the road features *railway crossing* and *cobblestone* but still good dividable. However, *railway crossing*, *manhole cover* and *smooth surface* are very close and show some misclassifications. The reason is that the vehicle vibrations due to these road features in simulation are similar.

The results of testing the classifier for different types of vehicles and vehicle set-ups are shown in Table 6.5. Overall, the generalization error remain good with a maximum of 10.2 % for S-Class.

The classifier from BMW 116d is used to predict unseen data, which are simulated from models of different vehicles, loads and suspensions driving over a 2 *km*-long road with random size of attributes. The testing results are shown in Table 6.6. It is obvious that the classifier performs good with the same vehicle model. However, the accuracy of the classifier declines from 90% to 81% with the increasing load on the vehicle. For other vehicles and set-ups, the performance measures decline greatly, which means the mass, size and

Figure 6.17: Classes separated by two aggregated features from 15 single features.

Table 6.5: Performance of classifiers for different vehicles.

parameters of spring and damping have an individual effect on the accuracy of the classifier. The results indicate, that the classifier, which is trained with BMW 116d, is not robust and the performance for the application on other vehicle set-ups greatly declines.

The performance measures improve, if the original classifier is trained incorporating the parameters and set-ups of the vehicles as features. Therefore, three vehicle model variations are considered as features, the load of the vehicle, the model of the vehicle and the type of suspension. The new classifier is trained on a road of length 180 km with a wide variation of vehicle loads, models, and type of suspension. Analogous to above, this classifier is tested on the presented vehicle models. The results (Table 6.7) suggest that this classifier is more robust against different vehicle models compared to the classifier solely

Table 6.6: Performance of the classifier for BMW 116d applied on different vehicle set-ups and vehicles.

based on data from one specific vehicle. However, the performance measures for slightly changes of the set-up, e.g. the increase of load, decrease compared to results shown in Table 6.6. Overall, the lowest value for performance measures is 73.0 and the highest value for generalization error is 24.5 %.

Table 6.7: Performance of the classifier for various vehicle models applied on different vehicle set-ups and vehicles.

### **6.4.2 Variation of classification parameters**

The influence of the position of the inertial sensor in the vehicle body in the simulation is shown in Figure 6.18. The best accuracy can be achieved with output obtaining from position S2, namely in the middle above the front or rear axle.

Moreover, the number of selected features and the order of the polynomial kernel also have an effect on the classification. In the practical application

Figure 6.18: Influence of the sensor position on the accuracy of the classification.

on the vehicle, the classifier should at least maintain at a high level no matter how the load of the vehicle changes. Thus each data flow contains the output simulated by a selected vehicle model driving across different road attributes with a series of loads from 0 kg to 400 kg. The features are selected with MANOVA and the number of features is set from 10 to 30. The classification is performed with support vector machine (SVM) with a polynomial kernel with orders 1 to 5. The results, respectively for the BMW 116d with anti-roll bar, are shown in Table 6.8. The results suggest that the maximum accuracy can be achieved with a kernel order of 2 and 25 road features, which leads to 13.85 % misclassification. Using 25 features and kernel order 3 is the second best option with 14.38 % misclassification. The results indicate that the classes are better separable with nonlinear classification functions. Furthermore, a higher amount of features, e.g. 30, seems to lead to over-fitting.

Table 6.8: Misclassifications in % of classifier with varying kernel order and number of features.

### **6.4.3 Discussion of extended simulation approach**

The proposed extended simulation approach enables to investigate the influence of parameter variation of vehicles on the classification performance. The simulation consists of a full car model with 10 degrees of freedom (DOF), various extensions, such as active suspension, a modified point contact tire model, dependent on vehicle velocity and length of obstacles, and road models including road features and defects. It is shown that it matches actual measured data, and therefore is a good representation of the real-world. Supervised machine learning techniques involve a costly and laboriously collection of labeled data for training and testing. The simulation reduces these costs, and it can be used to investigate and quantify the impact of specific vehicle parameters and settings on the classification results. Furthermore, the best parameters for the data processing and classification can be determined or tested.

Further improvements are to incorporate additional elements of a real vehicle into the vehicle model to the point of a commercial complex simulation, such as CarMaker [192]. With the exact set up and behavior of real vehicles in the simulation, classifiers can be trained in simulation before assigning the classifier to real vehicles.

## **6.5 Multiple vehicle combination**

The performance results of the proposed combination strategies for different scenarios, which are described in subsection 4.4.3, are shown and discussed in this section. Figure 6.19a shows the difference between the estimators for **Scenario 1**, where the precision matrix for the combination strategy and for all vehicles is identical. The *vote with precision matrix*, which incorporates the precision derived from the confusion matrix for all classes, is the one with fewest wrong predictions. Already with one vehicle, the proportion of wrong predictions is below 20 %, which means that each segment is classified correctly on average. With increasing number of vehicles, the number of wrong predictions for *vote with precision matrix* decreases faster compared to *majority vote* or *diagonal vote*. Between latter two estimates no big difference can be identified. Figure 6.19b represents the results of the prediction based on the vote with precision matrix split into the classes. The result shows a high impact of the class 5, *C*5, on the number of wrong predictions because the precision of the vehicles to predict class 5, π55, is only 0.73 and lower compared to other classes (Table 4.11).

Figure 6.19: Results of vehicle combination: Scenario 1. Each vehicle predicts the ground truth based on the global precision matrix.

In **Scenario 2**, each vehicle has an unknown precision matrix but the combination of the classifications is still performed with the global precision matrix (Table 4.11). As shown in Figure 6.20, again *vote with precision matrix* is the most promising estimate. When comparing one vehicle, which drives multiple times over the road and classifies the segments (Figure 6.20a), and multiple vehicles (Figure 6.20b), the latter one achieves better results. The reason is that the fixed precision matrix of only one vehicle can not compensate the difference to the global one and the proportion of wrong predictions converges towards 10 %. However, five or more different vehicles perform better and surpass 10 % (Figure 6.20b).

Figure 6.21 shows the influence of a vehicle with *broken sensor* from **Scenario 3** on the number of wrong predictions. In Figure 6.21a, the first vehicle has or has not an equally distributed precision matrix and therefore randomly predicts the classes. Above five vehicles, in average less than 10 % of the road is classified wrongly. Above ten vehicles the impact of the bad precision matrix can be ignored. Figure 6.21b shows the results of a simulation, in which the third vehicle has an equally distributed precision matrix, which is referred to as *broken sensor*. The blue line shows the proportion of misclassifications with the third vehicle with a *broken sensor* and no adjustments. As soon as the output of the third vehicle is included in the simulation, the number of wrong

Figure 6.20: Results of vehicle combination: Scenario 2. Unknown distribution of classification due to different precision matrices for vehicles.

classifications increases greatly. With increasing the number of vehicles with good precision matrices, the number of wrong predictions converges again to 0. The green line shows the results of the simulation with the identification of the vehicle with the bad precision matrix and the adjustment of the precision matrix in the back-end for this vehicle with the proposed Algorithm 4.4 in subsection 4.4.3. With this adjustment, the increase of number of wrong predictions is not as high as without adjustment and the number of misclassifications faster converges to 0. Summarized, this method improves the results of our multiple vehicle fusion and can identify vehicles with bad classification outputs.

Figure 6.21: Results of vehicle combination: Scenario 3. Classification with and without broken sensor. a: 1st vehicle with and without broken sensor. b: 3rd vehicle with broken sensor with and without automated adjusting precision matrix for this vehicle.

The results of **Scenario 4**, where the ground truth changes at one time sample, can be seen in Figure 6.22. After the change of the ground truth at time sample *t* = 5, the *vote with precision matrix* without forgetting needs 4 more time stamps to reduce the proportion of wrong classified segments below 10 %. When using the forget factor, already in the next time stamp an error rate below 2 % can be reached. In contrast, also correct classifications and good results are forgotten by the algorithm. However, the impact is small, which shows the results at time sample *t* = 2,3,4, where the green line is only slightly above the blue line.

Figure 6.22: Results of vehicle combination: Scenario 4. Change of the road conditions and simulated with and without forgetting of previous classifications.

#### **Conclusion and discussion**

With the presented methods to identify trajectories from different vehicles for each road road segment, an automatic annotating of sensor data with the label of the ground truth is possible. Hereby, the label of road segments, known or predicted from already trained vehicles, is transferred to new and untrained vehicles. For this purpose, the trajectories of multiple vehicles are compared and road segments, driven by various vehicles, are identified. Problems can occur with poor GPS signal. However, this can be avoided by applying an integrated navigation system, which is based on the sensor data fusion of GPS and MEMS-IMU to provide an accurate position of the vehicle [193, 194]. Modern vehicles are already equipped with integrated navigation systems. Furthermore, the difference of the time duration for obtaining the ground truth of a specific road segment and the time when a new vehicle drives this road segments and receives the label should be small. In the meantime, the ground truth could be changed, e.g. the condition of the road segment is significantly decreased or maintained. The investigation of methods to combine the prediction of various vehicles for specific road segments show, that the combination based on the precision matrix of the classifier outperforms the combination based on *majority vote*. Various extreme scenarios are simulated, e.g. a sudden change of the ground truth or wrong predictions of a vehicle, to quantify the impact on the classification results. Moreover, methods to overcome these extreme situations are presented. The combined processing is mainly performed in simulation. The next steps are to apply these methods on a fleet of vehicles.

# **7 Conclusion**

This work presents novel methods to comprehensively and automatically estimate the road condition with vehicle based sensors. To acquire the data from the sensors, a novel measurement system was developed [195]. The installation and operating process of the measurement system requires almost no manual intervention and the data can be automatically transferred to a central data base. A road unevenness index (RUI) is estimated with an inertial sensor in the vehicle body and road roughness index (RRI) with an acoustic sensor in the tire cavity. The estimation of the indices are based on the measurement of the vibration of the vehicle and tire with the mentioned sensors. It was shown from experiments under controlled conditions, that the vibrations are not only dependent on road roughness or unevenness but also on influences from the environment or vehicle operation, especially velocity. To decrease the effect of these influences, the indices are estimated for specific bins or intervals, which are constructed based on the main influences. The comparison with ground truth data suggest that the estimators conform with the actual road condition. The estimators of various vehicles can be combined and visualized on maps. Furthermore, the trend of the road condition can be identified with a robust regression over time for each road segment to be investigated.

Supervised learning data processing is introduced for the inertial and acoustic sensor to classify road features and road roughness. The major novelties include a feature selection with multivariate analysis of variances (MANOVA) to reduce the complexity and avoid over-fitting of the classifier. Two classifiers for the inertial sensor data are developed to estimate both, road surface type and road attributes. The prediction of the methods were observed and the results suggest a good performance.

Moreover, methods are introduced to realize an automatic transmission of labels to new measurement vehicles to train their classifier. With the same methods, road segments and lanes can be identified which are driven by various vehicles and methods to combine the output of these vehicles are investigated to improve the prediction rate. Overall, the methods are combined in a Matlab toolbox and graphic user interface for evaluation of data sets with alternative parametrization by non-experts.

Summarized, this work achieves the following goals

The developed methods can improve or partially substitute the current practice for road infrastructure evaluation to increase the automation rate and to reduce the costs and response rate. The results of the developed methods show that the inertial sensor and acoustic sensor represent a very good option to collect information of the tire road contact at low costs and over wide area. The data processing methods can run on electronic control units (ECUs) of modern vehicles, which serve as sensor platforms. Moreover, as the developed system is of modular design, the number and type of sensors and sensor modality can be varied. Lastly, a vehicle fleet can be equipped with the presented system as it is to estimate the condition of the road infrastructure. For example, the German Federal Environment Agency currently deploys seven measuring devices, developed in this work, and applies the presented methods, to automatically and comprehensively monitor the road condition.

# **List of Figures**

# **List of Tables**

# **List of Algorithms**

# **Bibliography**

## **Supervised seminar, bachelor, and master thesis**

# **Publications**

## **Peer-reviewed journal papers**

\* contributed equally

## **Peer-reviewed conference papers**

[195] J. Masino, M. Frey, F. Gauterin, and R. Sharma, "Development of a highly accurate and low cost measurement device for Field Operational Tests", in *IEEE International Symposium on Inertial Sensors and Systems*, Piscataway, New Jersey, USA: IEEE, 2016, pp. 74–77. doi: 10.1109/ISISS.2016.7435548.

## **Other**

## **Karlsruher Schriftenreihe Fahrzeugsystemtechnik (ISSN 1869-6058)**

Herausgeber: FAST Institut für Fahrzeugsystemtechnik

Die Bände sind unter www.ksp.kit.edu als PDF frei verfügbar oder als Druckausgabe bestellbar.

Die Bände sind unter www.ksp.kit.edu als PDF frei verfügbar oder als Druckausgabe bestellbar.

ISBN 978-3-7315-1004-8

## Karlsruher Schriftenreihe Fahrzeugsystemtechnik

The work provides novel methods to process inertial sensor and acoustic sensor data for road condition estimation and monitoring with application in vehicles, which serve as sensor platforms. Furthermore, methods are introduced to combine the results from various vehicles for a more reliable estimation.

A systematic, continuous, and comprehensive monitoring of existing roads is becoming increasingly important. Yet, current practice is laborious and time-consuming as most steps of the process are done manually. A novel measuring device for vehicles is developed to autonomously acquire a large set of real world data on public roads. In addition, an extended simulation approach is presented to record data under controlled conditions to investigate the impact of parameter variations. Furthermore, the methods for data processing are implemented in a user friendly Matlab toolbox to allow an automatic evaluation.

Overall, the developed methods can improve or partially substitute the current practice to evaluate the road infrastructure. This work contributes concepts and methods to allow an automatic and comprehensive road condition monitoring. The data processing methods can run on electronic control units of modern vehicles, which are equipped with corresponding sensors. However, the presented system consisting of the developed measuring device and Matlab toolbox can also be applied on vehicle fleets as it is.

ISSN 1869-6058 ISBN 978-3-7315-1004-8 J. Masino

Road Condition Estimation with Vehicle Based Sensors

**Band 82**