Automated crack propagation measurement on asphalt concrete specimens
  using an optical flow-based deep neural network
This article proposes a deep neural network, namely CrackPropNet, to measure
crack propagation on asphalt concrete (AC) specimens. It offers an accurate,
flexible, efficient, and low-cost solution for crack propagation measurement
using images collected during cracking tests. CrackPropNet significantly
differs from traditional deep learning networks, as it involves learning to
locate displacement field discontinuities by matching features at various
locations in the reference and deformed images. An image library representing
the diversified cracking behavior of AC was developed for supervised training.
CrackPropNet achieved an optimal dataset scale F-1 of 0.755 and optimal image
scale F-1 of 0.781 on the testing dataset at a running speed of 26
frame-per-second. Experiments demonstrated that low to medium-level Gaussian
noises had a limited impact on the measurement accuracy of CrackPropNet.
Moreover, the model showed promising generalization on fundamentally different
images. As a crack measurement technique, the CrackPropNet can detect complex
crack patterns accurately and efficiently in AC cracking tests. It can be
applied to characterize the cracking phenomenon, evaluate AC cracking
potential, validate test protocols, and verify theoretical models.
Zehui Zhu, Imad L. Al-Qadi
Automated Crack Propagation Measurement On Asphalt Concrete Specimens Using an Optical Flow-Based Deep Neural Network

###### Abstract

This article proposes a deep neural network, namely CrackPropNet, to measure crack propagation on asphalt concrete (AC) specimens. It offers an accurate, flexible, efficient, and low-cost solution for crack propagation measurement using images collected during cracking tests. CrackPropNet significantly differs from traditional deep learning networks, as it involves learning to locate displacement field discontinuities by matching features at various locations in the reference and deformed images. An image library representing the diversified cracking behavior of AC was developed for supervised training. CrackPropNet achieved an optimal dataset scale F-1 of 0.755 and optimal image scale F-1 of 0.781 on the testing dataset at a running speed of 26 frame-per-second. Experiments demonstrated that low to medium-level Gaussian noises had a limited impact on the measurement accuracy of CrackPropNet. Moreover, the model showed promising generalization on fundamentally different images. As a crack measurement technique, the CrackPropNet can detect complex crack patterns accurately and efficiently in AC cracking tests. It can be applied to characterize the cracking phenomenon, evaluate AC cracking potential, validate test protocols, and verify theoretical models.

A RESEARCH ARTICLE

sphalt Concrete, crack propagation, digital image correlation, optical flow, deep learning.

## 1 Introduction

Approximately 95 percent of paved roads in the United States are surfaced with asphalt. Cracking is a common mode of failure in asphalt concrete (AC) pavements. Many tests have been developed to assess the cracking potential of AC materials. Accurate monitoring of crack initiation and propagation during testing is crucial.

Contact tools like a linear variable differential transformer (LVDT), extensometers, strain gauges, and crack mouth opening displacement (CMOD) clip gauges are the most widely used methods to monitor crack propagation and opening in AC cracking tests. However, these tools only provide localized information, as the measurement location must be decided before testing. As shown in Figure 1, the CMOD clip gauge is attached at the bottom of the specimen when conducting the low-temperature semi-circular bending (SCB) test (Li & Marasteanu, 2010). As such, the crack opening is
only recorded at that location. Similarly, the above-mentioned contact techniques may only provide indirect crack propagation and opening measurements. For example, the load-line displacement (LLD), measured by an extensometer mounted vertically at the surface of the specimen, is used to estimate crack propagation speed. However, a such approximation is insufficient to describe the cracking phenomenon, as a crack tends to choose a path around the aggregate as it grows (Doll, Ozer, Rivera-Perez, Al-Qadi, & Lambros, 2017). In addition, orienting the contact devices is time-consuming and requires experience, especially on small specimens. Moreover, routine calibrations are needed to ensure accurate measurement. Therefore, developing an easy-to-use, accurate, and full-field crack measurement technique for AC cracking tests is imperative.

Low-level computer vision-based crack detection methods have been proposed. The most popular algorithms include thresholding, image segmentation, filtering, and blob extraction (Hartman & Gilchrist, 2004; Oliveira & Correia, 2009; Wang, Zhang, Wang, Braham, & Qiu, 2018; Ying & Salari, 2010; A. Zhang, Li, Wang, & Qiu, 2013). However, the limitation of these methods is obtaining accurate results under complex imaging environments. Deep learning, especially deep convolution neural network (CNN), has been widely used to detect and categorize cracks (Cha, Choi, & Buyukozturk, 2017; Fei et al., 2019; A. Zhang et al., 2017; L. Zhang, Yang, Zhang, & Zhu, 2016). However, these models were mainly developed for visible mature cracks; the ground truth verification relied on visual recognition. This renders the models mentioned above unsuitable for monitoring crack propagation in AC cracking tests, where small cracks in the early stages are critical but often difficult to visualize.

The digital image correlation (DIC) technique has the potential to overcome these challenges. The DIC is an optical method that measures full-field displacement and strain. Because surface cracks are defined as displacement field discontinuities, cracks with varying sizes could be located, given an accurate displacement field. A few attempts have been made to measure cracks using DIC. Due to complex crack growth, locating cracks based on DIC-measured displacement or strain field is a challenge. Current methods rely on the strain or displacement thresholding, which requires significant post-processing efforts and empirical knowledge (Buttlar et al., 2014; Safavizadeh & Kim, 2017). In addition, DIC analysis involves computationally expensive optimization, making it unsuitable for real-time applications such as crack propagation measurement in AC testing, where hundreds or thousands of images need to be analyzed. These limitations have limited the implementation of DIC as an automated

Figure 1: Low-temperature SCB test setup (Li & Marasteanu, 2010).
crack measurement technique for AC cracking tests.

This article proposes a deep neural network to automatically measure crack propagation during testing based on the optical flow concept. Compared to the existing techniques discussed above, it offers an accurate, flexible, efficient, and low-cost solution. It can accurately measure crack propagation from hundreds of images collected by low-cost cameras in less than one minute.

This paper has seven sections, and they are organized as the following: section one discusses the background and motivation of this study; section two presents the development of the database; section three introduces the architecture of the deep neural network; section four explains the training strategy; section five presents the evaluation of the proposed network; section six discusses advantages and possible applications of the CrackPropNet, and the conclusions and recommendations are presented in section seven.

## 2 Data Preparation

As shown in Figure 2, the data preparation process consists of four steps:

1. Collect raw images;
2. Compute displacement fields using DIC;
3. Label ground-truth crack edges;
4. Inspect and verify ground-truth labels.

Implementation details are described in the following sections.

### Raw Image Collection

Raw images were collected while conducting the Illinois Flexibility Index Test (I-FIT), as shown in Figure 4 (Ozer, Al-Qadi, Lambros, et al., 2016; Ozer, Al-Qadi, Singhvi, et al., 2016). An extensive testing program covering a wide range of testing conditions and materials was developed. The goal was to develop an extensive image database covering AC's diversified cracking behavior. All experiments were displacement controlled. Load-line displacement was used on room-temperature tests, while CMOD was used under low temperatures to provide better crack propagation stability (Doll et al., 2017). Two different testing temperatures: -12 and 25degC; and four different loading rates: 0.7, 6.25, 25, and 50 mm/min were considered. The fracture behavior of AC is time- and temperature-dependent. A more brittle failure is expected at lower temper

Figure 2: Data preparation procedure.
atures or higher loading rates (Al-Qadi et al., 2015). As shown in Figure 3, a total of 53 AC mixes were tested. They had different N-designs, binder types and content, aggregate mineralogy, and the amount of recycled materials. The I-FIT specimens were prepared either from lab-compacted cylindrical pills or field cores. All specimens had SCB geometry, as shown in Figure 1, while their thickness, notch length, and air void range from 25mm to 60mm, 10mm to 35mm, and 1% to 12%, respectively. For each I-FIT specimen, a speckle pattern consisting of a white layer of paint and a random black pattern on top was applied (Doll et al., 2017).

Two CCD (Charge Coupled Device) cameras were positioned perpendicularly to the surface of the I-FIT specimen to collect images during the test: a Point Grey Gazelle 4.1MP Mono (\(2048\times 2048\) pixels, 150 frames per second-fps) and an Allied Vision Prosilica GX6600 (\(6576\times 4384\) pixels, 4 fps) with a Tokina AT-X Pro Macro 100 2.8D lens. The Gazelle has a faster acquisition rate but a lower resolution than the Prosilica. The former is generally used in experiments where the materials can be considered homogeneous, while the latter aims to study damage zone evolution in heterogeneous materials such as AC. The database intentionally includes images taken from cameras with significantly different resolutions to ensure better generalization of the deep neural network.

### Compute Displacement Fields Using DIC

The displacement fields were first computed using DIC. The DIC works by tracking pixels in a sequence of images. This is achieved using area-based matching, which extracts gray value correspondences based on their similarities. First, a reference image was taken at the unloaded state, and an area of interest was selected. Then, a subset of pixels is compared to a deformed image taken at a loaded state to identify the best match. Finally, the deformation of a point in the subset can be computed using

Figure 3: Mix properties.
Equation 1, which allows for translation, rotation, shear, and combinations.

\[\begin{cases}x_{i}^{\prime}=x_{i}+u+u_{x}\Delta x+u_{y}\Delta y\\ y_{j}^{\prime}=y_{j}+v+u_{x}\Delta x+u_{y}\Delta y\end{cases} \tag{1}\]

As shown in Figure 5, \(x_{i}\) and \(y_{j}\) are Cartesian coordinates of a point \(Q(x_{i},y_{j})\) in the reference image; \(x_{i}^{\prime}\) and \(y_{j}^{\prime}\) refer to its coordinates in the deformed image; \(u\) and \(v\) denote the corresponding displacement components of the reference subset center \(P(x_{0},y_{0})\) in the x- and y- direction, respectively; \(u_{x}\), \(u_{y}\), \(v_{x}\), \(v_{y}\) are the first-order displacement gradients of the reference subset; \(\Delta x=x_{i}-x_{0}\) and \(\Delta y=y_{j}-y_{0}\). To provide adequate spatial resolution to resolve the displacement distribution between and within aggregate particles, the subset size used for correlation was carefully chosen for each test following the algorithm proposed by Pan, Xie, Wang, Qian, and Wang (2008).

### Ground-Truth Crack Edges Labeling

Once the displacement field was obtained, potential crack edges could be located following the method proposed by Zhu and Al-Qadi (2023). Figure 6 shows the displacement field (\(u\)) contour plot measured by DIC for an I-FIT specimen surface, where a crack is visible in the area of interest. The reference and deformed images were taken with the Gazelle camera when conducting the I-FIT test at 25\({}^{\circ}\)C with a 50mm/min loading rate. A subset size of \(23\times 23\) with a correlation point spacing of 11 pixels

Figure 4: Experiment set up.

Figure 5: Area-based matching (Pan, Qian, Xie, & Asundi, 2009).
was used for DIC analysis. This resulted in a spatial resolution of approximately 25 \(\mu\)m/pixel and produced roughly \(175\times 157\) square microns in the area of interest.

Then, the first-order derivative of the opening displacement \(u_{x}\) was obtained by filtering the displacement field with a \([-1,1]\) kernel. For example, Figure 7 plots \(u_{x}\) along three discrete \(y\) axes (\(y=496\), 1497, and 1816). A large \(u_{x}\) indicates the material separation between the two correlation points, suggesting a crack may present. Thus, potential crack edges could be found by locating the corresponding correlation points in the deformed image, as shown in Figure 8. Please note that the marked crack edges may not match the actual crack edges exactly because the correlation point spacing is usually larger than 1 pixel in DIC analysis. However, the effect is negligible as the spatial resolution is typically smaller than 25 \(\mu\)m/pixel, which results in an error of less than 0.3 mm. In this article, the target value for a crack edge pixel is 1, while the rest is 0.

Figure 6: Contour plot of horizontal displacement (\(u\)).

Figure 7: First-order derivative of the opening displacement field \(u_{x}\) along three discrete \(y\) axes.
### Verification of Ground-Truth Labels

The above-described procedure may falsely label some pixels as crack edges under certain circumstances. For example, suppose an I-FIT specimen has irregularities and holes on the surface that cannot be painted or create shadows. In that case, the error in DIC measurements will increase compared to experiments on flat surfaces. This may lead to falsely labeled crack edge pixels. To make the ground-truth crack edges as accurate as possible, every image went through two rounds of inspection:

1. Automated inspection: for a sequence of deformed images, if pixel \(A\) was labeled as a crack edge in frame \(F_{t_{0}}\), but not in the following frames \(F_{t_{1:n}}\), the label would be corrected. In contrast, if pixel \(A\) was not labeled as crack edge in frame \(F_{t_{n}}\), but was labeled in previous \(F_{t_{0:n-1}}\) and following frames \(F_{t_{n+1:2n}}\), the label would be corrected;
2. Manual inspection: the ground-truths were visually inspected and verified.

### Image Library Summary

An image library, made of pairs of images with and crack edge labels, was developed for supervised learning. It consisted of 2,560 frame pairs. The image library represented the diversified cracking behavior of AC. Images were collected in the past eight years by four different operators. The original images collected by the Gazelle and the Prosilica camera have resolutions of \(6576\times 4384\) and \(2048\times 2048\), respectively. The original image was downsized to \(1024\times 1024\) by min-pooling and cropping to balance computational overhead and accuracy. This resulted in a spatial resolution of approximately 0.05 mm/pixel. Because the correlation point spacing in DIC analysis was typically larger than 10 pixels, the labeled ground truth crack edges were discontinuous. To provide more accurate ground-truth labels and reduce the computational cost, the ground-truth crack edge map was downsized from the original resolution to \(128\times 128\). This resulted in a spatial resolution of approximately 0.4 mm/pixel, which is adequate in this task, where the measurement area is larger than \(50\times 50\)mm. Figure 9 illustrates

Figure 8: Deformed image with crack edges marked in red dots.
a sequence of images together with their ground-truth crack edge maps. They were collected while conducting an I-FIT test on a typical AC mix (an Illinois N90 mix) at 25 mm/min and 25degC.

To evaluate the sufficiency of the developed image library, Table 1 compared it with datasets used in previous studies for relative tasks. The size of the developed image library is comparable to existing datasets. It is worth noting that StrainNet uses synthetic images, which have the advantage of developing large datasets quickly and inexpensively. However, real images were used in this paper because of the complexity of crack shapes in AC testing.

## 3 Network Architecture

Training a deep CNN from scratch requires a large dataset and significant computational power, which is impossible in most situations. In practice, pre-trained networks could be used as initialization or feature extractors for the task of interest. Because surface cracks are defined as displacement field discontinuities, crack propagation measurement in AC fracture testing can be accomplished by stacking edge detection layers on pre-trained networks for optical flow estimation.

This section provides a brief overview of existing networks on optical flow estimation. The proposed network architecture is discussed in detail.

\begin{table}
\begin{tabular}{l|l|l|l} \hline \hline
**Dataset** & **Number of frames** & **Resolution** & **Type** \\ \hline \multicolumn{4}{l}{_Task: Pavement crack detection_} \\ \hline CRACK500 (Yang et al., 2019) & 500 & 2000\(\times\)1500 & Real \\ \hline GAPs384 (Eisenbach et al., 2017) & 1,969 & 1920\(\times\)1080 & Real \\ \hline CrackNet (A. Zhang et al., 2017) & 2,000 & 1024\(\times\)512 & Real \\ \hline CrackNet-V (Fei et al., 2019) & 3,083 & 1024\(\times\)512 & Real \\ \hline \multicolumn{4}{l}{_Task: Optical flow estimation_} \\ \hline KITTI2015 (Menze \& Geiger, 2015) & 800 pairs & 1242\(\times\)375 & Real \\ \hline Sintel (Butler, Wulff, Stanley, \& Black, 2012) & 1064 pairs & 960\(\times\)540 & Real \\ \hline \multicolumn{4}{l}{_Task: DIC with deep learning_} \\ \hline StrainNet (Boukhtache et al., 2021) & 363 reference frames & 256\(\times\)256 & Synthetic \\ \hline \hline Proposed & 2,560 pairs & 1024\(\times\)1024 & Real \\ \hline \end{tabular}
\end{table}
Table 1: Comparison with datasets used in previous studies for relative tasks.

Figure 9: Representative image pairs with ground-truth labels from the image library.
### CNN-Based Methods for Optical Flow Estimation

Optical flow is the pattern of apparent motion of objects due to the relative motion between an observer and a visual scene (Warren and Strelow, 2013). Traditional energy-minimization-based approaches involve computationally expensive optimization, making them unsuitable for large-scale real-time applications such as crack propagation measurement in AC testing, where hundreds of images need to be processed.

Another promising approach is the fast and end-to-end trainable CNN framework. Dosovitskiy _et al._ proposed two CNNs: FlowNetS and FlowNetC, to learn optical flow from a synthetic dataset (Dosovitskiy et al., 2015). As shown in Figure 10, FlowNetS stacks the reference and deformed images together and feeds them through a rather generic network to extract optical flow. FlowNetC creates separate processing streams for the reference and deformed images to generate two feature maps. Then, it resembles them with a correlation layer that performs multiplicative patch comparisons. However, FlowNetS and FlowNetC have problems with small displacements and noisy artifacts in estimated optical flow fields. To improve the performance, Ilg _et al._ developed FlowNet2.0 by stacking multiple FlowNetS and FlowNetC (Ilg et al., 2017). It reduces the estimation error by more than 50% compared to FlowNet and has been proven to be efficient in many other applications such as motion segmentation and action recognition. FlowNet2.0 outperforms other state-of-art networks, such as SpyNet, RecSpyNet, and LiteFlowNet (Hu et al., 2018; Hui et al., 2018; Ranjan and Black, 2017) in terms of accuracy.

The above studies demonstrate that CNNs are powerful in estimating optical flow. This inspired the development of similar networks to solve analogous problems in other fields. For example, Boukhtache _et al._ developed StrainNet to retrieve displacement and strain fields from pairs of reference and deformed images. It uses FlowNetS as the backbone and achieves comparable accuracy as DIC with a significant improvement in computing time (Boukhtache et al., 2021).

Figure 10: FlowNetS and FlowNetC Dosovitskiy et al. (2015).
### FlowNetS and FlowNetC

Both FlowNetS and FlowNetC consist of a contracting and an expanding part. Although the two networks adopt different approaches in contracting, they share the same expanding part. The two parts of the network are discussed in detail below.

#### 3.2.1 Contracting

FlowNetS concatenates the reference and deformed images together as input and lets the network learn how to process the image pair to extract motion information. In contrast, FlowNetC creates separate processing streams for the reference and deformed images to generate two feature maps. Then, it resembles them with a correlation layer that performs multiplicative patch comparisons. Given two feature maps \(\mathbf{f}_{1}\), \(\mathbf{f}_{2}\), with dimension \(c(\text{number of channels})\times w(\text{width})\times h(\text{height})\), the correlation layer compares patches in \(\mathbf{f}_{1}\) and \(\mathbf{f}_{2}\) as below:

\[c(\mathbf{x}_{1},\mathbf{x}_{2})=\sum_{\mathbf{o}\in[-k,k]\times[-k,k]}\langle \mathbf{f}_{1}(\mathbf{x}_{1}+\mathbf{o}),\mathbf{f}_{2}(\mathbf{x}_{2}+ \mathbf{o})\rangle \tag{2}\]

\(\mathbf{x}_{1}\) and \(\mathbf{x}_{2}\) denote the center square patch of size \(K\coloneqq 2k+1\) in \(\mathbf{f}_{1}\) and \(\mathbf{f}_{2}\), respectively. Because the correlation operation is fundamentally equivalent to convolving data with other data, it has no trainable weights. To reduce computation cost, the maximum displacement is constrained to \(d\), which means that for each location \(\mathbf{x}_{1}\), correlations \(c(\mathbf{x}_{1},\mathbf{x}_{2})\) are only computed in a neighborhood of size \(D\coloneqq 2d+1\). The output size of the correlation layer is \(D^{2}\times w\times h\).

#### 3.2.2 Expanding

As shown in Figure 12, to refine the coarse pooled representation and obtain a dense flow field, fractionally-strided convolution is applied to feature maps first. Then, they are concatenated with corresponding feature maps from the contracting part. This operation preserves high-level features as well as retains fine local features. The process is repeated four times, with each step doubling the resolution. A bi-linear up-sampling with a factor of four is performed at the end to obtain the original image resolution.

Figure 11: Proposed network architectures.
### Proposed Network Architecture

As shown in Figure 11, the proposed network combines one FlowNetC and two FlowNetS. First, reference (\(I_{r}\)) and deformed (\(I_{d}\)) images are fed into FlowNetC to generate an estimated flow field (\(w_{1}=(u_{1},v_{1})^{\top}\)) at the original image resolution. Second, subsequent FlowNetS gets reference image, deformed image, estimated flow field \(w_{1}\), warped deformed image (\(\tilde{I}_{d,1}(x,y)=I_{d}(x+u_{1},y+v_{1})\)), and brightness error field (\(e_{1}=||\tilde{I}_{d,1}-I_{r}||\)); and outputs an estimated flow field (\(w_{2}=(u_{2},v_{2})^{\top}\)) at the original image resolution. The concatenated input allows FlowNetS to assess the previous error more easily and compute an incremental update. Third, a modified FlowNetS receives reference image, deformed image, estimated flow field \(w_{2}\), warped deformed image (\(\tilde{I}_{d,2}(x,y)=I_{d}(x+u_{2},y+v_{2})\)), and brightness error field (\(e_{2}=||\tilde{I}_{d,2}-I_{r}||\)); and outputs an edge probability map for which the resolution is eight times smaller than the original image. The modified FlowNetS differs from FlowNetS in the expanding part:

* The last \(3\times 3\) convolution layer is cut. Inspired by Richer Convolutional Features for edge detection (Liu, Cheng, Hu, Wang, & Bai, 2017), layers shown in Figure 13 are added.
* Sigmoid units are connected to the final layer to generate an edge probability map.

Figure 12: Refinement part.

Figure 13: Edge detection layers.
## 4 Training

### Class-Balanced Loss Function

Because the distribution of crack-edge/non-crack-edge pixels greatly varies and is heavily biased: more than 90% of the ground truth pixels are non-crack-edge. A cost-sensitive loss function must be considered to balance the loss between positive (crack-edge)/negative (non-crack-edge) classes. Specifically, the following (Equation 3) class-balanced cross-entropy loss function was used:

\[\begin{split} L(W)&=\alpha\sum_{j\in Y_{-}}\log(1- \Pr(X_{j};W))\\ &+\beta\sum_{j\in Y_{+}}\log\Pr(X_{j};W)\end{split} \tag{3}\]

in which

\[\begin{split}\alpha&=\gamma+\frac{|Y_{+}|}{|Y_{+}|+| Y_{-}|}\\ \beta&=\lambda\cdot\frac{|Y_{-}|}{|Y_{+}|+|Y_{-}|} \end{split} \tag{4}\]

\(Y_{+}\) and \(Y_{-}\) denote the positive and negative sample sets, respectively. The hyperparameters \(\gamma\) and \(\lambda\) are used to balance positive and negative samples. \(X_{j}\) represents the activation value at each pixel \(j\), \(\Pr(X)\) is the standard sigmoid function, and \(W\) denotes all parameters in the network.

### Data Augmentation

Data augmentation is an often-used strategy to improve model generalization (Krizhevsky, Sutskever, & Hinton, 2012). The augmentations used in this study include geometric transformation: horizontal flip, as well as changes in brightness, contrast, saturation, and hue. It is worth mentioning that the same transformations were applied to both reference and deformed images. The augmentation was performed online during network training.

The brightness, contrast, and saturation factors are sampled uniformly from \([0.95,1.05]\); the hue factor is chosen from \([-0.05,0.05]\).

### Training Strategy

The following strategy was used in training the network:

1. The AdamW was chosen as the optimization method because it showed faster convergence than standard stochastic gradient descent with momentum in this task (Loshchilov & Hutter, 2017). The recommended parameters: \(\beta_{1}=0.9\) and \(\beta_{2}=0.999\) were used, and the weight decay coefficient was set as \(1e^{-4}\).
2. Fairly small mini-batches of six-image pairs were used.
3. The training started with a learning rate of \(5e^{-5}\), and it was divided by 2 every 5 epochs. The network was trained for 40 epochs.
4. To monitor over-fitting during training, the dataset was randomly split into 2,248 training and 312 validation pairs.

### Evaluation Metrics

Because of the similarity with edge detection, it is intuitive to directly leverage its evaluation criteria for this task. Given a crack edge probability map, a threshold is needed to generate the crack edge map. Two commonly used strategies are optimal dataset scale (ODS) and optimal image scale (OIS). The former uses a fixed threshold for all images in the dataset, while the latter employs an optimal threshold for each image (Liu et al., 2017; Xie and Tu, 2015). This paper used the F-1 (\(\frac{2\cdot Precision\cdot Recall}{Precision+Recall}\)) of both ODS and OIS to assess the network's performance. They were calculated using Equation 5 (Yang et al., 2019). It is worth noting that, unlike previous studies, zero tolerance was allowed for correct matches between ground truth and prediction (Liu et al., 2017; Xie and Tu, 2015; Yang et al., 2019).

\[\text{ODS F}=\max\{\frac{2P_{t}\cdot R_{t}}{P_{t}+R_{t}}:t=0.01,0.02, \ldots,0.99\} \tag{5}\] \[\text{OIS F}=\frac{1}{N_{i}}\sum_{i}^{N_{i}}\max\{\frac{2P_{t}^{ i}\cdot R_{t}^{i}}{P_{t}^{i}+R_{t}^{i}}:t=0.01,0.02,\ldots,0.99\}\]

The \(t\) denotes the threshold, \(i\) refers to the index of an image, and \(N_{i}\) is the total number of images. \(P_{t}\) and \(R_{t}\) represent precision and recall for the chosen threshold \(t\), respectively. Precision refers to the proportion of identified crack edge pixels that were correct, while Recall represents the fraction of crack edge pixels identified correctly. It is challenging to achieve high Precision and high Recall simultaneously because they often conflict with each other. A high F-1 can only be achieved when both Precision and Recall are high.

### Training Result

The training took 13 hours on an _NVIDIA TESLA V100_ GPU. Figure 14 shows the class-balanced cross-entropy loss decay curve and the validation F-1 curve. To compute the F-1 during training, a fixed threshold of 0.5 was used to generate edge maps from edge probability maps instead of using ODS or OIS strategies. The highest F-1 was observed at the \(37^{th}\) epoch, and the corresponding model was considered optimal. Figure 15 shows the precision-recall curve of the final model. The trained model achieved \(\text{ODS}=0.769\) and \(\text{OIS}=0.772\) on the validation dataset.

## 5 Testing and Evaluation

### Testing Result

A testing dataset consisting of 188 frame pairs was developed to validate the trained model further. The images were collected using the Gazelle and the Prosilica cameras
while conducting I-FIT tests. The same ground-truth crack edges labeling procedure was followed. CrackPropNet provided running speeds of 6fps and 26fps on a _NVIDIA TESLA P100_ and a _TESLA V100_ GPU, respectively. The trained model achieved \(\text{ODS}=0.755\) and \(\text{OIS}=0.781\) on the testing dataset. The validation and testing images' performance were similar, suggesting that the over-fitting problem was avoided. Figure 16 shows F-1s on each frame pair. The trained model generally performed well in most frame pairs. Less than 2% of edge map predictions had F-1s smaller than 0.4. Moreover, the model performed exceptionally well in differentiating the frames with no cracks from those with cracks, which indicates its robustness in capturing crack initiation. Figure 17 shows edge map predictions with F-1s lower than 0.4. It could be noticed that all of them happened in the early stage of crack development.

Figure 18 provides a visualization of a sequence of CrackPropNet-measured crack edges. They were intentionally selected to be shown here because of their lower-than-average F-1s. The final model produced high-quality crack edges. The lower-than-average F-1 was mainly due to the nature of edge detection, where predicted edges are expected to be coarser than ground-truths (Liu et al., 2017; Xie & Tu, 2015).

Figure 14: Training progress.

Figure 15: Precision-recall curve evaluated on the validation dataset.
### Noise Robustness Evaluation

Some noise is always present in digital images, especially for those taken by non-industrial low-cost cameras. Because the training images were collected using high-performance hardware, it is critical to evaluate the noise robustness of the trained model.

Figure 16: F-1s on each frame pair.

Figure 17: CrackPropNet-measured cracks with low F-1s.

Figure 18: Examples of CrackPropNet-measured crack propagation.
Random Gaussian noise was injected to frame pairs in the testing dataset based on the Gaussian noise model:

\[P(g)=\sqrt{\frac{1}{2\pi\sigma^{2}}}e^{-\frac{(a-\mu)^{2}}{2\sigma^{2}}} \tag{6}\]

\(\mu\) and \(\sigma\) denote mean and standard deviation, respectively. \(g\) refers to the gray value. Three \(\sigma\) (5,15,25) values were used to simulate various degrees of noise: low, medium, and high. Figure 19 shows images with different levels of random Gaussian noises injected.

Table 2 shows the model performance on noise-injected images in the testing dataset. As would be expected, the measurement accuracy decreases as \(\sigma\) increases. The trained model performs well on images with low to medium noise levels.

### Ideal-Ct

To evaluate the generalization of the trained model, a small dataset consisting of 91 frame pairs was developed. The images were collected using the Prosilica (\(6576\times 4384\) pixels, 4 fps) while conducting the indirect tensile cracking test (IDEAL-CT), as shown in Figure 20. The spatial resolution was about 35 \(\mu\)m/pixel. The test was performed at 25\({}^{\circ}\)C and 50 mm/min LLD on a cylindrical specimen of 62 mm in thickness and 150 mm in diameter. The IDEAL-CT is fundamentally different from the I-FIT test. The

\begin{table}
\begin{tabular}{c|c|c} \hline \hline \(\sigma\) & **ODS** & **OIS** \\ \hline
5 & 0.6643 & 0.7720 \\ \hline
15 & 0.5918 & 0.6586 \\ \hline
25 & 0.5292 & 0.5915 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Model performance on testing images with noise injected.

Figure 19: Images with different levels of random Gaussian noises were injected.
former is a strength test requiring no notch, while the latter is a fracture test with a pre-crack (notch). Most I-FIT specimens have a single, well-defined crack path, unlike the IDEAL-CT specimens with multiple crack paths (Al-Qadi, Said, Ali, & Kaddo, 2021). Moreover, as shown in Figure 21, the images contained blurred backgrounds, which poses a new challenge to the trained model.

Figure 21 illustrates a sequence of images with their ground-truth crack edge maps. They were collected while conducting an IDEAL-CT test on an AC mix with high asphalt binder replacement (20%) at 50 mm/min and 25degC.

The trained model achieved \(\text{ODS}=0.588\) and \(\text{OIS}=0.605\) on the evaluation dataset. Figure 21 provides a visualization of a sequence of CrackPropNet-measured crack edges. Overall, the trained model showed promising accuracy on a dataset that is fundamentally different from the training dataset. The measurement accuracy increased as the crack propagated downwards. The trained model was able to measure fine details of mature cracks, as shown in Figure 21, frames 3 and 4. Figure 22 shows F-1s on each frame pair. The relatively low overall F-1 was mainly due to the poor performance on small cracks in the early stage of development, where multiple crack paths were presented in an IDEAL-CT strength test specimen.

The promising accuracy of the CrackPropNet in the case of IDEAL-CT indicated the model was well-trained to locate displacement field discontinuity, which is the

Figure 21: Representative IDEAL-CT image pairs with ground-truth and CrackPropNet-measured crack edges.

Figure 20: IDEAL-CT setup.
definition of crack. As would be expected, the CrackPropNet could provide a relatively accurate measurement of crack propagation in other AC cracking tests regardless of the cracking mechanisms.

### Application: Crack-Propagation Speed

As a crack measurement technique, the CrackPropNet can detect complex crack patterns accurately and efficiently in AC cracking tests. The trained model was applied to calculate crack-propagation speed in a fracture test as a case study to demonstrate its usefulness.

Crack-propagation speed is one of the main AC cracking characteristic factors. A large crack-propagation speed after initiation indicates the mix is brittle and prone to cracking. Most state-of-art AC cracking potential prediction indices rely on an approximate crack-propagation speed. For example, according to AASHTO T393, the flexibility index (FI) from the Illinois-flexibility index test (I-FIT) uses the post-peak inflection-point slope from the Load-LLD curve to proxy the crack-propagation speed. The speed was assumed constant (Al-Qadi et al., 2015). With the help of the trained model, the true crack-propagation speed could be easily derived and used to calculate cracking indices.

This case study included two plant-produced AC mixes, and their design details are summarized in Table 3. Raw images were collected while conducting the I-FIT test at 50 mm/min and 25\({}^{\circ}\)C. Four replicates were used for each mix. It was expected that mix two would have a much higher crack propagation speed than mix one because:

* Mix two had significantly lower asphalt content than mix one.
* Mix two used recycled materials, while mix one did not.

Figure 23 shows the CrackPropNet-measured and ground-truth mean crack-propagation speed of AC mixes one and two. The speed was calculated by tracking the crack front and averaged along the crack path. As would be expected, the mean crack-propagation speed measured on mix two specimens was 72% faster than that on mix one, indicating that mix two is more prone to cracking than mix one. The CrackPropNet-measured mean crack-propagation speed was similar to the ground-truth crack-propagation speed. The trained model achieved a mean absolute error of 1.08 mm/s on the tested specimens. Moreover, the CrackPropNet captured the AC material-inherent crack propagation speed variability.

Figure 22: F-1s on each frame pair of the IDEAL-CT dataset.
## 6 Discussion

As summarized in Table 4, the CrackPropNet offers an accurate, flexible, efficient, and low-cost solution for crack propagation measurement in AC cracking tests. To measure crack propagation on AC specimen surfaces, CrackPropNet only needs a series of images collected by a low-cost camera while conducting cracking tests and a computer with GPU for post-processing.

Although the CrackPropNet was trained on an image database of I-FIT tests, its promising measurement accuracy in the case of IDEAL-CT suggested that it could provide a relatively accurate measurement of crack propagation in other AC cracking tests. This is because the architecture of the CrackPropNet was designed to learn to locate displacement field discontinuities (i.e., cracks) regardless of the cracking mechanism.

\begin{table}
\begin{tabular}{l|l|l|l|l} \hline \hline Technique & Accuracy & Flexibility & Efficiency & Cost \\ \hline Contact Tools (e.g., LVDT, clip gauge) & High & Low & Medium & \(\sim\)81,000 \\ \hline Low-Level Computer Vision (e.g., thresholding) & Low & Medium & Medium & \(\sim\)8500 \\ \hline Digital Image Correlation (Zhu \& Al-Qadi, 2023) & High & High & Low & \(\sim\) 810,000 \\ \hline CrackPropNet & Higha  & High & High & \(\sim\) 81,500 \\ \hline \end{tabular}
\end{table}
Table 4: Comparison between the CrackPropNet and other crack measurement techniques.

Figure 23: CrackPropNet-measured and ground-truth mean crack-propagation speed of mix one and two.

\begin{table}
\begin{tabular}{l|l|l} \hline \hline Property & Mix 1 & Mix 2 \\ \hline Type & SMA & Dense-Graded \\ \hline Binder Grade & PG 70-22 & PG 64-22 \\ \hline Asphalt Content (\%) & 7.3 & 4.98 \\ \hline NMAS(mm) & 4.75 & 12.5 \\ \hline ABR (\%) & 0 & 20 \\ \hline VMA (\%) & 18.5 & 14.6 \\ \hline \end{tabular}
\end{table}
Table 3: Mix Design Details for AC Mixes Used in This Study.
CrackPropNet has many applications. Examples are listed below:

* Compare AC mixes' cracking potential. In a C* fracture test, a video of the specimen surface is recorded during the test, and crack propagation is measured via visual recognition, which is subjective and time-consuming (Stempihar, 2013). Instead, one can use CrackPropNet to measure crack propagation. An AC mix with a faster crack propagation speed indicates that it is more prone to cracking.
* Validate test protocols. To validate the testing protocol of the single-edge notched beam (SENB) test, wire crack detection gauges were glued to the specimen surface to monitor crack propagation (Wagoner, Buttlar, & Paulino, 2005). The gauges only provided localized information. Instead, CrackPropNet can be used for full-field crack propagation measurement.
* Derive new or calibrate existing cracking indices. The FI, the primary outcome of the I-FIT, introduced the post-peak inflection-point slope to proxy the crack-propagation speed after initiation. With the development of CrackPropNet, the actual crack propagation can be efficiently measured and used to compute cracking indices. Similarly, it can be used to calibrate existing indices.

## 7 Conclusions

This article proposes an efficient deep neural network, namely CrackPropNet, to measure crack propagation on AC specimens during testing. The proposed approach provides accuracy, flexibility, efficiency, and cost-effectiveness compared to other techniques, including contact measurements, low-level computer vision, and DIC.

CrackPropNet involves learning to locate displacement field discontinuities (i.e., cracks) by matching features at various locations in the reference and deformed images. The input of CrackPropNet includes a reference and a deformed image, and a crack edge probability map is generated as the output. This was accomplished by stacking edge detection layers on a pre-trained optical flow estimation network consisting of a FlowNetC and two FlowNetS.

An image library was developed for supervised learning. It represents the diversified AC cracking behavior. CrackPropNet could provide running speeds of 6fps and 26fps on a _NVIDIA TESLA P100_ and a _TESLA V100_ GPU, respectively. Besides, it achieved promising measurement accuracy with an ODS F-1 of 0.755 and an OIS F-1 of 0.781 on the testing dataset.

It was demonstrated in an experiment that low to medium-level Gaussian noises had a limited impact on the measurement accuracy of CrackPropNet. Besides, the model showed promising performance on a fundamentally different dataset, which consisted of images collected while conducting the IDEAL-CT strength test. A case study demonstrated that CrackPropNet could accurately calculate crack-propagation speed, one of the main AC cracking characteristics.

CrackPropNet has many applications, including characterizing the cracking phenomenon, evaluating AC cracking potential, validating test protocols, and verifying theoretical models. The promising performance of CrackPropNet suggests that an optical flow-based deep learning network offers a robust solution in accurately and efficiently measuring crack propagation on AC specimens.

The followings are recommended for further research:

* Images collected during the process of crack propagation possess sequential nature. It is worth investigating if deep recurrent optical flow neural networks could
boost the accuracy.
* The accuracy and generalization of CrackPropNet are limited by the size of the database, which could be expanded by collecting more images from various AC tests.
* Because of AC's material-inherent variability, it is suggested to monitor both sides of a specimen for the test replicates to obtain a reliable crack propagation measurement.
* It is recommended to verify existing cracking indices using CrackPropNet. It would enable contractors and transportation agencies to assess the cracking potential of AC mixes more accurately and efficiently.

## Data Availability Statement

Examples of the image database and pre-trained CrackPropNet are available at [https://github.com/zehuiz2/CrackPropNet](https://github.com/zehuiz2/CrackPropNet).

## Author Contributions

The authors confirm their contribution to the paper as follows: study conception and design: Zehui Zhu and Imad L. Al-Qadi; data collection: Zehui Zhu; analysis and interpretation of results: Zehui Zhu and Imad L. Al-Qadi; draft manuscript preparation: Zehui Zhu and Imad L. Al-Qadi. All authors reviewed the results and approved the final version of the manuscript.

## Acknowledgment

The authors would like to thank Jose Julian Rivera Perez, Berangere Doll, Uthman Mohamed Ali, and Maxwell Barry for their help in preparing test specimens and collecting raw images. The contents of this report reflect the view of the authors, who are responsible for the facts and the accuracy of the data presented herein.

## Disclosure statement

The authors report there are no competing interests to declare.
