Automated Detection of Potholes in Pavements using Deep Neural Networks with Fine Tuning and Data Augmentation Techniques

doi:10.21203/rs.3.rs-4437158/v1

Download PDF

Research Article

Automated Detection of Potholes in Pavements using Deep Neural Networks with Fine Tuning and Data Augmentation Techniques

https://doi.org/10.21203/rs.3.rs-4437158/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Automatic detection of potholes for pavement condition assessment leads to great savings in expenses and allows a better spending of resources destined to road infrastructure management. Out of all the available pothole detection techniques, the use of image-based methods within neural networks frameworks is the one that has offered the best balance between cost and accuracy. Convolutional Neural Networks (CNN) are deep neural network models specially designed for image processing problems. They have achieved remarkable results in many applications. Inspired by the success of such models, in this paper, we have evaluated the performance of 8 CNN methods in the task of pothole detection from pavement images. Along with the evaluation of the CNN architectures, we have also tested the impact of using pre-trained models with fine tuning procedures and data augmentation techniques. Such procedures enable the use of fewer training images. Given the high cost and labor intensiveness of obtaining labeled images, the annotated images were intentionally randomly selected to reduce the dataset from its original size. The dataset of images is from Brazilian roads acquired from a typical setup of a camera attached to a vehicle. The results indicate that pre-trained CNN models with fine tuning constitute a promising technique for pothole detection in pavements, especially when a large amount of labeled data is not available.

Pothole Detection

Deep Neural Networks

Transfer Learning

Pavement condition assessment is a fundamental task in road maintenance programs. Budget restrictions and high maintenance costs produce a need for rationalized efforts for adequate maintenance and reduced life cycle costs (Meneses et al., 2013; Meneses and Ferreira, 2013; Liu et al., 2015; Babashamsi et al., 2016). With the increasing number of vehicles, pavement monitoring has received more attention given the high number of accidents related to poor road conditions (Chen et al., 2020). Such a problem is even more noticeable in developing countries, where financial resources for road maintenance are insufficient (Hoang, 2018).

Among all pavement distresses, potholes are one of the most common and dangerous (Omanovic et al., 2013; Akula et al., 2019). They are defined as bowl-shaped depressions in the pavement surface. Aging, heavy traffic, poor drainage, thin asphalt layer, and weak substructure are frequent causes of potholes (Ryu et al., 2015). In maintenance programs, this distress is generally manually detected by inspectors of agencies during periodical surveys (Hoang, 2018). Although such procedures may lead to trustworthy results, the task is labor-intensive and time-consuming. In recent years, many researchers have developed automatic pothole detection procedures (Omanovic et al., 2013; Harikrishnan and Gopi, 2017; Ye et al., 2019; Anand et al., 2018; Varona et al., 2019; Chen et al., 2020; Majidifard et al., 2020). This is also ongoing in many places in Brazil (Branco and Segantine, 2015; Espindola et al., 2021; Serafim et al., 2022; Antunes and Dantas, 2023), particularly in a partnership between Academia and the Court of Audits of the Ceará State (TCE, 2023).

Pothole detection methods can be grouped into three main categories: (i) vibration-based methods (Harikrishnan and Gopi, 2017; Varona et al., 2019); (ii) 3D reconstruction methods; and (iii) image-based methods (Anand et al., 2018; Ye et al., 2019; Chen et al., 2020). Vibration-based strategies rely on the use of accelerometers on the vehicle to identify potholes using the vibration signatures generated when passing over a pothole. Although this method has a reduced cost and is simple to implement, detecting pothole signatures may be challenging due to a large variety of surface shapes (Omanovic et al., 2013). 3D reconstruction (Li et al., 2009; Zhang et al., 2014) methods use laser scanners to construct accurate digital models of pavement surfaces. The application of this method is limited by the high cost of laser scanners and also the high computational cost required to process the large amount of points collected by the scanner. Finally, in the image-based approach, one uses images collected by regular cameras, which are analyzed with image processing techniques. Since image-based methods exhibit a good balance between cost and accuracy, such approach is followed by most of the recent works (Lekshmipathy and Velayudhan, 2020).

Image-based pothole detection is challenging because potholes have various shapes and other objects (shadows, horizontal signaling objects etc.) may be misclassified as a pothole under complex real-world conditions (Ye et al., 2019). Hence, over the years, several researchers developed works using various methods for pothole detection. The first papers on this topic mainly rely on the use of image processing techniques to extract features from raw images. This process is highly influenced by the knowledge of a specialist and the features are often related to image texture and color. Since the process is specialist-dependent, the feature design is not fully automated and is usually termed as a handcrafted procedure. After that, classifiers such as Neural Networks (NN) and Support Vector Machines (SVM) are used to classify pothole image segments (Koch and Brilakis, 2011; Koch et al., 2013; Omanovic et al., 2013; Ryu et al., 2015). More recently, researchers have focused on the use of Deep Neural Networks (DNN) (Anand et al., 2018; Suong and Jangwoo, 2018; Ye et al., 2019; Chen et al., 2020), which stack several layers of artificial neurons to enable the learning of complex patterns from data. DNNs arise as promising methods for the task of learning from images, as more data become available along with computational power with a fair cost. Apart from the remarkable accuracy, one important advantage of DNNs is that no separate feature extraction step is required. Thus, the process is not specialist-dependent, but fully automated.

Although much work has already been done in image-based pothole detection, two important aspects were neglected in previous works. First, available literature tends to focus on locating potholes within an image of a pavement surface. For road maintenance programs this task is unnecessarily complex. A decision-maker need primarily the information that an image contains a pothole and not the precise number of potholes and their respective positions on the image. Such modification of the original problem may not only align goals from researchers and practitioners, but also turn the problem into an easier machine learning task. The second aspect is related to the use of pre-trained DNNs. In such a strategy, a DNN is trained with large amounts of data for a particularly difficult image-based task. Then, given another image recognition task, one shall use several trained intermediate (hidden) layers of the aforementioned DNN and only adapt the final layers according to the new task at hand. That approach, named deep transfer learning, has led to remarkable performance in many areas, with the benefit of requiring a reduced number of training samples (Zoph et al., 2016; Kaya et al., 2017; Yang et al., 2018; Kim et al., 2020). It should also be emphasized that proper training of DNNs would require millions of labeled images, not often available for pavement applications. In such cases, the use of transfer learning (hidden layers of pre-trained neural models are used) is common, where only the parameters of the final layers are adjusted for the new task. Furthermore, data augmentation techniques can be applied to enhance the neural network's generalization capabilities. Alterations in backgrounds and textures are recognized to complicate the classification process, particularly when dealing with roads (Mikolajczyk and Grochowski, 2018; Chen et al., 2022).

This paper aims to evaluate several pre-trained DNN architectures in the detection of images with potholes and select best strategies for automated detection. The images were collected in a real-world scenario, where a camera is mounted in a car that performed a survey on roads of a Brazilian state. All images were manually labeled by two independent specialists. On the basis of our computational experiments, we can state that most pre-trained DNNs have achieved promising results, even though only 360 labeled images were used for training. The ResNet101V2 was the one that performed better, with a test accuracy of 89.44%.

The remaining of this paper is organized as follows. Section 2 presents the related work. Section 3 shows some basic concepts of deep transfer learning. The proposed methodology is presented in Section 4 and the empirical results are shown in Section 5. Section 6 contains the conclusions and final remarks.

The design of image-based pothole detection algorithms followed the recent trends in image processing. Seminal works (Koch and Brilakis, 2011; Koch et al., 2013; Omanovic et al., 2013; Ryu et al., 2015) are based on the classical image recognition procedure. First, a subset of the images is selected for training and several image enhancement techniques are used. After that, one can extract image features related to edges, texture and shape so that a pothole can be detected.

Koch et al. (2011) proposed the use of texture and shape features along with a simple threshold-based classifier, and Koch et al. (2013) presented an improved version. Omanovic et al. (2013) also considers shape-based features and identifies potholes using a clustering technique. Ryu et al. (2015) applies morphological operations and noise filtering to improve the original images. Then, the potholes are detected using shape and texture features.

Recently, the state-of-the-art of many image processing tasks were improved by the use of deep learning methods based on convolutional neural networks, a biologically inspired architecture tailored to handle image data (LeCun et al., 2010; Krizhevsky et al., 2012). The remarkable results obtained by deep learning models are explained by the capability of such models to extract relevant features in an automatic way (Goodfellow et al., 2016). In pothole detection, many researchers have devoted their efforts to adapt and evaluate deep neural networks for this task. Successful examples can be found in Anand et al. (2018), Suong and Jangwoo (2018), Chen et al. (2020) and Ye et al. (2019).

Suong and Jangwoo (2018) proposed a deep neural architecture based on the YOLOv2 (Redmon and Farhadi, 2016) object detection network. The proposed model reduces the computational cost of YOLOv2 and improves precision and recall metrics in the evaluated experiments. Anand et al. (2018) and Chen et al. (2020) have adapted well-known deep neural networks for the pothole detection task. In another reference (Anand et al., 2018), the proposed model consists of a dictionary learning convolution layer after a SqueezeNet (Iandola et al., 2017). The resulting model outperformed other deep models and traditional image processing techniques. Chen et al. (2020) proposed a location-aware network based on the ResNet50 (He et al., 2016). The method outperformed other deep models, achieving 94% accuracy (correctly classified 94% of the images with and without potholes). A new architecture based on convolutional layers is also presented in the literature for pothole detection (Ye et al., 2019). Although the model obtained good results, its performance was only assessed in images that focused on small pavement areas, thus compromising its applicability in real-world scenarios, such as done in another work with real images obtained in Brazilian roadways (Serafim et al., 2022).

It is important to point out that all aforementioned deep models comprise millions of parameters that shall be adjusted during training. As a result, the training procedure would require millions of labeled images to work properly, which tends not to be available in the case for real-world data from pavement management. To overcome such a problem, a popular strategy is the use of transfer learning, where hidden layers of pre-trained neural models are used and only the parameters of the final layers are adjusted for the new task. In the work herein, such a strategy is used, leading to training procedures with reduced computational cost and requiring less training examples. Moreover, as detailed later, data augmentation techniques were applied to obtain better performance even in scenarios where few labeled images are available for training.

3.1. Deep neural networks

Artificial neural networks (ANNs) were proposed as a computing model inspired by the biological brain’s neural networks (Rosenblatt, 1958). Their structure consists basically in a collection of small processing units, called artificial neurons, that are intertwined by weighted connections. In this setting, the artificial neurons would represent the biological neurons, and the weighted connections, like the synapses of the biological brains, the strength of the signal between those neurons. In most ANN models, artificial neurons are organized in layers where neurons of a given layer are connected to the neurons of the next layer. Neural Networks with more than three layers are often referred as Deep Neural Networks (DNN) and the study of DNNs is named Deep Learning (DL).

Deep learning models are powerful mathematical methods that became very popular in recent years due to its remarkable performances in computer vision, automatic translation, text generation and many other challenging tasks (Goodfellow et al., 2016). Among all deep models, Convolutional Neural Networks (CNN) are the current state-of-the-art model for image processing (Xie et al., 2020).

CNNs can be defined as a class of ANN where the neurons have the convolution operator as the main building block. In each layer, the original image is convolved with several kernels (small matrices) in a way that each neuron can be seen as a feature detector in the original image. Such a procedure is widely used in image processing, but in CNN models the elements of the kernels are learned as the result of an optimization procedure. Figure 1 presents a general architecture of a CNN.

The first convolutional layer takes the whole image and convolves it with several kernels. After such operation, a nonlinear function, named activation function, is used to highlight the most important elements of the convoluted images. Then, the resulting images are used as input of the next convolutional layer. The whole procedure is repeated until the last convolutional layer. Usually, a number of different kernels is applied in each layer to extract various features of the images. Often, CNN architectures include pooling layers between convolutional layers. The pooling technique consists of procedures to reduce the dimensionality of the images in each layer, so that the number of parameters, and thus the complexity of the ANN, is reduced.

After the convolutional layers, the CNN includes a fully connected layer that is similar to the classical Multilayer Perceptron (MLP). The fully connected layer takes the processed images and provides a prediction of the target. For instance, if the problem is to detect potholes, the fully connected layer will output the probabilities of having or not a pothole in a given image.

A CNN is therefore a mathematical model with up to millions of parameters that can be used to build a mapping between images and predefined classes. The power of such models relies on the fact that the parameters are adjusted to build a mapping in a set of classified images (training set). The adjustment procedure, named training phase, is modeled as an optimization problem where one will find the parameters that minimize a loss function related to the difference between model predictions and the classes of each image of the training set.

Recently, several CNN architectures have achieved good results in many image recognition problems. VGG16 (Simonyan and Zis-serman, 2015), VGG19 (Simonyan and Zisserman, 2015), ResNet50 (He et al., 2015), ResNet50V2 (He et al., 2016), ResNet101 (He et al., 2015), ResNet101V2 (He et al., 2016), Xception (Chollet, 2017) and InceptionV3 (Szegedy et al., 2015) are some of the most popular CNN architectures. All above-mentioned architectures have many layers with several kernels, thus such models comprise millions of parameters. Therefore, the training phase requires a large set of images to properly adjust the parameters. This fact imposes an important limitation to various applications where a training set of this size is unavailable. To overcome this limitation, transfer learning strategies are widely used (Tan et al., 2018).

3.2. Transfer learning

Transfer learning in DNNs consists of using the knowledge of a neural network trained for a specific task (source task) to help another neural network solve a different but similar task (target task). In its most simple formulation, the transfer learning procedure consists of using the first layers of a trained model and simply adjusting the parameters of the final layers according to the training set of the target task. The success of such a procedure is explained by the fact that the first layers of a DNN work as simple feature detectors that may be useful in a wide range of applications.

In modern CNN architectures it is common that, in addition to the architecture, authors also provide CNNs with trained parameters for challenging image recognition tasks. With those trained models, one can build a new CNN with the same architecture for a different task by training the last layers for the target task while maintaining the parameters of the first layers.

An optional step in transfer learning is fine-tuning (Yang et al., 2018; Kim et al., 2020). It consists of unfreezing the entire trained model, and re-train it for the target task with a low learning rate. This can potentially achieve significant improvements, by incrementally adapting the features extracted by the initial layers to the target task.

3.3. Data augmentation

State-of-the-art deep learning models usually have several millions of tunable parameters, which constitutes a challenging optimization task. The use of the aforementioned transfer learning strategies to obtain pre-trained parameters eases the burden of the learning procedure. However, when few images are available during training, the optimization of the last layers and the fine-tuning step can be hindered. Such results may occur due to overfitting, a state where a learning model adapts itself too closely to the training examples and cannot generalize well for unseen patterns and make predictions.

Data augmentation is a common technique to overcome the above issue (Mikolajczyk and Grochowski, 2018) In the context of image classification, it aims to generate new image examples for training by applying label-preserving transformations such as cropping, rotating, flipping and zooming to the original available data (Perez and Wang, 2017; Shorten and Khosh-goftaar, 2019). Since these transformations are randomly parameterized, a large amount of data can be generated this way. It is worth mentioning that data augmentation strategies were one of the factors that allowed the use of larger CNN models in recent years, which led to important breakthroughs in computer vision (Krizhevsky et al., 2012). Furthermore, data augmentation may induce invariances to simple transformations, a valuable feature to turn the final model more robust (Quiroga et al., 2018).

4.1. Data Generation

To build the image dataset for this research, a continuous visual survey was performed using a vehicle equipped with two cameras, a GPS, an odometer and a computer system to concentrate all information in a single database. The vehicle and related pieces of equipment used in the experiments are shown in Fig. 2. The vehicle, operated at an average speed of 40 km/h, collects yearly images of almost 10,000 km along the Brazilian state of Ceará. While traveling, the vehicle collects georeferenced images of the pavement every 1m and stores them for subsequent in-office analysis for classification of defects for pavement management applications. Broader images every 20m are also captured, but in the performed experiments we have considered only the pavement images.

In possession of the aforementioned image data, the pavement analyst observes them one by one and takes notes on the presence of defects. In this work, we have considered only the annotated potholes. For all the experiments we have labeled a set of 449 images, being 202 of pavements with potholes, and 247 of pavements without potholes. The annotated images were randomly selected among all collected images.

4.2. Computational experimental procedure

In the computational experiments carried out, the dataset is a balanced subset of the original one. Composed of 449 images, 60% of this new dataset was used for training the models, 20% for validation and the remaining 20% for testing. The training set is used to adjust the parameters of the neural network and the validation set is used to build a training stop criterion. The performance is evaluated on the testing set. The pre-trained weights of some models require data to be normalized from (0, 255) to a range (− 1, 1). To increase the size of the training dataset and improve the capability of the models in generalization, three data augmentation techniques were used in each image: random horizontal flip, random rotation with a factor of 0.1 and random zoom with a height factor of 0.15.

Table 1 presents the accuracy metrics for all evaluated DNN models. We have also included the confusion matrices for the best and worst execution of each model. In the table, TP (True Positives) are images without potholes that are correctly classified by the NN. TN (True Negatives) are images with potholes that are correctly classified by the NN. FP (False Positives) are images without potholes that are incorrectly classified and FN (False Negatives) are images with potholes that are incorrectly classified.

As it can be noticed, most models achieved average accuracy near or higher than 80%, with ResNet101V2 achieving a remarkable accuracy of 89.44%. We have also verified that, in most cases, the standard deviations are low, which indicate the consistency of the results along the different runs. Such consistency is also observed in the confusion matrices, where the worst and best results are comparable. It is important to highlight that in practical situations the decision-maker is interested in the current situation of a large pavement section. In that sense, using an automatic pothole recognition method that may misclassify some images is not an important issue.

Table 1

– Accuracy metrics and confusion matrices (for the best and worst executions) for the evaluated DNN models.
Model	Accuracy (%)	Confusion matrices
		Worst case				Best case
		TP	TN	FP	FN	TP	TN	FP	FN
ResNet101V2	89.44 ± 3.23	45	32	4	8	47	38	2	2
VGG19	86.96 ± 3.3	48	25	1	15	44	36	5	4
Xception	81.12 ± 2.79	39	30	10	10	44	31	5	9
InceptionV3	86.29 ± 1.93	39	36	10	4	43	37	6	3
VGG16	88.54 ± 1.65	44	32	5	8	43	37	6	3
ResNet50V2	88.31 ± 1.83	42	35	7	5	45	36	4	4
ResNet101V2	72.13 ± 1.3	34	29	15	11	38	28	11	12
ResNet50V2	68.99 ± 5.8	42	11	7	29	39	28	10	12

To illustrate the effect of the fine-tuning step, Fig. 3 shows an image matrix comprising the misclassified examples after the fine tuning. Such a result was obtained by the best ResNet101V2 execution. The first row (images 1 to 7) comprises images with potholes, while the second row (images 8 to 14) contains images of pavements without potholes. The columns indicate the novel (i.e, after fine-tuning) classifications of images that were misclassified before the fine-tuning step.

By observing Fig. 3, one can see that 11 out of 14 images that were misclassified before fine-tuning were afterwards correctly identified. It is interesting to notice that ResNet101V2 was affected by shades and anomalous illumination conditions before fine-tuning and it shows a more robust behavior to such conditions after the fine-tuning procedure.

In this work the performance of different deep neural networks (8 CNN architectures) for pothole identification in pavement images was evaluated to select the best methodology and improve it for the application as a task for pavement condition assessment. Such automation is important to enhance road maintenance programs, especially in terms of cost savings and resource usage. The analyzed dataset consisted of 449 pavement images with (202) or without (247) potholes. Given the scarce data labeled properly for deep learning application, this work focused also on data augmentation techniques (3 data augmentation strategies for achieving better generalization), applied to only 360 labeled patterns, which is a contribution of this work. Our approach considered pre-trained models, with the initial training procedure consisting of adjusting the parameters of the last two layers of each CNN for 200 epochs. After that, we performed a fine-tuning in all layers for 100 epochs. The results indicated that almost all CNN models achieved accuracy of at least 80% and the ResNet101V2 model had the best overall performance. The fine-tuning procedures had a positive impact on the final result, since it reduced the number of misclassifications (from 14 to 3 of the misclassified test images). Since the results have been obtained with a small set of labeled images, they indicate that the use of transfer learning techniques and data augmentation are promising for the pothole detection task. Overall, the results provided a good accuracy that can effectively translate into best budget management at the cost of relatively little computational effort. For future work, we intend to label more images and verify the performance of CNN networks on such extended databases. Moreover, we consider using a similar methodology to improve other relevant pavement distresses, such as patches and cracks, whose presence is commonly used in the calculation of pavement quality indices.

7. Acknowledgements

The authors acknowledge the support received from the Fundação Cearense de Apoio ao Desenvolvimento Científico e Tecnológico (FUNCAP) through the Cientista Chefe program. This program played a crucial role in the research described in this study, providing research funding, scholarships, supplying vehicles, and acquiring equipment. Additionally, this research wouldn't have been feasible without the financial support of both the National Council for Scientific and Technological Development (CNPq) and the Fundação Cearense de Apoio ao Desenvolvimento Científico e Tecnológico (FUNCAP).

Agarap, A.F., 2019. Deep learning using rectified linear units (relu). arXiv:1803.08375.
Akula, A., Bhatia, Y., Rai, R., Gupta, V., Aggarwal, N., Aukla, A., 2019. Convolutional neural networks based potholes detection using thermal imaging. Journal of King Saud University - Computer and Information Sciences doi:10.1016/j.jksuci.2019.02.004.
Anand, S., Gupta, S., Darbari, V., Kohli, S., 2018. Crack-pot: Autonomous road crack and pothole detection, in: Proceedings of the 2018 Digi-tal Image Computing: Techniques and Applications (DICTA), pp. 1–6. doi:10.1109/DICTA.2018.8615819.
Antunes, William; Dantas, Maria José Pereira. Detecção de anomalias em pavimentos asfálticos utilizando redes neurais para segmentação de imagens. Anais do Simpósio Brasileiro de Pesquisa Operacional. Vol 55 - 161117. São Paulo, 2023.
Babashamsi, P., Md Yusoff, N.I., Ceylan, H., Md Nor, N.G., Salarzadeh Jenatabadi, H., 2016. Evaluation of pavement life cycle cost analysis: Review and analysis. International Journal of Pavement Research and Technology 9, 241 – 254. doi:https://doi.org/10.1016/j.ijprt.2016.08.004.
Branco, Luiz Henrique Castelo; Segantine, Paulo César Lima. "Veículos aéreos não tripulados como sensores remotos auxiliando na detecção de defeitos em pavimentos." Anais do XVII Simpósio Brasileiro de Sensoriamento Remoto. Paraíba, 2015.
Chen, H., Yao, M., Gu, Q., 2020. Pothole detection using location-aware convolutional neural networks. International Jour-nal of Machine Learning and Cybernetics 11, 899–911. URL: https://app.dimensions.ai/details/publication/pub.1124841205, doi:10.1007/s13042-020-01078-7.
Chen, N., Xu, Z., Liu, Z., Chen, Y., Miao, Y., Li, Q., Hou, Y., Wang, L., 2022. Data augmentation and intelligent recognition in pavement texture using a deep learning. IEEE Transactions on Intelligent Transportation Systems 23, 25427–25436. doi:10.1109/TITS.2022.3140586.
Chollet, F., 2017. Xception: Deep learning with depthwise separable convolutions. arXiv:1610.02357.
Chollet, F. et al., 2015. Keras. https://keras.io.
Espíndola, A.C., Freitas, G.T.M., and Nobre Júnior, E.F. "Pothole and patch detection on asphalt pavement using deep convolutional neural network.". Proceedings of the joint XLII Ibero-Latin-American Congress on Computational Methods in Engineering and III Pan-American Congress on Computational Mechanics, ABMEC-IACM CILAMCE-PANACM-2021, Anais de Congresso, 2021.
Goodfellow, I.J., Bengio, Y., Courville, A., 2016. Deep Learning. MIT Press, Cambridge, MA, USA.
Harikrishnan, P., Gopi, V., 2017. Vehicle vibration signal process-ing for road surface monitoring. IEEE Sensors Journal PP, 1–1. doi:10.1109/JSEN.2017.2719865.
He, K., Zhang, X., Ren, S., Sun, J., 2015. Deep residual learning for image recognition. arXiv:1512.03385.
He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778.
He, K., Zhang, X., Ren, S., Sun, J., 2016. Identity mappings in deep residual networks. arXiv:1603.05027.
Hoang, N.D., 2018. An artificial intelligence method for asphalt pavement pothole detection using least squares support vector machine and neural network with steerable filter based feature extraction. Advances in Civil Engineering 2018, 1–12. doi:10.1155/2018/7419058.
Iandola, F.N., Moskewicz, M.W., Ashraf, K., Han, S., Dally, W.J., Keutzer, K., 2017. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and ¡1mb model size. ArXiv abs/1602.07360.
Kaya, H., Gürpınar, F., Salah, A.A., 2017. Video-based emotion recognition in the wild using deep transfer learning and score fusion. Image and Vision Computing 65, 66 – 75. doi:https://doi.org/10.1016/j.imavis.2017.01.012. multimodal Sentiment Analysis and Mining in the Wild Image and Vision Computing.
Kim, R.H., Nomikou, S., Coudray, N., Jour, G., Dawood, Z., Hong, R., Esteva, E., Sakellaropoulos, T., Donnelly, D., Moran, U., Hatzimemos, A., Weber, J.S., Razavian, N., Aifantis, I., Fenyo, D., Snuderl, M., Shapiro, R., Berman, R.S., Osman, I., Tsirigos, A., 2020. A deep learning approach for rapid mutational screening in melanoma. bioRxiv URL: https://www.biorxiv.org/content/early/2020/08/19/610311, doi:10.1101/610311.
Koch, C., Brilakis, I., 2011. Pothole detection in asphalt pavement images. Advanced Engineering Informatics 25, 507 – 515. doi:https://doi.org/10.1016/j.aei.2011.01.002. special Section: Engineering informatics in port operations and logistics.
Koch, C., Jog, G.M., Brilakis, I., 2013. Automated pothole distress assessment using asphalt pavement video data. Journal of Computing in Civil Engineering 27, 370–378. doi:10.1061/(ASCE)CP.1943-5487.0000232.
Krizhevsky, A., Sutskever, I., Hinton, G.E., 2012. Imagenet classification with deep convolutional neural networks, in: Advances in neural information processing systems, pp. 1097–1105.
LeCun, Y., Kavukcuoglu, K., Farabet, C., 2010. Convolutional networks and applications in vision, in: Proceedings of 2010 IEEE international symposium on circuits and systems, IEEE. pp. 253–256.
Lekshmipathy, J., S.N., Velayudhan, S., 2020. Vibration vs. vision: best approach for automated pavement distress detection. International Journal of Pavement Research and Technology PP, 1–9. doi:10.1007/s42947-020-0302-y.
Li, Q., Yao, M., Yao, X., Xu, B., 2009. A real-time 3D scanning system for pavement distortion inspection. Measurement Science and Technology 21, 015702. doi:10.1088/0957-0233/21/1/015702.
Liu, R., Smartz, B.W., Descheneaux, B., 2015. LCCA and environmental lCA for highway pavement selection in Colorado. International Journal of Sustainable Engineering 8, 102–110.
Majidifard, H., Adu-Gyamfi, Y., Buttlar, W.G., 2020. Deep ma-chine learning approach to develop a new asphalt pavement con-dition index. Construction and Building Materials 247, 118513. doi:https://doi.org/10.1016/j.conbuildmat.2020.118513.
Meneses, S., Ferreira, A., 2013. Pavement maintenance programming considering two objectives: maintenance costs and user costs. International Journal of Pavement Engineering 14, 206–221. doi:10.1080/10298436.2012.727994.
Meneses, S., Ferreira, A., Collop, A., 2013. Multi-objective decision-aid tool for pavement management. Proceedings of the Institution of Civil Engineers - Transport 166, 79–94. doi:10.1680/tran.10.00026.
Mikolajczyk, A., Grochowski, M., 2018. Data augmentation for improving deep learning in image classification problem, in: 2018 international interdisciplinary PhD workshop (IIPhDW), IEEE. pp. 117–122.
Omanovic, S., Buza, E., Huseinovic, A., 2013. Pothole detection with image processing and spectral clustering, in: Proceedings of the 2nd International Conference on Information Technology and Computer Networks.
Perez, L., Wang, J., 2017. The effectiveness of data augmentation in image classification using deep learning. arXiv preprint arXiv:1712.04621 .
Quiroga, F., Ronchetti, F., Lanzarini, L., Bariviera, A.F., 2018. Revisiting data augmentation for rotational invariance in convolutional neural networks, in: International Conference on Modelling and Simulation in Management Sciences, Springer. pp. 127–141.
Redmon, J., Farhadi, A., 2016. YOLO9000: better, faster, stronger. CoRR abs/1612.08242. URL: http://arxiv.org/abs/1612.08242, arXiv:1612.08242.
Rosenblatt, F., 1958. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review 65, 386.
Ryu, S.k., Kim, T., Kim, Y.R., 2015. Image-based pothole detection system for its service and road management system. Mathematical Problems in Engineering 2015, 1–10. doi:10.1155/2015/968361.
Serafim, M.O.; A. de Sousa, C.; Almeida, L.; Babadopulos, L.; Gomes, J.S. Automated Detection of Defects and Vertical Signs on Roadways Using Images Produced by Drivers. Journal of Testing and Evaluation. 51. 20220298. 10.1520/JTE20220298. 2022.
Shorten, C., Khoshgoftaar, T.M., 2019. A survey on image data augmentation for deep learning. Journal of Big Data 6, 60.
Simonyan, K., Zisserman, A., 2015. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556.
Suong, L.K., Jangwoo, K., 2018. Detection of potholes using a deep convolutional neural network. Journal of Universal Computer Science 24, 1244–1257.
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z., 2015. Rethinking the inception architecture for computer vision. arXiv:1512.00567.
Tan, C., Sun, F., Kong, T., Zhang, W., Yang, C., Liu, C., 2018. A survey on deep transfer learning, in: International conference on artificial neural networks, Springer. pp. 270–279.
Tribunal de Contas do Estado do Ceará - TCE. Tribunal de Contas do Estado do Ceará: 2020-2023. ISBN 978-65-980276-0-5. Fortaleza, 2023.
Varona, B., Monteserin, A., Teyseyre, A., 2019. A deep learning approach to automatic road surface monitoring and pothole detection. Personal and Ubiquitous Computing doi:10.1007/s00779-019-01234-z.
Xie, Q., Luong, M.T., Hovy, E., Le, Q.V., 2020. Self-training with noisy student improves imagenet classification, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Yang, Z., Yu, W., Liang, P., Guo, H., Xia, L., Zhang, F., Ma, Y., Ma, J., 2018. Deep transfer learning for military object recognition under small training set condition. Neural Computing and Applications doi:10.1007/s00521-018-3468-3.
Ye, W., Jiang, W., Tong, Z., Yuan, D., Xiao, J., 2019. Convolutional neural network for pothole detection in asphalt pavement. Road Materials and Pavement Design 0, 1–17. doi:10.1080/14680629.2019.1615533.
Zhang, Z., Ai, X., Chan, C.K., Dahnoun, N., 2014. An efficient algorithm for pothole detection using stereo vi-sion, in: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 564–568. URL: https://app.dimensions.ai/details/publication/pub.1093762941, doi:10.1109/icassp.2014.6853659.
Zoph, B., Yuret, D., May, J., Knight, K., 2016. Transfer learning for low-resource neural machine translation, in: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, As-sociation for Computational Linguistics, Austin, Texas. pp. 1568–1575. doi:10.18653/v1/D16-1163.

No competing interests reported.

Download PDF

Editorial decision: Revision requested
01 Jul, 2024
Reviews received at journal
21 Jun, 2024
Reviewers agreed at journal
17 Jun, 2024
Reviewers agreed at journal
16 Jun, 2024
Reviews received at journal
14 Jun, 2024
Reviewers agreed at journal
12 Jun, 2024
Reviewers agreed at journal
12 Jun, 2024
Reviewers invited by journal
10 Jun, 2024
Editor assigned by journal
20 May, 2024
Submission checks completed at journal
18 May, 2024
First submitted to journal
17 May, 2024

You are reading this latest preprint version

Automated Detection of Potholes in Pavements using Deep Neural Networks with Fine Tuning and Data Augmentation Techniques

Status:

Version 1

Abstract

Figures

1. Introduction

2. Related Works

3. Theoretical Background

3.1. Deep neural networks

3.2. Transfer learning

3.3. Data augmentation

4. Methodology

4.1. Data Generation

4.2. Computational experimental procedure

5. Results and Discussion

6. Conclusions

Declarations

References

Additional Declarations

Status:

Version 1