EasyChair Preprint

№ 9151

AI Enabled Road Health Monitoring System for

Smart Cities.

Arpit Kumar Bhatt and Susham Biswas

EasyChair preprints are intended for rapid

dissemination of research results and are
integrated with the rest of EasyChair.

October 26, 2022

 AI ENABLED ROAD HEALTH MONITORING SYSTEM FOR
SMART CITIES.
Arpit Kumar Bhatt1,*, Susham Biswas2
1Dept. of Computer Science& Engineering, Rajiv Gandhi Institute of Petroleum Technology Jais, Amethi, Uttar Pradesh, INDIA-
(1arpitkb, 2susham) @rgipt.ac.in

Abstract

Automatic crack identification is a difficult issue, that has been explored for decades, due to the complexity of roadway
networks. Any pavement that has cracks in it’s beginning to lose its surface integrity. Therefore, fracture identification
and management are essential duties because cracks that spread severely damage structures. Manual inspection is
limited to places that can be accessed by people and is based on the expert's prior knowledge. This paper describes the
methods used for evaluation of roads on pothole and crack detection in the highway pavement. The objective of this
research work is to create a precise pavement health smart monitoring system that requires a phone camera-based
monitoring and with the help of profiling of whole cross section of the road. 1 km stretch area in RGIPT campus is
considered for this study. In this research work road pavement is converted in 3D point cloud with the help of mobile
camera. Large pavement area is divided into number of grids to study the nature of terrain and various features of
roads. With the help of point cloud every grid is studied for their coordinate axes as well RGB values to identify the
variation of z-coordinates as we proceed forward. We have tried to bring automation in road health monitoring using
AI-ML. Identification and feature extraction is performed using classification in depth variation as well as
corresponding change in RGB value of generated point cloud. Obtained results are very accurate with the help of
image processing, and results are also verified with actual pavement surface.

Keywords: Pothole Detection, Artificial Intelligence, Image Processing, point cloud, Road Health Monitoring

Introduction

One of the most important components of a person's existence in the modern world is their ability to travel.
The condition of the road surface becomes a major problem because poorly maintained roads are a major cause of
accidents. The greatest option for road repairs may be provided by gathering various real-time data regarding the state
of the roads. Through participation and interaction with people, this information can be gathered. To address this issue,
numerous tactics have been employed. Instead of such expensive devices, we can employ a variety of sensors that are
already built into our cellphones. To locate potholes on the road, utilize a straightforward smart phone that is available
to everyone. Our main objective is to lessen the amount of human work needed to find potholes using a simple, useful,
and affordable technology, which will decrease accidents caused by potholes. To address this issue, numerous tactics
have been employed. Instead of such expensive devices, we can employ a variety of sensors that are already built into
our cellphones. To locate potholes on the road, utilize a straightforward smart phone that is available to everyone. Our
main objective is to reduce the amount of human work needed to find potholes using a simple, useful, and affordable
technology, which will decrease accidents caused by potholes.

Potholes on the road's surface indicate serious structural flaws [1]. They are a result of the road surface's
expansion and contraction as rainwater percolates to the ground surface [2]. The areas of the road that are already
damaged worsen due to tyre vibration. As a result, driving is not possible on the road's surface [3]. When potholes
expand as a result of the movement of subsurface materials brought on by vehicle activity, a vicious circle is formed
[4]. Traffic accidents must be avoided by swiftly repairing road potholes [5]. Due to recent advances in machine
learning, automated road pothole detecting systems are now feasible [6–9]. Convolutional neural networks (CNNs)
have advanced semantic segmentation as a technique for locating potholes in the road [5] and have produced
compelling results. The architecture shown below (Figure 1) gives a clear picture of the work that will be done in this
project. The general outline of how this system operates is that a mobile camera will be used to scan the pavement of
the road before being processed using artificial intelligence and image processing. We will get our output once the
procedure is finished.

Fig. 1 General Architecture of AI Based Road Monitoring System

Unsafe and comfortable travelling is severely hampered by poor road conditions. However, it is important to
identify this condition first. Humans can report road damage to central authorities on their own as the first method of
detection. Even though it involves the most human involvement, that approach has the highest accuracy when
assuming that people are fair. Based on the intensity of use, statistical analysis can be utilized to determine the
approximative damage probabilities of road segments. Surface analysis, a novel technique for monitoring roads that
makes use of various devices such as Ground Penetrating Radar (GPR) and other tools, was developed for the same
objective. Unfortunately, the equipment needed for the above procedure is extremely expensive, which limits its
applicability. Participatory sensing could improve the resolution and scope of the data that is gathered as an alternative.

1. Related Work
In [1] pothole detection is explained using 2D and 3D modelling, using computer vision as well as image analysis.
A vigorous algorithm is used for pothole detection accurately and computationally efficient. The recreated 3D road
surface is used to extract the point clouds of the identified potholes, and according to the experimental findings, the
proposed system's accuracy for pothole detection is about 98.7%, and its total pixel-level accuracy using image
processing is about 99.6%. According to [3] The lack of 3-D industrial images that use alternative methodologies for
use of transportation is the primary driving force behind this article. The dearth of literature comparing the relative
benefits and drawbacks of various imaging techniques for various pavement distress measurement scenarios also
emphasizes the need for this investigation. In order to facilitate a quick initial technology selection and deployment,
this overview will raise awareness of the 3-D imaging techniques that are currently accessible. In [4] Road pothole
detection continues to be primarily done manually by certified inspectors. The experimental findings show that, first,
the transformed disparity in terms of depth, images become more informative; second, our best performing
implementations, AA-UUNet and AA-RTFNet, for road pothole detection; and third, the training set augmentation
technique based on adversarial domain adaptation not only enhances the accuracy of the state-of-the-art semantic
segmentation algorithms. In continuation with this we found in [5] this article describes a pothole identification
technique that is both segmented and estimated based on road disparity maps. In order to generalize perspective
transformation, we first add the stereo rig roll angle into shifting distance computation. After that, semiglobal matching
is effectively used to estimate the road disparities. Then, a disparity map transformation algorithm is used to make it
easier to identify the areas of the damaged road. The transformed disparities are then collected into a set of super
pixels using basic linear iterative clustering. Finding the super pixels whose intensities are below an adaptively set
threshold allows for the detection of the potholes. The suggested approach is put into practice using CUDA on an
NVIDIA RTX 2080 Ti GPU. The testing findings show that the state-of-the-art accuracy and efficiency of our
suggested road pothole identification technique. In [6] there have been various attempts to build a device that can
automatically identify and detect potholes, which may help to enhance survey effectiveness and pavement condition
through early detection and prompt remediation. In this paper, we examine existing pothole detection techniques,
analyze them, and suggest a viable course for building a pothole detection technique that can quickly and effectively
identify potholes. Study [7] suggests in order to detect potholes, this study deployed and tested various deep learning
architectures. The photographs used for training were gathered using a cellphone mounted on the car's windshield,
along with several images acquired from the internet to improve the database's size and variety. To find potholes in
real time, different object identification techniques are used and contrasted. With 81% recall, 85% precision, and
85.39% mean Average Precision, YOLOv4 had the best performance. Processing happened at a rate of 20 frames per
second. Potholes might be located by the system up to 100 meters distant from the camera. By anticipating potholes,
the technology can increase driver safety and enhance self-driving car performance. In [8] we have observed that The
Pothole Patrol system takes advantage of the natural mobility of the participating vehicles, opportunistically collecting
data from vibration and GPS sensors, and processing the data to evaluate road surface conditions. On seven cabs
operating in the Boston region, we have installed P2. We demonstrate that we can extract potholes and other severe
road surface irregularities from accelerometer data using a straightforward machine-learning approach. We have
created a detector that only incorrectly identifies potholes on good road segments 0.2% of the time using a careful
combination of training data and signal properties. Our analysis of data from tens of thousands of taxi trips across vast
distances demonstrates that our technology is capable of accurately identifying several real potholes in and around the
Boston area. Manual inspection of reported potholes reveals that more than 90% of them have road irregularities that
require repair after clustering to further reduce bogus detections. In [9] The most recent research on whether it is
feasible to employ stereovision technology to undertake an extensive survey of pavement quality is presented in this
publication. Multiple high-resolution digital cameras and related modern equipment are included in the current Digital
Highway Data Vehicle (DHDV). It has been demonstrated that a three-dimensional (3D) surface model of pavements
can be created for the entire pavement lane-width, allowing for the extraction of comprehensive condition data. The
report also discusses earlier efforts to create 3D surface representations of pavements. The research concludes that
there are still several difficulties in applying such technology in the field. In [10] author explains about 3D laser
technology to develop point cloud. In order to produce precise digital representations of existent items, 3D laser
scanners use a technique that makes use of reflected laser pulses. One potential use for laser scanning technology in
3D survey is the detection of pavement distresses, such as potholes, widespread utility cuts, or repairs. Traditional
methods of measuring and evaluating pavement distresses are quite laborious and constrictive since they necessitate
lane or even road closures. With the use of a grid-based processing strategy, the study's exact 3D point-cloud points
with elevations were extracted while concentrating on distress features during scanning. According to the experimental
findings, the extent and coverage of distresses can be precisely and automatically assessed to determine the required
quantities of filled materials. This application, which is a first attempt, can help pavement engineers monitor pavement
performance and determine how much money will be needed for repairs. In [11] author suggest that poorly maintained
roads lead to decreased productivity, increased rolling resistance, increased fuel consumption, increased mechanical
wear, and unsafe operating conditions. The methods for repairing pavement and the resources needed to keep the
current road networks in excellent functioning order are a problem for road management organizations. The objective
of this research project is to develop a low-cost smart road health monitoring system that employs smartphone sensors
and camera-based monitoring to pinpoint the road segment that needs repair. In [12] Routine pavement surveys do not
employ information on pavement 3D properties, mostly because of technological issues with hardware acquisition
systems and software algorithms. As a result, most pavement distress surveys are not automated. The application of
the stereovision technology to pavement imaging is a novel suggestion made in the study. The goal is to identify the
road segment that needs maintenance by reconstructing the entire 3D pavement surface from a pair of photos taken
by phone sensors.

Methodology

The pothole detector and road monitoring methods are both included in the Pothole detection subsystem,
which also enables HDL code production. The algorithm is governed by four input variables. Character maps are
written into a RAM by the Processor Behavioral subsystem so they can be used as overlay labels. To determine the
reason of the fault, this design computes the centroid. A text label and a marker are added by the model, which also
places a marker in the middle of the flaw. Using a filter that chooses a polygonal region of interest and turns the
surrounding area black is a useful tactic. Using the filter before edge detection would cause the edges of the mask to
become strong lines, which would create false positives at the detector, therefore the order of operations is important
here. The road directly in front of the car and a trapezoidal part of it up ahead are the only places in the input image
where it might hit a pothole. Figure-2 is general flowchart of this research work which explains from capturing image
as input to the whole processing of the methodology. Captured image is converted into point cloud to complete
analysis.

Fig. 2 General Flowchart of AI Based data processing.

 An approach is developed using image processing for the detection of pothole in transportation engineering.
The procedure of the work is explained with help of below flowchart. Input image is provided with help of normal
camera, after pre-processing and feature extraction in MATLAB required calculations are performed. Thus, based on
point cloud data and extracted features potholes of the road can be identified. Figure – 3 is a flowchart explaining the
procedure of work, which takes place in six sub-parts to collect data, and then after performing all the necessary
calculations out will be obtained.

Fig. 3 Data Processing to develop a pothole detection Technique.

The Pothole HDL subsystem transforms the RGB input image to intensity, then implements bilateral filtering
as well as edge detection. The roadway area is chosen by the subsystem with a trapezoidal mask. Following that, the
design performs a morphological approach and determines the centroid coordinates for every probable pothole. The
detector identifies the largest size pothole in each frame and extricates the center coordinates. The input stream and
the coordinates' time are synchronized using the pixel stream aligner. The Overlay 512x512 subsystems then apply
various channel overlays to the frame to provide a text label and a pothole center marker. Figure-4 clearly explains
about what parameters are considered for the analysis of image using its threshold values in algorithm to get very
accurate outputs to detect existing potholes of the road.

Fig. 4 Flowchart including threshold in Road Health Monitoring System.

 Three input parameters for the subsystem are adjustable while it is operating. The edge detection portion of
the technique is controlled by the gradient intensity parameter, Gradient Threshold. The overlays, which include the
text and the fiducial marker, are colored differently when the Cartoon RGB setting is used. The Area Threshold setting
determines how many marked pixels must be present in the detection window in order for it to be labelled as a pothole.
Linear cracks and other defects that aren't a danger to the road will be seen if this value is too low. Only the biggest
hazards will be seen if it is too high.

Result

The input RGB pixel stream is divided by the model so that a portion of it can carry on toward the overlay
blocks. The detector must first translate from RGB to intensity. Since the image is the RGB's input data type, the RGB
to Intensity block automatically chooses uint8 for the output data type. High visual frequency of images and smaller
road undulations are to be reduced by the algorithm. While there are many ways to achieve this, using a bilateral filter
has the advantage of maintaining edges while minimizing noise and smaller areas. Figure-5 represents pixel values of
road surface which is separated by deterioration of the pavement. Thus, based on the pixel value we can identify the
potholes, as pixel values reduces as pothole present at a particular portion of the road.

Fig. 5 Pothole Extraction based on pixel value.

Utilizing a smartphone for data collection, images are then transformed into 3D point clouds using Agisoft.
Each stratum of the road has been examined with the use of a 3D point cloud. We applied our suggested model as it
is presented in this study. Road surface information is collected in July 2022. Figure -6 depicts the road's initial
condition before it deteriorated over time from use.
 Fig. 6 Actual image of road condition.

A histogram plotted from the collected image in MATLAB is another way for identifying potholes in smart
transportation systems. Both the RGB value and the pixel value are used to plot this histogram. The plotted histogram
value shows that potholes are located where the color of the road transitions into a grey tone. Figure-6 clearly
represents the higher peaks at certain places where deterioration of road pavements has been taken place. Histograms
are plotted on behalf of the pixel values of the input images.

Fig. 7 Histogram plot of road pavement for pothole detection.

 MATLAB 11th Gen Intel(R) Core (TM) i5-11500 is used. programme, a distributed pavement plot with more
than one million points was produced. To make a dispersed plot, surface coordinates and their RGB values are used.
The photo clearly shows a pothole. It is possible to see a visible pothole along its depth in a MATLAB pavement
scatter plot. The road directly in front of the car and a portion of the road ahead are the only places in the input image
where it might encounter a pothole. The camera position at the time of image capture determines the precise
coordinates.

Fig. 8 General Architecture of AI Based Road Monitoring System

To examine the nature of the topography and different road parameters, a large pavement area is divided into
a number of grids. As we move on, each grid is examined using the point cloud for its coordinate axes and RGB values
to determine how the z-coordinates vary. Entire area is devided into more than two hundred grids and then analysis of
each grid has been taken places. This study provide very precise knowledge about the terrain nature of the road
pavement. Undulated portion, uneven nature of the topography, slopy nature of the ground and down slope of the road
has been identified, which is very crucial for our study to predict the nature of the ground which will take place.
Figure-9 is graphical respresentation of grid formartion of the surface using MATLAB. Theshold value for this study
is decided based on the mean value of each grid. A certain plane is identified below the level surface of the road as
pothole. Yellow portion of the graphical representation represents top most portion the road, whereas blue portion
represents lower portion or pothole portion of the road. The RGB original pixel stream and the streaming data from
the detector are sent to the overlay subsystems by the pixel stream aligner block. Without having any knowledge of
the latency of those blocks, the aligner makes up for the processing delay that all the earlier steps in the detection
algorithm added.

Fig. 9 Grid formation of the pavement.

The axes x, y, z and RGB value representation of the 3D point cloud created from the photos. Potholes can
be located in smart cities by observing variations on the Z-axis. The change of the Z axis will be minimal or nonexistent
at smooth pavement surfaces, but it will increase where there are potholes. As a result, potholes can be located by
looking at the Z axis fluctuation. Only when the threshold value falls below the Z-axis average can potholes be
detected. The accelerometer dataset's captured spikes are normalised at their crests. Table 1 details the study's mean z
value as well as the depth at which a surface deterioration qualifies as a pothole. Four different places are considered
for study which are tabled below in table-1 with its mean as well as threshold value to identify pothole of the road
surface.

Sr. Mean Z value in Value of Z axis (Threshold) to

No. mm identify Pothole in mm
1. Rajiv Gandhi Plaza 73.0619 77.00
2. Gate - 1 75.1315 77.00
3. Academic Block -1 79.3726 77.00
4. Gate -2 83.8706 77.00

Table -1 Pothole identification at different places of the study for Z axis value.
 Fig. 10 General Architecture of AI Based Road Monitoring System

Thus, after creating 3D point cloud, we have obtained more than one million points, with the help of these
points scattered plot of payment surface is plotted in MATAB along its width. In this way, we have analyzed the
performance of the presented pothole detection algorithm technique qualitatively & quantitatively in both manners.
The proposed research algorithm is executed in MATLAB 11th Gen Intel(R) Core (TM) i5-11500. Figure -11
presents the scattered plot of road surface, a data gap is seen in some portion of the plot, which is pothole of the
road. 3D point cloud provides co-ordinates of every point of that location. Such places it lacks in providing a clear
plot there may be a depreciation of hump in the pavement surface.

Fig. 11 General Architecture of AI Based Road Monitoring System.

 2. Conclusion
Based on an evaluation of the current road conditions, the anomalous road conditions can be located and saved in
the traffic center. In order to control their driving behaviors and increase care, comfort, and efficiency, drivers can
employ early warning signals to get information about adjacent roads from other vehicles. It is possible to make the
roads safer and easier for the people to travel from one place to another. In the modern world, problems with road
surfaces are a cause of public concern, because safety of people is directly linked to this. The government payouts
millions of dollars for repairing and maintenance of the roads. The health and safety of passengers are the prime
priority of the concerned authorities., that is why the safety and consistency of the vehicles themselves, depend on a
well-maintained road network. the user with terms. An expensive and time-consuming operation, identification of
areas that need repairs or reworking is required for pavement maintenance, according to study. This study developed
an Artificial Intelligence and image processing-based cloud application that can help with pavement management to
speed up and reduce the cost of the procedure. The findings of result demonstrate a very high degree of precision and
accuracy in real-time quality detection of the road parameters. The aim of this work was to investigate an artificial
intelligence and image processing application for detection such as pavement condition, monitoring, and user
screening. This developed system provides 80-85% accurate result as it is verified physically actual condition of road
after completing all the required computations.

References
[1] Rui Fan et al., “Pothole detection based on disparity transformation and road surface modeling,” IEEE
Transactions on Image Processing, vol. 29, pp. 897– 908, 2019.
[2] John S Miller et al., “Distress identification manual for the long-term pavement performance program,”
Tech. Rep., 2003.
[3] Senthan Mathavan et al., “A review of three-dimensional imaging technologies for pavement distress
detection and measurements,” IEEE TITS, 2015.
[4] Rui Fan et al., “We learn better road pothole detection: from attention aggregation to adversarial domain
adaptation,” in European Conference on Computer Vision. Springer, 2020, pp. 285–300.
[5] Rui Fan et al., “Rethinking Road surface 3d reconstruction and pothole detection: From perspective
transformation to disparity map segmentation,” IEEE Transactions on Cybernetics, 2021.
[6] B. X. Yu and X. Yu, "Vibration-Based System for Pavement Condition Evaluation," in Applications of
Advanced Technology in Transportation, 2006.
[7] K. D. Zoysa, G. P. Seneviratne, W. W. A. T. Shihan and C. Keppitiyagama, "A Public Transport System Based
Sensor Network for Road Surface Condition Monitoring," in SIGCOMM07: ACM SIGCOMM 2007
Conference, Kyoto, 2007.
[8] L. G. B. H. R. N. S. M. H. B. Jakob Eriksson, "The Pothole Patrol: Using a Mobile Sensor Network for Road
Surface Monitoring," in Mobisys08: The 6th International Conference on Mobile Systems, Applications,
and Services, Breckenridge, 2008.
[9] K. C. P. Wang, "Challenges and Feasibility for Comprehensive Automated Survey of Pavement Conditions,"
in Eighth International Conference on Applications of Advanced Technologies in Transportation
Engineering (AATTE), Beijing, 2004.
[10] Chowdhury, Mashrur, and Adel W. Sadek. "Advantages and limitations of artificial intelligence." Artificial
intelligence applications to critical transportation issues 6.3, pp.360-375, (2012).
[11] Bhatt, A., Bharadwaj, S., Sharma, V. B., Dubey, R., & Biswas, S. AN OVERVIEW OF ROAD HEALTH
MONITORING SYSTEM FOR RIGID PAVEMENT BY TERRESTRIAL LASER SCANNER. The International Archives
of Photogrammetry, Remote Sensing and Spatial Information Sciences, 43, pp.173-180, (2022).
[12] K. T. Chang, J. R. Chang and J. K. Liu, "Detection of Pavement Distresses Using 3D Laser Scanning
Technology," in International Conference on Computing in Civil Engineering 2005, Cancun, 2005.
[13] Z. Hou, K. C. Wang and W. Gong, "Experimentation of 3d Pavement Imaging Through Stereovision," in
International Conference on Transportation Engineering 2007, Chengdu, 2007.
[14] C. Koch and I. Brilakis, "Pothole detection in asphalt pavement images," Advanced Engineering
Informatics, vol. 25, no. 3, pp. 507-515, 2011.
[15] G. M. Jog, C. Koch, M. Golparvar-Fard and I. Brilakis, "Pothole Properties Measurement through Visual 2D
Recognition and 3D Reconstruction," in International Conference on Computing in Civil Engineering,
Florida, 2012.
[16] L. Huidrom, L. K. Das and S. Sud, "Method for automated assessment of potholes, cracks and patches from
road surface video clips," Procedia - Social and Behavioral Sciences, vol. 104, pp. 312-321, 2013.
[17] Aparna, Y. Bhatia, R. Rai, V. Gupta, N. Aggarwal and A. Akula, "Convolutional neural networks-based
potholes detection using thermal imaging," Journal of King Saud University – Computer and Information
Sciences, 2019.
[18] Bianchini, P. Bandini and D. W. Smith, "Interrater Reliability of Manual Pavement Distress Evaluations,"
Journal of Transportation Engineering, vol. 136, no. 2, pp. 165-172, 2010.
[19] O. Mendoza, P. Melin and G. Licea, "A New Method for Edge Detection in Image Processing using Interval
Type-2 Fuzzy Logic," in 2007 IEEE International Conference on Granular Computing, California, 2007.
[20] O. Mendoza, P. Melin and G. Licea, "A New Method for Edge Detection in Image Processing using Interval
Type-2 Fuzzy Logic," in 2007 IEEE International Conference on Granular Computing, California, 2007.
[21] Bhat, P. Narkar, D. Shetty and D. Vyas, "Detection of Potholes using Image Processing Techniques," IOSR
Journal of Engineering, vol. 2, pp. 52-56, 2018.
[22] M. Muslim, D. Sulistyaningrum and B. Setiyono, "Detection and counting potholes using morphological
method from road video," AIP Conference Proceedings, vol. 2242, no. 1, pp. 3-11, 2020.
[23] Chowdhury, Mashrur, and Adel W. Sadek. "Advantages and limitations of artificial intelligence." Artificial
intelligence applications to critical transportation issues 6.3, pp.360-375, (2012).
[24] I. Poola, How Artificial intelligence is impacting real life every day, Int. J. Adv. Res. Dev. 2 (10) (2017) 96–
100. Retrieved October 12, 2019.