See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/355821299

Structural Crack Detection from Benchmark Data Sets Using Pruned Fully
Convolutional Networks

Article in Journal of Structural Engineering · November 2021

DOI: 10.1061/(ASCE)ST.1943-541X.0003140

CITATIONS READS
0 78

6 authors, including:

Jin Tao Si Yuan Ma

Zhejiang University Zhejiang University
10 PUBLICATIONS 154 CITATIONS 1 PUBLICATION 0 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Si Yuan Ma on 01 December 2021.

The user has requested enhancement of the downloaded file.

 Data Paper

Structural Crack Detection from Benchmark Data Sets

Using Pruned Fully Convolutional Networks
X. W. Ye 1; T. Jin 2; Z. X. Li 3; S. Y. Ma 4; Y. Ding 5; and Y. H. Ou 6

Abstract: Crack inspection is a crucial but labor-intensive work of maintenance for in-service bridges. Recently, the development of fully
convolutional network (FCN) provides pixel-wise semantic segmentation, which is promising as a means of automatic crack detection.
Downloaded from ascelibrary.org by Zhejiang University on 11/30/21. Copyright ASCE. For personal use only; all rights reserved.

However, the demand for numerous training images with pixel-wise labels poses challenges. In this study, a benchmark data set called
a bridge crack library (BCL) containing 11,000 pixel-wise labeled images with 256 × 256 resolution was established, which has 5,769
nonsteel crack images, 2,036 steel crack images, 3,195 noise images, and their labels. It is aimed at crack detection on multiple structural
materials including masonry, concrete, and steel. The raw images were collected by multiple cameras from more than 50 in-service bridges
during a period of 2 years. Various crack images with numerous crack forms and noise motifs in different scenarios were collected. Quality
control measures were carried out during the raw image collection, subimage cropping, and subimage annotation steps. The established BCL
was used to train three deep neural networks (DNNs) for applicability validation. The results indicate that the BCL could be applied to
effectively train DNNs for crack detection and serve as a benchmark data set for performance evaluation of DNN models. DOI:
10.1061/(ASCE)ST.1943-541X.0003140. © 2021 American Society of Civil Engineers.
Author keywords: Structural health monitoring; Structural crack detection; Image data set; Deep learning; Fully convolutional networks.

Introduction Realizing the importance of crack detection for bridges and the
drawbacks of current crack detection measures, both the industry
Bridges are one of the most crucial infrastructures that play an im- and academia have devoted efforts to automatic crack detection re-
portant role in public transportation systems. However, the harsh search (Xu et al. 2019). Recently, the development of deep
field environment poses great challenges to the durability of in- learning-based approaches have brought powerful tools to crack
service bridge structures (Ni et al. 2012; Ye et al. 2013). During detection on images (Hinton and Salakhutdinov 2006; LeCun
their whole service life cycle, the bridges will suffer from multiple et al. 2015; Ye et al. 2019a, b). A variety of deep neural network
negative impacts such as vehicle load, wind load, temperature (DNN) architectures have been developed, and among them fully
change, and acid rain. Consequently, there might be all kinds of convolutional network (FCN) stands out for its capacity in con-
damage induced by the harsh environment, and cracking is seen ducting pixel-wise semantic segmentation (Ronneberger et al.
as one of the most pervasive types of damage. Furthermore, cracks 2015). This characteristic is quite suitable for crack detection, be-
might lead to further damage, including stress concentration, steel- cause the geometric parameters of cracks such as length, width, and
bar corrosion, and spalling, which would gradually reduce struc- direction are important references for risk assessment decisions re-
tural strength and service time (Ni et al. 2010; Ye et al. 2012). garding maintenance.
Therefore, crack detection is an important task in the regular However, the utilization of DNNs demands large numbers of
work of bridge inspection (Spencer et al. 2019). However, the training data sets, and in this case, large numbers of bridge crack
traditional measures of bridge crack detection via on-site manual images are required. Unlike the pavement of roads, which has flat
inspection are labor-intensive, time-consuming, and sometimes and continuous surfaces and guarantees convenient accessibility,
hazardous. bridges are complicated structures. While it is possible to obtain
road crack images by vehicle-mounted cameras, it is difficult
1
to access the surface of bridge components. Manual inspection
Professor, Dept. of Civil Engineering, Zhejiang Univ., Hangzhou with ladder trucks is inevitable, which makes the process labor-
310058, China (corresponding author). ORCID: https://orcid.org/0000
intensive, expensive, and time-consuming. Moreover, FCN re-
-0003-0012-5842. Email: [email protected]
2
Ph.D. Candidate, Dept. of Civil Engineering, Zhejiang Univ., quires pixel-wise labels, which increases the difficulty of building
Hangzhou 310058, China. Email: [email protected] a data set. Some groups have adopted a transfer learning strategy or
3
Master’s Candidate, Dept. of Civil Engineering, Zhejiang Univ., a data augmentation method to reduce the necessity for the amount
Hangzhou 310058, China. Email: [email protected] of bridge crack images (Spencer et al. 2019). Nonetheless, a bridge
4
Ph.D. Candidate, Dept. of Civil Engineering, Zhejiang Univ., crack data set with adequate numbers of pixel-wise labeled crack
Hangzhou 310058, China. Email: [email protected] images and a rich diversity of on-site scenarios will serve as
5
Ph.D. Candidate, Dept. of Civil Engineering, Zhejiang Univ., an important resource for deep learning-based crack detection
Hangzhou 310058, China. Email: [email protected] research.
6
B.E. Candidate, Dept. of Civil Engineering, Zhejiang Univ., Hangzhou
This study aims to provide a basis for deep learning–based
310058, China. Email: [email protected]
Note. This manuscript was submitted on November 16, 2020; ap-
bridge crack detection research by the establishment of a pixel-wise
proved on June 2, 2021; published online on August 27, 2021. Discussion labeled image data set. Hence, an image data set called a bridge
period open until January 27, 2022; separate discussions must be sub- crack library (BCL) that contains 11,000 pixel-wise labeled images
mitted for individual papers. This paper is part of the Journal of Struc- was established. The data set is based on raw images collected from
tural Engineering, © ASCE, ISSN 0733-9445. inspection of in-service bridges during a span of 2 years. The raw

© ASCE 04721008-1 J. Struct. Eng.

J. Struct. Eng., 2021, 147(11): 04721008

 images were collected and examined by inspection engineers, collected 50 raw images of bridges and developed a pixel-wise la-
and 1,400 raw images of different structural components with dif- beled data set with multiple resolutions by data augmentation ap-
ferent materials were selected. These selected images were cropped proaches. Zhang et al. (2020a) collected 2,206 raw images with
to a resolution of 256 × 256 and elaborately labeled manually to different resolutions to generate a 2,206-image data set annotated
build the BCL. Note that the 11,000 subimages were directly with a bounding box. Dorafshan and Azari (2020) developed an
cropped and not processed by any data augmentation method to annotated impact echo data set containing 2016 impact echo data
maintain the originality of the raw crack images. Crack detection for the detection of the subsurface defects on bridge decks. Mei and
experiments using three different DNNs and the BCL were carried Gul (2020) compared nine crack image data sets with emphases on
out to validate the effectiveness and applicability of the data set in the numbers, resolutions, and types of annotation of the crack im-
model training. The results indicate that the BCL could be used to ages. Meanwhile, Majidifard et al. (2020) summarized nine crack
train robust DNNs for crack detection of in-service bridges. image data sets with emphases on the numbers, sources, and photo
angles of the crack images.
In spite of the invested efforts, the shortage of crack images re-
Review of Related Works mains one of the most common issues challenging an adequate
Downloaded from ascelibrary.org by Zhejiang University on 11/30/21. Copyright ASCE. For personal use only; all rights reserved.

training of DNNs. Hence, scholars have proposed multiple ap-

Regarding the detection results of image-based tasks, there are proaches to compensate for the shortage of crack images. Among
mainly two kinds of DNN architecture adopted by researchers, them, transfer learning and data augmentation are often applied in
i.e., the convolutional neural network (CNN) (Cha et al. 2017; practice (Dung and Anh 2019). When a transfer learning strategy is
Tong et al. 2017; Hoang et al. 2018) and the FCN (Huang et al. adopted, the DNNs will be trained by a large amount of structural
2018; Alipour et al. 2019; Ye et al. 2019a). The CNN can detect images first and then fine-tuned by a small amount of crack images.
cracks in a patch or bounding box manner, while the FCN can detect VGG (Gopalakrishnan et al. 2017; Dung and Anh 2019) and
crack regions without nearby accessory areas (Ju et al. 2020; Li et al. AlexNet (Dorafshan et al. 2018a) are often adopted as the base
2020). As for the detailed DNN models, multiple kinds of model models of transfer learning. The strategy of data augmentation ex-
architecture have been applied to crack detection research, such pands the data set by several image processing methods such as
as VGG (Yang et al. 2018), SegNet (Sajedi and Liang 2019; image stretching, image flip, rotation, and the adding of salt and
Zhang et al. 2019b), Faster R-CNN (Cha et al. 2018), Mask R- pepper noise (Alipour et al. 2019). The previous strategies might
CNN (Kim and Cho 2019), ResNet (Song et al. 2019), MobileNet mitigate the shortage of crack images; however, the high dimen-
(Song et al. 2019), GoogLeNet (Ni et al. 2019), Inception sional abstract features related to crack or noise motifs in crack im-
(Daneshgaran et al. 2019; Feng et al. 2019), RetinaNet (Ochoa- ages from the real situation would not be obtained or imitated
Ruiz et al. 2020), and YOLO (Du et al. 2020; Deng et al. 2021; completely.
Majidifard et al. 2020; Song et al. 2020; Zhang et al. 2020a). Be- Overall, there exist many crack image data sets and compensat-
cause geometric parameters are essential for the assessment of crack ing measures for the improvement of the image amount. The cur-
risks, image processing techniques are often applied to extract crucial rent data sets have the following features: (1) the majority of these
crack parameters such as length, width, and direction (Kim and Cho crack data sets were built on the road cracks on the surface of the
2018). Additionally, machine learning algorithms are used as well to asphalt pavement, unlike the bridge cracks on the surface of ma-
filter noise to improve accuracy and robustness (Yeum et al. 2019). sonry, concrete, or steel structures; (2) as for some bridge crack data
To utilize DNNs for crack detection, the establishment of sets, the hand-crafted data augmentation methods were used to re-
an image data set is one of the most important preconditions lease the shortage of image numbers but they could not contribute
(Spencer et al. 2019). The acquisition of a vast amount of raw crack greatly to the diversity of the data set; and (3) the majority of the
images is critical to building the image data set. To access the sur- annotation types were in a patch or bounding box manner, not in a
face of high buildings, unmanned aerial vehicles (UAV) were uti- pixel-wise manner.
lized in several studies (Kang and Cha 2018; Kim and Cho 2018).
To collect images of cracks in road pavement, vehicle-mounted
cameras were adopted by many groups (Maeda et al. 2018; Tong Establishment of Crack Image Data Set
et al. 2018; Zhang et al. 2018a, Bang et al. 2019; Park et al. 2019).
To collect images of cracks in tunnels, railway-based robots
Collection of Raw Images
were leveraged during the collection process (Xue and Li 2018;
Gao et al. 2019; Li et al. 2019). Besides, it was also possible to Deep learning is a kind of representational learning that could make
collect online images with appropriate processes for data set estab- it possible for a network architecture to learn the features from raw
lishment (Gao and Mosalam 2020). Image collection is secure data for the task of recognition (LeCun et al. 2015). However, due
and efficient when the access to the surfaces of target structures to the numerous learnable parameters waiting for modification in a
is easily available, such as the way vehicle-mounted cameras DNN architecture, there is a great demand for a vast amount of
collect images of cracks in pavement. training data. Therefore, to train a crack detection network, huge
Based on the collected raw images of cracks, multiple crack numbers of crack images are necessary. In addition, the cracks
image data sets have been established by the aforementioned ap- in real situations might appear in multiple forms, such as different
proaches (Ryu et al. 2020). In terms of the data sets focusing on lengths, widths, and directions. Moreover, there are plenty of noise
bridge defects, Kim and Cho (2018) developed a data set contain- motifs near the cracks on structural surfaces including the water
ing 42,000 patchwise annotated images with 227 × 227 resolution. spots, plants, stains, markers, and welding lines. Thus, a satisfac-
Dung and Anh (2019) collected 458 raw images and established a tory data set of crack images should have a large number of those
data set consisting of 40,000 patchwise annotated subimages and images containing diverse forms of cracks and multiple noise
600 pixel-wise annotated subimages. Dorafshan et al. (2018b) motifs.
collected 230 raw images of concrete bridges and developed a In order to establish this kind of data set, inspection of more
patchwise labeled data set that consisted of 56,000 subimages with than 50 in-service bridges located in Hangzhou, China, including
256 × 256 resolution for crack detection. Alipour et al. (2019) individual bridges and viaducts was conducted during the period

© ASCE 04721008-2 J. Struct. Eng.

J. Struct. Eng., 2021, 147(11): 04721008

 Table 1. Parameters of photographing devices
Device brand Model Censor type Censor size (mm) Resolution (pixels)
Canon EOS 77D CMOS 22.3 × 14.9 6,000 × 4,000
Canon EOS 800D CMOS 22.3 × 14.9 6,000 × 4,000
Nikon D5100 CMOS 23.6 × 15.6 4,928 × 3,264
Sony DSC-HX60 CMOS 7.7 5,184 × 3,888
Vivo X23 CMOS 6.9 4,032 × 3,204

2018–2020. The collection process was carried out by multiple images were taken by different photographing devices. The main
two-person teams, i.e., one of them was a full-time inspection en- parameters of the devices concerning the quality of images are
gineer and the other was a member of this study. All of the inspec- shown in Table 1. As shown in Table 1, a mobile phone, a
tion engineers involved in this process of collection have more than consumer-grade camera, and a digital single lens reflex were used
Downloaded from ascelibrary.org by Zhejiang University on 11/30/21. Copyright ASCE. For personal use only; all rights reserved.

3 years of inspection experience and all of the team members of this for photographing, and these covered the possible devices used in
study have a degree in civil engineering. daily inspection activities.
The period of photographing covered four seasons of a year, The bridges inspected include foot bridges, municipal bridges,
which provided diverse conditions of weather and illumination. and viaduct bridges. The forms of bridges consist of arch bridges,
The inspection was mainly conducted between 9 a.m. and 5 p.m. simply supported beam bridges, and continuous girder bridges, as
on sunny or cloudy days. The widths of cracks on the surface of in- illustrated in Fig. 1. Identical to the inspection requirement, the
service bridges are tiny, typically within millimeters. In order to structural components of the raw images contained the guardrail,
clearly record the forms of the cracks, the photographing distance the pavement, the beam, the abutment, and the pier. Typical raw
ranged from 0.5 to 8.0 m, according to different scenarios. To ac- images of structural components containing cracks are shown in
cess the surface of structural components in different altitudes, the Fig. 2. Note that plenty of data sets focusing on crack images
manual photographing was assisted by extension rods, ladder of asphalt pavement already exist. Therefore, the images taken
trucks, and boats. Owing to the existence of multiple teams, the in the inspection focused on concrete and masonry materials.

Fig. 1. Typical structural forms of inspected bridges: (a) arch bridge; (b) simply supported beam; (c) three-span continuous bridge; and (d) multispan
continuous bridge. (Images by authors.)

© ASCE 04721008-3 J. Struct. Eng.

J. Struct. Eng., 2021, 147(11): 04721008

Downloaded from ascelibrary.org by Zhejiang University on 11/30/21. Copyright ASCE. For personal use only; all rights reserved.

Fig. 2. Typical structural components of inspected bridges: (a) guardrail; (b) pavement; (c) beam; (d) abutment; and (e) pier.

Such diversity of bridge structures and components aims at having water spots, shadows, markers, welding lines, stains, and corrosion.
a qualified representation of the vast diversity of in-service bridges. Overall, 1,400 raw images were collected from the bridge inspection
The rich diversity could benefit the robustness of the trained DNNs, for the establishment of the bridge crack library. Among them, 1,000
improve their capacity for generalization and reduce issue of nonsteel crack images and 180 steel crack images were cropped and
overfitting. labeled in a pixel-wise manner. The rest of the 200 nonsteel crack
Based on the inspection of bridges carried out by the dedicated images and 20 steel crack images were preserved for image tests.
teams, as many as 1,500 images focusing on bridge cracks were
obtained. The process of selecting the images was carried out man-
ually to eliminate the raw images that were dim, underexposed, or Image Cropping
overexposed. Based on the laborious process of collection and se- The training images are used for the DNNs to learn and extract the
lection, 1,200 raw crack images of different bridge structural com- abstract features in order to separate the desired targets from noise
ponents made of masonry and concrete were selected, as illustrated motifs. Thus, the training images should reflect the complicated
by Fig. 2. The cameras and the corresponding recorded structural forms of the target and noise motifs in the real world. In tasks
components are listed in Table 2. of crack detection, the cracks are narrow and slender patterns while
Corresponding to the cameras used, the resolutions of the 1,200 the noise motifs, such as water spots, welding lines or shadows, are
nonsteel crack images are 4,032 × 3,204; 5,184 × 3,888; 4,928 × much wider. Therefore, the size of the training images should be
3,264; and 6,000 × 4,000 respectively. In order to cover the raw large enough to cover complete forms of crack regions. Otherwise,
crack images of steel bridges, 200 raw images provided by the first the cropped cracks will only take up a small portion of the original
International Project Competition for Structural Health Monitoring regions that fail to depict the forms of the cracks and the noise mo-
were also added. The raw crack images of steel bridges were also tifs. According to the sizes of the raw images, the cracks, and noise
collected by bridge inspectors and the resolutions are 4,928 × 3,264 motifs, the size of the cropped images was designated to have a
and 5,152 × 3,864, respectively. The collected raw images are resolution of 256 × 256 pixels in this study. Such an arrangement
shown in Fig. 3. It can be seen that these images not only contain makes it convenient to generate smaller training images such as
cracks, but also contain multiple noise motifs, such as scratches, 64 × 64 resolution or 32 × 32 resolution if necessary, while it
is difficult to do the opposite. A program run on Matlab was
developed to crop the picked crack subimage by clicking on the
crack area on the screen with a mouse, as illustrated in Fig. 4. Five
Table 2. Cameras and corresponding structural components
members of this study were in charge of the cropping process;
Component Guardrail Pavement Beam Abutment Pier Summary among them, two members did the cropping operation and three
Canon 77D 26 32 76 214 63 411 other members examined the quality of the cropped subimages
Canon 800D 32 1 124 7 26 190 independently. In this way, a subimage was cropped by one
Nikon 221 132 4 127 3 487 member and examined by three members, respectively. Only those
Sony 4 18 0 66 2 90 subimages that were accepted by all three independent examiners
Vivo 6 9 1 6 0 22 were kept for labeling, and the quality examination process is
Summary 289 192 205 420 94 1,200
shown in Fig. 5.

© ASCE 04721008-4 J. Struct. Eng.

J. Struct. Eng., 2021, 147(11): 04721008

Downloaded from ascelibrary.org by Zhejiang University on 11/30/21. Copyright ASCE. For personal use only; all rights reserved.

Fig. 3. Collected raw images: (a) raw images from inspection; and (b) raw images from IPC-SHM.

accurate and precise labeling of the crack edges highly significant.

While in the semantic segmentation for objects with a large area on
the images, such as the detection of vehicles, which might take up
30% or more of the whole area of the image, the errors in anno-
tation along the edges are less influential.
It is inevitable that the manual labeling processes carried out by
different people or the same people at different moments are em-
pirical and inhomogeneous. Therefore, measures should be taken to
reduce the incorrect labeling at the edge of the cracks to avoid false
learning samples. Given this consideration, a labeling kit, but not a
mouse, was used to depict the area of the crack in order to combat
the inadvertent errors in labeling, as shown in Fig. 7. It consists of a
pen and a board that can be skillfully utilized to depict the forms of
the cracks.
Photoshop (PS) was utilized for labeling and the labeling kit was
connected to PS as the input device for depicting the crack accu-
rately. The process of labeling and examination is shown in Fig. 8
and the state of the image during that process is shown in Fig. 9.
When the cropped subimages were labeled, they were zoomed-in
Fig. 4. Cropping process. on the screen of the computer display. Then, the pen was used to
depict the crack area as accurately and completely as possible, es-
pecially along the edges of the cracks. Similar to the quality control
Image Labeling for the process of cropping the subimages, the labeling operation
was carried out by two people and checked by three independent
On the crack images, the crack area occupies merely a tiny portion examiners. Labels returned by any of the examiners were checked
of the whole area of the image. As illustrated in Fig. 6(c), the crack and relabeled. In terms of the time dedicated to labeling, most of the
only occupies 2.83% of the whole area of the image. Furthermore, time was taken up by the labeling process, the examination process,
the crack is so narrow that the width takes merely several pixels, as and the relabeling process. To label a subimage with cracks that had
shown in Fig. 6(a). Consequently, the imbalanced distribution of a resolution of 256 × 256 pixels, the labeling process shown in
cracks and backgrounds will lead to the result that the deep Fig. 10 took around 2 min. The examination time took about
learning-based crack detection performance is quite sensitive to 30 s and the relabeling time for each subimage took approximately
errors in annotation of crack areas. The narrow width makes the 1 min. Thus, the total time spent for one labeling of the subimage

Fig. 5. Quality examination process of subimages.

© ASCE 04721008-5 J. Struct. Eng.

J. Struct. Eng., 2021, 147(11): 04721008

Downloaded from ascelibrary.org by Zhejiang University on 11/30/21. Copyright ASCE. For personal use only; all rights reserved.

Fig. 6. Proportion of the area of a crack: (a) amplified crack area; (b) training image; and (c) crack area.

with cracks, if well labeled initially, was about 2.5 min; otherwise,
it took at least 4 min. The labeling of noncrack images was much
easier as the entire area was labeled as background.
Finally, based on the aforementioned considerations, there are
5,769 images of nonsteel cracks obtained from raw images of
bridge inspections and 2,036 images of steel cracks obtained from
raw images of IPC-SHM. In order to compensate for the lack of
steel-related images and to improve the robustness of the trained
DNNs, 25,000 noncrack images of steel structures containing noise
motifs were cropped as well. Among them, 3,195 noise images that
contain crack-like motifs were selected and kept to enrich the data
set for the improvement of the robustness against noise motifs in
the real-world environments. The noise images kept and those
abandoned are shown in Fig. 10.
After the previous processes, there were 11,000 labeled images
with 256 × 256 pixel resolution selected for the establishment of
the BCL, and this number includes 5,769 nonsteel crack images,
2,036 steel crack images, 3,195 noise images, and their labels,
Fig. 7. Labeling kit.
as shown in Fig. 11.

Fig. 8. Process of labeling and examination.

Fig. 9. Image state during the labeling and examination process.

© ASCE 04721008-6 J. Struct. Eng.

J. Struct. Eng., 2021, 147(11): 04721008

Downloaded from ascelibrary.org by Zhejiang University on 11/30/21. Copyright ASCE. For personal use only; all rights reserved.

Fig. 10. Selection of noncrack noise images: (a) noise images kept; and (b) abandoned noise images.

Fig. 11. Training images in the BCL: (a) cropped subimages; and (b) image labels.

Comparison with Current Crack Data Sets pavement crack images, mostly of asphalt pavement. The compari-
son was evaluated according to the following aspects: the quality of
Inspired by Mei and Gul (2020) and Majidifard et al. (2020), this the raw images (sources, amounts, and resolutions), the quality
study summarizes and compares 21 crack data sets established by of images in the data sets (amounts, originalities, and resolutions),
recent research on roads and bridges, as listed in Table 3. The and the type of annotation.
bridge category consists of crack data sets built on the raw images The raw images of cracks contain the information of photograph-
of bridge structures that do not contain any asphalt pavement of ing distances, illumination conditions, and weather conditions,
traffic lanes. The road category is made up of sealed or unsealed which would affect the quality of the images in the established data

© ASCE 04721008-7 J. Struct. Eng.

J. Struct. Eng., 2021, 147(11): 04721008

 Table 3. Summary of different data sets for road and bridge cracks
Raw images Image data set
Structure type Reference No. Resolution No. Resolution Type of annotation
Bridge Kim and Cho (2018) 7,195 Not listed 42,000 227 × 227 Patch
Zhang et al. (2020a) 2,206 1,280 × 960 2,206 Multiscale Bounding box
4,000 × 3,000
This studya (BCL) 1,400 6,000 × 4,000 11,000 256 × 256 Pixel
5,184 × 3,888
5,152 × 3,864
4,928 × 3,264
4,032 × 3,204
Dung and Anh (2019)a 458 4,032 × 3,024 40,600 227 × 227 Pixel (600)
Patch (40,000)
Dorafshan et al. (2018b) 230 4,068 × 3,456 56,092 256 × 256 Patch
Downloaded from ascelibrary.org by Zhejiang University on 11/30/21. Copyright ASCE. For personal use only; all rights reserved.

Alipour et al. (2019) 50 6,000 × 4,000 7,520 1,024 × 1,024 Pixel

45,402 512 × 512
209,801 256 × 256
Road Du et al. (2020)a 45,788 Not listed 45,788 2,400 × 1,500 Bounding box
Ochoa-Ruiz et al. (2020) 16,750 600 × 600 251,250 600 × 600 Bounding box
Maeda et al. (2018)a 9,053 600 × 600 9,053 600 × 600 Patch bounding box
Majidifard et al. (2020)a 7,237 640 × 640 7,237 640 × 640 Bounding box
Zhang et al. (2019a) 6,500 4,096 × 2,048 4,000 1,024 × 512 Pixel
Chen et al. (2020) More than 2,000 4,608 × 2,592 10,000 512 × 512 Mask
1,024 × 1,024
Gopalakrishnan et al. (2017)a 1,056 3,072 × 2,048 1,056 3,072 × 1,452 Image
Zhang et al. (2018b) 800 4,000 × 2,000 150,000 200 × 200 Patch
Park et al. (2019) 664 1,920 × 1,080 36,000 40 × 40 Patch
Mei and Gul (2020)a 600 1,920 × 1,080 600 1,920 × 1,080 Pixel
Bang et al. (2019)a 527 1,920 × 1,080 527 1,920 × 1,080 Pixel
Tong et al. (2017) 500 Not listed 6,832 256 × 256 Patch
Tong et al. (2018) Not listed Not listed 8,000 200 × 200 Patch bounding box
Hoang et al. (2018) Not listed Not listed 400 150 × 150 Patch
Sajedi and Liang (2019)a Not listed Not listed 118 480 × 320 Pixel
a
The data set is publicly available.

set. Some data sets adopted the online images as source images to subimages are cropped into raw images that preserve the originality
increase the number of raw images. Yet, online images might be of crack images. It maintains the potential for users to implement
compressed and might lose part of the original information of the and test their own methods of data augmentation without disturb-
uploaded raw images. Meanwhile, all of the raw images for the ance. Additionally, the type of annotation of the BCL is in a pixel-
BCL were collected on-site by engineers with experience in crack wise manner while most of the crack data sets listed in Table 3 are
detection. To guarantee the sufficiency and diversity of the estab- in a bounding or patch manner. Because the cracks are closely re-
lished image data sets, the numbers and resolutions, and the types lated to the safety of bridges and the geometric information of
of material of the raw images are essential parameters. As shown in cracks is of special importance to the assessment of the state of
Table 1, the BCL has a relatively large size among the bridge crack the safety of bridges, the pixel-wise annotation of the BCL could
data sets with regard to the numbers and resolutions of raw images. contribute greatly to the quantitative assessment of bridge cracks.
As for the diversity regarding the structural components, some data Overall, the uniqueness of the proposed data set is reflected in
sets contain cracks on some of the types of structural components, the following aspects: (1) all of the raw images of bridge cracks
such as beam cracks, while the BCL contains cracks on guardrails, were collected by trained bridge inspection engineers during peri-
pavements, beams, abutments, and piers. As for the diversity regard- odic inspection of real-world bridges; (2) these bridge crack images
ing structural materials, some data sets contain concrete or steel, were rich in terms of raw image numbers and diversity covering
while the BCL covers cracks on masonry, concrete, and steel struc- different types of bridge structural components and containing a
tures. Such sufficiency should help to improve the robustness and the large number of noise motifs; (3) all of these images were cropped
capability for generalization of the subsequently trained DNNs. into raw subimages that were not processed by any data augmen-
When compared with all of the data sets in Table 1, the rank of tation, which preserves the originality of the subimages and allows
the BCL decreases in terms of the numbers and resolutions. Yet, users to implement further image augmentation without disturb-
most of the large image data sets are asphalt pavement-based ones ance; and (4) the annotation type is in a pixel-wise manner that
whose surface cracks are different from the cracks on the surface of could facilitate the quantitative assessment of bridge cracks.
structural components made of masonry, concrete, or steel.
To compensate for the shortage of crack images in the data
sets, expansion approaches such as data augmentation were used Data Set–Based Evaluation of Crack Detection
in some studies. They were not likely to be able to synthesize Performance
the real situation of the cracks completely by methods such as im-
age stretching, image flip, and rotation. Conversely, the originality In this study, the main purpose for the establishment of the BCL is
of crack images has been changed. As for the BCL, all of the to provide a benchmark data set for DNN-based automatic local

© ASCE 04721008-8 J. Struct. Eng.

J. Struct. Eng., 2021, 147(11): 04721008

 crack detection of in-service bridges. In order to validate the appli- experiments carried out on a desktop but not the high-performance
cability of the BCL in training DNNs, evaluation of crack detection server.
performance using three different DNN models was carried out. The performance of crack detection of the three DNNs reflected
The three DNN models were the pruned crack recognition network by indices is listed in Table 6. According to the indices, the PCR-
(PCR-Net) proposed by this study, the VGG-based FCN (Long Net and the VGG-based FCN have similar performances and are
et al. 2015), and the Deeplab V3 (Chen et al. 2018). slightly better than Deeplab V3. The model size of the PCR-Net
The basic architecture of the PCR-Net was adopted from is 47.6% of the VGG-based FCN and 5.1% of the Deeplab V3.
a model proposed by Liu (2019). It was developed from the Meanwhile, the speed of the PCR-Net is 2.26 times that of the
U-Net, which is famous for semantic segmentation and frequently VGG-based FCN and 5.12 times that of Deeplab V3. For a direct
utilized in structural health monitoring (SHM) (Zhang et al. 2020b). understanding of the performance of crack detection, the results of
As investigated by previous studies, model pruning is an efficient image test for crack detection are illustrated in Fig. 12.
choice for obtaining lightweight DNN models with a smaller size The DNNs have successfully detected the crack area and over-
(Wu et al. 2019). Thus, a pruning method for network slimming come the noise motifs including the welding lines, red rules, black
proposed by Liu et al. (2017) was applied to reduce the size of markers, black letters, white letters, and stains. Yet, within Fig. 12
Downloaded from ascelibrary.org by Zhejiang University on 11/30/21. Copyright ASCE. For personal use only; all rights reserved.

the basic architecture. In terms of the problem of binary classifica- the dark and line-like motifs on the images in the row labeled Con-
tion between the crack and the background, cross entropy was crete (b) give the DNNs problems in the accurate detection of
adopted, which has been widely used by many research groups cracks. Overall, as seen from the indices and the performance of
(Spencer et al. 2019). As for the optimization algorithm, the Adam the image test, the BCL could be used successfully in training dif-
method, proposed by Kingma and Ba (2015), was adopted in the ferent DNNs for crack detection, and larger models tend to make
training process. When it comes to the evaluation of performance, more conservative predictions than smaller ones. In order to have a
the rate of accuracy, the rate of precision, the rate of recall, the mean deep insight into the difference of the performance of crack detec-
intersection over union (MeanIoU) and the F-measure were taken tion among the DNNs, a detailed comparison of image tests was
for evaluation in this study. carried out and is illustrated in Fig. 13.
The number and types of images used for training, validation The original image is a raw image with 4,938 × 3,264–pixel res-
and testing are shown in Table 4. Among the 11,000 images in olution, and the rows in Fig. 13 labeled Section 1 and Section 2 are
the BCL, 9,900 images were used for training and 1,100 images two parts of the original image that contain cracks with different
widths. Overall, the three DNNs were able to detect the main body
were used for validation. Therefore, the proportion for training
of the cracks, which are relatively wide. However, the VGG-based
and validation was 9:1. As for testing, full size raw crack images
FCN and the PCR-Net also consider the junction line as a crack,
were utilized to obtain a direct cognition of the performance of
while Deeplab V3 is immune to this noise motif. Despite the ro-
crack detection.
bustness against noise, Deeplab V3 missed many narrow cracks
The training process was conducted on a Linux server, the in-
that the other two DNNs successfully detected. This indicates that
formation regarding hardware and software is listed in Table 5. In
Deeplab V3 is more conservative than the others. The detection of
order to distinguish the computational efficiency of models with
cracks in the row marked Section 1 that has narrow cracks shows
different sizes, the efficiency was calculated by crack detection
the conservativeness of Deeplab V3. The results of the detection of
Deeplab V3 have few false positives, yet the depiction of the geom-
etry of the crack is not as complete as that of the other two DNNs. A
Table 4. Images for training more obvious illustration can be found in the results of the detec-
Training Validation Testing tion in the row marked Section 2 that has tiny cracks. Deeplab V3
Model (images) (images) (images) does not detect anything, while the VGG-based FCN detects a few
PCR-Net 9,900 1,100 Full size, raw scattered elements of the crack and the PCR-Net detects more
FCN (VGG-based) elements. This conservativeness might be helpful in reducing false
Deeplab V3 positives, but tiny cracks might be missed.

Conclusions
Table 5. Hardware and software for training
This study presents the process of the establishment of a data set of
Item Description bridge crack images called a bridge crack library. It is aimed at
Hardware CPU: Intel(R) Core(TM) i7-8700K [email protected] providing a benchmark data set for an automatic detection of local
GPU: 2 × NVIDIA TITAN Xp/GDDR5X 12GB cracks in in-service bridges. Quality control measures were imple-
RAM: 32GB mented during the establishment of the BCL. Additionally, noise
Software Ubuntu version 16.4.5 LTS images without cracks were selected to provide samples of noise
Pytorch version 1.1.0 motifs. Evaluation of the performance of crack detection using
Python version 3.5.0
three DNN models was conducted to assess the applicability
OpenCV version 3.4.2
and effectiveness of the BCL in training different DNN models.

Table 6. Performance of crack detection of the three DNN models

Model MeanIoU Precision Recall Accuracy F1 Model size (million) Efficiency (s/image)
PCR-Net 0.5693 0.7661 0.6891 0.9880 0.7255 9 11.7
FCN (VGG-based) 0.5630 0.7977 0.6567 0.9882 0.7204 18.9 26.4
Deeplab V3 0.4757 0.8103 0.5353 0.9864 0.6447 178 59.9

© ASCE 04721008-9 J. Struct. Eng.

J. Struct. Eng., 2021, 147(11): 04721008

Downloaded from ascelibrary.org by Zhejiang University on 11/30/21. Copyright ASCE. For personal use only; all rights reserved.

Fig. 12. Image test of the three DNN models.

Fig. 13. Detailed image test of the three DNN models.

Based on this study, several conclusions can be addressed, as components, and contain multiple noise motifs such as markers,
follows: spots, stains, corrosion, water spots, shadows, and welding
• The quality, quantity, and diversity of the image data set are lines.
critical for the performance of deep learning-based crack detec- • Quality control measures were carried out during the collection
tion methods. Given this consideration, 1,200 raw images of the of the raw images, during subimage cropping, and during the
BCL were collected from inspection of all kinds of in-service processes of the annotation of the subimages in order to guar-
bridge structures and 200 steel images were collected from antee the quality of the crack images in the data set. Noise im-
the first International Project Competition for Structural Health ages that contained crack-like motifs were selected and kept to
Monitoring. The raw images cover the majority of structural improve the robustness against noise motifs in the real world.

© ASCE 04721008-10 J. Struct. Eng.

J. Struct. Eng., 2021, 147(11): 04721008

 • Based on the establishment of processes, a BCL was developed image segmentation.” In Proc., European Conf. on Computer Vision.
that contained 11,000 pixel-wise labeled images with resolu- Berlin: Springer.
tions of 256×256, among which there were 5,769 nonsteel crack Chen, T. Y., Z. H. Cai, X. Zhao, C. Chen, X. F. Lianga, T. R. Zou, and
images, 2,036 steel crack images, 3,195 noise images, and their P. Wang. 2020. “Pavement crack detection and recognition using the
architecture of segNet.” J. Ind. Inf. Integr. 18 (Jun): 100144. https://doi
labels. The BCL contains crack images of multiple bridge struc-
.org/10.1016/j.jii.2020.100144.
tural components including the guardrail, the pavement, the
Daneshgaran, F., L. Zacheo, F. Di Stasio, and M. Mondin. 2019. “Use of
beam, the abutment, and the pier, which are made of masonry, deep learning for automatic detection of cracks in tunnels: Prototype-2
concrete, or steel. developed in the 2017-2018 time period.” Transp. Res. Rec. 2673 (9):
• The established BCL was used to train three DNNs (PCR-Net, 44–50. https://doi.org/10.1177/0361198119845656.
VGG-based FCN, and Deeplab V3) for structural crack detec- Deng, J. H., Y. Lu, and V. C. S. Lee. 2021. “Imaging-based crack detection
tion. The results indicated that these DNNs could detect cracks on concrete surfaces using you only look once network.” Struct. Health
satisfactorily against plenty of noise motifs. As a result, the pro- Monit. 20 (2): 484–499. https://doi.org/10.1177/1475921720938486.
posed data set could be used to train DNNs for pixel-wise de- Dorafshan, S., and H. Azari. 2020. “Deep learning models for bridge deck
tection of structural cracks, and serve as a benchmark data set evaluation using impact echo.” Constr. Build. Mater. 263 (Dec):
Downloaded from ascelibrary.org by Zhejiang University on 11/30/21. Copyright ASCE. For personal use only; all rights reserved.

for performance evaluation of DNN models. 120109. https://doi.org/10.1016/j.conbuildmat.2020.120109.

Dorafshan, S., R. J. Thomas, and M. Maguire. 2018a. “Comparison of deep
convolutional neural networks and edge detectors for image-based
crack detection in concrete.” Constr. Build. Mater. 186 (Oct): 1031–
Data Availability Statement 1045. https://doi.org/10.1016/j.conbuildmat.2018.08.011.
Dorafshan, S., R. J. Thomas, and M. Maguire. 2018b. “SDNET2018: An
Some or all data, models, or code generated or used during the annotated image dataset for non-contact concrete crack detection using
study are available in a repository or online in accordance with deep convolutional neural networks.” Data Brief 21 (Dec): 1664–1668.
funder data retention policies. The image data set is stored in https://doi.org/10.1016/j.dib.2018.11.015.
the Harvard dataverse and can be visited through the following link: Du, Y. C., N. Pan, Z. H. Xu, F. W. Deng, Y. Shen, and H. Kang. 2020.
https://doi.org/10.7910/DVN/RURXSH. “Pavement distress detection and classification based on YOLO net-
It is uploaded as a zip file and consists of three categories: non- work.” Int. J. Pavement Eng. 1–14. https://doi.org/10.1080/10298436
steel crack images, steel crack images, and noise images. In each of .2020.1714047.
Dung, C. V., and L. D. Anh. 2019. “Autonomous concrete crack detection
the categories, there are two folders, one contains the color images
using deep fully convolutional neural network.” Autom. Constr.
and the other contains the corresponding labels. Users can down- 99 (Mar): 52–58. https://doi.org/10.1016/j.autcon.2018.11.028.
load the file and decompress the zip file to get the images and Feng, C. C., H. Zhang, S. Wang, Y. L. Li, H. R. Wang, and F. Yan. 2019.
labels. “Structural damage detection using deep convolutional neural network
and transfer learning.” KSCE J. Civ. Eng. 23 (10): 4493–4502. https://
doi.org/10.1007/s12205-019-0437-z.
Acknowledgments Gao, X. W., M. Jian, M. Hu, M. Tanniru, and S. Q. Li. 2019. “Faster multi-
defect detection system in shield tunnel using combination of FCN and
The work described in this paper was jointly supported by the Basic Faster RCNN.” Adv. Struct. Eng. 22 (13): 2907–2921. https://doi.org/10
Science Center Program for Multiphase Evolution in Hypergravity .1177/1369433219849829.
of the National Natural Science Foundation of China (Grant Gao, Y. Q., and K. M. Mosalam. 2020. “PEER Hub ImageNet: A large-
No. 51988101), the National Natural Science Foundation of China scale multiattribute benchmark data set of structural images.” J. Struct.
Eng. 146 (10): 04020198. https://doi.org/10.1061/(ASCE)ST.1943
(Grant Nos. 51822810 and 51778574), and the Zhejiang Provincial
-541X.0002745.
Natural Science Foundation of China (Grant No. LR19E080002).
Gopalakrishnan, K., S. K. Khaitan, A. Choudhary, and A. Agrawal.
The authors thank Professor Hui Li of the Harbin Institute of Tech- 2017. “Deep convolutional neural networks with transfer learning
nology, China for providing part of the crack image data used in for computer vision-based data-driven pavement distress detection.”
this study during the First International Project Competition for Constr. Build. Mater. 157 (Dec): 322–330. https://doi.org/10.1016/j
Structural Health Monitoring (IPC-SHM 2020). .conbuildmat.2017.09.110.
Hinton, G. E., and R. R. Salakhutdinov. 2006. “Reducing the dimension-
ality of data with neural networks.” Science 313 (5786): 504–507.
References https://doi.org/10.1126/science.1127647.
Hoang, N. D., Q. L. Nguyen, and V. D. Tran. 2018. “Automatic recognition
Alipour, M., D. K. Harris, and G. R. Miller. 2019. “Robust pixel-level crack of asphalt pavement cracks using metaheuristic optimized edge detec-
detection using deep fully convolutional neural networks.” J. Comput. tion algorithms and convolution neural network.” Autom. Constr.
Civ. Eng. 33 (6): 04019040. https://doi.org/10.1061/(ASCE)CP.1943 94 (Oct): 203–213. https://doi.org/10.1016/j.autcon.2018.07.008.
-5487.0000854. Huang, H. W., Q. T. Li, and D. M. Zhang. 2018. “Deep learning based
Bang, S., S. Park, H. Kim, and H. Kim. 2019. “Encoder-decoder network image recognition for crack and leakage defects of metro shield tunnel.”
for pixel-level road crack detection in black-box images.” Comput.- Tunnelling Underground Space Technol. 77 (Jul): 166–176. https://doi
Aided Civ. Infrastruct. Eng. 34 (8): 713–727. https://doi.org/10.1111 .org/10.1016/j.tust.2018.04.002.
/mice.12440. Ju, H. Y., W. Li, S. S. Tighe, Z. C. Xu, and J. Z. Zhai. 2020. “CrackU-net: A
Cha, Y. J., W. Choi, and O. Buyukozturk. 2017. “Deep learning-based novel deep convolutional neural network for pixelwise pavement crack
crack damage detection using convolutional neural networks.” Com- detection.” Struct. Control Health Monit. 27 (8): e2551. https://doi.org
put.-Aided Civ. Infrastruct. Eng. 32 (5): 361–378. https://doi.org/10 /10.1002/stc.2551.
.1111/mice.12263. Kang, D. H., and Y. J. Cha. 2018. “Autonomous UAVs for structural health
Cha, Y. J., W. Choi, G. Suh, S. Mahmoudkhani, and O. Buyukozturk. 2018. monitoring using deep learning and an ultrasonic beacon system with
“Autonomous structural visual inspection using region-based deep geo-tagging.” Comput.-Aided Civ. Infrastruct. Eng. 33 (10): 885–902.
learning for detecting multiple damage types.” Comput.-Aided Civ. In- https://doi.org/10.1111/mice.12375.
frastruct. Eng. 33 (9): 731–747. https://doi.org/10.1111/mice.12334. Kim, B., and S. Cho. 2018. “Automated vision-based detection of cracks on
Chen, L. C., Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. 2018. concrete surfaces using a deep learning technique.” Sensors 18 (10):
“Encoder-decoder with Atrous separable convolution for semantic 3452. https://doi.org/10.3390/s18103452.

© ASCE 04721008-11 J. Struct. Eng.

J. Struct. Eng., 2021, 147(11): 04721008

 Kim, B., and S. Cho. 2019. “Image-based concrete crack assessment using Song, Q., Y. Q. Wu, X. S. Xin, L. Yang, M. Yang, H. M. Chen, C. Liu,
mask and region-based convolutional neural network.” Struct. Control M. J. Hu, X. S. Chai, and J. C. Li. 2019. “Real-time tunnel crack analy-
Health Monit. 26 (8): e2381. https://doi.org/10.1002/stc.2381. sis system via deep learning.” IEEE Access 7 (May): 64186–64197.
Kingma, D. P., and J. L. Ba. 2015. “Adam: A method for stochastic opti- https://doi.org/10.1109/ACCESS.2019.2916330.
mization.” Preprint, submitted December 22, 2014. https://arxiv.org/abs Song, W. D., G. H. Jia, H. Zhu, D. Jia, and L. Gao. 2020. “Automated
/1412.6980. pavement crack damage detection using deep multiscale convolutional
LeCun, Y., Y. Bengio, and G. Hinton. 2015. “Deep learning.” Nature features.” J. Adv. Transp. 2020: 6412562. https://doi.org/10.1155/2020
521 (7553): 436–444. https://doi.org/10.1038/nature14539. /6412562.
Li, C., P. J. Xu, L. J. L. Niu, Y. Chen, L. S. Sheng, and M. C. Liu. 2019. Spencer, B. F., V. Hoskere, and Y. Narazaki. 2019. “Advances in computer
“Tunnel crack detection using coarse-to-fine region localization and vision-based civil infrastructure inspection and monitoring.” Engineer-
edge detection.” Wiley Interdiscip. Rev.: Data Min. Knowl. Discovery ing 5 (2): 199–222. https://doi.org/10.1016/j.eng.2018.11.030.
9 (5): e1308. https://doi.org/10.1002/widm.1308. Tong, Z., J. Gao, Z. Q. Han, and Z. J. Wang. 2018. “Recognition of asphalt
Li, G., B. Ma, S. H. He, X. L. Ren, and Q. W. Liu. 2020. “Automatic tunnel pavement crack length using deep convolutional neural networks.”
crack detection based on U-net and a convolutional neural network with Road Mater. Pavement Des. 19 (6): 1334–1349. https://doi.org/10
alternately updated clique.” Sensors 20 (3): 717. https://doi.org/10.3390 .1080/14680629.2017.1308265.
Downloaded from ascelibrary.org by Zhejiang University on 11/30/21. Copyright ASCE. For personal use only; all rights reserved.

/s20030717. Tong, Z., J. Gao, and H. T. Zhang. 2017. “Recognition, location, measure-
Liu, Q. 2019. “U-Net implementation in PyTorch.” Accessed December 12, ment, and 3D reconstruction of concealed cracks using convolutional
2019. https://github.com/Qiuyan918/Unet_Implementation_PyTorch/blob neural networks.” Constr. Build. Mater. 146 (Aug): 775–787. https://
/master/Unet_Implementation_PyTorch.ipynb. doi.org/10.1016/j.conbuildmat.2017.04.097.
Liu, Z., J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang. 2017. “Learning Wu, R. T., A. Singla, M. R. Jahanshahi, E. Bertino, B. J. Ko, and D. Verma.
efficient convolutional networks through network slimming.” In Proc., 2019. “Pruning deep convolutional neural networks for efficient edge com-
2017 IEEE Int. Conf. on Computer Vision. New York: IEEE. puting in condition assessment of infrastructures.” Comput.-Aided Civ. In-
Long, J., E. Shelhamer, and T. Darrell. 2015. “Fully convolutional networks frastruct. Eng. 34 (9): 774–789. https://doi.org/10.1111/mice.12449.
for semantic segmentation.” In Proc., 2015 IEEE Conf. on Computer Xu, Y., S. Y. Wei, Y. Q. Bao, and H. Li. 2019. “Automatic seismic damage
Vision and Pattern Recognition. New York: IEEE. identification of reinforced concrete columns from images by a region-
based deep convolutional neural network.” Struct. Control Health
Maeda, H., Y. Sekimoto, T. Seto, T. Kashiyama, and H. Omata. 2018. “Road
Monit. 26 (3): e2313. https://doi.org/10.1002/stc.2313.
damage detection and classification using deep neural networks with
smartphone images.” Comput.-Aided Civ. Infrastruct. Eng. 33 (12): Xue, Y. D., and Y. C. Li. 2018. “A fast detection method via region-based
fully convolutional neural networks for shield tunnel lining defects.”
1127–1141. https://doi.org/10.1111/mice.12387.
Comput.-Aided Civ. Infrastruct. Eng. 33 (8): 638–654. https://doi.org
Majidifard, H., P. Jin, Y. Adu-Gyamfi, and W. G. Buttlar. 2020. “Pavement
/10.1111/mice.12367.
image datasets: A new benchmark dataset to classify and densify pave-
Yang, X. C., H. Li, Y. T. Yu, X. C. Luo, T. Huang, and X. Yang. 2018.
ment distresses.” Transp. Res. Rec. 2674 (2): 328–339. https://doi.org
“Automatic pixel-level crack detection and measurement using fully
/10.1177/0361198120907283.
convolutional network.” Comput.-Aided Civ. Infrastruct. Eng. 33 (12):
Mei, Q. P., and M. Gul. 2020. “A cost effective solution for pavement crack
1090–1109. https://doi.org/10.1111/mice.12412.
inspection using cameras and deep neural networks.” Constr. Build.
Ye, X. W., T. Jin, and P. Y. Chen. 2019a. “Structural crack detection using
Mater. 256 (Sep): 119397. https://doi.org/10.1016/j.conbuildmat.2020
deep learning-based fully convolutional networks.” Adv. Struct. Eng.
.119397.
22 (16): 3412–3419. https://doi.org/10.1177/1369433219836292.
Ni, F. T., J. Zhang, and Z. Q. Chen. 2019. “Pixel-level crack delineation in
Ye, X. W., T. Jin, and C. B. Yun. 2019b. “A review on deep learning-based
images with convolutional feature fusion.” Struct. Control Health
structural health monitoring of civil infrastructures.” Smart Struct. Syst.
Monit. 26 (1): e2286. https://doi.org/10.1002/stc.2286. 24 (5): 567–585. https://doi.org/10.12989/sss.2019.24.5.567.
Ni, Y. Q., X. W. Ye, and J. M. Ko. 2010. “Monitoring-based fatigue reli- Ye, X. W., Y. Q. Ni, T. T. Wai, K. Y. Wong, X. M. Zhang, and F. Xu. 2013.
ability assessment of steel bridges: Analytical model and application.” “A vision-based system for dynamic displacement measurement of
J. Struct. Eng. 136 (12): 1563–1573. https://doi.org/10.1061/(ASCE)ST long-span bridges: Algorithm and verification.” Smart Struct. Syst.
.1943-541X.0000250. 12 (3–4): 363–379. https://doi.org/10.12989/sss.2013.12.3_4.363.
Ni, Y. Q., X. W. Ye, and J. M. Ko. 2012. “Modeling of stress spectrum Ye, X. W., Y. Q. Ni, K. Y. Wong, and J. M. Ko. 2012. “Statistical analysis of
using long-term monitoring data and finite mixture distributions.” stress spectra for fatigue life assessment of steel bridges with structural
J. Eng. Mech. 138 (2): 175–183. https://doi.org/10.1061/(ASCE)EM health monitoring data.” Eng. Struct. 45 (Dec): 166–176. https://doi.org
.1943-7889.0000313. /10.1016/j.engstruct.2012.06.016.
Ochoa-Ruiz, G., A. A. Angulo-Murillo, A. Ochoa-Zezzatti, L. M. Aguilar- Yeum, C. M., J. Choi, and S. J. Dyke. 2019. “Automated region-of-interest
Lobo, J. A. Vega-Fernandez, and S. Natraj. 2020. “An asphalt damage localization and classification for vision-based visual assessment of
dataset and detection system based on RetinaNet for road conditions civil infrastructure.” Struct. Health Monit. 18 (3): 675–689. https://doi
assessment.” Appl. Sci. 10 (11): 3974. https://doi.org/10.3390/app .org/10.1177/1475921718765419.
10113974. Zhang, A., K. C. P. Wang, Y. Fei, Y. Liu, C. Chen, G. W. Yang, J. Q. Li,
Park, S., S. Bang, H. Kim, and H. Kim. 2019. “Patch-based crack detection E. H. Yang, and S. Qiu. 2019a. “Automated pixel-level pavement crack
in black box images using convolutional neural networks.” J. Comput. detection on 3D asphalt surfaces with a recurrent neural network.” Com-
Civ. Eng. 33 (3): 04019017. https://doi.org/10.1061/(ASCE)CP.1943 put.-Aided Civ. Infrastruct. Eng. 34 (3): 213–229. https://doi.org/10
-5487.0000831. .1111/mice.12409.
Ronneberger, O., P. Fischer, and T. Brox. 2015. “U-Net: Convolutional net- Zhang, A., K. C. P. Wang, Y. Fei, Y. Liu, S. Y. Tao, C. Chen, J. Q. Li, and
works for biomedical image segmentation.” In Proc., 18th Int. Conf. on B. X. Li. 2018a. “Deep learning-based fully automated pavement crack
Medical Image Computing and Computer Assisted Intervention. detection on 3D asphalt surfaces with an improved CrackNet.” J. Com-
Berlin: Springer. put. Civ. Eng. 32 (5): 04018041. https://doi.org/10.1061/(ASCE)CP
Ryu, E., J. Kang, J. Lee, Y. Shin, and H. Kim. 2020. “Automated detection .1943-5487.0000775.
of surface cracks and numerical correlation with thermal-structural Zhang, C. B., C. C. Chang, and M. Jamshidi. 2020a. “Concrete bridge sur-
behaviors of fire damaged concrete beams.” Int. J. Concr. Struct. Mater. face damage detection using a single-stage detector.” Comput.-Aided Civ.
14 (1): 1–12. https://doi.org/10.1186/s40069-019-0387-3. Infrastruct. Eng. 35 (4): 389–409. https://doi.org/10.1111/mice.12500.
Sajedi, S. O., and X. Liang. 2019. “A convolutional cost-sensitive crack Zhang, K. G., H. D. Cheng, and B. Y. Zhang. 2018b. “Unified approach to
localization algorithm for automated and reliable RC bridge inspec- pavement crack and sealed crack detection using preclassification based
tion.” In Proc., 10th New York City Bridge Conf. Boca Raton, FL: on transfer learning.” J. Comput. Civ. Eng. 32 (2): 04018001. https://doi
CRC Press. .org/10.1061/(ASCE)CP.1943-5487.0000736.

© ASCE 04721008-12 J. Struct. Eng.

J. Struct. Eng., 2021, 147(11): 04721008

 Zhang, L. X., J. K. Shen, and B. J. Zhu. 2020b. “A research on an Zhang, X. X., D. Rajan, and B. Story. 2019b. “Concrete crack detection
improved Unet-based concrete crack detection algorithm.” Struct. using context-aware deep semantic segmentation network.” Comput.-
Health Monit. 20 (4): 1864–1879. https://doi.org/10.1177/14759217 Aided Civ. Infrastruct. Eng. 34 (11): 951–971. https://doi.org/10
20940068. .1111/mice.12477.
Downloaded from ascelibrary.org by Zhejiang University on 11/30/21. Copyright ASCE. For personal use only; all rights reserved.

© ASCE 04721008-13 J. Struct. Eng.

View publication stats J. Struct. Eng., 2021, 147(11): 04721008