See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/371253385

Deep learning-based welding image recognition: A comprehensive review

Article  in  Journal of Manufacturing Systems · June 2023

DOI: 10.1016/j.jmsy.2023.05.026

CITATIONS READS

0 252

3 authors:

Tianyuan Liu Pai Zheng

The Hong Kong Polytechnic University The Hong Kong Polytechnic University
31 PUBLICATIONS   271 CITATIONS    197 PUBLICATIONS   4,880 CITATIONS   

SEE PROFILE SEE PROFILE

Bao Jinsong
Donghua University
161 PUBLICATIONS   1,901 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Proactive Human-Robot Collaboration (GRF & AIR@InnoHK & Endowed Young Scholars Scheme, HKSAR) View project

Digital twin View project

All content following this page was uploaded by Tianyuan Liu on 05 June 2023.

The user has requested enhancement of the downloaded file.

 Journal of Manufacturing Systems 68 (2023) 601–625

Contents lists available at ScienceDirect

Journal of Manufacturing Systems

journal homepage: www.elsevier.com/locate/jmansys

Review

Deep learning-based welding image recognition: A comprehensive review

Tianyuan Liu a, Pai Zheng a, *, Jinsong Bao b, *
a
Department of Industrial and Systems Engineering, The Hong Kong Polytechnic University, 999077, Hong Kong, China
b
College of Mechanical Engineering, Donghua University, Shanghai 201620, China

A R T I C L E I N F O A B S T R A C T

Keywords: The reliability and accuracy of welding image recognition (WIR) is critical, which can largely improve domain
Welding system experts’ insight of the welding system. To ensure its performance, deep learning (DL), as the cutting-edge arti­
Computer vision ficial intelligence technique, has been prevailingly studied and adopted to empower intelligent WIR in various
Image recognition
industry implementations. However, to date, there still lacks a comprehensive review of the DL-based WIR
Deep learning
(DLBWIR) in literature. Aiming to address this issue, and to better understand its development and application,
Convolutional neural network
Explainable AI this paper undertakes a state-of-the-art survey of the existing DLBWIR research holistically, including the key
technologies, the main applications and tasks, and the public datasets. Moreover, possible research directions are
also highlighted at last, to offer insightful knowledge to both academics and industrial practitioners in their
research and development work in WIR.

number of publications is 123 % (((24− 7)/7 + (46− 24)/24 +

(62− 46)/46)/3). Furthermore, as of the end of June, 52 articles have
1. Introduction
been published in 2022. This indicates that the research on DLBWIR is
still at a high rate of growth. Fig. 2. plots the unit locations of the arti­
Machine vision plays a critical role in assuring weld quality through
cle’s participants, from which it can be seen that the research team came
seam tracking [1] and online monitoring [2] and NDT [3] post welding.
from 27 countries in Asia, North America, Europe, South America,
Its core, WIR, has gone through two main stages, per the use of feature
Oceania and Africa, indicating that the topic has attracted global interest
engineering or DL models [4]. With rapid developments of third wave AI
(Since an article may involve research institutions from more than one
innovation and high-performance computing infrastructure [5,6], ∑ ∑
country, thus locationi numberi > yearj numberj , as shown later). In
DLBWIR has become a dominant research hotspot [7,8].
response to this statistic, the authors argue that this is driven by smart
In literature, the concept of DL was first introduced by Hinton et al.
manufacturing strategic plans such as the American Industrial Internet,
[9] in 2006. Then, the overwhelming dominance of AlexNet in 2012 led
German Industry 4.0 and Made in China 2025. AI technology with DL as
to the explosion of DL in the CV domain [10]. Not until 2016, when
the core is the supporting element of these strategic planning. As a
AlphaGo defeated the world Go champion Lee Sedol, DL has started to
result, research related to DLBWIR as one of the instantiated scenarios
attract ever-increasing attention from industry [11]. Wang et al. [12]
for smart manufacturing also shows the trends in Fig. 1 and the distri­
summarized the commonly used DL structures in manufacturing,
bution of research in Fig. 2.
namely CNN, RNN, AE and DBN. Nevertheless, to the authors’ best
Completeness is the fundamental for review work. This paper covers
knowledge, there still lacks comprehensively reviews on the DLBWIR as
all relevant journal articles published from January 2016, when DL first
the fundamental basis for researchers and industrial practitioners in this
entered the field of WIR. Table 1 summarizes the search topics, types of
domain. Motivated by it, this paper presents a holistic review of recently
reference, source of databases and reference screening methods in this
published articles on DLBWIR, selected by the Web of Science database
review. As a result, 230 articles were derived. Next, the authors assessed
from January 2016 to June 2022, as shown in Fig. 1.
the relevance of each paper to the review topic based on the content of
As DL came to the forefront of industry in 2016, Günther et al. [13]
each article’s abstract, introduction, method and conclusion. For
conducted initial explorations and mapped out a DL-based intelligent
example, some articles contain the keywords “DL” or “welding image”,
welding framework. After a year of precipitation, more scholars began to
but they do not include DL and welding image as a whole, which were
publish technical articles ever since 2018. During the outbreak phase
therefore excluded from further review. Eventually, 192 articles
(last three years, 2019–2021), the average annual growth rate of the

* Corresponding authors.
E-mail addresses: [email protected] (P. Zheng), [email protected] (J. Bao).

https://doi.org/10.1016/j.jmsy.2023.05.026
Received 13 September 2022; Received in revised form 4 March 2023; Accepted 29 May 2023
Available online 2 June 2023
0278-6125/© 2023 The Society of Manufacturing Engineers. Published by Elsevier Ltd. All rights reserved.
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

Nomenclature GAN Generative adversarial network

DSC Depthwise separable convolution
WIR Welding image recognition NDT Non-destructive testing
DL Deep learning ME Model ensemble
DLBWIR DL-based WIR AM Attention mechanism
AI Artificial intelligence CAM Class activation mapping
CV Computer vision GradCAM Gradient-weighted class activation mapping
CNN Convolutional neural network BN Batch normalization
RNN Recurrent neural network KCV K-fold cross-validation
AE Auto encoder LSTM Long short-term memory
DBN Deep belief network Acc Accuracy
PRW Pre-welding MSE Mean squared error
IW In-welding DE Detection error
POW Post-welding MAP Mean average precision
CCD Charge coupled device GOF Goodness of fit
TFT Time-frequency transform Rec Recall
OM Optical microscope IOU Intersection over union
ROI Region of interest TE Tracking error
PreP Preprocessing PRMSE Pixel root mean square error
NormL Normalization SS Structural similarity
DA Data augmentation DWIQI Duplex wire-type image quality indicator
TL Transfer learning KG Knowledge graph
MML Multi-modal learning WDKG Welding domain knowledge graph

view, welding images can be obtained directly through visual sensing, or

indirectly through other sensing methods. The division of welding stages
and typical welding images corresponding to each stage are shown in
Fig. 3. In general, the WIR of PRW, IW, and POW is used to solve the
welding tracking, welding process monitoring and defect detection
tasks, respectively.

2.1.2. Why image-based

One of the main characteristics of image compared to other data
types is its spatial nature so that it can contain comprehensive features
such as position, shape, size and texture that reflect the state of the
welding process producing the welds or directly depict the welds.
However, acquiring an image that sufficiently reflects the target welding
information is often challenging and such challenge varies per require­
ment, welding state and manufacturing environment. In response to this
Fig. 1. Statistics on the number of articles published each year. problem, scholars have used different approaches to acquire welding
images to ensure specific engineering requirements be met. Du et al.
[15] scanned the weld joint with a laser strip and imaged it with a CCD
published in IEEE, Elsevier, Springs, Taylor & Francis, MDPI, ASME,
camera to obtain the position of the weld seam. Liu et al. [16] imaged
AWS, SAGE, and Hindawi were selected.
molten pool and keyhole from the front by a CCD camera to identify
different penetration levels in laser welding. Jiao et al. [17] simulta­
2. Connotations and motivations
neously collected the front and back information of the molten pool to
observe the penetration state. Li et al. [18] collected both the back image
2.1. Welding image
and the front laser reflection image of the molten pool to obtain suffi­
cient penetration information. Lu et al. [19] used a CCD camera to image
2.1.1. What is welding image
workers’ behavior to judge whether they conform to welding specifi­
Welding is a manufacturing process and technique for joining metals
cations. Knaak et al. [20] observed the surface defects of the molten pool
or other thermoplastic materials such as plastics by means of heat, high
by mid-wave infrared imaging technology. Jin et al. [21] converted the
temperature or pressure. There are abundant visual information sources,
current signal into a two-dimensional image by means of TFT, and used
either being related to or reflecting the quality of the weld produced,
it to visually display the information of the forming quality of the
from all three stages: PRW, IW, and POW. Such information helps assure
backside of the weld. For the problem of welding fault identification, Lee
the welding process operate appropriately to produce desired welds and
et al. [22] designed an interpretable TFT method. Li et al. [23] reflected
decide if produced welds meet quality requirements. Per literature,
the molten pool size information by simulated laser point reflection
welding image can be defined as “grid-type data of single or multiple
image. Yang et al. [24] photographed the appearance of the weld by
channels obtained by direct or indirect methods in the three stages of PRW,
CCD to observe its surface forming quality. In the references [25–27],
IW and POW, which can reflect the characteristics of weld seam and welding
the internal defect information of the weld was obtained by imaging
quality and promote welding process improvement” [7,14]. Therefore, its
methods of X-ray, microscope and TFT, respectively. Table 2 summa­
core is to reflect welding quality information, which is characterized by
rizes the different ways in which welding image has been acquired in
grid-type data from any of the welding stages. From the source point of
literature and their contents.

602
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

2.1.3. How to obtain welding image Table 1

It can be seen from Fig. 4 that the current research is mainly focused Systematic method on screening relevant articles.
on obtaining image data in the stage of IW and POW. This is mainly Searching Index Specific Content
because the technology of seam tracking for PRW is currently at a more
Search topics TS = ((weld image OR molten pool OR weld penetration OR
mature stage [38], while online closed-loop monitoring for IW and weld globule transition OR weld seam OR weld radiographic
intelligent NDT for POW are still challenging tasks. Fig. 4 provides sta­ OR weld defect) AND (deep learning OR DL OR CNN OR
tistics on the direct and indirect ways of acquiring welding images convolutional neural network))
during the different stages. One can find that the CCD dominates Article type Scientific/technical articles published in peer-reviewed
journals and published online
welding image acquisition technology and has been widely used in PRW, Search period From January 2016 to June 2022
IW and POW. The authors attribute this statistic to the fact that CCD Database Science citation index expanded (SCIE) and Emerging sources
imaging is low cost, rich information, no contact with the workpiece, citations index (ESCI)
and wide application scenarios [39,40]. Screening The relevance with the research topic as judged by the contents
procedure of abstract, introduction, method and conclusion of every
In PRW stage, the current research mainly uses laser stripes to scan
papers.
the weld bead and groove, and then uses CCD to shoot the laser stripes to Classification Key technologies of current development of DLBWIR (as shown
reflect the welding quality information. scheme in Section 3), applications of DLBWIR (as shown in Section 4)
In IW stage, CCD is used to directly photograph the melting area to and available public datasets (as shown in Section 5)
observe the dynamic process of this area from the outside. In addition,
other sensing methods such as sound and electricity are used to obtain
2.2. Welding image recognition
one-dimensional signals in the welding process, and then two-
dimensional welding images are obtained through TFT. It also simu­
After obtaining the welding image, the abstract feature information
lates the image during the welding process based on the theoretical
reflecting the welding quality such as size, shape and category, can be
model.
recognized from the direct information including brightness, color and
In POW stage, the CCD is mainly used to take images of the
pixel distribution contained in the welding image by the WIR technol­
appearance of the weld seam in order to observe the quality of the
ogy. According to the actual task requirements, this paper divides the
surface formation. However, this welding image does not show the in­
WIR task into four levels as shown in Fig. 5: 1) Classification. It aims to
ternal defects of the welding seam. Based on the OM, internal defects can
discover the object category in the welding image, which is an image-
be seen from the weld profile in a destructive way, while the internal
level recognition task (What?). 2) Detection. It aims to find the position
defects can be seen from the image in a non-destructive way based on
of the object in the welding image, which is a target-level recognition
the X-ray scanning method. In addition, one-dimensional signals of weld
task (Where?). 3) Segmentation. It aims to calculate the size of the object
formation can be obtained through sound or ultrasound, and then two-
in the welding image, which is a pixel-level recognition task (How
dimensional welding image can be obtained through TFT.
much?). 4) Tracking. It aims to solve when and where the target is in the
welding image, which is a target-level recognition task for dynamic
images (When and where?). It is worth noting that the above four types of

Fig. 2. Geographical distribution of published works in DLBWIR worldwide.

603
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

effective method for image recognition, it has been studied very little in
the WIR field. Therefore, based on the definition of DL, the paradigm of
supervised DLBWIR can be summarized in Fig. 6. In practice, three
major ways of obtaining welding image labels are: expert experience-
based manual labelling [46], welding process-based automatic label­
ling [47] and the high precision inspection-based automatic labelling
[48]. After acquiring the welding image data and the labels corre­
sponding to the different tasks, a certain hypothesis function is learned
from the hypothesis space defined by the structure of the model for the
different tasks as the final model according to the training method. The
process is supported by the platform provided by the hardware and
software environment. This well-trained model is an approximation of
the real model.

2.4. Related surveys

In view of the important role of WIR in welding systems, scholars

Fig. 3. Conceptual scope of welding image: (a) division of welding stages, and have reviewed the literature in the field from different perspectives.
(b) typical welding image. Table 4 summarizes the scope, taxonomy and relevance to DLBWIR of
different review articles.
tasks are not completely independent. For example, the tasks of the The following conclusions can be drawn from a comparative analysis
second and third levels generally also include the tasks of the first level. of recently published review articles. The academic community gener­
Therefore, welding image classification is the most fundamental recog­ ally believes that DL, especially CNN, has significant advantages in WIR
nition task, and there are relatively many studies on this level. and is the research direction to solve the WIR problem. Despite the
significant contributions of each reviewed survey, these review articles
2.3. Deep learning either present DL as a future perspective for solving WIR problems or analyze
some of the DLBWIR literature as it relates to individual welding stages.
2.3.1. What is deep learning Accordingly, this paper will provide a comprehensive review of the
Table 3 lists several typical definitions of DL. Although scholars have DLBWIR literature involved in the three stages of PRW, IW and POW.
defined DL from multiple perspectives such as structure, process, pur­ Although people are very curious to know "what are the difficulties and
pose, scope, function, and form, they lack a description of DL from an challenges in DLBWIR?", actually, this question is not easy to answer and
implementation perspective. Therefore, in conjunction with engineering may even be over-generalized. Since the welding image at different
practice, this paper refers to the data, model, training and platform stages and the WIR tasks in different scenarios have totally different
technologies involved in the DLBWIR process together with the basic objectives and constraints, their difficulties may vary from each other.
connotation of DL mentioned in Table 3 below. As such, it is difficult to generalize existing research from the perspective
of unifying issues and challenges. Therefore, this paper will summarize
2.3.2. Why deep learning-based the key technical components of DLBWIR from an overall perspective in
Assuming that the welding image is x and the welding quality in­ Section 3, as well as the challenges and corresponding solutions brought
formation is Q, the process of WIR is to seek f so that f(x) = Q. In by the characteristics of welding image to the application of DL. In
general, the development of WIR has gone through two stages: feature Section 4, the engineering requirements, vision tasks, difficulties, solu­
engineering-based and DL-based. In the feature engineering-based tions and results of DLBWIR tasks in different welding stages will be
method, it generally follows the technical route of image denoising, dissected. Given the key position of data in DLBWIR, this paper will
image enhancement, object segmentation, feature extraction, feature summarize the public datasets in the field and analyze their features in
selection, and feature recognition, that is, f(x) = Section 5. Finally, an outlook on future directions, both in terms of
fRec (fSel (fExt (fSeg (fEnh (fDen (x)))))). This idea decomposes the process of literature and authors, will be presented in Section 6.
finding f into several sub-tasks. Although it has strong interpretability, it
has the following shortcomings. First, the process involves too many 3. Key technologies of DLBWIR
intermediate subtasks. This not only spreads the optimization goal, but
also fails to achieve the global optimum. Second, the process relies too Based on the paradigm of DLBWIR, this section summarizes the key
much on expert empirical knowledge. This not only makes it difficult to technologies involved in DLBWIR from four aspects: data, model, training
give full play to the value of data, but also requires higher multidisci­ and platform.
plinary knowledge collaboration. The DL-based method can directly
establish the end-to-end mapping from x to Q, that is, f(x) can be directly 3.1. Data technology
derived from DL(x). This modelling approach requires only one objec­
tive function to be optimized and eliminates intermediate steps and 3.1.1. Welding image itself
cumulative errors. In addition, the process can be data-driven to adap­ As shown in Fig. 7, the key technologies for the welding image itself
tively learn and extract abstract visual features of the object, avoiding in the literature mainly include ROI, PreP, and NormL.
the need for a manual experience-driven feature extraction process. Table 5 summarizes the specific ROI, PreP, and NormL methods
Therefore, this idea can not only achieve global optimization, but also involved in the existing literature. Welding image generally has the
give full play to the value of data and improve the efficiency of model characteristics of high pixel, small object, and large background. ROI
development. refers to finding out the region of interest in the original image and
identifying it. Therefore, the ROI operation on the original welding
2.3.3. How to perform DLBWIR image can make the model focus on learning the characteristics of the
As DLBWIR is based on a DL approach to solving WIR tasks, DLBWIR key area while reducing the amount of calculation. Furthermore, as the
is largely in line with the technical framework followed by DL in other input image size is fixed for some classical network structures and pre-
areas of image recognition. Although unsupervised learning is also an trained models, ROI methods become one of the most commonly used

604
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

Table 2
Comparison of typical welding image at different stages.
Ref. Year Stage Method Sketch Map Image content Welding information

[15] 2019 PRW CCD Single-stripe laser, a single weld joint and groove. Weld joint position

2021 IW CCD Front of keyhole Penetration state

[28]

2021 Front of molten pool and keyhole Penetration state

[16]

2020 Front and back of molten pool Penetration state

[17]

2020 The back side of the melt pool, and the front laser stripe. Penetration state
[18]

2019 Weld seams are captured in three directions by multiple Penetration state
[29]
specular reflections.

2020 Protective products for manual welding Work safety and welding processes
[19]

2022 Infrared imaging of the molten pool Penetration state

[30]

2021 Mid-wave infrared imaging of the molten pool Surface defects

[20]

2020 TFT Current signal Weld back forming

[21]

2021 Vibration signal Internal defects in welded joint

[31]

2020 Sound signal Penetration state

[32]

2019 Eddy current signal Internal defects in weld seam

[33]

2022 Acceleration, displacement, current and voltage. Interpretable transformation to reflect welding failure
[22]
mechanisms

(continued on next page)

605
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

Table 2 (continued )
Ref. Year Stage Method Sketch Map Image content Welding information
2021 Simulation Molten pool Internal defects in weld seam
[34]

2021 Laser reflection point Size of molten pool

[23]

2021 POW CCD Weld appearance Surface defects in weld seam

[24]

2020 Multi-stripe laser scan multiple weld bead and groove Surface defects in weld seam
[35]

2021 X-ray Inside the weld seam Internal defects in weld seam
[25]

2020 OM Micro-organization Internal defects in weld seam

[26]

2021 TFT Sound signal Internal defects in weld seam

[36]

2020 Ultrasound signal Internal defects in weld seam

[27]

2022 Time-of-flight diffraction Internal defects in weld seam

[37]

Fig. 4. The way of acquiring welding image in different welding stages.

techniques for welding image data itself. 3.1.2. Welding image dataset
Due to the interference of arc light, spatter, and vibration in the In addition to the problems of the welding image itself, there is also a
welding environment, PreP methods represented by filtering, denoising, widespread problem of small samples at the dataset level, which brings a
and enhancement have also been extensively researched. The PreP re­ huge obstacle to the in-depth application of the DL methods. From the
sults in high quality welding image that facilitates the learning of feature data point of view, scholars have mainly addressed this problem through
information. DA, TL, and GAN. Fig. 8 plots the number of occurrences of the three
Meanwhile, the purpose of welding image NormL is to compress the methods in the literature.
grey scale range, thus reducing the search space and speeding up the Table 6 summarizes the specific DA, TL, and GAN methods involved
learning and convergence of the DL model. in the existing studies.
The idea of DA is to artificially transform the original image in the
spatial, greyscale and color domains to obtain new samples of welding

606
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

Fig. 5. Four basic tasks of WIR: (a) classification, (b) detection, (c) segmentation, and (d) tracking.

Table 3
Typical definitions of DL.
Ref. Year Definition Keywords Highlights

[41] 2009 DL is about learning feature Feature Structure,

hierarchies with features from hierarchies, process
higher levels of the hierarchy composition
formed by the composition of
lower level features.
[42] 2014 DL is a class of machine learning Machine Scope,
algorithms that uses multiple learning, process
layers to progressively extract multiple layers
higher-level features from the
raw input.
[43] 2015 DL (also known as deep Hierarchical Function,
structured learning or learning, form
hierarchical learning) is data Fig. 6. The overall paradigm of DLBWIR.
learning data representations, as representations
opposed to task-specific
algorithms. Learning can be datasets from the perspective of relevance to the target dataset.
supervised, semi-supervised or Moreover, GAN [72] is to learn the distribution of existing data
unsupervised. through generative network, and to expand the dataset by generating
[44] 2015 DL is a multi-level feature Multi-level, Structure,
learning method that uses feature learning process
new welding image samples based on the learned distribution. The
simple but non-linear commonly used GAN structures in DLBWIR are DGCAN [73], CGAN
components to gradually [74], CycleGAN [75], BAGAN [76] and WGAN [77].
abstract the features of each
layer (starting from the original
data) into higher-level features. 3.2. Model technology
[45] 2018 DL is not only the process of Learning Purpose
learning the relationship relationship,
3.2.1. Backbone
between two or more variables, learning
but also the process of learning knowledge
After acquiring the welding image, a suitable model structure needs
the knowledge that controls the to be selected to learn the feature patterns embedded in the welding
relationship and making the image. Fig. 9 summarizes the backbone structures that have been used
knowledge meaningful. for different DLBWIR tasks. It is worth noting that some studies have
optimized and improved some hyperparameters of the backbone ac­
image to expand the dataset [70]. DA is the simplest manner to imple­ cording to the specific problem (e.g., changing the number of layers and
ment and the process of learning the various transformations greatly the number of convolutional kernels, etc.), but the core of the network
improves the inductive bias of the model, making it the most commonly still belongs to that backbone, so it is counted as such for statistical
used method for inadequate welding image data. In practice, there are purposes.
both offline and online ways to achieve DA. Among all backbone structures, Lenet5 [96] has received the most
Meanwhile, the idea of TL is to use the knowledge learned by the applications, mainly because the structure is the simplest and contains
model to solve the source task based on the source dataset to improve the fewest parameters to be trained, which is more suitable for WIR with
the target task based on the target dataset [71]. Essentially, the model small samples. It also contains fewer key operations and techniques and
structure is first pre-trained based on the source dataset and then is therefore more scalable. Resnet [97] introduces the residual structure
fine-tuned based on the target dataset. Where a source dataset is a to improve the trainability of deeper DL models. VGG [98] maintains the
dataset with a rich sample size and label information that is different size of the receptive field by reducing the size of the convolution kernel
from the target dataset. The target dataset is a dataset that currently and increasing the depth of the network. Alexnet [10] not only imple­
holds insufficient data volume and label information to solve the target ments an efficient parallel computing structure, but also proposes clas­
task. Source datasets can be classified into out-of-domain and in-domain sical training techniques to prevent overfitting of the model. Mobilenet
[99] decomposes the classic convolution operation into two stages of

607
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

Table 4
Review articles related to DLBWIR.
Ref. Year Scope Taxonomy Relation to DLBWIR

[38] 2019 Weld seam Sensing technology, The authors found

tracking task in feature extraction that vision is the most
the PRW stage technology and widely used sensing
application scenarios method for seam
for seam tracking tracking task, and
task. believe that AI
technology is the
future direction of
quality monitoring
and process
optimization.
[4] 2021 Monitoring Sensing, The authors provide a
task in the IW preprocessing, brief analysis of the
stage segmentation, feature CNN-based WIR Fig. 7. Key technologies for welding image itself.
definition, extraction, literature in the
and recognition feature recognition
techniques for WIR. stage and argue that Table 5
DL techniques are the The specific technologies for welding image itself and their comparison.
future direction of
WIR. Ref. Year Key elements Method Pros Cons
[7] 2020 Laser welding Visual sensing The authors consider [52] 2021 Crop ROI Improve the Loss of semantic
monitoring technology and visual AI and in particular [53] 2019 Estimating ROI processing information.
task in the IW monitoring DL-based quality from center efficiency of
stage application scenarios. monitoring and point and arc of DL and
evaluation system to the promote DL to
be the most transmission learn key
interesting and [54] 2020 Resize features of
challenging future [55] 2019 Crop and down- welding
direction. sampling images.
[49] 2022 Laser welding Visual sensing, A summary of [56] 2021 Crop and resize
monitoring feature analysis and modeling methods [57] 2020 Crop, resize and
task in the IW modelling techniques. based on DL and CNN binarization
stage in the laser welding [58] 2019 Filtering, PreP Improve the This is a non-
literature is enhancement, quality of end-to-end
presented. and welding operation and
[50] 2020 NDT task in the Preprocessing, The application of DL, segmentation images. lacks
POW stage segmentation, feature especially CNN [59] 2021 Denoising and interaction with
extraction/selection methods, in the contrast DL-based
and recognition feature recognition enhancement welding image
techniques. stage is briefly [60] 2020 Histogram feature learning
summarized. equalization process.
[51] 2021 Vision sensing Vision sensing The authors believe and gray scale
task in the application scenarios. that visual analysis stretch
PRW, IW and modules can [61] 2021 Deblur
POW stages significantly reduce [62] 2020 Two-channel
the dependence of synthesis
welding systems on [63] 2020 Image
the operator. AI and alignment
especially CNN are [64] 2021 Gauss low-pass
used as future filter and
research directions to median filter
fully exploit welding [65] 2021 Spatial domain
features. filtering and
[8] 2020 Intelligent Enabling The authors identify gradient edge
welding system technologies, AI and DL as key extraction
platform technologies enabling technologies [66] 2019 The gray value NormL The model Lack of
and typical for intelligent range is (0, 1) solution space consideration
application scenarios. welding system. [34] 2021 The gray value is reduced to for adaptability
[14] 2020 Intelligent The welding The authors believe range is (− 0.5, speed up to activation
welding system manufacturing that DL is an effective +0.5) model learning functions.
process is broken approach to solving [67] 2020 Ƶ-NormL and
down into three complex modelling [68] 2020 Zero-center convergence.
stages: process pre- problems in the NormL
design, optimized welding [69] 2022 Normalize
design and in-line manufacturing. along the time
monitoring, and the axis
challenges and
possible solutions for
each stage are
depthwise convolution and pointwise convolution to reduce the amount
condensed through
formal descriptions.
of parameters and calculations, making it possible to deploy the DL
model on the mobile terminal.
On one hand, R-CNN [100–103] and SSD [104] are the three most
commonly used families of DL structures for object detection. The
R-CNN series is two-stage detector. Although it is slower, it has accuracy

608
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

Table 6
The specific technologies for welding image dataset and their comparison.
Ref. Year Key elements Method Pros Cons

[78] 2022 Stick DA Simple, Insufficient

[79] 2021 Brightness and intuitive and diversity of
cropping easy to samples
[18] 2020 Translation and operate
rotation
[20] 2021 Mirroring and
rotation
[80] 2021 Mirroring and
adding noise
[61] 2021 Lighting, rotation,
cropping, adding
noise, flipping,
saturation,
contrast, resize,
and cutting
Fig. 8. Key technologies for welding image dataset. [81] 2021 Flipping, color
jittering and
affine
advantages. Meanwhile, YOLO and SSD series are one-stage detectors
transformation
with the advantage of fast speed. The YOLO series offers a significant [82] 2020 Rotation, flipping,
increase in computing speed with less loss of accuracy, and the models in and the aspect
this series are always being maintained and updated, making them the ratio
most widely used in the DLBWIR field. [24] 2021 Affine
transformation,
On the other hand, FCN [105] and U-Net [106] are two classical gamma
image segmentation structures, both consisting entirely of convolutional transformation,
layers. FCN uses point-by-point summation to fuse multi-scale features, linear
and U-Net uses concatenate to fuse multi-scale features [107]. Also, transformation,
resize, and adding
Mask R-CNN [108] is a flexible structure that can be oriented to clas­
noise
sification, detection and segmentation. U-Net has been the most studied [67] 2020 Shearing,
application for its ability to maintain high segmentation accuracy even skewing, flipping
with small samples. However, the current solution for the tracking task and elastic
is to convert it into a temporal object detection or segmentation task, distortion
[83] 2020 Rotation,
and there is still a lack of application of network structures such as HCF shearing, shifting,
[109], MDNet [110], ECO [111], etc. that are directly modelled for the zooming, and
tracking task. channel
substitution
[84] 2021 Rotation, flipping,
3.2.2. Improved structure
color jittering,
The improved structure based on backbone have also been widely shifting, four
studied and applied in DLBWIR. We summarize the improved structure images are
that appear in the literature into four categories: MML, ME, AM, and cropped and
explainability. stitched into one
[85] 2021 Flipping, cutmix,
Although visual sensing has the advantage of being rich in infor­ and mosaic
mation, there are also some characteristic information that cannot be [86] 2019 Transferred from TL Improving the The network
obtained by visual sensing, such as the internal quality information of the out-of-domain initialization structure for
the weld during the welding process. In response to this problem, MML dataset without performance the target task
fine turning of model needs to be the
can utilize the complementarity between multiple modalities to learn a
[87] 2021 Transferred from same as the
more comprehensive feature representation [112]. The shared repre­ the dataset in the pretrained
sentation between multiple modalities allows the DL model to have a domain network
stronger inductive preference for certain hypotheses and can therefore [88] 2020 Transferred from structure.
serve to limit the hypothesis space. the dataset with
1,600,000 images
The welding image is formed by a combination of strongly coupled in the domain.
factors and the features learned by a single DL model may not be able to [89] 2019 Transferred from
describe the complex patterns embedded in the welding image. The the dataset in the
main idea of ME is to train multiple DL models simultaneously and then domain, and
conduct self-
have multiple models make voting decisions, thus improving the overall
supervised pre-
decision accuracy. training.
In addition to key objects, non-main information such as background [90] 2021 Transferred from
and interference widely exist in welding image, which presents an the dataset in the
obstacle to the DL model learning the main features. The AM allows for domain, and
conduct
more weighting of important feature information, thus preventing unsupervised pre-
important information from being masked by noise. As such, the AM not training.
only helps the model to decide which features in the welding image need [91] 2022 Comparing the
more attention, but also to allocate the limited computational resources effect of TL based
on different
to the more important parts. The imposed domain of attention in
backbone
DLBWIR involves spatial [113], channel [114], spatial channel mixing
(continued on next page)
[115] and text semantic [16].

609
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

Table 6 (continued ) summarizes the optimizer and training techniques involved in the
Ref. Year Key elements Method Pros Cons DLBWIR model training process.
The optimizer guides the model parameters to update the appro­
[92] 2020 Convolutional GAN The new It is difficult to
layers replace samples are learn a
priate size in the correct direction in the process of DL backpropagation,
perceptrons more diverse reliable data so that the updated parameters make the loss function value continue to
[26] 2020 Add conditional distribution approach the global minimum, which plays an important role in the
information to the while the model training process. It can be seen from the definition that the core of
generator and original
the optimizer includes two aspects, namely the optimization direction
discriminator welding image
[93] 2022 No image pairs data is and the update step size. Different optimizers appearing in the literature
required insufficient. are optimized and improved for these two. It can be seen from Fig. 10
[94] 2022 As an that Adam and SGD are more commonly used at present. This is because
enhancement tool Adam has the advantage of fast convergence speed, SGD has the
to restore the
balance of the
advantage of strong generalization ability, and each DL framework has
dataset better support for these two optimizers. A detailed comparison of DL
[95] 2020 Solve the problem optimizers can be found in the reference [140]. For the problem of weld
of collapse mode appearance detection, Li et al. [141] conducted a comparative study on
the effects of different optimizers. For the TIG welding defect identifi­
cation problem, Sekhar et al. [142] compared four optimizers. However,
The current DL is generally regarded as a black box model with poor
there is a lack in DLBWIR to analyze the characteristics of the welding
explainability. The literature [116] argues that a key component of AI
image data and select the optimizer accordingly.
systems is the ability to explain decisions, recommendations, pre­
The use of batch training can improve memory utilization and
dictions, actions and the process. DLBWIR mainly adopts the methods of
accelerate the training process of the model. It is the most widely used
feature map visualization and class activation mapping [117,118]
training technique under the condition of rapid development of hard­
respectively trying to explain what the model has learned and what is
ware technology. Compared with training based on single sample,
the basis for the model’s decision-making for individuals. Table 7
training based on batch can lock the direction of gradient descent faster
summarizes the commonly used model improvement methods in
and cause smaller training shock. Batch training has applications in
DLBWIR.
multiple tasks such as classification, detection, segmentation, and
tracking. However, the selection of batch size in the current research is
3.3. Training technology
based on experience, and there is no clear basis to select a batch size
suitable for specific DLBWIR tasks.
After acquiring the dataset and determining the hypothesis space
BN first calculates the mean and variance of the current batch, then
from the model structure, the model needs to be properly trained in
normalizes the current batch, and finally introduces a scale factor and a
order to find a hypothesis function that performs well. Fig. 10

Fig. 9. Backbone commonly used in DLBWIR: (a) classification, (b) detection, (c) segmentation, (d) tracking, and (e) others.

610
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

Table 7
The specific technologies for improved structure and their comparison.
Ref. Year Key elements Method Pros Cons

[119] 2021 Infrared thermal and visual image MML Complementary information, The primary and secondary relationship
[120] 2022 High-speed camera and coherent Limiting the hypothesis space between modalities is not clear.
light imaging
[121] 2022 Image and temperature
[122] 2020 Image and sound
[123] 2022 Eddy current detection and 3D
laser scanning
[124] 2020 Image, sound, voltage and current
[16] 2021 Image and text
[125] 2022 Handcrafted and learned features
[95] 2020 Inception and MobileNet ME Voting decisions with high robustness The training process is difficult and inefficient.
[126] 2021 Lenet5 and MLP
[127] 2019 Alexnet, Densenet, Resnet, VGG,
and Capsnet
[128] 2018 Lenet5 and Lenet5
[20] 2021 CNN, GRU, SVM and KNN
[16] 2021 Label text AM Pay attention to important information and optimize There is a lack of research on task justification
[129] 2019 Spatial the allocation of resources. and where to add it.
[130] 2020 Channel
[56] 2021 Spatial channel mixing
[131] 2022 Spatiotemporal and channel
attention
[132] 2021 Spatial correlation and Senet
[133] 2022 Bilinear attention
[134] 2022 Two modal cross-attention
[135] 2022 Four modal cross-attention
[136] 2022 Self-attention
[137] 2018 Feature map visualisation Explainability Visualization of the model’s learned knowledge and Lack of semantic-level explanations.
[136] 2022 Attention visualization basis for decision making.
[138] 2020 GradCAM
[82] 2020 CAM and Guided GradCAM
[139] 2022 Multi-scale fusion features-based
CAM

Fig. 10. Key technologies of the model training process.

translation factor to retain the learned features [143]. This method Wang et al. [153] used a Monte Carlo cross-validation method. The
makes the distribution of each layer’s input data relatively stable, re­ advantage of this method is that the limited sample space can be utilized
duces the model’s dependence on network initialization, accelerates the as much as possible and data interference can be eliminated to the
model’s convergence speed, and reduces the risk of model overfitting. greatest extent. However, the current literature does not consider the
Dropout refers to making some neurons stop working with a certain original data distribution when dividing the sub-datasets. Therefore, it is
probability when the signal propagates forward [144]. This can make necessary to consider a hierarchical cross-validation method that com­
the model not too dependent on some local features, thereby improving bines data distribution in future applications.
the generalization ability. However, dropout is designed for a more The essence of parameter regularization is to limit the parameter set
generalized neural network, ignoring the correlation of parameters in to be optimized, thereby standardizing the training direction of the
the convolution kernel. Therefore, Pan et al. [145] used a more targeted model, and limiting the scale of the hypothesis space to prevent the
Dropblock [146] method. model from overfitting. L1 [154] and L2 [155] parameter regularization
KCV generally divides the original dataset into k sub-datasets, a methods are two commonly used in the literature. However, the pa­
single sub-dataset is used for the verification, and other sub-datasets are rameters in the convolution kernel have a local correlation, and the L1
used for training model. KCV is repeated k times, and the result of and L2 regularization methods lack consideration of this correlation.
averaging k times can get a single estimate [147–152]. In addition, Therefore, Liu et al. [156] took into account the characteristics of the

611
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

local correlation of the parameters in the convolution kernel, and de­

signs a coarse-grained parameter regularization method for the classi­
fication of molten pool image.
The basic idea of the early-stopping method is to stop training when
the model’s performance on the validation set begins to decline [157]. In
this way, a model with better generalization performance can be ob­
tained. From the perspective of the limitation of the hypothesis space, it
can be considered that this method limits the parameter space of the
optimization process to a small neighborhood of the initial parameters,
so it can achieve a similar effect to L2 parameter regularization. The
disadvantage of early-stopping is that different methods are not taken to
solve the two problems of optimizing the loss function and reducing the
verification loss, which may face the problem of insufficient training Fig. 12. Realization environment of DLBWIR research.
loss.
complementarity at the same stage in the existing literature, and there is
3.4. Platform technology no research on multi-source asynchronous sensing to achieve cross-stage
information reasoning.
An important reason for the popularity and application of DL tech­ For data technology, the existing research includes two levels of key
nology in the industry is that a large number of excellent DL frameworks technologies for the welding image itself and the welding image dataset.
have been open sourced, which has brought convenience to the industry The widespread problems of pixel redundancy, visual interference, and
in terms of implementation. Fig. 11 summarizes the DL framework used drastic grey-scale changes in welding image have been addressed by
in DLBWIR related literature. Tensorflow [158] has a professional existing research through three types of methods: ROI, Prep, and NormL,
visualization platform for the training process, and its symbolic pro­ respectively. In response to the small size of the welding image dataset,
gramming features are more compatible with the underlying computer existing data technology has reduced the impact of small samples
code than other DL frameworks and easier to deploy in industrial sce­ through DA, TL, and GAN. However, the ROI, Prep, and NormL tech­
narios, hence it is currently the most widely used in DLBWIR [159]. niques for the welding image itself lack interaction with the DL training
Keras [160] and Pyorch [161] have also been widely used due to the process. The DA, TL, and GAN techniques for the welding image dataset
convenience brought by their highly modular packaging. In view of the are lacking in combination with the characteristics of welding image.
fact that most of the personnel who carry out WIR task come from the For example, Zhang et al. [29] added different noise types for different
engineering field, the Matlab framework [162], which is more popular penetration states and used a rotated-based DA method combining the
in the engineering field, has also been used in several ways. image characteristics of the three views.
Through the development of DL, it can be found that computing For model technology, existing research has been carried out mainly
power has always been a key factor hindering the development and on the basis of backbone structure and improved structure in the DL
landing of DL. Therefore, it can be seen from Fig. 12 that most of the field. The Lenet5 with simple structure and strong scalability is the most
studies in DLBWIR clearly indicate that there is a GPU accelerated widely used in DLBWIR. In order to further improve the advantages of
environment. Further, nearly 1/2 of the literature that specifies an DLBWIR, three improvement ideas of MML, AM and ME are respectively
operating system chooses Linux due to its high stability, scalability and used to supplement feature information sources, strengthen the learning
open source. In the absence of a local GPU acceleration environment, the of key features in welding image, and improve the basis for decision-
solution in reference [82,163] is to implement GPU acceleration based making.
on Google’s Colab cloud server. In order to obtain a DLBWIR model with strong generalization
ability, existing research includes the main techniques for the training
3.5. Brief summary process of DL models. However, the relevant training techniques lack
integration with the characteristics of the welding image and the
This section provides an overview of the key technologies involved in optimal choice of hyperparameters.
DLBWIR based on the DLBWIR research paradigm. For platform technology, open source frameworks and hardware
For sensing technology, existing research covers the three stages of acceleration environments provide platform support for the develop­
PRW, IW and POW, which can be used to acquire welding image in both ment of DLBWIR.
direct and indirect ways. Two types of images reflecting both internal
and external information about the weld can be acquired. However,
there are few studies on multi-source sensing to achieve information

Fig. 11. The DL framework used in DLBWIR.

612
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

4. Applications of DLBWIR the problem of small visual differences between different penetration
states by designing a label semantic attention that uses discriminative
This section provides an overview of the current status of DLBWIR inter-label texts to guide the model in learning discriminative visual
applications from a WIR task type perspective. Fig. 13 summarizes the features. In the reference [ 165,351-dimensional features were first
amount of DLBWIR posts in different task applications. The statistics extracted manually, followed by a stacked sparse autoencoder to
were based on 192 papers that had been searched and filtered by the establish a mapping relationship between the weld state and the input
rules described in the Introduction. The different tasks were distin­ features. Xue et al. [81] used Mobilenet as the skeleton network, which
guished on the basis of the supervisory information and task scenarios as greatly reduces the model size and offers the possibility of practical
described in Section 2.2. As the classification task is the basis and pre­ deployment of the model. The references [138,166] have improved the
requisite for several other tasks, the development of technology oriented explainability of models through model decision basis visualization and
towards the classification task generally precedes the other tasks. And feature map visualization respectively.
the cost of image-level labeling for classified task is lower than other The classification task for POW focuses on the identification of
tasks. Therefore, the applications oriented to classification task account external or internal defects in the weld seam to enable intelligent quality
for the vast majority, and applications oriented to object detection, inspection task. In the reference [ 167], three resampling methods were
image segmentation, target tracking, and other tasks decrease in turn. designed and compared for the problem of defect sample imbalance in
weld radiographic image recognition. Jiang et al. [168] designed four
4.1. Welding image classification pooling methods for different weld defect types and different regions in
the image. Kumaresan et al. [169] used a TL strategy to address the
Welding image classification is to discover the category of objects problem of a small sample of weld seam radiographic image. Miao et al.
contained in the welding image, which is a coarse-grained WIR task. [170] devised a two-stage identification method based on the fact that
The classification task for PRW is mainly the recognition of the bevel defects in weld appearance do not occur frequently, i.e. identifying the
shape and then the adjustment of the welding process based on the presence or absence of defects and then identifying the specific type of
recognition results. Tian et al. [164] first used different laser sources to defect. In the reference [36], a one-dimensional signal was first acquired
obtain weld bevel profile features and then designed a 10-layer CNN to using the acoustic emission technique, then a two-dimensional image
classify the bevel types. In the reference [15], a 6-layer CNN was was obtained by TFT, and then a CNN was built to identify whether the
designed to identify the bevel type in response to strong noise interfer­ welded steel pipe was subject to a natural gas leak. Zhang et al. [171]
ence in the field of view, and to narrow the parameter search for sub­ designed a 7-layer CNN and classified the weld microstructure images
sequent tasks accordingly. Cruz et al. [57] designed a 9-layer CNN to corresponding to the three defects. Feng et al. [132] firstly learned the
identify whether the parts to be welded are aligned. difference between normal and defect samples through comparative
The classification task for IW is mainly to identify the state of the learning, and secondly filtered the same features and extracts the
melting zone, and the purpose is to provide a basis for the traceability different features of the image pairs by means of the spatial correlation
and online control of welding quality. To better predict the melt depth, AM, which provides an effective attention for learning the differences
Jiao et al. [17] synthesised the current image with the previous 1/6 and between images of different armature welding surfaces. Hu et al. [172]
2/6 s images to reflect the dynamic welding phenomenon and built a adopted ELU activation function and grayscale adaptive pooling strat­
9-layer CNN to identify the image features. Zhang et al. [29] used egy in the task of X-ray defect recognition. To address the small sample
different data augmentation methods for different welding defects and problem faced by weld radiographic defect detection tasks, Di and Li
designed a 10-layer CNN to achieve classification of the three penetra­ [173] investigated the effect of freezing different layers on transfer
tion states. Miao et al. [33] designed a two-stage detection strategy to learning methods. Zhang et al. [174] weighted the loss function for the
improve the efficiency of CNN-based defect recognition for welding problem of imbalanced defect categories in weld radiographic images.
image acquired by TFT. The references [124,131] introduced temporal To address the problem of low quality of X-ray images, Liu et al. [175]
information in the identification of weld states and convert disturbing firstly embed defect images into local and global feature spaces, and
information for a single image into useful information to support model secondly maintain the consistency of the feature space based on a new
decisions. Liu et al. [137] found that the molten pool features extracted consistency strategy.
by the convolution kernel were redundant through feature map visual­ In summary, the main application scenarios of welding image clas­
ization, so it was integrated into the LSTM network to adaptively fuse sification include the identification of bevel types, melt zone states and
the redundant features. In order to learn richer molten pool features, Liu weld defects. The current research mainly focuses on issues such as noise
et al. [156] proposed a coarse-grained parameter regularization method, interference, lack of welding feature information, insufficient data vol­
taking into account that the weight parameters in the convolution kernel ume, small visual differences, high calculation costs, and poor explain­
have local correlations, which facilitated the convolutional kernel to be ability. However, the current research has been modeling DLBWIR as a
trained in the direction of increasing variance. Liu et al. [16] addresses single-label classification, and the actual welding image often contains a

Fig. 13. Posting volume for different DLBWIR tasks.

613
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

variety of label categories. Therefore, the authors believe that the ac­ in the welding image at the same time, which is a fine-grained WIR task.
curacy and interpretability of the classification model can be improved In view of the fact that the regression task can obtain specific values, this
by modeling DLBWIR as a multi-label classification task. section considers the regression problem as an object detection task and
summarizes it.
Lin et al. [176] first used a CNN to initially localize the laser spot
4.2. Welding object detection area, then increased the optical magnification of the camera and used
template matching for high precision localization. Wang et al. [177] first
Object detection is to determine the category and location of objects

Table 8
Application of DL in welding image object detection.
Ref. Year Stage Issue and challenge Method and result

Engineering Visual task Difficulty and challenge Backbone Key elements Data sources,
requirement volumes and
metrics

[176] 2018 IW Welding quality Laser spot detection Incomplete laser spot Not Two-stage detection of first coarse and Laboratory, 400
online image, mentioned then fine and Acc of 97.96
monitoring large field of view and %
complex calculation.
[177] 2021 Prediction of Back-bead width Weld width is difficult to Lenet5 Simultaneous acquisition of front and Laboratory,
weld forming prediction measure directly back images of the molten pool. Define 28494 and MSE
quality the width detection on the back of the of 0.096
weld as a regression problem.
[178] 2021 Welding quality The monitoring of Global information is Resnet Non-local attention Laboratory, DE of
online weld width and difficult to extract 0.1805 mm
monitoring reinforcement
[34] 2021 Welding quality Weld defect Lack of dataset YOLO Multiple dataset construction methods Simulation, 3892
online detection and MAP of 85 %
monitoring
[179] 2021 Laser beam Top and bottom The laser beam YOLO Dual vision sensing Laboratory, 240
absorbance diameter detection absorptance and GOF of 99.76
monitoring of keyhole inside a keyhole varies %
rapidly
[119] 2021 Welding quality Penetration status Arc interference and Faster Dual vision sensing and TL Laboratory, 2832
online recognition small sample size R-CNN and Acc of 95.58
monitoring %
[19] 2020 Safety monitoring Welding protective The object scale is SSD Multi-scale feature fusion and DSC Production,
of welding work equipment detection different and the model is 10000 and MAP
complex of 87.45 %
[180] 2022 Welding quality Penetration status The penetration state Lenet5- Fusion of timing information Laboratory,
online recognition changes dynamically and LSTM 28560 and DE of
monitoring is difficult to measure 3 mm
directly
[35] 2020 POW Automatic NDT Weld line detection Strong noise and complex SSD Parallel structured light and DSC Laboratory,
model 10000 and DE of
1 mm
[181] 2021 Connector quality Solder joint Small sample size, small Faster DA, TL and the region clustering-based Laboratory, 1654
detection detection object and the model is R-CNN anchor box generation and MAP of 94.1
complex %
[84] 2021 Body welding Solder joint Strong interference, small YOLO Feature pyramid, DA and DSC Production, 400
quality control detection object, small sample size and MAP of 89.32
and the model is complex %
[182] 2022 Automatic NDT Weld defect Low contrast and strong YOLO Contrast-limited adaptive histogram Production, 256
detection noise equalization, non-local mean and MAP of 51.2
denoising and GIOU loss %
[91] 2022 Weld seam Weld seam detection Small sample size Faster DA and TL Laboratory, 5000
polishing R-CNN and Acc 0 f 91.68
%
[78] 2022 Automatic NDT Weld defect Small sample size YOLO Stick-based DA Public dataset,
detection 3000 and MAP of
79.33 %
[183] 2022 Automatic NDT Weld defect Small object Faster Bidirectional feature pyramid Laboratory, MAP
detection R-CNN of 85.5 %
[184] 2022 Quality Weld defect Fine-grained YOLO Label smoothing Laboratory, 1100
inspection of detection and MAP of 90.1
body welded %
joints
[185] 2021 Rail quality Flash butt weld Small object YOLO Multi-scale feature fusion Production, 4529
detection detection and MAP of 87.41
%
[61] 2021 Steel pipe quality Weld defect Small sample size and YOLO Multiple DA methods and single stage Production,
detection detection real-time detection 30672 and MAP
of 98.7 %
[186] 2019 PCB board quality Solder joint Simultaneous multiple Mask Mask R-CNN and TL Laboratory, 88
detection detection recognition tasks with R-CNN and MAP of 97.4
small sample size %

614
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

used CNN to extract the visual features of the molten pool, and then while obtaining refined recognition results. Therefore, segmentation
established the correspondence between the visual features and the task is often used in quantitative detection of objects to be identified in
width of the back weld to achieve the key feature detection of the molten POW images.
pool object. In the reference [ 178], after CNN extracted the visual Yang et al. [187] designed an attention dense convolutional block
features of the molten pool, a regression method was used to simulta­ and used pixel segmentation to extract accurate laser stripes in a
neously detect the width and reinforcement of the deposited layer. In the strongly disturbed environment. Wang et al. [153] designed a segmen­
reference [34], six auxiliary datasets were created for the small sample tation network with a channel AM and a separable structure for
problem, and both porosity and slag defects were effectively detected extracting laser stripes.
based on YOLOv4. Kim et al. [179] collected the images of the top Mi et al. [188] used dilated convolution to increase the receptive
keyhole and the bottom aperture simultaneously, and then measured the field and designed a semantic segmentation network to segment the
aperture size based on YOLOv4, and finally established the relationship molten pool and the splash. In the literature [52], a segmentation
between the aperture size and the absorption rate of the laser beam. A network Epnet was designed to calculate the molten pool width and then
dual-input Faster R-CNN model with infrared thermal and visual images predict the forming width of the weld. Tan et al. [189] first divided the
as input, including a simultaneous feature extraction module, a con­ welding image into small blocks, and then designed CNN and threshold
volutional descriptor selection module and a synthetic feature-based neural network to segment each small block.
recognition module, was developed in the reference [119]. Lu et al. Yang et al. [190] proposed a branching structure for encoding spatial
[19] used an idea based on multi-scale feature fusion to improve the SSD and contextual features, respectively, for the small target and cross-scale
structure and apply it to welders’ behaviour detection. Yu et al. [ 180] problems of appearance defects. Ling et al. [191] designed a parallel
developed a CNN-LSTM model and extracted sufficient penetration in­ network to obtain semantic differences for the problem of small and
formation from the molten pool image sequence. irregular targets of weld defects, and used the Lovasz-softmax loss
Dong et al. [35] combined Mobilenet and SSD structures to achieve function in the training process. Li and Li [192] proposed a
lightweight detection of weld lines. In the reference [181], a high-frequency feature enhancement network for the problem of insig­
clustering-based anchor generation method was proposed and combined nificant pixel variation and small area of defects on the welded steel
with Faster R-CNN to achieve connector weld joint defect detection. Dai surface. Pandiyan et al. [193] designed an encoding-decoding type CNN
et al. [84] not only used a lightweight Mobilenet as the backbone of to realize the segmentation of four types of welds, which provides a basis
YOLOv3, but also designed a feature pyramid with cross-scale connec­ for the optimization of robot grinding parameters. Liu et al. [194] pro­
tions and employed a complete IOU loss to speed up model convergence posed a fusion block to extract the most important features in the three
and applied it to quality detection of resistance spot welds. Aiming at the dimensions of the normal map, and then constructed a CNN to segment
non-derivable problem of the IOU loss function, Yun et al. [182] first the key points of the welded stud from the normal map. Oh and Ki [26]
enhanced the X-ray image through contrast-limited adaptive histogram used the first generator to generate the weld segmentation map, and the
equalization, and secondly used GIOU as the loss function for training second generator converted it into an OM image, which realized the
the YOLO model. Liu et al. [91] used DA and TL methods to solve the direct prediction of the cross-sectional welding image from the laser
problem of insufficient weld appearance image data, and compared the welding process parameters. Due to the lack of image data of weld
effects of TL based on different backbone. Mery et al. [78] designed a DA appearance, Yang et al. [24] segmented the weld area based on U-Net,
method called Stick. For the problem of small defect targets in X-ray and then merged with the workpiece image to achieve data augmenta­
images, Chen et al. [183] adopted a bidirectional fusion feature pyramid tion. In order to segment laser welding defects of different sizes and
structure. Li et al. [184] adopted the label smoothing technique for the shapes, Zhu et al. [195] proposed a high-precision and high-efficiency
problem of small variation in the features of appearance defective. In the lightweight segmentation algorithm, which mainly includes a feature
reference [185], a minimum scale prediction network was designed to extraction module, an attention module, a localisation module and a
improve the detection accuracy of the model for small-scale objects and boundary anti-aliasing module. Jang et al. [196] regarded Resnet as the
applied to the detection of welding defects in railway tracks. In order to backbone of the segmentation network, and realized the measurement
identify multiple defects in the weld seam of steel pipes, Yang et al. [61] of acicular ferrite content that had a significant impact on the me­
used a single-stage inspection structure YOLOv5 to detect 8 types of chanical properties of carbon steel. In the reference [197], a Mask
defects. Wu et al. [186] first pre-trained the skeleton network Resnet101 R-CNN network for defect segmentation of weld X-ray images was
on the COCO dataset, and then detected the location of solder joint constructed based on transfer learning for the problem of small sample
defects based on Mask R-CNN. Table 8 summarizes the key elements of size of weld X-ray images. Zhang et al. [ 198] firstly proposed a com­
DL in the welding image object detection tasks. The identification of key pressed U-Net to segment the weld pad and the weld joint, secondly a
elements is based on the targeted approach used by the literature to template based approach to confirm the validity of each weld joint and
address the difficulty and challenge, the innovation point in the litera­ finally a heuristic algorithm to determine the defect type. In order to
ture, and the associated point with the theme of this paper. obtain more precise details of defects, Yang et al. [25] added cross-scale
In summary, the main application scenarios of welding object skip connections to U-Net. Wang and Shen [199] proposed a
detection include the detection of process quality, forming quality, weld boundary-aware semantic segmentation method, which combined
position and defect location. In addition to the common problems spatial pyramid pooling and spatial AM to improve the segmentation
similar to classification task, the current research also includes small accuracy and robustness of the welding area. For the problem of com­
object and cross-scale issues. However, although the current object plex background and low contrast of X-ray images, Yang et al. [200,201]
detection tasks are modeling and positioning the labels and positions of proposed an attention fusion module and a cross-scale connection
multiple objects in the welding image, the relationship between objects method. Dong et al. [90] proposed a novel unsupervised local deep
is not considered. Therefore, the authors believe that the reliability of feature learning method based on image segmentation to build a
DLBWIR can be improved through modeling relationship between network which can extract useful features from an image. Nowroth et al.
multiple objects. [202] proposed an improved method combining dilated convolution
and pyramidal pooling for segmenting OM images. Table 9 summarizes
4.3. Welding image segmentation the key elements of DL in the application of welding image segmentation
task.
Segmentation is the most refined WIR task, and its essence is a pixel- In summary, the main application scenarios of welding object seg­
by-pixel classification task. Due to the pixel-by-pixel classification mentation include segmentation of weld key points, melt zone objects,
characteristic of segmentation task, the model inference speed is slow and defect objects. In addition to facing common challenges similar to

615
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

Table 9
Application of DL in welding image segmentation.
Ref. Year Stage Issue and challenge Method and result

Engineering requirement Visual task Difficulty and challenge Backbone Key elements Data sources, volumes
and metrics

[187] 2022 PRW Weld seam tracking Laser stripe Strong interference, across U-Net Pixel segmentation Laboratory, 192 and
segmentation scales and unbalance task, BiConvLSTM Acc of 99.4 %
module and weighted
loss
[153] 2022 Weld seam tracking Laser stripe Strong interference and real- Erfnet Channel attention and Laboratory, 737 and
segmentation time DSC segmentation success
rate of 96 %
[188] 2021 IW Directed energy Segmentation of Strong interference and real- PSPnet Dilated convolution and Laboratory, 700 and
deposition monitoring molten pool and time pyramid pooling Acc of 94.71 %
spatter
[52] 2021 Width monitoring of Segmentation of Small sample size, real-time Enet DA, DSC and pyramid Laboratory, 600 and
cladding layer in arc molten pool and difficulty in contour pooling Acc of 94.18 %
additive manufacturing extraction
[189] 2020 Powder bed laser melting Segmentation of Strong interference and small VGG Threshold neural Laboratory, 210 and
process monitoring spatter sample size network and processing Rec of 80.48 %
in blocks
[190] 2022 POW Safety valve welding Segmentation of Small sample size and across Proposed Spatial features, Laboratory, 4708 and
quality inspection weld seam scales contextual features and Mean IOU of 86.704
feature fusion %
[191] 2022 PCB solder joint quality Solder joint defect Small sample size, small object Deeplab Parallel network and Laboratory, 340 and
inspection segmentation and irregular shapes Lovasz-softmax Acc of 96.78 %
[192] 2022 Surface defect detection Segmentation of Defects are insignificant and U-Net High-frequency Production, 1580 and
of welded rebar welding seam defect small in size components of low- Mean IOU of 86.2 %
level features
[193] 2019 Robot polishing endpoint Segmentation of Strong interference, unbalance Segnet DA, class- Laboratory, 2000 and
detection weld seam and robustness weighted loss and Acc of 99.35 %
regularization
[194] 2020 Welding stud detection Segmentation of There are noise spots, missing FCN Channel separation and Laboratory, 1427 and
stud key point spots, and the need to measure feature fusion Rec of 0.98 %
multiple studs simultaneously.
[26] 2020 Weld microstructure Segmentation of Small sample size and complex FCN CGAN and two-stage Laboratory, 1638 and
detection weld section OM image content image generation GOF of 93.6 %
[24] 2020 Weld position detection Segmentation of Complex and diverse industrial U-Net DA based on segmented Laboratory, 1940 and
weld seam environment image fusion Acc of 96.2 %
[195] 2021 Welding defect detection Segmentation of Defect size and shape are U-Net Multi-scale AM and DSC Production, 7263 and
of battery cover safety welding defect variable and require real-time Mean IOU of 84.67 %
hole
[196] 2020 Acicular ferrite content Segmentation of Small sample size and FCN DA and category Laboratory, 144 and
measurement weld microstructure imbalance sensitive loss Mean IOU of 86.2 %
[197] 2018 Automatic NDT Segmentation of Simultaneous multiple Mask Mask R-CNN and TL Public dataset, 88 and
welding seam defect recognition tasks with small R-CNN MAP of 85 %
sample size
[198] 2020 Quality inspection of Segmentation of Small sample size and real- U-Net DA and network Laboratory, 37,877
laser solder joints in solder joint and time compression and Acc of 99.85 %
battery production solder pad
[25] 2021 Automatic NDT Segmentation of Small sample size and across U-Net DA and skip Public dataset, 20 and
welding seam defect scales connections Acc of 99.8 %
[199] 2020 Automatic NDT Segmentation of Small sample size, strong U-Net DA, AM and atrous Production, 300 and
welding seam defect interference and variable spatial pyramid pooling Acc of 93.7 %
object shape
[200] 2022 Automatic NDT Segmentation of Complex backgrounds, U-Net Attention fusion block Public dataset, 20 and
welding seam defect poor contrast, weak texture, and cross-scale Acc of 99.8 %
and class imbalance connection
[90] 2021 Automatic NDT Segmentation of Difficulty in labeling data U-Net Unsupervised Production and
welding seam defect simulation, 24,111
and Acc of 89.55 %
[202] 2022 Weld microstructure Segmentation of Small sample size and across Deeplab TL, dilated convolution Laboratory, 282 and
detection weld section scales and pyramid pooling Mean IOU of 76.88 %

classification and detection tasks, this application is particularly affected approach is to first detect the area of the target, and then calculate the
by the difficulty of pixel-level labelling and the difficulty of segmenting object position to achieve the tracking task [203]. The end-to-end DL
pixels near boundaries. Therefore, the authors suggest that weakly su­ structure has not been applied, such as [204,205]. Furthermore, the
pervised welding image segmentation based on image-level labelling current research focuses on the seam tracking task of PRW.
information could be investigated in the future. In addition, the hard-to- Zou et al. [206] first input the welding image into the convolutional
segment samples can be mined explicitly by modelling the contextual long and short-term memory network, and then detected the welding
relationships of the boundary pixels. feature points through the similarity matching between multiple feature
layers. Zou and Zhou [86] firstly used CNN to learn welding image
4.4. Welding object tracking features, and secondly proposed a weld feature point detection method
based on multi-correlation filter collaboration considering the continu­
In the object tracking task of the welding image, the current ity of feature points in adjacent frames. Zou et al. [87] used the SSD

616
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

model to learn global and local weld features and combined it with RNN generate computed tomography results from eddy current testing re­
to learn contextual information of visual features to achieve weld sults. Aiming at the problem of industrial fakes caused by repeated im­
tracking under strong noise interference. Xiao et al. [207] used RCNN to aging in the field of NDT, Gao et al. [ 213] proposed an X-ray image
identify the weld type and detect the laser stripe ROI, then designed similarity comparison algorithm based on Siamese neural network.
different centre extraction algorithms for different types of welds and Table 11 summarizes the scenarios and key elements of DL in other
proposed a priori model to ensure the stability of the algorithm. In order subdivided WIR tasks.
to identify laser stripes and track welds in a strong interference envi­
ronment, Zhang et al. [208] designed a full convolutional network based
4.6. Brief summary
on the idea of segmentation. Li et al. [209] first used Mask R-CNN to
detect the weld position and then extracted the weld line to assist the
The application of DL in welding image classification, object detec­
robot in its inspection. Table 10 summarizes the key elements of DL in
tion, image segmentation and target tracking has realized the instanti­
the welding object tracking task.
ation of DLBWIR. However, different task types have different
In summary, the main application scenario for welding object
applicable scenarios. Table 12 summarizes the applicable scenarios for
tracking is the weld seam tracking task by capturing key points/objects
different task types and the pros and cons of the application process. In
in the area to be welded, and the current research focuses on the
general, DL mainly faces challenges at three levels of data (small sample,
problems of strong arc light interference and small samples. As far as
imbalance), feature (strong interference, fine-grained) and model (real-
application scenarios are concerned, the authors believe that the task
time, explainability) in various WIR tasks, and related solutions are
can be applied to the object tracking scenario of the IW stage. In terms of
developed around the key technologies mentioned in Section 3. How­
research points, the authors believe that tracking multiple objects in
ever, related research focuses on the application of DL methods in WIR,
welding image will be an interesting direction.
which lacks the guidance of welding domain knowledge.

4.5. Others 5. Public datasets for DLBWIR

Due to the difficulty of dataset construction, the DL method is less Welding image datasets are the basis for DLBWIR research. The
used in subdivided WIR tasks. publication of large datasets such as ImageNet [214] for classification
Wang et al. [92] designed a multi-scale feature fusion semantic tasks, COCO [215] or object detection tasks, SBD [216] for segmentation
segmentation network Res-Seg, which detected the edge of the molten tasks, and LaSOT [217] for tracking tasks is an important reason for the
pool with good continuity, and laid the foundation for the calculation of rapid development of DL in the CV field. Table 13 summarizes the public
width for molten pool and weld seam. Wang et al. [130] designed a datasets that exist in the field of DLBWIR. However, the public datasets
generative network to predict the future molten pool image, and then of DLBWIR still have the following shortcomings: (a) small amount of
input it into the regression network to guide weld reinforcement infor­ data; (b) few label categories; (c) unbalanced data; (d) lack of datasets
mation. In order to avoid extensive analytical calculations, Li et al. [23] for subdivided tasks.
used a LSTM network to establish a mapping mechanism from 2D im­
aging points in the imaging plane to 3D reflection points on the molten 6. Future perspectives
pool surface, enabling data-driven 3D reconstruction of the molten pool
surface. In the reference [31], a transfer learning method based on 6.1. Perspectives from the literature
multi-objective instance weighting was designed to detect welding
quality from the perspective of equipment fault diagnosis and was Approximately 50 % of the examined 192 articles gave a clear
validated with robotic spot welding tests. In order to know the crystal­ research outlook. As shown in Fig. 14, this section summarizes their
lographic orientation during welding, Singh et al. [ 210] established a relevant future perspectives in terms of the DLBWIR paradigm as well as
generative model from ultrasonic wave travel times to crystallographic system integration in the following six areas.
orientations. Mirzapour et al. [ 211] proposed a fast and flexible
denoising CNN for the denoising problem of weld radiographic images. 6.1.1. Sensing
In order to perform NDT with a low-cost method, Rohkohl et al. [ 212] Visual sensing has the advantage of containing a wealth of

Table 10
Application of DL in welding image object tracking.
Ref. Year Stage Issue and challenge Method and result

Engineering Visual task Difficulty and challenge Backbone Key elements Data sources, volumes and
requirement metrics

[206] 2020 PRW Weld seam Welding line Strong interference Faster Fusion of timing information and Laboratory, 46819 and TE
tracking feature point R-CNN periodically initialize network of 0.5 mm
detection
[86] 2019 Weld seam Welding line Strong interference and VGG Fusion of timing information, Laboratory, 0 and TE of
tracking feature point small sample size correlative filtering and TL 0.25 mm
detection
[87] 2021 Weld seam Welding line Strong interference and SSD Combine timing information, Laboratory, 40000 and TE
tracking feature point small sample size multi-scale features, DA and TL of 0.5 mm
detection
[207] 2019 Weld seam Weld line center Strong interference and Faster Two-step extraction and type- Laboratory, 4000 and TE of
tracking point detection robustness R-CNN related weld centre extraction 0.28 mm
method.
[208] 2019 Weld seam Weld line position Strong interference and FCN TL and multi-scale feature fusion Laboratory, 1000 and
tracking detection small sample size PRMSE of 3.87
[209] 2022 POW Robot inspection Weld seam position Complex structure and Mask Multi-scale feature fusion Laboratory and
detection non-salient objects R-CNN production, 1500 and TE of
20 mm

617
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

Table 11
Application of DL in other WIR tasks.
Ref. Year Stage Issue and challenge Method and result

Engineering Visual task Difficulty and challenge Backbone Key elements Data sources,
requirement volumes and metrics

[92] 2020 IW Weld width Edge detection of Small sample size and arc FCN DA and multi-scale feature fusion Laboratory, 1100
prediction molten pool interference and TE of 0.2 mm
[130] 2020 Weld Generation of Non-salient features Prednet Combine timing information and Laboratory, 25830
reinforcement molten pool image channel attention and SS of 93.27 %
prediction
[23] 2021 3D reconstruction 2D imaging points Strong interference, small LSTM Combine timing information and Laboratory and
of molten pool are mapped to 3D sample size and difficult to train the model based on the simulation, 5000
reflection points measure directly. simulation data and MSE of 0.002
[31] 2021 Welding product Robot spot welding Small sample size and Resnet Case-based TL Laboratory, 2160
fault diagnosis defect diagnosis transferability issue and Acc of 98.33 %
[210] 2022 Prediction of weld Image generation Determining the DNN Build a generative model from Simulation, 10000
tissue structure microstructure of weld ultrasonic wave travel times data and within 3◦
seam by NDT methods to crystallographic orientation.
[211] 2021 POW Weld radiographic Image denoising Image is blurry and details FFDnet Selective image enhancement Public dataset and
image recognition are not noticeable production, DWIQI
of 96 %
[212] 2022 Automatic NDT Image generation Comprehensive and U-Net Generate high-cost test results Laboratory, 294 and
inexpensive online from low-cost test results Acc of 93.7 %
inspection
[213] 2022 Fake film Image similarity Small sample size and VGG Comparisons by block Production, 342 and
identification comparison larger image resolution Acc of 93.458 %

multi-modal data types, 2) simultaneous multi-modal data acquisition,

Table 12
and 3) multi-modal data alignment.
Comparison of applications of DLBWIR in different tasks.
Task Application Pros Cons 6.1.2. Dataset
scenario
DL models have more parameters to be trained than traditional
Classification Recognition of Welding image Unable to achieve machine learning models. Increasing the amount of data in the welding
groove type, annotation is quantitative
image not only prevents overfitting of the DL model, but also improves
penetration state simple and wide recognition and poor
and weld defect range of explainability. the generalization ability of the model. The references [17,164,224,225]
type. applications. suggest that future research will require the creation of large-scale,
Detection Recognition of Preliminary Welding image is high-quality welding image datasets. This includes extending the label
the object quantitative difficult to label by categories of the welding image data, extending the test conditions to
location of recognition can be object and the number
obtain comprehensive welding image data, and extending the size of the
melting zone and achieved to meet of objects and
welding defects. the needs of backgrounds is not dataset based on both. In addition, Dai et al. [94] identified few-shot
automated balanced. learning and zero-shot learning as future research directions for
detection. DLBWIR.
Segmentation Quantitative It can realize Pixel-by-pixel
recognition of refined recognition annotation of welding
melting zone and provide image is difficult. 6.1.3. Model structure
objects and weld quantitative basis Model inference time . A typical intelligent welding systems contains multiple modules and
defect objects. for welding process is long. has very limited storage and computing resources that can be allocated
optimization. It has to the vision module. Therefore, the references [195,219,226] consider
strong
small and fast DLBWIR models as the way forward, resulting in easy
explainability.
Tracking Bevel and weld It can realize the Existing research deployment at the edge and on mobile as well as real-time applications.
seam key point recognition of achieves object In addition, most of the current DLBWIR is a data-driven model and
tracking. moving object. tracking through lacks the participation of expert knowledge. This not only causes a waste
frame-by-frame
of expert knowledge, but also makes it difficult to move towards strong
detection, which is
computationally intelligent welding [227]. Therefore, the related literature believes that
expensive. The scope the model structure driven by data and knowledge will be an important
of application is research direction, such as adding manual features [28], physical and
limited. mathematical principles [228,229], and welding mechanism [230].
Others Welding image It can solve the Welding image
denoising, subtask annotation is difficult
generation, and requirements in the and has limited 6.1.4. Training
edge detection. system task. application. Hyperparameters are crucial for the training, convergence and
recognition ability of the model. However, most of the current literature
is based on empirical selection of hyperparameters. The references [131,
information, however, there are some features that cannot be obtained 167] believes that the hyperparameters involved in the DLBWIR model
by visual sensing, such as information about defects inside the weld (such as learning rate, batch size, optimizer, etc.) need to be compared
seam during the welding process. Although the references [32,223] and optimized to obtain better recognition results.
launched a preliminary study on multi-modal sensing, it had only been a In addition, the references [20,230] believe that the problem of
simple fusion of multi-modal information at the classification stage and imbalance training is often faced in WIR tasks. To solve this problem, it
lacked an in-depth development of MML. Therefore, Jiang et al. [119] is necessary to design a class-related loss function to balance the prob­
decomposed this problem into three sub-problems: 1) selecting lem of drift in the learning process caused by the sample size.

618
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

Table 13
Public datasets for DLBWIR.
Ref. Year Stage Amount Label Task URL

[187] 2022 PRW 192 Laser stripe Segmentation https://aistudio.baidu.com/aistudio/

datasetdetail/106021/
[218] 2019 IW 33,254 Good weld, burn through, contamination, lack of fusion, Classification https://www.kaggle.com/danielbacioiu/tig-
misalignment and lack of penetration aluminium-5083
[55] 2019 45,058 Good weld, burn through, contamination, lack of fusion, lack of Classification https://www.kaggle.com/danielbacioiu/tig-
shielding gas and high travel speed stainless-steel-304
[219] 2021 4163 Short circuit and no short circuit Classification https://doi.org/10.17632/2nyjpb89bf.1
[220] 2015 POW 88 Non-defective, lack of penetration, porosity, slag inclusion and Detection and https://domingomery.ing.puc.cl/material/
crack Segmentation gdxray/
[171] 2019 162 Under-melt, beautiful-weld and over-melt Classification https://buffalo.box.com/s/
13ccf2flyaqnlzfcx0egy2vfwm0ym3sv
[193] 2019 2000 Four weld removal states and background Classification and https://data.mendeley.com/datasets/
Segmentation 2pcnt8kpw9/1
[198] 2020 37,877 Four tasks for welding pad and welding spot segmentation, Segmentation and https://pan.baidu.com/s/1c82KVsjTieMp-
three tasks for qualifying battery, obliquely placed welding Classification pYCCn8l5w
pad, welded through welding spot and placed-over-highly
electrode tab classification.
[221] 2021 37 Good and defect Classification https://www.mdpi.com/2075–4701/11/4/535/s1
[222] 2021 297 Spatter, lack of fusion and normal Segmentation https://www.kaggle.com/ruthka/maskrcnn-
mswelddefect
[61] 2021 3408 Blowhole, undercut, broken arc, crack, overlap, slag inclusion, Classification and https://github.com/huangyebiaoke/steel-pipe-
lack of fusion, and hollow bead Detection weld-defect-detection/releases/tag/1.0
[191] 2022 340 Missing solder, missing proteus, solder short, overturned Segmentation https://pan.baidu.com/share/init?
direction and solder ball. surl=_xLYqLI8HwxcsBO3i0wsyQ

6.1.6. Integration
The WIR module does not exist independently, but exists in the robot
welding system, the robot welding production cell, and the
manufacturing system. Therefore, Nogay and Akinci [235] consider the
DL model as one of the modules in the robot welding cell, and emphasize
its horizontal integration with other modules such as motion modules.
Wang et al. [236] emphasize the vertical integration between DLBWIR
module and control system. Cruz et al. [57] and Chu et al. [237]
emphasize the end-to-end integration of existing models with produc­
tion lines and even information systems through internet of things to
achieve interconnectivity.

6.2. Perspectives from the authors

6.2.1. Public datasets and pre-trained models

Data is the source of power for DL. While the importance of datasets
Fig. 14. Summary of research outlooks addressed in the literature. is also addressed in the DLBWIR literature and the creation of large,
high-quality datasets is identified as future work, there is a lack of
6.1.5. Well-trained model emphasis on the openness of datasets. From the history of DL in the field
In the current research, most of the welding images are collected of natural images, one can easily see the important role played by public
based on small-scale welding processes. Therefore, Nomura et al. [231] datasets in driving the development. Therefore, it is believed that it will
indicate that improving the generalization of the final model will be a be an important task to establish an open welding image data set.
research direction, that is, to ensure that the model has the generaliza­ Furthermore, in order to build a reliable welding image dataset, the
tion ability under variable operating conditions. Fioravanti et al. [232] authors believe that three aspects should be emphasized. In the dataset
believe that online learning methods should be explored to improve the design stage, both its practical values in solving domain problems and its
generalization ability of the model. In addition, Yang et al. [61] argue academic merits in advancing research have to be considered holisti­
that it is necessary to carry out few-shot learning research for the current cally. In the image acquisition stage, raw images containing rich infor­
situation of small samples of welding image, and automatically update mation about the quality of the weld should be acquired through the use
the system as new samples are added to the design model. of auxiliary light, filters and other ancillary equipment. In the image
Liu et al. [156] believe that many simplifications and idealization labelling stage, several experts should simultaneously label and vote on
assumptions have been made in the current DLBWIR research, such as the final label to improve the accuracy and consistency of the label. A
welding image data from the same distribution, and the model will not high-quality dataset should be characterized by a large amount of data, a
be disturbed and attacked. This will lead to insufficient flexibility and balanced sample size for each category, a large number of label cate­
robustness of our algorithm [233,234]. In the future, we should model gories, and a high label confidence, to satisfy the task of refined
more general situations and conduct attack tests. recognition.
Liu et al. [131] argue that the current DLBWIR is a black box model, It is extremely difficult to establish a large data set for each sub­
and people have no way of knowing the decision basis and reliability of divided scenario. The idea in the CV field for this problem is to pre-train
the model, which limits the application of DLBWIR in safety-sensitive the model structure on a large public dataset, and then fine-tune the
tasks such as aerospace and autonomous driving. Therefore, weights learned by the model based on the data in the target domain.
improving the explainability of the model is essential for developers to The pre-training model has played an important role in promoting image
optimize the model and increase the confidence of users. recognition tasks in the subdivision field. Therefore, it is not only

619
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

necessary to establish a public dataset in the field of welding image, but the KG, but also helps developers discover the causal relationship be­
also to pre-train a batch of classic models for WIR tasks in subdivision tween the welding image and the welding state. Thus, giving the DL
fields and teams that lack large-scale test conditions. model semantic explainability and causality reasoning capabilities, it
provides a basis for developers to improve the model and optimize the
6.2.2. Semantic explainability for DLBWIR welding process.
The welding process is carried out under clear process guidance. The This part takes the radiographic image detection of welding seams of
welding image contains rich welding semantic information, which is a aerospace structural parts as an example to illustrate the semantic
carrier that reflects the welding state. However, DLBWIR is an end-to- explainability and causal reasoning framework of DLBWIR based on KG.
end way to establish the association relationship between the welding As shown in Fig. 15, it can be divided into three layers: data layer,
image and the welding state. This black box model not only lacks the knowledge layer, and explanation layer. The data layer mainly includes
reflection of the decision-making basis of the model, but also lacks an welding context data and multi-modal data. The multi-modal data in the
explicit expression of the causal relationship between the welding image process design stage mainly includes process cards and text documents.
and the welding state. Although the references [82,137] used feature The multi-modal data in the simulation stage includes temperature field
map visualization and class activation mapping methods to show users simulation data, stress field simulation data, and molten pool flow
the features learned by DL and the basis for making decisions for specific simulation data. The PRW stage mainly includes point cloud and image
examples, this explanation is a coarse-grained explanation oriented by data. The IW stage includes weld pool image data, instantaneous current
human visual perception. This explanation neither include the job se­ data, welding sound data, and so on. The multi-modal data in the POW
mantics in the welding process, nor the reasoning and revealing of the stage includes weld radiographic image data, weld actual width data,
causal relationship of the welding status information [139]. Therefore, weld metallographic image data, etc. After acquiring welding data, in
people still cannot intuitively capture the decision-making basis of the the knowledge layer, knowledge extraction, knowledge representation,
model through this visual explanation and optimize the welding process semantic alignment, knowledge fusion, and knowledge storage opera­
accordingly. tions are performed on each modal data to form knowledge modules of
As a highly readable external knowledge carrier, the KG provides a each welding stage and synthesize to form the WDKG. In the explanation
way for the semantic expression of the welding manufacturing process layer, there are two parts that can be explained by semantics and the
[133]. Through KG, users can infer the associated path as the semantic basis for decision-making. Through causal reasoning technology,
explanation of the model decision. This type of association path not only various path dependencies that produce various defects can be found,
expresses the semantic information of the entities and relationships in thereby forming semantic explainability.

Fig. 15. KG-empowered semantic explanation for DLBWIR. The multimodal data involved in the welding context is integrated in the data layer. In the knowledge
layer, the sub-KG corresponding to each welding stage is constructed. Each sub-KG is fused at the explanation layer, and the basis for explanation reasoning of
welding defects is provided.

620
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

As shown in Fig. 15, for a weld seam, visual explanation can provide [3] Yan Z, Xu H, Huang P. Multi-scale multi-intensity defect detection in ray image of
weld bead. NDT E Int 2020;116:102342. https://doi.org/10.1016/j.
the basis for decisions made by the DL model. However, this description
ndteint.2020.102342.
is imprecise and does not contain engineering semantics. The fusion of [4] Fan X, Gao X, Liu G, Ma N, Zhang Y. Research and prospect of welding
knowledge modules formed by welding contexts can lead to semantic monitoring technology based on machine vision. Int J Adv Manuf Tech 2021;115:
explanation. In terms of semantic explanation, if the DL-based target 3365–91. https://doi.org/10.1007/s00170-021-07398-4.
[5] Sahoo S, Lo CY. Smart manufacturing powered by recent technological
detection algorithm detects porosity defects, it can correspond to se­ advancements: a review. J Manuf Syst 2022;64:236–50. https://doi.org/
mantic knowledge such as stress concentration, impure shielding gas, 10.1016/j.jmsy.2022.06.008.
too much welding current, etc. Suppose the DL-based target detection [6] Wang J, Xu C, Zhang J, Zhong R. Big data analytics for intelligent manufacturing
systems: a review. J Manuf Syst 2022;62:738–52. https://doi.org/10.1016/j.
algorithm detects an unperforated defect. In that case, it can correspond jmsy.2021.03.005.
to the semantic knowledge of low molten pool temperature, too small [7] Cai W, Wang J, Jiang P, Cao L, Mi G, Zhou Q. Application of sensing techniques
bevel angle, too small welding current, too large welding speed, etc. The and artificial intelligence-based methods to laser welding real-time monitoring: A
critical review of recent literature. J Manuf Syst 2020;57:1–18. https://doi.org/
explanation formed by semantic association of the causes of welding 10.1016/j.jmsy.2020.07.021.
defects is for front-line operators, and its essence is the semantic [8] Wang B, Hu SJ, Sun L, Freiheit T. Intelligent welding system technologies: State-
knowledge formed by in-depth annotation of multi-stage and multi- of-the-art review and perspectives. J Manuf Syst 2020;56:373–91. https://doi.
org/10.1016/j.jmsy.2020.06.020.
modal information. [9] Hinton GE, Osindero S, Teh YW. A fast learning algorithm for deep belief nets.
Neural Comput 2006;18(7):1527–54. https://doi.org/10.1162/
7. Conclusion neco.2006.18.7.1527.
[10] Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep
convolutional neural networks. Commun Acm 2017;60(6):84–90. https://doi.
DLBWIR is becoming a core technology to provide forecasting and org/10.1145/3065386.
decision making in intelligent welding systems. This paper reviewed a [11] Silver D, Huang A, Maddison CJ, Guez A, Sifre L, Driessche GVD, et al. Mastering
the game of Go with deep neural networks and tree search. Nature 2016;529
total of 192 previous publications to summarize the state-of-the-art of (7587):484–9. https://doi.org/10.1038/nature16961.
the DLBWIR research and application. The main contributions of this [12] Wang J, Ma Y, Zhang L, Gao RX, Wu D. Deep learning for smart manufacturing:
work are summarized as follows: methods and applications. J Manuf Syst, 48(C); 2018. p. 144–56. 〈https://doi.
org/10.1016/j.jmsy.2018.01.003〉.
[13] Günther J, Pilarski PM, Helfrich G, Shen H, Diepold K. Intelligent laser welding
1) The key enabling technologies for the DLBWIR, such as sensing, data, through representation, prediction, and control learning: an architecture with
model, training, and platform were outlined. Moreover, the charac­ deep neural networks and reinforcement learning. Mechatronics 2016;34:1–11.
teristics and scope of application of the relevant technologies were https://doi.org/10.1016/j.mechatronics.2015.09.004.
[14] Zhang Y, Yang YP, Zhang W, Na SJ. Advanced welding manufacturing: a brief
summarized. analysis and review of challenges and solutions. J Manuf Sci Eng 2020;142(11):
2) The current applications of DL in different WIR tasks are summa­ 110816. https://doi.org/10.1115/1.4047947.
rized, based on which, the most challenging is the lack of guidance [15] Du R, Xu Y, Hou Z, Shu J, Chen S. Strong noise image processing for vision-based
seam tracking in robotic gas metal arc welding. Int J Adv Manuf Tech 2019;101
from welding domain knowledge. (5):2135–49. https://doi.org/10.1007/s00170-018-3115-2.
3) Two promising research works, namely opening datasets and se­ [16] Liu T, Bao J, Zheng H, Wang J, Yang C, Gu J. Learning semantic-specific visual
mantic explainability are highlighted, which are the key to promote representation for laser welding penetration status recognition. Sci China Technol
Sc 2021:1–14. https://doi.org/10.1007/s11431-021-1848-7.
the in-depth research and practical application of DLBWIR. [17] Jiao W, Wang Q, Cheng Y, Yu R, Zhang Y. Prediction of weld penetration using
dynamic weld pool arc images. Weld J 2020;99(11):295–302. https://doi.org/
Despite the rapid growth, DLBWIR remains a rapidly evolving field. 10.29391/2020.99.027.
[18] Li C, Wang Q, Jiao W, Johnson, Zhang Y. Deep learning-based detection of
Many pressing issues should be addressed to enhance its viability in penetration from weld pool reflection images. Weld J 2020;99(9):239–45.
practice. In that regard, we hope this paper can provide insightful https://doi.org/10.29391/2020.99.022.
knowledge and guide to attract more researchers in addressing the [19] Lu H, Li C, Chen W, Jiang Z. A single shot multibox detector based on welding
operation method for biometrics recognition in smart cities. Pattern Recogn Lett
future directions of the DLBWIR research and application.
2020;140:295–302. https://doi.org/10.1016/j.patrec.2020.10.016.
[20] Knaak C, von Eßen J, Kröger M, Schulze F, Abels P, Gillner A. A spatio-temporal
ensemble deep learning architecture for real-time defect detection during laser
Declaration of Competing Interest welding on low power embedded computing boards. Sens-Basel 2021;21(12):
4205. https://doi.org/10.3390/s21124205.
The authors declare that they have no known competing financial [21] Jin C, Shin S, Yu J, Rhee S. Prediction model for back-bead monitoring during gas
metal arc welding using supervised deep learning. IEEE Access 2020;8:
interests or personal relationships that could have appeared to influence 224044–58. https://doi.org/10.1109/ACCESS.2020.3041274.
the work reported in this paper. [22] Lee J, Noh I, Lee J, Lee SW. Development of an explainable fault diagnosis
framework based on sensor data imagification: a case study of the robotic spot-
welding process. IEEE Trans Ind Inf 2022;18(10):6895–904. https://doi.org/
Acknowledgement 10.1109/TII.2021.3134250.
[23] Li L, Cheng F, Wu S. An LSTM-based measurement method of 3D weld pool
This research is partially funded by the Mainland-Hong Kong Joint surface in GTAW. Measurement 2021;171:108809. https://doi.org/10.1016/j.
measurement.2020.108809.
Funding Scheme (MHX/001/20), Innovation and Technology Commis­ [24] Yang L, Fan J, Liu Y, Li E, Peng J, Liang Z. Automatic detection and location of
sion (ITC), Hong Kong Special Administration Region of the People’s weld beads with deep convolutional neural networks. IEEE Trans Instrum Meas
Republic of China, National Key R&D Programs of Cooperation on Sci­ 2021;70:1–12. https://doi.org/10.1109/TIM.2020.3026514.
[25] Yang L, Wang H, Huo B, Li F, Liu Y. An automatic welding defect location
ence and Technology Innovation with Hong Kong, Macao and Taiwan algorithm based on deep learning. NDT E Int 2021;120:102435. https://doi.org/
(SQ2020YFE020182), Ministry of Science and Technology (MOST) of 10.1016/j.ndteint.2021.102435.
the People’s Republic of China, and the Centrally Funded Postdoctoral [26] Oh S, Ki H. Cross-section bead image prediction in laser keyhole welding of AISI
1020 steel using deep learning architectures. IEEE Access 2020;8:73359–72.
Fellowship Scheme (1-YXBM), The Hong Kong Polytechnic University.
https://doi.org/10.1109/ACCESS.2020.2987858.
[27] Yan Y, Liu D, Gao B, Tian G, Cai Z. A deep learning-based ultrasonic pattern
References recognition method for inspecting girth weld cracking of gas pipeline. IEEE Sens J
2020;20(14):7997–8006. https://doi.org/10.1109/JSEN.2020.2982680.
[28] Wu D, Hu M, Huang Y, Zhang P, Yu Z. In situ monitoring and penetration
[1] Luo Z, Dai JS, Wang C, Wang F, Tian Y, Zhao M. Predictive seam tracking with
prediction of plasma arc welding based on welder intelligence-enhanced deep
iteratively learned feedforward compensation for high-precision robotic laser
random forest fusion. J Manuf Process 2021;66:153–65. https://doi.org/
welding. J Manuf Syst 2012;31(1):2–7. https://doi.org/10.1016/j.
10.1016/j.jmapro.2021.04.007.
jmsy.2011.03.005.
[29] Zhang Z, Wen G, Chen S. Weld image deep learning-based on-line defects
[2] Zhang Y, Zhao Z, Zhang Y, Bai L, Wang K, Han J. Online weld pool contour
detection using convolutional neural networks for Al alloy in robotic arc welding.
extraction and seam width prediction based on mixing spectral vision. Opt Rev
2019;26(1):65–76. https://doi.org/10.1007/s10043-018-0479-3.

621
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

J Manuf Process 2019;45:208–16. https://doi.org/10.1016/j. digital image processing and deep learning techniques. Sens-Basel 2020;20(16):
jmapro.2019.06.023. 4505. https://doi.org/10.3390/s20164505.
[30] Yu R, He H, Han J, Bai L, Zhao Z, Lu J. Monitoring of back bead penetration based [58] Zhu H, Ge W, Liu Z. Deep Learning-based classification of weld surface defects.
on temperature sensing and deep learning. Measurement 2022;188:110410. Appl Sci-Basel 2019;9(16):3312. https://doi.org/10.3390/app9163312.
https://doi.org/10.1016/j.measurement.2021.110410. [59] Deng H, Cheng Y, Feng Y, Xiang J. Industrial laser welding defect detection and
[31] Lee K, Han S, Pham VH, Cho S, Choi HJ, Lee J, et al. Multi-objective instance image defect recognition based on deep learning model developed. Symmetry-
weighting-based deep transfer learning network for intelligent fault diagnosis. Basel 2021;13(9):1731. https://doi.org/10.3390/sym13091731.
Appl Sci-Basel 2021;11(5):2370. https://doi.org/10.3390/app11052370. [60] Li Y, Hu M, Wang T. Weld image recognition algorithm based on deep learning.
[32] Ren W, Wen G, Xu B, Zhang Z. A novel convolutional neural network based on Int J Pattern Recogn 2020;34(08):2052004. https://doi.org/10.1142/
time–frequency spectrogram of arc sound and its application on GTAW S0218001420520047.
penetration classification. IEEE T Ind Inf 2020;17(2):809–19. https://doi.org/ [61] Yang D, Cui Y, Yu Z, Yuan H. Deep learning based steel pipe weld defect
10.1109/TII.2020.2978114. detection. Appl Artif Intell 2021;35(15):1237–49. https://doi.org/10.1080/
[33] Miao R, Gao Y, Ge L, Jiang Z, Zhang J. Online defect recognition of narrow 08839514.2021.1975391.
overlap weld based on two-stage recognition model combining continuous [62] Cheng Y, Wang Q, Jiao W, Yu R, Chen S, Zhang Y, et al. Detecting dynamic
wavelet transform and convolutional neural network. Comput Ind 2019;112: development of weld pool using machine learning from innovative composite
103115. https://doi.org/10.1016/j.compind.2019.07.005. images for adaptive welding. J Manuf Process 2020;56:908–15. https://doi.org/
[34] Gantala T, Balasubramaniam K. Automated defect recognition for welds using 10.1016/j.jmapro.2020.04.059.
simulation assisted TFM imaging with artificial intelligence. J Nondestruct Eval [63] Wang D, Dong Y, Pan Y, Ma R. Machine vision-based monitoring methodology for
2021;40(1):1–24. https://doi.org/10.1007/s10921-021-00761-1. the fatigue cracks in U-rib-to-deck weld seams. IEEE Access 2020;8:94204–19.
[35] Dong Z, Mai Z, Yin S, Wang J, Yuan J, Fei Y. A weld line detection robot based on https://doi.org/10.1109/ACCESS.2020.2995276.
structure light for automatic NDT. Int J Adv Manuf Tech 2020;111(7):1831–45. [64] Yang L, Fan J, Huo B, Liu Y. Inspection of welding defect based on multi-feature
https://doi.org/10.1007/s00170-020-05964-w. fusion and a convolutional network. J Nondestruct Eval 2021;40(4):1–11.
[36] Song Y, Li S. Gas leak detection in galvanised steel pipe with internal flow noise https://doi.org/10.1007/s10921-021-00823-4.
using convolutional neural network. Process Saf Environ 2021;146:736–44. [65] Tao X, Peng L, Tao Y, Ye C. Inspection of defects in weld using differential array
https://doi.org/10.1016/j.psep.2020.11.053. ECT probe and deep learning algorithm. IEEE Trans Instrum Meas 2021;70:1–9.
[37] Zhi Z, Jiang H, Yang D, Gao J, Wang Q, Wang X, et al. An end-to-end welding https://doi.org/10.1109/TIM.2021.3099566.
defect detection approach based on titanium alloy time-of-flight diffraction [66] Choi SG, Hwang I, Kim YM, Kang B, Kang M. Prediction of the weld qualities
images. J Intell Manuf 2022. https://doi.org/10.1007/s10845-021-01905-w. using surface appearance image in resistance spot welding. Met-Basel 2019;9(8):
[38] Rout A, Deepak BBVL, Biswal BB. Advances in weld seam tracking techniques for 831. https://doi.org/10.3390/met9080831.
robotic welding: a review. Robot Cim-Int Manuf 2019;56:12–37. https://doi.org/ [67] Dong X, Taylor CJ, Cootes TF. Defect detection and classification by training a
10.1016/j.rcim.2018.08.003. generic convolutional neural network encoder. IEEE Trans Signal Proces 2020;68:
[39] Lei T, Rong Y, Wang H, Huang Y, Li M. A review of vision-aided robotic welding. 6055–69. https://doi.org/10.1109/TSP.2020.3031188.
Comput Ind 2020;123:103326. https://doi.org/10.1016/j. [68] Ajmi C, Zapata J, Elferchichi S, Zaafouri A, Laabidi K. Deep learning technology
compind.2020.103326. for weld defects classification based on transfer learning and activation features.
[40] Zhang Y, Wang Q, Liu Y. Adaptive intelligent welding manufacturing. Weld J Adv Mater Sci Eng 2020:1574350. https://doi.org/10.1155/2020/1574350.
2021;100(1):63–83. https://doi.org/10.29391/2021.100.006. [69] Guo S, Wang D, Chen J, Feng Z, Guo W. Predicting nugget size of resistance spot
[41] Bengio Y, Montreal U. Learning deep architectures for AI. Found Trends Mach Le welds using infrared thermal videos with image segmentation and convolutional
2009;2(1):1–87. https://doi.org/10.1561/2200000006. neural network. J Manuf Sci Eng 2022;144(2):021009. https://doi.org/10.1115/
[42] Deng L, Yu D. Deep learning: methods and applications. Found Trends Signal 1.4051829.
2014;7(3–4):197–387. https://doi.org/10.1561/2000000039. [70] Shorten C, Khoshgoftaar TM. A survey on image data augmentation for deep
[43] Schmidhuber J. Deep learning in neural networks: an overview. Neural Netw learning. J Big Data 2019;6(1):1–48. https://doi.org/10.1186/s40537-019-0197-
2015;61:85–117. https://doi.org/10.1016/j.neunet.2014.09.003. 0.
[44] Lecun Y, Bengio Y, Hinton GE. Deep learning. Nature 2015;521(7553):436–44. [71] Pan SJ, Yang Q. A survey on transfer learning. IEEE Trans Knowl Data En 2009;22
https://doi.org/10.1038/nature14539. (10):1345–59. https://doi.org/10.1109/TKDE.2009.191.
[45] Zhang W, Yang G, Lin Y, Ji C, Gupta MM. On definition of deep learning. In: IEEE [72] Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, et al.
World Automation Congress; 2018. p. 232–6. 〈https://doi.org/10.23919/WAC. Generative adversarial nets. Commun Acm 2020;63(11):139–44. https://doi.org/
2018.8430387〉. 10.1145/3422622.
[46] Oh S, Jung M, Lim C, Shin S. Automatic detection of welding defects using Faster [73] Radford A, Metz L, Chintala S. Unsupervised representation learning with deep
R-CNN. Appl Sci-Basel 2020;10(23):8629. https://doi.org/10.3390/ convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434;
app10238629. 2015.
[47] Zhang Z, Li B, Zhang W, Lu R, Wada S, Zhang Y. Real-time penetration state [74] Mirza M, Osindero S. Conditional generative adversarial nets. arXiv Prepr arXiv
monitoring using convolutional neural network for laser welding of tailor rolled 2014;1411:1784.
blanks. J Manuf Syst 2020;54:348–60. https://doi.org/10.1016/j. [75] Zhu JY, Park T, Isola P, Efros AA. Unpaired image-to-image translation using
jmsy.2020.01.006. cycle-consistent adversarial networks. In: IEEE International Conference on
[48] Shevchik S, Quang T, Meylan B, Farahani F, Olbinado M, Back A, et al. Supervised Computer Vision; 2017. p. 2242–51. 〈https://doi.org/10.1109/ICCV.2017.244〉.
deep learning for real-time quality monitoring of laser welding with X-ray [76] Mariani G, Scheidegger F, Istrate R, Bekas C, Malossi C. BAGAN: data
radiographic guidance. Sci-Rep-UK 2020;10:3389. https://doi.org/10.1038/ augmentation with balancing gan. arXiv preprint arXiv; 2018. p. 1803.09655.
s41598-020-60294-x. [77] Arjovsky M, Chintala S, Bottou L. Wasserstein generative adversarial networks.
[49] Wu D, Zhang P, Yu Z, Gao Y, Zhang H, Chen H, et al. Progress and perspectives of In: Proceedings of the 34th international conference on machine learning; 2017.
in-situ optical monitoring in laser beam welding: sensing, characterization and p. 214–23.
modeling. J Manuf Process 2022;75:767–91. https://doi.org/10.1016/j. [78] Mery D, Kaminetzky A, Golborne L, Figueroa S, Saavedra D. Target detection by
jmapro.2022.01.044. target simulation in X-ray testing. J Nondestruct Eval 2022;41:21. https://doi.
[50] Hou W, Zhang D, Wei Y, Guo J, Zhang X. Review on computer aided weld defect org/10.1007/s10921-022-00851-8.
detection from radiography images. Appl Sci-Basel 2020;10(5):1878. https://doi. [79] Cai W, Jiang P, Shu LS, Geng SN, Zhou Q. Real-time laser keyhole welding
org/10.3390/app10051878. penetration state monitoring based on adaptive fusion images using
[51] Xu Y, Wang Z. Visual sensing technologies in robotic welding: recent research convolutional neural networks. J Intell Manuf 2021. https://doi.org/10.1007/
developments and future interests. Sens Actuat A-Phys 2021;320:112551. s10845-021-01848-2.
https://doi.org/10.1016/j.sna.2021.112551. [80] Cheng YC, Wang QY, Jiao WH, Xiao J, Chen SJ, Zhang YM. Automated
[52] Wang Y, Lu J, Zhao Z, Deng W, Han J, Bai L, et al. Active disturbance rejection recognition of weld pool characteristics from active vision sensing. Weld J 2021;
control of layer width in wire arc additive manufacturing based on deep learning. 100(5):183–92. https://doi.org/10.29391/2021.100.015.
J Manuf Process 2021;67:364–75. https://doi.org/10.1016/j. [81] Xue B, Chang B, Du D. Multi-output monitoring of high-speed laser welding state
jmapro.2021.05.005. based on deep learning. Sens-Basel 2021;21(5):1626. https://doi.org/10.3390/
[53] Park JK, An WH, Kang DJ. Convolutional neural network based surface inspection s21051626.
system for non-patterned welding defects. Int J Precis Eng Man 2019;20(3): [82] Xia C, Pan Z, Fei Z, Zhang S, Li H. Vision based defects detection for keyhole TIG
363–74. https://doi.org/10.1007/s12541-019-00074-4. welding using deep learning with visual explanation. J Manuf Process 2020;56:
[54] Yang Y, Pan L, Ma J, Yang R, Zhu Y, Yang Y, et al. A high-performance deep 845–55. https://doi.org/10.1016/j.jmapro.2020.05.033.
learning algorithm for the automated optical inspection of laser welding. Appl [83] Ajmi C, Zapata J, Martínez-Álvarez JJ, Doménech G, Ruiz R. Using deep learning
Sci-Basel 2020;10(3):933. https://doi.org/10.3390/app10030933. for defect classification on a small weld X-ray image dataset. J Nondestruct Eval
[55] Bacioiu D, Melton G, Papaelias M, Shaw R. Automated defect classification of 2020;39(3):1–13. https://doi.org/10.1007/s10921-020-00719-9.
SS304 TIG welding process using visible spectrum camera and machine learning. [84] Dai W, Li D, Tang D, Jiang Q, Wang D, Wang H, et al. Deep learning assisted
NDT E Int 2019;107:102139. https://doi.org/10.1016/j.ndteint.2019.102139. vision inspection of resistance spot welds. J Manuf Process 2021;62:262–74.
[56] Ma G, Yu L, Yuan H, Xiao W, He Y. A vision-based method for lap weld defects https://doi.org/10.1016/j.jmapro.2020.12.015.
monitoring of galvanized steel sheets using convolutional neural network. [85] Zhou C, Li D, Wang P, Sun J, Huang Y, Li W. ACR-Net: attention integrated and
J Manuf Process 2021;64:130–9. https://doi.org/10.1016/j.jmapro.2020.12.067. cross-spatial feature fused rotation network for tubular solder joint detection.
[57] Cruz YJ, Rivas M, Quiza R, Beruvides G, Haber RE. Computer vision system for IEEE Trans Instrum Meas 2021;70:1–12. https://doi.org/10.1109/
welding inspection of liquefied petroleum gas pressure vessels based on combined TIM.2021.3094837.

622
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

[86] Zou Y, Zhou W. Automatic seam detection and tracking system for robots based [114] Hu J, Shen L, Albanie S, Sun G, Wu E. Squeeze-and-excitation networks. IEEE
on laser vision. Mechatronics 2019;63:102261. https://doi.org/10.1016/j. Trans Pattern Anal 2020;42(8):2011–23. https://doi.org/10.1109/
mechatronics.2019.102261. TPAMI.2019.2913372.
[87] Zou Y, Zhu M, Chen X. A robust detector for automated welding seam tracking [115] Woo S, Park J, Lee JY, Kweon IS. CBAM: convolutional block attention module.
system. J Dyn Sys Meas Control 2021;143(7):071001. https://doi.org/10.1115/ In: European Conference on Computer Vision; 2018. p. 3–19.
1.4049547. [116] Biran O, Cotton C . Explanation and justification in machine learning: a survey. In:
[88] Gonzalez-Val C, Pallas A, Panadeiro V, Rodriguez A. A convolutional approach to International Joint Conference on Artificial Intelligence; 2017. p. 8–13.
quality monitoring for laser manufacturing. J Intell Manuf 2020;31(3):789–95. [117] Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D.. Grad-cam:
https://doi.org/10.1007/s10845-019-01495-8. visual explanations from deep networks via gradient-based localization. In: IEEE
[89] Yang L, Wang Z, Gao S, Shi M, Liu B. Magnetic flux leakage image classification International Conference on Computer Vision; 2017. p. 618–626. 〈https://doi.
method for pipeline weld based on optimized convolution kernel. org/10.1109/ICCV.2017.74〉.
Neurocomputing 2019;365:229–38. https://doi.org/10.1016/j. [118] Zhou BL, Khosla A, Lapedriza A, Oliva A, Torralba A. Learning deep features for
neucom.2019.07.083. discriminative localization. In: IEEE Conference on Computer Vision and Pattern
[90] Dong X, Taylor CJ, Cootes TF. Automatic aerospace weld inspection using Recognition; 2016. p. 27–30. 〈https://doi.org/10.1109/CVPR.2016.319〉.
unsupervised local deep feature learning. Knowl-Based Syst 2021;221:106892. [119] Jiang R, Xiao R, Chen S. Prediction of penetration based on infrared thermal and
https://doi.org/10.1016/j.knosys.2021.106892. visual images during pulsed GTAW process. J Manuf Process 2021;69:261–72.
[91] Liu C, Chen S, Huang J. Machine vision-based object detection strategy for weld https://doi.org/10.1016/j.jmapro.2021.07.046.
area. SCI Program-Neth 2022;2022:1188974. https://doi.org/10.1155/2022/ [120] Ma D, Jiang P, Shu L, Geng S. Multi-sensing signals diagnosis and CNN-based
1188974. detection of porosity defect during Al alloys laser welding. J Manuf Syst 2022;62:
[92] Wang Y, Han J, Lu J, Bai L, Zhao Z. TIG stainless steel molten pool contour 334–46. https://doi.org/10.1016/j.jmsy.2021.12.004.
detection and weld width prediction based on res-seg. Met-Basel 2020;10(11): [121] Jamnikar ND, Liu S, Brice C, Zhang X. In-process comprehensive prediction of
1495. https://doi.org/10.3390/met10111495. bead geometry for laser wire-feed DED system using molten pool sensing data and
[93] Ai J, Tian L, Bai L, Zhang J. Data augmentation by a CycleGAN-based extra- multi-modality CNN. Int J Adv Manuf Tech 2022;121:903–17. https://doi.org/
supervised model for nondestructive testing. Meas Sci Technol 2022;33:045017. 10.1007/s00170-022-09248-3.
https://doi.org/10.1088/1361-6501/ac3ec3. [122] Wu D, Huang Y, Zhang P, Yu Z, Chen H, Chen S. Visual-acoustic penetration
[94] Dai W, Li D, Tang D, Wang H, Peng Y. Deep learning approach for defective spot recognition in variable polarity plasma arc welding process using hybrid deep
welds classification using small and class-imbalanced datasets. Neurocomputing learning approach. IEEE Access 2020;8:120417–28. https://doi.org/10.1109/
2022;477:46–60. https://doi.org/10.1016/j.neucom.2022.01.004. ACCESS.2020.3005822.
[95] Le X, Mei J, Zhang H, Zhou B, Xi J. A learning-based approach for surface defect [123] Miao R, Shan Z, Zhou Q, Wu Y, Ge L, Zhang J, et al. Real-time defect identification
detection using small image datasets. Neurocomputing 2020;408:112–20. of narrow overlap welds and application based on convolutional neural networks.
https://doi.org/10.1016/j.neucom.2019.09.107. J Manuf Syst 2022;62:800–10. https://doi.org/10.1016/j.jmsy.2021.01.012.
[96] Lecun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to [124] Chen C, Xiao R, Chen H, Lv N, Chen S. Prediction of welding quality
document recognition. Prpc IEEE 86(11); 1998. p. 2278–324. 〈https://doi. characteristics during pulsed GTAW process of aluminum alloy by multisensory
org/10.1109/5.726791〉. fusion and hybrid network model. J Manuf Processes, 68(B); 2020. p. 209–24.
[97] He K, Zhang X, Ren SQ, Sun J. Deep residual learning for image recognition. In: 〈https://doi.org/10.1016/j.jmapro.2020.08.028〉.
IEEE conference on computer vision and pattern recognition; 2016. p. 770–778. [125] Hou W, Rao L, Zhu A, Zhang D. Feature fusion for weld defect classification with
〈https://doi.org/10.1109/CVPR.2016.90〉. small dataset. J Sens 2022;2022:8088202. https://doi.org/10.1155/2022/
[98] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale 8088202.
image recognition. arXiv Prepr arXiv 2015;1409:1556. [126] Kershaw J, Yu R, Zhang YM, Wang P. Hybrid machine learning-enabled adaptive
[99] Howard AG Zhu M, Chen B, Kalenichenko D, Wang WJ, Weyand T., et al. welding speed control. J Manuf Process 2021;71:374–83. https://doi.org/
MobileNets: efficient convolutional neural networks for mobile vision 10.1016/j.jmapro.2021.09.023.
applications. arXiv preprint arXiv: 1704.04861; 2017. [127] Feng Y, Chen Z, Wang D, Chen J, Feng Z. DeepWelding: a deep learning enhanced
[100] Girshick R, Donahue J., Darrell T., Malik J. Rich feature hierarchies for accurate approach to GTAW using multisource sensing images. IEEE Trans Ind Inf 2019;16
object detection and semantic segmentation. In: IEEE Conference on Computer (1):465–74. https://doi.org/10.1109/TII.2019.2937563.
Vision and Pattern Recognition; 2014. p. 580–7. 〈https://doi.org/10.1109/CVP [128] Cai N, Cen G, Wu J, Li F, Wang H, Chen X. SMT solder joint inspection via a novel
R.2014.81〉. cascaded convolutional neural network. IEEE Trans Comp Pack Man 2018;8(4):
[101] Girshick, R. Fast R-CNN. In: IEEE International Conference on Computer Vision. 670–7. https://doi.org/10.1109/TCPMT.2018.2789453.
2015: 1440–1448. 〈https://doi.org/10.1109/ICCV.2015.169〉. [129] Yang L, Wang Z, Gao S. Pipeline magnetic flux leakage image detection algorithm
[102] Ren SQ, He K, Girshick R, Sun J. Faster R-CNN: towards real-time object detection based on multiscale SSD network. IEEE Trans Ind Inf 2019;16(1):501–9. https://
with region proposal networks. IEEE Trans Pattern Anal 2017;39(6). https://doi. doi.org/10.1109/TII.2019.2926283.
org/10.1109/TPAMI.2016.2577031. 1137-49. [130] Wang Y, Zhang C, Lu J, Bai L, Zhao Z, Han J. Weld reinforcement analysis based
[103] Redmon J, Divvala S, Girshick R, Farhadi A. You only look once: unified, real- on long-term prediction of molten pool image in additive manufacturing. IEEE
time object detection. In: IEEE Conference on Computer Vision and Pattern Access 2020;8:69908–18. https://doi.org/10.1109/ACCESS.2020.2986130.
Recognition; 2016. p. 779–88. 〈https://doi.org/10.1109/CVPR.2016.91〉. [131] Liu T, Wang J, Huang X, Lu Y, Bao J. 3DSMDA-Net: an improved 3DCNN with
[104] Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, et al. SSD: single shot separable structure and multi-dimensional attention for welding status
multibox detector. In: European Conference on Computer Vision; 2016. p. 21–37. recognition. J Manuf Syst 2022;62:811–22. https://doi.org/10.1016/j.
〈https://doi.org/10.1007/978-3-319-46448-0_2〉. jmsy.2021.01.017.
[105] Shelhamer E, Long J, Darrell T. Fully convolutional networks for semantic [132] Feng T, Huang S, Liu J, Wang J, Fang X. Welding surface inspection of armatures
segmentation. IEEE Trans Pattern Anal 2017;39(4):640–51. https://doi.org/ via CNN and image comparison. IEEE Sens J 2021;21(19):21696–704. https://
10.1109/TPAMI.2016.2572683. doi.org/10.1109/JSEN.2021.3079334.
[106] Ronneberger O, Fischer P. Brox T. U-Net: convolutional networks for biomedical [133] Li Q, Yang B, Wang S, Zhang Z, Tang X, Zhao C. A fine-grained flexible graph
image segmentation. In: International conference on medical image computing convolution network for visual inspection of resistance spot welds using cross-
and computer assisted intervention; 2015. p. 234–41. 〈https://doi.org/10.1007/9 domain features. J Manuf Process 2022;78:319–29. https://doi.org/10.1016/j.
78-3-319-24574-4_28〉. jmapro.2022.04.025.
[107] Yu R, Kershaw J, Wang P, Zhang Y. Real-time recognition of arc weld pool using [134] Li W, Qi J, Sun H. TAF2-Net: triple-branch attentive feature fusion network for
image segmentation network. J Manuf Process 2021;72:159–67. https://doi.org/ ultrasonic flaw detection. IEEE Trans Instrum Meas 2022;71:1–12. https://doi.
10.1016/j.jmapro.2021.10.019. org/10.1109/TIM.2022.3150592.
[108] He K, Gkioxari G, Dollár P, Girshick R. Mask R-CNN. In: IEEE International [135] Li Z, Chen H, Ma X, Chen H, Ma Z. Triple pseudo-siamese network with hybrid
Conference on Computer Vision; 2017. p. 2961–9. 〈https://doi.org/10.1109/ attention mechanism for welding defect detection. Mater Des 2022;217:110645.
ICCV.2017.322〉. https://doi.org/10.1016/j.matdes.2022.110645.
[109] Ma C, Huang J, Yang X, Yang M. Hierarchical convolutional features for visual [136] Wang Z, Chen H, Zhong Q, Lin S, Wu J, Xu M, et al. Recognition of penetration
tracking. In: IEEE International Conference on Computer Vision; 2015. p. state in GTAW based on vision transformer using weld pool image. Int J Adv
3074–3082. 〈https://doi.org/10.1109/ICCV.2015.352〉. Manuf Tech 2022;119:5439–52. https://doi.org/10.1007/s00170-021-08538-6.
[110] Nam H, Han B. Learning multi-domain convolutional neural networks for visual [137] Liu T, Bao J, Wang J, Zhang Y. A hybrid CNN–LSTM algorithm for online defect
tracking. In: IEEE Conference on Computer Vision and Pattern Recognition; 2016. recognition of CO2 welding. Sens-Basel 2018;18(12):4369. https://doi.org/
p. 4293–4302. 〈https://doi.org/10.1109/CVPR.2016.465〉. 10.3390/s18124369.
[111] Danelljan M, Bhat G, Khan F, Felsberg MECO.: efficient convolution operators for [138] Božič A, Kos M, Jezeršek M. Power control during remote laser welding using a
tracking. In: IEEE Conference on Computer Vision and Pattern Recognition; 2017. convolutional neural network. Sens-Basel 2020;20(22):6658. https://doi.org/
p. 6931–6939. 〈https://doi.org/10.1109/CVPR.2017.733〉. 10.3390/s20226658.
[112] Nazir Q, Shao C. Online tool condition monitoring for ultrasonic metal welding [139] Liu T, Zheng H, Bao J, Zheng P, Wang J, Yang C, et al. An explainable laser
via sensor fusion and machine learning. J Manuf Process 2021;62:806–16. welding defect recognition method based on multi-scale class activation mapping.
https://doi.org/10.1016/j.jmapro.2020.12.050. IEEE Trans Instrum Meas 2022;71:1–12. https://doi.org/10.1109/
[113] Jaderberg M, Simonyan K, Zisserman A, Kavukcuoglu K Spatial transformer TIM.2022.3148739.
networks. In: International Conference on Neural Information Processing [140] Ruder S. An overview of gradient descent optimization algorithms. arXiv preprint
Systems; 2015. p. 2017–25. arXiv:1609.04747; 2016.

623
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

[141] Li Y, Mu H, Polden H, Li H, Wang L, Xia C, Pan Z. Towards intelligent monitoring [169] Kumaresan S, Aultrin KSJ, Kumar SS, Anand MD. Transfer learning with CNN for
system in wire arc additive manufacturing: a surface anomaly detector on a small classification of weld defect. IEEE Access 2021;9:95097–108. https://doi.org/
dataset. Int J Adv Manuf Tech 2022;120:5225–42. https://doi.org/10.1007/ 10.1109/ACCESS.2021.3093487.
s00170-022-09076-5. [170] Miao R, Jiang Z, Zhou Q, Wu Y, Gao Y, Zhang J, et al. Online inspection of narrow
[142] Sekhar R, Sharma D, Shah P. Intelligent classification of tungsten inert gas overlap weld quality using two-stage convolution neural network image
welding defects: a transfer learning approach. Front Mech Eng 2022;8:824038. recognition. Mach Vis Appl 2021;32(1):27. https://doi.org/10.1007/s00138-020-
https://doi.org/10.3389/fmech.2022.824038. 01158-2.
[143] Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by [171] Zhang B, Jaiswal P, Rai R, Guerrier P, Baggs G. Convolutional neural network-
reducing internal covariate shift. In: International Conference on Machine based inspection of metal additive manufacturing parts. Rapid Prototyp J 2019;25
Learning; 2015. p. 448–56. (3):530–40. https://doi.org/10.1108/RPJ-04-2018-0096.
[144] Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: a [172] Hu A, Wu L, Huang J, Fan D, Xu Z. Recognition of weld defects from X-ray images
simple way to prevent neural networks from overfitting. J Mach Learn Res 2014; based on improved convolutional neural network. Multimed Tools Appl 2022;81:
15(1):1929–58. 15085–102. https://doi.org/10.1007/s11042-022-12546-3.
[145] Pan H, Pang Z, Wang Y, Wang Y, Chen L. A new image recognition and [173] Di X, Li L. Image defect recognition method based on deep learning network. Mob
classification method combining transfer learning algorithm and mobilenet model Inf Syst 2022;2022:5696334. https://doi.org/10.1155/2022/5696334.
for welding defects. IEEE Access 2020;8:119951–60. https://doi.org/10.1109/ [174] Zhang Z, Liu W, Sun X. Image recognition of limited and imbalanced samples
ACCESS.2020.3005450. based on transfer learning methods for defects in welds. Proc Inst Mech Eng B-J
[146] Ghiasi G, Lin TY, Le Q.V. Dropblock: a regularization method for convolutional Eng 2022. https://doi.org/10.1177/09544054221082779.
networks. arXiv preprint arXiv:1810.12890; 2018. [175] Liu J, Liu X, Qu F, Zhang H, Zhang L. A defect recognition method for low-quality
[147] Yang L, Jiang H. Weld defect classification in radiographic images using unified weld image based on consistent multiscale feature mapping. IEEE Trans Instrum
deep neural network with multi-level features. J Intell Manuf 2020;32(2):459–69. Meas 2022;71:1–11. https://doi.org/10.1109/TIM.2022.3171609.
https://doi.org/10.1007/s10845-020-01581-2. [176] Lin CS, Huang YC, Chen SH, Hsu YL, Lin YC. The application of deep learning and
[148] Chen Y, Su S, Yang H. Convolutional neural network analysis of recurrence plots image processing technology in laser positioning. Appl Sci-Basel 2018;8(9):1542.
for anomaly detection. Int J Bifurcat Chaos 2020;30(01):2050002. https://doi. https://doi.org/10.3390/app8091542.
org/10.1142/S0218127420500029. [177] Wang Q, Jiao W, Wang P, Zhang Y. A tutorial on deep learning-based data
[149] Zhang B, Liu S, Shin YC. In-process monitoring of porosity during laser additive analytics in manufacturing through a welding case study. J Manuf Process 2021;
manufacturing process. Addit Manuf 2019;28:497–505. https://doi.org/10.1016/ 63:2–13. https://doi.org/10.1016/j.jmapro.2020.04.044.
j.addma.2019.05.030. [178] Wang Y, Xu X, Zhao Z, Deng W, Han J, Bai L, et al. Coordinated monitoring and
[150] Liu J, Guo F, Zhang Y, Hou B, Zhou H. Defect classification on limited labeled control method of deposited layer width and reinforcement in WAAM process.
samples with multiscale feature fusion and semi-supervised learning. Appl Intell J Manuf Process 2021;71:306–16. https://doi.org/10.1016/j.
2021:1–16. https://doi.org/10.1007/s10489-021-02917-y. jmapro.2021.09.033.
[151] Li S, Gao J, Zhou E, Pan Q, Wang X. Deep learning-based fusion hole state [179] Kim H, Nam K, Oh S, Ki H. Deep-learning-based real-time monitoring of full-
recognition and width extraction for thin plate TIG welding. Weld World 2022;66: penetration laser keyhole welding by using the synchronized coaxial observation
1329–47. https://doi.org/10.1007/s40194-022-01287-4. method. J Manuf Process 2021;68:1018–30. https://doi.org/10.1016/j.
[152] Xiao M, Yang B, Wang S, Chang Y, Li S, Yi G. Research on recognition methods of jmapro.2021.06.029.
spot-welding surface appearances based on transfer learning and a lightweight [180] Yu R, Kershaw J, Wang P, Zhang Y. How to accurately monitor the weld
high-precision convolutional neural network. J Intell Manuf 2022. https://doi. penetration from dynamic weld pool serial images using CNN-LSTM deep
org/10.1007/s10845-022-01909-0. learning model? IEEE Robot Autom Let, 7(3); 2022. p. 6519–25. 〈https://doi.
[153] Wang B, Li F, Lu R, Ni X, Zhu W. Weld feature extraction based on semantic org/10.1109/LRA.2022.3173659〉.
segmentation network. Sens-Basel 2022;22(11):4130. https://doi.org/10.3390/ [181] Zhang K, Shen H. Solder joint defect detection in the connectors using improved
s22114130. Faster-RCNN algorithm. Appl Sci-Basel 2021;11(2):576. https://doi.org/
[154] Park MY, Hastie T. L1–regularization path algorithm for generalized linear 10.3390/app11020576.
models. J R Stat Soc B 2007;69(4):659–77. https://doi.org/10.1111/j.1467- [182] Yun GH, Oh SJ, Shin SC. Image preprocessing method in radiographic inspection
9868.2007.00607.x. for automatic detection of ship welding defects. Appl Sci-Basel 2022;12(1):123.
[155] Cortes C, Mohri M, Rostamizadeh A. L2 regularization for learning kernels. arXiv https://doi.org/10.3390/app12010123.
Prepr arXiv 2012;1205:2653. [183] Chen Y, Wang J, Wang G. Intelligent welding defect detection model on improved
[156] Liu T, Bao J, Wang J, Zhang Y. A coarse-grained regularization method of R-CNN. IETE J Res 2022:1–10. https://doi.org/10.1080/
convolutional kernel for molten pool defect identification. J Comput Inf Sci Eng 03772063.2022.2040387.
2020;20(2):021005. https://doi.org/10.1115/1.4045294. [184] Li H, Li B, Wen X, Wang H, Wang X, Zhang S, et al. A detection and configuration
[157] Prechelt L. Early stopping-but when? Neural Netw: Tricks Trade 1998:55–69. method for welding completeness in the automotive body‑in‑white panel based
https://doi.org/10.1007/3-540-49430-8_3. on digital twin. Sci Rep-UK 2022;12:7929. https://doi.org/10.1038/s41598-022-
[158] Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J., et al. Tensorflow: a system 11440-0.
for large-scale machine learning. In: Operating Systems Design and [185] Chen Z, Wang Q, Yang K, Yu T, Yao J, Liu Y, et al. Deep learning for the detection
Implementation; 2016. p. 265–83. and recognition of rail defects in ultrasound B-scan images. Transp Res Rec 2021;
[159] Parvat A., Chavan J., Kadam S., Dev S., Pathak V. A survey of deep-learning 2675(11):888–901. https://doi.org/10.1177/03611981211021547.
frameworks. In: International Conference on Inventive Systems and Control; [186] Wu H, Gao W, Xu X. Solder joint recognition using mask R-CNN method. IEEE
2017. p. 1–7. 〈https://doi.org/10.1109/ICISC.2017.8068684〉. Trans Comp Pack Man 2019;10(3):525–30. https://doi.org/10.1109/
[160] Gulli A, Pal S. Deep learning with Keras. Packt Publishing Ltd; 2017. TCPMT.2019.2952393.
[161] Paszke A, Gross S, Massa F, Lerer A, Bradury J, Chanan G, et al. Pytorch: an [187] Yang L, Fan J, Huo B, Li E, Liu Y. Image denoising of seam images with deep
imperative style, high-performance deep learning library. Adv Neural Inf Process learning for laser vision seam tracking. IEEE Sens J 2022;22(6):6098–107.
Syst 2019;32:8026–37. https://doi.org/10.1109/JSEN.2022.3147489.
[162] Han CM, Kim TW, Choi HW. Assessment of laser joining quality by visual [188] Mi J, Zhang Y, Li H, Shen S, Yang Y, Song C, et al. In-situ monitoring laser based
inspection, computer Simulation, and deep learning. Appl Sci-Basel 2021;11(2): directed energy deposition process with deep convolutional neural network.
642. https://doi.org/10.3390/app11020642. J Intell Manuf 2021:1–11. https://doi.org/10.1007/s10845-021-01820-0.
[163] Mishra A, Patti A. Deep convolutional neural network modeling and laplace [189] Tan Z, Fang Q, Li H, Liu S, Zhu W, Yang D. Neural network based image
transformation algorithm for the analysis of surface quality of friction stir welded segmentation for spatter extraction during laser-based powder bed fusion
joints. ADCAIJ-Adv Distrib C, 10(3); 2021. 3–7-320. 〈http://hdl.handle.net/103 processing. Opt Laser Technol 2020;130:106347. https://doi.org/10.1016/j.
66/147243〉. optlastec.2020.106347.
[164] Tian Y, Liu H, Li L, Yuan G, Feng J, Chen Y, et al. Automatic identification of [190] Yang Y, He Y, Guo H, Chen Z, Zhang L. Semantic segmentation supervised deep-
multi-type weld seam based on vision sensor with silhouette-mapping. IEEE Sens learning algorithm for welding-defect detection of new energy batteries. Neural
J 2020;21(4):5402–12. https://doi.org/10.1109/JSEN.2020.3034382. Comput Appl 2022. https://doi.org/10.1007/s00521-022-07474-0.
[165] Zhang Y, You D, Gao X, Wang C, Li Y, Gao PP. Real-time monitoring of high- [191] Ling Z, Zhang A, Ma D, Shi Y, Wen H. Deep siamese semantic segmentation
power disk laser welding statuses based on deep learning framework. J Intell network for PCB welding defect detection. IEEE Trans Instrum Meas 2022;71:
Manuf 2019;31:799–814. https://doi.org/10.1007/s10845-019-01477-w. 1–11. https://doi.org/10.1109/TIM.2022.3154814.
[166] Jia CB, Liu XF, Zhang GK, Zhang Y, Yu CH, Wu CS. Penetration/keyhole status [192] Li Z, Li D. A high-frequency feature enhancement network for the surface defect
prediction and model visualization based on deep learning algorithm in plasma detection of welded rebar. Struct Control Health 2022;29(8):e2983. https://doi.
arc welding. Int J Adv Manuf Tech 2021;117(11):3577–97. https://doi.org/ org/10.1002/stc.2983.
10.1007/s00170-021-07903-9. [193] Pandiyan V, Murugan P, Tjahjowidodo T, Caesarendra W, Manyar OM, Then DJH.
[167] Hou W, Wei Y, Jin Y, Zhu C. Deep features based on a DCNN model for classifying In-process virtual verification of weld seam removal in robotic abrasive belt
imbalanced weld flaw types. Measurement 2019;131:482–9. https://doi.org/ grinding process using deep learning. Robot Cim-Int Manuf 2019;57:477–87.
10.1016/j.measurement.2018.09.011. https://doi.org/10.1016/j.rcim.2019.01.006.
[168] Jiang H, Hu Q, Zhi Z, Gao J, Gao Z, Wang R, et al. Convolution neural network [194] Liu H, Yan Y, Song K, Chen H, Yu H. Efficient optical measurement of welding
model with improved pooling strategy and feature selection for weld defect studs with normal maps and convolutional neural network. IEEE Trans Instrum
recognition. Weld World 2021;65(4):731–44. https://doi.org/10.1007/s40194- Meas 2020;70:1–14. https://doi.org/10.1109/TIM.2020.3024389.
020-01027-6. [195] Zhu Y, Yang R, He Y, Ma J, Guo H, Yang Y, et al. A lightweight multiscale
attention semantic segmentation algorithm for detecting laser welding defects on

624
T. Liu et al. Journal of Manufacturing Systems 68 (2023) 601–625

safety vent of power battery. IEEE Access 2021;9:39245–54. https://doi.org/ Pattern Recognition; 2019. p. 5374–83. 〈https://doi.org/10.1109/CVPR.201
10.1109/ACCESS.2021.3064180. 9.00552〉.
[196] Jang J, Van D, Jang H, Baik DH, Yoo SD, Park J, et al. Residual neural network- [218] Bacioiu D, Melton G, Papaelias M, Shaw R. Automated defect classification of
based fully convolutional network for microstructure segmentation. Sci Technol aluminium 5083 TIG welding using HDR camera and neural networks. J Manuf
Weld Joi 2020;25(4):282–9. https://doi.org/10.1080/13621718.2019.1687635. Process 2019;45:603–13. https://doi.org/10.1016/j.jmapro.2019.07.020.
[197] Ferguson MK, Ronay A, Lee YTT, Law KH. Detection and segmentation of [219] Martínez RT, Bestard GA, Silva AMA, Alfaro SCA. Analysis of GMAW process with
manufacturing defects with convolutional neural networks and transfer learning. deep learning and machine learning techniques. J Manuf Process 2021;62:
Smart Sustain Manuf 2018;2:1–43. https://doi.org/10.1520/SSMS20180033. 695–703. https://doi.org/10.1016/j.jmapro.2020.12.052.
[198] Zhang H, Di X, Zhang Y. Real-time CU-Net-based welding quality inspection [220] Mery D, Riffo V, Zscherpel U, Mondragón G, Lillo I, Zuccar I, et al. GDXray: the
algorithm in battery production. IEEE Trans Ind Electron 2020;67(12):10942–50. database of X-ray images for nondestructive testing. J Nondestruct Eval 2015;34
https://doi.org/10.1109/TIE.2019.2962421. (4):42. https://doi.org/10.1007/s10921-015-0315-7.
[199] Wang L, Shen Q. Visual inspection of welding zone by boundary-aware semantic [221] Hartl R, Bachmann A, Habedank JB, Semm T, Zaeh MF. Process monitoring in
segmentation algorithm. IEEE T Instrum Meas 2020;70:1–9. https://doi.org/ friction stir welding using convolutional neural networks. Met-Basel 2021;11(4):
10.1109/TIM.2020.3010665. 535. https://doi.org/10.3390/met11040535.
[200] Yang L, Fan J, Huo B, Li E, Liu Y. A nondestructive automatic defect detection [222] Kandala RR, Abhinay A, Aechana S.; 2021. 〈https://www.kaggle.com/ruthka/ma
method with pixelwise segmentation. Knowl-Based Syst 2022;242:108338. skrcnn-mswelddefect〉.
https://doi.org/10.1016/j.knosys.2022.108338. [223] Zhang Y, You D, Gao X, Zhang N, Gap PP. Welding defects detection based on
[201] Yang L, Song S, Fan J, Huo B, Li E, Liu Y. An automatic deep segmentation deep learning with multiple optical sensors during disk laser welding of thick
network for pixel-level welding defect detection. IEEE Trans Instrum Meas 2022; plates. J Manuf Syst 2019;51:87–94. https://doi.org/10.1016/j.
71:1–10. https://doi.org/10.1109/TIM.2021.3127645. jmsy.2019.02.004.
[202] Nowroth C, Gu T, Grajczak J, Nothdruft S, Twiefel J, Hermsdorf J, et al. Deep [224] Cheng Y., Chen S., Xiao J., Zhang Y. Dynamic estimation of joint penetration by
learning-based weld contour and defect detection from micrographs of laser beam deep learning from weld pool image. Sci Technol Weld Joi, 26(4); 2021. p.
welded semi-finished products. Appl Sci-Basel 2022;12(9):4645. https://doi.org/ 279–85. 〈https://doi.org/10.1080/13621718.2021.1896141〉.
10.3390/app12094645. [225] Shang J, An W, Liu Y, Han B, Guo Y. Oil pipeline weld defect identification system
[203] Zou Y, Li J, Chen X, Lan R. Learning siamese networks for laser vision seam based on convolutional neural network. KSII T Internet Inf, 14(3); 2020. p.
tracking. J Opt Soc Am A Opt Image Sci Vis 2018;35(11):1805–13. https://doi. 1086–103. 〈https://doi.org/10.3837/tiis.2020.03.010〉.
org/10.1364/JOSAA.35.001805. [226] Yang Y, Yang R, Pan L, Ma J, Zhu Y, Diao T, et al. A lightweight deep learning
[204] Ma C, Huang JB, Yang X, Yang MH. Hierarchical convolutional features for visual algorithm for inspection of laser welding defects on safety vent of power battery.
tracking. In: IEEE International Conference on Computer Vision; 2015. p. Comput Ind 2020;123:103306. https://doi.org/10.1016/j.
3074–82. 〈https://doi.org/10.1109/ICCV.2015.352〉. compind.2020.103306.
[205] Nam H, Han B. Learning multi-domain convolutional neural networks for visual [227] Zhang B, Zhu J, Su H. Toward the third generation of artificial intelligence. Sci Sin
tracking. In: IEEE Conference on Computer Vision and Pattern Recognition; 2016. Inf 2020;50(9):1281–302. https://doi.org/10.1360/SSI-2020-0204.
p. 4293–302. 〈https://doi.org/10.1109/CVPR.2016.465〉. [228] Kästner L, Ahmadi S, Jonietz F, Jung P, Caire G, Ziegler M, et al. Classification of
[206] Zou Y, Lan R, Wei X, Chen J. Robust seam tracking via a deep learning framework spot-welded joints in laser thermography data using convolutional neural
combining tracking and detection. Appl Opt 2020;59(14):4321–31. https://doi. networks. IEEE Access 2021;9:48303–12. https://doi.org/10.1109/
org/10.1364/AO.389730. ACCESS.2021.3063672.
[207] Xiao R, Xu Y, Hou Z, Chen C, Chen S. An adaptive feature extraction algorithm for [229] Maldonado-Ramirez A, Rios-Cabrera R, Lopez-Juarez I. A visual path-following
multiple typical seam tracking based on vision sensor in robotic arc welding. Sens learning approach for industrial robots using DRL. Robot Cim-Int Manuf, 71;
Actuat A-Phys 2019;297:111533. https://doi.org/10.1016/j.sna.2019.111533. 2021. p. 102130. 〈https://doi.org/10.1016/j.rcim.2021.102130〉.
[208] Zhang Y, Yang G, Wang Q, Ma L, Wang Y. Weld feature extraction based on fully [230] Guo R, Liu H, Xie G, Zhang Y. Weld defect detection from imbalanced
convolutional networks. Chin J Lasers 2019;46(3):0302002. https://doi.org/ radiographic images based on contrast enhancement conditional generative
10.3788/CJL201946.0302002. adversarial network and transfer learning. IEEE Sens J 2021;21(9):10844–53.
[209] Li J, Jin S, Wang C, Xue J, Wang X. Weld line recognition and path planning with https://doi.org/10.1109/JSEN.2021.3059860.
spherical tank inspection robots. J Field Robot 2022;39(2):131–52. https://doi. [231] Nomura K, Fukushima K, Matsumura T, Asai S. Burn-through prediction and weld
org/10.1002/rob.22042. depth estimation by deep learning model monitoring the molten pool in gas metal
[210] Singh J, Tant K, Mulholland A, MacLeod C. Deep learning based inversion of arc welding with gap fluctuation. J Manuf Process 2021;61:590–600. https://doi.
locally anisotropic weld properties from ultrasonic array data. Appl Sci-Basel org/10.1016/j.jmapro.2020.10.019.
2022;12(2):532. https://doi.org/10.3390/app12020532. [232] Fioravanti CCB, Centeno TM, Delgado MRDBDS. A deep artificial immune system
[211] Mirzapour M, Movafeghi A, Yahaghi E. Quantitative weld defect sizing using to detect weld defects in DWDI radiographic images of petroleum pipes. IEEE
convolutional neural network-aided processing of RT images. Insight 2021;63(3): Access 2019;7:180947–64. https://doi.org/10.1109/ACCESS.2019.2959810.
141–5. https://doi.org/10.1784/insi.2021.63.3.141. [233] Dung CV, Sekiya H, Hirano S, Okatani T, Miki C. A vision-based method for crack
[212] Rohkohl E, Kraken M, Schönemann M, Breuer A, Herrmann C. How to detection in gusset plate welded joints of steel bridges using deep convolutional
characterize a NDT method for weld inspection in battery cell manufacturing neural networks. Autom Constr 2019;102:217–29. https://doi.org/10.1016/j.
using deep learning. Int J Adv Manuf Tech 2022;119. https://doi.org/10.1007/ autcon.2019.02.013.
s00170-021-08553-7. 4829-2843. [234] Sassi P, Tripicchio P, Avizzano CA. A smart monitoring system for automatic
[213] Gao W, Li C, Yang T, Li L, Li X. A method for detection of rare industrial fake film welding defect detection. IEEE T Ind Electron 2019;66(12):9641–50. https://doi.
based on deep learning. Secur Commun Netw 2022; 2022. 3253190. 〈https://doi. org/10.1109/TIE.2019.2896165.
org/10.1155/2022/3253190〉. [235] Nogay HS, Akinci TC. Classification of operation cases in electric arc welding
[214] Deng J, Dong W, Socher R, Li LJ, Li K., Fei-Fei L. ImageNet: a large-scale wachine by using deep convolutional neural networks. Neural Comput Appl 2021;
hierarchical image database. In: IEEE Conference on Computer Vision and Pattern 33(12):6657–70. https://doi.org/10.1007/s00521-020-05436-y.
Recognition; 2009. p. 248–55. 〈https://doi.org/10.1109/CVPR.2009.5206848〉. [236] Wang Q, Jiao W, Zhang YM. Deep learning-empowered digital twin for visualized
[215] Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan, et al. Microsoft COCO: weld joint growth monitoring and penetration control. J Manuf Syst 2020;57:
common objects in context. In: European Conference on Computer Vision; 2014. 429–39. https://doi.org/10.1016/j.jmsy.2020.10.002.
p. 740–55. 〈https://doi.org/10.1007/978-3-319-10602-1_48〉. [237] Chu Y, Feng D, Liu Z, Zhao Z, Wang Z, Xia X, et al. Hybrid-learning-based
[216] Gould S, Fulton R, Koller D . Decomposing a scene into geometric and operational visual quality inspection for edge-computing-enabled IoT system.
semantically consistent regions. In: International Conference on Computer Vision; IEEE Internet Things 2022;9(7):4958–72. https://doi.org/10.1109/
2009. p. 1–8. 〈https://doi.org/10.1109/ICCV.2009.5459211〉. JIOT.2021.3107902.
[217] Fan H, Lin L, Yang F, Chu P, Deng G, Yu S, et al. Lasot: a high-quality benchmark
for large-scale single object tracking. In: IEEE Conference on Computer Vision and

625

View publication stats