Acta Materialia 255 (2023) 119086

Contents lists available at ScienceDirect

Acta Materialia
journal homepage: www.elsevier.com/locate/actamat

Full length article

A unified microstructure segmentation approach via human-in-the-loop

machine learning
Juwon Na a,b , Se-Jong Kim b , Heekyu Kim a , Seong-Hoon Kang b , Seungchul Lee a,c,d ,∗
a
Department of Mechanical Engineering, Pohang University of Science and Technology, 77 Cheongam-ro, Pohang, Republic of Korea
b
Korea Institute of Materials Science, 797 Changwon-daero, Changwon, Republic of Korea
c
Graduate School of Artificial Intelligence, Pohang University of Science and Technology, 77 Cheongam-ro, Pohang, Republic of Korea
d
Institute of Convergence Research and Education in Advanced Technology, Yonsei University, 50 Yonsei-ro, Seoul, Republic of Korea

ARTICLE INFO ABSTRACT

Dataset link: https://github.com/iai-postech/O Microstructure segmentation, a technique for extracting structural statistics from microscopy images, is an
penSource/tree/main/weakly_supervised_micro essential step for establishing quantitative structure–property relationships in a broad range of materials
structure_segmentation/data_files/Low carbon research areas. However, the task is challenging due to the morphological complexity and diversity of structural
steel micrographs
constituents. While recent breakthroughs in deep learning have led to significant progress in microstructure
Keywords: segmentation, there remain two fundamental challenges to its further diffusion: prohibitive annotation costs
Quantitative microstructural analysis and an unreliable decision-making process. To tackle these challenges, we propose a human-in-the-loop
Image analysis machine learning framework for unified microstructure segmentation, which leverages recent advances in
Microstructure both weakly supervised learning and active learning techniques. The key idea behind our approach lies in
Electron microscopy the integration of human and machine capabilities to make not only precise but also reliable microstructure
Artificial neural networks segmentation at minimal annotation costs. Extensive experiments demonstrate the generality of our framework
across different material classes, structural constituents, and microscopic imaging modalities.

1. Introduction segmentation, the process of assigning each pixel in a microscopic

image to one of structural constituents, is essential for quantitative
Over the last century, the importance of structure, both atomic and microstructure analysis since structural quantities such as size, volume
microscopic, has been highlighted in the field of materials science and fraction, and aspect ratio are directly measured from a segmented
engineering by the fact that a material’s macroscopic properties are micrograph [12]. However, microstructure segmentation remains chal-
closely related to its microscopic structure [1,2]. With the rapid de- lenging due to the morphological complexity and diversity of structural
velopment of structure characterization and quantification techniques, constituents as well as the low-contrast and non-illumination nature of
materials scientists are now able to establish theoretical or empirical microscopic imaging systems [13].
models for so-called structure–property relationships [3,4] (e.g., grain Concurrent with the recent limelight of deep learning, state-of-the-
boundary [5] and particle [6] strengthening mechanism). The knowl- art algorithms have been applied to microstructure systems to expand
edge of structure–property relationships enables the design and dis- the scope of traditional microstructure analysis [14]. While existing
covery of new materials by tailoring properties through the control of works have proven valuable for a wide range of microstructure segmen-
structure during material processing and service. This is a fundamen-
tation tasks [15–25], there exist two main drawbacks to their practical
tal paradigm in a broad range of materials research areas, including
use. (1) Prohibitive annotation costs: Previous approaches, which are
metallic materials, functional materials, nanomaterials, and so on [7].
studied in supervised settings, are brittle and sensitive to changes
Structure quantification is the cornerstone for making structure–
in task specifications regarding material class, structural constituent,
property linkages. Structural information is typically represented as
and microscopic imaging modality [16]. For example, switching from
micrographs obtained by microscopic imaging techniques such as op-
the dendritic structure in aluminum alloy to the grain structure in
tical microscope (OM) [8], scanning electron microscope (SEM) [9],
or transmission electron microscope (TEM) [10], and then analyzed pure iron, changing imaging modality from an optical to an electron
quantitatively through image processing pipeline [11]. Microstructure microscope, or even adjusting imaging settings (e.g., magnification and

∗ Corresponding author at: Department of Mechanical Engineering, Pohang University of Science and Technology, 77 Cheongam-ro, Pohang, Republic of Korea.
E-mail addresses: [email protected] (J. Na), [email protected] (S.-J. Kim), [email protected] (H. Kim), [email protected] (S.-H. Kang),
[email protected] (S. Lee).

https://doi.org/10.1016/j.actamat.2023.119086
Received 4 March 2023; Received in revised form 4 June 2023; Accepted 10 June 2023
Available online 14 June 2023
1359-6454/© 2023 Acta Materialia Inc. Published by Elsevier Ltd. All rights reserved.
J. Na et al. Acta Materialia 255 (2023) 119086

Fig. 1. Human-in-the-loop machine learning framework for microstructure segmentation. The framework involves two main components: weakly supervised learning and active
learning. (a) Our approach for weakly supervised microstructure segmentation exploits only an input micrograph and its corresponding scribble annotation to yield microstructure
segmentation results, which enables a significant reduction in annotation effort. (b) The active learning strategy for microstructure segmentation entails alternating between training
a model on available annotation and requesting new labels for unlabeled pixels with high uncertainty, which ensures reliable decision-making in microstructure segmentation.

resolution) necessitates a re-labeled dataset to maintain their effec- even better than recently published supervised learning approaches.
tiveness in the face of a modified task specification. This indicates (2) We demonstrate that our active learning strategy brings additional
that supervised learning is unsuitable for its adoption in real-world improvements in both the model’s performance and reliability. This
applications due to the prohibitive annotation costs [26]. (2) Unreliable is accomplished in an active fashion, in the sense that the labeling
decision-making: For industrial and safety–critical applications, making process is driven by a model itself rather than by human intuition. (3)
reliable model predictions is the most important objective, as incorrect As an extension of our proposed approach, we show that after training
predictions might have severe consequences. The black-box nature of with a single reference micrograph, our proposed model is capable
deep learning contributes to unreliable decision-making while building of segmenting unseen micrographs taken from different sections or
a predictive model. Existing studies are unable to not only assess the positions within a sample. We emphasize that all of these findings are
reliability of model predictions but also understand or correct how a essential for practical-purpose microstructure segmentation. Hence, our
model has made its prediction. framework has the potential to accelerate the paradigm shift in mi-
In this work, we present a unified microstructure segmentation crostructure segmentation from a low-throughput and manual process
approach via a human-in-the-loop machine learning framework. Our to a high-throughput and automated one.
key insight is the integration of human and machine capabilities in
order to achieve accurate and reliable microstructure segmentation at 2. Methods
minimal annotation costs, which is impossible or difficult for machines
and humans to accomplish alone. Specifically, our framework involves 2.1. Weakly supervised semantic segmentation
two main components: weakly supervised learning and active learning.
(1) To minimize the amount of labeled data, we present a weakly We utilize fully convolutional networks as the backbone model,
supervised approach for microstructure segmentation, in which scribble which extracts the feature vector from each pixel and then classifies it
annotation by domain experts is regarded as weak supervision. (2) To into one of several semantic classes. The networks have the following
maximize the model’s performance gain and reliability, we employ an layers in order: four convolutional modules, a 1 × 1 convolutional layer,
active learning strategy that interactively queries domain experts to and a batch normalization layer. Each convolutional module consists
label new data. Consequently, our proposed approach involves minimal of a 3 × 3 convolutional layer, rectified linear unit (ReLU) activation
annotation costs while ensuring reliable decision-making in data-driven function, and a batch normalization layer. A schematic representation
microstructure segmentation, thereby overcoming the two aforemen- of the networks is depicted in Fig. 1(c), and structural information is
tioned drawbacks. We provide an overall illustration of the framework listed in Table 1.
in Fig. 1. The objective of weakly supervised segmentation is to assign a
Extensive quantitative and qualitative experiments are carried out semantic label to each pixel, despite that only partial supervision is
to assess our framework from a practical perspective. (1) We demon- available during training. The key of the problem is how to propagate
strate the generality of our weakly supervised microstructure segmenta- annotations from sparsely labeled pixels to unlabeled pixels. To this
tion approach across different material classes, structural constituents, end, we consider the following three constraints: (1) scribble infor-
and microscopic imaging modalities. The results are comparable to or mation, (2) feature similarity, and (3) spatial continuity. For the sake

2
J. Na et al. Acta Materialia 255 (2023) 119086

Table 1 annotation effort by making judicious suggestions on the most effective

Network structure.
annotation areas. Second, the resulting labels are added to the latest
Layer Output shape Kernel size Channel size Pad Stride
version of the database. Third, the segmentation model is retrained on
Input (None, None, 1) – – – – the expanded database. These procedures are repeated until it reaches
Conv block 1–4 (None, None, 64) 3 64 1 1
the desired performance.
Conv layer 5 (None, None, 64) 1 𝑞 1 1
Output (None, None, 𝑞) – – – –

None denotes arbitrary size, and 𝑞 denotes the number of structural constituents. 2.3. Scribble annotation guidance for grain segmentation

Grain segmentation poses more significant challenges compared to

of simplicity, we will now refer to { }𝑁 as { }, where 𝑁 denotes the gold nanoparticle segmentation and ultrahigh carbon steel segmen-
𝑖=1
number of pixels in a microscopic image. In addition, we will refer to tation due to the need for detecting blurred or missing boundaries
the networks’ response vectors as {𝑟𝑖 ∈ R𝑞 }, pseudo labels obtained by while eliminating spurious scratches and noise. To assist with this
applying the argmax function to {𝑟𝑖 } as {𝑝𝑖 ∈ R𝑞 }, and scribble labels as task, we present a two-step guidance on drawing scribble annotations
{𝑠𝑖 ∈ R𝑞 }, where 𝑞 denotes the number of structural constituents. As a specialized for grain segmentation. The first step involves generating
constraint on scribble information, the cross-entropy loss is introduced initial scribble annotations based on a visible grain and grain boundary
only for scribbled pixels, as follows: region, followed by obtaining an initial segmentation result (Supple-
mentary Fig. 1). The second step focuses on refining the annotations
∑
𝑁 ∑
𝑞
𝐿𝑠𝑐𝑟 = − 𝑠𝑖,𝑗 log 𝑟𝑖,𝑗 ⋅ 1 {𝑟𝑖 = scribbled pixel}, (1) by considering regions where the initial segmentation fails to detect
𝑖=1 𝑗=1 blurred or missing boundaries and eliminate scratches and noise (Sup-
plementary Fig. 2). It should be noted that even after following this
where 1 {⋅} represents the 0–1 indicator function. As a constraint on
feature similarity, the cross-entropy loss between {𝑟𝑖 } and {𝑝𝑖 } is con- two-step guidance, the active learning strategy can be employed to
sidered for unlabeled pixels, which ensures that pixels with the same further enhance the performance and reliability of grain segmentation.
(different) semantics are mapped to the same (distinctive) features. It
is defined as follows: 2.4. Evaluation metrics
𝑁 ∑
∑ 𝑞
𝐿𝑠𝑖𝑚 = − 𝑝𝑖,𝑗 log 𝑟𝑖,𝑗 ⋅ 1 {𝑟𝑖 ≠ scribbled pixel}. (2) There are several standard metrics for image segmentation, such as
𝑖=1 𝑗=1 precision, recall, intersection over union (IOU), and Dice coefficient.
As a constraint on spatial continuity, total variation (TV) [27] loss Both IoU and Dice coefficient are essential methods for quantifying
with L1 norm is introduced, which enhances the feature similarity of the overlap between a segmentation map and its corresponding ground
spatially continuous pixels. It is defined as follows: truth map. They are defined as follows:

∑
𝑊 ∑
−1 𝐻−1 ‖𝐴 ∩ 𝐵‖
𝐼𝑜𝑈 (𝐴, 𝐵) = , (6)
𝐿𝑐𝑜𝑛 = ‖𝑟𝑤+1,ℎ − 𝑟𝑤,ℎ ∥1 +‖𝑟𝑤,ℎ+1 − 𝑟𝑤,ℎ ∥1 , (3) ‖𝐴 ∪ 𝐵‖
𝑤=1 ℎ=1
2‖𝐴 ∩ 𝐵‖
where 𝑊 and 𝐻 represent the width and height of a micrograph. 𝐷𝑖𝑐𝑒(𝐴, 𝐵) = , (7)
‖𝐴‖ + ‖𝐵‖
Finally, the above three constraints are balanced by hyper-parameters,
as follows: where 𝐴 and 𝐵 are two segmentation masks for a given class, and ∩, ∪
are the intersection and union operators. Precision and recall are attrac-
𝐿 = 𝐿𝑠𝑐𝑟 + 𝛼𝐿𝑠𝑖𝑚 + 𝛽𝐿𝑐𝑜𝑛 (4) tive metrics for segmentation quality due to their sensitivity to over and
We manually search for the hyper-parameters. For low carbon steel under-segmentation. Over-segmentation results in low precision scores
and ultrahigh carbon steel segmentation, we set 𝛼 = 0.5 and 𝛽 = 0.001. while under-segmentation results in low recall scores. They are defined
For gold nanoparticle segmentation, we set 𝛼 = 0.5 and 𝛽 = 0.5. Our as follows:
networks are trained using the Adam optimizer [28], with a learning 𝑇𝑃
𝑃 𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = , (8)
rate of 5 × 10−2 during 2 × 104 iterations. The Xavier uniform initial- 𝑇𝑃 + 𝐹𝑃
izer [29] is used as the kernel initializer. Using an NVIDIA TITAN Xp 𝑇𝑃
GPU, network training takes about 5 min with a 512 × 512 micrograph, 𝑅𝑒𝑐𝑎𝑙𝑙 = , (9)
𝑇𝑃 + 𝐹𝑁
while evaluating takes only a few seconds.
where 𝑇 𝑃 , 𝑇 𝑁, 𝐹 𝑃 , and 𝐹 𝑁 represent true positive, true negative,
false positive, and false negative pixels, respectively.
2.2. Active learning strategy
We evaluate the performance of our segmentation model using the
above evaluation metrics. To ensure the reliability of our evaluation,
Our active learning strategy entails alternating between training
we conduct 10 different runs, each with a different seed, for all the
a model on available annotation and requesting additional informa-
segmentation tasks. We then report the mean of the results, along with
tive labels for uncertain pixels. Specifically, we construct a three-step
strategy for active learning: (1) uncertainty estimation and annotation the standard error of the mean (S.E.M.).
(2) update database (3) retraining and inference. First, we estimate
per-pixel uncertainties using a margin sampling technique. Once a 2.5. Dataset
segmentation model produces pixel-wise class probabilities, a margin
sampling technique measures the difference between the likelihood of The cementite particle dataset is publicly available [15] and con-
first and second probable classes, where a low difference indicates high sists of 24 SEM micrographs with spatial dimensions of 645 × 475.
uncertainty. The margin is defined as follows: The UHCS dataset is freely available [30] and contains 961 SEM
micrographs with dimensions of 645 × 480. The gold nanoparticle
𝑀𝑎𝑟𝑔𝑖𝑛(𝑟𝑖 ) = 𝑟𝑖 (𝑞1 ) − 𝑟𝑖 (𝑞2 ), (5)
dataset is freely available [20] and contains 215 HRTEM micrographs
where 𝑞1 and 𝑞2 are the highest and second-highest scoring classes with dimensions of 512 × 512. The low carbon steel dataset contains
under a response vector 𝑟𝑖 . Subsequently, a domain expert annotates the 16 OM micrographs with spatially varied dimensions, measured at a
regions where the model makes uncertain predictions, which reduces magnification of 50× after nital etching of low carbon steel.

3
J. Na et al. Acta Materialia 255 (2023) 119086

Fig. 2. Microstructure segmentation of ultrahigh carbon steels from SEM micrographs. The columns represent input micrographs, scribble annotations, segmentation maps,
and segmentation overlays, with different structural constituents shown in different colors. The first row shows a binary segmentation task for cementite particles, while the second
and third rows show a multi-class segmentation task for more complex microstructures. A segmentation overlay depicts a segmentation map overlaid on an input micrograph.

Table 2 Fig. 2 illustrates the results of UHCS microstructure segmentation.

Cementite particle segmentation performance.
The columns represent input micrographs, scribble annotations, seg-
IOU Precision Recall
mentation maps, and segmentation overlays, where different structural
Mean S.E.M. Mean S.E.M. Mean S.E.M. constituents are shown in different colors. The first row depicts a binary
DeCost et al. [15] 0.55 – 0.75 – 0.70 – segmentation task for cementite particles, while the subsequent rows
Ours 0.71 0.031 0.80 0.024 0.88 0.030
illustrate a multi-class segmentation task for more complex microstruc-
Data are calculated across 24 SEM images. tures. The results demonstrate that our model successfully partitions the
micrographs into the relevant regions of structural constituents in all
tasks, despite only receiving scribble annotation as partial supervision.
3. Results and discussion Specifically, the model is able to identify not only coarse-grained mi-
crostructures such as the proeutectoid cementite network (red) but also
3.1. Weakly supervised microstructure segmentation fine-grained microstructures such as small cementite particles (purple)
and Widmanstätten cementite (blue). Furthermore, it is effective in
Our weakly supervised learning approach for microstructure seg- recognizing non-trivial background structures such as ferritic matrix
mentation considers scribble annotation by domain experts as weak (yellow) and pearlite (green). We report quantitative segmentation
supervision [31]. Therefore, the annotators need only make a few scrib- performance for cementite particles in Table 2. As image segmentation
bles on the regions where they feel confident and can easily draw. In metrics, precision, recall, and region intersection over union (IOU) are
this scenario, our proposed approach exploits only an input micrograph employed, where a higher score indicates better performance. When
and its corresponding scribble annotation to make complete and clear compared to the state-of-the-art method [15], which relies on super-
microstructure segmentation, while requiring only a comparatively vised learning, the quantitative results clearly indicate our model’s
little amount of training and inference time. superiority on all metrics. Consequently, we reach the conclusion that
We demonstrate the versatility of our proposed segmentation ap- densely labeling all pixels in images may be excessively redundant for
proach on three different cases: SEM micrographs of ultrahigh carbon microstructure segmentation.
steel, OM micrographs of low carbon steel, and TEM micrographs of
gold nanoparticle. Each represents a distinct microstructure segmen-
tation problem in terms of material class, structural constituent, and 3.1.2. OM micrographs of low carbon steel
microscopic imaging modality. When compared to SEM, OM is a more straightforward and cost-
effective approach for microstructural characterization and analysis.
3.1.1. SEM micrographs of ultrahigh carbon steel OM micrograph segmentation tasks have primarily focused on grain
SEM has been an essential analytical technique for studying a boundary detection in polycrystalline metals and alloys. Herein, we
wide range of materials. Herein, we explore an SEM micrograph seg- investigate a grain segmentation problem for low carbon steel.
mentation task for ultrahigh carbon steels (UHCSs). The benchmark The grain segmentation results are shown in Fig. 3, along with
dataset for UHCS segmentation from SEM micrographs is publicly avail- comparisons to three standard methods, including OTSU threshold,
able [15,30]. The benchmark micrographs exhibit several structural Canny edge detection, and watershed segmentation. Defects that appear
constituents such as cementite particles, ferritic matrix, Widmanstätten in the optical micrograph, such as blurred or missing boundaries,
cementite, proeutectoid cementite network, and pearlite. spurious scratches, and noise, are enlarged within each image. Our

4
J. Na et al. Acta Materialia 255 (2023) 119086

Fig. 3. Microstructure segmentation of low carbon steels from OM micrographs. We compare our model with several standard methods for grain boundary detection, including
the OTSU threshold method, the Canny edge detection method, and the watershed segmentation method. The grain boundaries are indicated in blue, while the grains are indicated
in red. The white boxes indicate defects in grain structure such as blurred or missing boundaries, spurious scratches, and noise, which are enlarged in each image below. (For
interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Fig. 4. Microstructure segmentation of gold nanoparticles from TEM micrographs. The columns represent input micrographs, scribble annotations, segmentation maps, and
segmentation overlays, respectively. The crystalline regions are indicated by red scribbles, while the background regions are indicated by blue scribbles. A segmentation overlay
depicts a segmentation map overlaid on an input micrograph. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this
article.)

scribble annotation encourages that the blurred or missing boundary our model performs strongly compared to the standard methods, delin-
pixels belong to the grain boundary class (blue), while the scratch and eating clear and complete grain boundaries. The watershed method,
noise pixels belong to the grain class (red). The results demonstrate that although widely used for grain boundary detection, has limitations

5
J. Na et al. Acta Materialia 255 (2023) 119086

Fig. 5. Active learning for microstructure segmentation. The columns represent scribble annotations, segmentation maps, and uncertainty maps, respectively, where different
structural constituents of UHCSs are shown in different colors. The uncertainty maps visualize per-pixel uncertainties on the segmentation maps. The rows depict iterative optimization
processes.

Table 3 Table 4
Grain segmentation performance. Nanoparticle segmentation performance.
Dice Precision Recall Dice Precision Recall
Mean S.E.M. Mean S.E.M. Mean S.E.M. Mean S.E.M. Mean S.E.M. Mean S.E.M.
OTSU 0.80 – 0.84 – 0.76 – Groschner et al. [20] 0.89 – 0.95 – 0.84 –
Canny 0.75 – 0.66 – 0.87 – Ours 0.91 0.014 0.97 0.010 0.85 0.017
Watershed 0.77 – 0.79 – 0.76 –
Data are calculated across 43 TEM images.
Ours 0.96 0.013 0.95 0.011 0.95 0.017

Data are calculated across 16 OM images.

our weakly supervised learning approach makes an improvement over

the recent supervised learning work [20].
such as generating spurious curvatures and missing small 3-sided grain
sections. The qualitative results are supported by the quantitative per- 3.2. Active learning for microstructure segmentation
formance presented in Table 3. Furthermore, we observe that defects in
the input micrograph have little impact on our segmentation outcomes. While deep learning has established the de facto standard for mi-
The reasonable correspondence implies that scribble annotation enables crostructure segmentation, the black-box nature of deep learning poses
our model to incorporate the subjectivity of domain experts. significant challenges to its practical adoption, particularly due to its
unreliable decision-making. Active learning has recently considerable
3.1.3. TEM micrographs of gold nanoparticle attention as a potential alternative to concerns about the reliability of
TEM is considered the most popular technique in characterizing machine-centric methods. We present an active learning strategy for
nanomaterials. Herein, we aim to partition gold nanoparticle HRTEM microstructure segmentation, which interacts with domain experts to
images into crystalline and background regions. The benchmark dataset request additional informative labels for uncertain pixels. Fig. 1(b).
for gold nanoparticle segmentation is publicly available [20]. illustrates the procedure for active learning.
Fig. 4 illustrates the nanoparticle segmentation results. The columns Fig. 5 depicts active learning results for UHCS microstructure seg-
represent input micrographs, scribble annotations, segmentation maps, mentation. The columns represent scribble annotations, segmentation
and segmentation overlays, respectively. The results demonstrate that maps, and uncertainty maps, respectively, while the rows depict iter-
our model only naively trained on scribbles can localize the boundaries ative optimization processes. The uncertainty maps visualize per-pixel
between crystalline (red) and background (blue) regions. This indicates uncertainties on the segmentation maps based on a margin sampling
that our proposed model induces a bias toward spatial relations be- technique, which provides valuable feedback for the subsequent label-
tween pixels. The quantitative results presented in Table 4 show that ing process. Using the estimated uncertainty as a criterion for active

6
J. Na et al. Acta Materialia 255 (2023) 119086

Fig. 6. Unseen micrograph segmentation. The columns depict the reference micrograph for the model’s pre-training and three different unseen micrographs. The rows represent
input micrographs, segmentation overlay, and particle size distribution, respectively.

learning, our strategy alternates between training a model on available reliable decisions for microstructure segmentation? (3) Can a pre-
annotation and asking domain experts to label uncertain regions within trained microstructure segmentation model be generalized to unseen
a segmentation map. The results demonstrate that it progressively micrographs?
makes a better segmentation over a few iterations. Specifically, we To address these questions, we proposed a novel framework for mi-
observe a strong correlation between the segmentation performance crostructure segmentation through human-in-the-loop machine learn-
and our uncertainty estimation throughout the iterative processes.
ing, which capitalizes on both human and machine capabilities. To the
This suggests that, with iterations, the model not only improves the
best of our knowledge, this is the first work on unified microstructure
segmentation performance but also alleviates per-pixel uncertainties.
segmentation. We adopted scribble annotation as a means of human–
machine interaction within this framework. Specifically, our framework
3.3. Unseen micrograph segmentation
made three remarkable contributions. First, we decreased annotation
Microstructure retains consistency in shape and topology across costs by shifting the training paradigm of data-driven microstructure
sections (longitudinal and cross-section) or positions within a sample. segmentation from supervised to weakly supervised learning. Second,
As an extension of our proposed approach, we examine whether a we demonstrated that our active learning strategy makes drastic im-
model pre-trained with a single micrograph as a reference can segment provements in both the performance and reliability of microstructure
unseen micrographs taken from different positions. segmentation. Finally, we showed that a model pre-trained on a single
Fig. 6 displays the results of unseen micrograph segmentation of reference micrograph can be generalized to statistically equivalent
cementite particles. The rows represent input micrographs, segmenta- unseen micrographs. We underline that our contributions can, in prin-
tion overlay, and particle size distribution, respectively. The columns ciple, be generalized to a broad range of microstructure segmentation
depict the reference micrograph for the model’s pre-training and three tasks. We believe that this study paves the way for a unified approach to
different unseen micrographs. The results demonstrate that the pre- high-throughput quantitative microstructure analysis for accelerating
trained model is effective not only for the reference micrograph but microstructure design and processing optimization.
also for micrographs that are unseen but statistically equivalent to
the reference micrograph. It can also be confirmed from the particle
size distribution on the prediction and the ground truth, showing Code availability
that the two distributions overlap. Consequently, it indicates that the
effectiveness of our proposed method extends to unseen micrographs
Our source code is available at https://github.com/iai-postech/Ope
at a single image scribble annotation cost.
nSource/tree/main/weakly_supervised_microstructure_segmentation.
4. Conclusion
Declaration of competing interest
The materials community’s need for effective and efficient high-
throughput microstructure analysis served as the driving force be-
hind this research. We began by asking the following fundamental The authors declare that they have no known competing finan-
questions: (1) How can we mitigate the prohibitive annotation costs cial interests or personal relationships that could have appeared to
for microstructure segmentation? (2) How can neural networks make influence the work reported in this paper.

7
J. Na et al. Acta Materialia 255 (2023) 119086

Data availability [13] E.A. Holm, R. Cohn, N. Gao, A.R. Kitahara, T.P. Matson, B. Lei, S.R.
Yarasi, Overview: Computer vision and machine learning for microstructural
characterization and analysis, Metall. Mater. Trans. A (2020) 1–15.
The dataset for low carbon steel micrographs is available at http
[14] A. Agrawal, A. Choudhary, Deep materials informatics: Applications of deep
s://github.com/iai-postech/OpenSource/tree/main/weakly_supervised learning in materials science, MRS Commun. 9 (3) (2019) 779–792.
_microstructure_segmentation/data_files/Low carbon steel micrograph [15] B.L. DeCost, B. Lei, T. Francis, E.A. Holm, High throughput quantitative metallog-
s. raphy for complex microstructures using deep learning: A case study in ultrahigh
carbon steel, Microsc. Microanal. 25 (1) (2019) 21–29.
[16] A. Durmaz, M. Müller, B. Lei, A. Thomas, D. Britz, E. Holm, C. Eberl, F. Mücklich,
Acknowledgments P. Gumbsch, A deep learning approach for complex microstructure inference,
Nature Commun. 12 (1) (2021) 6272.
This work was supported by the National Research Foundation of [17] M. Li, D. Chen, S. Liu, F. Liu, Grain boundary detection and second phase
Korea (NRF) grant funded by the Korea Government (MSIT) [grant segmentation based on multi-task learning and generative adversarial network,
Measurement 162 (2020) 107857.
numbers 2020R1A2C1009744]; the Ministry of Trade, Industry and
[18] J.P. Horwath, D.N. Zakharov, R. Mégret, E.A. Stach, Understanding important
Energy (MOTIE) (Development of Meta Soft Organ Module Manufac- features of deep learning models for segmentation of high-resolution transmission
turing Technology without Immunity Rejection and Module Assembly electron microscopy images, Npj Comput. Mater. 6 (1) (2020) 1–9.
Robot System), Republic of Korea [grant numbers 20012378]; and [19] C. Shen, C. Wang, M. Huang, N. Xu, S. van der Zwaag, W. Xu, A generic high-
the Fundamental Research Program of the Korea Institute of Materials throughput microstructure classification and quantification method for regular
SEM images of complex steel microstructures combining EBSD labeling and deep
Science [grant numbers PNK9130]. learning, J. Mater. Sci. Technol. 93 (2021) 191–204.
[20] C.K. Groschner, C. Choi, M.C. Scott, Machine learning pipeline for segmentation
Appendix A. Supplementary data and defect identification from high-resolution transmission electron microscopy
data, Microsc. Microanal. (2021) 1–8.
[21] D.S. Bulgarevich, S. Tsukamoto, T. Kasuya, M. Demura, M. Watanabe, Pattern
Supplementary material related to this article can be found online
recognition with machine learning on optical microscopy images of typical
at https://doi.org/10.1016/j.actamat.2023.119086. metallurgical microstructures, Sci. Rep. 8 (1) (2018) 1–8.
[22] J. Yeom, T. Stan, S. Hong, P.W. Voorhees, Segmentation of experimental datasets
References via convolutional neural networks trained on phase field simulations, Acta Mater.
(2021) 116990.
[1] W.D. Callister, D.G. Rethwisch, Materials Science and Engineering: An [23] S.M. Azimi, D. Britz, M. Engstler, M. Fritz, F. Mücklich, Advanced steel
Introduction, Vol. 9, Wiley New York, 2018. microstructural classification by deep learning methods, Sci. Rep. 8 (1) (2018)
[2] C. Smith, A History of Metallography: The Development of Ideas on the Structure 1–14.
of Metals before 1890, MIT Press, 1988. [24] H. Kim, J. Inoue, T. Kasuya, Unsupervised microstructure segmentation by
[3] K.J. Kurzydlowski, B. Ralph, The Quantitative Description of the Microstructure mimicking metallurgists’ approach to pattern recognition, Sci. Rep. 10 (1) (2020)
of Materials, Vol. 3, CRC Press, 1995. 1–11.
[4] R.T. DeHoff, F.N. Rhines, Quantitative Microscopy, Mcgraw-Hill, New York, [25] Y. Ju, S. Li, X. Yuan, L. Cui, A. Godfrey, Y. Yan, Z. Cheng, X. Zhong, J. Zhu,
1968. A macro-nano-atomic–scale high-throughput approach for material research, Sci.
[5] E. Hall, The deformation and ageing of mild steel: III discussion of results, Proc. Adv. 7 (49) (2021) eabj8804.
Phys. Soc. Sec. B 64 (9) (1951) 747. [26] M. Muller, D. Britz, F. Mucklich, Machine learning for microstructure classifi-
[6] C.S. Smith, Grains, phases, and interfaces: An introduction of microstructure, cation: How to assign the ground truth in the most objective way, Adv Mater.
Trans. Metall. Soc. AIME 175 (1948) 15–51. Processes 179 (1) (2021) 16–22.
[7] M. Ge, F. Su, Z. Zhao, D. Su, Deep learning analysis on microscopic imaging in [27] D. Strong, T. Chan, Edge-preserving and scale-dependent properties of total
materials science, Mater. Today Nano 11 (2020) 100087. variation regularization, Inverse Problems 19 (6) (2003) S165.
[8] M.W. Davidson, M. Abramowitz, Optical microscopy, in: Encyclopedia of Imaging [28] D.P. Kingma, J. Ba, Adam: A method for stochastic optimization, in: 3rd
Science and Technology, vol. 2, (no. 1106–1141) John Wiley & Sons, Inc. International Conference on Learning Representations, 2015.
Hoboken, NJ, USA, 2002, p. 120. [29] X. Glorot, Y. Bengio, Understanding the difficulty of training deep feedforward
[9] P.J. Goodhew, J. Humphreys, R. Beanland, Electron Microscopy and Analysis, neural networks, in: Proceedings of the Thirteenth International Conference on
CRC Press, 2000. Artificial Intelligence and Statistics, Vol. 9, 2010, pp. 249–256.
[10] B. Fultz, J.M. Howe, Transmission Electron Microscopy and Diffractometry of [30] B.L. DeCost, M.D. Hecht, T. Francis, B.A. Webler, Y.N. Picard, E.A. Holm,
Materials, Springer Science & Business Media, 2012. UHCSDB: Ultrahigh carbon steel micrograph database, Integr. Mater. Manuf.
[11] L. Wojnar, Image Analysis: Applications in Materials Engineering, Crc Press, Innov. 6 (2) (2017) 197–205.
2019. [31] W. Kim, A. Kanezaki, M. Tanaka, Unsupervised learning of image segmentation
[12] B. Midtvedt, S. Helgadottir, A. Argun, J. Pineda, D. Midtvedt, G. Volpe, based on differentiable feature clustering, IEEE Trans. Image Process. 29 (2020)
Quantitative digital microscopy with deep learning, Appl. Phys. Rev. 8 (1) (2021) 8055–8068.
011310.