IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL.

20, 2023 6011905

A Data-Related Patch Proposal for Semantic

Segmentation of Aerial Images
Lianlei Shan , Graduate Student Member, IEEE, Guiqin Zhao , Jun Xie, Peirui Cheng , Member, IEEE,
Xiaobin Li, Member, IEEE, and Zhepeng Wang

Abstract— Large-size images cannot be directly put into GPU data. Moreover, random and sequential sampling are blind
for training and need to be cropped to patches due to GPU (cannot make different decisions based on data distribution),
memory limitation. The commonly used cropping methods before leading to the need to crop a large number of patches to cover
are random cropping and sequential cropping, which are crude
and fatally inefficient. First, categories of datasets are often all the patterns, which greatly increases the training time and
imbalanced, and just simple cropping misses an excellent oppor- so is low efficiency. We draw our work from the imbalance
tunity to make the data distribution balanced. Second, the and the inefficiency.
training needs to crop a large number of patches to cover all
patterns, which greatly increases the training time. This problem
is of great practical hazards but is often overlooked by previous A. Imbalance
works. The optimal solution is to generate valuable patches. Most segmentation datasets are imbalanced [2], [3].
Valuable patches refer to the value to network training, i.e., In Deepglobe [2], “agriculture” accounts for 56.76%, while
the value of this patch for the convergence of the network,
and the improvement of the accuracy. To this end, we propose “water” only accounts for 3.74%. In ISPRS Potsdam, the
a data-related patch proposal strategy to sample high valuable largest category impervious surface is 32 times larger than
patches. The core idea is to score each patch according to the the smallest class cluster. There are two strategies to solve the
accuracy of each category, so as to perform balanced sampling. imbalance problem: resampling and reweighting. At present,
Compared with random cropping or sequential cropping, our people mainly pay attention to the latter one and propose
method can improve the segmentation accuracy and accelerate
the training vastly. Moreover, our method also shows great advan- focal loss [4], gradient harmonizing mechanism (GHM) [5],
tages over the loss-based balanced approaches. Experiments on online hard example mining (OHEM) [6] and other excellent
Deepglobe and Potsdam show the excellent effect of our method. works. The main idea of these works is to reweight losses
Index Terms— Large-size images, patch proposal, semantic calculated from different samples. However, compared with
segmentation. the indirect loss-based reweighting methods, the resampling
approach can directly change the data distribution, and thus is
I. I NTRODUCTION simple and effective. Random or sequential cropping cannot
seize the opportunity to perform data adjustment, but our
W ITH the advancement of photography and sensor tech-
nologies, more and more large-size (high-resolution)
images are accessible. The demand for efficient and effec-
method is data-related so it can balance the data distribution
with a wet finger.
tive processes of this kind of image has been increased
in remote sensing analysis. However, due to GPU memory B. Inefficiency
limitation, large-size images need to be cropped to patches Due to the need to cover all kinds of textures and colors
during training [1]. And the two main cropping approaches are in the original image, the number of patches cropped out is
random cropping and sequential cropping, which only execute often very large. Without enough patches for training, the
the “crop” operation and do no other filters. These rough results of the same network will drop a lot. Some works with
operations miss one great opportunity to balance the training state-of-the-art performance like HRNet with object-contextual
representations (OCRs) [7] cannot avoid this decline, indi-
Manuscript received 24 February 2023; revised 30 May 2023; accepted 4 cating that the improvement of network architecture cannot
July 2023. Date of publication 25 October 2023; date of current version
8 November 2023. This work was supported by the Science and Technology ease or solve the problem. Therefore, filtering those recurring
Innovation (STI) 2030-Major Projects under Grant 2021ZD0201404. (Corre- patches (those with low training value) is an effective way to
sponding author: Zhepeng Wang.) improve efficiency.
Lianlei Shan and Guiqin Zhao are with the School of Computer Science
and Technology, University of Chinese Academy of Sciences, Beijing 100049, According to the above analysis, the key to solve the
China (e-mail: [email protected]; [email protected]. problem is to generate patches with high training values.
edu.cn). The high training value means the cropped patches can make
Jun Xie and Zhepeng Wang are with Lenovo Research, Beijing 100085,
China (e-mail: [email protected]; [email protected]). networks get a competitive accuracy on all the categories
Peirui Cheng and Xiaobin Li are with the Key Laboratory of Network and converge in the shortest possible time. Specifically, the
Information System Technology (NIST), Aerospace Information Research proposed patches can cover all the patterns, and there are no
Institute, Chinese Academy of Sciences, Beijing 100190, China (e-mail:
[email protected]; [email protected]). redundant and repeated patches. Moreover, the patch proposal
Digital Object Identifier 10.1109/LGRS.2023.3327390 strategy should give a specific emphasis to the difficult case.
1558-0571 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: CHINA UNIVERSITY OF MINING AND TECHNOLOGY. Downloaded on April 16,2024 at 07:29:57 UTC from IEEE Xplore. Restrictions apply.
 6011905 IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 20, 2023

To this end, we design a data-related patch proposal approach

to generate patches that meet the above-mentioned require-
ments. In summary, the main contributions of our work are as
follows.
1) We note the drawbacks of random/sequential cropping
and propose a data-related patch proposal strategy for
the segmentation of large-size images.
2) The proposed strategy can directly balance the distri-
bution of categories and thus greatly improve model
performance without changing network structure or Fig. 1. Data distribution and classifiers obtained by different sampling meth-
ods. Random and sequential sampling cannot change the original unbalanced
training strategy. data distribution, resulting in the final classifier being heavily biased toward
3) The strategy can filter a large number of redundant large categories. (Bottom) Our method can balance categories to obtain a
patches, thus greatly shortening the training time. balanced and accurate classifier to avoid errors.

II. R ELATED W ORK

A. Patch Proposal Strategy
Some works do simple filters for the cropped patches in
the previous patch proposal strategies. Li et al. [8] selected
more discriminative patches via the clustering algorithm to
Fig. 2. Display of different patches. The boxes in the figure represent patches.
obtain more features to match the image tag. Isensee et al. [9]
ensured that more than (1/3) of the pixels in a batch sample
are foreground pixels. Yang and Min [10] used salient object to models and thus define the criterion for a good proposal
detection to extract foreground first and then to make sure strategy. Then we introduce how to evaluate and filter the
the foreground takes up a certain proportion. However, [8] is patches to make distribution balanced. Finally, we introduce
difficult to cluster for various kinds of complex images, while the overall process of the patch proposal.
[9] and [10] are only for two-class tasks. Our method does not
require preoperations such as training or clustering. Moreover,
our method can be applied to the segmentation of any number A. Damage of Unbalanced Data Distribution
of categories. Fig. 1 shows the data distribution using different sampling
methods and the classifier trained based on the distributions.
B. Solutions to Imbalance The sampling method at the top is random or sequential
cropping. Since the original dataset is imbalanced and random
Data imbalance includes category imbalance and difficult or sequential cropping cannot modify the initial distribution,
and easy sample imbalance. The previous methods alleviate the sampled data are also imbalanced. Thus, the obtained
the imbalance mainly through loss reweighting or changing classifier is biased toward the larger category. Given the feature
the network structure. Focal loss [4] applied weights to dif- x of a pixel, it should belong to the sixth class. But according
ferent samples to balance the foreground and the background. to the segmentation head, i.e.,
Moreover, the high confidences are reduced to alleviate the
imbalance of difficult and easy samples. GHM [5] found the r = argmax(wi · x) (1)
i
problem of oversensitivity to extreme samples in focal loss
and used piecewise function to solve the problem. OHEM [6] it is misclassified as the first category (as shown in the top
proposed a structure for OHEM. During training, the network classifier of Fig. 1).
runs twice. In the first step, the network calculates the sample A proper patch proposal strategy (sampler) is to balance
difficulty but does not calculate the loss and then calculates each category as shown in the bottom part of the figure so that
the loss according to the difficulty obtained in the first step. x can be correctly classified as the sixth class. So how to set
Bilateral-branch network (BBN) [11] proposed a two-branch the sampling strategy to make the data balanced is a problem
network structure to deal with class imbalance (long-tail to be solved. We use a scoring strategy to solve this problem,
distribution). However, the loss reweight-based method has no and the specific operation is introduced in Section III-B.
obvious effect on the extreme imbalance, and changing the
network structure to adapt to the class imbalance will reduce B. Heuristic Patch Scoring Strategy
the universality of the network. On the contrary, resample-
In this section, we introduce how to balance the data
based methods do not require any additional cost, so it is
distribution through a reasonable patch scoring strategy.
highly cost-effective.
1) Balance Within the Patch: The patches C and H in Fig. 2
only contain the “agriculture” category, so the two patches are
III. M ETHOD extremely unbalanced. The rest of the patches have balanced
In this section, we introduce how to get high-quality categories. Therefore, C and H should have low scores (the
patches. We first introduce the damage of data imbalance first to be eliminated), while other patches should have high

Authorized licensed use limited to: CHINA UNIVERSITY OF MINING AND TECHNOLOGY. Downloaded on April 16,2024 at 07:29:57 UTC from IEEE Xplore. Restrictions apply.
 SHAN et al.: DATA-RELATED PATCH PROPOSAL FOR SEMANTIC SEGMENTATION OF AERIAL IMAGES 6011905

scores. So, to express this difference, we design balanced score

Sc within the path, i.e.,
tm − tl
Sc = 1 − (2)
tm
Fig. 3. Overall process of patch proposal. The image is first downsampled and
where tm denotes the number of pixels of the maximal category then scored according to the strategy obtained in Section III-B. Next, we sort
(the category with the most pixels) in the patch, and tl denotes the obtained score map and select the largest K pixels. After selecting k pixels,
we will also save the index (i.e., the corresponding x and y coordinates) of
the number of pixels of the minimal category. We only count the corresponding pixel. Then according to the zoom ratio, it can correspond
the category whose pixel number accounts for more than 5% to a certain position of the original size image. Taking the obtained position
of the total pixel number of the patch to avoid the influence of as the center point, the corresponding patch can be obtained through to the set
length and width. Finally, we use nonmaximum suppression (NMS) to filter
individual pixels. If there is only one category in the patch, tl out the patches with large overlap.
is 0 so the score value is 0. The closer the tm and tl , the more
balanced the category distribution, and the greater the score
value, which is exactly in line with our previous setting. the scope of the image), each pixel will get a score value
2) Balance Between Patches: As can be seen from the (calculated by the (4) in Section III-B for the patch centered
figure, although C and H are filtered out, the proportion on this pixel), and a score map will be obtained as shown in the
of the “water” category (marked by blue) is still low when figure. Second, we sort the score map from the largest to the
the remaining patches are combined for training. Therefore, smallest and select the pixels with K ’s largest values, where
appropriate attention should be paid to the minimal categories, K is a hyperparameter. In this way, K patch proposals are
which is consistent with the idea of sampling more foreground obtained. Finally, NMS is used to remove the patches with
objects in the focal loss. In conclusion, patch D in the figure large overlaps. Nevertheless, the implementation of NMS is
should be given a higher score. To this end, we propose the simpler than that in object detection [12], i.e., we sort the patch
balanced score Sd between patches, i.e., from large to small according to the score value, and if the
PN 1 √ intersection of union (IoU) of the new patch and the previous
i=1 pi × ni
Sd = 1 √ (3) patch exceeds a threshold, the new patch will be discarded.
pmin
× n Now given the locations of selected patches, we can crop
where pi represents the proportion of the ith category to the the corresponding patches in the source image and ground
whole training set, and N represents the number of categories truth via the locations and sent the cropped patches to any
of the whole training set. n i represents the pixel number of segmentation network for training.
the ith category in the patch. n is the total number of pixels of
the patch. pmin represents the proportion of the category with IV. E XPERIMENTS
the minimal proportion for one specific patch. A. Implementation Details
It can be observed that the rarer the pixel, the higher the We conduct experiments with the PyTorch framework [13]
calculated Sd , which satisfies the need. For example, patch and based on the MMSegmentation codebase [14]. All the
D has more pixels of the “water” category, and the “water” experiments are performed on a workstation with two NVIDIA
category occupies a small proportion in the whole dataset, GTX Titan 2080Ti 12G GPU cards. We use the pretrained
so the score obtained by (3) for patch D will be larger than ResNet-50 [15] as our backbone network. We train all our
other patches, which meets our need. Inspired by works of models with the Adam [16] optimizer with the learning rate set
long-tailed distribution [11], we use the square root function to 4e−4. We decay the learning rate with cosine annealing [17]
to prevent the overfitting of small categories. To sum up, (3) to the minimum value of 1e−6 in the last epoch (we do not
meets the requirement of paying proper attention to small perform any warm restarts). The weight decay is set to 1e−4.
categories. The predicted semantic segmentation map is upsampled to the
3) Comprehensive Score: The two equations are added original size for evaluation. The network parameters except
together to obtain the comprehensive score S as follows: the backbone are initialized by the method in [18]. The patch
S = λ1 × Sc + λ2 × Sd (4) size is set to 512 × 512. If there is no special explanation,
we select the top 100 score values (K = 100) from the score
where λ1 and λ2 are the weighting factors that balance the map for the next operation.
two fractions, and we do sufficient experiments to show the
effect of different weighting factors to the final results in B. Experimental Results
Section IV-B.
We demonstrate the advantages of our method from aspects
of improving model accuracy and shortening training time.
C. Whole Procedure 1) Accuracy: We use deeplabv3+ [19] and OCRNet
Given the comprehensive score, we can get the valuable [21] segmentation networks to conduct experiments on two
patches, and the whole process is shown in Fig. 3, which can datasets: Deepglobe [2] and ISPRS Potsdam. The experimental
be divided into three steps. First, the images are downsampled results are shown in Table I. rand&seq represents the mixed
to 256 × 256 (a hyperparameter), and except for the edge area use of random and sequential cropping. It can be observed that
(the patch centered on the pixel in the edge area will exceed our method has quite an obvious improvement in accuracy.

Authorized licensed use limited to: CHINA UNIVERSITY OF MINING AND TECHNOLOGY. Downloaded on April 16,2024 at 07:29:57 UTC from IEEE Xplore. Restrictions apply.
 6011905 IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 20, 2023

TABLE I
R ESULTS ON D IFFERENT S EGMENTATION N ETWORKS

Fig. 4. Training process of DeepLabV3+ on Deepglobe via different

sampling methods. The black dots represent the moment of convergence.

Fig. 5. High-resolution results. Cyan for urban, yellow for agriculture, purple
for rangeland, green for forest, blue for water, white for barren land, and black
for fog which is a neglected category. Large area segmentation errors occurred
in the previous work, and our results are already very close to the label.

TABLE III
E FFECTS OF D IFFERENT R ATIOS OF λ1 AND λ2

TABLE II
R ESULTS ON D IFFERENT S EGMENTATION N ETWORKS
ON THE D EEPGLOBE DATASET

(as shown by the black dots in the figure). However, methods

with our sampling will converge in about 50 epochs, so our
training time is (2/3) of the original, which is a considerable
reduction.
4) Visual Results: Fig. 5 shows the results in high res-
olution, and we compared it with the best previous work.
The category imbalance in Deepglobe is more serious, so the As can be seen, the main wrong reason for the previous
improvement is greater. For a benchmark dataset such as work is predicting some categories that are not agriculture
Potsdam, The improvements are still obvious. as agriculture. This is because agriculture accounts for more
2) Results on Other Segmentation Networks: Our previous than 50% of the total dataset, so even the previous approach
experiments are all based on DeepLabV3+ and OCRNet, and suffers from this problem. By comparison, we can see that
we have not verified the effectiveness of our method on other our approach has solved this problem successfully, and our
semantic segmentation networks. Therefore, in this section, results are very close to the label, which fully demonstrates
we choose U-Net [24] and PSPNet [23], which have sound the effectiveness of our work.
effects in the field of remote sensing segmentation, to do the
experiment. The experimental results are shown in Table II, C. Ablation Study
and it can be seen that our method can also play an effect 1) Different Weighting Factors: The final results of different
on different segmentation networks. This shows our method λ values in (4) are shown in Table III. 2:1 is the best choice,
is effective for different kinds of segmented networks. and using only one of them will not bring much improvement,
3) Training Time: The training process of DeepLabV3+ on which proves the indispensability of both.
Deepglobe via different sampling methods is shown in Fig. 4. 2) Compared With Loss-Based Balance: Our method uses
The symbol of our training convergence is that the mIoU resample to readjust data distribution to avoid the problem
change in successive five epochs is less than 0.2. As can be caused by data imbalance. Similarly, there are many ways
seen from the figure, the mIoU of the random and sequential to eliminate the influence by loss reweighting. Therefore,
models varies greatly and will converge after the 75th epoch it is necessary to compare the effect of these two methods.

Authorized licensed use limited to: CHINA UNIVERSITY OF MINING AND TECHNOLOGY. Downloaded on April 16,2024 at 07:29:57 UTC from IEEE Xplore. Restrictions apply.
 SHAN et al.: DATA-RELATED PATCH PROPOSAL FOR SEMANTIC SEGMENTATION OF AERIAL IMAGES 6011905

TABLE IV [3] M. Cordts et al., “The cityscapes dataset for semantic urban scene
C OMPARISON B ETWEEN S AMPLE -BASED BALANCE understanding,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.
AND L OSS -BASED BALANCE (CVPR), Jun. 2016, pp. 3213–3223.
[4] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for
dense object detection,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV),
Oct. 2017, pp. 2999–3007.
[5] B. Li, Y. Liu, and X. Wang, “Gradient harmonized single-stage detector,”
in Proc. AAAI Conf. Artif. Intell., vol. 33, 2019, pp. 8577–8584.
[6] A. Shrivastava, A. Gupta, and R. Girshick, “Training region-based
object detectors with online hard example mining,” in Proc. IEEE Conf.
Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 761–769.
[7] Y. Yuan, X. Chen, X. Chen, and J. Wang, “Segmentation transformer:
Object-contextual representations for semantic segmentation,” 2019,
arXiv:1909.11065.
[8] Y. Li, J. Wu, and Q. Wu, “Classification of breast cancer histology
TABLE V images using multi-size and discriminative patches based on deep
learning,” IEEE Access, vol. 7, pp. 21400–21408, 2019.
E XPERIMENTAL R ESULTS W ITH D IFFERENT PATCH N UMBERS
[9] F. Isensee, P. F. Jaeger, S. A. A. Kohl, J. Petersen, and K. H. Maier-Hein,
“NnU-Net: A self-configuring method for deep learning-based biomed-
ical image segmentation,” Nature Methods, vol. 18, no. 2, pp. 203–211,
Feb. 2021.
[10] H. Yang and K. Min, “A saliency-based patch sampling approach for
deep artistic media recognition,” Electronics, vol. 10, no. 9, p. 1053,
Apr. 2021.
[11] B. Zhou, Q. Cui, X.-S. Wei, and Z.-M. Chen, “BBN: Bilateral-branch
network with cumulative learning for long-tailed visual recognition,”
The experimental results are shown in Table IV. Among in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR),
Jun. 2020, pp. 9716–9725.
them, super-parameters in focal loss, GHM, and OHEM are
[12] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-
consistent with the original paper. It can be observed that time object detection with region proposal networks,” in Proc. Adv.
compared with the method of loss reweighting, our method Neural Inf. Process. Syst., 2015, pp. 91–99.
is more direct and has more obvious effects. [13] A. Paszke et al., “Automatic differentiation in PyTorch,” in Proc. NIPS,
2017, pp. 1–4.
3) Results With Different K (Number of Patches): After [14] K. Chen et al., “MMSegmentation: Open MMLAB segmentation toolbox
the score map is obtained, the number of patches (K in and benchmark,” 2019, arXiv:1906.07155.
Section III-C) with high scores is selected as a hyperparame- [15] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for
image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.
ter. The experimental results are shown in Table V. It can be (CVPR), Jun. 2016, pp. 770–778.
observed that 100 is a threshold. When the number exceeds [16] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
100, most of the newly added patch proposals are filtered out 2014, arXiv:1412.6980.
[17] I. Loshchilov and F. Hutter, “SGDR: Stochastic gradient descent with
due to the existence of NMS, so the improvement of the result warm restarts,” 2016, arXiv:1608.03983.
can be ignored. [18] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers:
Surpassing human-level performance on ImageNet classification,” in
Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Dec. 2015, pp. 1026–1034.
V. C ONCLUSION
[19] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam,
In this letter, we propose a data-related patch proposal “Encoder–decoder with atrous separable convolution for semantic
strategy, which can directly change the data distribution to image segmentation,” in Proc. Eur. Conf. Comput. Vis. (ECCV), 2018,
pp. 801–818.
avoid the loss of accuracy caused by imbalance. In addition, [20] Z. Sun, Z. Zhang, M. Chen, Z. Qian, M. Cao, and Y. Wen, “Improving
filtering out meaningless patches for training will also accel- the performance of automated rooftop extraction through geospatial
erate network convergence and thus shorten training time. stratified and optimized sampling,” Remote Sens., vol. 14, no. 19,
p. 4961, Oct. 2022.
More importantly, the proposed method can be applied to [21] Y. Yuan, X. Chen, and J. Wang, “Object-contextual representations for
all the segmentation networks and so contains a wide range semantic segmentation,” in Computer Vision—ECCV. Glasgow, U.K.:
of application significance. And due to our unique sampling Springer, 2020, pp. 173–190.
[22] E. Xie et al., “SegFormer: Simple and efficient design for semantic
method, multiple categories exist in each patch, so it is very segmentation with transformers,” in Proc. Adv. Neural Inf. Process. Sys.
suitable for contrastive learning methods to play a role. (NIPS), vol. 34, Dec. 2021, pp. 12077–12090.
[23] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing
R EFERENCES network,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR),
Jul. 2017, pp. 6230–6239.
[1] W. Chen, Z. Jiang, Z. Wang, K. Cui, and X. Qian, “Collaborative [24] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional net-
global-local networks for memory-efficient segmentation of ultra-high works for biomedical image segmentation,” in Proc. Int. Conf. Med.
resolution images,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Image Comput. Comput.-Assist. Intervent. Cham, Switzerland: Springer,
Recognit. (CVPR), Jun. 2019, pp. 8916–8925. 2015, pp. 234–241.
[2] I. Demir et al., “DeepGlobe 2018: A challenge to parse the earth [25] F. Milletari, N. Navab, and S.-A. Ahmadi, “V-Net: Fully convolutional
through satellite images,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern neural networks for volumetric medical image segmentation,” in Proc.
Recognit. Workshops (CVPRW), Jun. 2018, pp. 172–17209. 4th Int. Conf. 3D Vis., Oct. 2016, pp. 565–571.

Authorized licensed use limited to: CHINA UNIVERSITY OF MINING AND TECHNOLOGY. Downloaded on April 16,2024 at 07:29:57 UTC from IEEE Xplore. Restrictions apply.