Abandoned-Cart-Vision: Abandoned Cart Detection

Using a Deep Object Detection Approach in a

Shopping Parking Space
Mark P. Melegrito Alvin Sarraga Alon Sammy V. Militante
Department of Electronics Engineering Digital Transformation Center, STEER Hub College of Engineering and Architecture
2021 IEEE International Conference on Artificial Intelligence in Engineering and Technology (IICAIET) | 978-1-6654-2899-6/21/$31.00 ©2021 IEEE | DOI: 10.1109/IICAIET51634.2021.9573963

Technological University of the Philippines Batangas State University University of Antique

Manila, Philippines Batangas City, Philippines Antique, Philippines
[email protected] [email protected] [email protected]

Yolanda D. Austria Myriam J. Polinar Maria Concepcion A. Mirabueno

Department of Computer Engineering College of Engineering and Architecture Department of Computer Engineering
Adamson University Bohol Island State University Adamson University
Manila, Philippines Tagbilaran, Bohol, Philippines Manila, Philippines
[email protected] [email protected] [email protected]

Abstract— Nowadays, seeing a large number of shopping technologies. Security professionals use a passive technique
carts abandoned in the parking lot is a typical occurrence at to search the collected videos for suspicious actions or objects
every supermarket. After being used by customers who left their [6]. As a result, an active monitoring mechanism is required
shopping carts in the parking lot and never returned. This study
presents a technique for detecting abandoned carts in parking
to improve the protection of people and things in public areas
lots. The proposed identification of abandoned shopping carts in by identifying possible suspicious situations and issuing
parking areas enables supermarket management to quickly security alerts [7].
respond to consumer requirements for shopping carts while also
providing enough parking space for vehicles. In this study, the If an object is purposefully left somewhere and remains
YOLOv3 model, a state-of-the-art deep transfer learning object untouched/unused for an extended length of time, it is
identification method, is utilized to construct a shopping cart
detection model. Upon the result of the study, the detection
considered unattended [8]. These items might be suspicious
model has a training and validation accuracy of 92.17 % and or not. Increased security concerns need the implementation
93.80 %, respectively, with an mAP value of 93.00 %, according of effective surveillance systems capable of detecting and
to the study's findings. Because of its outstanding performance, recognizing suspicious things in public spaces [9]. Security
the proposed model is suitable for video surveillance equipment. alerts assist security professionals in taking appropriate
The system achieved a total testing accuracy of 100 %, with security precautions promptly. One such technology is
detection per frame accuracy ranging from 40.03 % to 65.03 %.
unattended object detection, which analyzes a sequence of
Keywords—shopping cart, deep learning, object detection, video frames to detect one or more unattended items [10].
YOLOv3, abandoned shopping cart The separation of an abandoned or purposefully left object
from numerous other things in the scene is a key difficulty in
I. INTRODUCTION the object identification approach [11].
With its wide relevance in real-world applications,
particularly for abandoned items, video surveillance is one of In this contemporary environment, going to supermarkets
the most challenging study topics in the field of computer and shopping malls to buy products has become a part of
vision [1]. Theft detection, unattended item detection, fight people's daily routines, with commodity consumption
detection, intrusion detection, person detection, and other reaching new highs as people's standard of living rises [12].
applications are among them [2]. Video surveillance is the However, many individuals overlook the importance of
primary requirement in light of growing security risks and returning shopping carts after each use, leaving them
breaches, as well as the limits of human surveillance [3]. abandoned in parking areas [13]. When a person leaves a
shopping cart or other similar item unattended in any area, the
Theft, fights, snatching, bombs hidden in abandoned suggested technique examines it, determines the object's most
baggage, and other security concerns are common in public likely position, and marks it as unattended. The objective of
locations [4]. All of these occurrences are linked to human the proposed approach is to develop a shopping cart
activities. As a result, in addition to object detection, public abandonment model that uses a deep learning-based object
places require a video surveillance system to detect detection mechanism. On a dataset of shopping carts, the
suspicious human activity [5]. The cause of each occurrence proposed model is trained using an appropriate data
is identified after it has occurred using current surveillance augmentation-based pre-processing technique. The goal of

978-1-6654-2899-6/21/$31.00 ©2021 IEEE

this model is to strengthen the detecting process.

The proposed approach also aids security personnel by

giving situational awareness and allowing them to respond
quickly in an emergency. The research's primary contribution
is a method for detecting unattended items by probing ‘k'
consecutive video frames and then generating an
identification, and to assist the supermarket management to
easily come to the needs of the shopping carts by the
customers inside the supermarket and also to provide
adequate parking space for the vehicles. The significance of
this study is for the store owners as well as the consumers to
have easier access to the shopping carts and the parking space Fig. 2. Abandoned shopping cart dataset.
when needed. The present shopping cart detection is limited
A. Dataset Generation and Preparation
to parking lots only. The presented method is evaluated on a
video dataset as well as a few real-time streaming videos There are 320 images in the dataset that was utilized in this
taken using a camera in the lab. investigation. As indicated in Fig. 2, eighty percent (80%)
was utilized for training and twenty percent (20%) was used
The following is how the rest of the paper is organized: for validations. The testing datasets come from a real-time
Section 2 explains the methodology of the proposed streaming video captured using a lab camera.
approach. Section 3 is devoted to the experimental analysis,
whereas Section 4 is devoted to conclusions and future work. B. Dataset Annotation
II. METHODOLOGY The researchers annotated and identified objects using a
software annotation tool [14], then labeled the datasets as
The primary goal of the research is to identify abandoned
shopping carts in parking spaces. The research was able to shown in Fig. 3. A rectangular bounding box was generated
create a block diagram as illustrated in Fig. 1 by implementing in the shopping cart region for both the training and validation
the core requirements provided. images, and it was utilized to annotate the datasets. Every
image has the label "shopping-cart" on it. The result was an
XML file with the cart's coordinates.

C. Model Training
Because it offers the benefits of high detection accuracy,
accurate location, and quick speed, the YOLOv3 algorithm is
an enhancement over the YOLOv1 and YOLOv2 algorithms.
It can identify small targets and has high resilience to
environmental situations, especially when multi-scale
prediction methods are used, and as a result, it has become a
modern research focus [15]-[17]. Fig. 4 depicts the YOLOv3
algorithm's network topology [18]. The residual network is
mostly used to improve the feature extraction network, while
the fundamental backbone network is upgraded from
Darknet-19 to Darknet-53 to extract features and get more
detailed feature information.

Fig. 1. System workflow. Fig. 3. Shopping cart dataset annotation.

 E. Testing and Inference
The researchers employed a new batch of images from
video camera footage captured in supermarket parking lots
for the testing and inference process. Because they were not
among the 300 pictures used for training and validation, these
4429 images were employed to prevent biases in the testing
accuracy results. The accuracy testing was carried out
utilizing (2).

# !"#
' 100 (2)
$ "% # %% !"# &

III. RESULTS AND ANALYSIS

This section discussed the results of the training,
validation, and testing.
A. Training and Validation Results
The training had a training loss of less than 35% and a
validation loss of less than 10% when it started. It ended on
Fig. 4. Network structure diagram of YOLOv3. epoch 24 with training and validation losses of less than 5.0 %
and 11%, respectively. As demonstrated in Fig. 5, training
The proponents selected YOLOv3 (as shown in Fig. 4)
because it is a more frequently used network in business epoch 19 had the lowest training and validation loss score of
today and many lightweight networks are built on it. The less than 5%, whereas training epoch 1 had the greatest
main reason for selecting YOLOv3 is its simple structure and training and validation loss score.
ease of usage.
The Loss lessens as the training period lengthens, as
The study used GPU-based cloud computing resources to shown in Fig. 5. Because as the training progresses, it learns
train the annotated custom dataset. It is critical to ensure that from the datasets provided. The val loss, on the other hand,
the GPU runtime is selected. After that, the researcher varies, despite the fact that the loss decreases as the epoch
installed additional object detection dependencies and progresses. A validation set was used to validate a specific
libraries, and then uploaded the annotated dataset to the model, but it is also used to evaluate subsequent models.
Cloud-Computing service for training. The researcher
employed transfer learning from a pre-trained YOLOv3 As illustrated in Fig. 6, the validation data set was
model in the training to ensure that the trained custom models employed to offer an objective evaluation of the model fit.
had excellent detection accuracy. When setting the model hyperparameters, the validation data
was used to offer an evaluation of the model fit to the testing
dataset. If the epoch increases, the value of the mAP will also
D. Model Evaluation increase. The value of the mAP will increase as the epoch
New models are saved in Assessment based on a increases. The mAP number represents the dataset
reduction in validity loss. In certain cases, the smaller the validation's precision as a percentage. Because the mAP is
loss, the more effective the model is to detect an object in equivalent to 1 (100 percent), the validity number is near to
images. However, some versions can suffer over-fitting and the mAP's highest possible value.
yet have lower losses. To ensure that the best-customized
detection model is chosen, the researcher evaluated the mAP
(mean Average Precision); the higher the mAP, the better the
detection accuracy of the model. By doing so, the researcher
had to choose the best-trained model with the highest mAP.

Therefore, we define mAP as (1):

(1)

where Q is the number of queries in the set and AP(q) is the

average precision (AP) for a given query, q. The mAP metric,
which we use to evaluate experiments with the YOLOv3
model, is frequently used as a final metric to compare results.
Fig. 5. Training (X = Epoch; Y Axis = Loss, Val_Loss).
Fig. 6. Chart for evaluation.

Epoch 1 has the lowest mAP performance of 36.54 in this

graph. Epoch 9 had the best results, with an mAP value of
0.930 or 93% as shown in Fig. 6. In addition, epoch 9 also
got 92.17% and 93.80% for training and validation scores as
reflected in Fig. 5.
B. Video Image Testing
To assess model epoch 9's inference, the research team
prepared a video clip. The video dataset for the deployment
and testing location came from camera video from grocery
parking lots. Fig. 7 shows the detection accuracy per frame of
the video testing.
In the short clip video displayed in the graph, there are
4468 image-frames where variation or fluctuation in
detection accuracy is graphically portrayed. The detection
accuracy of each frame ranges from 40.03 to 65.03 %, and
the model has a total testing accuracy of 100 % since it
correctly recognizes the shopping carts (see Fig. 8).
The model inference of image testing is also shown in Fig.
8 to check that the model is operating appropriately. In video
image testing, the system only demonstrates that it is 100%
functional and that it can recognize the shopping carts that
have been abandoned by consumers.

Fig. 8. Image/Video frames testing results (a) (b) (c) (d) and (e).

IV. CONCLUSION AND FUTURE WORKS

The research of identifying abandoned shopping carts in
parking areas enables supermarket management to respond
swiftly to customer requests for shopping carts while also
Fig. 7. Video image testing chart (X=Image Frames; Y=Accuracy Score).
providing adequate parking space for vehicles. The suggested
model is useful for video surveillance equipment for
detecting abandoned objects due to its exceptional [8] N. Dwivedi, D. Singh and D. Kushwaha, "An Approach for Unattended
Object Detection through Contour Formation using Background
performance. As a result, the research found that the analyses' Subtraction", Procedia Computer Science, vol. 171, pp. 1979-1988,
main goals had been met: to develop an artificial intelligence 2020. doi: 10.1016/j.procs.2020.04.212
system that can identify abandoned shopping carts. [9] C. Wang, J. Liu, Y. Chen, H. Liu and Y. Wang, "Towards In-baggage
The research attained an mAP of 0.93 in model epoch 9. Suspicious Object Detection Using Commodity WiFi", 2018 IEEE
Conference on Communications and Network Security (CNS), 2018.
This shows that, even with a small number of datasets, doi: 10.1109/cns.2018.8433142
YOLOv3 is an effective approach for detecting abandoned [10] L. Jadhav and B. Momin, "Detection and identification of
objects in parking spaces, particularly shopping carts. When unattended/removed objects in video surveillance", 2016 IEEE
tested, the system achieved a high performing score of 100% International Conference on Recent Trends in Electronics, Information
& Communication Technology (RTEICT), 2016. doi:
overall testing accuracy. 10.1109/rteict.2016.7808138
[11] M. Peker, "Unattended object detection based on blob tracking", 2017
ACKNOWLEDGMENT 25th Signal Processing and Communications Applications Conference
The authors are grateful to the Computing Resources of (SIU), 2017. doi: 10.1109/siu.2017.7960458
Batangas State University's STEER Hub (Science, [12] N. Sondhi, "Segmenting & profiling the deflecting customer:
understanding shopping cart abandonment", Procedia Computer
Technology, Engineering, and Environment Research Hub), Science, vol. 122, pp. 392-399, 2017. doi: 10.1016/j.procs.2017.11.385
Digital Transformation Center Lab. [13] U. Wagner, C. Ebster, U. Eske and W. Weitzl, "The Influence of
Shopping Carts on Customer Behavior in Grocery Stores", Marketing
ZFP, vol. 36, no. 3, pp. 165-175, 2014. doi: 10.15358/0344-
REFERENCES 1369_2014_3_165
[1] V. Tsakanikas and T. Dagiuklas, "Video surveillance systems-current [14] H. Alon, M. Ligayo, M. Melegrito, C. Franco Cunanan and E. Uy II,
status and future trends", Computers & Electrical Engineering, vol. 70, "Deep-Hand: A Deep Inference Vision Approach of Recognizing a
pp. 736-753, 2018. doi: 10.1016/j.compeleceng.2017.11.011 Hand Sign Language using American Alphabet", 2021 International
[2] P. Cocca, F. Marciano and M. Alberti, "Video surveillance systems to Conference on Computational Intelligence and Knowledge Economy
enhance occupational safety: A case study", Safety Science, vol. 84, pp. (ICCIKE), 2021. doi: 10.1109/iccike51210.2021.9410803
140-148, 2016. doi: 10.1016/j.ssci.2015.12.005. [15] H. Alon, "Eye-Zheimer: A Deep Transfer Learning Approach of
[3] H. Luo, J. Liu, W. Fang, P. Love, Q. Yu and Z. Lu, "Real-time smart Dementia Detection and Classification from NeuroImaging", 2020
video surveillance to manage safety: A case study of a transport mega- IEEE 7th International Conference on Engineering Technologies and
project", Advanced Engineering Informatics, vol. 45, p. 101100, 2020. Applied Sciences (ICETAS), 2020. doi:
doi: 10.1016/j.aei.2020.101100 10.1109/ICETAS51660.2020.9484315
[4] I. DAHI, M. CHIKR EL MEZOUAR, N. TALEB and M. ELBAHRI, [16] L. Lacatan, R. Santos, J. Pinkihan, R. Vicente and R. Tamargo, "Brake-
"An edge-based method for effective abandoned luggage detection in Vision: A Machine Vision-Based Inference Approach of Vehicle
complex surveillance videos", Computer Vision and Image Braking Detection for Collision Warning Oriented System", 2021
Understanding, vol. 158, pp. 141-151, 2017. doi: International Conference on Computational Intelligence and
10.1016/j.cviu.2017.01.008 Knowledge Economy (ICCIKE), 2021. doi:
10.1109/iccike51210.2021.9410750
[5] V. Singh, S. Singh and P. Gupta, "Real-Time Anomaly Recognition
Through CCTV Using Neural Networks", Procedia Computer Science, [17] J. Dioses, Jr, "Bottle-SegreDuino: An Arduino Frequency-Based Bin
vol. 173, pp. 254-263, 2020. doi: 10.1016/j.procs.2020.06.030 for Tin Can and Plastic Bottle Segregation using an Inductive
Proximity Effect", International Journal of Advanced Trends in
[6] R. Parry, M. Pino, C. Faull and L. Feathers, "Acceptability and design Computer Science and Engineering, vol. 9, no. 4, pp. 5451-5454, 2020.
of video-based research on healthcare communication: Evidence and doi: 10.30534/ijatcse/2020/184942020
recommendations", Patient Education and Counseling, vol. 99, no. 8,
pp. 1271-1284, 2016. doi: 10.1016/j.pec.2016.03.013 [18] J. Redmon and A. Farhadi, "YOLOv3: An Incremental
Improvement", arXiv, 2018.
[7] D. Singh, S. Paroothi, M. Rusia and M. Ansari, "Human Crowd
Detection for City Wide Surveillance", Procedia Computer Science,
vol. 171, pp. 350-359, 2020. doi: 10.1016/j.procs.2020.04.036