International Conference on Communication and Signal Processing, April 4-6, 2019, India

Deep Learning based Automated Billing Cart

Ragesh N, Giridhar B, Lingeshwaran D, Siddharth P and K P Peeyush

Abstract—Nowadays, shopping malls have become an integral Load cell is a weight measuring device. It uses the electrical
part of life and people in cities often go to shopping malls in and mechanical properties of metal. When force is applied
order to purchase their daily requirements. In such a place, the over the metal, there is a change in the shape and an elastic
environment must be made hassle-free. Our system is mainly
deformation occurs, which in turn causes the resistance of the
designed for edible objects like fruits and vegetables. For edible
products like vegetables and fruits, bar-codes and RFID tags gauge to change. The basic idea behind the load cell is
cannot be used as they have to be stuck on each of the items and Wheatstone bridge, if there is a change in resistance between
the weight of each item has to be individually measured. The two terminals, then there is a potential drop.
proposed system consists of a camera which detects the The weight sensor used here is a load cell interfaced using
commodity using Deep Learning techniques and a load cell which HX711 amplifier module. HX711 is a 24-bit ADC module
measures the weight of the commodity attached to the shopping
designed for weigh scaled and can be directly interfaced with
cart. This system will generate the bill when the customer scans
the item in front of the camera which is fixed on to the Cart. the load cell. The input multiplexer selects either Channel A or
B differential input to the low-noise programmable gain
Index Terms—object detection, YOLO, load cell, smart amplifier. There is an on-chip power supply regulator that
shopping, smart cart. eliminates the need for external power supply. There is no
external programming required for HX711.
I. INTRODUCTION The rest of the paper is organized as below. Section II, III

T HE Neural Networks have taken the world to next level.

we are seeing neural networks daily and even we are
and IV describes the related work, methodology and system
description respectively. Section V says about the working of
the system. At last, Section VI concludes the paper with
using them. From Apple’s Siri to Google’s Assistant, all are
made of neural networks. Neural networks for object detection conclusion.
has been used in here.
Humans can see an object and are able to identify what II. RELATED WORK
object it is and the location of the object. The human neural This is probably the first time that object detection is used
system is fast and accurate enough to identify the object for shopping purposes. But object detection can be used for
instantly. Implementation of the human visual system in various purposes.
computers using neural networks have been helping us to do In 2014, Sowmya V., P., S. K., and Deepika, J [1] used
things like driving cars without specialized sensors. Convolutional Neural Networks to predict the probability of
There are many methods for implementation of object an image getting higher number of likes on Instagram. Image
detection. Methods like R-CNN use region proposal to detection and classification involves two major steps, training
generate bounding box and then run a classifier throughout the and testing. In order to train their system, they’ve converted
bounding box. Then the duplications are eliminated using the images into grayscale and resized them as data is more
post-processing technique. R-CNN is a slow method for object important for the network. A better trained image will result in
detection. low Mean Squared Error (MSR) by using enough number of
For this reason, we use YOLO model (You Only Look epochs.
Once). In YOLO, a single convolutional network predicts In 2018, E. K. Jose and Veni, S [2] used YOLO to track
multiple bounding boxes and class probabilities for them. vacant spaces in parking lots. Image identification suffers
from issues like irregular illumination, climatic variation and
There are several advantages for using this YOLO model.
perspective distortion. They have implemented motion
Yolo is extremely fast when compared to other models for
tracking using Kalman filter. Then, the current frame, that is
object detection. The mAP of YOLO is more than twice of
motion-free, is cropped and fed to pre-trained YOLO model
other real time object detection models. for counting and localization of cars.
Joseph Redmon and Ali Farhadi introduced YOLOv2 [3], a
Ragesh N, Giridhar B, Lingeshwaran D, Siddharth P and K P Peeyush are new algorithm for object detection. At 40 FPS, YOLOv2 gets
with the Department of Electronics and Communication Engineering, Amrita 78.6 mAP, running significantly faster than other object
School of Engineering, Coimbatore, Amrita Vishwa Vidyapeetham, India detection algorithms. This method simultaneously trains for
[email protected] [email protected]
[email protected] [email protected] object detection and classification and that’s what makes it
[email protected] faster than other methods.

978-1-5386-7595-3/19/$31.00 ©2019 IEEE

0779
Authorized licensed use limited to: KIT Library. Downloaded on March 27,2023 at 12:31:15 UTC from IEEE Xplore. Restrictions apply.
 Ruinian Li, Tianyi Song, Nicholas Capurso, Jiguo Yu, values to predict: x, y, w, h, and confidence in each bounding
Jason Couture, and Xiuzhen Cheng [4] designed a smart box. The (x, y, w, h) represent the rectangle box containing an
shopping system based on IoT. They have attached a RFID tag object. While training the dataset we used annotation file to
to each product. When the customer wants to buy a product, feed this data. The confidence represents the probability of the
the RFID tag in the product, when placed in the cart, will be existence of object in the grid. We only predict one class in
scanned using an RFID reader that has been fixed to the every grid cell. We obtain a class-specific confidence scores
cart[5]. for each box, these scores describe the probability of one class
Makoto Ikeda and Leonard Barolli [6] used Deep Neural appearing in the box[9,10].
Network (DNN) to identify the vegetable type. DNN has a
deep hierarchy that connected multiple internal layers for IV. SYSTEM DESCRIPTION
feature detection and representation learning. Deep Learning
uses a pixel level of the image as input value, and acquire the A. Load Cell
characteristic that is most suitable by learning. The resistance change in a load cell is very small so that it is
In 2017, Vishal G. Nandanwar [7] used Load cell as the magnified by the resistive imbalance in the Wheatstone Bridge
weight measuring device along with HX711 interfaced with shown in Fig. 2. In a load cell, the value of three of the
Arduino with ability to do logical operations and data to be resistors are known and the fourth resistor is a variable
stored. resistor. When the load cell is loaded with a weight, the
change in strain of the strain gauge causes a change in
III. METHODOLOGY resistance that can be seen by change in potential difference.
The method we used for detection of object is YOLO. The The load cell has been calibrated up to 5 Kg.
image is divided into multiple grids and the bounding boxes
are placed in the image, and further predicted if there is any
object inside the grid shown in Fig. 1.
The YOLO of course is prepared with Pascal VOC dataset,
in which the scale size of objects is large, straightforwardly
utilizing YOLO to distinguish objects of little scale isn't
appropriate. We train the YOLO model to identify the little
scale objects. The framework receives to separate the picture
into S x S network. A framework cell identified as positive if
there is any article present in the cell. Network cells predicts
the bounding boxes and confidence score[8].

Fig. 2. Calibration of Load Cell

The calibration curve is as shown in Fig. 3.

Fig. 1. Detection with YOLO is a regression problem. S X S grids are

formed for each image. Every grid formed is used to create bounding boxes,
confidence score and probability of the detected object of belonging to a
particular class. The prediction is determined using the formula:
S X S (B X 5 + C) tensor [8].

Certainty reflects how sure the container B contains an

item. In the event that there is no article existed, the certainty Fig. 3. Calibration Curve
scores set be zero. Where, the intersection over union (IOU) is B. Object Detection Using YOLOv2
determined by utilizing the anticipated using the predicted We are using YOLOv2 for object detection shown in Fig. 4.
mask and the ground truth. IOU is given by the proportion of YOLOv2 partitions the image into several grids and predicts
territory of cover and area of union (ground truth). We have 5

0780
Authorized licensed use limited to: KIT Library. Downloaded on March 27,2023 at 12:31:15 UTC from IEEE Xplore. Restrictions apply.
the bounding box for interested object in each grid. First, in completion of training is indicated by the creation of
order to facilitate object detection in our system, Tensorflow checkpoint files.
and keras libraries were installed. YOLO is implemented
using darknet. V. WORKING OF THE SYSTEM
The statements of Tensorflow library are used for The system’s major components are the Camera, Arduino,
implementing darknet. For implementing object detection Load cell, LCD display. The system here is designed in such a
using YOLO, certain software packages have been installed. way that object of one kind is detected. That is one sort of a
First, Microsoft visual studio package was installed in the vegetable is placed in front of the camera. The vegetable is
system. Then, CUDA and CuDNN by Nvidia were installed. detected using the Object detection technique we are using
Packages like matplotlib for creating 2D graphs and plots, that is YOLO. After detecting the object’s class a signal is sent
Cython for implementing darknet in python, scikit-learn for to the Arduino with the serial communication techniques, and
machine learning, pillow for python imaging library, pandas all the items of that same kind are placed in the cart.
for providing fast, flexible, and expressive data structures After the Arduino receives the signal about the class of the
designed to make working with structured and time series object, it has load-cell connected to it which measures the
data, beautifulsoup4 for scraping information from web pages, weight of the total items present in the cart. Now the rate of
h5py for providing both a high- and low-level interface to the each vegetable is predefined by the user, which means that
HDF5 library from python. each class of objects have a particular rate. After the load-cell
measures weight the Arduino computes the total cost
according to the rates defined by the user.
We have provided 3 buttons for the user:
-Reset
-Next
-Finish
These buttons are provided for the user to have a hassle free
experience. The RESET button is used for clearing all the
items in the cart if there is any mistake in the bill. The NEXT
button is used to detect the next object, that is when one kind
of vegetable’s rate is computed, then we need to provide the
system with the next object. The FINISH button is used when
all the items the customer wants to buy are evaluated by the
cost. The block diagram of the cart is shown in Fig. 5.
Connections given in the circuit is, Load cell is connected to
Fig. 4. Working of YOLOv2 the Vcc, Ground, Pin 4 – Data (OUT), Pin 5 - Clock.
The datasets are taken in high resolution in order to obtain
accurate results. The datasets taken are used to train the
system that had to be used to detect objects. In order to train
our system, the datasets are converted to XML files which
were JPG files previously.
The XML files that are converted are called annotation
files. These files contain the coordinates of the bounding
boxes for the image and the labels of those images. This
conversion is carried out using a software created called
Label-Img. For example, if we want our system to detect
carrot, then the dataset containing the image of carrot is taken
and edited using the Label-Img. Bounding box surrounding
the image of our desired object is drawn and then labelled as
carrot and saved.
A XML file will be created after saving. When we open the
XML file, we can see the coordinates and the label that was
given by us. After XML file is created for all datasets, the
program is run using cfg and weight files. The configuration Fig. 5. Block Diagram for the Cart
(cfg) file contains the layers of the neural network and the
weights file is used for updating weights after training of the VI. CONCLUSION
given dataset. Once the training is done, checkpoint files are The system that we have designed can detect only 3 objects
created, which contains the modified weights. namely, Potato, Tomato and Carrot. The accuracy of the
These checkpoint files are accessed before the object object detection is 70%. In this paper, we proposed a new,
detection algorithm is run on the testing dataset. The hassle-free shopping experience. The system has been mainly

0781
Authorized licensed use limited to: KIT Library. Downloaded on March 27,2023 at 12:31:15 UTC from IEEE Xplore. Restrictions apply.
designed for edible objects like fruits and vegetables. The [3] Redmon and A. Farhadi, "YOLO9000: Better, Faster, Stronger," 2017
IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
RFID tags that are being used nowadays, can’t be stuck to
Honolulu, HI, 2017, pp. 6517-6525.
edible objects like fruits and vegetables. In case if that is to be [4] R. Li, T. Song, N. Capurso, J. Yu, J. Couture and X. Cheng, "IoT
done, the weights of the objects must be predefined. The Applications on Secure Smart Shopping System," in IEEE Internet of
customers doesn’t have a choice to buy according to their Things Journal, vol. 4, no. 6, pp. 1945-1954, Dec. 2017.
[5] C. Phanikrishna and A. V. N. Reddy, "Contour tracking based knowledge
requirement. Also, in India, Barcodes and barcode readers are
extraction and object recognition using deep learning neural networks,"
the widely used method for shopping. Practically, Barcodes 2016 2nd International Conference on Next cpp. 352-354.
cannot be given to each and every vegetable and fruits that the [6] Y. Sakai, T. Oda, M. Ikeda and L. Barolli, "A Vegetable Category
customer wants to buy. Like RFID tags, these Barcodes Recognition System Using Deep Neural Network," 2016 10th
International Conference on Innovative Mobile and Internet Services in
doesn’t give the user choice to buy according to their
Ubiquitous Computing (IMIS), Fukuoka, 2016, pp. 189-192.
requirement. Thus, using object detection for this purpose, [7] V. G. Nandanwar, M. K. M. and R. S. Ankushe, "Portable Weight
gives the customer a freedom to buy and pay for exactly what Measuring Instrument," 2017 International Conference on Recent Trends
they want. in Electrical, Electronics and Computing Technologies (ICRTEECT),
Warangal, 2017, pp. 44-48.
[8] W. Xibao, B Xu etc., ‘’Visual Target Detection Based on YOLO Network
REFERENCES Structure’’Boletín Técnico, Vol.55, Issue 9, pp.06-13.2017.
[1] Sowmya V., P., S. K., and Deepika, J., “Image Classification Using [9] Esteban, P. G., Baxter, P., Belpaeme, T., Billing, E., Cai, H., Cao, H. L.,
Convolutional Neural Networks”, International Journal of Scientific & ... & Fang, Y. (2017). How to build a supervised autonomous system for
Engineering Research , vol. 5, no. 6, p. 06/2014, 2014. robot-enhanced therapy for children with autism spectrum disorder.
[2] E. K. Jose and Veni, S., “YOLO classification with multiple object Paladyn, Journal of Behavioral Robotics, 8(1), 18-38.
tracking for vacant parking lot detection”, Journal of Advanced Research [10] Kumar, R., Gopalakrishna, K., & Ramesha, K. (2013). Intelligent
in Dynamical and Control Systems, vol. 10, pp. 683-689, 2018. shopping cart. International Journal of Engineering Science and
Innovative Technology, 2(4), 499-507.

0782
Authorized licensed use limited to: KIT Library. Downloaded on March 27,2023 at 12:31:15 UTC from IEEE Xplore. Restrictions apply.