2021 International Conference on Computer Communication and Informatics (ICCCI -2021), Jan.

27 – 29, 2021, Coimbatore, INDIA

AI-Based Navigator for hassle-free navigation

for visually impaired
Nihal Bhandary Kaif Siddique
Student, Dept. of Computer Engineering, Student, Dept. of Computer Engineering,
Vivekanand Education Society’s Institute of Technology, Vivekanand Education Society’s Institute of Technology,
Chembur, India. Chembur, India.
[email protected] [email protected]

Anish Adnani Dr. Sharmila Sengupta

Student, Dept. of Computer Engineering, Associate Professor, Dept. of Computer Engineering,
Vivekanand Education Society’s Institute of Technology, Vivekanand Education Society’s Institute of Technology,
Chembur, India. Chembur, India.
[email protected] [email protected]

Supriya Patil
Student, Dept. of Computer Engineering,
Vivekanand Education Society’s Institute of Technology,
Chembur, India.
[email protected]

Abstract—There are about 285 million visually impaired people Although it can be very useful in outdoor settings, indoor
in the world. Though they manage to do their daily chores, with environments create different problems.
support but face many difficulties. The proposed solution
provides indoor and outdoor assistance to the visually impaired. In indoor environments, due to lack of environmental cues [eg.
Conventionally they generally use a cane for proper and hassle- landmarks] and randomly placed objects cane would be of little
free navigation. The system would aid them with a routine task use. Some people use accessible maps like tactile maps to help
like walking, aid while writing, going up and down the stairs, navigate indoors. Accessible maps help in improving and
searching for common objects like keys, brushes, chairs, etc. speeding up the process of building mental maps, using tactile
Voice-based aid would be provided to them through speakers and maps is difficult when the person is not familiar with raised
sensors. The ultrasonic sensor helps in recognizing objects within surfaces and tactile representations (e.g., lines, graphs, and
the vicinity which might not be captured by the cameras and Braille) which takes a lot of effort to learn [2].
produce an alerting sound for the user to avoid any sudden
bump. This system will be comparatively economical and Another popular method for navigation is the use of wearable
accessible. computer devices and mobile devices [2]. Since it does not
require the user to be familiar with their surroundings
Keywords- Convolutional Neural Networks (CNN), computer beforehand and is dynamic. The proposed work is based on this
vision, neural networks, Jetson Nano particular way of navigation. The three important features that
the proposed system would provide are (1) distance estimation,
I. INTRODUCTION (2) orientation, and (3) obstacles and hazards.
Considering that a significant portion of the global The proposed navigator intends to help in guiding visually
population has problems with the vision, navigating in the impaired people mainly in indoor environments.
indoor spaces still remains a challenge. While there are
attempts to design our indoor environments to the needs of II. LITERATURE SURVEY
visually impaired people, the cost and reach are some of the
limiting factors. There have been numerous studies conducted using
different sensors, algorithms, and models for assisting visually
Cane is one of the most used aids by sight impaired people impaired people in navigation. While some of them required
for navigation. With the help of the cane, a person can perceive processors with higher processing power others required a
the environment around them by hitting at different positions in good internet connection and their surroundings to be mapped
front of them. beforehand. The existing systems also lacked three features
essential in the mobility of a visually impaired person i.e.
distance estimation, orientation, and obstacles and hazards.

978-1-7281-5875-4/21/$31.00 ©2021 IEEE

 2021 International Conference on Computer Communication and Informatics (ICCCI -2021), Jan. 27 – 29, 2021, Coimbatore, INDIA

The navigation system [3] consists of 2 parts: a wearable path. The main drawback is that these tags need to be inserted
part and a schematic of the area where the navigation system beforehand and also it does not detect randomly placed objects.
works by guiding the user. The indoor space is divided into It does provide distance estimation but that is limited to the
cells with each cell containing IR transmitters. The location of objects with chips embedded in them [5].
these transmitters is mapped onto a matrix and using the IR
signals received, a processor calculates the position of the user. “Navigation System for Blind - Third Eye” is an AI-based
The drawback is that the area of navigation is limited to the system. It uses sonar sensors for estimating the distance, IR for
number of transmitters. Furthermore, if we intend to increase obstacle avoidance, and cameras for orientation. The model
the range then it would put a load on the processor as it would used for detection and classification falls short on accuracy and
be required to perform more calculations. Finally, it does not it can be improved using other models [7]. This system uses
help the user in avoiding obstacles or in distance estimation ultrasonic sensors, a microcontroller, and aids the user by
[3]. synthetic speech output. It uses vibrations to alert the person of
an obstacle in his/her immediate vicinity. Depending on the
The system “NAVI” assists the blind and visually impaired distance from the obstacle, the vibrations either increase or
by using sound commands. The system uses RGB sensors and decrease in intensity. This system does not help in orienting the
IR sensors to detect objects which can be seen as well as which user in their surroundings but is a good aid for obstacle
cannot be seen. It employs a pathfinding algorithm to help to avoidance [8].
assist in navigation. One of the pitfalls is that the system
expects a robust processor with high-performance capabilities Real-time Object detection [10] uses a YOLO model and
as well as it needs an array of sensors for operation. Although it phone camera/web camera to capture a video feed of the
succeeds in orientation and obstacle avoidance it fails in surroundings. The major drawback of this paper is that it
providing the distance estimation [4]. depends on the phone/ laptop for orienting the camera. The
paper relies on the user for positioning the camera towards
This system uses Radio-frequency identification (RFID) for where they are moving, this can be difficult under given
both indoor and outdoor navigation. It uses RFID tags circumstances.
embedded in objects, these tags store electronic information
and need not be in the line of sight of the user. The user is
given a cane with the receiver and by interacting with the
detectors on the floor, a braille message is generated which
would help in identifying their location or any obstacles in the

TABLE I. COMPARISON OF EXISTING AND PROPOSED NAVIGATION MODELS FOR PEOPLE WITH VISION IMPAIRMENT

Performance Paper[1] Paper[2] Paper[3] Paper[4] Paper[5] Paper[6] Proposed

Parameters System

Internet No Yes Yes No No No No

required

Algorithms Custom Pathfinding Differential OpenCV A custom YOLO and YOLO v5

used algorithm algorithm Global module in algorithm based OpenCV for model for
using Positioning Python on the inputs object object
matrices System from the sensors detection. detection

Sensors IR & IR & RGB RFID Tag IR sensor, Stereoscopic Camera Cameras
Ultrasonic sensors reader sonar sonar system, and
receivers sensor, accelerometer Ultrasonic
and sensors
camera

Outdoor Indoor Both Both Both Both Both Indoor

/Indoor
 2021 International Conference on Computer Communication and Informatics (ICCCI -2021), Jan. 27 – 29, 2021, Coimbatore, INDIA

Fig. 2 System Architecture of the proposed system

III. PROPOSED SYSTEM

The system consists of cameras, a processor [Jetson B. Methodology
Nano], an ultrasonic sensor, and a microcontroller [arduino 1) YOLO: Object detection
nano]. The proposed system then relays the suggestions to The camera continuously records and sends the frames of
the user via headphones. the video to the processor, where the frames undergo the
following processes.
A. System Architecture
Fig. 2 shows the block diagram of the proposed system. a) Data Preparation
The cameras act as an eye of the system. The cameras are at The incoming video stream is broken into a series of
head level and the user can either fit it onto the glasses or a frames of images which are then put into a buffer. The input
cap. The cameras capture the image of the immediate images from the buffer are resized to 448 x 448. The image
surroundings of the user and then sends it into the processor is then divided into grid cells. The cells in the grid will help
[Jetson Nano], where the images undergo preprocessing in classifying the regions where the probability of an object
before it gets fed into the CNN model. The detected objects is high. The COCO Dataset which has 80 classes trained on
are then classified into known classes, the distance is the yolov5 model.
estimated. After the processor is done with the above
process, the synthetic speech module conveys the b) Bounding boxes and confidence calculations
information to the user through headphones. Each cell in the grid will predict some bounding boxes
and the confidence value for the bounding boxes. The
The ultrasonic sensor has a velcro strap, the user has the confidence value is the probability that the bounding boxes
option to either stick it to his shoes or the cane. The sensor contain an object. This generates a mesh of bounding boxes
on the ground level helps in detecting the obstacles in the and their confidence values. The bounding boxes below the
path. The ultrasonic sensor relays the information to a threshold confidence values are removed. The final image
microcontroller [arduino uno] where the distance of the contains the bounding boxes with a significant confidence
obstacle is calculated. The microcontroller is programmed to value.
beep when that distance is less than the predetermined
distance. The beep will alert the user of any impending
obstacle in their path.
 2021 International Conference on Computer Communication and Informatics (ICCCI -2021), Jan. 27 – 29, 2021, Coimbatore, INDIA

followed by 2 fully connected layers. It also contains max-

pooling layers and activation layers between the
convolutional layers. The initial convolutional layers of the
network extract features from the image while the fully
connected layers predict the output probabilities and
coordinates [8].

IV. IMPLEMENTATION
Jetson Nano is the main processing part of the system.
The cameras and speakers are connected to the processor.
Using the high processing power of GPU, Jetson Nano can
run our model to detect and alert the user. The entire neural
network is written in Pytorch and executed in anaconda
environment. The model was trained on the COCO dataset
for object detection. COCO is a large-scale object detection,
Fig. 3 Object detection steps segmentation, and captioning dataset [9].
For the obstacle avoidance part of our system, an
c) Class probabilities Arduino Uno microcontroller was employed in the
Class probabilities for each grid cell are calculated. This system. It is the microcontroller board that is equipped with
value along with the confidence value of bounding boxes a digital and analog input/output pin. It can be programmed
from earlier is then multiplied. The bounding box has a low in Arduino software. The Arduino is used with an ultrasonic
value and is then removed from the image. We end up with sensor. The Ultrasonic sensor emits ultrasound at 40000Hz
the class probability map as shown in Fig. 4 [8]. The objects which travels through the air and if there is an object its path
detected are then sent to the speech module, where it is will bounce back to the module. The Arduino is
relayed to the user using headphones. programmed to beep when the object comes in the preset
range of ultrasonic sensor which is done using an
2) Obstacle detection using ultrasonic sensor Arduino. AI-based Navigator for hassle-free navigation for
The ultrasonic sensor consists of two parts - a transmitter the visually impaired is developed and integrated as
and a receiver. The transmitter continuously transmits sound mentioned in the design. The system has 4 separate
waves. When an object is in the path of a sound wave, part of components: Jetson nano, camera, Arduino uno, and
it is transmitted back. The microcontroller keeps track of the ultrasonic sensor.
transmitted time and the received time. The time difference
is then used to calculate the distance between the object and V. RESULTS
the sensor. The microcontroller then runs this value into a
program, if the value is found to be less than the threshold AI-based Navigator for hassle-free navigation for the
value, it sends a signal to the buzzer. The buzzer then alerts visually impaired is developed and integrated as mentioned
the user of the presence of an object. in the design. The system has 4 separate components: jetson
nano, camera, Arduino uno, and ultrasonic sensor.
C. YOLO object detection model Figure 5 and 6 shows the output from the object detection
YOLO is a real-time object detection CNN model. It is model using YOLO.
the most effective way to detect objects and it does it in a Following tasks were performed.
single neural network. It is capable of achieving high
accuracy while running in real-time, it can process at a rate • Detecting the distance of the object.
of 45 frames per second.
• Detecting the type of the object.
The neural network has 24 fully connected layers

Fig. 4 YOLO Architecture

 2021 International Conference on Computer Communication and Informatics (ICCCI -2021), Jan. 27 – 29, 2021, Coimbatore, INDIA

• Describing the foreign environment using speakers.

• Alerting the users by producing a beep if some
object is in close proximity.

Fig. 7 Arduino component

VI. CONCLUSION
The proposed system can detect objects and guide the
user in indoor environments. The main aim was to assist a
Fig. 5 System detecting multiple objects on table visually impaired person in navigation without any costly
infrastructure or equipment. The proposed system can
The ultrasonic sensor and Arduino were used for
achieve its objectives (orientation, distance estimation,
measuring the distance between the user and the objects. obstacle avoidance) essential for helping them to navigate.
The limitations in the system are the memory and processing
power of the processor. Even though Jetson Nano has high
processing capabilities it still is not enough to run big neural
networks. The number of classes of objects and the speed of
the system could have been better if a large neural network
was trained on a bigger dataset. The larger neural network
would require a more powerful processor, which would
increase the overall cost of the system thereby making it
difficult to distribute among the common populace. Even
with the existing neural network, the system does well in
assisting them for basic navigation around the indoor spaces.

VII. FUTURE SCOPE

The proposed system is able to detect objects and relay them
to the user. A path finding module can be added to the
system. This aspect can aid the user in guiding through
particular paths within their indoor settings. Additionally, the
number of detectable classes can be increased by training it
Fig. 6 Shelf of books detected by the system on the new datasets.
Further, it might be possible to increase the accuracy and
The ultrasonic sensor uses an ultrasonic sound wave frame per second rate of the system. The existing system
which reverts when it touches some obstacle. And thereby does well on most parameters laid down for the use by
giving information about the distance where the object is visually impaired people. These can be improved by
located. employing better suited algorithms and larger datasets.
 2021 International Conference on Computer Communication and Informatics (ICCCI -2021), Jan. 27 – 29, 2021, Coimbatore, INDIA

REFERENCES

[1] World report on vision, ISBN 978-92-4-151657-0.

[2] Watthanasak Jeamwatthanachai, Mike Wald, Gary Wills, “Indoor
navigation by blind people: Behaviors and challenges in unfamiliar
spaces and buildings” for British Journal of Visual Impairment.
[3] M. I. Islam, M. M. H. Raj, S. Nath, M. F. Rahman, S. Hossen and M.
H. Imam, "An Indoor Navigation System for Visually Impaired
People Using a Path Finding Algorithm and a Wearable Cap," 2018
3rd International Conference for Convergence in Technology (I2CT),
Pune, 2018, pp. 1-6, DOI: 10.1109/I2CT.2018.8529757.
[4] C. K. Lakde and P. S. Prasad, "Navigation system for visually
impaired people," 2015 International Conference on Computation of
Power, Energy, Information and Communication (ICCPEIC),
Chennai, 2015, pp. 0093-0098, DOI:
10.1109/ICCPEIC.2015.7259447.
[5] M. Nassih, I. Cherradi, Y. Maghous, B. Ouriaghli, and Y. Salih-Alj,
"Obstacles Recognition System for the Blind People Using RFID,"
2012 Sixth International Conference on Next Generation Mobile
Applications, Services and Technologies, Paris, 2012, pp. 60-63,
DOI: 10.1109/NGMAST.2012.28.
[6] Samartha Koharwal, Samer Bani Awwad, Aparna Vyakaranam,
“Navigation System for Blind -Third Eye”, International Journal of
Innovative Technology and Exploring Engineering (IJITEE)ISSN:
[7] M. Bousbia-Salah, A. Redjati, M. Fezari, and M. Bettayeb, "An
Ultrasonic Navigation System for Blind People," 2007 IEEE
International Conference on Signal Processing and Communications,
Dubai, 2007, pp. 1003-1006, DOI: 10.1109/ICSPC.2007.4728491.
[8] Redmon, J., Divvala, S., Girshick, R., and Farhadi, A., “You Only
Look Once: Unified, Real-Time Object Detection”, arXiv e-prints,
2015.
[9] Lin, T.-Y., “Microsoft COCO: Common Objects in Context”, arXiv
e-prints, 2014.
[10] S. Vaidya, N. Shah, N. Shah, and R. Shankarmani, "Real-Time Object
Detection for Visually Challenged People," 2020 4th International
Conference on Intelligent Computing and Control Systems (ICICCS),
Madurai, India, 2020, pp. 311-316, DOI:
10.1109/ICICCS48265.2020.9121085.