PERSONAL ASSISTANT DEVICE FOR BLIND

THROUGH VOICE
ABSTRACT
Billions of devices belonging to different manufacturers and domains are connected to the
Internet of Things. The amount of consumer equipment that a single person can manage,
however, is limited. Additionally, personal information is spread across several sources, which
prevents a seamless integration. Intelligent personal assistants are a solution to facilitate the
integration of online services and data sources. However, the management of equipment is still
challenging due to difficulty of interaction, security information integration, among other
problems. This paper proposes the architecture and implementation of an intelligent personal
assistant based on the Swarm, a decentralized platform for heterogeneous smart devices. Our
proposal focuses on ease of interaction and semantic data integration.
 TABLE OF CONTENTS

CHAPTER NO. TITLE PAGE NO.

ABSTRACT i
v
LIST OF FIGURES
LIST OF SYMBOLS vii
xi
LIST OF ABBREVIATIONS
LIST OF TABLES xii

1. CHAPTER 1 : INTRODUCTION
1.1 GENERAL
1.2 OBJECTIVE
1.3 EXISTING SYSTEM
1.3.1 DISADVANTAGES
1.4 LITERATURE SURVEY
1.5 PROPOSED SYSTEM
1.5.1 ADVANTAGES
2. CHAPTER 2 :PROJECT DESCRIPTION
2.1 GENERAL
2.2 METHODOLOGIES
2.2.1 MODULES NAME
2.2.2 MODULES EXPLANATION
2.3 TECHNIQUE OR ALGORITHM
2.3.1 PSEDUO CODE
3. CHAPTER 3 : REQUIREMENTS
3.1 GENERAL
3.2 HARDWARE REQUIREMENTS
3.3 SOFTWARE REQUIREMENTS
3.4 FUNCTIONAL REQUIREMENTS
3.5 NON FUNCTIONAL REQUIREMENTS
4. CHAPTER 4 : SYSTEM DESIGN
 4.1 GENERAL
4.2 UML DIAGRAMS
4.2.1 USE CASE DIAGRAM
4.2.2 CLASS DIAGRAM
4.2.3 OBJECT DIAGRAM
4.2.4 COMPONENT DIAGRAM
4.2.5 DEPLOYMENT DIAGRAM
4.2.6 SEQUENCE DIAGRAM
4.2.7 COLLABORATION DIAGRAM
4.2.8 STATE DIAGRAM
4.2.9 ACTIVITY DIAGRAM
4.2.10 DATA FLOW DIAGRAM
4.2.11 E-R DIAGRAM
4.2.12 SYSTEM ARCHITECTURE
5. CHAPTER 5 :SOFTWARE SPECIFICATION
5.1 PYTHON
5.2 HISTORY OF PYTHON
5.3 IMPARTANCE OF PYTHON
5.4 FEATURES OF PYTHON
5.5 LIBRARIES USED
CHAPTER 6 : IMPLEMENTATION
6. 6.1 GENERAL
6.2 IMPLEMENTATION
7. CHAPTER 7 : SNAPSHOTS
7.1 GENERAL
7.2 VARIOUS SNAPSHOTS
8. CHAPTER 8 : SOFTWARE TESTING
8.1 GENERAL
8.2 DEVELOPING METHODOLOGIES
8.3 TYPES OF TESTING
8.3.1 UNIT TESTING
 8.3.2 FUNCTIONAL TESTING
8.3.3 SYSTEM TESTING
8.3.4 PERFORMANCE TESTING
8.3.5 INTEGRATION TESTING
8.3.6 ACCEPTANCE TESTING
8.4 BUILD THE TEST PLAN
9. CHAPTER 9 :
APPLICATIONS AND FUTURE ENHANCEMENT
9.1 APPLICATIONS
9.2 FUTURE ENHANCEMENTS
10. CHAPTER 10 :
10.1CONCLUSION
10.2 REFERENCES
 LIST OF FIGURES
FIGURE NO NAME OF THE FIGURE PAGE NO.
2.1 Pseudo Code

4.1 Use case Diagram

4.2 Class Diagram

4.3 Object Diagram

4.4 Component Diagram

4.5 Deployment Diagram

4.6 Sequence diagram

4.7 Collaboration diagram

4.8 State Diagram

4.9 Activity Diagram
4.10 Data flow diagram
4.11 E-R Diagram
4.12 Architecture Diagram
 LIST OF SYSMBOLS
NOTATION
S.NO NAME NOTATION DESCRIPTION

Class Name

1. Class Represents a collection of

similar entities grouped
+ public -attribute together.

-private -attribute

# protected

+operation

+operation

+operation

NAME Associations represents

Class A Class B
static relationships between
2. Association classes. Roles represents
the way the two classes see
Class A Class B each other.

3. Actor It aggregates several classes

into a single classes.

Class A Class A Interaction between the

Aggregation system and external
4. environment
Class B Class B
5.
Relation uses Used for additional process
(uses) communication.

Extends relationship is used

when one use case is similar
6. Relation extends to another use case but does
(extends) a bit more.

Communication between
7. Communication
various use cases.

State of the processs.

8. State
State

9. Initial State Initial state of the object

10. Final state F inal state of the object

11. Control flow

Represents various control
flow between the states.

12. Decision box Represents decision making

process from a constraint
 Interact ion between the
13. Usescase system and external
Usecase environment.

Represents physical
modules which are a
collection of components.
14. Component

Represents physical
15. Node modules which are a
collection of components.

A circle in DFD represents

a state or process which has
16. Data been triggered due to some
event or acion.
Process/State

Represents external entities

17. External entity
such as
keyboard,sensors,etc.

Represents communication
Transition
18. that occurs between
processes.

19. Object Lifeline Represents the vertical

dimensions that the object
communications.
 Represents the message
20. Message Message
exchanged.
 LIST OF ABBREVATION

S.NO ABBREVATION EXPANSION

1. ML Machine Learning

2. SVM Support Vector Machine

3. CNN Convolutional Neural Networks

4. ANN Artificial Neural Networks

5. AI Artificial Intelligence

6. DNN Deep Neural Networks

 CHAPTER 1
INTRODUCTION
1.1 GENERAL
Advances in technology have helped in the automation of everyday tasks. Scheduling meetings,
reserving hotels, and paying bills are now easily performed by autonomous software agents on
behalf of humans. These software agents, called intelligent personal assistants (IPA), are capable
of gathering information and triggering actions from online services
In today's fast-paced world, efficiency is key. Advances in technology, particularly Artificial
Intelligence (AI), have opened doors to automating our mundane tasks, freeing up valuable time
and mental energy.
Imagine a personalized AI assistant that understands your voice commands, anticipates your
needs, and seamlessly executes them across various online services - scheduling meetings,
booking hotels, paying bills, and much more.
This project aims to build such an assistant, your trusty AI concierge, empowering you to focus
on what truly matters.

1.2 OBJECTIVE OF THE PROJECT

The objective of the project is to interact with system which will help us by taking voice
command and execute that command and give the response based on the command which is
given by user. This project's objective is to develop an Intelligent Personal Assistant (IPA)
capable of

Understanding natural language voice commands: The IPA will interpret user commands
expressed in everyday language, eliminating the need for rigid keyword phrases. .Responding
intelligently and contextually: It will not only execute commands but also provide insightful
responses based on user intent and the broader context of the interaction.

Triggering actions across online services: The IPA will connect with various online services
(e.g., calendars, booking platforms, payment gateways) to execute user-requested actions, like
scheduling appointments, booking travel, and paying bills.
IPAs have been used in the context, such as an interactive programming approach in a cloudbased
architecture, and the application of natural language processing for simplifying user interaction. A
decentralized and opportunistic solution, compatible with the Swarm heterogeneity, however, has not
been proposed yet.

1.3 METHODOLOGIES

1.3.1 MODULES NAME:

1. Communication Agent 2. Personal Database 3. Task Manager 4. Service Manager 5. Execute

Command

1.3.2 MODULES EXLANATION

1. Communication Agent: The Communication Agent is responsible for the user interaction, using
different methods, such as natural language, gestures and touch screens. The current version of the
Swarm Assistant uses natural language processing (NLP) for voice and text. This agent provides a
multiplatform mobile web interface, and a backend that maps intents in natural language to actual
commands in the Swarm network. Intents are associated to phrases that have the same semantic
meaning.

2. Personal Database: The Personal database stores information for the Swarm Assistant to act
proactively on behalf of a person. That includes location, interests, preferences, and relationships. This
module is a semantic database, which stores ontology of concepts and relationships related to the
person domain. It leverages interoperability through semantic inference

3. Task Manager: Conversion of Speech-to-Text and Text-to-Speech is performed by task manager.

Speech-to-Text

Far from a being a fad, the overwhelming success of speech-enabled products like Amazon Alexa has
proven that some degree of speech support will be an essential aspect of household tech for the
foreseeable future. If you think about it, the reasons why are pretty obvious. Incorporating speech
recognition into your Python application offers a level of interactivity and accessibility that few
technologies can match. The accessibility improvements alone are worth considering. Speech
recognition allows the elderly and the physically and visually impaired to interact with state-of-the-art
products and services quickly and naturally—no GUI needed!

4. Service Manager: Analysis of commands in datasets which are stored by user and matching them
with user requested voice and arrange all the things which are ready to execute the commands if they
are available.

5. Execute Command After finding the match for the given command, run the respective command
and send the resource allocation for particular application are within the console only depending up on
type application we need to access.

1.4 TECHNIQUE USED OR ALGORITHM USED

Swarm assistants

• Intelligent personal assistants (IPA), are capable of gathering information and triggering actions from
online services

. • Agent provides a multiplatform mobile web interface, and a backend that maps intents in natural
language to actual commands in the Swarm network.

Speech-to-Text Conversation:

Speech recognition is an interdisciplinary subfield of computer science and computational linguistics that
develops methodologies and technologies that enable the recognition and translation of spoken language
into text by computers. It is also known as automatic speech recognition (ASR), computer speech
recognition or speech to text (STT). It incorporates knowledge and research in the computer science,
linguistics and computer engineering fields. The term voice recognition or speaker identification refers to
identifying the speaker, rather than what they are saying. Recognizing the speaker can simplify the task of
translating speech in systems that have been trained on a specific person's voice or it can be used to
authenticate or verify the identity of a speaker as part of a security process.

1. Convert voice into bits: a. Sound waves are one-dimensional. At every moment in time, they have
a single value based on the height of the wave. b. Record of the height of the wave at equally-
spaced points and it called sampling
 2. Quick sidebar on Digital Sampling

a. Sampling is crating approximation of the original sound wave and some data will be lost

b. To recover that data Nyquist theorem, and with math is used to perfectly reconstruct the original
sound wave from the spaced-out samples

3. Pre-processing our Sampled Sound Data

a. Now it will generate array of samples and each sample representing sound waves amplitude at
1/16000 th of a second interval.
b. Then that samples information need to given to neural network.
c. But trying to recognize speech patterns by processing these samples directly is difficult.
d. To reduce the complexity we need to do pre-processing to voice data.
e. Then we need to group the samples by some milliseconds time period
f. By that it will create a short recording and even though it is some difficult for different
frequency signals.
g. Because the voice will contain different pitch (low, medium, and high) and when ever those
pitches are mixed it will generate complex sound of human speech.
h. For neural networks to process, the complex sound will break apart this complex sound wave
into it’s component parts.
i. Fourier Transform breaks apart the complex sound wave into the simple sound waves that make
it up.
j. A neural network can find patterns in this kind of data more easily than raw sound waves.

4. Recognizing Characters from Short Sounds

a. Now we can feed the audio to our neural networks. The input to the neural network will be 20
millisecond audio chunks. b. For each little audio slice, it will try to figure out the letter that corresponds
the sound currently being spoken. c. For this we are using Recurrent Neural Network, which is having
memory that influences future predictions. d. That’s because each letter it predicts should affect the
likelihood of the next letter it will predict too. e. By memory of previous predictions helps the neural
network make more accurate predictions going forward. f. After feeding it we’ll end up with a mapping
of each audio chunk to the letters most likely spoken during that chunk.
Nyquist Sampling Theorem:

It is a theorem in the field of digital signal processing which serves as a fundamental bridge between
continuous-time signals and discrete-time signals. It establishes a sufficient condition for a sample rate
that permits a discrete sequence of samples to capture all the information from a continuous-time
signal of finite bandwidth.

Fourier Transform:

It is a mathematical transform which decomposes a function (often a function of time, or a signal) into
its constituent frequencies, such as the expression of a musical chord in terms of the volumes and
frequencies of its constituent notes. The term Fourier transform refers to both the frequency domain
representation and the mathematical operation that associates the frequency domain representation to
a function of time.

Recurrent Neural Network:

A recurrent neural network (RNN) is a class of artificial neural networks where connections between
nodes form a directed graph along a temporal sequence. This allows it to exhibit temporal dynamic
behavior. Derived from feedforward neural networks, RNNs can use their internal state (memory) to
process variable length sequences of inputs. This makes them applicable to tasks such as unsegmented,
connected handwriting recognition or speech
 CHAPTER 2

LITERATURE SURVEY

Title: Swarm OS control plane: An architecture proposal for heterogeneous and organic
networks

Author: Laisa C. P. Costa ; Jan Rabaey ;

Year: 2015

Description:

Computing swarms, consisting of smart networked sensors and actuators in the connected world,
enable an extension of the info-sphere into the physical world. We call this extended cyber-
physical info-sphere Swarm. This work presents a proposal for the Swarm Framework
Architecture. Envisioning the Swarm ecosystem, challenges and characteristics were identified
and a control plane framework was proposed. In order to exercise this framework, a use case was
designed and the system simulated. The work presents a computational control plane architecture
proposition for organic and heterogeneous networks leveraging the Swarm concept.

Title: The semantic Mediation for the Swarm: An adaptable and organic solution for the Internet
of Things

Author: Laisa C. C. De Biase ; Pablo C. Calcina-Ccori

Year: 2017

Description:

This paper presents the architecture of the Mediation Service, which is a service being proposed
to support interoperability in the Internet of Things and Swarm ecosystem. Mediation uses
semantics and this paper presents strategies for service discovery using semantics. A software
architecture is presented and results in a solution to generate a matching degree of service
requests and services being offered.

Title: Intelligent Personal Assistants Based on Internet of Things Approaches

Author: J. Santos, J. J. P. C. Rodrigues

Year: 2016

Description:

The Internet has emerged as a key network to make information accessible quickly and easily,
revolutionizing how people communicate and interact with the world. The information available
on the Internet about a given subject may be extensive, allowing the development of new
solutions to solve people's day-to-day problems. One such solution is the proposal of intelligent
personal assistants (IPAs), which are software agents that can assist people in many of their daily
activities. IPAs are capable of accessing information from databases to guide people through
different tasks, deploying a learning mechanism to acquire new information on user
performance. IPAs can improve the assistance they offer to users by collecting information
autonomously from objects that are available in the surrounding environment. To make this idea
feasible, IPAs could be integrated into ubiquitous computing environments in an Internet of
Things (IoT) context. Therefore, it is necessary to integrate wireless sensor networks with the
Internet properly, considering many different factors, such as the heterogeneity of objects and the
diversity of communication protocols and enabling technologies. This approach fulfills the IoT
vision. This paper surveys the current state of the art of IoT protocols, IPAs in general, and IPAs
based on IoTs.

Title: Addressing the need to capture scenarios, intentions and preferences: Interactive
intentional programming in the smart home

Author: M. Funk, L.-L. Chen, S.-W

Year: 2018
Description:

The Internet of Things (IoT) and connected products have become part of the advance of
ubiquitous technology into personal and professional living spaces, such as the smart home.
What connectivity and distributed computing have made possible, is still programmed only
according to more or less simplified rule systems (or in traditional code); the mapping between
what end users intend or would value and what can be expressed in rules is not straightforward.
This article analyzes the temporal, preferential, technical, and social complexity of mapping end-
user intent to rules, and it suggests new concepts to better frame information that needs to be
captured to create smart-home systems that better match users’ intents. We need a new approach
aimed at first capturing end users’ intentions and potential usage scenarios, then providing this
information to a control system that learns to resolve intentions and scenarios for available
devices in the context. The new approach should deconstruct and rebuild IoT-related
programming at a higher level of abstraction that allows end users to express long-term
intentions and short-term preferences, instead of programming rules. Based on related work, a
first-person perspective and analysis of current smart-home programming practices, the concept
of Interactive Intentional Programming (IIP), is introduced and discussed.

Title: Seamless human-device interaction in the internet of things

Author: Eugenio Rubio-Drosdov ; Daniel Díaz-Sánchez

Year: 2017

Description:

The Internet of Things will bring a scenario in which interaction between humans and devices
will be critical to allow people to use, monitor or configure Internet of Things devices.
Interactions in such applications are based on traditional graphical interfaces. Devices that accept
interaction based on Natural Language, e.g., through voice commands, can understand basic
human orders or answering questions whenever user expressions fit into a known language
pattern. Some devices can understand natural language voice commands but require
sophisticated voice assistants located in the cloud, which raises significant privacy concerns.
Others devices which handle voice-processing locally can perform a very limited local
recognition system, requiring users to be familiar with words the system can process. The
purpose of this work is to diminish the complexity of Natural Language processing in the context
of IoT. The solution posited in this article allows Internet of Things devices to offload Natural
Language processing to a system that improves the use of Natural Language and alleviates the
need to learn or remember specific words or terms intended for triggering device actions. We
have evaluated the feasibility of the design with a proof-of-concept implemented in a home
environment and it was tested by real users.

Pseudo Code:

Pseudo Code
 CHAPTER 3
SYSTEM ANALYSIS

3.1 EXISTING SYSTEM

• A central element is called Swarm Broker, which provides semantic registry and discovery of services,
an attribute-based access control system, and a CoAPHTTP proxy. The network size and complexity of
the Swarm, however, poses usability challenges, as human users may not be able to easily interact with
the plethora of available services. In this context, an approach to abstract the Swarm complexity is
needed.

3.2 DISADVANTAGES:

• Complexity is high

3.3 PROPOSED SYSTEM:

• The Swarm offers new opportunities for building a personal assistant in a consumer electronics network,
such as opportunistic gathering of devices for user interaction, for example an agenda may notify about an
appointment and show it in the dynamically recruited TV. This work proposes the creation of an
intelligent personal assistant to provide rich interaction with a network, taking advantage of the Swarm
platform.

3.4 ADVANTAGES:
• Time consumption is less

3.5 GENERAL
We can see from the results that on each database, the error rates are very low due to the
discriminatory power of features and the regression capabilities of classifiers. Comparing the
highest accuracies (corresponding to the lowest error rates) to those of previous works, our
results are very competitive.
3.6 HARDWARE REQUIREMENTS
The hardware requirements may serve as the basis for a contract for the implementation of the
system and should therefore be a complete and consistent specification of the whole system.
They are used by software engineers as the starting point for the system design. It shouls what
the system do and not how it should be implemented.

• PROCESSOR : DUAL CORE 2 DUOS.

• RAM : 4GB DD RAM
• HARD DISK : 250 GB

3.7 SOFTWARE REQUIREMENTS

The software requirements document is the specification of the system. It should include both a
definition and a specification of requirements. It is a set of what the system should do rather than
how it should do it. The software requirements provide a basis for creating the software
requirements specification. It is useful in estimating cost, planning team activities, performing
tasks and tracking the teams and tracking the team’s progress throughout the development
activity.

• Operating System : Windows 7/8/10

• Platform : Spyder

• Programming Language : Python, HTML

• Front End : Django

3.8 FUNCTIONAL REQUIREMENTS

A functional requirement defines a function of a software-system or its component. A function is

described as a set of inputs, the behavior, Firstly, the system is the first that achieves the standard
notion of semantic security for data confidentiality in attribute-based deduplication systems by
resorting to the hybrid cloud architecture.
3.9 NON-FUNCTIONAL REQUIREMENTS

EFFICIENCY

Our multi-modal event tracking and evolution framework is suitable for multimedia documents
from various social media platforms, which can not only effectively capture their multi-modal
topics, but also obtain the evolutionary trends of social events and generate effective event
summary details over time. Our proposed mmETM model can exploit the multi-modal property
of social event, which can effectively model social media documents including long text with
related images and learn the correlations between textual and visual modalities to separate the
visual-representative topics and non-visual-representative topics.
 CHAPTER 4
SYSTEM DESIGN
4.1 GENERAL
Design Engineering deals with the various UML [Unified Modelling language] diagrams
for the implementation of project. Design is a meaningful engineering representation of a thing
that is to be built. Software design is a process through which the requirements are translated into
representation of the software.

4.2 UML DIAGRAMS

4.2.1 USE CASE
EXPLANATION:

The main purpose of a use case diagram is to show what system functions are performed for
which actor. Roles of the actors in the system can be depicted. The above diagram consists of
user as actor. Each will play a certain role to achieve the concept.

4.2.2 CLASS DIAGRAM

Fig 4.2: Class Diagram

EXPLANATION

In this class diagram represents how the classes with attributes and methods are linked together
to perform the verification with security. From the above diagram shown the various classes
involved in our project.
4.2.3 OBJECT DIAGRAM

USER SERVER

VOICE-TO-TEXT

ASSISTANT DATABASE

Fig 4.3: Object Diagram

EXPLANATION:

In the above digram tells about the flow of objects between the classes. It is a diagram that
shows a complete or partial view of the structure of a modeled system. In this object diagram
represents how the classes with attributes and methods are linked together to perform the
verification with security.
4.2.4 COMPONENT DIAGRAM

User Provide Voic Get Data Voice to

e Command Text

Find Verify Get Store Data

Answer Command Command

Open Execute Get Assist

Application Application

Fig 4.4: Component Diagram

4.2.5 DEPLOYMENT DIAGRAM

User Voice Get Voice to

Comma Data Text

Find Verify Get Store

Answer Comma Comma Data

Open Execute Get

Applicat Applicat Assist

Fig 4.5: Deployment Diagram

4.2.6 SEQUENCE DIAGRAM

User Server Database

Provide Voice

Get Data

Voice to Text

Store Command

Get Command

Verify Command

Execute Command

Open Application

Provide Assist

Fig 4.6: Sequence Diagram

EXPLANATION:

Activity diagrams are graphical representations of workflows of stepwise activities and

actions with support for choice, iteration and concurrency. In the Unified Modeling Language,
activity diagrams can be used to describe the business and operational step-by-step workflows of
components in a system. An activity diagram shows the overall flow of control.

4.2.7 COLLABORATION DIAGRAM

2: Get Data
3: Voice to Text
6: Verify Command
7: Execute Command
8: Open Application

1: Provide Voice
User Server

9: Provide Assist

5: Get Command
4: Store Command

Databas
e

Fig 4.7: Collaboration Diagram

4.2.8 STATE DIAGRAM

User

Voice Command

Get Data

Voice to Text

Store Data

Verify Command

Open Application

Get Assist

Fig 4.8: State Diagram

EXPLANATION:

State diagram are a loosely defined diagram to show workflows of stepwise activities and
actions, with support for choice, iteration and concurrency. State diagrams require that the
system described is composed of a finite number of states; sometimes, this is indeed the case,
while at other times this is a reasonable abstraction. Many forms of state diagrams exist, which
differ slightly and have different semantics.

4.2.9 ACTIVITY DIAGRAM

User

Voice Command

Voice to Text

Store Data

Get Command

Open Application

Get Assist

Fig 4.9: Activity Diagram

EXPLANATION:

Activity diagrams are graphical representations of workflows of stepwise activities and

actions with support for choice, iteration and concurrency. In the Unified Modeling Language,
activity diagrams can be used to describe the business and operational step-by-step workflows of
components in a system. An activity diagram shows the overall flow of control.

4.2.10 DATA FLOW DIAGRAM

LEVEL 0

Voice
User
Command

Get
Voice to Text Store Data Assistance

Get and Verify

Command

LEVEL 1

Voice Command

Voice to Text Store Data Application

Get Voice Command Get Command from

Command and get information, store database, Verify the
Data in Database command

Fig 4.10: Data Flow Diagrams

Get
Assistance
EXPLANATION:

A data flow diagram (DFD) is a graphical representation of the "flow" of data through an
information system, modeling its process aspects. Often they are a preliminary step used to create an
overview of the system which can later be elaborated. DFDs can also be used for the visualization of data
processing (structured design).

A DFD shows what kinds of data will be input to and output from the system, where the data will
come from and go to, and where the data will be stored. It does not show information about the timing of
processes, or information about whether processes will operate in sequence or in parallel.

4.2.11E-R DIAGRAM

Reminde Commands

Assist Verify

Applicatio Stored

Data
UserNam
Voice

User Command Store

Text
Comman

Fig 4.11: E-R Diagram

EXPLANATION:
Entity-Relationship Model (ERM) is an abstract and conceptual representation of data. Entity-
relationship modeling is a database modeling method, used to produce a type of conceptual
schema or semantic data model of a system, often a relational database.

4.2.12 SYSTEM ARCHITECTURE

Fig 4.12: System Architecture

Explanation:

User who wants control the device with his/her commands. User tries to interact with the
system which will control all the devices or execute the commands of user. User will interact
with system by using multiple interfaces (voice and text). User can choose any one interface to
interact with the system. Then user will pass the command and it will verify with the personal
database where the system have all the commands to verify. The user command will verify with
that database and it is given to communication agent. Which will provide whatever the action
that user wants to perform with the system.

Swarm network is overlay network which handles control and data traffic related to
services. Swarm broker which provides semantic registry and discovery of services, an attribute-
based access control system. A web crawler (also known as a web spider or web robot) is a
program or automated script which browses the World Wide Web in a methodical, automated
manner. Swarm assistant is system which is combination of multiple interfaces, communication
agent, personal database, and WebCrawler. And it will connect to swarm broker which is a
registry that have operations or services to do. And it will act mediator between the system and
network.

Swarm assistant is an assistant device which is used to work based on the voice
commands of the user. Initially user will give the voice command to the system. System will
convert the voice command to text. Then based on the text command the communication agent
will verify with the personal database. If the data is matched with personal database, then the
action is given to swarm broker to execute.

If user want to listen songs then user will give the voice command that play music.
According to that command the system will select the song play that song. If user wants to access
facebook at that time it will go web crawler from there it will open the social networking sites.
 CHAPTER 5
DEVELOPMENT TOOLS
5.1 PYTHON

Python is a high-level, interpreted, interactive and object-oriented scripting language. Python is

designed to be highly readable. It uses English keywords frequently where as other languages
use punctuation, and it has fewer syntactical constructions than other languages.

5.2 HISTORY OF PYTHON

Python was developed by Guido van Rossum in the late eighties and early nineties at the
National Research Institute for Mathematics and Computer Science in the Netherlands.

Python is derived from many other languages, including ABC, Modula-3, C, C++, Algol-68,
SmallTalk, and Unix shell and other scripting languages.

Python is copyrighted. Like Perl, Python source code is now available under the GNU General
Public License (GPL).

Python is now maintained by a core development team at the institute, although Guido van
Rossum still holds a vital role in directing its progress.

5.3 IMPORTANCE OF PYTHON

• Python is Interpreted − Python is processed at runtime by the interpreter. You do not
need to compile your program before executing it. This is similar to PERL and PHP.

• Python is Interactive − You can actually sit at a Python prompt and interact with the
interpreter directly to write your programs.

• Python is Object-Oriented − Python supports Object-Oriented style or technique of

programming that encapsulates code within objects.
 • Python is a Beginner's Language − Python is a great language for the beginner-level
programmers and supports the development of a wide range of applications from simple
text processing to WWW browsers to games.

5.4 FEATURES OF PYTHON

• Easy-to-learn − Python has few keywords, simple structure, and a clearly defined
syntax. This allows the student to pick up the language quickly.

• Easy-to-read − Python code is more clearly defined and visible to the eyes.

• Easy-to-maintain − Python's source code is fairly easy-to-maintain.

• A broad standard library − Python's bulk of the library is very portable and cross-
platform compatible on UNIX, Windows, and Macintosh.

• Interactive Mode − Python has support for an interactive mode which allows interactive
testing and debugging of snippets of code.

• Portable − Python can run on a wide variety of hardware platforms and has the same
interface on all platforms.

• Extendable − You can add low-level modules to the Python interpreter. These modules
enable programmers to add to or customize their tools to be more efficient.

• Databases − Python provides interfaces to all major commercial databases.

• GUI Programming − Python supports GUI applications that can be created and ported
to many system calls, libraries and windows systems, such as Windows MFC,
Macintosh, and the X Window system of Unix.

• Scalable − Python provides a better structure and support for large programs than shell
scripting.
Apart from the above-mentioned features, Python has a big list of good features, few are listed
below −

• It supports functional and structured programming methods as well as OOP.

• It can be used as a scripting language or can be compiled to byte-code for building large
applications.

• It provides very high-level dynamic data types and supports dynamic type checking.

• IT supports automatic garbage collection.

• It can be easily integrated with C, C++, COM, ActiveX, CORBA, and Java.

5.5 LIBRARIES USED IN PYTHON

• numpy - mainly useful for its N-dimensional array objects.

• pandas - Python data analysis library, including structures such as dataframes.

• matplotlib - 2D plotting library producing publication quality figures.

• scikit-learn - the machine learning algorithms used for data analysis and data mining
tasks.

Figure : NumPy, Pandas, Matplotlib, Scikit-learn

 CHAPTER 6

IMPLEMENTATION AND TESTING

6.1 GENERAL

Coding:

import pyttsx3

import webbrowser

import smtplib

import random

import speech_recognition as sr

import wikipedia

import datetime

import wolframalpha

import os

import sys

import glob

engine = pyttsx3.init('sapi5')

client = wolframalpha.Client('Your_App_ID')
voices = engine.getProperty('voices')

engine.setProperty('voice', voices[len(voices)-1].id)

def speak(audio):

print('Computer: ' + audio)

engine.say(audio)

engine.runAndWait()

def greetMe():

currentH = int(datetime.datetime.now().hour)

if currentH >= 0 and currentH < 12:

speak('Good Morning’)

if currentH >= 12 and currentH < 18:

speak('Good Afternoon!')

if currentH >= 18 and currentH !=0:

speak('Good Evening!')

greetMe()

speak('Hello Sir, I am your digital assistant LARVIS the King Sreekanth!')

speak('How may I help you?')

def myCommand():
r = sr.Recognizer()

with sr.Microphone() as source:

print("Listening...")

r.pause_threshold = 1

audio = r.listen(source)

try:

query = r.recognize_google(audio, language='en-in')

print('User: ' + query + '\n')

except sr.UnknownValueError:

speak('Sorry sir! I didn\'t get that! Try typing the command!')

query = str(input('Command: '))

return query

if __name__ == '__main__':

while True:

query = myCommand();

query = query.lower()

if 'open youtube' in query:

speak('okay')

webbrowser.open('www.youtube.com')
 elif 'open google' in query:

speak('okay')

webbrowser.open('www.google.co.in')

elif 'open facebook' in query or 'open fb' in query:

speak('okay')

webbrowser.open('www.facebook.com')

elif 'open gmail' in query or 'open mail' in query:

speak('okay')

webbrowser.open('www.gmail.com’)

elif 'open twitter' in query :

speak('okay')

webbrowser.open('www.twitter.com')

elif "what\'s up" in query or 'how are you' in query:

stMsgs = ['Just doing my thing!', 'I am fine!', 'Nice!', 'I am nice and full of energy']

speak(random.choice(stMsgs))

elif 'email' in query:

speak('Who is the recipient? ')

recipient = myCommand()
if 'me' in recipient:

try:

speak('What should I say? ')

content = myCommand()

server = smtplib.SMTP('smtp.gmail.com', 587)

server.ehlo()

server.starttls()

server.login("Your_Username", 'Your_Password')

server.sendmail('Your_Username', "Recipient_Username", content)

server.close()

speak('Email sent’)

except:

speak('Sorry Sir! I am unable to send your message at the’)

elif 'nothing' in query or 'abort' in query or 'stop' in query:

speak('okay')

speak('Bye Sir, have a good day.')

sys.exit()
elif 'hello' in query:

speak('Hello Sir')

elif 'bye' in query:

speak('Bye Sir, have a good day.')

sys.exit()

elif 'play music' in query:

music_folder = 'music'

music=glob.glob("music/*.mp3")

#music = ['Enni_Soni', 'Psycho_Saiyaan','Deewana_Tune','Enna_Sona','Sanam_Re']

random_music = music_folder +'\\'+ random.choice(music) + '.mp3'

os.system('start '+random.choice(music))

speak('Okay, here is your music! Enjoy!')

sys.exit()

else:

query = query

speak('Searching...')

try:

try:

res = client.query(query)
 results = next(res.results).text

speak('WOLFRAM-ALPHA says - ')

speak('Got it.')

speak(results)

except:

results = wikipedia.summary(query, sentences=2)

speak('Got it.')

speak('WIKIPEDIA says - ')

speak(results)

except:

webbrowser.open('www.google.com')

speak('Next Command! Sir!')

6.2 GENERAL
The purpose of testing is to discover errors. Testing is the process of trying to discover
every conceivable fault or weakness in a work product. It provides a way to check the
functionality of components, sub assemblies, assemblies and/or a finished product It is the
process of exercising software with the intent of ensuring that the Software system meets its
requirements and user expectations and does not fail in an unacceptable manner. There are
various types of test. Each test type addresses a specific testing requirement.

6.3 DEVELOPING METHODOLOGIES

The test process is initiated by developing a comprehensive plan to test the general
functionality and special features on a variety of platform combinations. Strict quality control
procedures are used. The process verifies that the application meets the requirements specified in
the system requirements document and is bug free. The following are the considerations used to
develop the framework from developing the testing methodologies.

6.4 TYPES OF TESTS

6.4.1 UNIT TESTING
Unit testing involves the design of test cases that validate that the internal program logic
is functioning properly, and that program input produce valid outputs. All decision branches and
internal code flow should be validated. It is the testing of individual software units of the
application .it is done after the completion of an individual unit before integration. This is a
structural testing, that relies on knowledge of its construction and is invasive. Unit tests perform
basic tests at component level and test a specific business process, application, and/or system
configuration. Unit tests ensure that each unique path of a business process performs accurately
to the documented specifications and contains clearly defined inputs and expected results.
6.4.2 FUNCTIONAL TEST
Functional tests provide systematic demonstrations that functions tested are available as
specified by the business and technical requirements, system documentation, and user manuals.
Functional testing is centered on the following items:
Valid Input : identified classes of valid input must be accepted.
Invalid Input : identified classes of invalid input must be rejected.
Functions : identified functions must be exercised.
Output : identified classes of application outputs must be exercised.
Systems/Procedures : interfacing systems or procedures must be invoked.
6.4.3 SYSTEM TEST
System testing ensures that the entire integrated software system meets requirements. It
tests a configuration to ensure known and predictable results. An example of system testing is the
configuration oriented system integration test. System testing is based on process descriptions
and flows, emphasizing pre-driven process links and integration points.

6.4.4 PERFORMANCE TEST

The Performance test ensures that the output be produced within the time limits,and the
time taken by the system for compiling, giving response to the users and request being send to
the system for to retrieve the results.

6.4.5 INTEGRATION TESTING

Software integration testing is the incremental integration testing of two or more
integrated software components on a single platform to produce failures caused by interface
defects.
The task of the integration test is to check that components or software applications, e.g.
components in a software system or – one step up – software applications at the company level –
interact without error.

6.4.6 ACCEPTANCE TESTING

User Acceptance Testing is a critical phase of any project and requires significant
participation by the end user. It also ensures that the system meets the functional requirements.

Acceptance testing for Data Synchronization:

 ➢ The Acknowledgements will be received by the Sender Node after the Packets are
received by the Destination Node
➢ The Route add operation is done only when there is a Route request in need
➢ The Status of Nodes information is done automatically in the Cache Updation process

6.5 BUILD THE TEST PLAN

Any project can be divided into units that can be further performed for detailed
processing. Then a testing strategy for each of this unit is carried out. Unit testing helps to
identity the possible bugs in the individual component, so the component that has bugs can be
identified and can be rectified from errors.
 CHAPTER 7
OUTPUT SCREENS
GENERAL:

This project is implements like application using python and the Server process is maintained
using the SOCKET & SERVERSOCKET and the Design part is played by Cascading Style
Sheet.

SNAPSHOTS
 When user given command to open e-mail then it will take the command and verify with
database and executed the command. Then open e-mail by using web crawler.

When user wants book a bus ticket then user will send the voice command to open abhibus or
book bus then it will open the bus booking web site.
 CHAPTER 8
CONCLUSION

This paper proposed architecture for a personal assistant that fulfills the challenges of interacting
with the system to perform the operations that user wants to execute. It was built on top of the
Swarm platform, and added natural language processing and semantic integration of personal
data gathered from different sources.

8.1 APPLICATIONS

1. Biometric system
2. Alexa

8.2 FUTURE ENHANCEMENT

Future work will investigate the performance, security and privacy of the system, along with an
analysis of the costs of data acquisition and inference. Finally, we seek to include opportunistic
use of resources and automatic service composition.
 REFERENCES

[1] L. C. P. Costa, J. Rabaey, A. Wolisz, M. Rosan, and M. K. Zuffo, “Swarm os control plane:
an architecture proposal for heterogeneous and organic networks,” IEEE Transactions on
Consumer Electronics, vol. 61, no. 4, pp. 454–462, November 2015.

[2] L. C. De Biase, P. C. Calcina-Ccori, F. S. Silva, and M. K. Zuffo, “The semantic mediation

for the swarm: An adaptable and organic solution for the internet of things,” in Consumer
Electronics (ICCE), 2017 IEEE International Conference on. IEEE, 2017, pp. 78–79.

[3] J. Santos, J. J. P. C. Rodrigues, J. Casal, K. Saleem, and V. Denisov, “Intelligent personal

assistants based on internet of things approaches,” IEEE Systems Journal, vol. 12, no. 2, pp.
1793–1802, June 2018.

[4] M. Funk, L.-L. Chen, S.-W. Yang, and Y.-K. Chen, “Addressing the need to capture
scenarios, intentions and preferences: Interactive intentional programming in the smart home,”
International Journal of Design, vol. 12, no. 1, pp. 53–66, 2018.

[5] E. Rubio-Drosdov, D. D´ıaz-Sanchez, F. Almen ´ arez, P. Arias-Cabarcos, ´ and A. Mar´ın,

“Seamless human-device interaction in the internet of things,” IEEE Transactions on Consumer
Electronics, vol. 63, no. 4, pp. 490–498, November 2017.

[6] J. Santos, J. Rodrigues, B. Silva, J. Casal, K. Saleem, and V. Denisov, “An iot-based mobile
gateway for intelligent personal assistants on mobile health environments,” vol. 71, 03 2016.

[7] A. A. de Freitas, M. Nebeling, X. Chen, J. Yang, A. S. K. Karthikeyan Ranithangam, and A.

K. Dey, “Snap-to-it: A user-inspired platform for opportunistic device interactions,” in
Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM,
2016, pp. 5909–5920.