Machine Learning for Beginners

The Ultimate Guide to Learn and Understand

Machine Learning – A Practical Approach to
Master Machine Learning to Improve and
Increase Business Results

Anderson Coen
© Copyright 2019 - Anderson Coen - All rights reserved.
The content contained within this book may not be reproduced, duplicated or
transmitted without direct written permission from the author or the
publisher.

Under no circumstances will any blame or legal responsibility be held against

the publisher, or author, for any damages, reparation, or monetary loss due to
the information contained within this book, either directly or indirectly.

Legal Notice:
This book is copyright protected. It is only for personal use. You cannot
amend, distribute, sell, use, quote or paraphrase any part, or the content
within this book, without the consent of the author or publisher.

Disclaimer Notice:
Please note the information contained within this document is for educational
and entertainment purposes only. All effort has been executed to present
accurate, up to date, reliable, complete information. No warranties of any
kind are declared or implied. Readers acknowledge that the author is not
engaging in the rendering of legal, financial, medical or professional advice.
The content within this book has been derived from various sources. Please
consult a licensed professional before attempting any techniques outlined in
this book.
 Description
By reading this document, the reader agrees that under no circumstances is
the author responsible for any losses, direct or indirect, that are incurred as a
result of the use of information contained within this document, including,
but not limited to, errors, omissions, or inaccuracies.
What is machine learning? Does it really help businesses provide better
services and earn more? How can I improve my business processes to
increase ROI (Return On Investment)? I am unable to focus on important
tasks because I am getting bogged down by menial tasks. If you are
confronted by one or more of these questions, this book is for you!
You should also get this book if you have heard machine learning but
couldn’t start because it looks too overwhelming. This book will demonstrate
it’s quite easy, and many situations have similar solutions – the only things
needed are a bit of conceptual knowledge and some free time.
Learn machine learning and data analysis concepts through practical
examples coded in Python. The following libraries are covered in great detail:
● NumPy
● SciPy
● Sklearn (Scikit-learn)
● Pandas
● TensorFlow
● Matplotlib
There’s also a chapter dedicated for an introduction to Raspberry Pi. If you
are interested in robotics and automation, this chapter is not to be missed.
Machine learning is being used in industries that are not directly related to
computer science. If you are new to Python, there is also an introductory
chapter that covers the basics, so you can start with data analysis and
machine learning as soon as possible. Buy this book and start your machine
learning journey today!
 Table of Contents

Introduction
Chapter 1: Machine Learning
What Is Machine Learning Really?
Types of Machine Learning
More Categories of Machine Learning
Machine-Learning Challenges
Present Day Examples
Chapter 2: Coding with Python
Fundamental Programming
Setup
Initializing PyCharm
Data Structures
Advanced Programming
Revisiting Mathematical Concepts
Vectors
Matrices
Basic Statistics
Probability
Distribution
Data Analysis – Foundation of Machine Learning
Python Libraries for Data Analysis
The Pandas Framework
Machine-Learning Projects
Predicting If a Country’s GDP is Related to Its Better Life Index
Predicting Real Estate Prices for Investment
Chapter 3: Working with Raspberry Pi
What is Raspberry Pi?
 Selecting the Model
Hardware Components
First Project
Installation and Setup
Remote Access to Raspberry Pi
Using Camera with Raspberry Pi
Sending and Receiving Signals Using GPIO of the Raspberry Pi
Chapter 4: Working with TensorFlow
The Projects
Project #1: Predicting Student Grade
Project #2: Predicting Student Grade
Project #3: Neural Network using TensorFlow
Chapter 5: Advanced Machine Learning
A Corporate Project
Create a Predictive Chat Bot
Conclusion
Where Will Machine Learning Be in the Next 20 Years?
References
Appendix A: Machine-Learning Concepts
 Introduction
Flying without wings, breathing underwater without gills, climbing
mountains, inventing cures for incurable diseases, taming elemental powers,
landing on the moon, the list of human feats is never-ending. But, there is one
thing that has always eluded mankind – predicting the future. The biggest
disadvantage humans have is never knowing when they are going to die.
There is no way to predict future events with certainty. With all the
technological advancements, the biggest questions are still unanswered. Will
human beings ever be able to correctly predict the future?
Let me ask you a question. Did anyone in the past come close to predicting
the future? How can you know without access to relevant information? The
relevant information required to make a decision is generated from analyzing
accumulated in the past is called data analysis. If we take the same process a
little further, create trends from existing data and use them to predict future
outcomes, we enter the domain called data science.
And, it’s not just about making decisions; for humans, it’s a matter of
survival and growth. Every child relies heavily on data analysis to perform
experiments, gather environment responses, and analyze them to learn and
adapt. When a mother scolds her child because he threw the bowl across the
room, he learns to not do that again. When the same mother hugs and kisses
the child when the first word they say is “ma,” the child learns that to speak
this word is happiness, and will always feel happy saying it for the rest of his
life.
So, where does machine learning fits on all this? Machines are great at doing
repetitive tasks, but a programmer has to write a script to tackle every
possible scenario for the machine to work efficiently. What if, just like a
child, the machine could learn from their experiences and experiments?
Instead of telling the machine what to do for all possible situations, the
machine is told a set of rules. In case an unfamiliar situation happens, the
machine will determine the appropriate response using the rules and gauge
the environment response. If it was favorable, the machine will record it and
apply the same response in a similar situation later on. If an unfavorable
response is met, the machine will use the set of rules again and start over if a
similar situation happens again.
We know that real-life scenarios are much more complex than machine
logics. Applying the same logic on similar situations isn’t suitable when
dealing with complex variables, such as humans. Consider a robot, Ms. X,
who is the maid of a household. The first time she meets her charge, she tries
to joke in front of the little Jana, and she bursts into laughter. Ms. X recorded
this as a successful experiment and during dinner the same night and cracked
the same joke. Jana’s older brother Jack commented, “That’s the lamest joke
ever.” Ms. X is now confused. A human being wouldn’t be confused in these
circumstances; it knows a joke might be funny to one person but not for
someone else. You might think this is just basic, but in fact, learning this
intricacy is one of the things that sets humans at the top of the animal
kingdom.
You might have seen a sci-fi movie or read a futuristic novel that talks about
a hive mind. A collection of robots or machines that have come together to
unlock unlimited processing power gaining abilities like future prediction,
imminent threat detection, then eventually conquering and controlling the
human race. Well, it might be or not be true in the future; one thing is for
certain, we are moving towards fully enlightened independent robots that can
thrive on their own.
The project of machine learning has one main goal – to give machines the
degree of learning and inference capabilities humans possess. Maybe when
machines reach that level, they would be able to accurately predict the demise
of humans and this planet.
I assume you have some basic knowledge about computers and programming
languages before you started reading this book. Don’t worry, every topic I
cover in this book, I will explain right from the basics. I am also assuming
you have access to a good, stable internet connection and an upgraded
computer system in good condition. Data analysis and machine learning
algorithms put a lot of strain on the computer resources. My Master’s degree
thesis was to improve an already well-optimized algorithm. The data set was
huge and the algorithm was very complex. I didn’t have access to a computer
with a powerful configuration. I still remember how I had to leave the
computer on for 20-22 hours for the program to finish processing. If you
want to learn machine learning, start saving, and invest in the latest computer
system.
 Chapter 1: Machine Learning
Watch the Terminator movie series if you haven’t. It indirectly portrays
robots following the age old saying, “If at first you don’t succeed, try, try
again!” The machine network, SkyNet, created by scientists to facilitate
humans in research and development, gets affected by illusions of grandeur.
Assuming machines are the dominant race, SkyNet attacks humans. One
thing leads to another and SkyNet has to send robots to the past to neutralize
one human threat that was the cause of Skynet’s destruction in the future.
SkyNet sends one robot, who, after a lot of trouble and the hero getting killed
in the process, gets sent to the scrapyard by the heroine. SkyNet learns from
the encounter and the next time sends an upgraded robot. Yes, machine
learning at its finest! Well, without recalling anything else from the movie
franchise, here’s the moral of the entire series: machines are faster at
processing in a given direction but humans are faster at adapting in fluid
situations. No matter what kind of robot SkyNet threw out of the production
facility, the human protagonists, albeit taking heavy losses and many times
aided by some machines, always adapted and defeated the evil machines. The
first installment of this movie franchise was released in 1980s starring Arnold
Schwarzenegger as the time travelling robot. It was an instant hit.
But, why I am talking about a movie franchise in a book about machine
learning? The general public in the 1980s had no idea about fully automated
robots. In 2019, it looks very likely we are going to see fully automated
robots in two or three decades. Many things that are now part of everyone’s
lives or look very much possible in the near future have been inspired by
movies. Movie and television franchises like Star Wars and Star Trek have
influenced scientific development for decades. Therefore, it’s important not
to ignore this avenue of inspirations.
Do you remember a movie or TV show where the plot revolves around
machine learning?

What Is Machine Learning Really?

Often considered a part of Artificial Intelligence (AI), machine learning is

described as the reliance of a machine on pattern detection and situational
inference instead of a predetermined set of instructions and the scientific
study of statistical models and algorithms that enable machines to achieve
instructional independence.
Artificial Intelligence, you might be thinking, why I didn’t say that before?
Now, that’s a phrase you would have heard in many movies and TV shows.
Sci-fi movies based on deep space exploration have spaceships operated by
AI that enables the human crew to hibernate in special pods for decades. But,
as is the twist in almost every one of them, the crew is forcibly awakened due
to a catastrophic accident the ship has experienced due to a problem in AI.
But what does it mean that machine learning is a part of AI? For anything to
be considered intelligent, it must have several qualities, and learning-on-the-
go is just one of them. It means a machine might not have AI even though it
learns from the collected data.
The human mind is the most complex and brilliant machine in the known
world. What makes the human mind the best machine in the world up till
now? The ability to relate complex situations and infer a possible course-of-
action. The human mind can imagine and discuss abstract matters. In case of
immediate danger to the human, the mind can shut off all non-essential
systems to prioritize survival protocols. An organic machine that can
arguably operate at the speed of light? That’s definitely a marvel.
But, the human mind is limited by the body that holds it. The best analogy for
the human mind and body is attaching a lightspeed engine to a tortoise; the
potential is infinite but limited by the host’s physical characteristics. This
leads to the answers of a question everything thinks about when starting to
learn machine learning.
“Why are we trying to give machines the ability to learn and make decisions
on their own?” I thought about this question a lot, and I assume, before
moving forward, you would also like to know the answer to this question.
The blunt answer is, humans are lazy with limited physical strength.
1. Humans don’t like performing repetitive tasks, it gets boring fast.
2. An average human can’t lift more than its body weight.
3. Humans are slow with numbers and symbols in general.
4. Humans require frequent breaks.
5. The performance of a human being is directly proportional to its mood
 or mental health.
6. An average human being has a running speed of 45 km/h and no human
can sustain that speed for long.
7. Humans require a very specific thriving environment just to survive.
To overcome all these challenges, humans invented machines. From a wheel
to a crane, spaceships to submarines, humans have developed machines for
every imaginable challenge possible. This is an amazing feat, but also
highlights a serious problem. Humans have to develop a machine specific to
each challenge or modify it to meet the requirements of another challenge.
The Challenger 2 tank can’t do what a Lamborghini can do, a Typhoon can’t
do what Saturn V can do. You might say, I am comparing apples to oranges;
okay, another example, can you drive a Jetta and Lamborghini? Here are a
few of the reasons why the world is looking towards machine learning.
1. Creating a new machine every time a new feature is required is
financially expensive and a huge strain on resources. Remember the
classical typewriter? How difficult it was to type and maintain the
machine, how difficult it was to add erasing capabilities to the
typewriter? A digital keyboard not only made the typing and
corrections faster but, also, reduced maintenance costs in the long run.

2. If the input data is very large in size, it is difficult to find a connection

 without the help of a machine. Consider a police officer who has to
manually go through thousands of records to match the photo of a
suspect captured by a security camera. With image processing and
machine learning, a software will be able to search the records for the
right person, even if you have a side pose of the suspect in the captured
image.
3. How many times does a designed prototype ace the test but fails on the
very first day of real-world application? Back it goes to the testing
facility, wasting some more millions of taxpayer dollars on
experiments. It usually happens because the prototype designers failed
to consider a specific real-world scenario during the design and testing.
What if the machine had the capacity to modify itself in unexpected
conditions? It would save millions of dollars in prototyping and
experimentation stages.

Types of Machine Learning

There are three broad types of machine learning:

1. Supervised learning
2. Unsupervised learning
3. Reinforcement learning

Supervised Learning
A type of machine learning where the machine learns the system by training
on a set of sample data that lead to the desired output. Once the machine is
ready, live data is fed, and the machine predicts future outcomes using the
relation it has established during the training. Supervised learning is a
popular type of machine learning in beginners. Most people starting with
machine learning find supervised learning easier to understand. Why
wouldn’t they? This is how humans learn too!
Classification
An example would be an email filtering system that sends and directs emails
sent to you to either your inbox or spam folder. This type of supervised
learning is called classification where labels are given to data. The system
will be trained on existing emails that have been correctly directed, so the
system can build a relation and automatically direct emails in the future. A
system that classifies data into two classes (spam or inbox in this case) is
called a binary classification system.
If you have the Outlook.com email service from Microsoft, you might have
noticed, there's no way to set a spam filter that most email services provide.
Instead, you have to manually move emails from inbox to spam folder and
vice versa. The email filter of Outlook.com learns by monitoring your
actions, and with time, the spam filtering becomes smarter. It might feel like
a hassle in the start, but it becomes very convenient on the long-term as you
don’t have to worry about blocking every single email address or every
keyword in the spam filter. Of course, in a complex system like Outlook.com,
labeling an email as spam or not is not the only classification task. All such
systems that have various classifications are called multi-class classification
system. A better example of a multi-class classification is gesture recognition
now widely available in smartphones. During setup, the smartphone trains on
user gestures to recognize and differentiate personalized versions of gestures.
Regression
The machine learning applied to systems that take some inputs and produce
continuous outputs is called regression analysis. The system is defined by a
number of (explanatory) variables that are considered to affect the system in a
way that it continuously produces a response variable (outcome). The
analysis finds the relation between the explanatory and outcome variables to
predict future system behavior.
A real-life example would to predict the SAT scores of a few students by
sampling their life and study patterns. Using the data gathered from students
who have already given the test as training for the prediction model, the
learning model will have the ability to predict which future students will have
more chances of passing the test using their behavioral data.
Reinforcement Learning
In reinforcement learning, there is a sort of feedback mechanism (or function)
that provides reward signals when the machine-learning model interacts with
a given system. The signals are used to gradually improve the learning
model. We can say this is the automated trial-and-error method of machine
learning. Reinforcement learning differs from the supervised learning in the
sense that in the former learning model, the machine directly interacts with
the system rather than using previous user-system interactions to learn and
predict future outcomes.
The perfect example of reinforcement learning would be a strategy game
where the computer learns the moves made by the player and gradually learns
to finally beat every move of the player.
Unsupervised Learning
In both supervised and reinforcement machine learning analysis, we would
already know the system’s response to a given set of inputs either by
sampling previous interactions or having the machine interact with the
system. But, what if we have to deal with a data set without knowing what
outcome it results in or what feedback it gives when applied to a system? We
have to resort to unsupervised learning analysis, where meaningful
information is extracted from a data set without knowing the outcome or
feedback of the system. In other words, the data is analyzed without
considering any of the system’s attributes to predict future behavior of the
system.
Clustering
A technique of categorizing data into subgroups (clusters) without any prior
knowledge of the data is called clustering. For example, online marketers
would like to segment the audience based upon their interests without
knowing how they would interact or have interacted with their marketing
campaign. This helps marketers to predict the audience response and create
targeted marketing campaigns for different audience segments.
Another practical application of clustering would be to find anomalies in a
system. For example, a bank would deploy a learning model that tracks all
credit card transactions of a client creating a spending profile. Using this
profile, the system would be able to detect outlier transactions that might be a
case of theft or fraud.

More Categories of Machine Learning

We can also categorize machine learning systems on their ability to

generalize. In simpler terms, when predicting an action or decision for a
future event, it should be able to generalize that event as the same or similar
to an event it encountered during training. Performing well during the
training doesn’t equate to performing well during live action.
So, how do machine learning systems generalize? There are two approaches
to this.
Instance-based
The easiest way of learning something is to learn by heart. But, this approach
becomes tedious over time because memory is limited (in case of machines)
and flawed (in the case of humans). A better way is to perform generalization
to reach a new conclusion. For example, you know that you need 5 minutes
to walk from your house to the corner store. Now, if you want to visit Ben
who lives just across the corner store, you would generalize that you would
need roughly 5 to 6 minutes to reach his house (the extra minute to cross the
road). Let’s take the example of the email filtering system that learns by
tracking your actions of marking different emails as spam. The first phase
would be to learn by heart and mark all identical emails as spam. The next
phase would be to find a measure of similarity between the emails marked as
spam and automatically generalize future emails according to that similarity
model. The measure of similarity can be anything, such as specific words in
the subject or specific email domains, even the time of reception can be used
to find similarity. This will not be a 100% accurate filtering system, but
makes things much easier for the end-user.
Model-based
Another way of generalization is to create a model for the system during
training and use that model for predicting future events and outcomes. For
example, we would like to know if there’s any relation between money and
happiness in people’s lives. We gather the required data and plot it to see if
there’s any trend. If there’s a visible trend, albeit with a few outliers, we can
devise a model according to the trend. If the trend is linear, we can start with
a simpler linear model. Here’s an example.
happiness = �� 0 + �� 1 x money

The �� 0 + �� 1 are two linear model parameters that we need to find

value for before we can start using this model for prediction. To find the
values of these parameters, we can test the performance of our model by
either using a utility function that tells how good our model is, or a cost
function that tells how bad our model is. For linear models, the cost function
makes more sense as we can determine how far the model predictions are
from the training data. Using a linear-regression algorithm for training, the
gap is minimized that results in the best possible values for the model
parameters.
Once the model parameters are decided, we can use it for predictions. In a
later chapter, we are going to take a look at the Python script to create a linear
model to find if money and happiness are truly related.
Sometimes your model will not make good predictions. In that case, you
might want to include more data attributes to the model or use a better model
(such as polynomial-regression model instead of the linear one).

Machine-Learning Challenges

Machine learning is heavily dependent on the training data and the model
(and algorithm) chosen to represent the system. Let’s see what can happen
that fails your machine-learning project.
Insufficient Training Data
Humans can work with very small data set and reach the correct conclusions.
You touch a hot iron and you will probably be cautious around an iron for the
rest of your lives. You might see dark clouds and a fearful storm once in your
life and whenever you see those same clouds again, you will know a storm is
coming. This process starts right from the childbirth and keeps going until the
death. Machines are not so smart, you might say. They require considerable
data to train a machine in using a particular prediction algorithm. For
example, for a successful image recognition project, you might need access to
millions of images as training data.
Various researches in the last two decades have found the same conclusion:
instead of spending resources on better algorithms, spend resources on
collection of large data sets because, given the right amount of data, many
algorithms will reach the same conclusion.
Similar but Not the Same Training Data
For example, you write a script to predict traffic congestion on major New
York highways during rush hours but train it using data collected from Los
Angeles. The data might be of similar type but has no relation with the
scenario we will use our script to predict. We have to make sure the training
data represents the right scenarios.
Poor Quality or Skewed Data
The training data must be free of or contain minimal errors, noise, and/or
outliers. Training a model using erroneous data will affect its performance in
making the right predictions. A better approach would be to clean the training
data, and in fact, most data scientists spend more time preparing the training
data than coding the model and algorithm. Important considerations must be
need in how to deal with outliers (discard or normalize) and missing data
(ignore or fill in with values such as mean, etc.).
Another aspect of sampling training data is to make sure the data isn’t
skewed or biased. For example, how many times you have seen poll results
and accompanied predictions that you know are completely wrong. The most
obvious reason is that the participants of the poll aren’t representative of the
whole situation. For example, if 1,000 US residents from Idaho took part in a
poll, you can’t use this data to make predictions for the entire USA.
Say No to Overfitting
Humans tend to blow up the extent of their experience and apply
indiscriminately. How many times the restaurant waiter or manager has come
to your table and ask for your feedback and you have replied, “Everything
was amazing!” even though the fries weren’t good. Consider there are 100
people who order fries every day from the restaurant, the waiters and
manager are able to collect feedback from 60 people and only 5% of those
told them the fries are bad. The restaurant would discard those three reviews
thinking they are outliers even though they are indicative of a serious issue.
This is how overgeneralization can truly ruin a successful business.
Another example would be you going on a tourist vacation to a foreign
country. The taxi driver you hired from the airport to your hotel charges steep
rates. Due to the frustrating experience, you might form the opinion that all
taxi drivers in that country are manipulators, and you might even avoid taking
a taxi for the rest of your trip. Of course, this is a case of overgeneralized
opinion that doesn’t portray the facts.
Machines can also fall into the same pit; in machine learning it’s called
overfitting. Overfitted data can be a serious issue because the model will
perform exceptionally well on such data during training but fail miserably
when given live data. The process of reducing such overfitting risks is called
regularization.
Underfitting Is Trouble Too
Underfitting is the opposite of underfitting and is caused by the model not
able to properly detect the subtleness of training data. Relating money with
happiness is an issue that’s too complex to accurately predict; there are
infinite variables that can affect the relationship. Therefore, a model to
predict this behavior will never be 100% accurate because it cannot
accommodate all those variables.
Bonus: Testing
How do you test a prediction model after training? You wouldn’t want to ship
it to the users for testing because if they have paid for the product, and the
prediction results are coming up as wrong, your business will suffer. In some
industries, this is the norm; for example, gaming studios release alpha and
beta versions that are free to play and test and then predict performance once
the game is available for purchase. But in other industries, it’s important to
test before releasing the product to the public. One of the basic procedures to
perform testing is to divide the training data into two parts, a training set and
a test set. This way, you don’t need to worry about testing data after you have
trained your model using the training set.

Present Day Examples

Machine learning is still rapidly evolving, but it has become vital in many
industries. Machine learning is helping professionals in various fields to
reach new conclusions that were impossible before. Here are a few examples.
Image Recognition
The modern age is all about images and videos. With the boom of social
media, the selfie craze and the hugely improved smartphone cameras,
everyone captures an image every day. Have you ever uploaded photos on
Facebook and saw the people in them get automatically tagged? Smartphones
have now facial locks so you can lock/unlock your phone using you face.
There are applications that converts handwriting into typed words by learning
the user’s hand strokes.
Speech Recognition
We are very close to getting fully smart homes that you will be able to
control just by using your voice. There are already services like Alexa that
recognize voice commands and perform the actions. Depending upon the
make of your smartphone, there is a virtual assistant that you can activate to
give voice commands. There is also high-level dictation software used by
professionals that learn the characteristics of the user’s voice and offer high
speed speech to text service.
Medical Applications
Machine learning has enabled medical professionals to find new cures and
predict viral outbreaks. By analyzing tracked information of the patients,
accurate predictions can be made remotely in regards to health deterioration
and onset of illness. Applying the image recognition benefits of machine
learning with medical imaging techniques such as MRI, etc., helps detect
diseases at a nascent age that a doctor’s naked eye might miss.
Finance and Trading
Machine learning can also help with stock trading and making better financial
decisions through accurate market predictions. Various algorithm techniques
are used to find correlation between different signals and used to formulate a
short-term investment strategy.
Product Associations
It is very important for businesses to find the impact of one product over the
other. For example, McDonalds might want to predict what impact
discontinuing an item would have on the sales of other items before making
the decision.
Classification
Government and private institutions like to classify people and clients
according to their risk potential. For example, your bank would like to use
your financial history to predict your future before accepting your loan
application.
Customer Sentiment
Businesses would like the predict customer sentiment before launching a new
brand. This is extremely useful to choose the correct time for launch to
maximize customer participation. This might include competitor analysis and
case studies of similar businesses.
Artificial Intelligence
Machine learning is considered a stepping stone in achieving complete
artificial intelligence. Consider a system that’s fully aware of all its
parameters and already knows the future outcomes if any of those parameters
change. It would require tremendous processing power but will help humans
achieve unimaginable results. There are governments already considering the
use robots to avoid losing precious human lives in wars. How about sending
robot in space that have the same mind power of a human but aren’t
constrained by the physical attributes of human body? We might be able to
mine different planets of the solar system for resources that are already
running out on Earth.
 Chapter 2: Coding with Python
Python is a multi-purpose language. It means it can be used to code and
resolve issues in various applications. Starting out as a simple to write, simple
to read programming language, it has grown into the most popular language
in the data analysis and machine-learning industries.
But, why did Python become so popular in data analysis compared to
languages with similar capabilities? Data analysis can be used in any field,
including finance and medical. People who are not professional programmers
have to crunch numbers for analysis and research. A programming language
that’s easier to code but powerful enough to handle complex data generated
in real-life applications was bound to become popular.
The open-source nature of the programming language also helped spread its
use because many online and offline instructors adopted Python to teach
various algorithms and programming techniques. When you are new, there is
usually a lot of confusion and you need as much resources as you can find.
Due to numerous, highly varied applications of the programming language,
organizations and volunteers have published different packages (usually
called distributions) to facilitate faster and easier startup. If you want to learn
the basics of programming and Python, you can start with using any online
Python environment. One such example is https://repl.it/languages/python3

Fundamental Programming

For this book, I have chosen to use PyCharm and Anaconda with Python 3.7.
PyCharm is a very popular and sophisticated Integrated Development
Environment (IDE) that provides various advantages over IDLE, the standard
Python IDE. Anaconda is a specialized distribution that provides all the
libraries related to data analysis and machine learning by default. It helps
programmers avoid wasting time on finding and installing the required
libraries that don’t come with the standard Python distribution.
Anaconda also provides a package management system “conda” that helps
create virtual environments. Virtual environments are used to install different
setups for different projects on the same computer system. It helps avoid
conflicts between packages. If you are going to only work on one application
for the rest of your life (which is rarely the case), you can skip virtual
environment setup. But, it’s highly recommended.

Setup

Setting up PyCharm and Anaconda is a bit tricky, so make sure you go

through the process with utmost care. One small mistake can break the
connection between PyCharm and Anaconda.
Install PyCharm
Go to the website URL https://www.jetbrains.com/pycharm/download/ to
download the latest version of PyCharm for your system. The link will
automatically redirect to the correct URL for your operation system, for
example, for me running Windows, the URL becomes
https://www.jetbrains.com/pycharm/download/#section=windows.
Select the “community” version, so you don’t have to pay in the future. After
the download is complete, start the installation process. Make sure to check
the “Add launchers dir to the PATH” option, in the window below.

I installed PyCharm in the same directory where Python is installed but that’s
optional.
Install Anaconda
Go to the URL https://www.anaconda.com/distribution/, download the
correct installer (I chose the one for Windows – 64 bit), and install once the
download is complete. Note that Anaconda is a huge distribution without
hundreds, if not thousands, of Python libraries. After-install size is a
staggering figure of 2.5 Gigabytes (GB). During the install, you will see the
following screen; make sure you check the box that says “Add Anaconda to
my PATH environment variable” even though it isn’t advised. Checking this
option will make your life a lot easier using Anaconda in the future.

The installation can take a considerable amount of time depending upon the
computational power of your computer system. After the installation is done,
we can test if Anaconda is working correctly. Open Windows start menu and
type “Anaconda Prompt”. Type “python” and Python will start.
Bonus: Install TensorFlow
TensorFlow is a great platform to build and train machine-learning models.
Let’s install it on our system using a virtual environment. What was a virtual
environment? To create isolated spaces to keep required tools, libraries, etc.,
separate for different projects, virtual environments are used. This is very
useful for programming languages that can be used for multiple application
fields. Imagine yourself working on two projects, one on machine learning
and one on developing a 2D game. Creating a virtual environment would help
keep things organized for both projects.
Let’s open Windows command prompt. We are going to use Anaconda to
create a virtual environment and install TensorFlow in it. The latest version
of TensorFlow 2.0 is still not fully compatible with standard Python 3.7. You
can find so many programmers running into issues trying to install
TensorFlow 2.0 on Python 3.7. This is another reason to use Anaconda to
install TensorFlow; you will avoid any compatibility issues. Enter the
following command in the Windows command prompt to create a virtual
environment and install the platform in it.
conda create -n tf TensorFlow
“-n” means new, “tf” is the name we gave to the virtual environment getting
created with “TensorFlow” installed in it. During the install, you might be
asked to proceed with installing some new packages. Just type “y” and hit
“ENTER” to continue. Once complete, you can activate this newly created
virtual environment with the following command.
conda activate tf
You will now see “(tf)” written before the current path in the command
prompt. It tells you are working in a virtual environment.
Bonus: Install Keras
Keras is a high-level API for creating neural networks using Python. While
we are in the same virtual environment “tf,” we can invoke the standard
Python package manager “pip” to install Keras.
pip install keras
Once everything is installed and ready, we can jump to PyCharm to finalize
the setup and starting coding.
Initializing PyCharm

Search “pycharm” in Windows start menu and you will see an option that
says “JetBrains PyCharm Community Edition...”. Select the option and wait
for a window to pop up. This is how it would look like.
Click on “Create New Project” to start. It will ask for a location and name. I
used “tensorEnv” as project name; you can choose whatever name you like.

Now, the PyCharm interface will load. Right-click where it says “tensorEnv”
in the “Project” and select “New” and then “Python File”. Enter a name; I
used “test.py”.
There’s two more steps to do. First is setting the right Python interpreter for
the project and then adding a configuration to the project. Click on “File”,
and select “Settings”. In the new pop up, select “Project Interpreter” under
“Project: tensorEnv”.

Click on the gear icon on the right side of the screen, select “Add...” You will
see a new window on your screen.
Select “Conda Environment” and then select “Existing environment”. In the
above image, you are seeing the following value in the “Interpreter” field.
D:\LearningNewSkills\Python\Anaconda3\envs\tf\python.exe
For you, that field might be empty. We have to navigate and select the correct
interpreter “pythonw.exe” for our project (notice the ‘w’). Hint: “python.exe”
and “pythonw.exe” are usually in the same folder. Remember “tf” in the
above path is the environment we created using Windows command prompt.
Replace “tf” with the environment name you used then when looking for the
right folder. Once the right file is selected, click “OK” until you are back on
the main PyCharm interface.
Now, we have to setup the configuration. Click on “Add Configuration” on
the top-right side of the screen.

Click on the ‘+’ sign and then select “Python”.

In the new window, add the name you want for the configuration. Usually,
the name should be the same as the file you created before, so we are going to
use “test”. In the “Script path” field, click on the folder icon and navigate to
the file you created, which is “test.py” in our case. Once the file is selected,
click “OK” until you are back on the main PyCharm interface.
If you are thinking this is quite complex, you are right. Good news is, you
don’t need to do all the steps every time you start a new project. You also
don’t need to use PyCharm, which is just the IDE. You can install Anaconda,
TensorFlow, and Keras on the standard IDLE. But, PyCharm is worth all the
trouble we just went through.
Let’s do the “Hello, World” program. In the test.py we created, write the
following line of code.
print('Hello, World!')
To run the script, click on “Run” and then “Run ‘test’” (you will see your
Python file name instead of test).

You will see the output in the “Run” section.

Other important thing to mention here is the “Terminal”, which is the

Windows command prompt, so you can perform system action without
leaving PyCharm. Don’t forget the “Python Console”, that you can use to
execute a single line of code, very useful for testing. There is also the
“TODO”, which is a to-do list for PyCharm; let’s leave it for later.
“Process finished with exit code 0” is Python interpreter’s way of telling us
that everything worked without any errors. We can suppress this output,
which we will look into that later.
We are going to come back to this “tensorENV” when we start with machine
learning. Using the above steps, create a new virtual environment and call it
“basics”. Create a new project “Basics” and add the file “firstprog.py” in it.
We are going to print “Climate change is real!” 10 times.
for i in range(0, 11):
print('Climate change is real!')
You will see the following output.
Climate change is real!
Climate change is real!
Climate change is real!
Climate change is real!
Climate change is real!
Climate change is real!
Climate change is real!
Climate change is real!
Climate change is real!
Climate change is real!
Climate change is real!

Process finished with exit code 0

We have used a “for” loop to perform an operation (“print()” method) several
times. Another method used here is the range() that creates a sequence when
given a starting and ending point. The example proves that range() stops the
sequence one value short of the ending point (stopped at 10 when 11 was
given as ending point).
We can do the same with a “while” loop which is beneficial in situations
where we don’t know beforehand how many times the operation must be
performed.
i=1

while i < 11:

print('Climate change is real!')
i += 1
Let’s write a program to print a ladder of ‘#’ to the number of levels user
wants.
i = int(input('Enter the level of ladders you want to create: '))

for x in range(1, i+1):

spac = ' ' * (i-1)
char = '#' * x
print('%s%s' % (spac, char))
i -= 1
The output of the above code will be like below if user inputs 10.
Enter the level of ladders you want to create: 10
#
##
###
####
#####
 ######
#######
########
#########
##########
It looks more like a cone because space and “#” don’t take up equal physical
space in a word processor. In PyCharm, here’s how the output looks.

In this script, we have involved the user using the input() method. We have
used the int() to perform string to integer conversion because input() always
registers the user input as a string. “i -= 1” is a short version of “i = i -1”.

Data Structures
Lists and Tuples
Lists are the simplest data structure in Python, but that doesn’t mean it’s
primitive.
aLst = [1, 3 , 5.55, 2, 4 ,6, 'numbers', 'are', 'not', 'letters']

print('Number of elements in the list: ', len(aLst))

for itm in aLst:
 print(itm)
The script will output the following.
Number of elements in the list: 10
1
3
5
2
4
6
numbers
are
not
letters
Let’s do some more.
bLst = ['letters','are','forever',7,8,9]
cLst = aLst + bLst

print('Number of elements in the list: ', len(cLst))

for itm in cLst:
print(itm)
The output is below.
1
3
5
2
4
6
numbers
are
not
letters
letters
are
forever
7
8
9
The “+” is an arithmetic operator, but by using it on two (or more) lists, it
joins all the lists. There are many other ways of concatenating lists like join()
and append(). In this script, we have used len() method to count the number
of elements in a list. All data structures in Python are iterable, which is
evident with the use of “for” loop on the list.
The order of elements in a list is fixed and governed by the order they are
added/appended to the list, called index of that element. The first element of a
list has zero index.
cLst[0]
1
clst[len(cLst)-1]
9
We can slice a list using the indices that returns a new list. As we will see
soon, list is not the only data structure in Python to support slicing.
cLst[0:5] #will return elements with index of zero to index of four
[1, 3, 5, 2, 4]
A new element is always added at the end of the list.
cLst.append(‘New element’)
print(cLst)
[1, 3, 5, 2, 4, 6, 'numbers', 'are', 'not', 'letters', 'letters', 'are', 'forever', 7, 8, 9,
'New element']
We can also remove elements from a list using various methods, including
pop() where you can pass the index of the element you want to remove from
the list. If no index is provided, the last element is removed.
cLst.pop()
print(cLst)
[1, 3, 5, 2, 4, 6, 'numbers', 'are', 'not', 'letters', 'letters', 'are', 'forever', 7, 8, 9]
Here is another reason why Python is so popular in data science applications,
it is very smart with memory management. The biggest challenge for
memory management is to handle duplication and decide when it’s essential
and when it’s superfluous. Here is one example of Python’s memory
management in action.
cLst.append('New element') #add back the popped element
dLst = cLst
dLst.pop()
print(cLst)
print(dLst)
[1, 3, 5, 2, 4, 6, 'numbers', 'are', 'not', 'letters', 'letters', 'are', 'forever', 7, 8, 9]
[1, 3, 5, 2, 4, 6, 'numbers', 'are', 'not', 'letters', 'letters', 'are', 'forever', 7, 8, 9]
The assignment operators work a little differently in Python if applied to a
data structure. Instead of creating two lists with identical elements, Python
creates a new pointer “dLst” that points to the original list “cLst”.
“dLst.pop()” therefore changes the original list “cLst”. We can say cLst and
dLst are two names of the same entity.
The elements in a list can repeat.
dLst = [1, 1, 2, 3, 4, 5, 5]
print(dLst)
[1, 1, 2, 3, 4, 5, 5]
We can also reverse and sort a list.
eLst = [5,7,2,4,7889,7984,21,4,8]

print(list(reversed(dLst))) #get a new list with elements in reversed order but

without sorting, doesn't modify the original list

eLst.sort(reverse=True) #descending sorting of list, modifies original list

print(eLst)

print(sorted(eLst)) #returns an ascending sorted list, doesn't modify original

list
The above print methods will output the following, respectively:
[5, 5, 4, 3, 2, 1, 1]
[7984, 7889, 21, 8, 7, 5, 4, 4, 2]
[2, 4, 4, 5, 7, 8, 21, 7889, 7984]
Tuples are just like lists but have following distinctions
1. Tuples are unordered – elements don’t have fixed position (index)
2. Tuples are immutable – elements can’t be added or removed later
aTup = (1, 2, 3, 'first')
print(aTup)
(1, 2, 3, 'first')
Tuples don’t support methods that change the tuple, which make them ideal
for someone to pass information to others.
bTup = (3, 4, 5, 'second')
cTup = aTup + bTup
print(cTup)
(1, 2, 3, 'first', 3, 4, 5, 'second')
print(len(cTup)) #
8
print(cTup.count(3)) #count how many times “3” is in the tuple
2
print('first' in cTup) #check if “first” is an element of the tuple
True
We can create new tuples out of existing ones.
dTup = (2, 5, 33, 5, 7)
eTup = tuple(reversed(dTup))
print(eTup)
The output is.
(7, 5, 33, 5, 2)
Strings
Strings are an immutable collection of characters. Strings are a basic data
type in Python but also behave like a data structure (list). This is the reason
many data structure methods are applicable on strings. One of the reasons
Python has taken over as the best data analysis programming language is the
ease it offers in manipulating strings.
aStr = "This is a\nsimple string"
bStr = r"This is a\nraw string. Special characters like \n are taken as simple
characters."

print(aStr)
print(bStr)
The above will result in the following output:
This is a
simple string
This is a\nraw string. Special characters like \n are taken as simple characters.
Note how the first string takes two lines because the “\n” is taken as a
newline.
We have already seen that when taking input from the user, the input data is
always taken as a string, and in some cases, we have to perform type
conversion to get the correct data.
inp = input("Enter a number: ")
print(type(inp))
Here is a test of the above code.
Enter a number: 110
<class 'str'>
Even though the input looks like a number, it isn’t, and we have to convert to
integer.
inp = int(inp)
Likewise, we can convert other data types into string.
intNum = 245
cStr = str(intNum)
print(type(cStr), cStr)
The output will be.
<class 'str'> 245
Let’s perform some data structure operations on a string.
# get number of characters in a string
print(len(aStr))
print(len(bStr))

# get specific character using its index

print(aStr[len(aStr)-1]) #last character of aStr

# slicing a string to get a new string

print(aStr[0:4]) #last index is exclusive
print(aStr[-6:]) #negative indices can be used
The respective outputs will be.
23
81
g
This
string
Now, a few methods specific to the string data type.

# some string specific methods

print(aStr.lower()) #returns new string with all lowercase letters
print(cStr.strip()) #removes whitespace from start and end of string
print(aStr.replace("\n"," ")) #replaced newline with a space
print(bStr.split(r"\n")) #creates a list of substrings, note the "\n" passed as a
raw string
print("<missing>".join(bStr.split(r"\n"))) #join a list to create a string
print(aStr + bStr) #concatenate two strings
print("\"%s\" is a string" %bStr)
The respective outputs are below.
this is a
simple string
This is another string
This is a simple string
['This is a', 'raw string. Special characters like ', ' are taken as simple
characters.']
This is a<missing>raw string. Special characters like <missing> are taken as
simple characters.
This is a
simple stringThis is a\nraw string. Special characters like \n are taken as
simple characters.
"This is a\nraw string. Special characters like \n are taken as simple
characters." is a string
We can also use a loop on a string.
for char in aStr:
print(char, end=" == ")
The output is.
T== h== i== s== == i== s== == a==
== s== i== m== p== l== e== == s== t== r== i== n== g==
Dictionaries
A very unique key-value pair data structure, dictionaries are mutable,
ordered, and iterable.
aDict = {
"a":"A member",
"b":"B member",
0:[2, 4, 6],
1:[3, 5, 7]
}

print(aDict["a"])
print(aDict[0])
print(aDict[0][2])
The output is.
A member
[2, 4, 6]
6
Let’s change a value and then a key.
aDict["a"] = "First member"
print(aDict["a"])
aDict["1st"] = aDict.pop("a")
print(aDict)
The output for both print methods is below.
First member
{'b': 'B member', 0: [2, 4, 6], 1: [3, 5, 7], '1st': 'First member'}
Note, the new key-value pair now appears at the end of the dictionary.
We can get all the keys and values as separate lists from a dictionary.
print(aDict.keys())
print(aDict.values())
The output is a little strange-looking, see below.
dict_keys(['a', 'b', 0, 1])
dict_values(['A member', 'B member', [2, 4, 6], [3, 5, 7]])
The “dict_keys” and “dict_values” are view objects and don’t reside in the
memory. You can say they are merely references to the original dictionary.
You cannot iterate over these view objects, but you can check membership.
print('B member' in aDict.values())
The output is.
True
Checking if a key is in a dictionary is simpler.
print("a" in aDict)
The output is.
True
If we try to get a value using a key that is not present in a dictionary, an error
occurs.
print(aDict["c"])
The output will be.
KeyError: 'c'
There is a safe method, get(). If the key is not found, the default message is
returned instead of raising an error.
print(aDict.get("c","Key not found"))
The output will be.
Key not found
Sets
A dictionary with only keys and no values is called a set. This is the closest
you will get to mathematical sets in Python. Sets are mutable, iterable,
unordered, and all elements must be unique. Because the sets are unordered,
the set elements don’t have fixed indices; therefore, retrieving elements with
indices and slicing doesn’t work.
aSet = {'data collection', 'data analysis', 'data visualization'}
bSet = set([1, 2, 3])
cSet = {1, 1, 2, 3}

print(aSet)
print(bSet)
print(cSet)

print(2 in bSet)
print((bSet == cSet)) #ignored duplicate element leading to cSet and bSet
having the same elements
The output of the above code will be.
{'data collection', 'data analysis', 'data visualization'}
{1, 2, 3}
{1, 2, 3}
True
True
Let’s see what more we can do with sets.
#set operations
print(bSet & cSet) #set intersection – return common elements
print(aSet | bSet) #set union – join sets
print(bSet - cSet) #set difference – return elements of bSet that are not in
cSet, null set in this case
cSet.add(4) #add a single element, ignored if element already present in the
set
print(cSet)
cSet.remove(4) #raises error if element is not found in the set, differs from
pop() as it doesn't return the removed element
print(cSet)
cSet.discard(4) #ignored if element isn't found in the set instead of raising an
error
nums = [4, 5, 6]
cSet.update(nums) #adds multiple elements to the set at the same time
print(cSet)
cSet.clear() #removes all elements making the set empty
print(cSet)
The output is.
{1, 2, 3}
{'data visualization', 1, 2, 'data collection', 3, 'data analysis'}
set()
{1, 2, 3, 4}
{1, 2, 3}
{1, 2, 3, 4, 5, 6}
set()

Advanced Programming
Python is a high-level language that offers various distinct functionalities
such as comprehensions and iterators. But, first, let’s look at how to create
functions and handle exceptions.
Creating and Using Functions
Functions are code blocks that are explicitly called to perform specific tasks.
They can, optionally, take single or multiple inputs and return a single output.
The best practice is that a function should not have more than two inputs
(also called function arguments). The single output can be a data structure if
more than one value needs to be returned.
Here is a complete script with the function “checkodd()” that takes an array
of integer numbers and decides if each element is odd or even.
def checkodd(num):
arr = []
for itr in num:
if(itr%2 != 0):
arr.append(str(itr))
return arr

def main():
inputArr = []
for i in range(1, 11):
inp = int(input("Enter an integer number: "))
inputArr.append(inp)
oddNums = checkodd(inputArr)
print("The odd numbers in the user inputs are: ", ",
".join(oddNums))

if __name__ == "__main__":
main()
Here is the result of a test I did.
Enter an integer number: 3
Enter an integer number: 5
Enter an integer number: 4
Enter an integer number: 2
Enter an integer number: 6
Enter an integer number: 9
Enter an integer number: 8
Enter an integer number: 7
Enter an integer number: 12
Enter an integer number: 31
The odd numbers in the user inputs are: 3, 5, 9, 7, 31
Let’s discuss the different aspects of this code.
The definition of a function should appear before it’s call. It’s not a
requirement but rather a good practice rule. I had to pass 10 inputs to the
function, so I combined all the inputs in an array and passed to the function. I
applied the same logic to the function return.
The “if __name__ == "__main__":” is where the execution of the script
starts. You can say it acts like a pointer because every script you write can be
imported into another script as a module. In that case, Python needs to know
where the script execution should start.
This a very simple example where we could have achieved the results
without writing a custom function.
Variable Scope
When writing functions, it’s important to understand the scope of a variable.
It means where you have declared a variable and where it’s available. A
variable can be either local or global depending upon where they are
declared.
In the above script, “arr” is a local variable (list) of function checkodd(). If
we try to access it from main() function, we will get an error. In the same
way, the list “inputArr” is local to the main() function, if we try to access it
from checkodd(), we will get an error. Here’s a benefit of using PyCharm. If
you try to access a variable that isn’t available in a function, PyCharm will
give an error warning.

If, we do want to access a variable across functions, we have to explicitly

declare it as a global variable. To access “inputArr” inside checkodd()
function definition, we have to change the main() definition to below.
def main():
global inputArr #making the variable global
inputArr = [] #declaring it an empty list, unfortunately, we cannot
declare the values and scope of the variable in the same line
for i in range(1, 11):
inp = int(input("Enter an integer number: "))
inputArr.append(inp)
oddNums = checkodd(inputArr)
print("The odd numbers in the user inputs are: ", ",
".join(oddNums))
If you don’t want to use the “global” keyword, we have to relocate the
“inputArr” declaration to the top of the script. Here’s the complete updated
script.
inputArr = []
def checkodd(num):
arr = []
print(inputArr)
for itr in num:
if(itr%2 != 0):
arr.append(str(itr))
return arr

def main():
for i in range(1, 11):
inp = int(input("Enter an integer number: "))
inputArr.append(inp)
oddNums = checkodd(inputArr)
print("The odd numbers in the user inputs are: ", ",
".join(oddNums))

if __name__ == "__main__":
main()
The output now, with some random inputs below.
Enter an integer number: 45
Enter an integer number: 2
Enter an integer number: 36
Enter an integer number: 785
Enter an integer number: 1
Enter an integer number: 5
Enter an integer number: 2
Enter an integer number: 3
Enter an integer number: 5
Enter an integer number: 7
[45, 2, 36, 785, 1, 5, 2, 3, 5, 7]
The odd numbers in the user inputs are: 45, 785, 1, 5, 3, 5, 7
List Comprehensions
In previous examples, we have used loops to iterate over a list. Python offers
a faster way that’s more powerful than using a loop on a list in many
situations. It is called list comprehension.
Let’s create a function to populate an empty list with prime numbers between
1 and 100.
import math
primeNums = []

def populatePrime(num):
max_divizr = 1 + math.floor(math.sqrt(num))
for divizr in range(2, max_divizr):
if(num % divizr == 0):
return
primeNums.append(str(num))
return

def main():
 for i in range(2, 101):
populatePrime(i)
print("The prime numbers between 1 and 100 are: ", ",
".join(primeNums))

if __name__ == "__main__":
main()
The script imports a standard library in Python “maths” to implement a faster
algorithm to find prime numbers in a given range. Mathematically, we know
that an integer cannot be a prime number if it’s divisible by a number less
than or equal to its square root. We also know that range() never includes the
maximum value given. We have declared a global list “primeNums” to
collect the prime numbers. We can make the code more dynamic by adding
an input() asking the user to enter the maximum range the prime numbers
should be found in. We have to change the main() function definition to
below.
def main():
inp = int(input("Enter an integer number. The prime numbers will
be output up until this number: ")) + 1
for i in range(2, inp):
populatePrime(i)
print("The prime numbers between 1 and 100 are: ", ",
".join(primeNums))
Notice the one we added to the user input to get the correct maximum range
value with the variable “inp”.
We can use list comprehensions instead of using the nested for-if in the
populatePrime() function. Here’s the updated function definition.
import math
def populatePrime(num):
return [str(x) for x in range(2, num) if all(x % y for y in range(2, 1
+ math.floor(math.sqrt(x))))]

def main():
inp = int(input("Enter an integer number. The prime numbers will
be output up till this number: "))
primeNums = populatePrime(inp)
print("The prime numbers between 1 and 100 are: ", ",
".join(primeNums))

if __name__ == "__main__":
main()
The output of the above code if user enters 100 is.
Enter an integer number. The prime numbers will be output up till this
number: 100
The prime numbers between 1 and 100 are: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29,
31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97
As you can see, the results are the same. List comprehensions offer, in most
cases, faster execution times but are very hard to read for people who are new
to Python. This is evident from the below line of code.
[str(x) for x in range(2, num) if all(x % y for y in range(2, 1 +
math.floor(math.sqrt(x))))]
You don’t have to use list comprehensions if you don’t want to, but when
dealing with large data sets, the execution times do matter.

Revisiting Mathematical Concepts

Vectors
An object with magnitude and specific direction is called a vector. We can
visualize a vector as an arrow figure whose length represents the magnitude
and the arrowhead indicating its direction. The opposite of vector quantities
are scalar quantities that have only magnitude.
One of the easiest examples of scalar and vector quantities is speed and
velocity. For example, a car is travelling 50 kilometers every hour towards
south. The speed of the car is 13.89 m/s and it’s velocity is 50 km/hr North.

There is one unique vector that doesn’t have a particular direction, the zero
vector. Represented with a bolded 0, the zero vector has zero magnitude and
doesn’t point in a specific direction.
Why are vectors important? Let’s consider distance vs. displacement.
Without direction, distance covered doesn’t portray the complete story. You
might have traveled 10 kms to the mall and 10 kms back to your home,
travelling a total distance of 20 kms. But, your displacement is zero because
you ended up where you started. It is this reason displacement is always used
for navigational purposes.
Vector Operations
Moving a vector while keeping its magnitude and direction intact is called
translation. Translation doesn’t change a vector and helps in performing
vector operations.
Addition
To add two vectors a and b, which is written as a + b, we translate vector b
so its tail touches the head of vector a. Now, join the tail of vector a with the
head of vector b, this new vector shows the result of a + b.
Addition operation obeys the cumulative law; it means the order of operators
doesn’t matter.

Subtraction
Subtraction is also addition between two vectors but direction of the second
vector is reversed. So, to calculate b - a, we actually perform b + (-a).
Scalar Multiplication
Scalar multiplication is multiplying a vector with a real number (called
scalar). The direction of the vector remains the same but the magnitude of the
vector increases by the scalar value.
Vector Multiplication
There are two different types of vector multiplication, dot product and cross
product.
Dot product is used to find how much one vector is in the same direction as
another vector. Dot product doesn’t have anything to do with the magnitude
of the vectors. A positive result means both vectors have similar direction;
zero indicates both vectors are perpendicular; while a negative results shows
both vectors have almost opposite direction. Dot product always return only a
number.
Cross product of two vectors results in a new vector that is perpendicular to
the vectors and a magnitude equal to the area of parallelogram created by the
two vectors. If two vectors are parallel, the cross product is zero, and if the
vectors are perpendicular, the cross product is maximum. The direction of the
new vector can be easily found using the right hand rule. If you point your
index and middle fingers in the direction of two vectors, the direction of your
upright thumb represents the direction of new vector.

Matrices
An arrangement of numbers, symbols, and/or their combinations in rows and
columns forming a rectangular array is called matrix. A matrix with 3 rows
and 3 columns with all elements equal to one is written as follows.
Matrix Operations
Various scalar operations including addition and subtraction can be
performed on a matrix. A matrix can also be operated along with another
matrix.

“17” and “4” comprise the primary diagonal while “6” and “3” comprise the
secondary diagonal.
Identity Matrix
A square matrix that has all elements in primary diagonal set as one and all
elements in the secondary diagonal set as zero is called an identity matrix.
Here is a 2x2 identity matrix.

Scalar Multiplication
If a matrix is multiplied by a number, it’s called scalar multiplication.

Matrix Addition
We can add two matrices which requires both matrices to have same number
of rows and columns. The element of one matrix is added with the element of
the other matrix with the corresponding position.

Matrix Multiplication
Two matrices can be multiplied only if the number of columns of first matrix
is equal to the number of rows of second matrix. The resultant matrix will
have a number of rows of first matrix and number of columns of second
matrix.

Matrix Determinant
Determinant is a special characteristic of a matrix that helps in various
applications such as finding the inverse of a matrix to solve some linear
equations. The matrix must be square, i.e., same number of rows and
columns. The determinant can be zero. Let’s consider the below matrix.

The determinant of this matrix is found by multiplying elements in the

secondary diagonal and subtracting the result from the multiplication of
elements in the primary diagonal. The operation gives the determinant of the
above matrix as: 17 x 4 - 6 x 3 = 50.
Matrix Inversion
If the determinant of a matrix is zero, we cannot find its inverse. In all other
cases, we reciprocate the determinant and multiply by the adjugate of the
original matrix. To find the adjugate of a matrix, we swap the position of
elements in primary diagonal and change the signs of elements in secondary
diagonal.

If we multiply the above matrix with the original matrix, we will get an
identity matrix.
Bonus: we can write a vector as a single column or single row matrix. This
way, all matrix operations can be applied to vectors.
Matrix Transpose
Transposing a matrix involves switching the elements in the rows to columns.
Let’s consider the following matrix.

The transpose of the above matrix is.

Basic Statistics

You cannot perform data analysis and machine learning without basic
statistics. In most machine-learning applications, you train your script to
analyze an entire data set through a single, yet varying perspective and find
related potential predictors. The statistics that involves a single
variate/variable is called univariate statistics.
Univariate statistics are largely based on linear models which are heavily
involved in machine learning.
Outlier
Have you ever thought about the concept of superheroes? People among us
(the data set) but very different? So much different that they outside the
characteristic boundaries of all else. An average untrained American male
can lift around 155 lbs. How much superman can lift? According to the
comics, 2 billion tons! If we include superman in the calculation of how
much an average American male can lift, the result will not be truly
representative of the general population (data set).
Detecting outliers in a data set and what to do with them is very important. In
most cases, outliers are discarded because they don’t represent the data set
correctly. This is because outliers usually happen due to an error while
gathering data. But, that’s not always the case. Let’s take an example. An
industrial oven bakes cookies, and a sensor is used to monitor the oven’s
temperature. The oven has two doors; Door A is used to feed cookie dough
into the oven while Door B is used to takeout baked cookies. The sensor
records the temperature in Fahrenheit every second. Here are the readings.
349.5, 350.5, 350.1, 350, 150, 348.5, 349, 350, 349.5, 149.25, 351.5, 350,
349.5, 350.1, 149.7
Something strange is happening; every 5 seconds, the temperature drops well
below the desired temperature range to bake cookies (ideally the temperature
should be 350°F). There can be two possibilities:
1. The sensor is malfunctioning.
2. Heat is leaking.
1 is a possibility, but the data is too precise to indicate any malfunction. The
temperature drops every 5 seconds like clockwork. 2 is a greater possibility,
but why is it happening all the time? Something must be happening every 5
seconds to cause this heat loss. After some careful process investigation, it
was found that the temperature drops steeply because both the oven doors, A
and B, are opened at the same time. Opening both the doors by a few seconds
apart resolves the issue to a great extent.
Average
The concept of average is to find the centerpoint of a data set. Why? The
centerpoint of a data set tells a lot about the important characteristics of the
data set.
Mean
The most common average is the mean. It is calculated by summing all the
elements in a data set and dividing that sum by the number of elements in the
data set. Remember when we said “an average untrained American male can
lift around 155 lbs?” It was a mean of the weight lifted by a specific number
of untrained American males. Mean gives as a general idea of the entire data
set.
349.5, 350.5, 350.1, 350, 150, 348.5, 349, 350, 349.5, 149.25, 351.5, 350,
349.5, 350.1, 149.7
These were the temperatures recorded every second in an industrial oven.
How to calculate the mean? We add all the readings and divide by the
number of readings. The mean comes up as 4647.15/15 = 309.81°F. The
temperature should have remained around 350°F for the best results.
Median
Median represents the true center by position in a data set. The data set is first
sorted in ascending order. If the data set has an odd number of values, the
median is the value that has an equal number of values on both sides. Let’s
find the median in our industrial oven example. First, we sort the data set in
ascending order.
149.25, 149.7, 150, 348.5, 349, 349.5, 349.5, 349.5, 350, 350, 350, 350.1,
350.1, 350.5, 351.5
There are 15 recorded values which is an odd number. The median in this
case is 349.5, the value at 8th position because it has an equal number of
values (7) on both sides. If we had an even number of values, we would have
to calculate the mean of the two values that have an equal number of values
on both sides. For example, here’s how many kilometers I drove every day
for the last eight days.
101, 215, 52, 87, 64, 33, 459, 16
Let’s sort the data set.
16, 33, 52, 64, 87, 101, 215, 459
The two middle values are 64 and 87. The median in this case is ( 64+87 ) / 2
= 75.5 kms
Mode
The most repeated (common) value in a data set is called mode. If none of the
values repeat, the data set doesn’t have a mode. In our industrial oven
example, 350 appears three times and hence is the mode of the data set.
Variance
It is the measure of variability of each value in the data set with respect to its
mean. It is used in investments and revenues to optimize various assets to
achieve a target average. Let’s take an example. You have a physical store, an
online store and a kiosk in the mall. Your physical store generates a revenue
of $30k a month, your online store generates a revenue of $24k a month and
the kiosk generates $12k a month. Your average revenue is $22k each month.
To see which asset contributes more to your average revenue, we use the
formula:
sum of ( asset revenue - mean revenue )2 / number of assets.
For our stores, the variance is, ( ( 30 - 22 )2 + ( 24 - 22 )2 + ( 12 - 22 )2 ) / 3 =
56
The lower the variance the lesser away the individual revenue contributions
are from the average revenue.
Standard Deviation
We find standard deviation by taking the square root of variance. For the
above example, the standard deviation is 7.48. Standard deviation shows
what’s the extent of deviation of the data set as a whole from its mean.
Probability

The likelihood or chance of something to happen is called probability. In

machine learning, probability is one of the most important concepts at play.
In most real-world scenarios, it is impossible to tell what will happen next
accurately. It is this reason, whenever we talk about future events, we talk
about the likelihood of something happening. A simple example is the
weather forecast where the weatherperson suggests or expects something
might happen, but many times it doesn’t happen the same way.
There are events where probability doesn’t apply; for example, universal
truths like the sun will always rise in the morning (unless the world is going
to end). Probability only makes sense for events (or experiments) where more
than one outcome can occur. The probability of such an event can be
expressed mathematically as:
Probability = number of ways an outcome can happen / total number of
possible outcomes
Probability lies between zero and one; zero meaning there’s zero chance and
one meaning something is imminent. It also means probability is never
negative. The events or experiments should be repeatable infinite amount of
times without any external interference.
Another important thing to remember is that finding probability in each event
may or may not affect the probability found in the consecutive events. For
example, when flipping a coin, there's an equal 50% chance (0.5 probability)
the outcome will be either a head or a tail. If we get a head on the first flip,
that doesn’t mean on the next flip that we are going to get a tail. The
probability again resets to 50% for either a heads or tails outcome on each
flip (or event or experiment as its generally called).
So, is there an example where each consecutive event changes the probability
of the next event? Yes, there is. Let’s the example of a complete deck of 52
cards. There are four ace cards, 12 face cards and 36 number cards. Let’s
shuffle the cards so there’s randomness and fairness in the system. What is
the probability of drawing an ace from the deck? In the first draw, the
probability is 4/52. Assume that the card drawn was Queen of Hearts, if we
perform a second draw without putting this Queen card back in the deck, the
total possibilities reduce to 51 in the second draw. The probability of drawing
an ace is now 4/51. If no ace is drawn even after 12 draws and not putting the
drawn card back in the deck, the probability of drawing an ace on the
thirteenth draw will be 4/40. This is an example of a simple system where the
probability of an occurrence changes with each event.

Distribution

Distribution can be of many types, but for this book, whenever we talk about
distribution, we are talking about probability distribution unless explicitly
stated otherwise. Let’s consider an example.
We are going to flip a coin three times, taken as one event, and record the
outcome. We will repeat the process until we have all the possible unique
outcomes. Here’s a list of all unique outcomes. H H H
HHT
HTH
THH
HTT
THT
TTH
T T TWhat is the probability of finding exactly one head? We look at the
possible outcomes and count all the outcomes that have exactly one head. Out
of total eight possible outcomes, we have three possibilities of getting exactly
one head, which makes the probability ⅜. What’s the probability of getting
zero heads? It is ⅛. Three heads? Also, ⅛. Two heads? The same as one
head, which is ⅜.
This is a simple example with a very small set of total possible outcomes. It
is easier to understand the different probabilities as fractions. But, consider a
data set that has millions of possible outcomes, which is the case in most real-
life scenarios. The fractions become too complex to understand. To make the
probability of different possible outcomes easier to understand, the
probability distributions are created. It is a visual representation that makes
more sense than looking at fractions.
Our current events are random yet discreet because only known values can
occur. The probability distribution is referred to as discrete. If we denote the
event as X, the following chart represents the discrete probability distribution
of random variable X.

Data Analysis – Foundation of Machine Learning

Before we do machine learning, we have to perform data analysis. There are
various stages in data analysis. For machine learning, we only require data
munging, but still, it’s a good idea to learn about data analysis before learning
about machine learning.
In simple terms, if the data is analyzed to understand what happened in the
past, the process is data analysis. If that analysis is built upon to predict
what’s going to happen in the future, it’s machine learning. There are many
overlapping concepts, and you can’t separate one from the other.
Let’s setup a project “data_analysis” in PyCharm, and we will start coding
with some basic data analysis.

Python Libraries for Data Analysis

Python provides a lot of libraries to source data depending upon its origin.
The biggest power of data analysis libraries is they support reading data from
the most common data sources by default. It means we don’t need another
library to just read data. Let’s start with our first data analysis library,
NumPy, which are used to create multi-dimensional data structures called
“numpy arrays.”
NumPy
import numpy as np

aArr = np.array([40.71, 74.01])

bArr = np.array([[91, 7, -14],
[3, 5, -7]])

print(aArr.ndim) #get dimension of array

print(bArr.ndim) #note that when creating multidimensional arrays, the
number of elements in each subarray must be the same
The output of the above script is:
1
2
You can say the “ndim” gives the number of rows in a numpy array. We can
get the number of rows and columns using the shape.
print(aArr.shape) #returns a tuple of rows and columns, for one-dimensional
array, only the number of columns are returned
print(bArr.shape)
The output is:
(2,)
(2, 3)
You might say we can do all that with nested lists. Even though numpy arrays
look like nested lists, we can do much more with them.
To get the number of elements in a numpy array, we can use the following
code:
print(bArr.size) #outputs 6 as there are total 6 elements in the bArr
We can use indices to access individual elements. Just like lists, the indices of
a numpy array start from zero but with a little bit different syntax.
print(bArr[0,2]) #outputs -14
Notice the difference, if it would have been a nested list, we would have used
“bArr[0][2]” to access -14.
Unlike lists, all the elements in a numpy array must be of same data type. If
not, numpy will try to convert all elements to the same data type, which is
usually string.
cArr = np.array([123, "is a", " number", True])
print(cArr)
The output will be:
['123' 'is a' ' number' 'True']
All the non-string values are converted to string. This is dangerous because
during data munging, if we program to look for boolean True, our script will
always fail for this array. Another observation, the commas are missing!
Does this have to do with numpy changing the data types of the values? Let’s
print the other numpy arrays to check.
print(aArr)
print(bArr)
[40.71 74.01 ]
[[ 91 7 -14]
[ 3 5 -7]]
So, this is the general behavior when printing the contents of an array. A little
bit more on the conversion, our array already had two strings in it. What if we
have an array that only has numbers and a boolean value?
dArr = np.array([555, True, False])
print(dArr)
The output is:
[555 1 0]
The boolean values were converted to the most similar number value.
Numpy arrays are immutable. Once created, we cannot add new values,
remove existing ones or swap values. However, we can create new arrays
from existing ones and make necessary changes during the process.
eArr = np.append(aArr, 555.55)
print(eArr)
The output is:
[ 40.71 74.01 555.55]
One more thing, any trailing zeros from a floating number are removed.
fArr = np.array([20.5, 14.00])
print(fArr)
The output is:
[20.5 14. ]
What if we want to append data in a multidimensional array?
gArr = np.append(bArr[0], 4) #new array is one dimensional
print(gArr)
bArr = np.append(bArr, 4) #converts multidimensional array to one
dimensional array
print(bArr)
The outputs are:
[ 91 7 -14 4]
[ 91 7 -14 3 5 -7 4]
We can also remove values that also results in a new array.
gArr = np.delete(gArr, 3) #takes the index of element to be removed. 3 is
index of 4 in gArr
print(gArr)
The output is:
[ 91 7 -14]
The numpy library also facilitates in quickly creating commonly used arrays.
zArr = np.zeros((2, 3)) #by default the data type is floating
print(zArr)

zArr2 = np.zeros((2, 3), dtype=int)

print(zArr2)
The outputs are:
[[0. 0. 0.]
[0. 0. 0.]]
[[0 0 0]
[0 0 0]]
We can create a 3D array.
oArr = np.ones((2, 4, 4)) #read as multidimensional (3D) array containing
two arrays each of 4 rows and 4 columns
print(oArr)
The output is:
[[[1. 1. 1. 1.]
[1. 1. 1. 1.]
[1. 1. 1. 1.]
[1. 1. 1. 1.]]

[[1. 1. 1. 1.]
[1. 1. 1. 1.]
[1. 1. 1. 1.]
[1. 1. 1. 1.]]]
We can also create arrays with elements having a specific sequence.
seqArr = np.arange(0, 257, 32) #create sequence by telling how much space
(step) should be between each element
print(seqArr)

linArr = np.linspace(0,256,9, dtype=int) #create sequence by telling number

of elements needed. numpy sets the appropriate step
print(linArr)
The outputs are:
[ 0 32 64 96 128 160 192 224 256]
[ 0 32 64 96 128 160 192 224 256]
A few points to note about “linspace()”:
 1. The maximum extreme is included in the sequence
2. If we don’t specify the data type, the sequence will have the default
floating type
What if we want to create an array with all elements having the same value?
We have a method for that as well.
samArr = np.full((3,3), 13)
print(samArr)
The code outputs the following:
[[13 13 13]
[13 13 13]
[13 13 13]]
We can also create an identity matrix.
idArr = np.eye(3, dtype=int) #identity matrices are always square, meaning
the same number of rows and columns hence providing single value for both
print(idArr)
The output is:
[[1 0 0]
[0 1 0]
[0 0 1]]
We can also create arrays with random integers and floating numbers.
ranIntArr = np.random.randint(2, 16, size=(3, 3)) #2 and 16 are limits so
numbers will be generated randomly between this range. (3, 3) is the size of
matrix returned
print(ranIntArr)

ranFltArr = np.random.random((3, 3)) #generate a 3x3 square matrix with

random float numbers
print(ranFltArr)
The outputs are (since we are using random methods, you might see different
numbers when you execute the above code):
[[ 3 14 6]
[10 10 4]
[ 2 13 3]]
[[0.34134875 0.2531188 0.83863728]
[0.78300571 0.47987474 0.88471375]
[0.69181843 0.17512459 0.70456541]]
Let’s perform some operations with numpy arrays.
aArr = np.array([
[2, 4, 6],
[6, 4, 2]
])
bArr = np.array([
[3, 5, 7],
[7, 5, 3]
])

#addition
print(np.add(aArr, bArr)) #also aArr + bArr

#subtraction
print(np.subtract(aArr, bArr)) #also aArr - bArr
#multiplication - not matrix multiplication
print(np.multiply(aArr, bArr)) #also aArr * bArr

#division
print(np.divide(aArr, bArr)) #also aArr / bArr
The output of the above script is:
[[ 5 9 13]
[13 9 5]]
[[-1 -1 -1]
[-1 -1 -1]]
[[ 6 20 42]
[42 20 6]]
[[0.66666667 0.8 0.85714286]
[0.85714286 0.8 0.66666667]]
We can also perform dot and cross products on arrays.
cArr = np.array([
[2, 4, 6],
[6, 4, 2],
[8, 9, 1]
])
dArr = np.array([
[3, 5, 7],
[7, 5, 3],
[8, 9, 1]
])
#dot product
print(np.dot(cArr, dArr))

#cross product
print(np.cross(cArr, dArr))
The outputs are:
[[82 84 32]
[62 68 56]
[95 94 84]]
[[-2 4 -2]
[ 2 -4 2]
[ 0 0 0]]
Transposing an array is very simple.
print(dArr.T) #transpose
The output is:
[[3 7 8]
[5 5 9]
[7 3 1]]
In many situations, you will have to sum of elements present in an array.
#sum all elements in an array
print(np.sum(dArr))

#sum all elements in the rows or columns

print(np.sum(dArr, axis=0)) #0 axis means elements in rows, 1 means
columns
print(np.sum(dArr, axis=1))
The outputs are:
48
[18 19 11]
[15 15 18]
Numpy library offers many more methods that can be applied to arrays. You
are encouraged to explore all the available methods by experimenting with
random arrays, it will help when dealing with practical tasks.
Matplotlib
When the data set is too large, it’s difficult to understand what the data is
telling you. Data visualization helps in this regard. You might not be able to
detect a trend looking at thousands of numbers, but a simple line chart will
instantly tell you what’s the trend.
Let’s create some charts using matplotlib library. But, first, we need to install
this library in the virtual environment “tf” we created. In PyCharm, towards
the bottom left side of your screen, you will see the “Terminal,” switch to it.
Run the command “activate tf”. You will now see (tf) written before the
command prompt. Now run “pip install matplotlib”, and the external library
will be downloaded and installed in our virtual environment.

We can now start plotting charts. Here’s an example of a simple line chart.
import matplotlib.pyplot as plt
import numpy as np
arr1 = np.array([2, 4, 6])
arr2 = np.array([3, 5, 7])

plt.plot(arr1, arr2)
plt.show()
The above script will output a straight line chart on the screen like this.

The plot has a few buttons on the bottom left side. You can reset the view
with the “Home” button, cycle through different views using the “Arrow”
buttons, move the plot around using the “Crosshair.” The “Zoom” button will
you let you zoom in (left click and drag) or out (right click and drag) of the
plot view. The “Configuration” buttons gives you more options to change the
plot view and looks like below. The last button, “Save,” lets you save the plot
as an image.

We can customize the plots by adding a title, labels, and a legend to our plot
view. Here is the updated script.
import matplotlib.pyplot as plt
import numpy as np

arr1 = np.array([2, 4, 6])

arr2 = np.array([3, 5, 7])

arr3 = np.array([-1, -3, -5])

arr4 = np.array([-2, -4, -6])

plt.plot(arr1, arr2, label="First Line")

plt.plot(arr3, arr4, label="Second Line")
plt.xlabel("X Axis")
plt.ylabel("Y Axis")
plt.title("Line Charts\nwith arbitrary values")

plt.legend()
plt.show()
The output now is:

Next, let’s work with bar charts and histograms. Many people think both are
the same, but our example will show how they differ. Here’s an example of
bar charts.
import matplotlib.pyplot as plt
import numpy as np

arr1 = np.array([2, 4, 6])

arr2 = np.array([3, 5, 7])

arr3 = np.array([3, 5, 7])

arr4 = np.array([12, 14, 16])

plt.bar(arr1, arr2, label="Bar Chart 1")

plt.bar(arr3, arr4, label="Bar Chart 2")

plt.title("Bar Chart\nwith arbitrary values")

plt.legend()
plt.show()
The output is:
We can see that bar charts can be easily used to visually compare two
different data sets.
A histogram is more often used to visualize the distribution of a data set over
some specified period (usually time period). Here’s a code to visualize the
ages of a population sample in age groups of 10 years.
import matplotlib.pyplot as plt
import numpy as np

arr1 = np.array([16, 55, 21, 45, 85, 57, 32, 66, 94, 12, 25, 29, 30, 32, 45, 16,
12, 74, 63, 18])
rangeBin = [0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100]
plt.hist(arr1, rangeBin, histtype='bar', rwidth=0.9, label="Age Groups")

plt.title(“Population Age Distribution")

plt.legend()
plt.show()
Here is the output chart.

To change the bar colors, we add a “color” argument to the “plt.hist()”

method.
plt.hist(arr1, rangeBin, histtype='bar', rwidth=0.9, label="Age Groups",
color="teal")
The “rwidth” is set to 0.9 to separate the bars a little; otherwise, they would
be plotted right next to each other forming a block.
Next up is scatter plot, which is usually used to show the relation between
two variables.
import matplotlib.pyplot as plt
import numpy as np

arr1 = np.array([2, 4, 6, 7, 3, 7, 5])

arr2 = np.array([3, 5, 7, 2, 4, 7, 9])

plt.scatter(arr1, arr2, label="Scatter Data", color="teal", marker="*", s=1000)

plt.title("Scatter Plot")

plt.legend()
plt.show()
The output of the above script will be:
In a scatter plot, we can choose the marker type and size. Here, we have used
a star, for a full list of available marker types, visit
https://matplotlib.org/3.1.1/api/markers_api.html.
Another chart you can create with the Python matplotlib library is the stack
plot which is used to show the contribution of several assets in achieving a
certain point. It’s much easier to understand if I show you an example.
import matplotlib.pyplot as plt
import numpy as np

daysArr = np.array([1, 2, 3, 4, 5, 6, 7])

sleepArr = np.array([7, 8, 7.5, 8.25, 4, 9, 8])

eatArr = np.array([2, 1.5, 1.25, 0.5, 3, 4, 0.75])
workArr = np.array([8, 9, 7.5, 8.25, 7, 0.25, 0.5])
chillArr = np.array([7, 5.5, 7.75, 7, 10, 10.75, 14.75])

plt.stackplot(daysArr, sleepArr, eatArr, workArr, chillArr, labels=["Sleep",

"Eat", "Work", "Chill"], colors=['m', 'c', 'r', 'k'])

plt.title("Stack Plot")

plt.legend()
plt.show()
The output is:
We have used color codes for the stack plot, ‘m’ for magenta, ‘c’ for cyan, ‘r’
for red, and ‘k’ for black.
Let’s move onto pie charts. Here’s our activities example using a pie chart.
import matplotlib.pyplot as plt
import numpy as np

daysArr = np.array([1, 2, 3, 4, 5, 6, 7])

sleepArr = np.array([7, 8, 7.5, 8.25, 4, 9, 8])

eatArr = np.array([2, 1.5, 1.25, 0.5, 3, 4, 0.75])
workArr = np.array([8, 9, 7.5, 8.25, 7, 0.25, 0.5])
chillArr = np.array([7, 5.5, 7.75, 7, 10, 10.75, 14.75])

slices = np.array([8, 0.75, 0.5, 14.75]) #taking last values of all activity arrays
to create an array detailing time spent on Sunday in different activities

plt.pie(slices, labels=["Sleep", "Eat", "Work", "Chill"], colors=['lightblue', 'r',

'm', 'teal'], explode=(0, 0.15, 0, 0), autopct="%1.1f%%")

plt.title("Pie Chart\nSunday activities")

plt.show()
The output is:
Pretty chill Sunday! Pie charts are a great way of showing how different
things add up to complete the whole picture. We have used “explode”
argument to make “Eating” stand out from the rest. The “autopct” argument
can take special formatting to show the percentage each activity takes up the
hours of my typical Sunday.
Data Acquisition
Up till now, we have declared random arrays to showcase different features
of numpy and matplotlib libraries. In real-life tasks, you will need to access
data from a certain source. The data set might be in a comma separated
values (.csv) file which is sometimes also saved as a simple text (.txt) file, or
maybe in another format like an excel (.xslx) file or maybe in XML or JSON
format. Also, the source file might be located on your computer or on the
internet.
How to acquire data in such cases? Let’s look at a few examples.
Get data from local text file
Navigate to your PyCharm project directory (you can see the link of project
folder besides the project name in Project pane of PyCharm) and create a file
“example.txt”. Add the following sample data to it, save and close the file.
1,45
2,23
3,12
4,54
5,74
6,21
7,5
8,68
9,24
10,8
There are many libraries that can be used to import data from a text/csv file.
We are going to show two approaches, using the standard csv library and
then the numpy library.
Here’s the script using standard csv library to read data and plot the data
using matplotlib.
import csv
import matplotlib.pyplot as plt

x = []
y = []

file = open('example.txt', 'r')

file_data = csv.reader(file, delimiter=",")

for row in file_data:

x.append(int(row[0]))
y.append(int(row[1]))

plt.plot(x, y)

plt.show()
file.close()
The output is:

Let’s use numpy to get the same result.

import numpy as np
import matplotlib.pyplot as plt

x, y = np.loadtxt('example.txt', delimiter=',', unpack=True)

plt.plot(x, y)

plt.show()
The output is the same. Yes, it’s that simple with numpy! On top of that, it’s
faster. The “unpack” argument tells numpy to map the read data to the given
variables.
Get data from internet
Most data accumulators provide APIs to enable programmers to
communicate with their data. We are going to connect with Quandl using
their API to download stock prices of certain companies and see the price
trend.
Our code will be divided into three parts (each part will be a separate
function).

1. The first part will acquire data through the API.

2. The second part will preprocess data so it’s ready for plotting.
3. The third and final part will plot the data as a line chart.
Here is the complete code.
import matplotlib.pyplot as plt
import urllib.request
from dateutil.parser import parse
api_key = "xxxxxxxxxxxxxxxxxx" #replace xxxxxxxxxxxxxxxxxx with your
API key

def get_data(comp):
stockPriceURL = "https://www.quandl.com/api/v3/data
sets/WIKI/"+comp+".csv?api_key="+api_key
getsource = urllib.request.urlopen(stockPriceURL).read().decode()
return getsource

def filter_data(rawData):
stockDate = []
stockClose = []
dataSplit = rawData.split('\n')
for row in dataSplit[1:]:
if row != "":
elems = row.split(',')
stockDate.append(parse(elems[0]))
stockClose.append(float(elems[4]))
return stockDate, stockClose

def plot_data(final_data):
plt.plot(final_data[0], final_data[1])

plt.xlabel("Year")
plt.ylabel("Closing Stock Price")
 plt.title("Closing Stock Price Trend for TESLA (TSLA)\nfor the
last ten yrs")

plt.show()

def main():
#company = input("Enter the company symbol you want to see
price trend of: ") #enable this if user input is required
company = "TSLA"
rawData = get_data(company)
final_data = filter_data(rawData)
plot_data(final_data)

if __name__ == "__main__":
main()
The output chart will be like this.
Let’s talk a bit about our code.
To use this code, we will have to first sign up with Quandl to use their API.
The sign up is completely free, so complete it by going to this link:
https://www.quandl.com/sign-up-modal?defaultModal=showSignUp
and activate the account. You will be assigned an API key that you can place
in the above code to use Quandl API services.
The data provided by Quandl has a header row. We have ignored the header
row by slicing the list containing the parsed data (dataSplit[1:]). In the above
script, I have focused on gathering dates (elems[0]) and the respective closing
price (elems[4]) to plot the chart. The data from the .csv file is read as strings.
We have to convert it to the correct format (date and floating) before trying to
plot them using matplotlib. The date conversion is done through parse of the
standard dateutil.parser library. The conversion of closing prices from string
to floating is straightforward.
Another thing to note in the script is how we have returned two lists
(stockDate and stockClose) in the function filter_data() and is gathered by the
single list final_data, which results in concatenation of returned lists.
This is not the only way to acquire data from Quandl. We could have used
numpy arrays instead of lists in our code that would have made things a little
faster. Quandl also offers a special library for Python “quandl” that you can
install using pip and use to access data from Quandl.
Let’s setup quandl library on Python and see how we can perform the actions
of the last script using the new library. Switch to “terminal” in PyCharm.
Activate the “tf” virtual environment if not already active, and then use the
following command to install quandl.
pip install quandl
Here is the code to display closing price trend for Apple (AAPL).
import numpy as np
import matplotlib.pyplot as plt
import quandl

api_key = "xxxxxxxxxxxxxxxxxx" #replace xxxxxxxxxxxxxxxxxx with your

API key

def get_data(comp):
getsource = quandl.get("EOD/"+comp+".4", returns="numpy",
authtoken=api_key, start_date="2009-01-01", end_date="2019-01-01")
return getsource

def filter_data(rawData):
dates = np.array([i[0] for i in rawData])
closep = np.array([i[1] for i in rawData])
 return dates, closep

def plot_data(final_data):
#plt.plot(final_data[0], final_data[1], label="Price")
plt.plot_date(final_data[0], final_data[1], '-', label="Price",
color=”gold”)

plt.xlabel("Half-Year")
plt.ylabel("Closing Stock Price")
plt.xticks(rotation=20)
plt.title("Closing Stock Price Trend for APPLE (AAPL)\nusing all
available data")
plt.subplots_adjust(bottom=0.15)

plt.legend()
plt.show()

def main():
#company = input("Enter the company symbol you want to see
price trend of: ") #enable this if user input is required
company = "AAPL"
rawData = get_data(company)
final_data = filter_data(rawData)
plot_data(final_data)
if __name__ == "__main__":
main()
The above will output the following chart.

We can see from the trend that in mid-2014, the stock prices crashed for
Apple. But, this wasn’t because the company went out of business, Apple
introduced split shares which allowed cheaper shares opening doors to micro
investors. Other than that, Apple’s stock price has a very distinctly repetitive
trend. Any experienced stock trader can look at this trend and find
opportunities of investing at the right time and making profits.
Let’s talk about our script. TSLA stocks are not available on the free Quandl
account so we will work with the available data of AAPL stocks. Quandly
offers a lot of stock data, but we are only interested in the End of Day closing
stock price, which is the fifth column of the Quandl data. In the get_data()
function, we import the stock data as a numpy recarray. Numpy recarrays are
different than normal numpy arrays because they can contain different data
types wrapped in a tuple. The start and end dates are used to set a range for
data, but in our example, both are useless because we have very limited set of
data available on our free Quandl account.
We actually don’t need to filter_data() function because the data is ready for
plotting. For the sake of showing a new technique, we have used this function
to create standard numpy arrays from the recarray Quandl generated. The
function returns both arrays and collected as a tuple by the main() function.
Note that this is not a good practice as we are dealing with a huge data set.
Can you think of a better way to pass data from filter_data() to main()?
In the plot_data() function, we have shown two different plotting methods,
one we already know plt.plot() and the other plt.plot_date(). The plot_date()
is very useful in plotting two quantities in which one is time. We have also
used some advanced plot formatting to make the plot look nicer. The labels
on the ‘x’ axis are rotated to avoid overlap and the plot margin from the
bottom is increased so nothing overflows out of the visible window. One
more thing, plot_date() usually plots a scatter plot instead of a line chart, we
have added the ‘-’ argument to force plot_date() in creating a line chart.
You might see the following warning because we didn’t explicitly set the
data format conversion. But, we can ignore this warning because our script is
correctly plotting the data.
To register the converters:
>>> from pandas.plotting import
register_matplotlib_converters
>>> register_matplotlib_converters()
warnings.warn(msg, FutureWarning)
The quandl library is a very powerful tool if you want to write scripts on
stocks and other finance applications. The free account has some API call
limitations, but you can download the entire data set of any symbol (company
traded in a stock market) using the API URL method to your computer and
then keep practicing different analysis and machine-learning techniques with
the offline data set.
A project
We have learned enough to consider a real-world scenario and attempt to
solve the issue(s) using the tools we have learned so far. Here’s a scenario.
We have been contracted by a potato chips manufacturing facility that makes
200g packets. We have to find if a randomly picked chips packet from the
assembly line has a net weight within the accepted tolerance range of ±10g.
We ask the facility to sample a random number of samples and record the
mean weight and standard deviation. The values come up as 197.9g and 5.5g
respectively.
We start by assuming the system probability has a normal distribution. We
have a complex and randomly sampled system so it’s impossible to ascertain
the probability of a specific event through the normal probability
determination methods. To find the probability of such as a complex system,
we create the probability distribution curve and find area under the curve.
In our code, we have used a library we haven't used so far, the “scipy”
library. The erf() from math library is used to estimate the error function of a
normal distribution system. We are also using the subplot().
# Import required libraries
from math import erf, sqrt
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm

# Declare given data

sysAvg = 197.9
sysDevStd = 5.5
sysLowTol = 190
sysHighTol = 210

''' Calculating probability '''

# Probability from Z=0 to lower bound
lowTolProb = erf((sysLowTol - sysAvg) / (sysDevStd * sqrt(2))) / 2

# Probability from Z=0 to upper bound

highTolProb = erf((sysHighTol - sysAvg) / (sysDevStd * sqrt(2))) / 2

# Calculate Interval Probabilities

pIn = highTolProb - lowTolProb
pOut = 1 - pIn

# Print data and results

print('===Given Data===')
print('Mean = %.1fg \nStandard Deviation: %.1fg' % (sysAvg, sysDevStd))
print('Tolerance range = %.1fg <--> %.1fg \n' % (sysLowTol, sysHighTol))
print('===Calculation Results===')
print('Lower Bound = %.4f' % lowTolProb)
print( 'Upper Bound = %.4f' % highTolProb)
print('Probability of finding a chips packet with weight within the tolerance
range (pIn) = %.1f%%' % (pIn*100)) # using %% to print a percent symbol
print('Probability of finding a chips packet with weight outside the tolerance
range (pOut) = %.1f%% \n' % (pOut*100)) # using %% to print a percent
symbol

''' Plotting distribution curve '''

# Calculate the Z-scores
z1 = (sysLowTol - sysAvg) / sysDevStd
z2 = (sysHighTol - sysAvg) / sysDevStd

# Calculate the plot values

x = np.arange(z1, z2, 0.001) # range of x-axis within tolerance range
x_all = np.arange(-4, 4, 0.001) # entire range of x-axis, both in and out of
tolerance range
# mean = 0, sysDevStd = 1, since Z-scores were calculated, norm.pdf
calculates the probability density function
y = norm.pdf(x, 0, 1)
y_all = norm.pdf(x_all, 0, 1)

# Build the plot

fig, ax = plt.subplots(figsize=(12, 9)) # set plot height and width
plt.style.use('fivethirtyeight') # set color scheme
ax.plot(x_all, y_all, label="Probability Range") # setting label here is
necessary, otherwise plt.legend() would throw an error

# Add custom styling to the plot

ax.fill_between(x, y, 0, alpha=0.3, color='yellow') # alpha argument is used
to create a shade of yellow color
ax.fill_between(x_all, y_all, 0, alpha=0.1, color='red') # alpha argument is
used to create a shade of red color
ax.set_xlabel('Standardized Mean')
ax.set_ylabel('Probability Density')
ax.set_title('System Distribution Curve')
# Show the plot
plt.legend()
plt.savefig('sysProbCurve.png') # Save the plot in current working directory,
this must be done before plot.show() or you might end up with a blank image
file
plt.show()
We are also saving the plot as a “png” image in the current working file. As
stated in the comment, we have to save the file before showing it on the
screen or we might end up with a blank image. In our plot, we have used
shading to highlight areas of interest. Yellow color shows the area of
probability to find a chips packet with weight within the tolerance range,
which is more than the area with red color, which shows the probability of
picking a packet outside the range.
The output of the above code is below.
===Given Data===
Mean = 197.9g
Standard Deviation: 5.5g
Tolerance range = 190.0g <--> 210.0g

===Calculation Results===
Lower Bound = -0.4246
Upper Bound = 0.4861
Probability of finding a chips packet with weight within the tolerance range
(pIn) = 91.1%
Probability of finding a chips packet with weight outside the tolerance range
(pOut) = 8.9%
There is a 91.1% chance of picking a chips packet from the production line
and its weight being within the tolerance range. Is it a high enough
percentage? It depends upon the production facility and the standards it has to
keep up with. Note that there are industrial standards and government
inspections to make sure the final product follows the claims made. The
weight of the packet is one such important parameter.

The Pandas Framework

“Pandas” framework provides even more flexibility and options when dealing
with large data sets. It is the most versatile data structure available, and you
can handle almost all data-related work with pandas. It is also better than
other data processing options; for example, if you are still thinking of using a
spreadsheet software to perform data analysis, you are missing on a much
faster option. One more fact, using pandas and numpy using Python to
perform data analysis and machine learning is as fast as using another
language like C++ because these libraries are based on the C language with a
Python wrapper. So, we have faster processing of C family languages and the
easy syntax of Python greatly enhancing coding experience and results.
The pandas framework also makes it easier to read data from any data source.
No matter the source, the data read has always the same format which makes
it easier for the scriptwriter to handle it.
To facilitate dealing with massive data sets, the pandas framework offers two
data structures, series and data frames. Let’s discuss the basics of both.
Series
What happens when the best features of dictionaries and numpy arrays are
combined together? The pandas framework “series” data structure is the
result. Let’s do some basic work with pandas series.
from pandas import Series
import pandas as pd

aSer = Series([2, 4, 6])

print(aSer)
print(aSer.values) # get all values as a list
print(aSer.index) # get all indices as a list

keys = ['frst', 'scnd', 'thrd']

bSer = Series([3, 5, 7], index=keys)

print(bSer)
print(bSer['frst']) # get element using key
print(bSer[0]) # get element using index

aDict = {
 'a': 1,
'b': 2,
'c': 0
}
cSer = Series(aDict) # creating a series from a dictionary

print(cSer)

ind = ['a', 'b', 'd']

dSer = Series(aDict, index=ind) # if key isn't in the dictionary, "NaN" is
added as value to that key

print(dSer)

print(pd.isnull(dSer['d'])) # check if an element has “NaN” value

dSer.name = 'Data'
dSer.index.name = 'Index'

print(dSer)

dSer.index = ['un', 'deux', 'nul'] # keys are mutable unlike standard

dictionaries
print(dSer)
The outputs of the above “print” statements are:
0 2
1 4
2 6
dtype: int64
[2 4 6]
RangeIndex(start=0, stop=3, step=1)
frst 3
scnd 5
thrd 7
dtype: int64
3
3
a 1
b 2
c 0
dtype: int64
a 1.0
b 2.0
d NaN
dtype: float64
True
Index
a 1.0
b 2.0
d NaN
Name: Data, dtype: float64
un 1.0
deux 2.0
nul NaN
Name: Data, dtype: float64
Time Series
Any quantity that can be measured for a period of time forms a time series.
This series is very important in different fields including finance, economics,
ecology, neuroscience, and physics. The quantity might be sampled after
fixed intervals (fixed frequency) or sampled at random times (irregular). We
are going to take a look at the following time series.
1. Timestamps – quantity sampled at specific time instants.
2. Fixed periods – quantity sampled after a fixed time period, like
monthly, annually, etc.
3. Intervals – quantity sampled for a range of time.
Timestamps are the most common and simplest time series.
The pandas framework provides access to various tools and algorithms to
work with large time series. You can easily resample, aggregate, and segment
a time series. The available resources are very useful in various applications
such as analyzing log files.
from datetime import datetime
from datetime import timedelta
from dateutil.parser import parse

now = datetime.now()
print(now) # the returned data has year, month, day, hour, minutes, seconds,
milliseconds, and microseconds

nexTime = now + timedelta(12) # 12 means the nexTime will be 12 days in

advance from now

print(nexTime)

print(str(nexTime)) # string looks the same as datatime format

print(nexTime.strftime('%Y-%m-%d')) # reformat datetime value

ranTime = '1990-03-07'

print(datetime.strptime(ranTime, '%Y-%m-%d')) # format string as datetime

print(parse(ranTime)) # format string that has datetime format as datetime

without specifying a format

print(parse(ranTime, dayfirst=True)) # put the day before month to follow

international locales
The respective outputs of the above print statements are:
2019-10-30 08:10:25.841218
2019-11-11 08:10:25.841218
2019-11-11 08:10:25.841218
2019-11-11
1990-03-07 00:00:00
1990-03-07 00:00:00
1990-07-03 00:00:00
In a time series, we deal with a set of time values usually used as a column or
axis index in a dataframe. We can create indexes using a list containing
strings that have date related data.
stringDates = ['1958-01-31', '1969-07-20']

print(pd.to_datetime(stringDates)) # converts an array of strings that look

like used as index for a dataframe

print(pd.to_datetime(stringDates + [None])) # we can use "none" as an index

which translates to NaT (Not a Time)

The outputs are:

DatetimeIndex(['1958-01-31', '1969-07-20'], dtype='datetime64[ns]',
freq=None)
DatetimeIndex(['1958-01-31', '1969-07-20', 'NaT'], dtype='datetime64[ns]',
freq=None)
Once we have the time indices, we can create a pandas series or dataframe
resulting in a time series. Here’s an example of a time series using series.
stringDates = ['1958-01-31', '1969-07-20']

print(pd.to_datetime(stringDates)) # converts an array of strings that look

like used as index for a dataframe
dateIndices = pd.to_datetime(stringDates + [None])

print(dateIndices) # we can use "none" as an index which translates to NaT

(Not a Time)

tSer = Series(np.random.randn(2), index=pd.to_datetime(stringDates))

tSer2 = Series(np.random.randn(3), index=dateIndices)

print(tSer)
print(tSer2)
print(tSer + tSer2) # performing arithmetic operations on time series is easy
print(tSer * tSer2) # notice how NaT becomes the first element
The outputs are:
DatetimeIndex(['1958-01-31', '1969-07-20'], dtype='datetime64[ns]',
freq=None)
DatetimeIndex(['1958-01-31', '1969-07-20', 'NaT'], dtype='datetime64[ns]',
freq=None)
1958-01-31 2.520516
1969-07-20 0.305652
dtype: float64
1958-01-31 -0.720893
1969-07-20 1.213476
NaT -0.229652
dtype: float64
NaT NaN
1958-01-31 1.799623
1969-07-20 1.519128
dtype: float64
NaT NaN
1958-01-31 -1.817023
1969-07-20 0.370901
dtype: float64
We can also perform index slicing on a time series. Time series can also
contain duplicate indices. Here’s a script that showcases bot concepts.
longtSer = Series(np.random.randn(500), index=pd.date_range('7/3/2016',
periods=500)) # create a long time series and populate with random numbers

print(longtSer.head()) # get first five rows

print(longtSer[:5]) # use index slicing to get the same result: first five rows
print(longtSer['2016-07']) # we can slice the data using the month or year or
even the day
print(longtSer.tail()) # get last five rows
print(longtSer[-5:]) # use negative index slicing to get the same result: last
five rows

dates = pd.DatetimeIndex(['1/1/2019', '1/1/2019', '1/2/2019', '1/2/2019',

'1/3/2019'])

duplSer = Series(np.arange(5), index=dates) # time series can have duplicate

indices
print(duplSer)
print(duplSer.index.is_unique) # check if there are any duplicate time indices

grupdSer = duplSer.groupby(level=0) # create a group object where values

with same indices are grouped together
print(grupdSer.mean()) # get mean of grouped values for each duplicate time
index
print(grupdSer.count()) # get count of grouped values for each duplicate time
index
The outputs are:
2016-07-03 2.008307
2016-07-04 0.953226
2016-07-05 -0.355272
2016-07-06 -1.002089
2016-07-07 -1.609062
Freq: D, dtype: float64
2016-07-03 2.008307
2016-07-04 0.953226
2016-07-05 -0.355272
2016-07-06 -1.002089
2016-07-07 -1.609062
Freq: D, dtype: float64
2016-07-03 2.008307
2016-07-04 0.953226
2016-07-05 -0.355272
2016-07-06 -1.002089
2016-07-07 -1.609062
2016-07-08 -0.383618
2016-07-09 -0.346427
2016-07-10 0.449504
2016-07-11 0.099086
2016-07-12 0.311142
2016-07-13 -1.197215
2016-07-14 -1.014129
2016-07-15 -0.260970
2016-07-16 1.270730
2016-07-17 1.118838
2016-07-18 -0.461846
2016-07-19 -0.152710
2016-07-20 1.274209
2016-07-21 -0.175187
2016-07-22 -0.724205
2016-07-23 0.278828
2016-07-24 0.413432
2016-07-25 1.234999
2016-07-26 -0.425415
2016-07-27 -1.788413
2016-07-28 3.147094
2016-07-29 0.216745
2016-07-30 -0.674562
2016-07-31 -1.958297
Freq: D, dtype: float64
2017-11-10 0.319045
2017-11-11 0.614589
2017-11-12 0.479498
2017-11-13 -0.287627
2017-11-14 -1.362115
Freq: D, dtype: float64
2017-11-10 0.319045
2017-11-11 0.614589
2017-11-12 0.479498
2017-11-13 -0.287627
2017-11-14 -1.362115
Freq: D, dtype: float64
2019-01-01 0
2019-01-01 1
2019-01-02 2
2019-01-02 3
2019-01-03 4
dtype: int32
False
2019-01-01 0.5
2019-01-02 2.5
2019-01-03 4.0
dtype: float64
2019-01-01 2
2019-01-02 2
2019-01-03 1
dtype: int64
Dataframes
The second data structure offered by pandas is the dataframes. It’s very fast
and can handle gigabytes of data, which makes dataframes the best choice to
use in machine learning.
Let’s see a few basic dataframe operations.
import pandas as pd

dayLst = ['Mon', 'Tue', 'Wed', 'Thur', 'Fri', 'Sat', 'Sun']

siteStats = {
"Day Number": [1, 2, 3, 4, 5, 6, 7],
"Visitors": [1405, 24517, 32415, 74512, 9541, 32145, 33],
"Bounce Rate": [65, 42, 54, 74, 82, 10, 35]
}

dFrame = pd.DataFrame(siteStats, index=dayLst) # using a list to assign

custom indices to the dataframe
dFrame2 = pd.DataFrame(siteStats)
dFrame2.set_index('Day Number', inplace=True) # assigning existing
dataframe column as the index

print(dFrame)
print(dFrame2)
The output is:
Day Number Visitors Bounce Rate
Mon 1 1405 65
Tue 2 24517 42
Wed 3 32415 54
Thur 4 74512 74
Fri 5 9541 82
Sat 6 32145 10
Sun 7 33 35
Visitors Bounce Rate
Day Number
1 1405 65
2 24517 42
3 32415 54
4 74512 74
5 9541 82
6 32145 10
7 33 35
One important thing to remember with pandas dataframe is that when you
apply any methods such as set_index(), you get a new dataframe and the
original dataframe remains intact. This is a safety feature so you don’t
accidentally overwrite the source data. If you want to overwrite the original
data set, you have to use the “inplace=True” argument.
We can access a specific column or columns of a dataframe. We can also
convert those columns to a numpy array, which leads to a question: can we
convert a numpy array to a dataframe? Yes, we can. Here’s a script that does
all that.
print(dFrame['Bounce_Rate']) # returns a single column of dataframe as a
series with the same index as the dataframe
print(dFrame.Bounce_Rate) # only works if column headers don't have
spaces, returns the same output as above print

print(dFrame[['Visitors', 'Bounce_Rate']]) # returns multiple columns of

dataframe as new dataframe with same index as the original

print(np.array(dFrame[['Visitors', 'Bounce_Rate']])) # returns multiple

columns of dataframe as multidim numpy array

dFrame3 = np.array(dFrame[['Visitors', 'Bounce_Rate']]) # convert a numpy

array to a pandas dataframe

print(dFrame3)
The output of the above script is:
Mon 65
Tue 42
Wed 54
Thur 74
Fri 82
Sat 10
Sun 35
Name: Bounce_Rate, dtype: int64
Mon 65
Tue 42
Wed 54
Thur 74
Fri 82
Sat 10
Sun 35
Name: Bounce_Rate, dtype: int64
Visitors Bounce_Rate
Mon 1405 65
Tue 24517 42
Wed 32415 54
Thur 74512 74
Fri 9541 82
Sat 32145 10
Sun 33 35
[[ 1405 65]
[24517 42]
[32415 54]
[74512 74]
[ 9541 82]
[32145 10]
[ 3335]]
As the output suggests, when we access a single column of a dataframe by
using the column header and we get a list. In our script, we created a
dataframe from a dictionary. The dictionary keys were set without spaces that
makes each column a parameter of the dataframe.
Reading Data
The pandas framework also provides out of the box support to get data from
most common data set file formats from both local or external sources. We
have already learned how to connect with Quandl using it’s API URL or the
quandl Python library to access data. What if the data is already downloaded
from Quandl and currently residing on your computer? We can also read data
from there. Login to your Quandl account by going to their website. Search
for “AAPL”, which is the stock price data for Apple Inc. Quandl provides
data export feature in many file formats. Let’s download in the .csv format to
the current PyCharm project folder.
import pandas as pd

''' # getting data from a CSV file on local computer'''

dataF = pd.read_csv('EOD-AAPL.csv')

dataF.set_index('Date', inplace=True) # set date as the index making this a

time series dataframe
dataF.to_csv("new.csv") # save dataframe data in a new csv file

print(dataF.head())

dataF = pd.read_csv('new.csv', index_col=0) # if we want to read csv data

and set the index at the same time, this is how it's done

print(dataF.head())

dataF.rename(columns={'Open': 'Open_Price'}, inplace=True) # renamed a

column, we can rename multiple columns by passing more key-values in the
columns dictionary

print(dataF.head())

dataF.to_csv("new2.csv", header=False) # save dataframe data in a new csv

file without the headers

dataF = pd.read_csv('new2.csv', names=['Date', 'Open', 'High', 'Low', 'Close',

'Volume', 'Dividend', 'Split', 'Adj_Open', 'Adj_High', 'Adj_Low', 'Adj_Close',
'Adj_Volume'], index_col=0) # setting column headers if csv file doesn't
have headers or if you want to rename columns
print(dataF.head())
The output of the above script is:
Date Open High Low ... Adj_Low Adj_Close Adj_Volume

2017-12-
28 171.00 171.850 170.480 ... 165.957609 166.541693 16480187.0
2017-12-
27 170.10 170.780 169.710 ... 165.208036 166.074426 21498213.0
2017-12-
26 170.80 171.470 169.679 ... 165.177858 166.045222 33185536.0
2017-12-
22 174.68 175.424 174.500 ... 169.870969 170.367440 16349444.0
2017-12-
21 174.17 176.020 174.100 ... 169.481580 170.367440 20949896.0
[5 rows x 12 columns]
Date Open High Low ...
Adj_Low Adj_Close Adj_Volume

2017-12-
28 171.00 171.850 170.480 ... 165.957609 166.541693 16480187.0
2017-12-
27 170.10 170.780 169.710 ... 165.208036 166.074426 21498213.0
2017-12-
26 170.80 171.470 169.679 ... 165.177858 166.045222 33185536.0
2017-12-
22 174.68 175.424 174.500 ... 169.870969 170.367440 16349444.0
2017-12-
21 174.17 176.020 174.100 ... 169.481580 170.367440 20949896.0

[5 rows x 12 columns]
Date Open_Price High Low ...
Adj_Low Adj_Close Adj_Volume

2017-12-
28 171.00 171.850 170.480 ... 165.957609 166.541693 16480187.0
2017-12-
27 170.10 170.780 169.710 ... 165.208036 166.074426 21498213.0
2017-12-
26 170.80 171.470 169.679 ... 165.177858 166.045222 33185536.0
2017-12-
22 174.68 175.424 174.500 ... 169.870969 170.367440 16349444.0
2017-12-
21 174.17 176.020 174.100 ... 169.481580 170.367440 20949896.0
[5 rows x 12 columns]
Date Open High Low ...
Adj_Low Adj_Close Adj_Volume

2017-12-
28 171.00 171.850 170.480 ... 165.957609 166.541693 16480187.0
2017-12-
27 170.10 170.780 169.710 ... 165.208036 166.074426 21498213.0
2017-12-
26 170.80 171.470 169.679 ... 165.177858 166.045222 33185536.0
2017-12-
22 174.68 175.424 174.500 ... 169.870969 170.367440 16349444.0
2017-12-
21 174.17 176.020 174.100 ... 169.481580 170.367440 20949896.0

[5 rows x 12 columns]
As you can see, all the print statements have given identical outputs. That
was the goal of the above script, to show different ways to read data from
.csv files set with slightly different data form and to normalize them to one
form.
Writing Data
Using the pandas framework, we can save dataframes in file formats other
than we imported it from. For example, let’s save our dataframe as an HTML
file.
dataF.to_html('dataF.html') # save dataframe as an HTML file
The new HTML file will be saved in the directory of current project. If you
open the file using a browser, you will see the following.
If you open the same HTML file in a text editor, you will see the following.
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>Open</th>
<th>High</th>
<th>Low</th>
<th>Close</th>
<th>Volume</th>
<th>Dividend</th>
<th>Split</th>
<th>Adj_Open</th>
<th>Adj_High</th>
<th>Adj_Low</th>
<th>Adj_Close</th>
<th>Adj_Volume</th>
</tr>
...
Getting Data from Internet
Here is a code that gets stock price data of Google (GOOG) from Yahoo
Finance site using the special pandas_datareader library. We have to first
install this library using pip. Switch to Terminal in PyCharm and make sure
“tf” environment is activated; if not, activate it with following command.
activate tf
Then run the pip command.
pip install pandas_datareader
Once installed, run the following script.
import pandas_datareader.data as web
import matplotlib.pyplot as plt
from matplotlib import style

rangeStart = '1/1/2009'
rangeStop = '1/1/2019'

dFrame = web.get_data_yahoo('GOOG', rangeStart, rangeStop)

print(dFrame.head()) #output first five rows of the dataframe

style.use('ggplot')
dFrame['Adj Close'].plot()
plt.show()
The output will be:
Date High Low ... Volume Adj Close

2009-01-02 160.309128 152.179596 ... 7248000.0 160.060059

2009-01-05 165.001541 156.911850 ... 9814500.0 163.412491
2009-01-06 169.763687 162.585587 ... 12898500.0 166.406265
2009-01-07 164.837143 158.779861 ... 9022600.0 160.403763
2009-01-08 161.987823 158.077484 ... 7228300.0 161.987823

[5 rows x 6 columns]
The plotted graph will look like the following. Note how it looks different
than our previous charts because we used the preset style “ggplot”.

Machine-Learning Projects
We have learned enough tools and procedures to start dealing with real-life
scenarios. Let’s do two projects that will show us how machine learning can
be beneficial in solving problems.

Predicting If a Country’s GDP is Related to Its Better Life Index

Remember when we talked about whether happiness and money are related?
Let’s write some script to find out the relationship between the two using data
of a few countries.
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import sklearn
# Load the data
oecd_bli = pd.read_csv("oecd_bli_2015.csv", thousands=',')
gdp_per_capita =
pd.read_csv("gdp_per_capita.csv",thousands=',',delimiter='\t',
encoding='latin1', na_values="n/a")
# Prepare the data
country_stats = prepare_country_stats(oecd_bli, gdp_per_capita)
X = np.c_[country_stats["GDP per capita"]]
y = np.c_[country_stats["Life satisfaction"]]
# Visualize the data
country_stats.plot(kind='scatter', x="GDP per capita", y='Life satisfaction')
plt.show()
# Select a linear model
model = sklearn.linear_model.LinearRegression()
# Train the model
model.fit(X, y)
# Make a prediction for Cyprus
X_new = [[22587]] # Cyprus' GDP per capita
print(model.predict(X_new)) # outputs [[ 5.96242338]]

Predicting Real Estate Prices for Investment

A real estate tycoon Mr. Gill from New York wants to expand his investment
portfolio and is actively considering making substantial investment in other
states of the US. Luckily, a common acquaintance introduced you to him and
he offered you a big paycheck contract. You have only job, to find if he
should invest in real estate of a specific state right away, delay, or stop
thinking about new investment ventures altogether?
The first step is to gather real estate pricing data for the area. We are going to
use Quandl as the data source. After we have the data, we will find
correlation of the Housing Price Index (HPI) of all the states with each other
and the national HPI. We will plot all the information to visualize the HPI
trends for the last 44 years.
import pandas as pd
import quandl
import matplotlib.pyplot as plt
from matplotlib import style

# HPI = Housing Price Index

api_key = "xxxxxxxxxxxxxxxxxxxxxxxxx" # replace the xxx with your api

key here
''' # only run this once then afterwards use the pickle file so we don't have to
query quandl API on every script run

usStates =
pd.read_html('https://simple.wikipedia.org/wiki/List_of_U.S._states#List')
usStates = usStates[0]['postal abbreviation[1]']

supFrame = pd.DataFrame()

for x in usStates['postal abbreviation[1]'][:]:

qry = 'FMAC/HPI_'+str(x)+'.1'
newdFrame = quandl.get(qry, authtoken=api_key)
newdFrame.rename(columns={'NSA Value': str(x)}, inplace=True)
# newdFrame[str(x)] = ( (newdFrame[str(x)] - newdFrame[str(x)]
[0]) / newdFrame[str(x)][0] ) * 100 # get percent change relative to starting
value, enable this line if this percent values are to be saved
if supFrame.empty:
supFrame = newdFrame
else:
supFrame = supFrame.join(newdFrame)

supFrame.to_pickle('supFrame.pickle')
#supFrame.to_pickle('supFrame2.pickle') enable this line if this percent
values are to be saved
'''
iniPercSupFrame = pd.read_pickle('supFrame2.pickle') # this is the
dataframe with percent change relative to starting value
'''relpercSupFrame = supFrame.pct_change() # this calculates the percent
change relative to the previous value
supFrame = pd.read_pickle('supFrame.pickle')
print(supFrame.head())
print(supFrame.tail())
print(relpercSupFrame.head())
print(relpercSupFrame.tail())
print(iniPercSupFrame.head())
print(iniPercSupFrame.tail())'''

'''supFrame.plot()
#relpercSupFrame.plot()
#iniPercSupFrame.plot()
style.use('fivethirtyeight')
plt.legend().remove()
plt.show()'''

''' # only run this once then afterwards use the pickle file so we don't have to
query quandl API on every script run
usFrame = quandl.get('FMAC/HPI_USA.1', authtoken=api_key)
usFrame['NSA Value'] = ( (usFrame['NSA Value'] - usFrame['NSA Value']
[0]) / usFrame['NSA Value'][0] ) * 100 # get percent change relative to
starting value
usFrame.to_pickle('usFrame.pickle')'''
usFrame = pd.read_pickle('usFrame.pickle') # this has housing price index
for the entire of USA

fig = plt.figure(figsize=(12, 10))

ax1 = plt.subplot2grid((1, 1), (0, 0))
style.use('fivethirtyeight')
#iniPercSupFrame.plot(ax=ax1)
#usFrame.plot(ax=ax1, color='k', linewidth=10)
#plt.legend().remove()
#plt.show() # the graph shows that there is definitely correlation between
HPI of different states and also of states and the national HPI

stateHPIcorr = iniPercSupFrame.corr() # find correlation of each state with

the other states
print(stateHPIcorr.head())
print(stateHPIcorr.describe()) # get a summary of the dataframe

#stateYR = iniPercSupFrame['TX'].resample('A').mean()
stateYR = iniPercSupFrame['TX'].resample('A').ohlc() # get open, high, low
and close along with the monthly HPI for the state of Texas
print(stateYR.head())
iniPercSupFrame['TX'].plot(ax=ax1, label='Monthly TX HPI')
stateYR.plot(ax=ax1, label='Yearly TX HPI')
plt.legend(loc=4) # to show legend at bottom right hand side of the plot
plt.show() # show relation between monthly HPI and annual mean HPI for
the state of Texas
Note that I have commented out some parts of the code, uncomment them
when you run the script for the first time. The outputs are below.
AL AK AZ ... WV WI WY
AL 1.000000 0.956590 0.946930 ... 0.986657 0.992753 0.957353
AK 0.956590 1.000000 0.926650 ... 0.977033 0.943584 0.989444
AZ 0.946930 0.926650 1.000000 ... 0.933585 0.945189 0.927794
AR 0.995986 0.974277 0.946899 ... 0.993022 0.988525 0.971418
CA 0.949688 0.935188 0.982345 ... 0.946577 0.951628 0.935299

[5 rows x 50 columns]
AL AK AZ ... WV WI WY
count 50.000000 50.000000 50.000000 ... 50.000000 50.000000 50.000000
mean 0.971637 0.947688 0.943983 ... 0.967408 0.967758 0.948303
std 0.022655 0.034883 0.022095 ... 0.027050 0.022545 0.036489
min 0.895294 0.814556 0.884497 ... 0.862753 0.901193 0.813296
25% 0.958299 0.937193 0.934415 ... 0.956899 0.954704 0.935313
50% 0.976359 0.955945 0.945212 ... 0.973251 0.971728 0.958090
75% 0.987204 0.965108 0.953891 ... 0.985205 0.985719 0.968983
max 1.000000 1.000000 1.000000 ... 1.000000 1.000000 1.000000

[8 rows x 50 columns]
Date open high low close

1975-12-31 0.000000 5.727672 0.000000 5.637840

1976-12-31 5.956903 13.843932 5.956903 13.843932
1977-12-31 14.362016 28.585277 14.362016 28.585277
1978-12-31 29.814172 48.705961 29.814172 48.705961
1979-12-31 50.274852 74.032622 50.274852 74.032622

The next phase is to introduce some more data attributes and then perform
machine learning to make some predictions.
''' # only run this once, then afterwards use the pickle file so we don't have to
query quandl API on every script run
mortggFrame = quandl.get('FMAC/MORTG', trim_start="1975-01-01",
authtoken=api_key)
mortggFrame['Value'] = ((mortggFrame['Value'] - mortggFrame['Value'][0]) /
mortggFrame['Value'][0]) * 100 # get percent change relative to starting
value
print(mortggFrame.head())
mortggFrame = mortggFrame.resample('D').mean()
mortggFrame = mortggFrame.resample('M').mean() # these two resampling
operations is a hack to shift data column from start of month to end of month
print(mortggFrame.head())

mortggFrame.to_pickle('mortggFrame.pickle')'''

mortggFrame = pd.read_pickle('mortggFrame.pickle')
mortggFrame.columns = ['Mort30yr']
print(mortggFrame.head())

''' only run this once, then afterwards use the pickle file so we don't have to
query quandl API on every script run
unEmp = quandl.get("USMISERY/INDEX.1", start_date="1975-01-31",
authtoken=api_key)
unEmp["Unemployment Rate"] = ((unEmp["Unemployment Rate"] -
unEmp["Unemployment Rate"][0]) / unEmp["Unemployment Rate"][0]) *
100.0 # get percent change relative to starting value

unEmp.to_pickle('unEmp.pickle')
'''
unEmp = pd.read_pickle('unEmp.pickle')
print(unEmp.head())
HPIMega = iniPercSupFrame.join([mortggFrame, unEmp])

HPIMega.dropna(inplace=True) # remove all rows that have even one NaN

print(HPIMega)

HPIMega.to_pickle('HPIMega.pickle')
HPIMega = pd.read_pickle('HPIMega.pickle')
The “HPIMega.pickle” is our final prepared data that we will use to perform
machine learning. The outputs generated by this section of code are:
Date Mort30yr

1975-01-31 0.000000
1975-02-28 -3.393425
1975-03-31 -5.620361
1975-04-30 -6.468717
1975-05-31 -5.514316
Date Unemployment Rate

1975-01-31 0.000000
1975-02-28 0.000000
1975-03-31 6.172840
1975-04-30 8.641975
1975-05-31 11.111111
Date AL AK ... Mort30yr Unemployment Rate
1975-01-31 0.000000 0.000000 ... 0.000000 0.000000
1975-02-28 0.572095 1.461717 ... -3.393425 0.000000
1975-03-31 1.238196 2.963317 ... -5.620361 6.172840
1975-04-30 2.043555 4.534809 ... -6.468717 8.641975
1975-05-31 2.754944 6.258447 ... -5.514316 11.111111
... ... ... ... ... ...
2016-05-31 266.360987 411.870585 ... -61.823966 -41.975309
2016-06-30 269.273949 412.844893 ... -62.142100 -39.506173
2016-07-31 271.680625 411.627829 ... -63.520679 -39.506173
2016-08-31 272.869925 409.194858 ... -63.520679 -39.506173
2016-09-30 272.330718 406.662586 ... -63.308590 -38.271605

[501 rows x 52 columns]

Let’s start with some machine learning. You can backup all the code we’ve
used up until now in another file or start with a new file from here on because
we have gathered the prepared data in a pickle. The new file can import the
pickle with these two lines of code.
import pandas as pd

HPIMega = pd.read_pickle('HPIMega.pickle')
We have to also install scikit-learn library to perform machine learning
operations. Go to the Terminal section of PyCharm and run command
“activate tf” if “tf” virtual environment is not active. Then, run the pip
command.
pip install scikit-learn
Here is the code.
import pandas as pd
import numpy as np
from statistics import mean
from sklearn import svm, preprocessing, model_selection

def addLabel(curHPI, futHPI):

if futHPI > curHPI:
return 1
else:
return 0

def movAvg(values):
return mean(values)

HPIMega = pd.read_pickle('HPIMega.pickle')
HPIMega.rename(columns={"NSA Value": "US_HPI"}, inplace=True)
HPIMega = HPIMega.pct_change() # because to predict future we should
have percent change relative to last value not the start value

HPIMega.replace([-np.inf, np.inf], np.nan, inplace=True) # replace infinities

with NaN
HPIMega['HPI_US_Fut'] = HPIMega['US_HPI'].shift(-1)
HPIMega.dropna(inplace=True) # remove all NaN values
HPIMega['label'] = list(map(addLabel, HPIMega['US_HPI'],
HPIMega['HPI_US_Fut']))

#HPIMega['custom_mort_mean'] =
HPIMega['Mort30yr'].rolling(10).apply(movAvg, raw=True) # example of
rolling apply

# features are 'Mort30yr' and 'Unemployment Rate'

X = preprocessing.scale(np.array(HPIMega.drop(['label', 'HPI_US_Fut'], 1)))

y = np.array(HPIMega['label'])

X_train, X_Test, y_train, y_test = model_selection.train_test_split(X, y,

test_size=0.2) # 20% of the data is reserved for testing instead of being used
completely for training

cLinFunc = svm.SVC(kernel='linear')

cLinFunc.fit(X_train, y_train)
print("Predicting accuracy: %.2f%%" % (cLinFunc.score(X_Test,
y_test)*100))
The above code gives the following output.
Prediction accuracy: 71.23%
The accuracy will fluctuate within a certain range. So a recap: we have
written a script that predicts the labels 0 and 1. Remember that label would
only be 1 if future HPI is greater than current HPI (meaning the real estate
prices will go higher in the future). Our script correctly predicted on roughly
71 instances out of 100 tries when the HPI will rise or drop in the future. We
can use some more data to the system, for example, US GDP, to further
enhance the accuracy.
Are you going to meet the real estate tycoon with your current prediction
model now or ask for more time, get more data, improve accuracy, and then
have a meeting? What are you going to do?
 Chapter 3: Working with Raspberry Pi
Ever heard of an overkill? Your computer or laptop might be a versatile
beast, but sometimes it can feel like using a tank to go to Walmart.
Depending upon your application, it might be easier to deal with a smaller,
reprogrammable computer that offers a specific set of hardware capabilities.
There have been various compact reprogrammable computers available for
decades that engineers and coding enthusiasts have used to develop various
solutions. Raspberry Pi is also one such computers.

What is Raspberry Pi?

Raspberry Pi is a credit-card-sized (a little thicker to be honest) minicomputer

that can be programmed to perform any actions. Three features make
Raspberry Pi standout from the competition:
1. Uses Python as coding language, which is much easier and powerful at
the same time.
2. Smaller size, which makes it ideal for on-to-go applications that offer
less power and space.
3. Readily accessible I/O ports to interact with other systems.
The applications of Raspberry Pi are so numerous that describing each one
needs a separate book. The most common uses coders have found for this
computer are
● automate home appliances
● remote control garage door
● control a robot to perform various tasks
● adding security system to your house
● add features to your car such as your own custom digital infotainment
system
Selecting the Model

The current model of Raspberry Pi is 4 Model B. For this book, we are going
to use the more common Model 3 B because you will find more content
about this model on the internet. You can purchase the minicomputer by
finding authorized resellers for your country on this web page
https://www.raspberrypi.org/products/raspberry-pi-3-model-b/
You will also have to purchase some peripherals. Here is a list.

1. A breadboard
2. A jumper wires kit (male to female is what we require for this
book)
3. A bunch of LED lights (green, yellow, red)
4. A bunch of resistors (one 1k Ω, one 2k Ω, three 300-1k Ω)
5. Distance sensor (two HC-SR04)
6. Raspberry Pi camera module
7. A USB microSD card reader for your computer if it doesn’t have a
reader slot
8. A microUSB power supply (2.1 A rating)
9. A microSD card with at least 16GB of storage capacity and an
adapter if you have purchased an older model
10. For development, you will need a monitor,
keyboard, and mouse

Hardware Components

The Model 3 B replaced the Model 2 B in early 2016. It has the following
hardware configuration:
● Quad Core 64bit CPU 1.2gHz Broadcom BCM2837
● 1GB Ram
● Wireless LAN and Bluetooth Low Energy (BLE) BCM43438 on board
● 100 Base Ethernet
● 40-pin GPIO (extended)
● 4 USB v2 ports
● Composite video port
● 4 Pole stereo audio port
● Full size HDMI
● CSI port to connect Raspberry Pi camera
● CSI port to connect Raspberry Pi touchscreen display
● MicroSD port
● MicroUSB power point that supports up to 2.5 A rated power sources
According to the manufacturers, this particular model complies with
following European standards.

1. Electromagnetic Compatibility Directive (EMC) 2014/30/EU

2. Restriction of Hazardous Substances (RoHS) Directive
2011/65/EU

First Project

The codes in this section are inspired by the work of “sentdex” Youtube
channel. The images used are also provided by the channel.

Installation and Setup

Go to the https://www.raspberrypi.org/downloads/ and select the NOOBS

because that’s the easier option to install the software. Installing Raspbian OS
directly on the Raspberry Kit is a bit overwhelming and not advised for
someone who is just starting with Raspberry Pi.. You will be redirected to
https://www.raspberrypi.org/downloads/noobs/ and from there, select the
NOOBS option. The NOOB Lite is faster to download but takes longer to
install on the Raspberry kit.
When the download is finished, extract the files from the zipped folder.
Before you flush the contents of the extracted folder to the microSD card for
Raspberry Pi, you need to correctly format the microSD card. Download and
install the official SD card formatter from
https://www.sdcard.org/downloads/formatter/ for your operating system.
Plug in the SD card to your computer and start the formatter, select the
correct Drive, click on the “Option” button and select “Full (Erase)” instead
of “Quick” for the “Format Type”. Also, select “On” for the “Format Size
Adjustment”. Click on “Ok” and then “Format”. Click “Ok” if you are asked
to confirm the action. It might take a minute for the formatting to complete,
exit the formatter application.
Let’s copy the contents of extracted NOOBS folder and paste it to the
microSD drive. It can take some time as the files are in gigabytes. To turn on
and off the Raspberry Pi, you simply plug in or out the power cord. For this
reason, you should connect all peripherals before you plug in the power
because if you connect the peripheral later, it might not get recognized. Once
the copy/paste operation is complete, we have to plug the microSD card in
the Raspberry Pi memory card slot which is on the bottom side of the kit.
Let’s connect a monitor and keyboard to the kit as well and plug in the
power. A red LED light will turn on the Raspberry Pi, indicating it’s turned
on. You will see installation wizard start on the monitor.
Go through the installation wizard and let the operating system installation
finish. It can take between 10-30 minutes for the process to complete. Once
the installation is complete, you will see the Raspbian desktop on the
monitor. Just like Windows, you can change the desktop background if you
prefer but that’s totally optional.
We have to update all the firmware of the Raspberry kit to the latest version.
For that, you need to connect the kit with WiFi (you can also plugin an
Ethernet cable if you don’t have WiFi). On the top left corner of your screen,
you will see an icon beside the Bluetooth icon that will show two lines with
red crosses. Click on that icon; a small popup will appear listing all the
available WiFi networks. Connect to your WiFi by entering the WiFi
password if set. Once connected to WiFi, open the terminal by pressing
CTRL + ALT + T and run the following command.
sudo apt-get update
Then,
sudo apt-get upgrade
Hopefully, the process will finish without any issues. But, if the update fails,
you have to perform a few prerequisite tasks (correct the update distribution
source link). Open the sources.list using the following command on the
terminal:
sudo nano /etc/apt/sources.list
Once the file is open, replace everything with the following two commands.
(Hint: Use CTRL+k to delete a line).
deb http://archive.raspbian.org/raspbian jessie main contrib non-free
deb-src http://archive.raspbian.org/raspbian jessie main contrib non-free
Use CTRL+X to close the file and select “Yes” when the system asks you to
save the file.
Press “Enter” again to keep the same filename. Let’s try upgrading the
distribution.
sudo apt-get dist-upgrade
After it’s completed successfully, we should rerun the original update and
upgrade commands. But, first, we should purge wolfram engine from the
system because it takes a lot of storage and doesn’t have any benefits.
sudo apt-get purge wolfram
Enter ‘y’ to confirm and it will be deleted from the memory card. One more
command to run here is, again enter ‘y’ to confirm.
sudo apt-get autoremove
Once that’s finished execution, let’s get back to updating and upgrading.
sudo apt-get update
sudo apt-get upgrade
Tip: if you want to break a process, press CTRL+C.

Remote Access to Raspberry Pi

We can access Raspberry Pi remotely on the same (local) network through

two major methods.
1. Remote desktop
2. Secure Shell (SSH)
In both methods, the first step is to make sure the Raspberry Pi is connected
to your wifi network.
SSH Method
We start by opening the terminal by pressing CTRL+ALT+T and running the
following commands.
ifconfig
You will see details about your internet connection. You are looking for the
“inet addr” which is your local IP address. Grab that IP address and then we
have to enable the SSH because it’s disabled by default. Open the Raspberry
configuration with the following command.
sudo raspi-config
Navigate to “Interfacing Options”, select “SSH”, you will be asked to
confirm, select “Yes” and SSH will be enabled. Select “Finish” to close the
Raspberry configuration. We can now remotely connect our computer with
Raspberry Pi. On Windows, we will require to download an application
called Putty. Here is the URL
https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html select the
correct version that matches your Windows system. Once downloaded, install
and start Putty. Note that the latest Windows 10 version supports SSH by
default so you might want to update your Windows if you don’t want to
install Putty.
On the Putty screen, enter the IP address you collected from the Raspberry Pi
and hit “Enter”. You will see a new black window. Use “pi” in the “login as”
and enter “raspberry” when asked for the password (these are the default
values so every time you connect using this information you will be shown
warning to change the login credentials). The connection will be established,
you might be asked to confirm a few things, select “Yes” to finish setting up
the remote connection.
Remote Desktop
This method is relatively easier to setup on Windows. If you are using the
latest Windows version, this option is also included by default. You just need
to install a remote desktop client on the Raspberry Pi. We are going to use
“xrdp”. Let’s install using the following command.
sudo apt-get install tightvncserver
And then,
sudo apt-get install xrdp
If you try to install “xrdp” client and it doesn’t go through, you might want to
remove a few (Virtual Network Computing) VNC servers that are already
added to the Raspberry Pi. Use the following command to do so.
sudo apt-get remove xrdp vnc4server tightvncserver
Once the installation is complete, let’s start “Remote Desktop” application on
Windows and enter the local IP address we had gathered before. Click “Yes”
if you are asked to confirm anything. After a few moments, you will see a
login screen, use “pi” and “raspberry” in the username and password fields,
respectively. You will see a screen showing a lot of updates, and if
everything goes right, you will see the Raspberry Pi desktop on your
computer screen. Congratulations, the connection is set.
Two important commands before we start doing some fun stuff. To reboot the
Raspberry Pi, use the following command.
sudo reboot
To completely shutdown the kit, use the following command.
sudo shutdown -h now

Using Camera with Raspberry Pi

Raspberry camera was last updated in spring 2016. It is an 8MP (Mega Pixel)
Sony IMX219 sensor camera that supports capture of high definition video
and images. You can also perform advanced video effects like time-lapse and
slow-motion. The new camera has better image quality even in low-lit
environments and exceptional color fidelity. The supported video formats are
1080p on 60 frames/sec, 720p on 60 frames/sec and the low-quality VGA90
that’s suitable for long video recording that can clog the memory.
Let’s plug the camera in the special camera slot and connect all other
peripherals before powering up the Raspberry Pi. Once everything's ready,
login to the Raspberry OS using the remote connection. Open the terminal
and run the following command.
cd Desktop/
We also have to enable camera interfacing by starting Raspberry
Configuration.
sudo raspi-config
Select “Interfacing Options”, “Camera” is the first option, select it, select
“Yes” to enable camera interface. Select “Finish” to close the configuration.
You might have to reboot the Raspberry Pi, go ahead if it needs to be done.
Once it powers back on, run the following command again if needed.
cd Desktop/
Run another command to create a script file on the desktop.
nano cameraex.py
This will open the script file. Let’s add the following script.
import picamera
import time

cam = picamera.PiCamera()
cam.capture(‘firstImage.jpg’)
Enter CTRL+X to close the script file, save it with the same name, and run
the script using the terminal with the following command.
python cameraex.py
On the desktop, there will be a new image file. Open it to see the result. We
can perform various image manipulations. For example, to vertically flip the
image, we can change the “cameraex.py” script like this.
import picamera
import time

cam = picamera.PiCamera()
cam.vflip = True
cam.capture(‘firstImage.jpg’)
Let’s record a video. We have to replace the cam.capture() with another line
of code.
import picamera
import time
cam = picamera.PiCamera()
#cam.vflip = True
#cam.capture(‘firstImage.jpg’)
cam.start_recording(‘firstvid.h264’)
time.sleep(5)
cam.stop_recording
Let’s run the script using the following command.
python cameraex.py
To view the video, we need to use a media player. Raspbarian OS comes
loaded with media player called “OMX Player”. Let’s call it in the terminal to
view the recorded video.
omxplayer firstvid.h264
We can also test the camera with a live feed. Remove all the code from the
script file and add the following lines, run the script from the terminal.
import picamera

cam = picamera.PiCamera()
cam.start_preview()
You should see a live video feed on your screen now. It might take a few
seconds to appear. Also, if you are remotely running the Raspberry OS, make
sure the monitor connected with the Raspberry Pi is turned on because all the
images and videos only show up in that monitor, not on the remotely
connected system monitor.
Sending and Receiving Signals Using GPIO of the Raspberry Pi

Sending Signal
Let’s set up a series circuit on the breadboard using a 300-1k Ω resistor, a red
LED (you can choose any other color), and the two male to female jumpers.
If you have no idea how to create circuits on a breadboard, here’s a tutorial
that can get you up to speed https://www.youtube.com/watch?
v=6WReFkfrUIk.
Make sure you remember the pins where you connect the female end of
jumpers on the GPIO. To make things easier, let’s connect the female end of
jumper at the GPIO23 position. Here is the map of GPIO pins on the
Raspberry Pi to make more sense.

Once the circuit is ready, power up the Raspberry Pi and let’s start with the
coding. Open the terminal, navigate to the desktop and just to double check,
we will install the GPIO RPI using the following command.
sudo apt-get install python-rpi.gpio
If the RPI is already installed, the terminal will let you know. Let’s create a
script file “ledex.py” and add the following script to it.
import RPI.GPIO as GPIO
import time
# female end of jumper connected at GPIO23

GPIO.setmode(GPIO.BCM)
GPIO.setup(23, GPIO.OUT)

GPIO.output(23, GPIO.HIGH)
time.sleep(5)
GPIO.output(23, GPIO.LOW)
GPIO.cleanup()
Save the script file and run it through the terminal using the following
command.
python ledex.py
You will be able to see the LED light up for 5 seconds and then turn off.
Receiving Signal
Let’s create a circuit using the HC SR04 distance sensor, four jumpers, and
three 1k Ω resistors. How does the distance sensor work? We can send a
sound signal (trigger) using the sensor which strikes a physical object in front
of the sensor and is received back by the sensor (echo). Since we know the
speed of sound, we can calculate the time difference between the trigger and
echo to calculate the distance between the sensor and the physical object.
Once everything is connected as shown in the image below, power up the kit
and start the terminal to run a few commands.
cd Desktop/
nano distancesensor.py
In the new script file, add the following script.
import RPI.GPIO as GPIO
import time

GPIO.setmode(GPIO.BCM)

TRIG = 4 # GPIO pin number where trigger of distance sensor is connected

ECHO = 18 # GPIO pin number where echo of distance sensor is connected

GPIO.setup(TRIG, GPIO.OUT)
GPIO.setup(ECHO, GPIO.IN)

GPIO.output(TRIG, True)
time.sleep(0.0005)
GPIO.output(TRIG, False)

while GPIO.input(ECHO) == False:

strtP = time.time()
while GPIO.input(ECHO) == True:
endP = time.time()
travlTime = endP - strtP
distance = travlTime / 0.000058
print(‘Distance: %.2f cm’, % (distance))
GPIO.cleanup()
Back on the terminal, run the script with the following command.
python distancesensor.py
Now, let’s move on to the real purpose of this project, to create an automatic
garage stop light that will be activated when you pull your car into the
garage. The light should be green if there’s space in the garage, turn yellow
as the car moves closer and turn red when there’s no space.
The circuit should be setup according to the given figure.
After making the connections, power up the Raspberry Pi and create a new
script file to host the following script.
import RPi.GPIO as GPIO
import time

GPIO.setwarnings(False) # doing this first, since we're using a while True.

GPIO.cleanup() # this resets the pins in the GPIO port

GPIO.setmode(GPIO.BCM)
# declaring pin connections
TRIG = 4
ECHO = 18
GREEN = 17
YELLOW = 27
RED = 22

# declaring input and output signals

GPIO.setup(TRIG,GPIO.OUT)
GPIO.setup(ECHO,GPIO.IN)

GPIO.setup(GREEN,GPIO.OUT)
GPIO.setup(YELLOW,GPIO.OUT)
GPIO.setup(RED,GPIO.OUT)

def greenLEDon():
GPIO.output(GREEN, GPIO.HIGH)
GPIO.output(YELLOW, GPIO.LOW)
GPIO.output(RED, GPIO.LOW)

def yellowLEDon():
GPIO.output(GREEN, GPIO.LOW)
GPIO.output(YELLOW, GPIO.HIGH)
GPIO.output(RED, GPIO.LOW)

def redLEDon():
 GPIO.output(GREEN, GPIO.LOW)
GPIO.output(YELLOW, GPIO.LOW)
GPIO.output(RED, GPIO.HIGH)

def getDistance(): # this is the code we developed before

GPIO.output(TRIG, True)
time.sleep(0.000005)
GPIO.output(TRIG, False)

while GPIO.input(ECHO) == False:

strtP = time.time()

while GPIO.input(ECHO) == True:

endP = time.time()

travlTime = endP - startP

# getting distance in centimeters

distance = travlTime / 0.000058

#print('Distance: %.2f cms' %(distance))

return distance
while True:
distance = getDistance()
time.sleep(0.005)
print(distance)

if distance >= 30:

greenLEDon()
elif 30 > distance > 10:
yellowLEDon()
elif distance <= 10:
redLEDon()
You can test this script by moving your hand closer and further from the
sensor. We can build upon this to create a script that will automatically open
and close the garage door when you pull up on the driveway. Can you think
of how machine learning can make these scripts give better results? Maybe
the distance measured over a period of time can be analyzed to profile your
driving behavior and make accurate predictions like how fast you would be
pulling up in the garage on a Monday evening coming back from work!
The above simple project was just an example showcasing the power of
Raspberry Pi. Feel free to experiment with Raspberry Pi and take on some
projects. Here are a few cool examples.
1. Create a weather station that monitors atmospheric parameters and
makes predictions.
2. Create a social media bot. You can work with Twitter as it’s a bit
easier.
3. Create a Minecraft game server. Good thing is, this project doesn't
require much hardware investment.
4. Create a time-lapse camera. Everyone loves taking cool videos!
 Chapter 4: Working with TensorFlow
Remember we briefly talked about TensorFlow at the start of this book but
left it for later? Well, it’s time to learn everything about TensorFlow. So,
what is TensorFlow? TensorFlow is a Python library for machine learning
and scientific computing. We have already set up all the requirements, so
let’s get started with a hands-on project.

The Projects

As a demonstration of how TensorFlow can be used to implement machine

learning in a real-life scenario, we will focus on a large data set that contains
student attributes along with previous grades and successfully train and test a
model to make accurate predictions on student performance in high school.
On PyCharm, go ahead and close all open projects. Now, you will see the
initial window and on the left sidebar, you will see the “tensorEnv” project,
click on it to open. This is the project we will be working on for this chapter.
To download the data, go to the link
https://archive.ics.uci.edu/ml/datasets/Student+Performance and click on
highlighted “Data Folder”. In the new page, select “student.zip” and
download to the folder anywhere on your computer. Now, unzip the folder
and grab the student-met.csv file and move it to the working directory of the
“tensorEnv” current project.

Project #1: Predicting Student Grade

We have to prepare the data set before creating a model based on it. We will
look to eliminate rows and/or columns that have missing data and discard
student attributes that we consider surplus. An important note: do not discard
too much data or you might end up with very bad prediction scores. The more
data you have, the more chances of creating an accurate prediction model.
For our project, we are going to relate the past student grades with their
weekly study time (studytime), previous class failures (failures), and school
absences (absences) to predict their future grade.
import pandas as pd
import numpy as np
import sklearn
from sklearn import linear_model

stdntFram = pd.read_csv('student-mat.csv', delimiter=";") # usually files

have ‘,’ as delimiter so we don’t need to set this but in this file, we have to
declare the correct data delimiter
#print(stdntFram.head()) # check the data frame to see if we need to prepare
the data before building the model

# G1, G2, G3 are the semester grades for every student

stdntFram = stdntFram[['G1', 'G2', 'G3', 'studytime', 'failures', 'absences']] #

discarding data that's currently irrelevant to our current project requirements.
You might want to experiment by keeping a few more attributes and see if
prediction scores get better

predict = 'G3' # goal is to predict this as close to the actual values, will
remove this column from stdntFram before training and testing

X = np.array(stdntFram.drop([predict], 1))
y = np.array(stdntFram[predict])

X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X,

y, test_size=0.2) # segment data set between training and testing data, 20%
testing data is the standard
linMod = linear_model.LinearRegression() # choose a model, linear in this
case

linMod.fit(X_train, y_train) # train the model

accuracy = linMod.score(X_test, y_test)

print("Prediction accuracy: %.2f%%" % (accuracy * 100)) # will fluctuate
within a range on subsequent script executions

allPred = linMod.predict(X_test)

for itr in range(len(allPred)):

print(allPred[itr], X_test[itr], y_test[itr]) # first is predicted by our
system and last value is what the value actually is
Look at the data closely and see what other observations you can make.
Modify the script to use more attributes and see if how closely other
attributes relate to the grades and if prediction accuracy can be further
improved.
Prediction accuracy: 87.09%
6.754745900887105 [7 8 4 0 8] 8
13.109798237848533 [14 13 3 1 12] 13
3.632451977653248 [6 5 3 1 0] 0
4.3744867252888895 [6 5 1 0 7] 6

Project #2: Predicting Student Grade

Let’s take on another project. This time we are going to implement the K-
Nearest Neighbors (KNN) algorithm to relate an integer ‘k’ with its closest
neighbors. We are going to deal with a data set that will require more
preprocessing than we did in the earlier project. The KNN algorithm is
usually used to classify data points, for example, a movies database can be
modeled to learn about good and bad movies and predict a future movie’s
classification as bad or good. In our project, we are going to classify cars in
view of certain car attributes.
Let’s go to https://archive.ics.uci.edu/ml/data sets/Car+Evaluation and
download the “Car Evaluation Data Set” as we did for the first project. We
are going to add a header to this file that will make it easier to use pandas to
read the data. Just open the data file in PyCharm and add the following line
as the first line of the file. Don’t forget to save the file afterwards.
buying,maint,door,persons,lug_boot,safety,class
Let’s start coding.
import sklearn
from sklearn.neighbors import KNeighborsClassifier
import pandas as pd
from sklearn import preprocessing

data = pd.read_csv("car.data")
#print(data.head()) # check the data frame to see if we need to prepare the
data before building the model

lblEnc = preprocessing.LabelEncoder()
buying = lblEnc.fit_transform(list(data["buying"]))
maint = lblEnc.fit_transform(list(data["maint"]))
door = lblEnc.fit_transform(list(data["door"]))
persons = lblEnc.fit_transform(list(data["persons"]))
lug_boot = lblEnc.fit_transform(list(data["lug_boot"]))
safety = lblEnc.fit_transform(list(data["safety"]))
clasify = lblEnc.fit_transform(list(data["class"]))

predict = "class" # optional

X = list(zip(buying, maint, door, persons, lug_boot, safety))

y = list(clasify)

X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X,

y, test_size=0.2)

knnModel = KNeighborsClassifier(n_neighbors=9)

knnModel.fit(X_train, y_train)

accuracy = knnModel.score(X_test, y_test)

print("Prediction accuracy: %.2f%%" % (accuracy * 100))

allPred = knnModel.predict(X_test)

classifyNames = ['unacc', 'acc', 'good', 'vgood']

for x in range(len(X_test)):
print("Predicted value: ", classifyNames[allPred[x]], "Test data: ",
X_test[x], "Actual data: ", classifyNames[y_test[x]])
The above code will output the following results.
Prediction accuracy: 90.46%
Predicted value: good Test data: (1, 3, 3, 1, 2, 1) Actual data: good
Predicted value: good Test data: (1, 0, 0, 0, 2, 0) Actual data: good
Predicted value: good Test data: (2, 1, 1, 0, 0, 1) Actual data: good
Predicted value: good Test data: (0, 3, 2, 2, 0, 0) Actual data: good
Predicted value: unacc Test data: (1, 1, 3, 1, 2, 2) Actual data: unacc
Predicted value: unacc Test data: (3, 1, 2, 2, 1, 2) Actual data: unacc
Predicted value: good Test data: (2, 0, 3, 1, 2, 1) Actual data: good
Predicted value: acc Test data: (1, 1, 3, 2, 0, 2) Actual data: acc
Predicted value: good Test data: (3, 0, 0, 1, 2, 1) Actual data: good
Predicted value: good Test data: (2, 0, 0, 0, 2, 2) Actual data: good
Predicted value: good Test data: (1, 1, 1, 2, 0, 1) Actual data: good
Predicted value: good Test data: (3, 0, 0, 1, 2, 0) Actual data: good
Predicted value: unacc Test data: (1, 1, 2, 2, 2, 2) Actual data: unacc
Predicted value: good Test data: (1, 2, 0, 1, 1, 1) Actual data: good
Predicted value: unacc Test data: (1, 0, 2, 1, 0, 2) Actual data: unacc
Predicted value: good Test data: (3, 2, 0, 0, 2, 0) Actual data: good
Predicted value: good Test data: (1, 2, 1, 1, 2, 1) Actual data: good
Predicted value: good Test data: (3, 3, 3, 2, 0, 1) Actual data: good
Predicted value: unacc Test data: (3, 1, 3, 1, 2, 0) Actual data: unacc
Predicted value: good Test data: (0, 3, 3, 0, 2, 2) Actual data: good
Predicted value: good Test data: (1, 3, 2, 1, 1, 2) Actual data: unacc
Predicted value: unacc Test data: (2, 3, 3, 2, 0, 0) Actual data: unacc
Predicted value: good Test data: (2, 0, 1, 0, 2, 2) Actual data: good
Predicted value: good Test data: (2, 1, 1, 0, 2, 2) Actual data: good
Predicted value: good Test data: (0, 2, 3, 1, 0, 0) Actual data: unacc
Predicted value: good Test data: (0, 2, 0, 1, 2, 2) Actual data: good
Predicted value: good Test data: (2, 0, 0, 0, 1, 1) Actual data: good
Predicted value: acc Test data: (1, 1, 2, 1, 0, 2) Actual data: acc
Predicted value: good Test data: (2, 3, 0, 2, 0, 0) Actual data: unacc
Predicted value: good Test data: (0, 3, 0, 0, 0, 1) Actual data: good
Predicted value: good Test data: (0, 0, 0, 0, 1, 0) Actual data: good
Predicted value: unacc Test data: (1, 2, 0, 1, 1, 0) Actual data: acc
Predicted value: good Test data: (1, 1, 0, 0, 1, 1) Actual data: good
Predicted value: good Test data: (0, 1, 0, 2, 2, 1) Actual data: good
Predicted value: good Test data: (0, 3, 1, 0, 1, 2) Actual data: good
Predicted value: good Test data: (3, 3, 2, 0, 2, 2) Actual data: good
Predicted value: good Test data: (2, 1, 0, 2, 2, 0) Actual data: good
Predicted value: unacc Test data: (2, 2, 1, 2, 0, 2) Actual data: unacc
Predicted value: good Test data: (0, 0, 1, 2, 0, 1) Actual data: good
Predicted value: good Test data: (2, 3, 1, 0, 1, 0) Actual data: good
Predicted value: good Test data: (3, 1, 1, 1, 2, 2) Actual data: good
Predicted value: good Test data: (2, 3, 0, 1, 0, 0) Actual data: unacc
Predicted value: acc Test data: (2, 1, 1, 1, 2, 0) Actual data: acc
Predicted value: good Test data: (3, 0, 3, 1, 0, 0) Actual data: good
Predicted value: good Test data: (2, 0, 1, 0, 2, 0) Actual data: good
Predicted value: good Test data: (1, 0, 2, 0, 1, 2) Actual data: good
Predicted value: good Test data: (0, 3, 0, 0, 2, 0) Actual data: good
Predicted value: good Test data: (3, 2, 0, 0, 1, 2) Actual data: good
Predicted value: good Test data: (0, 1, 0, 2, 0, 2) Actual data: unacc
Predicted value: good Test data: (0, 3, 3, 0, 1, 0) Actual data: good
Predicted value: good Test data: (3, 3, 0, 0, 1, 0) Actual data: good
Predicted value: good Test data: (3, 0, 2, 0, 2, 0) Actual data: good
Predicted value: good Test data: (0, 3, 3, 2, 2, 0) Actual data: good
Predicted value: good Test data: (1, 2, 0, 2, 1, 0) Actual data: acc
Predicted value: good Test data: (1, 1, 2, 1, 1, 1) Actual data: good
Predicted value: good Test data: (2, 2, 1, 1, 1, 2) Actual data: unacc
Predicted value: unacc Test data: (0, 1, 1, 1, 0, 0) Actual data: unacc
Predicted value: good Test data: (3, 1, 0, 1, 0, 0) Actual data: unacc
Predicted value: good Test data: (2, 0, 0, 2, 2, 1) Actual data: good
Predicted value: good Test data: (1, 2, 3, 0, 1, 1) Actual data: good
Predicted value: good Test data: (3, 2, 2, 0, 0, 1) Actual data: good
Predicted value: good Test data: (1, 2, 2, 0, 0, 1) Actual data: good
Predicted value: good Test data: (0, 2, 2, 1, 0, 0) Actual data: unacc
Predicted value: good Test data: (1, 0, 3, 0, 1, 0) Actual data: good
Predicted value: unacc Test data: (3, 2, 1, 1, 2, 0) Actual data: unacc
Predicted value: good Test data: (0, 0, 0, 0, 1, 2) Actual data: good
Predicted value: good Test data: (1, 3, 0, 2, 0, 0) Actual data: unacc
Predicted value: good Test data: (2, 3, 2, 1, 0, 1) Actual data: good
Predicted value: good Test data: (2, 2, 0, 1, 1, 1) Actual data: good
Predicted value: vgood Test data: (2, 1, 1, 2, 0, 0) Actual data: vgood
Predicted value: good Test data: (1, 3, 2, 2, 2, 1) Actual data: good
Predicted value: unacc Test data: (1, 1, 1, 1, 1, 2) Actual data: unacc
Predicted value: good Test data: (2, 3, 3, 0, 1, 1) Actual data: good
Predicted value: good Test data: (2, 3, 2, 0, 1, 2) Actual data: good
Predicted value: good Test data: (3, 0, 2, 2, 0, 1) Actual data: good
Predicted value: good Test data: (2, 2, 2, 2, 2, 1) Actual data: good
Predicted value: unacc Test data: (2, 1, 2, 2, 0, 2) Actual data: acc
Predicted value: good Test data: (3, 3, 1, 1, 2, 0) Actual data: good
Predicted value: unacc Test data: (2, 2, 3, 1, 0, 2) Actual data: unacc
Predicted value: good Test data: (3, 1, 0, 1, 0, 1) Actual data: good
Predicted value: good Test data: (3, 0, 1, 0, 1, 0) Actual data: good
Predicted value: good Test data: (2, 1, 0, 2, 0, 1) Actual data: good
Predicted value: unacc Test data: (2, 3, 2, 1, 2, 0) Actual data: unacc
Predicted value: good Test data: (0, 2, 0, 2, 2, 2) Actual data: good
Predicted value: unacc Test data: (2, 3, 3, 1, 0, 2) Actual data: unacc
Predicted value: good Test data: (2, 1, 2, 2, 2, 1) Actual data: good
Predicted value: good Test data: (3, 0, 3, 2, 2, 2) Actual data: good
Predicted value: acc Test data: (2, 1, 1, 2, 1, 0) Actual data: vgood
Predicted value: good Test data: (0, 3, 0, 1, 2, 0) Actual data: good
Predicted value: good Test data: (1, 1, 2, 0, 0, 0) Actual data: good
Predicted value: vgood Test data: (2, 1, 2, 2, 0, 0) Actual data: vgood
Predicted value: good Test data: (0, 3, 1, 0, 1, 1) Actual data: good
Predicted value: good Test data: (2, 1, 1, 0, 2, 0) Actual data: good
Predicted value: good Test data: (1, 3, 3, 0, 0, 1) Actual data: good
Predicted value: good Test data: (3, 0, 0, 2, 2, 1) Actual data: good
Predicted value: unacc Test data: (1, 0, 2, 2, 2, 0) Actual data: unacc
Predicted value: unacc Test data: (0, 0, 0, 2, 0, 0) Actual data: unacc
Predicted value: good Test data: (1, 1, 2, 0, 1, 2) Actual data: good
Predicted value: good Test data: (3, 3, 0, 2, 2, 0) Actual data: good
Predicted value: acc Test data: (2, 1, 2, 2, 1, 2) Actual data: acc
Predicted value: good Test data: (3, 3, 3, 0, 2, 1) Actual data: good
Predicted value: good Test data: (1, 3, 1, 0, 1, 0) Actual data: good
Predicted value: good Test data: (3, 3, 0, 1, 1, 0) Actual data: good
Predicted value: unacc Test data: (3, 1, 0, 1, 1, 0) Actual data: unacc
Predicted value: good Test data: (3, 1, 1, 0, 2, 0) Actual data: good
Predicted value: good Test data: (1, 1, 2, 2, 0, 1) Actual data: good
Predicted value: unacc Test data: (2, 2, 1, 1, 0, 2) Actual data: unacc
Predicted value: good Test data: (3, 3, 1, 0, 2, 0) Actual data: good
Predicted value: good Test data: (1, 0, 1, 2, 2, 1) Actual data: good
Predicted value: unacc Test data: (1, 1, 0, 1, 1, 2) Actual data: unacc
Predicted value: unacc Test data: (2, 0, 2, 1, 1, 0) Actual data: unacc
Predicted value: acc Test data: (1, 1, 2, 1, 2, 0) Actual data: acc
Predicted value: good Test data: (2, 3, 1, 0, 2, 2) Actual data: good
Predicted value: good Test data: (2, 1, 0, 0, 2, 2) Actual data: good
Predicted value: good Test data: (1, 3, 2, 1, 0, 1) Actual data: good
Predicted value: good Test data: (2, 0, 2, 1, 1, 1) Actual data: good
Predicted value: good Test data: (0, 0, 0, 0, 2, 0) Actual data: good
Predicted value: good Test data: (1, 0, 0, 0, 2, 1) Actual data: good
Predicted value: good Test data: (3, 1, 2, 0, 1, 1) Actual data: good
Predicted value: unacc Test data: (2, 1, 2, 2, 2, 2) Actual data: unacc
Predicted value: good Test data: (3, 2, 2, 1, 2, 1) Actual data: good
Predicted value: good Test data: (0, 2, 0, 1, 0, 1) Actual data: good
Predicted value: unacc Test data: (1, 2, 0, 1, 2, 0) Actual data: acc
Predicted value: good Test data: (3, 2, 0, 1, 1, 1) Actual data: good
Predicted value: good Test data: (0, 1, 0, 0, 2, 1) Actual data: good
Predicted value: good Test data: (1, 0, 2, 0, 1, 1) Actual data: good
Predicted value: good Test data: (2, 0, 3, 0, 0, 1) Actual data: good
Predicted value: good Test data: (0, 3, 1, 2, 1, 0) Actual data: good
Predicted value: good Test data: (1, 3, 2, 2, 2, 2) Actual data: good
Predicted value: good Test data: (1, 1, 0, 2, 1, 0) Actual data: acc
Predicted value: good Test data: (1, 2, 2, 2, 0, 1) Actual data: good
Predicted value: good Test data: (3, 1, 3, 0, 0, 1) Actual data: good
Predicted value: unacc Test data: (0, 2, 1, 2, 2, 0) Actual data: unacc
Predicted value: unacc Test data: (2, 3, 2, 2, 0, 2) Actual data: unacc
Predicted value: good Test data: (2, 3, 2, 0, 1, 1) Actual data: good
Predicted value: good Test data: (3, 3, 3, 1, 1, 1) Actual data: good
Predicted value: good Test data: (1, 2, 1, 2, 0, 1) Actual data: good
Predicted value: good Test data: (1, 0, 0, 0, 2, 2) Actual data: good
Predicted value: good Test data: (1, 2, 2, 0, 2, 1) Actual data: good
Predicted value: good Test data: (1, 3, 3, 0, 1, 1) Actual data: good
Predicted value: vgood Test data: (2, 0, 3, 2, 0, 0) Actual data: unacc
Predicted value: good Test data: (3, 0, 2, 0, 2, 1) Actual data: good
Predicted value: acc Test data: (2, 1, 0, 2, 0, 2) Actual data: acc
Predicted value: good Test data: (1, 1, 0, 1, 1, 0) Actual data: acc
Predicted value: good Test data: (0, 0, 1, 2, 2, 2) Actual data: good
Predicted value: good Test data: (0, 3, 2, 2, 2, 0) Actual data: good
Predicted value: good Test data: (2, 0, 3, 2, 1, 1) Actual data: good

Project #3: Neural Network using TensorFlow

In this project, we are going to create a neural network that will work on the
TensorFlow Keras Fashion MNIST image data set to classify clothing
images. We already have setup all the prerequisites so let’s start with the
coding.
Here’s the code. I have added comments wherever necessary.
import tensorflow as tf
from tensorflow import keras
import numpy as np
import matplotlib.pyplot as plt

imgData = keras.data sets.fashion_mnist # import the image data set

(img_train, lbl_train), (img_test, lbl_test) = imgData.load_data() # segment

data set into training and testing data

# declaring available cloth types

cloth_type = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt',
'Sneaker', 'Bag', 'Ankle Boot']

# normalizing images using scalar division

img_train = img_train / 255.0
img_test = img_test / 255.0
'''define model
- flatten = prepare image which is 2d/3d array so it can be passed through
individual 1d neurons
- dense = fully connect all neurons to create a layer
- relu = input layer, rectified linear unit, we can use other activation function
- softmax = output layer, pick all neuron values in a way they add up to one
(which also makes it show probability)
'''
neuralModel = keras.Sequential([
keras.layers.Flatten(input_shape=(28, 28)),
keras.layers.Dense(128, activation="relu"),
keras.layers.Dense(10, activation="softmax")
])

# create the model using the above definition, typical values are used for
optimizer, loss and metrics
neuralModel.compile(optimizer="adam",
loss="sparse_categorical_crossentropy", metrics=["accuracy"])

neuralModel.fit(img_train, lbl_train, epochs=5) # using higher epochs

doesn't improve the accuracy a lot

testLoss, testAcc = neuralModel.evaluate(img_test, lbl_test)

print("Tested Accuracy: %.2f%%" % (testAcc*100))
# perform prediction
allPred = neuralModel.predict(img_test)

# we can use following code to confirm our script is making correct

predictions
for itr in range(5):
plt.grid(False)
plt.imshow(img_test[itr], cmap=plt.cm.binary) # shows a better
version of the image
plt.xlabel("Actual: " + cloth_type[lbl_test[itr]])
plt.title("Prediction: " + cloth_type[np.argmax(allPred[itr])])
plt.show()
The output of the above script is:
Tested Accuracy: 88.0%
 Chapter 5: Advanced Machine Learning
If you are starting the last chapter of this book, it means you have a high
degree of understanding of all the underlying concepts of Python
programming and machine learning. If that’s not the case, it’s highly
recommended to revisit the previous chapters. This chapter is going to be
short and direct. We are going to take on a corporate level project.

A Corporate Project

You have an online store and want to optimize the chat service your support
staff uses, so outside office hours, a chat bot will take over chat service and
offer answers to most common queries.

Create a Predictive Chat Bot

The chat bot will learn the user queries and over time provide better answers.
Let’s start coding without any delay. Go ahead and install the “nltk” and
“tflearn” libraries in the virtual environment. I am going to use a json file to
give training and testing data to our chat bot. I have filled the json file with
some random strings that look like most common user queries during the chat
in the past.
Here’s the complete code inspired by the work of “techwithtim” Youtube
channel. The chat bot will have to process natural language, in Python, we
can use the nltk library for this purpose.
import nltk
from nltk.stem.lancaster import LancasterStemmer
stemmer = LancasterStemmer()

import numpy
import tflearn
import tensorflow
import json
import pickle

with open("intents.json") as file:

data = json.load(file)

try:
with open("data.pickle", "rb") as f:
words, labels, training, output = pickle.load(f)
except:
words = []
labels = []
docs_x = []
docs_y = []

for intent in data["intents"]:

for pattern in intent["patterns"]:
wrds = nltk.word_tokenize(pattern)
words.extend(wrds)
docs_x.append(wrds)
docs_y.append(intent["tag"])

if intent["tag"] not in labels:

labels.append(intent["tag"])
 words = [stemmer.stem(w.lower()) for w in words if w != "?"]
words = sorted(list(set(words)))

labels = sorted(labels)

training = []
output = []

out_empty = [0 for _ in range(len(labels))]

for x, doc in enumerate(docs_x):

bag = []

wrds = [stemmer.stem(w.lower()) for w in doc]

for w in words:
if w in wrds:
bag.append(1)
else:
bag.append(0)

output_row = out_empty[:]
 output_row[labels.index(docs_y[x])] = 1

training.append(bag)
output.append(output_row)

training = numpy.array(training)
output = numpy.array(output)

with open("data.pickle", "wb") as f:

pickle.dump((words, labels, training, output), f)

tensorflow.reset_default_graph()

net = tflearn.input_data(shape=[None, len(training[0])])

net = tflearn.fully_connected(net, 8)
net = tflearn.fully_connected(net, 8)
net = tflearn.fully_connected(net, len(output[0]), activation="softmax")
net = tflearn.regression(net)

model = tflearn.DNN(net)

try:
 model.load("model.tflearn")
except:
model.fit(training, output, n_epoch=1000, batch_size=8,
show_metric=True)
model.save("model.tflearn")
To start using the chat bot, we can add two functions to the script as shown
below.
def bag_of_words(s, words):
bag = [0 for _ in range(len(words))]

s_words = nltk.word_tokenize(s)
s_words = [stemmer.stem(word.lower()) for word in s_words]

for se in s_words:
for i, w in enumerate(words):
if w == se:
bag[i] = 1

return numpy.array(bag)

def chat():
print("Start talking with the bot (type quit to stop)!")
while True:
inp = input("You: ")
if inp.lower() == "quit":
 break

results = model.predict([bag_of_words(inp, words)])

results_index = numpy.argmax(results)
tag = labels[results_index]

for tg in data["intents"]:
if tg['tag'] == tag:
responses = tg['responses']

print(random.choice(responses))

chat()
 Conclusion
I hope you enjoyed this book. We started from the basics of Python and
worked towards implementing some pretty serious machine-learning
concepts. I also hope that after reading this book, you feel content with my
efforts and are motivated towards exploring more about machine learning on
your own.
There are various resources available online that you can follow to further
your journey in this field. You will find several good YouTube channels with
in-depth tutorials on machine learning.
No matter how you proceed now, keep this book handy because it will help
you find answers quickly. One of the problems learning with online resources
is that you will quickly forget where you saw something particular and waste
a lot of time searching for it. In all honesty, bookmarks are still a thing that
make books better.

Where Will Machine Learning Be in the Next 20 Years?

Machine learning is already penetrating every walk of life, and the trend will
continue to grow unless there’s a world-wide catastrophe that completely
stops the internet. Since someone remarked, “Data is the new oil,” industry
experts have argued how valuable user data truly is for businesses and why
there's a need to give everyone the ability to monetize access to their data.
The social media platform you joined for free just to connect with your
friends years ago now shows you targeted ads, which the advertisement
company is heavily charged, and on top of that, sells your data to data
aggregators. Do you think the transaction is fair?
There is definitely a need to monitor and control what data companies can
access and for what reason. The European (General Data Protection
Regulation) GDPR is a very good initiative in this regard. I hope more
countries follow suit and adapt GDPR to their local internet industry. Too
much good stuff is also bad for you!
But, the future is not gloom and doom. Machine learning is already making
lives easier and as more industries become aware of its capabilities, it will
become easier to gradually optimize their services. Consider a fully capable
artificially intelligent robot providing medical services in places where there's
an extreme shortage of doctors and health facilities. Imagine a house that’s
aware of your needs and correspondingly prepares everyday like clockwork
without any manual adjustments. The future feels good!
 References
Cortez, P. and Silva A. Using Data Mining to Predict Secondary School
Student Performance. In A. Brito and J. Teixeira Eds., Proceedings of 5th
FUture BUsiness TEChnology Conference (FUBUTEC 2008) pp. 5-12,
Porto, Portugal, April, 2008, EUROSIS, ISBN 978-9077381-39-7
Geron, Aurelien. Hands-On Machine Learning with Scikit-Learn &
TensorFlow. 2d. ed. 1005 Gravenstein Highway North, Sebastopol, CA
95472: O’Reilly Publication, 2016
Rashka, Sebastian. Python Machine Learning. 1st. ed. 35 Livery Street,
Birmingham B3 2PB, UK: Packt Publishing Ltd., 2015
 Appendix A: Machine-Learning Concepts
This will be a brief chapter where we will explore the theoretical background
of various machine-learning concepts that you will encounter from time to
time.
Data Scrubbing
A process that runs in the background to fix errors in the main memory
through periodical inspections is called data scrubbing. The errors are fixed
with redundant data, such as duplicating existing data or using checksums.
Data scrubbing removes varied and unique error instances with the same
error that makes it easier to filter and fix.
In data analytics and machine learning, we know that data integrity is vital to
get correct results from the implementation of an algorithm. Whenever data is
accessed from an unreliable source, it should first be error-checked before
any action is performed on it. Data is inherently dirty (having errors) and has
a tendency of fallback to dirtiness even after cleaning. Incorrect data can lead
to wrong decisions that in turn can cause huge business losses. Most
companies that utilize data analysis and machine learning in Customer
Relationship Management (CRM), marketing tools and Business Intelligence
(BI) software cite dealing with inconsistent data as the biggest issue. It’s like
putting low-grade gas in your car. It might sound better due to cheaper rates
and no immediate signs of significant decrease in performance. But, in the
long run, it’s going to ruin the car’s engine.
The quality of data used for analysis and predictions is the difference between
successful and failing businesses.
But, how you would error check without knowing if some data is correct or
not? You have to find inconsistencies in data or check for the possibility of
missing data. All computer systems have some sort of error checking so these
errors don’t affect the output. Data analysts and scientists have to take care of
data scrubbing themselves.
The future belongs to the Internet of Things (IoT) where all the systems will
be connected to provide seamless functionality. Every system produces a
mammoth amount of data and joining them together to create a supersystem
would mean every subsystem should be able to understand data generated by
another subsystem. Standardizing data production and transmission will be
necessary along with implementation of highly precise data scrubbing
methods.
In the long term, data scrubbing helps in determining the cause of errors that
must be fixed in order for the system to improve its performance. Even after
the errors are removed, data scrubbing must not be removed from the system.
Using machine learning with data scrubbing results in a self-learning
completely automatic error detection and correction system that can even
predict when an error will occur by analyzing system parameters. For
example, you have written a code that continuously downloads data from a
statistical website. The data scrubbing module actively checks the data to
ensure there’s no missing or erroneous data. The internet connection goes
down that leads to missing data (nulls and NaNs) being fed to the system.
The data scrubbing module checks for possible reasons and learns that it’s
due to a faulty internet connection. Now, whenever our script will start
getting missing data, the data scrubbing module will check the internet
connection. If it’s down, it will try to reconnect, and if that fails, it will pause
data import and notify the user through available methods of the problem.
This way, the data scrubbing module will keep self-learning whenever a new
error is met. Lastly, here’s a data scrubbing checklist for quick reference.
● Identify frequent sources of error and create a pattern. The sources can
be anything from bad internet connection to incorrect data import
settings.
● Check for all common data issues such as finding duplicates and
missing data. Investigate data with respect to the data standardization
rules to identify inconsistencies.
● Clean the data from all errors and inconsistencies.
● Update the data and system regularly, so they are always working with
the latest tools and standards.
● Secure access to the data and the system by deploying stringent data
security processes including staff training.
● Comply with all the regulatory laws and regulations governing the data.
An example is full compliance with GDPR for your website data.
● Enrich user experience by using high quality data that allows for better
user insights.
Neural Network
A set of algorithms created to mimic the working of a human mind is called
neural networks. Humans deduce outcomes using patterns and neural
networks also aim to use the same technique to reach conclusions.
Neural networks are chiefly used in classification and clustering. Neural
networks can be combined with other machine learning algorithms to predict
behavior of complex systems. Neural networks are made of numerous layers,
each can have different properties (arguments). Each layer in itself is made
from nodes, which are called neurons. The layers act as input and output of
the neural network. The neurons act as binary diodes that either switch on or
off in response to an input.
The layer structure helps neural networks to find relationship between data
attributes that isn’t possible with other single-layer predictive algorithms. The
input data goes through each layer just like a car goes through an automated
car washing facility.
Neural networks form the basis of deep learning because generally neural
network algorithms deploy three or more layers. The input of each layer is
the output of the previous layer (except the first layer that takes the raw data).
Many machine-learning experts consider neural networks as makeshift AI
because the layers act in a sequence on the data to reach a conclusion. Neural
networks are great at predicting the working of a complex system but fail to
establish the actual relationship between the input and the output.
Decision Trees
Decision trees contain a series of nodes much like a flowchart that the user
can use to predict the probability of an event by following the node path.
Decision trees are easier to understand for non-technical personnel. They are
not greatly affected by missing data or outliers, which means less time and
resources are needed to clean the data. There are some disadvantages as well.
One of the biggest disadvantages of using decision trees is overfitting.
Decision trees are better for systems that have non-linear relationships
between its attributes. The algorithms mostly rely on supervised machine
learning. Each internal node acts corresponds to a system attribute and each
leaf (external) node corresponds to a classification label. Decision trees are
also very similar nested sets of “if”, “then”, and “else” statements which
makes them easier to code. There are two types of decision trees.
 1. Continuous variable decision tree: a decision tree that has a continuous
target variable (regression algorithms)
2. Categorical variable decision tree: a decision tree where the target
variable has specific categories (classification algorithms)
If the system data has high variance (very frequent data changes in the data),
decision trees might be difficult but not impossible to implement. Decision
trees are the algorithm version of Julius Caesar’s famous quote “divide and
conquer.” The “sklearn” library provides a module “tree” that can be used to
create and work with decision trees in Python. Decision trees are inverse of
natural trees in the perspective of location of roots and leaves. In a decision
tree, the leaves (terminal nodes) lie at the bottom while roots are at the top. It
also means a decision tree has a top-to-bottom hierarchy.
Algorithm Selection
You have read about and coded various predictive algorithms throughout this
book. You might be wondering which algorithm is best at predicting a
situation. The answer is, it depends on a lot of factors. There is no “one for all
situations” algorithm. As a data scientist, it will be your responsibility to
analyze the system data and behavior and choose the best model you think
most closely represent a given system. You will have to find the relationship
between system attributes, usually it’s done by plotting a trend of different
system attributes. Once, the relationship is estimated, the correct model is
picked along with the algorithm ideal for the specific system attribute
relationship.
It is important to understand that you can misinterpret the data and pick the
wrong model, leading to an algorithm implementation that will not give
accurate and precise predictions. A good data scientist will always keep this
in mind and rigorously analyze the system attribute data from different
perspectives before moving to pick a predictive algorithm. To automate this
process, the data scientist might train different models using the data set and
compare prediction results to choose one of them.
Data Mining
The process of inspecting large data sets and finding patterns by combining
techniques of machine learning, database systems, and statistics is called data
mining. It is very useful to detect patterns in data that might not be easily
apparent. When we say data mining, we are actually not mining data, but
mining what the data represents.
The process of data mining has usually six parts (these processes are not
sequential).
1. Anomaly detection: The process of finding an outlier, deviation, spike,
drop, or a huge change from the majority of data points that might be
errors or interesting instances to investigate.
2. Dependency modeling: Also known as association rule learning, the
purpose is to find relationship between different attributes of a system.
3. Clustering: The procedure of finding groups or clusters of similar data
points in a system is called clustering without prior knowledge of data.
4. Classification: The process of labeling or generalizing new data
according to preset data structure is called classification. We have
already seen this in action in our book.
5. Regression: The process of finding a function model that fits the
entirety or majority of the data set. This helps in dependency modeling.
6. Summarization: This process provides a concise data set representation
through many methods including data visualization and report
generation.
Data mining can be used wherever you can find digital data. This has led to
several cases of data misuse. It is the responsibility of the data
accumulators/aggregators to make sure the data collected is only available for
the purposes the data is collected for. Implementation of stricter rules is a big
problem, but the EU is way ahead of North America in this respect.
Unfortunately, there will be no way to completely stop data misuse in the
future no matter how strict rules are implemented. This is one of the biggest
challenges in Internet of Things (IoT).