Jamhuriya University of Science & Technology (JUST)
CA416 - Principles of Data Science
Chapter 5
Practical Data Exploration and Visualization with
Pandas and Matplotlib packages.
Lecturer: XYZ
1
Jamhuriya University of Science & Technology (JUST)
Learning outcomes
By the end of this lecture, you will be able to:
Describe some core data analysis concepts including
dataframes, and data exploration.
Create and access main data structures in Python,
such as series and dataframes.
Perform exploratory data analysis in Python usig
Pandas library.
Build data visualizations with Matplotlib library
2
Jamhuriya University of Science & Technology (JUST)
Data science workflow – recap
Source:
https://www.dataquest.io/bl
og/what-is-data-science/ 3
Jamhuriya University of Science & Technology (JUST)
Types of variables
A variable is any characteristic or attribute that can be
quantitatively and qualitatively been measured
Variable
Numeric Categorical
Continuous Discrete Ordinal Nominal
Continuous – time, age, etc Ordinal – grades, rating, etc
Discrete – people, houses, etc Nominal – nationality, race, etc
4
Jamhuriya University of Science & Technology (JUST)
Exploratory data analysis
Exploratory Data Analysis (EDA) is an initial exploration of
the data to understand its characteristics, patterns,
correlations, and to identify any anomalies in the data.
Statistical data summarization and graphical visualizations
are two primary forms of EDA.
EDA is typically applied before formal data modeling and
helps inform the development of appropriate statistical models
or machine learning models.
5
Jamhuriya University of Science & Technology (JUST)
Pandas
Data analysis is normally performed over data stored in a
tabular format, e.g., Excel spreadsheet.
Each observation is recorded in a row and its each
attribute is recorded in a column (e.g., students’ and
their grades in each assignment)
Pandas is a Python library for manipulating data in tabular
format and comes with Python Anaconda.
In Pandas, data manipulation can be much more varied, it can
programmed more easily and performed more efficiently,
which is critical in large-scale projects
6
Jamhuriya University of Science & Technology (JUST)
Pandas
• Pandas is a high-level library built on NumPy, providing
tools that make it easier to work with real-world data:
– load data from a variety of sources (e.g. CSV, JSON, SQL).
– update (add, modify, delete etc) data
– select subsets of the data
– group data by a certain criterion
– clean and handling missing values or NANs
– visualize the data using different plotting tools
– perform statistical analysis of the data and
– export the data to other file formats or databases
• Pandas provides two main data structures, namely,
DataFrame (equivalent to a spreadsheet) and a Series
(equivalent to a column in a spreadsheet).
7
Jamhuriya University of Science & Technology (JUST)
Pandas Series
A Series is just a column is a dataframe or spreadsheet
Pandas series with indices
is created the list
the series attributes can
also be separately
accessed
Noice that the values
are just NumPy arrays 8
Jamhuriya University of Science & Technology (JUST)
Operations on series data
Please read the Pandas
documentation for detailed list of
statistical functions applicable on
series, as given in the additional
resource slide.
9
Jamhuriya University of Science & Technology (JUST)
Pandas Dataframe
A Pandas dataframe is a collection of Series ( a 2D data
structure with row & columns - effectively a spreadsheet )
Let's create an example dataframe with the population and area
values for some regions in Somalia
The head() returns the first few
rows of the dataframe. You can use
tail() to see the last few rows
The dataframe is just like
spreadsheet sheet with indices (as
the series) 10
Jamhuriya University of Science & Technology (JUST)
Indexing the dataframe records
Since we did not supply any particular values as the index,
a range of integers was used as the index for the dataframe
However, it may be convenient to set the region names as the
indices for our example dataframe.
The argument inplace=True is
an instruction to modify the
dataframe and as the result the
dataframe will now have two
columns
Region names are now used as the
dataframe indices.
11
Jamhuriya University of Science & Technology (JUST)
Hands-on exercise 1
We can create dataframes by supplying dictionaries with
identical sets of keys as arguments to DataFrame()
Can you represent each column (population and area)
as
dictionaries and create a dataframe from them ?
12
Jamhuriya University of Science & Technology (JUST)
Dataframe attributes
Using these attributes, one can
separately access df information
This shows that dataframe values are
just 2D NumPy arrays, and this is
why NumPy underpins Pandas and other
data science libraries
13
Jamhuriya University of Science & Technology (JUST)
Descriptive statistics
Average population and
area for all regions are
6.931368e+05 and
41079.333333 in order.
These statistics include
mean, quartiles, median,
total observations etc.
14
Jamhuriya University of Science & Technology (JUST)
Descriptive statistics
The info() method
provides a concise
description of the
dataframe
The shape() method
provides the shape of
the dataframe in terms
of the number of rows
and columns
15
Jamhuriya University of Science & Technology (JUST)
Matplotlib for plotting a dataframe
• The matplotlib is comprehensive package for
data visualization, and comes as part of Anaconda.
Pandas has a convenient integration with matplotlib, which
means that data contained in a dataframe can be plotted with
plot():
• You can select the plot type of your choice ( e.g., scatter, bar,
boxplot, pie, hist, …) corresponding to your data
• Please see the resource at the end for more information on
various plots and arguments of the plot() function
16
Jamhuriya University of Science & Technology (JUST)
Matplotlib for plotting a dataframe
• Let us now plot the data contained in our dataframe, df, by
simply calling its plot method:
Logy=True argument
enables us to scale the
y values
logarithmically.
Otherwise the scales
could have been very
different
Bar plots are useful
tools for viewing
categorical
17
variables
Jamhuriya University of Science & Technology (JUST)
Selecting dataframe columns
• Let us extract the area column/variable from our data
frame
The extracted columns are Pandas series type and can stored in a
different variable or processed separately 18
Jamhuriya University of Science & Technology (JUST)
Selecting dataframe cells
• We also extract a cell value of a dataframe
Notice that the cell is
accessed by its column
name and row index.
The index can be a
number and its format is
to put it in square
brakcets []
You can use the same
syntax to update the cell,
e.g. change the number
Again the extracted cell values can be separately processed.
19
Jamhuriya University of Science & Technology (JUST)
Slicing the dataframe
The iloc attribute is used
to access the rows and
columns by their integer
indices:
‘:’ means extract all –
columns (also rows)
The loc attribute is used
to access the rows and
columns by their string
indices: 20
Jamhuriya University of Science & Technology (JUST)
Adding columns to a dataframe
Remember this a
vectorised or element
wise math operation just
like NumPy arrays.
‘density’ columns is now
created and added to the
dataframe
Obviously Banaadir has the
highest density 21
Jamhuriya University of Science & Technology (JUST)
Adding columns to a dataframe
The backward slash ‘\’
enables the continuation
of the list definition.
A new column ‘capital’
is now created and
added to the dataframe
22
Jamhuriya University of Science & Technology (JUST)
Adding rows to a dataframe
This now adds a new
row with index
‘M_Shabelle’ to the
dataframe
23
Jamhuriya University of Science & Technology (JUST)
Conditional data selection
The selected data is a
dataframe itself.
Such conditional
extraction can be
applied to any other
dataframe column
24
Jamhuriya University of Science & Technology (JUST)
Conditional updating
This populates
the entire new
column with the
single value
‘low’
2525
Jamhuriya University of Science & Technology (JUST)
Conditional updating
The first index in
loc specifies the
rows to which the
change applies
and the second
argument
specifies the
column
26
Jamhuriya University of Science & Technology (JUST)
Conditional updating
We can also use the apply() function to apply some
operation to every row or every column in a dataframe.
The function takes a custom function as an argument, the
custom function takes either a row or a column at a time and
can return a modified row or column:
2727
Jamhuriya University of Science & Technology (JUST)
Conditional updating
The axis
argument
indicates
whether to
process the
dataframe by
columns (1)
or rows (0)
28
Jamhuriya University of Science & Technology (JUST)
Deleting dataframe columns
The drop() method can be
used to remove rows or
columns depending on the
axis we specifiy and
column/row we name
From this output, we can
see that density_status
column is now removed.
29
Jamhuriya University of Science & Technology (JUST)
Deleting dataframe rows
Like with columns, using
the inplace=True
argument means we are
updating the dataframe
and as the result the
dataframe will now have
fewer records or rows.
From this output, we can
see that M_Shabelle row is
now removed.
30
Jamhuriya University of Science & Technology (JUST)
Exploring categorical variables
The unique() function
returns the unique
values of the colum
The value_counts() method
returns the requicy of each
The value_counts() unique value
method returns a series
31
Jamhuriya University of Science & Technology (JUST)
Exploring numerical variables
The unique() function
returns the unique
values of the colum
The value_counts() method
returns the requicy of each
The value_counts() unique value
method returns a series
32
Jamhuriya University of Science & Technology (JUST)
Visualizing numerical variables
This plot shows that 5
regions has a population
ranging from about 375K
to approx.
620k
Histograms are useful
tools for viewing the
distribution of
variables
33
Jamhuriya University of Science & Technology (JUST)
Visualizing numerical variables
Maximum
Q3
Median IQR
Q1
Minimum
Boxplots are used the represent summary statistics (5 number
summary) and to compare summary of different datasets
The two variable or columns could be drawn on the same plot but they
have been plotted separately since their scales differ, 34
Jamhuriya University of Science & Technology (JUST)
Visualizing numerical variables
The rot and fontsize
arguments are used
here to rotate and
size the x labels.
Line plots are
primarily used for
viewing continuous
variables.
35
Jamhuriya University of Science & Technology (JUST)
Visualizing categorical variables
plt is an alias for
matplotlib Pyplot
module. The loc
argument to the legend
function sets the
location of the legend
Pie charts are primarily
using for visualizing
proportions of mostly
categorical variables
36
Jamhuriya University of Science & Technology (JUST)
Hands-on exercise 2
Suppose you have this data in a dictionary:
exam_data =
{
'name': ['Ali', 'Ahmed', 'Jama', 'Omar', 'Fatima', 'Mohamed',
'Mohamud', 'Malin', 'Farah', 'Samad'],
'score': [62.5, 79, 16.5, 65, 53, 81, 58, 45, 72, 66.5], 'attempts':
[1, 3,
2, 3, 2, 3, 1, 1, 2, 1],
'qualify': ['yes', 'no', 'yes', 'no', 'no', 'yes', 'yes', 'yes', 'no', 'yes']
}
Create a dataframe from this data and retrieve the following subsets of data:
1. The first three rows
2. The following three rows
3. The score for 'Mohamed'
4. The scores of all students who qualify and who made just one
attempt.
37
Jamhuriya University of Science & Technology (JUST)
References & reading resources
Data Analysis with Pandas
• https://pandas.pydata.org/docs/getting_started/
• https://pandas.pydata.org/pandas-docs/stable/index.html
• https://www.youtube.com/watch?v=5JnMutdy6Fw
• https://pandas.pydata.org/pandas-
docs/stable/reference/api/pandas.DataFrame.plot.html
• Chapter 3, Python Data Science Handbook, Jake
VanderPlas
• Chapter 5-6, Python for Data Analysis, Wes
McKinney.
Matplotlib
• https://matplotlib.org/tutorials/index.html
38
Jamhuriya University of Science & Technology (JUST)
39
