Complete Download Python Data Visualization Cookbook Milovanović PDF All Chapters
Complete Download Python Data Visualization Cookbook Milovanović PDF All Chapters
com
https://ebookname.com/product/python-data-visualization-
cookbook-milovanovic/
OR CLICK BUTTON
DOWNLOAD EBOOK
https://ebookname.com/product/data-visualization-with-python-and-
javascript-scrape-clean-explore-transform-your-data-1st-edition-
kyran-dale/
https://ebookname.com/product/python-cookbook-2nd-ed-edition-
beazley/
https://ebookname.com/product/python-machine-learning-
cookbook-1st-edition-joshi/
https://ebookname.com/product/misreading-plato-1st-edition-
matthew-clemente/
Research Design in Counseling 3rd Edition P. Paul
Heppner
https://ebookname.com/product/research-design-in-counseling-3rd-
edition-p-paul-heppner/
https://ebookname.com/product/artificial-intelligence-approaches-
tools-and-applications-approaches-tools-and-applications-1st-
edition-brent-m-gordon/
https://ebookname.com/product/the-christian-hope-brian-
hebblethwaite/
https://ebookname.com/product/plant-user-specification-guide-1st-
edition-james-hitchmough/
https://ebookname.com/product/special-edition-using-
wordperfect-12-laura-acklen/
Weapons Warfare Ancient Medieval John Powell
https://ebookname.com/product/weapons-warfare-ancient-medieval-
john-powell/
www.it-ebooks.info
Python Data
Visualization Cookbook
Igor Milovanović
BIRMINGHAM - MUMBAI
www.it-ebooks.info
Python Data Visualization Cookbook
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or
transmitted in any form or by any means, without the prior written permission of the publisher,
except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the
information presented. However, the information contained in this book is sold without
warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers
and distributors will be held liable for any damages caused or alleged to be caused directly or
indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies
and products mentioned in this book by the appropriate use of capitals. However, Packt
Publishing cannot guarantee the accuracy of this information.
ISBN 978-1-78216-336-7
www.packtpub.com
www.it-ebooks.info
Credits
Reviewers Proofreaders
Tarek Amr Amy Johnson
Simeone Franklin Lindsey Thomas
Jayesh K. Gupta
Indexer
Kostiantyn Kucher
Mariammal Chettiyar
Kenneth Emeka Odoh
Graphics
Acquisition Editor Abhinash Sahu
James Jones
Production Coordinator
Lead Technical Editor Shantanu Zagade
Ankita Shashi
Cover Work
Technical Editors Shantanu Zagade
Pratik More
Amit Ramadas
Ritika Singh
Copy Editors
Brandt D'Mello
Janbal Dharmaraj
Deepa Nambiar
Kirti Pai
Laxmi Subramanian
www.it-ebooks.info
About the Author
He is an Evangelist for high-quality systems design who holds strong interests in software
architecture and development methodologies. He is always persistent on advocating
methodologies that promote high-quality software, such as test-driven development,
one-step builds, and continuous integration.
He also possesses solid knowledge of product development. Having field experience and
official training, he is capable of transferring knowledge and communication flow from
business to developers and vice versa.
www.it-ebooks.info
About the Reviewers
Tarek Amr achieved his postgraduate degree in Data Mining and Information Retrieval from
the University of East Anglia. He has about 10 years' experience in Software Development.
He has been volunteering in Global Voices Online (GVO) since 2007, and currently he is the
local ambassador of the Open Knowledge Foundation (OKFN) in Egypt. Words such as Open
Data, Government 2.0, Data Visualisation, Data Journalism, Machine Learning, and Natural
Language Processing are like music to his ears.
Jayesh K. Gupta is the Lead Developer of Matlab Toolbox for Biclustering Analysis (MTBA).
He is currently an undergraduate student and researcher at IIT Kanpur. His interests lie in the
field of pattern recognition. His interests also lie in basic sciences, recognizing them as the
means of analyzing patterns in nature. Coming to IIT, he realized how this analysis is being
augmented by Machine Learning algorithms with various diverse applications. He believes
that augmenting human thought with machine intelligence is one of the best ways to advance
human knowledge. He is a long time technophile and a free-software Evangelist. He usually
goes by the handle, rejuvyesh online. He is also an avid reader and his books can be checked
out at Goodreads. Checkout his projects at Bitbucket and GitHub. For all links visit http://
home.iitk.ac.in/~jayeshkg/. He can be contacted at [email protected].
www.it-ebooks.info
Kostiantyn Kucher was born in Odessa, Ukraine. He received his Master's degree in
Computer Science from Odessa National Polytechnic University in 2012. He used Python
as well as Matplotlib and PIL for Machine Learning and Image Recognition purposes.
Kenneth Emeka Odoh performs research on state of the art Data Visualization
techniques. His research interest includes exploratory search where the users are
guided to their search results using visual clues.
When Kenneth is not writing source codes, you can find him singing at the Campion College
chant choir.
www.it-ebooks.info
www.PacktPub.com
Did you know that Packt offers eBook versions of every book published, with PDF and ePub
files available? You can upgrade to the eBook version at www.PacktPub.com and as a print
book customer, you are entitled to a discount on the eBook copy. Get in touch with us at
[email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up
for a range of free newsletters and receive exclusive discounts and offers on Packt books
and eBooks.
TM
http://PacktLib.PacktPub.com
Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book
library. Here, you can access, read and search across Packt's entire library of books.
Why Subscribe?
ff Fully searchable across every book published by Packt
www.it-ebooks.info
www.it-ebooks.info
Table of Contents
Preface 1
Chapter 1: Preparing Your Working Environment 5
Introduction 5
Installing matplotlib, NumPy, and SciPy 6
Installing virtualenv and virtualenvwrapper 8
Installing matplotlib on Mac OS X 10
Installing matplotlib on Windows 11
Installing Python Imaging Library (PIL) for image processing 12
Installing a requests module 14
Customizing matplotlib's parameters in code 14
Customizing matplotlib's parameters per project 16
Chapter 2: Knowing Your Data 19
Introduction 19
Importing data from CSV 20
Importing data from Microsoft Excel files 22
Importing data from fixed-width datafiles 25
Importing data from tab-delimited files 27
Importing data from a JSON resource 28
Exporting data to JSON, CSV, and Excel 31
Importing data from a database 36
Cleaning up data from outliers 40
Reading files in chunks 46
Reading streaming data sources 48
Importing image data into NumPy arrays 50
Generating controlled random datasets 56
Smoothing the noise in real-world data 64
www.it-ebooks.info
Table of Contents
Chapter 3: Drawing Your First Plots and Customizing Them 71
Introduction 72
Defining plot types – bar, line, and stacked charts 72
Drawing a simple sine and cosine plot 78
Defining axis lengths and limits 81
Defining plot line styles, properties, and format strings 84
Setting ticks, labels, and grids 89
Adding a legend and annotations 92
Moving spines to the center 95
Making histograms 96
Making bar charts with error bars 99
Making pie charts count 101
Plotting with filled areas 103
Drawing scatter plots with colored markers 105
Chapter 4: More Plots and Customizations 109
Introduction 109
Setting the transparency and size of axis labels 110
Adding a shadow to the chart line 113
Adding a data table to the figure 116
Using subplots 118
Customizing grids 121
Creating contour plots 125
Filling an under-plot area 128
Drawing polar plots 131
Visualizing the filesystem tree using a polar bar 134
Chapter 5: Making 3D Visualizations 139
Introduction 139
Creating 3D bars 139
Creating 3D histograms 143
Animating in matplotlib 146
Animating with OpenGL 150
ii
www.it-ebooks.info
Table of Contents
iii
www.it-ebooks.info
Table of Contents
iv
www.it-ebooks.info
Preface
The best data is the data that we can see and understand. As developers, we want to
create and build the most comprehensive and understandable visualizations. It is not always
simple; we need to find the data, read it, clean it, massage it, and then use the right tool to
visualize it. This book explains the process of how to read, clean, and visualize the data into
information with straight and simple (and not so simple) recipes.
How to read local data, remote data, CSV, JSON, and data from relational databases are all
explained in this book.
Some simple plots can be plotted with a simple one-liner in Python using matplotlib, but
doing more advanced charting requires knowledge of more than just Python. We need to
understand the information theory and human perception aesthetics to produce the most
appealing visualizations.
This book will explain some practices behind plotting with matplotlib in Python, statistics used,
and usage examples for different charting features we should use in an optimal way.
This book is written and the code is developed on Ubuntu 12.03 using Python 2.7,
IPython 0.13.2, virtualenv 1.9.1, matplotlib 1.2.1, NumPy 1.7.1, and SciPy 0.11.0.
Chapter 2, Knowing Your Data, introduces you to common data formats and how to read and
write them, be it CSV, JSON, XSL, or relational databases.
Chapter 3, Drawing Your First Plots and Customizing Them, starts with drawing simple plots
and covers some of the customization.
Chapter 4, More Plots and Customizations, follows up from previous chapter and covers more
advanced charts and grid customization.
www.it-ebooks.info
Preface
Chapter 6, Plotting Charts with Images and Maps, covers image processing, projecting data
onto maps, and creating CAPTCHA test images.
Chapter 7, Using Right Plots to Understand Data, covers explanations and recipes on some
more advanced plotting techniques such as spectrograms and correlations.
Chapter 8, More on matplotlib Gems, covers a set of charts such as Gantt charts, box plots,
and whisker plots, and also explains how to use LaTeX for rendering text in matplotlib.
Other software packages used in this book are IPython, which is an interactive Python
environment that is very powerful, and flexible. This can be installed using package
managers for Linux-based OSes or prepared installers for Windows and Mac OSes.
If you are new to Python installation and software installation in general, it is very much
recommended to use prepackaged scientific Python distributions such as Anaconda,
Enthought Python Distribution, or Python(X,Y).
Other required software mainly comprises of Python packages that are all installed using
the Python installation manager, pip, which itself is installed using Python's easy_install
setup tool.
You will need to know some general programming concepts, and any kind of programming
experience will be helpful. However, the code in this book is explained almost line by line. You
don't need math for this book; every concept that is introduced is thoroughly explained in plain
English, and references are available for further interest in the topic.
Conventions
In this book, you will find a number of styles of text that distinguish between different kinds of
information. Here are some examples of these styles, and an explanation of their meaning.
www.it-ebooks.info
Preface
Code words in text are shown as follows: "We packed our little demo in class DemoPIL,
so that we can extend it easily, while sharing the common code around the demo function,
run_fixed_filters_demo."
When we wish to draw your attention to a particular part of a code block, the relevant lines or
items are set in bold:
# tidy up tick labels size
all_axes = plt.gcf().axes
for ax in all_axes:
for ticklabel in ax.get_xticklabels() + ax.get_yticklabels():
ticklabel.set_fontsize(10)
New terms and important words are shown in bold. Words that you see on the screen, in
menus or dialog boxes for example, appear in the text like this: "We then set up a label for
the stem plot and the position of baseline, which defaults to 0."
Reader feedback
Feedback from our readers is always welcome. Let us know what you think about this
book—what you liked or may have disliked. Reader feedback is important for us to
develop titles that you really get the most out of.
www.it-ebooks.info
Preface
If there is a topic that you have expertise in and you are interested in either writing or
contributing to a book, see our author guide on www.packtpub.com/authors.
Customer support
Now that you are the proud owner of a Packt book, we have a number of things to help you to
get the most from your purchase.
Errata
Although we have taken every care to ensure the accuracy of our content, mistakes do happen.
If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be
grateful if you would report this to us. By doing so, you can save other readers from frustration
and help us improve subsequent versions of this book. If you find any errata, please report them
by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on
the errata submission form link, and entering the details of your errata. Once your errata are
verified, your submission will be accepted and the errata will be uploaded on our website, or
added to any list of existing errata, under the Errata section of that title. Any existing errata can
be viewed by selecting your title from http://www.packtpub.com/support.
Piracy
Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt,
we take the protection of our copyright and licenses very seriously. If you come across any
illegal copies of our works, in any form, on the Internet, please provide us with the location
address or website name immediately so that we can pursue a remedy.
We appreciate your help in protecting our authors, and our ability to bring you valuable content.
Questions
You can contact us at [email protected] if you are having a problem with any
aspect of the book, and we will do our best to address it.
www.it-ebooks.info
Preparing Your Working
1
Environment
In this chapter, we will cover the following recipes:
Introduction
This chapter introduces the reader to the essential tooling and installation and configuration
of them. This is a necessary work and common base for the rest of the book. If you have never
used Python for data and image processing and visualization, it is advised not to skip this
chapter. Even if you do skip it, you can always return to this chapter in case you need to
install some supporting tool or verify what version you need to support the current solution.
www.it-ebooks.info
Preparing Your Working Environment
Getting ready
We assume that you already have Linux (preferably Debian/Ubuntu or RedHat/SciLinux)
installed and Python installed on it. Usually, Python is already installed on the mentioned
Linux distributions and, if not, it is easily installable through standard means. We assume
that Python 2.7+ Version is installed on your workstation.
Almost all code should work with Python 3.3+ Versions, but because most
operating systems still deliver Python 2.7 (some even Python 2.6) we decided
to write the Python 2.7 Version code. The differences are small, mainly in
version of packages and some code (xrange should be substituted with range
in Python 3.3+).
We also assume that you know how to use your OS package manager in order to install
software packages and know how to use a terminal.
matplotlib requires NumPy, libpng, and freetype as build dependencies. In order to be able to
build matplotlib from source, we must have installed NumPy. Here's how to do it:
Install NumPy (at least 1.4+, or 1.5+ if you want to use it with Python 3) from
http://www.numpy.org/.
NumPy will provide us with data structures and mathematical functions for
using it with large datasets. Python's default data structures such as tuples,
lists, or dictionaries are great for insertions, deletions, and concatenation.
NumPy's data structures support "vectorized" operations and are very efficient
for use and for executions. They are implemented with Big Data in mind and
rely on C implementations that allow efficient execution time.
SciPy, building on top of NumPy, is the de facto standard's scientific and
numeric toolkit for Python comprising great selection of special functions and
algorithms, most of them actually implemented in C and Fortran, coming from
the well-known Netlib repository (see http://www.netlib.org).
www.it-ebooks.info
Chapter 1
If you are using RedHat or variation of this distribution (Fedora, SciLinux, or CentOS)
you can use yum to perform same installation:
$ su -c 'yum-builddep python-matplotlib'
How to do it...
There are many ways one can install matplotlib and its dependencies: from source, from
precompiled binaries, from OS package manager, and with prepackaged python distributions
with built-in matplotlib.
Most probably the easiest way is to use your distribution's package manager. For Ubuntu that
should be:
# in your terminal, type:
$ sudo apt-get install python-numpy python-matplotlib python-scipy
If you want to be on the bleeding edge, the best option is to install from source. This path
comprises a few steps: Get the source, build requirements, and configure, compile,
and install.
Download the latest source from code host www.github.com by following these steps:
$ cd ~/Downloads/
$ wget https://github.com/downloads/matplotlib/matplotlib/matplotlib-
1.2.0.tar.gz
$ tar xzf matplotlib-1.2.0.tar.gz
$ cd matplotlib-1.2.0
$ python setup.py build
$ sudo python setup.py install
www.it-ebooks.info
Preparing Your Working Environment
How it works...
We use standard Python Distribution Utilities, known as Distutils, to install matplotlib from
source code. This procedure requires us to previously install dependencies, as we already
explained in the Getting ready section of this recipe. The dependencies are installed using
the standard Linux packaging tools.
There's more...
There are more optional packages that you might want to install depending on what your data
visualization projects are about.
No matter what project you are working on, we recommend installing IPython—an Interactive
Python shell that supports PyLab mode where you already have matplotlib and related
packages, such as NumPy and SciPy, imported and ready to play with! Please refer to
IPython's official site on how to install it and use it—it is, though, very straightforward.
virtualenv is an open source project started by Ian Bicking that enables a developer to isolate
working environments per project, for easier maintenance of different package versions.
For example, you inherited legacy Django website based on Django 1.1 and Python 2.3, but at
the same time you are working on a new project that must be written in Python 2.6. This is my
usual case—having more than one required Python version (and related packages) depending
on the project I am working on.
www.it-ebooks.info
Chapter 1
virtualenv enables me to easily switch to different environments and have the same package
easily reproduced if I need to switch to another machine or to deploy software to a production
server (or to a client's workstation).
Getting ready
To install virtualenv, you must have workable installation of Python and pip. Pip is a tool for
installing and managing Python packages, and it is a replacement for easy install. We will
use pip through most of this book for package management. Pip is easily installed, as root
executes the following line in your terminal:
# easy_install pip
virtualenv by itself is really useful, but with the help of virtualenvwrapper, all this becomes
easy to do and also easy to organize many virtual environments. See all the features at
http://virtualenvwrapper.readthedocs.org/en/latest/#features.
How to do it...
By performing the following steps you can install the virtualenv and virtualenvwrapper tools:
3. You will probably want to add the following line to your ~/.bashrc file:
source /usr/loca/bin/virtualenvwrapper.sh
ff mkvirtualenv ENV: This creates virtual environment with name ENV and
activates it
ff workon ENV: This activates the previously created ENV
ff deactivate: This gets us out of the current virtual environment
www.it-ebooks.info
Preparing Your Working Environment
In case you are not satisfied with EPD or cannot use it for other reasons such as versions
distributed with it, there is a manual (read: harder) way of installing Python, matplotlib,
and its dependencies.
Getting ready
We will use the Homebrew project that eases installation of all software that Apple did not
install on your OS, including Python and matplotlib. Under the hood, Homebrew is a set of
Ruby and Git that automate download and installation. Following these instructions should
get the installation working. First, we will install Homebrew, and then Python, followed by
tools such as virtualenv, then dependencies for matplotlib (NumPy and SciPy), and finally
matplotlib. Hold on, here we go.
How to do it...
1. In your Terminal paste and execute the following command:
ruby <(curl -fsSkL raw.github.com/mxcl/homebrew/go)
After the command finishes, try running brew update or brew doctor to verify that
installation is working properly.
2. Next, add the Homebrew directory to your system path, so the packages you install
using Homebrew have greater priority than other versions. Open ~/.bash_profile
(or /Users/[your-user-name]/.bash_profile) and add the following line to
the end of file:
export PATH=/usr/local/bin:$PATH
3. You will need to restart the terminal so it picks a new path. Installing Python is as
easy as firing up another one-liner:
brew install python --framework --universal
4. Now, you need to update your path (add to the same line):
export PATH=/usr/local/share/python:/usr/local/bin:$PATH
5. To verify that installation worked, type python --version at the command line,
you should see 2.7.3 as the version number in the response.
10
www.it-ebooks.info
Chapter 1
6. You should have pip installed by now. In case it is not installed, use easy_install
to add pip:
$ easy_install pip
7. Now, it's easy to install any required package; for example, virtualenv and
virtualenvwrapper are useful:
pip install virtualenv
pip install virtualenvwrapper
9. Verify that everything is working. Call Python and execute the following commands:
import numpy
print numpy.__version__
import scipy
print scipy.__version__
quit()
Getting ready
There are two ways of installing matplotlib on Windows. The easier way is by installing
prepackaged Python environments such as EPD, Anaconda and Python(x,y). This is the
suggested way to install Python, especially for beginners.
11
www.it-ebooks.info
Preparing Your Working Environment
The second way is to install everything using binaries of precompiled matplotlib and required
dependencies. This is more difficult as you have to be careful about the versions of NumPy
and SciPy you are installing, as not every version is compatible with the latest version of
matplotlib binaries. The advantage in this is that you can even compile your particular
versions of matplotlib or any library as to have the latest features, even if they are not
provided by authors.
How to do it...
The suggested way of installing free or commercial Python scientific distributions is as easy as
following the steps provided on the project's website.
If you just want to start using matplotlib and don't want to be bothered with Python versions
and dependencies, you may want to consider using the Enthought Python Distribution (EPD).
EPD contains prepackaged libraries required to work with matplotlib and all the required
dependencies (SciPy, NumPy, IPython, and more).
As usual, we download Windows Installer (*.exe) that will install all the code we need to start
using matplotlib and all recipes from this book.
Let me shortly explain how we would install matplotlib using precompiled Python, NumPy,
SciPy, and matplotlib binaries. First, we download and install standard Python using official
MSI Installer for our platform (x86 or x86-64). After that, download official binaries for NumPy
and SciPy and install them first. When you are sure that NumPy and SciPy are properly
installed, then we download the latest stable release binary for matplotlib and install it by
following the official instructions.
There's more...
Note that many examples are not included in the Windows installer. If you want to try the
demos, download the matplotlib source and look in the examples subdirectory.
12
www.it-ebooks.info
Chapter 1
Some popular features of PIL are fast access to data, point operations, filtering, image
resizing, rotation, and arbitrary affine transforms. For example, the histogram method
allows us to get statistics about the images.
PIL can also be used for other purposes, such as batch processing, image archiving, creating
thumbnails, conversion between image formats, and printing images.
PIL reads a large number of formats, while write support is (intentionally) restricted to the
most commonly used interchange and presentation formats.
How to do it...
The easiest and most recommended way is to use your platform's package managers.
For Debian/Ubuntu use the following commands:
$ sudo apt-get build-dep python-imaging
$ sudo pip install http://effbot.org/downloads/Imaging-1.1.7.tar.gz
How it works...
This way we are satisfying all build dependencies using apt-get system but also installing
the latest stable release of PIL. Some older versions of Ubuntu usually don't provide the
latest releases.
On RedHat/SciLinux:
# yum install python-imaging
# yum install freetype-devel
# pip install PIL
There's more...
There is a good online handbook, specifically, for PIL. You can read it at
http://www.pythonware.com/library/pil/handbook/index.htm, or download the
PDF version from http://www.pythonware.com/media/data/pil-handbook.pdf.
There is also a PIL fork, Pillow, whose main aim is to fix installation issues. Pillow can be found
at http://pypi.python.org/pypi/Pillow and it is easy to install.
On Windows, PIL can also be installed using a binary installation file. Install PIL in your Python
site-packages by executing .exe from http://www.pythonware.com/products/pil/.
Now, if you want PIL used in virtual environment, manually copy the PIL.pth file and the PIL
directory at C:\Python27\Lib\site-packages to your virtualenv site-packages directory.
13
www.it-ebooks.info
Preparing Your Working Environment
Even though Python comes with the urllib2 module for work with remote resources and
supporting HTTP capabilities, it requires a lot of work to get the basic tasks done.
Requests module brings new API that makes the use of web services seamless and pain free.
Lot of the HTTP 1.1 stuff is hidden away and exposed only if you need it to behave differently
than default.
How to do it...
Using pip is the best way to install requests. Use the following command for the same:
$ pip install requests
That's it. This can also be done inside your virtualenv if you don't need requests for every
project or want to support different requests versions for each project.
Just to get you ahead quickly, here's a small example on how to use requests:
import requests
r = requests.get('http://github.com/timeline.json')
print r.content
How it works...
We sent the GET HTTP request to a URI at www.github.com that returns a
JSON-formatted timeline of activity on GitHub (you can see HTML version of that
timeline at https://github.com/timeline). After response is successfully read,
the r object contains content and other properties of the response (response code,
cookies set, header metadata, even the request we sent in order to get this response).
14
www.it-ebooks.info
Chapter 1
Getting ready
As we already said, matplotlib configuration is read from a configuration file. This file provides
a place to set up permanent default values for certain matplotlib properties, well, for almost
everything in matplotlib.
How to do it...
There are two ways to change parameters during code execution: using the dictionary of
parameters (rcParams) or calling the matplotlib.rc() command. The former enables us
to load already existing dictionary into rcParams, while the latter enables a call to a function
using tuple of keyword arguments.
Both examples are semantically the same. In the second sample, we define that all
subsequent plots will have lines with line width of 2 points. The last statement of the
previous code defines that the color of every line following this statement will be red,
unless we override it by local settings. See the following example:
import matplotlib.pyplot as plt
import numpy as np
s = np.sin(2 * np.pi * t)
# make line red
plt.rcParams['lines.color'] = 'r'
plt.plot(t,s)
c = np.cos(2 * np.pi * t)
# make line thick
15
www.it-ebooks.info
Preparing Your Working Environment
plt.rcParams['lines.linewidth'] = '3
plt.plot(t,c)
plt.show()
How it works...
First, we import matplotlib.pyplot and NumPy to allow us to draw sine and cosine
graphs. Before plotting the first graph, we explicitly set line color to red using plt.
rcParams['lines.color'] = 'r'.
Next, we go to the second graph (cosine function), and explicitly set line width to 3 points
using plt.rcParams['lines.linewidth'] = '3'.
Getting ready
If you don't want to configure matplotlib as the first step in your code every time you
use it (as we did in the previous recipe), this recipe will explain how to have different default
configurations of matplotlib for different projects. This way your code will not be cluttered
with configuration data and, moreover, you can easily share configuration templates with
your co-workers or even among other projects.
How to do it...
If you have a working project that always uses the same settings for certain parameters in
matplotlib, you probably don't want to set them every time you want to add a new graph code.
Instead, what you want is a permanent file, outside of your code, which sets defaults for
matplotlib parameters.
matplotlib supports this via its matplotlibrc configuration file that contains most of the
changeable properties of matplotlib.
16
www.it-ebooks.info
Chapter 1
How it works...
There are three different places where this file can reside and its location defines its usage.
They are:
ff Current working directory: This is where your code runs from. This is the place to
customize matplotlib just for your current directory that might contain your current
project code. File is named matplotlibrc.
ff Per user .matplotlib/matplotlibrc: This is usually in user's $HOME directory (under
Windows, this is your Documents and Settings directory). You can find out
where your configuration directory is using the matplotlib.get_configdir()
command. Check the next command.
ff Per installation configuration file: This is usually in your python site-packages.
This is a system-wide configuration, but it will get overwritten every time you reinstall
matplotlib; so it is better to use per user configuration file for more persistent
customizations. Best usage so far for me was to use this as a default template if
I mess up my user's configuration file or if I need fresh configuration to customize
for a different project.
The following one-liner will print the location of your configuration directory and can be run
from shell.
$ python -c 'import matplotlib as mpl; print mpl.get_configdir()'
ff axes: Deals with face and edge color, tick sizes, and grid display.
ff backend: Sets the target output: TkAgg and GTKAgg.
ff figure: Deals with dpi, edge color, figure size, and subplot settings.
ff font: Looks at font families, font size, and style settings.
ff grid: Deals with grid color and line settings.
ff legend: Specifies how legends and text inside will be displayed.
ff lines: It checks for line (color, style, width, and so on) and markers settings.
ff patch: Patches are graphical objects that fill 2D space, such as polygons and circles;
set linewidth, color, antialiasing, and so on.
ff savefig: There are separate settings for saved figures. For example, to make
rendered files with a white background.
ff text: This looks for text color, how to interepret text (plain versus latex markup)
and similar.
ff verbose: It checks how much information matplotlib gives during runtime: silent,
helpful, debug, and debug-annoying.
ff xticks and yticks: These set the color, size, direction, and labelsize for major and
minor ticks for x and y axes.
17
www.it-ebooks.info
Preparing Your Working Environment
There's more...
If you are interested in more details for every mentioned setting (and some that we did
not mention here), the best place to go is the website of matplotlib project where there is
up-to-date API documentation. If it doesn't help, user and development lists are always
good places to leave questions. See the back of this book for useful online resources.
18
www.it-ebooks.info
Knowing Your Data
2
In this chapter we will cover the following recipes:
Introduction
This chapter covers basics about importing and exporting data from various formats. Also
covered are ways of cleaning data, such as normalizing values, adding missing data, live data
inspection, and usage of some similar tricks to get data correctly prepared for visualization.
www.it-ebooks.info
Knowing Your Data
Python has a module called csv that supports reading and writing CSV files in various
dialects. Dialects are important because there is no standard CSV and different applications
implement CSV in slightly different ways. A file's dialect is almost always recognizable by the
first look into the file.
Getting ready
What we need for this recipe is the CSV file itself. We will use sample CSV data that you can
download from ch02-data.csv.
We assume that sample datafiles is in the same folder as the code reading it.
How to do it...
The following code example demonstrates how to import data from a CSV file. We will:
filename = 'ch02-data.csv'
data = []
try:
with open(filename) as f:
reader = csv.reader(f)
header = reader.next()
data = [row for row in reader]
except csv.Error as e:
print "Error reading CSV file at line %s: %s" % (reader.line_
num, e)
sys.exit(-1)
20
www.it-ebooks.info
Chapter 2
if header:
print header
print '=================='
How it works...
First, we import the csv module in order to enable access to required methods. Then, we
open the file with data using the with compound statement and bind it to the object f. The
context manager with statement releases us of care about the closing resource after we are
finished manipulating those resources. It is a very handy way of working with resource-like
files because it makes sure that the resource is freed (for example, that the file is closed)
after the block of code is executed over it.
Then, we use the csv.reader() method that returns the reader object, allowing us to
iterate over all rows of the read file. Every row is just a list of values and is printed inside
the loop.
Reading the first row is somewhat different as it is the header of the file and describes the
data in each column. This is not mandatory for CSV files and some files don't have headers,
but they are a really nice way of providing minimal metadata about datasets. Sometimes
though, you will find separate text or even CSV files that are just used as metadata,
describing the format and additional data about the data.
The only way to check what the first line looks like is to open the file and visually inspect it
(for example, see the first few lines of the file). This can be done efficiently on Linux using
bash commands such as head as follows:
$ head some_file.csv
During iteration of data, we save the first row in header, while we add every other row to the
data list.
Should any errors occur during reading, csv.reader() will generate an error that we can
catch and print the helpful message to the user, in order to help detection of errors.
There's more...
If you want to read about the background and reasoning for the csv module, the PEP-defined
document CSV File API is available at http://www.python.org/dev/peps/pep-0305/.
If we have larger files that we want to load, it's often better to use well-known libraries, such
as NumPy's loadtxt(), that cope better with large CSV files.
21
www.it-ebooks.info
Knowing Your Data
Note that we need to define a delimiter to instruct NumPy to separate our data as
appropriate. The function numpy.loadtxt() is somewhat faster than the similar function
numpy.genfromtxt(), but the latter can cope better with missing data, and you are able
to provide functions to express what is to be done during the processing of certain columns
of loaded datafiles.
Currently, in Python 2.7.x, the csv module doesn't support Unicode, and you
must explicitly convert the read data into UTF-8 or ASCII printable. The official
Python CSV documentation offers good examples on how to resolve data
encoding issues.
In Python 3.3 and later versions, Unicode support is default and there are no
such issues.
A common approach to importing data from Excel files is to export data from Excel into
CSV-formatted files and use the tools described in the previous recipe to import data using
Python from the CSV file. This is a fairly easy process if we have one or two files (and have
Microsoft Excel or OpenOffice.org installed), but if we are automating a data pipe for many
files (as part of an ongoing data processing effort), we are not in a position to manually
convert every Excel file into CSV. So, we need a way to read any Excel file.
Python has decent support for reading and writing Excel files through the project
www.python-excel.org. This support is available in the form of different modules
for reading and writing, and is platform independent; in other words, we don't have to
run on Windows in order to read Excel files.
The Microsoft Excel file format changed over time, and support for different versions is
available in different Python libraries. The latest stable version of XLRD is 0.90 at the
time of this writing and it has support for reading .xlsx files.
22
www.it-ebooks.info
Discovering Diverse Content Through
Random Scribd Documents
Athenian citizens were not allowed to marry foreign women, even
of reputation and virtue; this law was not strictly observed: the
penalty for the violation of it was fixed at one thousand drachms.
Simo mentions the epithet peregrina, as what Chremes said he had
heard Glycera called; but does not himself drop the slightest hint
against her, but, on the contrary, praises her modest demeanour; as
he must have been well aware, that she did not deserve such an
epithet, being her opposite neighbour, and having seen her abroad:
ξέναι, or strange women, when they appeared in public, were
obliged to wear striped dresses, to distinguish them from women of
innocent conversation.
NOTE 83.
NOTE 84.
Exit Sosia.
“Here we take our last leave of Sosia, who is, in the language of
the commentators, a protatick personage, that is, as Donatus
explains it, one who appears only once in the beginning (the
protasis) of the piece, for the sake of unfolding the argument, and is
never seen again in any part of the play. The narration being ended,
says Donatus, the character of Sosia is no longer necessary. He
therefore departs, and leaves Simo alone to carry on the action. With
all due deference to the ancients, I cannot help thinking this method,
if too constantly practised, as I think it is in our author, rather
inartificial. Narration, however beautiful, is certainly the deadest part
of theatrical compositions: it is, indeed, strictly speaking, scarce
dramatic, and strikes the least in the representation: and the too
frequent introduction of a character, to whom a principal person in
the fable is to relate in confidence the circumstances, previous to the
opening of the play, is surely too direct a manner of conveying that
information to the audience. Every thing of this nature should come
obliquely, fall in a manner by accident, or be drawn as it were
perforce, from the parties concerned, in the course of the action: a
practice, which, if reckoned highly beautiful in epic, may be almost
set down as absolutely necessary in dramatic poetry. It is, however,
more adviseable, even to seem tedious, than to hazard being
obscure. Terence certainly opens his plays with great address, and
assigns a probable reason for one of the parties being so
communicative to the other; and yet it is too plain that this narration
is made merely for the sake of the audience, since there never was a
duller hearer than Master Sosia, and it never appears, in the sequel
of the play, that Simo’s instructions to him are of the least use to
frighten Davus, or work upon Pamphilus. Yet even this protatick
personage is one of the instances of Terence’s art, since it was often
used in the Roman comedy, as may be seen even in Plautus, to
make the relation of the argument the express office of the
prologue.”—Colman.
Monsieur Baron does not dismiss Sosia here, but brings him on
the stage again; once in the third act, and once in the fourth. Sir R.
Steele introduces Humphrey again in the first act, and also in the
fifth. We are told by Donatus, that in the Andrian and Perinthian of
Menander, which are similar in the plot, the first scene is the same as
in Terence, but that in the Perinthian, the old man consults with his
wife instead of Sosia; and, in the Andrian he opens with a soliloquy.
NOTE 85.
NOTE 86.
NOTE 87.
NOTE 88.
The grinding-house.
Terence has rendered by the word pistrinum, the Greek
σωφρονιστήριον, or house of correction, whither criminals were sent
for the various terms of imprisonment proportioned to their offences.
Slaves, while in this prison, were employed chiefly in grinding corn,
which, from a deficiency of mechanical knowledge, was, in those
times, a very laborious employment. The Athenians, who were
universally celebrated for their kind and gentle treatment of slaves,
were very reluctant to proceed to severer punishments than
whipping or imprisonment: but when a flagrant delinquency
rendered it necessary to make an example, they either burned the
criminal with a hot iron, in the offending member, if possible; or put
on his feet a torturing instrument, called χοῖνιξ. If the law required
the criminal to suffer death, which happened in very few cases, he
was either hung, beaten to death with clubs, or cast into a deep pit,
called βάραθρον, filled at the bottom with sharp spikes. They
sometimes had recourse to other extraordinary modes of
punishment: but the before-mentioned were the most common.
NOTE 89.
NOTE 90.
NOTE 91.
NOTE 92.
NOTE 93.
NOTE 94.
NOTE 95.
NOTE 96.
NOTE 97.
I. From Terence.
V. From Horace.
X. From Anacreon.
Μέσον, ναῒ
ΦορήΜεθα σὺν Μελαίνᾳ,
ΧειΜῶνι Μοχθεῦντες Μεγάλῳ.
NOTE 98.
NOTE 99.
NOTE 100.
NOTE 101.
A. I. S. V. Pamphilus, Mysis.
This scene contains the third and last part of the narration, which
is entirely pathetic, and its length is very artificially and successfully
relieved by the figure called by the Greeks προσωποποια, which is
introduced with so many moving and pathetic graces, as afford
ample proof that Terence was as great a master of the passions, as
even Trabea, Attilius, and Cæcilius themselves, who were so highly
extolled by the ancients for their excellence in compositions of that
nature. Terence has admirably relieved the necessary length of his
narration in this play, by his judicious method of dividing it: the first
part is serious, (vide Note 65,) and raises our curiosity: the second
part is comic, (vide Note 89,) and excites our laughter; the third part
is pathetic, and moves our pity. The lines in which Pamphilus
describes the death of Chrysis are so extremely moving, that some
of the most eminent critics have considered them at least equal, if
not superior, to all attempts in the pathetic both ancient and
modern. The finest passage in M. Baron’s Andrienne is, (in my
opinion,) his imitation of the before-mentioned speech of Pamphilus:
and the inimitable beauty which so much strikes us in the French
copy ought to impress us with a just idea of the splendid merit of
the Latin original.
The whole speech is too long to be inserted here, the following
are extracts:
NOTE 102.
NOTE 103.
I shrewdly suspect that this daughter of Chremes is either
hideously ugly, or that something is amiss in her.
In the Latin aliquid monstri alunt, they breed up some monster.
This expression took its rise from the custom of exposing and
destroying monstrous and deformed children, (see Note 93) which
was required by law: therefore, those parents who resolved,
notwithstanding, to educate a child of that kind, were compelled to
do so with the utmost secrecy: hence, the phrase “alere monstrum,”
to breed up a monster, was used in Rome, to express any thing
done in great secrecy. Terence has, by no means, violated
probability, in representing Pamphilus as unacquainted with the
person of Philumena: though she had been contracted to him; as
Grecian women very seldom appeared abroad, and never, unveiled:
and it not unfrequently occurred, that the bridegroom was
introduced to the bride for the first time on the day of marriage.
NOTE 104.
She is in labour.
In the Latin, Laborat e dolore. Cooke thinks that these words
mean merely she is weighed down by grief: and argues, that if
Pamphilus had understood her words in any other sense, he would
have urged her to more haste; as he does, when she tells him that
she is going for a midwife. But laboro sometimes means to strive or
struggle, as in Ovid,
Also, in Horace,
———————“laborat
Lympha fugax trepidare.”
Od., B. II. O. 3. L. 11.
NOTE 105.
Can I suffer, that she, who has been brought up in the paths
of modesty and virtue, should be exposed to want, and,
perhaps, even to dishonour?
By the expression sinam coactum egestate ingenium immutarier?
shall I suffer her innocence to be endangered by want? I am inclined
to believe that Terence meant, the want of friends and protection,
and not poverty, because we are told afterwards, (Act IV.) that
Glycera was possessed of the property of Chrysis, which we are to
imagine, from what Crito says concerning it, to have been something
considerable. I believe egestate is often put for want of any kind. It
may appear somewhat enigmatical, that Terence should speak of the
liberal and virtuous education of Glycera, by such a person as
Chrysis was said to have been; but it is a circumstance in no wise
repugnant to the manners of the Greeks; as we see in the Eunuch in
the instance of Thais and Pamphila.
NOTE 106.
I call upon you, then, by the pledge of this hand you now
extend to me, and by the natural goodness of your
disposition.
Quod ego te per hanc dextram oro, et ingenium tuum. Some read
genium, by your genius, or by your good angel, and quote the
following passage from Horace in support of this reading:
NOTE 107.
NOTE 108.
Charinus, Byrrhia.
“These two characters were not in the works of Menander, but
were added to the fable by Terence, lest Philumena’s being left
without a husband, on the marriage of Pamphilus to Glycerium
should appear too tragical a circumstance.—Donatus.
Madame Dacier, after transcribing this remark adds, that it
appears to her to be an observation of great importance to the
theatre, and well worthy our attention.
Important as this dramatic arcanum may be, it were to be
wished, that Terence had never found it out, or, at least, that he had
not availed himself of it in the construction of the Andrian. It is plain
that the duplicity of the intrigue did not proceed from the imitation
of Menander, since these characters, on which the double plot is
founded, were not drawn from the Greek poet. Charinus and Byrrhia
are indeed but poor counterparts, or faint shadows of Pamphilus and
Davus; and, instead of adding life and vigour to the fable, rather
damp its spirit, and stop the activity of its progress. As to the tragical
circumstance of Philumena’s having no husband, it seems something
like the distress of Prince Prettyman[A], who thinks it a matter of
indifference, whether he shall appear to be the son of a king or a
fisherman, and is only uneasy lest he should be the son of nobody at
all. I am much more inclined to the opinion of an ingenious French
critic, whom I have already cited more than once, than to that of
Donatus or Madame Dacier. His comment in this underplot is as
follows:—
“It is almost impossible to conduct two intrigues at a time
without weakening the interest of both. With what address
has Terence interwoven the amours of Pamphilus and
Charinus in the Andrian! But has he done it without
inconvenience? At the beginning of the second act, do we not
seem to be entering upon a new piece? and does the fifth
conclude in a very interesting manner?”—Diderot.
It is but justice to Sir Richard Steele to confess, that he has
conducted the under-plot in the Conscious Lovers in a much more
artful and interesting manner than Terence in the play before us.
The part which Myrtle sustains (though not wholly unexceptionable,
especially the last act,) is more essential to the fable than Charinus
in the Andrian. His character also is more separated and
distinguished from Bevil, than Charinus from Pamphilus, and serves
to produce one of the best scenes[B] in the play.” Colman.
NOTE 109.
NOTE 110.
Byrrhia.—I beseech you, O Charinus, to wish for something possible,
since what you now wish for is impossible!
Terence always admirably preserves the characters of domestics,
in the style of the advice they give their masters, which is very often
conveyed in some trite adage, or formal apothegm. This is another
instance of our author’s art. Want of attention to the dialogue of the
inferior characters, is a frequent fault among dramatic writers; and
often proves hostile to the success of a piece, particularly of a
comedy, where it is absolutely essential.
NOTE 111.
“C. Byrrhia,
Quid tibi videtur? Adeon’ ad eum? B. Quidni? si nihil impetres,
Ut te arbitretur sibi paratum mœchum, si illam duxerit.”
The ingenious French editor, mentioned in Note 72, has given the
following elegant and delicate turn to this objectionable passage.
“C. Byrrhia,
Quid tibi videtur? Adeon’ ad eum? B. Quidni? ut, si nihil impetres,
Te sibi cavendum credat, si illam duxerit.”
NOTE 113.
NOTE 114.
Now, if either you, or Byrrhia here, can do any thing; in Heavens
name, do it; contrive, invent, and manage, if you can, that she
may be given to you.
It does not appear that Charinus and Byrrhia set any stratagem
on foot, in compliance with the wishes of Pamphilus, to break off the
treaty between Simo and Chremes; indeed, they are rather inactive
throughout the play, and the under-plot proceeds separately from
the principal plot: this, I attribute to Terence’s close imitation of
Menander, in what respects Pamphilus’s intrigue, as the characters of
Charinus and Byrrhia were added by Terence: Menander’s play being
written with a single plot; which was doubled by our author, in
compliance with the taste of his age. It is supposed that Terence’s
reputation for art was gained chiefly by his success in combining two
intrigues in one play: a mode of dramatic writing which the Romans
in those times considered a great novelty. The Stepmother is the
only play written by Terence, in which the plot is single, and though
critics in general argue with Volcatius,
“Sumetur Hecyra sexta ex his fabula,”
that it is not equal to the rest of his productions, many persons, very
eminent for their judgment, have attributed the superiority of the
other five plays, to the advantages they possess over the
Stepmother, both in portraiture of character, and in the conduct of
the catastrophe, and of the fable in general, rather than to any
additional attraction which they can derive from a double plot. The
Carin and Byrrhie of M. Baron, are, in every respect, the
counterparts of the Charinus and Byrrhia of Terence; but Sir R.
Steele has very much enlivened the character of Charinus; his Myrtle
is one of the most entertaining personages in the piece. Vide Notes
108, 159, 162, 163.
NOTE 115.
NOTE 116.
NOTE 117.
Every thing is quite still and quiet.