INTRODUCTION TO THE COURSE -

LABWORK
Week Lecture Content
11 Lab-work 1 Lab-work 1:
- Data with Python (install Python, input data into Python)
12 Lab-work 2 Lab-work 2:
- Graphics with Python (draw graphic by Python)
13 Lab-work 3 Lab-work 3:
- Statistical analyses with Python
14 Lab-work 4 Lab-work 4:
- Programming with Python in practice (part 1)
15 Lab-work 5 Lab-work 5:
- Programming with Python in practice (part 2)
16 Lab-work 6 Lab-work 6: Assignment (30%)

[email protected] 1
 APPLIED STATISTICS
IN ENVIRONMENT
COURSE CODE: ENEE1006IU

Lab-works
Applications of Python
Part 1 – Data
(3 credits: 2 is for lecture, 1 is for lab-work)
Instructor: TRAN THANH TU
Email: [email protected]

[email protected] 2
 Introduction to Python
Features of Python
• Interactive: one can launch a Python console and run instructions
directly it.
• Portable: available on most existing systems. It only requires a C
compiler to be ported to any new platform.
• Structure: functions, classes, modules.
• Many existing libraries for all sort of purposes.
• It is a case-sensitive language (Labour and labour have different
meaning)

[email protected] 3
 Introduction to Python
Install Python
Mac OS X:
Windows: n Python is already installed.
n Download Python from
n Open a terminal and run python
http://www.python.org or run Idle from Finder.
n Install Python.

n Run Idle from the Start Menu.

Linux:
n Chances are you already have
Python installed. To check, run
python from the terminal.
Install ipython n If not, install from your
Run cmd (make sure the directory is distribution's package system.
same as the one Python is stored)
Call py get-pip.py to install pip
Install Spyder (Editor for Python)
Call pip install ipython to install Ipython
[email protected] 4
 Introduction to Python
Syntax Rules
• The syntax is designed to be simplified as compared to other
languages like C/C++.
• Every compound instruction ends with ":"
• There are no blocks of code; blocks are implicitly created by
indentation.
• Expressions: usual arithmetic operators, named logic operators:
and, or, not.
• Assignments use the = sign but they don't have to end with ";"
• Comments start with # as in shell scripting.
• Variables are declared by assigning them a value and they are
local to the block where they appear first.

[email protected] 5
 Introduction to Python
Some Lessons
n There’s usually more than one way to solve a problem.

n Don’t rush to code the first idea that pops out of your
head. Think about the design and ask if there’s a better
way to approach the problem.
n Your first task is to find a correct algorithm. After that,
strive for clarity, simplicity, efficiency, scalability, and
elegance.
n One of the best ways to formulate an algorithm is to ask

yourself how you would solve the problem.

n This straightforward approach is often simple, clear, and

efficient enough.
n Try to debug, ask Google!

[email protected] 6
 Introduction to Python
References
n Python Homepage

• http://www.python.org
n Python Tutorial

• http://docs.python.org/tutorial/
n Python Documentation

• http://www.python.org/doc
n Python Library References

n http://docs.python.org/release/2.5.2/lib/lib.html

n Python Add-on Packages:

n http://pypi.python.org/pypi

[email protected] 7
 Introduction to Python
Define a new function
Function’s name parameters
prompt
>>> def greet(person):
print("Hello",person)
Function’s parts
print ("How are you?")
>>> greet("Terry") statements
Hello Terry
How are you?
>>> greet("Paula")
Hello Paula
How are you?
>>>
#this is for comments
comment
(not command à Python skips text from # to end of line)
[email protected] 8
 Introduction to Python
Spyder editor

Help Pane
Editor Pane

IPython Pane

[email protected] 9
 Introduction to Python
Strings
"hello"+"world" "helloworld" # concatenation
"hello"*3 "hellohellohello" # repetition
"hello"[0] "h" # indexing (first letter)
"hello"[-1] "o" # (from end)
"hello"[1:4] "ell" # slicing
len("hello") 5 # size
"hello" < "jello" 1 # comparison
"e" in "hello" 1 # search
"escapes: \n etc, \033 etc, \if etc"
'single quotes' """triple quotes""" r"raw strings"

[email protected] 10
 Introduction to Python
Lists
n Flexible arrays, not Lisp-like linked lists
a = [99, "bottles of beer", ["on", "the", "wall"]]
n Same operators as for strings
a+b, a*3, a[0], a[-1], a[1:], len(a)
n Item and slice assignment
a[0] # -> 99
a[1:2] # -> ["bottles of beer"]
del a[-1] # -> [98, "bottles of beer"]
n Use len() to get the length of a list
len(a) # -> 2

[email protected] 11
 Data with Python
Datatype
number = input("Enter a number: ")
#Enter a number: 3 number = eval(input("Enter an equation: "))
number #Enter an equation: 3 + 2
number
'3' A string
5
int
number = eval(input("Enter a number: ")) x, y = 2, 3
#Enter a number: 3 X
number 2
Y
3 An integer (no single quotes ' ')
3
x = 10
number = float
eval(input("Enter a number: ")) X = 5.7
#Enter a number: 3.7 print(x)
number 10
print(X)
3.7 A float (no single quotes ' ')
[email protected]
5.7 12
 Data with Python
Datatype
We can do various kinds of conversions between strings, integers
and floats using the built-in int, float, and str functions.

x = 10 y = "20" z = 30.0
float(x) float(y) int(z)
10.0 20.0 30
str(x) int(y) str(z)
'10' 20 '30.0'
integer è float string è float float è integer
integer è string string è integer float è string

[email protected] 13
 Data with Python
Comparison
Operators
n

== != > < >= <=

Operator Arithmetical
n Assignment
meaning
= += -= *= //= /= %= **=
+ addition
n Logical
- subtraction
and or not
* multiplication
n Bitwise
/ float division
& | ^ ~ >> <<
** exponentiation
n Membership
% remainder
in not in
abs() absolute value
n Identity
// integer division
is is not
[email protected] 14
 Data with Python
x=6
Examples y=2
print(x - y)
x=2+3 4
print(x) print(x/y)
5 3.0
print(5 * 7) print(x//y)
35 3
print("5" + "7")
57 print(x*y)
12
print(x**y)
36
print(x%y)
0
print(abs(-x))
6 15
[email protected]
 Data with Python To use a library, we
need to make sure
this line is in our
Math library program:
exp(x) Returns e ** x. import math
ceil(x) Returns the ceiling of x.
Importing a library
floor(x) Returns the floor of x.
makes whatever
copysign(x,y) Returns x with the same sign as y. functions are
fabs(x) Returns the absolute value of x. defined within it
factorial(x) Returns xfactorial. available to the
isinf(x) Return True if x is infinity. program.
ldexp(x, i) Returns x* (2 ** i).
log(x[, base]) Returns the logarithm of x to the given base. If base is
omitted, this function computes the natural logarithm.
log10(x) Returns the base 10 logarithm of x.
pow(x, y) Returns x** y.
sqrt(x) Returns the square root of x.
[email protected] 16
 Data with Python
List Operations

Operator Meaning
<seq> + <seq> Concatenation
<seq> * <int-expr> Repetition
<seq>[] Indexing
len(<seq>) Length
<seq>[:] Slicing
for <var> in <seq>: Iteration
<expr> in <seq> Membership (Boolean)

[email protected] 17
 Data with Python
List Operations
Method Meaning
<list>.append(x) Add element x to end of list.
<list>.sort() Sort (order) the list. A comparison function
may be passed as a parameter.
<list>.reverse() Reverse the list.
<list>.index(x) Returns index of first occurrence of x.
<list>.insert(i, x) Insert x into list at index i.
<list>.count(x) Returns the number of occurrences of x in
list.
<list>.remove(x) Deletes the first occurrence of x in list.
<list>.pop(i) Deletes the ith element of the list and
returns its value.
[email protected] 18
 Data with Python
Examples
t = (23, ‘abc’, 4.56, (2,3), ‘def’)
t[1:4]
(‘abc’, 4.56, (2,3))
t[:2] Slicing
(23, ‘abc’)
t[2:]
(4.56, (2,3), ‘def’)

t[:] [ : ] makes a copy of an entire sequence

(23, ‘abc’, 4.56, (2,3), ‘def’)

[email protected] 19
 Data with Python
Examples
lst = [1,2,3,4] Check if a certain value appears
3 in lst
True anywhere in a sequence

lst = [1,2,3,4]
lst[3]
4 Change the value in a list
lst[3] = "Hello"
lst
[1, 2, 3, 'Hello']

zeroes = [0] * 50 Produces a list containing 50 zeroes

names = ['ben', 'chen', 'yaqin'] Zipping lists together
gender = [0, 0, 1]
combine=zip(names, gender)
print(tuple(combine))
[('ben', 0), ('chen', 0), ('yaqin', 1)]
[email protected] 20
 Data with Python
Examples
lst = [3, 1, 4, 1, 5, 9] lst.count(1)
lst.append(2) 2
lst.remove(1)
lst lst
[3, 1, 4, 1, 5, 9, 2] [9, 5, 4, 3, 'Hello', 2, 1]
lst.pop(3)
lst.sort() 3
lst lst
[1, 1, 2, 3, 4, 5, 9] [9, 5, 4, 'Hello', 2, 1]
lst.reverse()
myList=[34, 26, 0, 10]
lst del myList[1]
[9, 5, 4, 3, 2, 1, 1] myList
lst.index(4) [34, 0, 10]
2 del myList[1:3]
lst.insert(4, "Hello") myList
lst [34]
[9, 5, 4, 3, 'Hello', 2, 1, 1]
[email protected] 21
 Data with Python
File I/O: open and close
file open(filename, mode)
While opening a file, you need to supply
n The name of the file, including the path preserve the
n The mode in which you want to open a file
contents

n Common modes are r (read), w (write), a (append)

Mode is optional, defaults to r clears the

existing contents
of the file
open(..) returns a file object
close() on the file object closes the file
n finishes any buffered operations

[email protected] 22
 Data with Python
Example of File I/O

[email protected] 23
 Data with Python
Example of File I/O
( )

( )

[email protected] 24
 Data with Python
Example of File I/O

( )

Note empty line due to '\n'

[email protected] 25
 Data with Python
Example of File I/O

Note the use of for ... in

for sequence

( )

]
[email protected] 26
 Data with Python
Example of File I/O

( )

( )

[email protected] 27
 Data with Python
Data processing
n The generic Python language has somewhat limited
capabilities for dealing with math and array (table) data
n Numpy is a library that allows for efficient processing of
large numerical datasets. It also creates a n-
dimensional data structure (array) that doesn't exist in
generic Python.
n Pandas is a Python library that builds on Numpy and
Matplotlib to make it easier to work with dataframes
(i.e. tables)

[email protected] 28
 Data with Python
DataFrame
Filter
#Calculate mean salary for each professor rank:
df_sub = df[ df['salary'] > 120000 ]
#Select only those rows that contain female
professors:
df_f = df[ df['sex'] == 'Female' ]
Slice
#Select column salary:
df['salary']
#Select column salary:
df[['rank','salary']]
[email protected] 29
 Data with Python
DataFrame
Select rows
#Select rows by their position:
df[10:20]
Select a range of rows
#Select rows by their labels:
df_sub.loc[10:20,['rank','sex','salary']]

Select a range of rows and/or columns

#Select rows by their labels:
df_sub.iloc[10:20,[0, 3, 4, 5]]

[email protected] 30
 Data with Python
DataFrame
Select a range of rows and/or columns
df.iloc[0] # First row of a data frame
df.iloc[i] #(i+1)th row
df.iloc[-1] # Last row
df.iloc[:, 0] # First column
df.iloc[:, -1] # Last column
df.iloc[0:7] #First 7 rows
df.iloc[:, 0:2] #First 2 columns
df.iloc[1:3, 0:2] #Second through third rows
and first 2 columns
df.iloc[[0,5], [1,3]] #1st and 6th rows and 2nd
and 4th columns 31
[email protected]
 Data with Python
DataFrame
Sort
# Create a new data frame from the original
sorted by the column Service
df_sorted = df.sort_values( by ='service')
df_sorted.head()

# Create a new data frame from the original

sorted by 2 columns Service and Salary
df_sorted = df.sort_values( by =['service',
'salary'], ascending = [True, False])
df_sorted.head(10)

[email protected] 32
 Data with Python
DataFrame
Missing values
# Read a dataset with missing values
flights =
pd.read_csv("http://rcs.bu.edu/examples/python/
data_analysis/flights.csv")

# Select the rows that have at least one

missing value
flights[flights.isnull().any(axis=1)].head()

[email protected] 33
 Data with Python
DataFrame
Missing values
df.method() description
dropna() Drop missing observations
dropna(how='all') Drop observations where all cells is NA
dropna(axis=1, Drop column if all the values are missing
how='all')
dropna(thresh = 5) Drop rows that contain less than 5 non-missing values

fillna(0) Replace missing values with zeros

isnull() returns True if the value is missing

notnull() Returns True for non-missing values

[email protected] 34
 Data with Python
Numpy

[email protected] 35
 Data with Python
Numpy – Create array

[email protected] 36
 Data with Python
Numpy – Create array

[email protected] 37
 Data with Python
Numpy reshape

[email protected] 38
 Data with Python
Numpy get elements with expression

[email protected] 39
 Data with Python
Numpy ravel and transpose

[email protected] 40
 Data with Python
Pandas
Data loading

[email protected] 41
 Data with Python
Pandas
Series examples

[email protected] 42
 Data with Python
Pandas
DataFrame examples

[email protected] 43
 Data with Python
Pandas

[email protected] 44
 Data with Python
Pandas

[email protected] 45
 Data with Python
Pandas
Aggregation Functions in Pandas
Aggregation - computing a summary statistic about each group,
i.e.
• compute group sums or means
• compute group sizes/counts

Common aggregation functions:

min, max
flights[['dep_delay','a
count, sum, prod
rr_delay']].agg(['min',
mean, median, mode, mad
'mean','max'])
std, var

[email protected] 46