Complete Course in Unix

Overview
File System
Shell Programming
Process Management
Inter Process Communication
System Administration
Utilities
Overview
 Characteristics of Unix System

Time-sharing
Multi-user, multi-process system
Written in high level language
Hierarchical file system
Consistent file formats
 Architecture of Unix System

System Kernel
Utilities and application software
File Structure
 Kernel Functions

Schedule Processes
Keep track of files
Control hardware devices
File System
 File System

Characteristics of file system

Types of files
File system layout
File structure related system calls
 File System
Characteristics of Unix File System

Hierarchical Structure
Consistent treatment of file data
Ability to create and delete files
Dynamic growth of files
Treatment of peripheral devices as files
 File System
Types of Files

Regular files
Directories
Special files
Pipes
Named Pipes
I/O devices
 File System
File System Layout

Boot block contains bootstrap code to initialize the

operating system
Super block describes the state of a file system
Inode list is a list of inodes
Data blocks contain file and administrative data
File structure related
system calls
 OPEN

#include <fcntl.h>

fd = open ( pathname, flags, modes)

pathname is the filename
flags indicate the type of open
modes give the file permission if it is being created
 CREAT

Creates a new file with the indicated filename and access

permission modes. If the file already exists, creat truncates
the file.

fd = creat ( filename, mode)

char *filename;
int mode;
 READ

Read returns the number of bytes it read into the buffer. It

is generally count or a value less than count if the number
of bytes left to be read are less than the count
number = read ( fd, buffer, count )
fd is the file descriptor
buffer is the address of the data structure
count is the no of bytes to be read
number is the no of bytes read
 WRITE

Write writes count bytes of data from user address buffer

to the file descriptor fd
number = write ( fd, buffer, count )
fd is the file descriptor
buffer is the address of the data structure
count is the no of bytes to be written
number is the no of bytes actually written
 LSEEK
Changes the position of read-write pointer
lseek ( fd, offset, reference )
int fd is the file descriptor
long offset is the byte offset
int reference indicates where the offset is to be considered from
reference values
0 set the pointer to offset bytes from beginning of file
1 increment the current value of pointer by offset
2 set the pointer to size of file plus offste bytes
 CLOSE

Close closes a file descriptor obtained from a prior open,

creat, dup, pipe or fcntl system call

close ( fd )
int fd;
 STAT

Stat returns status information about the specified file

stat ( filename, statbuffer )
char *filename is the absolute file name
struct stat *statbuffer is the address of the data structure
in the user process which will contain status information
 FSTAT

Fstat returns the status information of an open file whose

file descriptor is fd
fstat ( fd, statbuffer )
int fd is the file descriptor
struct stat *statbuffer is the address of the data structure
in the user process which will contain status information
 DUP

Dup duplicates the specified file descriptor, returning the

lowest available file descriptor
newfd = dup ( fd )
int fd is the file descriptor
int newfd is the new file descriptor
 LINK

Link gives another name filename2 to an existing file

filename1
link ( filename1, filename2 )
char *filename1 is an existing file
char *filename2 is the new name which will also point to
the source file name
 ACCESS
Checks if calling process has r,w or x permission for a file
access ( filename, access_mode )
char *filename is the name of the file
access_mode
04 read
02 write
01 read
00 existence
Return value
0 access allowed
-1 access not allowed
 CHMOD
Chmod changes the access permissions of the indicated file to the
specified mode
chmod(filename, mode)
char *filename
Modes
4000 setuid bit
2000 setgid bit
1000 sticky bit
0400 read for owner
0040 read for group
0004 read for others
 CHOWN
Chown changes the owner and the group of the indicated
file to the specified owner and group IDs
chown(filename, owner, group)
char *filename name of the file
int owner, group
 UMASK
Sets the file mode creation mask and return the old value.
When creating a file, permissions are turned off if the
corresponding bits in the mask are set
umask(mask)
int mask
 IOCTL

Ioctl does device-specific operations on the open device

whose file descriptor is fd.
ioctl ( fd, cmd, arg_pointer)
int fd;
int cmd is the device specific command
arg_pointer defines a parameter whose type depends on
the command
SHELL PROGRAMMING
 What is a Shell

Standard command and programming language

Interprets command lines
Runs other programs
Reads commands either from the prompt or from a file
 Shell Types

sh (bourne shell)
csh
ksh (korn shell)
bash(bourne again shell)
 Input/Output Redirection

Create Files
Append to files
Use files as input to the shell
Truncate files
 Input/Output Redirection
Redirector Operators

< Open the following file as stdin

> Open the following file as stdout
>> Append to the following file
2>&1 Merge file descriptor 2 with 1
| Pipe stdout into stdin
 File/Directory Commands
cd Change directory
ls List directorys contents
mkdir Create directory
rmdir Remove directory
cat Concatenate and write files contents
ln Create links to existing files
mv Move files
cp Copy files
rm Remove files
 Selection Commands
awk Pattern scanning and processing language
cut Select columns
diff Compare files
grep Select lines or rows
sed Edit streams of data
head Select lines from top in a file
tail Select lines from bottom in a file
wc Count words, characters and lines in a file
 Shell Variables
Special Variables
Environment Variables
Program Variables
 Special Variables
$# Number of positional arguments
$0 Program/command name invoked
$1,$2.. Positional parameters passed to the shell
$* Expands to $1 $2...
$@ Expands to $1 $2..
$? Return code from the last executed command
$$ PID of the current shell
 Environment Variables
HOME Pathname of the users login directory
PATH Shells search path for commands
PS1 Primary prompt
IFS Internal field separator
PWD Current working directory
 Program Variables
Normal variables using assignment operator

e.g. age=12
 Quotes
Single Quotes No evaluation whatsoever

Double Quotes No evaluation expect for $, \ and `

Backslash Character following backslash is

protected from evaluation
 Expr
Evaluates arguments and returns 0 or 1
Returns numeric value from an expression

E.g. if expr $a = $b
or if expr $str = abc
a=`expr $a + 1`
Foreground/Background Execution
Background processing by placing an & at the end
Background processes run effectively wrapped around by
nohup
loops run in background enclosed in parentheses
fg command moves a background job into foreground
 READ
Main input mechanism for a shell program
Gets a line from the standard input and can store the value
in a variable
Can read a file line by line
Control Structures
 If ..then ..else

If <condition>
then
<command>
[elif <condition>
<command>
else
<command>]
fi
 Case

case word in
pattern1) action1 ;;
pattern2|pattern3) action2 ;;
.......................................
.......................................
*) default action ;;
esac
 While .. do

while <condition>
do
commands
done
 test

Tests if an expression is true

Returns a 0 if true else returns a non-zero value
 test

Integer operators used with test

eq is equal to
ne is not equal to
lt is less than
le less than or equal to
gt greater than
ge greater than or equal to
 test
Syntax

test <expression>

or

[ <expression> ] alternative syntax

 test
File Operators

-d file exists and is a directory

-f file exists and is an ordinary file
-r file exists and is readable
-s file exists and has a non-zero size
-w file exists and is writable
-x file exists and is executable
 Filename Wildcards
* Match 0 or more characters
? Match exactly one character
[1,2,3] Match any character in the square brackets
[a-z] Match any characters between a to z
[!0-9] Match any character not in the square brackets
+(abc) Match one or more instance of abc
*(abc) Match zero or more instances of abc
 Functions

function identifier {list;}

or
identifier() {list;}
 Command Execution

Grave accents or backquotes

`command`
Executing in a separate environment
(command)
Simple execution
{command}
 Disk Usage Commands

df Gives the disk usage statistics of all the filesystems

du Summarizes disk usage
 Setuid/Setgid Permissions

This powerful feature enables the effective userid/groupid of

the calling process to be the same as the userid/groupid of
the owner of the file.
Used most frequently when a script needs to be executed as
root.
 Setuid/Setgid Permissions

The setuid bit is turned on like this

chmod 4000 filename
and the permissions then look like this
-rws------
For setgid
chmod 2000 filename
----rws---
 Sticky Bit

This is set to prevent files from being removed from a

directory by arbitrary users.
chmod 1000 directoryname
and the permissions look like this
drwxr-xr-t
In this case, only the owner of the directory, owner of the file
or the superuser can remove a file in the directory
 Find

Finds files that match a given set of parameters

find -name filename Finds files with the filename
find -perm mode Find files with given permissions
find -type c File types
find -user name Belonging to given user
find -size n Of size n
find -atime n Access time n days ago
 Find

Operators used with find

-print
find -name filename -print Prints the filename
-exec
find -name filename -exec rm -f {} \; Executes command
 File

Tells about the contents of the file

file filename
 Copy Files

cp command
example
cp file1 file2
 Copy Directories

cp -r|R command
Recursively copies the files under a directory and its
subdirectories

tar command
Use tar cvf to create a tar of the subdirectories and files
under a given directory
e.g. tar cvf tarname path of files/subdirectories
 Copy Directories

Untar a tar file using

tar xvf tarname untar location
 Mail

Used for sending mails

e.g.
cat msg | mail -s subject recipient
Note that the mail program expects its input from the stdin
 Cut

Selects columns or fields

-c for col
-f for field
-d for delimiter
 Sort

Sort divides each lines into fields at whitespace (blanks or

tabs) and sorts the lines by fields from left to right

sort +n -m
sorts the nth field, stops sorting at the mth field
 Process Status

Produces a report summarizing execution statistics for the

current processes.
E.g.
ps -ef gives general stats
ps -fu username gives stats about all the processes running
under the user username
 Time

Used to time a command

e.g.
time progname
 Uptime

Used to get a rough estimate of the system load

e.g.
Uptime
The result looks like this
12:40pm up 4 day(s), 23:59, 1 user, load average: 0.00,
0.00, 0.01
 Cron

Does periodic execution of jobs

Listing cron entries

crontab -l
Editing cron entries
crontab -e
e.g 20 10 * * * /$HOME/myscript
Process Related System
Calls
 EXEC
Execve executes the program file filename, overlaying
the address space of the executing process.Argv is an
array of character string parameters to the execed
program and envp is an array of character strings that
are the environment of the new process
execve(filename, argv, envp)
char *filename
char *argv[]
char *envp[]
 FORK

Fork creates a new process. The child process is a logical

copy of the parent process, except that the parents return
value from the fork is the process ID of the child, and the
childs return value is 0
 WAIT

Wait causes the process to sleep until it discovers a

child process that had exited or a process asleep in trace
mode. If wait_stat is not 0, it points to an address that
contains status information on return from the call.
wait(wait_stat)
int *wait_stat
 EXIT

Exit causes the calling process to terminate, reporting

the 8 low-order bits of status to its waiting parent. The
kernel may call exit internally, in response to certain
signals
exit(status)
int status
 Some other System Calls

getuid() Returns the real user ID

geteuid() Returns the effective user ID
getgid() Returns the real group ID
getegid() Returns the effective group
getpid() ID
getppid() Returns the process ID
Returns the parent process
ID
 SIGNAL

Software Interrupts
 Classification of Signals

Termination of a process
Process induced exceptions
Unrecoverable conditions during system calls
Unexpected error condition during a system call
Tracing execution of a process
 Handling Signals

Exit on receipt of a signal

Ignore the signal
Execute a particular function on receipt of a signal
 Signal Syntax

#include <signal.h>
signal(sig,function)
int sig;
void (*func)()
Signal allows the calling process to control
signal processing.
 Signal Syntax
Some values of sig
SIGHUP hangup
SIGINT interrupt
SIGQUIT quit
SIGKILL kill
SIGFPE floating point exception
SIGBUS bus error
SIGPIPE write on a pipe with no reader
SIGTERM software termination
 Signal Syntax

The function is interpreted as

SIG_DFL default operation. Process
terminates except for some
signals
SIG_IGN ignore the signal.kernel calls a
function in the process with
signal number as the argument.
SIGKILL cant be ignored
 Kill
Kill sends the signal sig to the process identified by pid
kill(pid,sig)
int pid,sig
pid > 0 send signal to process whose PID is pid
pid 0 send signal to processes whose process group ID is
pid of sender
pid -1 if effective UID of sender is of superuser send
signal to all processes otherwise to processes whose real
UID equals effective UID of the sender
Inter Process Communication
 Types of IPC

Pipes
Named Pipes
Messages
Shared Memory
Semaphores
Sockets
 Pipes

Allow data transfer in a FIFO manner

Allow synchronization of process execution
Use file system for data storage
Can be created using pipe() system call
Pipe manipulation is done using regular system calls for
files e.g. read(), write()
Only related processes can access a pipe
 Pipes
Syntax

pipe (file_descriptors)
where file_descriptors is a pointer to an array of 2 integers
which are the file descriptors for reading and writing the
pipe
 Named Pipes

Similar to pipes
Have directory entries
open system call used to open named pipes
mknod system call to create named pipes
Unrelated processes can also communicate using named
pipes
 MESSAGES

Allow processes to send data streams to arbitrary processes

Four system calls cover all the operations
msgget
msgsend
msgrec
msgctl
 msgget

Returns the message queue identifier assocaited with the key

msgget(key, flag)
key is the name of the message queue
flag is the modes which can be passed
 msgsnd
msgsnd (id, message, size, flag)
Sends message to the queue associated with the message queue identifier specified
by id
message is of the form
struct msgbuf
{
long mtype;
char mtext[];
}
Flags can be
IPC_NOWAIT- Return -1 if the message cant be stored
MSG_NOERROR- Message bigger than size is shortened with no error returned
 msgrcv
msgsnd (id, message, size, type, flag)
Reads message from the queue associated with the message
queue identifier specified by id
message is of the form
struct msgbuf
{
long mtype;
char mtext[];
}
 msgctl
Msgctl ( id, command, buffer )
Provides a variety of message control functions as specified by
command
Commands can be
IPC_STAT store information about queue in the buffer
IPC_SET Change the information in the buffer
IPC_RMID remove the message queue identifier specified
by id
 SHARED MEMORY

Allows processes to share part of their virtual address

space
System calls similar to the ones in messages
 shmget

int shmget ( key, size, flag )

key is the name of the shared memory

size is the size of shared memory segment in bytes
flag is a combination of
IPC_CREAT
IPC_EXCL
mode
 shmat

Inserts a shared memory segment into a process address space

char *shmat ( id, at_address, flag )

id is the shared memory segment identifier

size is the address in process address space where the shared
memory segment has to be attached
flag is a combination of
SHM_RDONLY read only
0 read,write
 shmdt

Removes a shared memory segment into a process address space

int shmdt ( dt_address )

dt_address is the address in the process address space of a

shared memory segment to be detached
 shmctl

shmctl ( id, cmd, buf)

Provides shared memory control operations as specified by the

cmd

cmd IPC_STAT
IPC_SET
IPC_RMID
SHM_LOCK
SHM_UNLOCK
 SEMAPHORE

Non-negative integer count used to coordinate access to

the resources
Initial semaphore count set to the number of free resources
Count decremented or incremented by the threads or
processes as and when they acquire or free resources.
Threads block at count zero till it becomes positive
 semget

semget ( key, num, flag )

Returns or creates the semaphore identifier associated with the
key
key is the name of the semaphore set
number defines the number of semaphores in the set
flag is a combination of
IPC_CREAT
IPC_EXCL
 semop
semop ( id, operations, number )
Performs operations on the semaphore
id is the semaphore identifier
operations is the array of semaphore operation structure
operation consists of
- semaphore number
-operation
-flags
number is the number of entries in operations
 semctl

semctl ( id, semnum, cmd, ... )

Performs operations on the semaphore as specified by the cmd
id is the semaphore identifier
Fourth argument is optional, depending upon the operation
requested.
If required, then its a buffer.
cmd GETVAL, SETVAL, GETPID, GETNCNT, GETZCNT
 SOCKET

Creates an endpoint for communication and returns a descriptor

int socket ( domain, type, protocol )
domain specifies communication domain
type type of communication over socket (virtual circuit or
datagram)
protocol protocol to control the communication
Other Socket System Calls
 Close

Closes the communication endpoint

close ( sd )
sd is the socket descriptor
 Bind

Assigns a name to an unnamed socket

int bind ( sd, sockname, namelength)
sd is the socket descriptor
sockname is the name of the socket
namelength is the length of the name
 Connect

Makes a connection to an existing socket

int connect ( sd, sockaddr, namelength)
sd is the socket descriptor
sockaddr is the name of the target socket
 Accept

Receives incoming requests for a connection

accept ( fd, addr, addrlen )
addr user data array which kernel fills with the address of the
connecting client
 send

Transmits data over a connected socket

send ( sd, msg, length, flag)
msg pointer to data being sent
length length of the data
 recv

Receives data over a connected socket

recv ( sd, buf, length, flag)
buf pointer to data being sent
length length of the data
 shutdown

Closes a socket connection

shutdown ( sd, mode )
mode specifies which side no longer permits data transmission
 System Administration

File System Administration

Machine Management
System Tuning and Performance Management
System Accounting
User Management
Backup/Restore Activities
Setting up Peripheral Devices
Routine Maintenance
 File System Administration

Creation and Maintenance of File Systems

File System consistency checks
Monitoring of disk space
Routine Maintenance such as removing inactive files
 File System Administration
Commands
TASK SHELL COMMAND
Check File System fsck
Display disk usage df
List files by size du
Identify FileSystem type fstyp
Create File System mkfs
Mount File System mount
Unmount File System umount
 Machine Management Commands

TASK SHELL COMMAND

Set boot defaults fltboot
Change to firmware shutdown
Power Off shutdown
Reboot shutdown
Display Configuration prtconf
Display System uname
 System States

System State Description

0 Powerdown State
s Single user
2 Multiuser
3 Start Remote File Share
5 Switch to firmware
6 Halt & Reboot OS
q Reexamine (/etc/inittab)
 Changing System States
Switching from multiuser to single user
SHUTDOWN -I S
INIT S
Switching to any state
INIT <System State>
Switching to multiuser
INIT 2
Turning system off
SHUTDOWN -I 0
 User and Group Management

Assigning and Maintaining logins and passwords

Organization of system resources to suit particular needs
of the users
Communicating with users
 User Management Commands

Task Command
Add User useradd
Add Group groupadd
Change password passwd
Listing users and groups logins, listusers
Modify user attributes usermod
Modify group attributes groupmod
Remove user userdel
Remove group groupdel
 Communicating with Users

Keeping users informed about system status and servicing

details
Using Wall to send messages to the users
Using utilities like mail
Message of the day facility using /etc/motd
SYSTEM UTILITIES
 MAKE

Provides a method for maintaining an up-to-date version of

programs that consist of a number of files generated in a
variety of ways
 MAKE
Operations

Find the target in the description file

Ensure that all the dependencies (files) of the target exist and
are up to date
Create the target in case any of the dependencies have been
modified
 MAKE
MakeFile

Comments All characters after a #

Continuation lines Lines ending in \
Macro Definition Identifier followed by = sign
 MAKE
Example MakeFile makedemo

#makedemo
prog : x.o y.o z.o
cc x.o y.o z.o -o prog
x.o : x.c defs.h
cc x.c
y.o : y.c defs.h
cc y.c
z.o : z.c defs.h
cc z.c
 MAKE
Macros

Macro Definition
name = value

Example
OBJECTS = x.o y.o z.o

prog : $(OBJECTS)
cc $(OBJECTS) -o prog
SYSTEM UTILITIES
 Grep

Searches the named files or the standard input and prints

each line that contains an instance of the pattern
 Grep
Usage

grep [options] pattern filenames

Options
-n prints the line numbers
-v inverts the sense of the test
-i case insensitive search
 Grep
Example

Locate variable in C source code

grep variable *.[ch]
See if abc is logged in
who | grep abc
Filenames that dont have temp in their names
ls | grep -v temp
List the subdirectories in the current directory
ls -ltr | grep ^d
 AWK

Scans a set of input lines, searching for lines that match any
set of patterns and take a specified action for each pattern on
each line that matches the pattern
 AWK
Usage

awk pattern-action statements optional list of input files

e.g. awk {print $1, $2} file1 file2
OR
awk -f programfile optional list of input files
where programfile is the file where the pattern-action
statements are stored
 AWK
Fields

Record is the sequence of characters separated by a newline

character
Fields in a record are separated by sequence of blanks or
tabs

$0 refers to the entire record

$1 refers to the first field
$2 refers to the second field
 AWK
Printing

{ print }
prints all the records

{print $1 $3}
prints the first and third fields of each of the input line
 AWK
BuiltIn Variables

ARGC number of command line arguments

ARGV array of command line arguments
FILENAME name of current input file
FNR record number in current file
NR number of current record
NF number of fields in the record
OFS Output field separator
ORS Output record separator
 AWK
Examples
Print last field of each input line
{ print $NF }
Print total number of input lines
END {print NR}
Print input lines with more than 4 fields
NF > 4
Print the total no of lines that match string abc
/abc/ {nlines++} END {print nlines}
 AWK
BuiltIn Arithmetic Functions

cos(x)
log(x)
exp(x)
rand(x)
sin(x)
sqrt(x)
srand(x)
 AWK
BuiltIn String Functions

index(s,t) returns position of t in s

length(s) returns length of s
split(s,a) splits into array a on FS
sub(s,r) substitutes s for first r
substr(s,p) returns suffix of string starting at position p
 AWK
Control Flow Statements

if (expression) stat1 else stat2

while (expression) stat
for (expr1;expr2;expr3) stat
do stat while (expr)
break
continue
next
exit(expr)
return(expr)
 SED
Stream Editor

Reads one line at a time from input files

Applies the commands from a list of commands one at a
time, in order, to each line
writes to standard output
 SED
Usage

sed list of ed commands filenames...

Example
sed s/UNIX/UNIX(TM)/g filenames
replaces UNIX by UNIX(TM) in all the file occurences
and writes the result to standard output
 SED
Commands
a\ text Appends text on the output before reading the
next input line
b label Branch to the : command bearing the label
c\ text change lines to the following text
d delete line; read next input line
p print
r read
w write
q quit