1

SECURITY IN
COMPUTING,
FIFTH EDITION
Chapter 5: Operating Systems

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 2

Chapter 5 Objectives
• Basic security functions provided by operating systems
• System resources that require operating system
protection
• Operating system design principles
• How operating systems control access to resources
• The history of trusted computing
• Characteristics of operating system rootkits

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 3

Operating System Functions

Users

User Interface
Synchronization,
Concurrency
Control, Deadlock
Operating Services Management,
System
Communication,
Accounting

Resource Allocation

Data

CPU
Memory

Program
Libraries
I/O Devices

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 4

History of Operating Systems

• Single-user systems, no OS
• Multiprogrammed OS, aka monitors
• Multiple users
• Multiple programs
• Scheduling, sharing, concurrent use
• Personal computers

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 5

Protected Objects
• Memory
• Sharable I/O devices, such as disks
• Serially reusable I/O devices, such as printers
• Sharable programs and subprocedures
• Networks
• Sharable data

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 6

OS Layered Design
Subprocesses of User Processes

User Processes

Compilers, Data base Managers

Utility Functions

File Systems, Device Allocation

Operating
System Scheduling, Sharing,
Memory Management

Synchronization, Allocation
Operating
System Security Functions
Kernel
Security
Kernel Hardware

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 7

Functions Spanning Layers

Least
Trusted
Code

User Authentication Module

User
Interface

User ID
Lookup

Most Authentication
Trusted Data Comparison
Code
Authentication
Data Updates
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 8

Modular OS Design
Users Users Users Users Users
User Mode
Privileged Mode
User Interface

Sec File Object A/V Net Backup Shell

System Services Interface

I/O Time Synch Memory Comm Sec

Primitive Services

Microkernel Kernel Mode Drivers

Hardware Interface and Abstraction

Hardware

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 9

Virtualization
• With virtualization, the OS presents each user with just
the resources that user should see
• The user has access to a virtual machine (VM), which
contains those resources
• The user cannot access resources that are available to
the OS but exist outside the VM
• A hypervisor, or VM monitor, is the software that
implements a VM
• Translates access requests between the VM and the OS
• Can support multiple OSs in VMs simultaneously
• Honeypot: A VM meant to lure an attacker into an
environment that can be both controlled and monitored
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 10

Separation and Sharing

• Methods of separation:
• Physical
• Temporal
• Logical
• Cryptographic
• Methods of supporting separation/sharing:
• Do not protect
• Isolate
• Share all or share nothing
• Share but limit access
• Limit use of an object

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 11

Hardware Protection of Memory

Addresses Memory
0
Hardware
Address Operating System
Limitation
n

n+ 1

Addressing User Program Space

Range

High
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 12

Fence Registers
Address Address
Limit Limit
Register Register
n +1 p +1

Addresses Memory Addresses Memory

0 0

Operating Operating
System Version 1 System Version 2

n p

n +1 p +1

User Program User Program

Space Space
Addressing Addressing
Range Range

High High
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 13

Base/Bounds Registers
Addresses Memory
0

Operating
Base Register System
n
n +1
n +1
User A
Program Space
Bounds Register p
p
p +1 User Program
User B Space
Program Space
q
q +1 User C
Program Space

High
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 14

Two Pairs of Base/Bounds Registers

Operating
System

User A
Data Base Program Space

User B
Data Bounds Data Space

User A
Data Space User Program
and Data
Space
User C
Program Space

Program Base User C

Data Space

Program Bounds User B

Program Space

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 15

Tagged Architecture
Tag Memory Word

R 0001
RW 0137
R 0099
X

X
X
X

X
R 4091
RW 0002

Code: R = Read-only RW = Read/Write

X = Execute-only
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 16

Segmentation
Physical Placement of
Program’s Segments
Logical Arrangement of
Program
Operating
MAIN System
Segments

SEG_A SUB

SUB MAIN

DATA_SEG
Segments for
Other Users

SEG_A

DATA_SEG

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 17

Segment Address Translation

Segment Translation Table
Address
MAIN c 0
Logical Program
SEG_A g
MAIN
SUB a a
SEG_A DATA_SEG h
b
FETCH<DATA_SEG,20>
c
SUB
d
DATA_SEG e
+ f
g
h

Location 20 within Segment DATA_SEG i

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 18

Paging
Page Translation Table Memory
Logical Program Page Address Address
0
Page 0 0 b a
Page 1 b
FETCH<4,37> 1 f Page 0
c Location
Page 2 2 i Page 4
d 37, Page 4
Page 3 3 l e
Page 7
Page 4 4 c f
Page 1
Page 5 5 g g
Page 5
h
Page 6 6 n
i
Page 2
Page 7 7 e j
k
l
Page 3
m
+
n
Page 6
o

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 19

Paged Segmentation
Segment Translation Table
Page Memory
Segment Table Address
0
MAIN
Page Translation Tables a
Logical Program SEG_A For Segment MAIN b
Page Address DATA_SEG Page 1
MAIN c
SUB
0 c MAIN Page 0
DATA_SEG 1 f d
SEG_A
e
FETCH<DATA_SEG,20> SEG_A Page 1
For Segment SEG_A
Page Address f
MAIN Page 1
SUB 0 n g
SEG_A Page 2
1 e h
DATA_SEG
2 g i
SUB Page 0
For Segment SUB j
Page Address k
0 i
l
DATA_SEG Page 0
For Segment DATA_SEG m
Page Address
20 = Page 0 n
0 l SEG_A Page 0
o
1 b
+
Segment DATA_SEG Word 20
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 20

Principles of Secure OS Design

• Simplicity of design
• OSs are inherently complex, and any unnecessary complexity only
makes them harder to understand and secure
• Layered design
• Enables layered trust
• Layered trust
• Layering is both a way to keep a design logical and understandable
and a way to limit risk
• Example: very tight access controls on critical OS functions, fewer
access controls on important noncritical functions, and few if any
access controls on functions that aren’t important to the OS

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 21

Kernelized Design
• A kernel is the part of the OS that performs the lowest-
level functions
• Synchronization
• Interprocess communication
• Message passing
• Interrupt handling
• A security kernel is responsible for enforcing the security
mechanisms of the entire OS
• Typically contained within the kernel

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 22

Reference Monitor
S
S
S

Opera
ting Sy
stem
Truste or
d Soft
war e
Opera
ting Sy
stem
o
Truste r
O d Soft
O war e
O
O Reference
Monitor
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 23

Trusted Systems
• A trusted system is one that has been shown to warrant
some degree of trust that it will perform certain activities
faithfully
• Characteristics of a trusted system:
• A defined policy that details what security qualities it enforces
• Appropriate measures and mechanisms by which it can enforce
security adequately
• Independent scrutiny or evaluation to ensure that the mechanisms
have been selected and implemented properly

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 24

History of Trusted Systems

Security Controls E.C. Information
for Computer Technology
Systems Security
Trusted Computer Evaluation Common
System Evaluation Criteria Criteria
Criteria

1970 1983 1991 1994

1972 1988 1992

Security
Technology Combined
British,
Planning Federal
German,
Study Criteria
French
Criteria
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 25

Trusted Computing Base (TCB)

User applications

Utilities

User request interpreter

User process coordination, synchronization
Non-TCB
User environment: objects, names (e.g., files)
User I/O
Procedures, user processes
Creation and deletion of user objects
Directories
Extended types
Segmentation, paging, memory management

Primitive I/O
Basic operations
TCB Clocks, timing
Interrupt handling
Hardware: registers, memory
Capabilities

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 26

Other Trusted System Characteristics

• Secure startup
• System startup is a tricky time for security, as most systems load
basic I/O functionality before being able to load security functions
• Trusted path
• An unforgeable connection by which the user can be confident of
communicating directly with the OS
• Object reuse control
• OS clears memory before reassigning it to ensure that leftover data
doesn’t become compromised
• Audit
• Trusted systems track security-relevant changes, such as
installation of new programs or OS modification
• Audit logs must be protected against tampering and deletion
From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 27

Rootkits
• A rootkit is a malicious software package that
attains and takes advantage of root status or
effectively becomes part of the OS
• Rootkits often go to great length to avoid being
discovered or, if discovered and partially
removed, to reestablish themselves
• This can include intercepting or modifying basic OS
functions

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 28

Rootkit Evading Detection

Will call’s
Intercepted Yes
result reveal
function call
rootkit?

No

Pass call to Execute call but

operating system monitor result and
function adjust as necessary

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.
 29

Summary
• OSs have evolved from supporting single users and
single programs to many users and programs at once
• Resources that require OS protection: memory, I/O
devices, programs, and networks
• OSs use layered and modular designs for simplification
and to separate critical functions from noncritical ones
• Resource access control can be enforced in a number of
ways, including virtualization, segmentation, hardware
memory protection, and reference monitors
• Rootkits are malicious software packages that attain root
status or effectively become part of the OS

From Security in Computing, Fifth Edition, by Charles P. Pfleeger, et al. (ISBN: 9780134085043). Copyright 2015 by Pearson Education, Inc. All rights reserved.