Operating System security

Security Policy
● A security policy is a statement of the security we
expect the system to enforce

● A system can be trusted only in relation to its security

policy
⚫ that is, to the security needs the system is
expected to satisfy

2
The Military Security Policy is based on
protecting classified information with respect
to confidentiality.
• each piece of information is ranked at a
particular sensitivity level: – unclassified –
restricted – confidential – secret – top secret
• each piece of information may be associated
with one (or more) projects, called
compartments.
• The combination is called the classification
or class of a piece of information.
A person has a clearance to access information up to a
certain level of sensitivity.
• The clearance of a person has the same form as the
classification of a piece of information:
• the need-to-know rule (principle of least privilege)
means that individuals shall only have access to those
data that they need in order to perform their jobs.
• the use of compartments helps to enforce the
need-to know rule.
• the user may not alter classifications, i.e. the policy
requires Mandatory Access Control (MAC)
Dominance Relation
● We say that s dominates o (or o is dominated by s) if o
<= s

For a subject s and an object o,

o <= s if and only if
rank(o) <= rank(s) and
compartments(o) is subset of compartments(s)

● A subject can read an object if the subject dominates

the object.

6
Trusted OS Design
● OS is a complex system
⚫ difficult to design
⚫ Adding the responsibility of security
enforcement makes it even more difficult
● Clear mapping from security requirements to the
design
● Design must be checked using formal reviews or
simulation
● Requirements 🡪 design 🡪 testing

7
Security Design Principles
● Least privilege
⚫ users, programs, fewest privilege possible
● Economy of mechanism
⚫ small, simple, straight forward
● Complete mediation
⚫ every attempt must be checked

8
Security Design Principles
● Ease of use
● Permission based
⚫ denial of access is the default
● Separation of privilege
⚫ more than one condition
● Least common mechanism
⚫ the risk of sharing

9
Security features in ordinary OS
● Authentication of users
⚫ password comparison
● Protection of memory
⚫ user space, paging, segmentations
● File and I/O device access control
⚫ access control matrix
● Allocation & access control to general objects
⚫ table lookup

10
Security features in ordinary OS
● Enforcement of sharing
⚫ integrity, consistency
● Fair service
⚫ no starvation
● Interprocess communication & synchronization
⚫ table lookup
● Protection of OS protection data
⚫ encryption, hardware control, isolation

CS 450/650 Lecture 21: Trusted Operating System

11
Security features of Trusted OS
● Identification and Authentication
● Mandatory and Discretionary Access Control
● Object reuse protection
● Complete mediation (all accesses are checked)
● Trusted path
● Accountability and Audit (security log)
● Intrusion detection (patterns of normal system
usages, anomalies)

12
Trusted Operating Systems
● OS part that performs lowest level functions
User
tasks
O
S
OS
Kernel

Hardwar
e

13
Security Kernel
● responsible for enforcing security mechanisms
of the entire OS
● Coverage
⚫ ensure that every access is checked
● Separation
⚫ security mechanisms are isolated from the rest of OS
and from user space 🡪 easier to protect
● Unity
⚫ all security mechanisms are performed by a single set
of code 🡪 easier to trace problems

14
Reference Monitor
● portion of a security kernel that controls accesses to
objects
● Collection of access controls for
O O O
⚫ Devices, Files, Memory, Interprocess
communication, Other objects Gate
S S S
● It must be
⚫ Always invoked when any object is accessed
⚫ Small enough
⚫ analysis, testing
⚫ Tamperproof
CS 450/650 Lecture 22: Trusted Operating System
15
Trusted Computing Base (TCB)
● Everything in the trusted OS necessary to enforce
security policy
● System element on which security enforcement
depends:
⚫ Hardware
⚫ processors, memory, registers, and I/O devices
⚫ Processes
⚫ separate and protect security-critical processes

CS 450/650 Lecture 22: Trusted Operating System

16
Trusted Computing Base (TCB)
● System element on which security enforcement
depends (cont):
⚫ Primitive files
⚫ security access control database,
identification/authentication data
⚫ Protected memory
⚫ reference monitor can be protected against
tampering
⚫ Interprocess communication
⚫ e.g., reference monitor can invoke and pass data
securely to audit routine
CS 450/650 Lecture 22: Trusted Operating System
17
TCB and Non-TCB Code
Applications

Utilities
User request interpreter
Non-TC
… B
Segmentation, paging, memory
management
Primitive I/O
Basic Operations
Clocks, timing
TC
Interrupt handling
B
Hardware:registers memory
Capabilities

CS 450/650 Lecture 22: Trusted Operating System

18
 Combined Security Kernel / OS System
OS Kernel:
User
- HW interactions tasks
- Access control O
S
OS
Kernel

Hardwar
e

OS:
Resource allocation
Sharing
Access control
Authentication functions Security activity
CS 450/650 Lecture 22: Trusted Operating System
19
 Separate Security Kernel
Security Kernel:
Access control User
tasks
Authentication functions O
S
Security Kernel

Hardwar
e

OS:
Resource allocation
Sharing
Hardware interactions

CS 450/650 Lecture 22: Trusted Operating System

20
Layered OS
User processes

Compilers, database

O Utility functions
S
File system, device allocation
Scheduling, sharing,
MM
Synchronization, allocation
OS kernel
Security kernel Security functions

Hardware

CS 450/650 Lecture 22: Trusted Operating System

21
Assurance
● Testing
⚫ based on the actual product being evaluated,
⚫ not on abstraction

● Verification
⚫ each of the system’s functions works correctly

● Validation
⚫ developer is building the right product
⚫ according to the specification

CS 450/650 Lecture 22: Trusted Operating System

22
Testing
● Observable effects versus internal structure
● Can demonstrate existence of a problem,
⚫ but passing tests does not imply absence of any
● Hard to achieve adequate test coverage within
reasonable time
⚫ inputs & internal states
⚫ hard to keep track of all states

● Penetrating Testing
⚫ tiger team analysis, ethical hacking
⚫ Team of experts in design of OS tries to crack system

CS 450/650 Lecture 22: Trusted Operating System

23
Formal verification
● The most rigorous method
● Rules of mathematical logic to demonstrate that a
system has certain security property

● Proving a Theorem
⚫ Time consuming
⚫ Complex process

CS 450/650 Lecture 22: Trusted Operating System

24
Validation
● Requirements checking
⚫ system does things it should do
⚫ also, system does not do things it is not supposed to do

● Design and code reviews

⚫ traceability from each requirement to design and
code components
● System testing
⚫ data expected from reading the requirement
document can be confirmed in the actual running
of the system
CS 450/650 Lecture 22: Trusted Operating System
25