CS 356 – Lecture 25 and 26

Operating System Security

Spring 2013
 Review
•  Chapter 1: Basic Concepts and Terminology
•  Chapter 2: Basic Cryptographic Tools
•  Chapter 3 – User Authentication
•  Chapter 4 – Access Control Lists
•  Chapter 5 – Database Security (skipped)
•  Chapter 6 – Malicious Software
•  Networking Basics (not in book)
•  Chapter 7 – Denial of Service
•  Chapter 8 – Intrusion Detection
•  Chapter 9 – Firewalls and Intrusion Prevention
•  Chapter 10 – Buffer Overflow
•  Chapter 11 – Software Security
•  Chapter 12 – OS Security
 Chapter 12

Operating System
Security
Operating System

§  each layer of code needs

measures in place to
provide appropriate
security services

§  each layer is vulnerable

to attack from below if
the lower layers are not
secured appropriately

Security Layers
 Measures
•  the 2010 Australian Defense Signals Directorate
(DSD) list the “Top 35 Mitigation Strategies”
•  over 70% of the targeted cyber intrusions
investigated by DSD in 2009 could have been
prevented
•  the top four measures for prevention are:
–  patch operating systems and applications using auto-
update
–  patch third-party applications
–  restrict admin privileges to users who need them
–  white-list approved applications
 Operating System Security
•  possible for a system to be compromised during the
installation process before it can install the latest
patches
•  building and deploying a system should be a planned
process designed to counter this threat
•  process must:
–  assess risks and plan the system deployment
–  secure the underlying operating system and then the key
applications
–  ensure any critical content is secured
–  ensure appropriate network protection mechanisms are
used
–  ensure appropriate processes are used to maintain
security
 System Security Planning

the first step in deploying a

new system is planning
plan needs to identify
appropriate
personnel and
training to install and planning should
manage the system include a wide
security assessment
of the organization

planning process needs to

determine security
requirements for the aim is to maximize
system, applications, data, security while
and users minimizing costs
System Security Planning Process
any additional security
the purpose of the system, measures required on the
the type of information who will administer the
system, and how they will system, including the use of
stored, the applications and
manage the system (via host firewalls, anti-virus or
services provided, and their
local or remote access) other malware protection
security requirements
mechanisms, and logging

what access the system has

the categories of users of to information stored on
the system, the privileges
other hosts, such as file or
they have, and the types of
database servers, and how
information they can access
this is managed

how access to the

how the users are information stored on the
authenticated
system is managed
 Operating Systems Hardening
•  first critical step in securing a system is to
secure the base operating system
•  basic steps
•  install and patch the operating system
•  harden and configure the operating system to
adequately address the identified security needs of the
system
•  install and configure additional security controls, such
as anti-virus, host-based firewalls, and intrusion
detection system (IDS)
•  test the security of the basic operating system to
ensure that the steps taken adequately address its
security needs
 Initial Setup and Patching

should stage
system and validate all
security begins patches on the
with the test systems
installation of initial installation before
should install the overall boot
the operating minimum process must deploying them
system necessary for the
desired system
also be secured
in production

the integrity and critical that the

full installation and source of any system be kept up
ideally new hardening process to date, with all
systems should be additional device
should occur driver code must critical security
constructed on a before the system related patches
protected network be carefully
is deployed to its installed
validated
intended location
 • Remove
Unnecessary
Services, •  when performing the initial
Applications, installation the supplied
Protocols defaults should not be
used
–  default configuration is set
to maximize ease of use
•  if fewer software packages and functionality rather
are available to run the risk than security
is reduced
–  if additional packages are
•  system planning process needed later they can be
should identify what is installed when they are
actually required for a required
given system
 •  system planning process should
consider:
• Configure
–  categories of users on the
Users, Groups, system
and
–  privileges they have
Authentication
–  types of information they can
access
–  how and where they are
defined and authenticated

•  not all users with access to a •  default accounts included as part

system will have the same of the system installation should be
access to all data and secured
resources on that system
–  those that are not required
•  elevated privileges should should be either removed or
be restricted to only those disabled
users that require them, and
then only when they are –  policies that apply to
needed to perform a task authentication credentials
configured
 • Install
• Configure
• Additional
• Resource
• Security
• Controls
• Controls
•  once the users and
groups are defined, •  further security
appropriate possible by installing
permissions can be set and configuring
on data and resources additional security
•  many of the security tools:
hardening guides –  anti-virus software
provide lists of –  host-based firewalls
recommended changes
to the default access –  IDS or IPS software
configuration –  application white-listing
 •  checklists are included in
• Test the security hardening guides

System •  there are programs

specifically designed to:
Security –  review a system to ensure
that a system meets the
basic security requirements
–  scan for known
vulnerabilities and poor
configuration practices
•  final step in the process of
initially securing the base •  should be done following
operating system is the initial hardening of the
security testing system
•  goal: •  repeated periodically as
–  ensure the previous part of the security
security configuration steps
are correctly implemented maintenance process
–  identify any possible
vulnerabilities
 Application Configuration
•  may include:
–  creating and specifying appropriate data storage areas for
application
–  making appropriate changes to the application or service default
configuration details
•  some applications or services may include:
–  default data
–  scripts
–  user accounts
•  of particular concern with remotely accessed services such as
Web and file transfer services
–  risk from this form of attack is reduced by ensuring that most of
the files can only be read, but not written, by the server
 Encryption Technology

is a key
enabling
technology
that may be if secure network
cryptographic
used to services are file systems are
secure data must be provided using another use of
if secure network encryption
both in configured TLS or IPsec services are
transit and and suitable public and provided using
private keys must SSH, appropriate
when stored appropriate server and client
cryptographic be generated for keys must be
keys created, each of them created
signed, and
secured
 Security Maintenance
•  process of maintaining security is
continuous
•  security maintenance includes:
–  monitoring and analyzing logging information
–  performing regular backups
–  recovering from security compromises
–  regularly testing system security
–  using appropriate software maintenance
processes to patch and update all critical
software, and to monitor and revise
configuration as needed
 automated can only inform you
analysis is about bad things
preferred that have already
happened

generates significant in the event of a system

volumes of information and breach or failure, system
it is important that administrators can more
sufficient space is quickly identify what
allocated for them happened

Logging
range of data acquired should key is to ensure you
be determined during the capture the correct
system planning stage data and then
information appropriately monitor
can be and analyze this data
generated
by the
system,
network and
applications
 Data Backup and Archive

performing regular needs and policy relating

backups of data is to backup and archive
a critical control backup archive should be determined
that assists with during the system
• the process of • the process of retaining
maintaining the making copies copies of data over planning stage
integrity of the of data at extended periods of • kept online or offline
system and user regular time in order to meet • stored locally or transported to a
intervals legal and operational remote site
data requirements to access •  trade-offs include ease of
• may be legal or past data implementation and cost
operational versus greater security and
requirements for the robustness against different
retention of data threats
 Linux/Unix Security
•  patch management
–  keeping security patches up to date is a widely
recognized and critical control for maintaining security
–  application and service configuration
–  most commonly implemented using separate text files for each
application and service
–  generally located either in the /etc directory or in the installation tree for
a specific application
–  individual user configurations that can override the system defaults are
located in hidden “dot” files in each user’s home directory
–  most important changes needed to improve system security are to
disable services and applications that are not required
 Linux/Unix Security
•  users, groups, and permissions
–  access is specified as granting read, write, and
execute permissions to each of owner, group, and
others for each resource
–  guides recommend changing the access
permissions for critical directories and files
–  local exploit
•  software vulnerability that can be exploited by an
attacker to gain elevated privileges
–  remote exploit
•  software vulnerability in a network server that could be
triggered by a remote attacker
 Linux/Unix Security

remote access controls logging and log rotation

•  several host firewall programs •  should not assume that the
may be used default setting is necessarily
•  most systems provide an appropriate
administrative utility to select
which services will be permitted
to access the system
 Linux/Unix Security
•  chroot jail
–  restricts the server’s view of the file system
to just a specified portion
–  uses chroot system call to confine a
process by mapping the root of the
filesystem to some other directory
–  file directories outside the chroot jail aren’t
visible or reachable
–  main disadvantage is added complexity
 Windows Security

patch management users administration

•  “Windows Update” and and access controls
“Windows Server •  systems implement
Update Service” assist discretionary access
with regular controls resources
maintenance and
should be used •  Vista and later systems
include mandatory integrity
•  third party applications controls
also provide automatic
update support •  objects are labeled as
being of low, medium, high,
or system integrity level
•  system ensures the
subject’s integrity is equal
or higher than the object’s
level
•  implements a form of the
Biba Integrity model
 Windows Security
Users Administration and Access Controls

Windows systems also

define privileges combination of share and
NTFS permissions may be
•  system wide and granted to user
accounts used to provide additional
security and granularity
when accessing files on a
shared resource

User Account Control (UAC) Low Privilege Service

•  provided in Vista and later systems Accounts
•  assists with ensuring users with •  used for long-lived service
administrative rights only use processes such as file, print, and
them when required, otherwise DNS services
accesses the system as a normal
user
 Windows Security
application and service
configuration

•  much of the configuration information

is centralized in the Registry
•  forms a database of keys and values that
may be queried and interpreted by
applications
•  registry keys can be directly modified
using the “Registry Editor”
•  more useful for making bulk changes
 Windows Security
–  other security controls
–  essential that anti-virus, anti-spyware, personal firewall, and other
malware and attack detection and handling software packages are
installed and configured
–  current generation Windows systems include basic firewall and
malware countermeasure capabilities
–  important to ensure the set of products in use are compatible

–  Windows systems also support a range of cryptographic

functions:
–  encrypting files and directories using the Encrypting File System (EFS)
–  full-disk encryption with AES using BitLocker

–  “Microsoft Baseline Security Analyzer”

–  free, easy to use tool that checks for compliance with Microsoft’s
security recommendations
 Virtualization
•  a technology that provides an abstraction
of the resources used by some software
which runs in a simulated environment
called a virtual machine (VM)
•  benefits include better efficiency in the use
of the physical system resources
•  provides support for multiple distinct
operating systems and associated
applications on one physical system
•  raises additional security concerns
 Virtualization Alternatives

application virtualization

full virtualization
allows
applications
written for one
environment to virtual machine monitor (VMM)
execute on some multiple full
other operating operating system
system instances execute
in parallel coordinates access between
hypervisor each of the guests and the actual
physical hardware resources
Native Virtualization Security Layers

User Apps User Apps User Apps

Guest O/S 1 Guest O/S 2

... Guest O/S n
Kernel Kernel Kernel

Hypervisor/ VMM
BIOS / SMM

Physical Hardware

Figure 12.2 Native Virtualization Security Layers

Hosted Virtualization Security Layers

User Apps User Apps

Guest O/S 1
... Guest O/S n
Other Kernel Kernel
User Apps
Hypervisor/ VMM

Host Operating System Kernel

BIOS / SMM

Physical Hardware

Figure 12.3 Hosted Virtualization Security Layers

 Virtualization Security Issues
•  security concerns include:
–  guest OS isolation
•  ensuring that programs executing within a guest
OS may only access and use the resources
allocated to it
–  guest OS monitoring by the hypervisor
•  which has privileged access to the programs and
data in each guest OS
–  virtualized environment security
•  particularly image and snapshot management
which attackers may attempt to view or modify
Securing Virtualization Systems
•  carefully plan the
security of the
virtualized system
•  secure all elements
organizations of a full
virtualization
using solution and
maintain their
virtualization security
•  ensure that the
should: hypervisor is
properly secured
•  restrict and protect
administrator
access to the
virtualization
solution
 Hypervisor Security
•  should be
–  secured using a process similar to securing an operating system
–  installed in an isolated environment
–  configured so that it is updated automatically
–  monitored for any signs of compromise
–  accessed only by authorized administration
•  may support both local and remote administration so must be
configured appropriately
•  remote administration access should be considered and
secured in the design of any network firewall and IDS
capability in use
•  ideally administration traffic should use a separate network
with very limited access provided from outside the
organization
Virtualization
Infrastructure access to VM

Security image and

snapshots
must be
carefully
controlled

access must
be limited to
just the
appropriate
guest

systems manage
access to hardware
resources
 Summary
•  system security planning •  Linux/Unix security
•  operating systems hardening –  patch management
–  initial setup and patching –  application configuration
–  remove unnecessary services –  users, groups, permissions
–  configure users and groups –  remote access
–  test system security –  security testing
•  application security •  windows security
–  application configuration –  patch management
–  encryption technology –  users administration and access
–  security maintenance controls
–  data backup –  application and service
–  virtualization security configuration
–  security testing
•  virtualization alternatives