Industrial Control System Cyber Incident Response Time Critical Analysis – Analysis will be performed to determine impacts based on

the threat(s) analyzed and provide options to stakeholders on ways to contain and
preserve the safety of operations.

Containment Considering Safety – Ensuring safety will be prioritized and considered

at each step of containment or change in the industrial environment, operational
This poster offers guidance on preparing for and performing cyber Incident Response (IR) for Industrial Control System (ICS) environments. technology. or engineering systems.

For the most effective industrial IR and established industrial NSM (Network Security Monitoring) program an updated ICS Asset Inventory Eradication, Recovery Considering Safety – This will involve removing threats such
as malware, adversary remote access, etc. in order to reestablish a safe and trusted
industrial process. This could require rebuilding the operating system, reloading
is best. See related ICS NSM Poster to assist with this control system network security monitoring and proactive defense. industrial software, uploading controller logic, etc.

Lessons Learned – This will involve applying knowledge, technology, personnel

resourcing, and process gaps to the ICS or IT/OT converged Cyber Incident Response
Plan.
DIFFERENCES BETWEEN IT SECURITY AND ICS SECURITY CRITICAL ICS ASSETS reduce the impact on safety and operations.
1. Preparation (test jump bag in ICS dev.) 5. Safe Containment – maintain
Information Sharing - Sharing key takeaways from incidents with the ICS community
Industrial engineering control system assets are often compared to traditional Industrial incident response should deploy proactive monitoring, baseline traffic 2. Establish a Safe and Defensible operations Incident Handlers - Cybersecurity and ICS field and technical personnel who may be and peers in the sector will help maintain the safety and reliability of operations
information technology (IT) assets. However, traditional IT assets focus on digital and system activity, and prioritize data acquisition from the critical assets in the Cyber Position 6. Safe Eradication, Recovery required to make environment and asset changes. These personnel handle evidence in facilities across other sectors globally. Key information would be in the form
data at rest or data in transit. Operating technology industrial control systems (OT/ environment. Critical industrial assets can be targeted with malware, and human 3. Integrated Identification & Detection 7. Lessons Learned acquisition, scope threats and infections, and undertake analyses, among other of adversary attack tactics, techniques and procedures observed, indicators of
ICS) manage, monitor, and control real-time engineering systems for physical input adversaries can cause negative impacts on the process by directly interacting with 4. Quick Triage on ICS impacts, safety tasks. compromise of a specific attack, and the campaign of malware capability used.
values and control output for physical actions in the real world. This is the primary the control environment using legitimate operational software with malicious intent.
Fire & Security, Safety and Law Enforcement – Teams prepared for physical first
difference between IT and OT/ICS systems, which have differing requirements, skills Several critical ICS assets are outlined below. At a minimum, access control, network ICS SAFE CYBER POSITION aid, emergency response, evacuation strategies for physical site safety, and efforts ICS INCIDENT RESPONSE PLAN
needed, and processes, including cyber incident response: traffic, and system changes should be regularly monitored, starting with these assets.
beyond the site.
Once triage of the cyber situation deems it necessary, a facility can decide to The convergence of information technology (IT) and operational technology (OT) can
Data Historian – A database that stores operational process records. It can be enter a safe and defensible cyber position. This would be determined by the level enable effective management of control systems. Convergence can improve uptimes,
abused to act as a pivot point from a compromised asset in IT to an asset in the ICS of risk deemed as acceptable by the facility owners, with input from ICS security, INDUSTRIAL CYBER INCIDENT RESPONSE PROCESS performance, quality and productivity, and access to business data about the process
network. engineering, safety, and other teams. The ICS Safe Cyber Position should be tested as to identify efficiencies – all of which leads to increased profits for those who adopt
part of the preparation phase of the ICS Incident Response Plan. It will enable more these solutions.
Engineering Workstation – A workstation that has software to program and change a in-depth threat containment and aid the eradication and recovery phases when it is
Programmable Logic Controller and other field device settings/configurations. safe to do so. Preparation While IT/OT convergence of both technology and workforce development
Safety & Planning perspectives poses unique challenges, it can also enable and drive a more realistic
Security Human Machine Interface - A visual interface between the physical process and
operators that is used to monitor, control, and change most any part of the industrial view of cyber threat detection and incident response that could further protect the
Incident industrial process.
process.
Response Disconnect/Disable connectivity to outside ICS Information Detection and
Sharing Identification
Programmable Logic Controllers - PLCs connect the physical hardware, run logic A converged incident response plan will consider available cybersecurity defenses
Isolate critical segments of the ICS
code to read the state or to change the state or a process, and interface with devices in both environments and work to reduce the impact of attacks through IT into ICS,
that make physical changes in the real world. Enable additional monitoring around rendering a more realistic view for detection and response. This can provide early
Cyber Security critical ICS assets warning signs of an attack that could impact or specifically target the industrial
Controls Skill sets
process.
Modify physical site procedures with additional
INDUSTRIAL INCIDENT RESPONSE – GETTING STARTED ICS Safe Cyber Position monitoring
The converged plan must prioritize safety, and defenders should understand security
Disable unused system and network services Lessons Evidence processes and technologies in order to detect where an adversary is in an ICS attack.
ICS asset inventory and control system network security monitoring are critical for Learned Acquisition ICS incident response plans are most effective when exercised on a frequent basic,
effective industrial incident response. Verify that the Safety Instrumented System is such as annually, through such initiatives as tabletop exercises.
System
Support functioning as expected and isolated if needed
Designs

INDUSTRIAL CYBER INCIDENT RESPONSE TRIAGE

ICS Network
ICS Asset ICS Incident Successful ICS incident response requires a clear understanding of roles,
Security Eradication,
Inventory Response responsibilities, physical safety, the engineering process, network visibility, industrial Recovery Time Critical
Monitoring protocols, and forensics capabilities. It also requires having a defensible cyber Considering Analysis SANS ICS COMMUNITY FORUM
position. Traditional incident response steps can be adapted to suit industrial control Safety
environments by considering and acquiring: Join the ICS Community Forum. Tips, tricks and Q&A to secure your ICS:
UNIQUE CONSIDERATIONS FOR ICS INCIDENT RESPONSE ICS Asset Inventory - An established ICS asset inventory of operational technology
Containment
Considering
devices and engineering assets will improve ICS Incident Response scenarios. • ystem memory from top critical ICS assets
S Safety https://ics-community.sans.org/signup
Unique Systems – Nontraditional computer systems with industrial and proprietary • Engineering field device (local) log events
Common methodologies to establish the inventory, physical inspection, passive
protocols. • Engineering field device configuration and logic (compare logic file hashes to
traffic analysis, configuration file analysis, active scanning, can be combined for
improved accuracy. For example, physical inspection takes advantage of face-to- baseline) SANS INDUSTRIAL CONTROL SYSTEM SECURITY CURRICULUM
Reliance on external vendor support – Engineering systems with external engineering • Engineering workstation field device programming software usage In industrial environments, safety to people, the environment and to the engineering
face security awareness and educational discussions on-site with engineering and
team support that may require special secure remote access.
operational teams. Augmented with passive network captures, it can create and • Engineering workstation removable media connections assets is goal #1. ICS410: ICS/SCADA Security Essentials
verify an inventory and provide network traffic to sift through for threat detection. • Operator workstations, HMI application, and system access logs
Legacy Systems – Devices that may not be suitable for patching or firmware updates, • Operator workstations and HMI remote access events Preparation & Planning – Expanding on traditional IT incident response, it will be Global Industrial Cyber Security Professional (GICSP)
or that are only available for patching or firmware updates to internal operating • Removable media connections for operator workstations critical to ensure that site safety teams are involved in cyber incident response
ICS Network Security Monitoring - Network Security Monitoring is a human-driven,
systems at infrequent times. • Remote access logs – VPN, Jumpbox, Access Control Lists across trusted zones on planning. External organizations such as ICS peers, government agencies, Information ICS456: Essentials for NERC Critical Infrastructure
proactive, and repeatable process of collection, detection, and analysis. While not
specific to ICS, NSM excels in control system networks because the environment is Firewalls, etc. Sharing and Analysis Centers (ISACs), and Computer Emergency Response Teams
Non-traditional operating systems – Purpose-built embedded and/or proprietary
usually more static and has fewer users than in traditional information technology (CERTs) will also need to be part of the overall plan. Tools for those teams in the
Protection
operating systems that are common in control environments where many traditional control system are to be tested in development environments at this stage. GIAC Critical Infrastructure Protection (GCIP)
(IT) environments. NSM is most effective with an established ICS asset inventory to
security defenses are not effective or applicable.
assist with an active approach control system thread detection and drives industrial ICS INCIDENT RESPONSE ROLES
Integrated Detection and Identification - Incident response teams will work with
Safety of people – The main goal for control systems is not confidentiality, integrity,
incident response to reduce impacts to operations, and safety of people, the
Conduct specific ICS tabletop scenarios with key teams to reinforce safety, roles, and other ICS security and engineering personnel on network security monitoring. Threat
ICS515: ICS Visibility, Detection, and Response
environment and engineering assets.
or availability, but rather safety, then integrity to trust operations, and availability. responsibilities. Effective industrial incident response teams have technical IT and identification can be conducted based on consuming and applying threat intelligence GIAC Response and Industrial Defense (GRID)
ICS Incident Response - NSM will lead to ICS Incident Response. An effective ICS IR ICS cybersecurity, engineering, overall facility, and safety backgrounds. to find and identify threats and impacts to systems and components for operations.
Protection of physical assets – Control systems that use physical components to
change the physical world. Impacts such as a cyber attack could result in physical
Plan will incorporate safety decisions at each step, has outlined communications
Incident Response Director - Interfaces with executive leadership team on the status Evidence Acquisition – Teams will use already-tested and deployed or available
ICS612: ICS Cybersecurity In-Depth
plans, a designated war room, out of band communication, a contact list of key
damage, safety implications, and environmental impacts. of an incident, resources, impacts, and options to maintain operations and safety. tools to quickly acquire meaningful forensics data from critical ICS assets to help
personnel - safety and engineering teams etc. Traditional IR steps need to be
adapted to consider safety impacts at each step in industrial environments: Lead Responder - Guides incident response personnel and quick triage/impact determine threats.
timeline analysis, and advises the Incident Response Director on available actions to
 Validation – Tabletop exercises validate readiness by comparing optimal defense
WHERE IS THE ADVERSARY IN THE ATTACK ICS INCIDENT RESPONSE TABLETOPS controls against existing controls. Areas in need of improvement are identified 1. Select one of the presented realistic ICS Incident Response Tabletop Scenarios
in industrial incident response plans and security and safety playbooks. for the organization’s next ICS IR exercise.
After reviewing detection data from NSM and gaining an initial understanding of An incident response tabletop is a paper-based exercise that facilitates security Simultaneously, tabletops help train both new and established team members about 2. Mature the process by creating specific scenarios based on the organizations’ ICS
the malicious actions, incident response steps will be determined by where the discussions across several teams and focuses on existing preparedness. A tabletop the industrial process and ICS-specific security. threat landscape by leveraging ICS threat intelligence, internal or external gap
adversary or threat is, what the impact has been already, and what the potential exercise can help verify deployed security technologies, controls, event monitoring, assessments, compliance reports, etc.
impact is moving forward for control system operations and safety. An essential and security processes to help identify areas for improvement. Beyond traditional Situational Awareness and Team Building – Reviewing threat intelligence with the 3. Custom scenarios should consider cyber to physical safety risks at every step
question is: “How far along is the adversary in the attack?” IT incident response tabletops, industrial tabletops must consider additional teams, teams involved educates them about adversary capabilities and attack techniques. and include the operational top critical ICS assets.
controls, and environments built for industrial operations, which have a different Regularly performing tabletops establishes and strengthens cross-departmental 4. Involve as many teams as practical, including Safety, Process Controls
Is the adversary stealing sensitive data to build a harmful industrial attack? mission than IT. relationships needed for incident response events that could span multiple Engineering, Operators, ICS Network Architects, ICS Security, Plant Management,
industrial sites across large geographic regions. etc.
Regularly conducted incident response tabletop exercises as part of a mature 5. Discuss, learn, act, and repeat. “ICS Defense Is Doable!”
ICS Security Program serve to identify weak points in security efforts and enable Practical Defense Actions – Tabletop exercises can identify gaps in such critical areas
Is the adversary attempting to move laterally towards the control environment? proactive defense to address the range of threats. as threat detection, data source collection, log correlation, network segmentation
changes, access control updates, security and safety process changes, and the ICS INCIDENT RESPONSE JUMPBAG
communication of roles and responsibilities. Effectiveness in all these areas is key
for a mature program. The results of tabletop exercises will directly improve overall The objective in industrial environments during a cyber incident is to maintain safety
Is the adversary attempting to elevate permissions to maintain a foothold in the control environment? response time, reduce impacts on the engineering process, and increase safety. and operations. Use these tools for quick analysis and triage to understand the
Resilience against Ransomware Ability to detect APTs using threat(s), operational impacts, and present options to facility owners to minimize loss
that could impact safety and modern attack methodologies and ensure safety. Store Jump Bag (ideally rolling protective cases) at critical site(s)
operations? targeting critical infrastructure? or deploy them with the IR team as they conduct IR.
Is the adversary attempting to enumerate and map out the control network? INDUSTRIAL IR TABLETOPS KICK-START GUIDE
• Data acquisition tools (prioritize memory)
ICS Operational Cyber • Laptops with Security Onion, REMnux, SIFT
Incident Response • Identify detection gaps • Baseline images of critical ICS assets
Is the adversary attempting to communicate with field devices to disable safety protections or affect quality assurance? Preparedness Validate • Improved understanding of • Hashes of field device logic/configuration files
Incident operations • Log, packet analysis, and timeline tools
Critical ICS assets Possible to run the industrial Response • Training • Approved digital camera (no photo metadata)
protected, regularly monitored process in manual mode, Readiness • Hardcopy ICS incident response playbooks, network diagrams
Is the adversary attempting to disrupt, manipulate, or damage physical assets or cause harm to people or the environment? to enable safety of people and completely isolated from IT? • Site physical safety training certificates
production? • Network/converter cables (USB <-> Serial)
Awareness - • Contact list for safety, engineering, integrators, security, emergency response
ICS Attacks team
Has the adversary established a C2 and is the adversary enumerating the control network, laterally moving in the environment, attempting to access a Adversary • Out-of-band communications, handheld radios on site
Human Machine Interface, or accessing PLCs or other field devices on programming service ports? ICS incident response tabletops are much like the pre-game practice drills that Capabilities • Communicate & • Forensically clean USBs, external drives
sports teams run before a game. Like pre-game drills, ICS incident response designate roles and • CD-ROM drives and discs
scenarios are designed to test all that will be needed once the game begins. In this Improved responsibilities for ICS • Personal protective equipment (PPE) for safety
case, however, the game is the serious business of cybersecurity, and it requires ICS Detection & security • Malware analysis tools (static, automated)
defense capabilities, safety processes, and cyber preparedness. These proactive Safety • Focus sessions with
exercises test the effectiveness of an ICS Security Program prior to an attack. several teams –
WHEN TO INITIATIVE ICS INCIDENT RESPONSE Convergence
Tabletops are conducted in roundtable discussions guided by an Incident Response • Educate Other Teams ICS INCIDENT RESPONSE MUST-HAVES
Plan, knowledge of the engineering processes, and an understanding of the existing • Building cross-team
Analysis drawing on ICS NSM will contribute to escalating the factors that will ultimately determine when industrial incident response steps are to be invoked. Use the
ICS security defenses. Weak points are identified and assigned to be addressed relationships for Incident 1. ICS-Specific Incident Response Plan
questions listed above to help determine the potential risk that an intrusion will disrupt the industrial process or safety, and to understand the progression of an attack
immediately to strengthen the program. ICS incident response tabletops provide a Response Execute realistic tabletop exercises driven by sector-specific threat intelligence or
already in progress. Answers to these questions will help drive defense steps and shift to potential incident response steps.
high return on investment in several important areas: gaps identified in your facility.

2. ICS-Specific Network Security Monitoring

Ensure “plant floor” network visibility with ICS deep-packet inspection to drive
incident response or proactive threat hunting. Network visibility capabilities should
go beyond just querying about indicators of compromise and include capabilities to
assist with analyzing threat tradecraft.

Exfiltration of Physical Damage

3. Trained ICS-Specific Security Defenders

Malicious Code, Manipulation of

Trained ICS cybersecurity personnel who understand the nuances between

Loss of visibility of Loss of process

traditional IT and ICS security, the ICS mission, safety, the engineering process, and
sensitive to assets or safety
ICS protocols and active defense procedures.

Unauthorized control system

industrial system control process controls concerns
Access Detected operations ICS INCIDENT RESPONSE IN PRACTICE
information Successful ICS incident response requires a clear understanding of roles,
responsibilities, physical safety, the engineering protocols and process, network
visibility, detection, and forensics capabilities. Facilities benefit when having a
tested safe defensible cyber position. Consider adapting traditional IR steps to suit
industrial control environments:

• Acquire forensics data from key ICS assets

• Quickly triage to understand the threat via static or automated malware analysis
Malicious Code, Exfiltration of sensitive Loss of visibility of control Loss of process controls Manipulation of control Physical Damage to assets or • Execute the Safe Cyber Position
• Contain threats while running operations
Unauthorized Access industrial system process A cyber incident affecting the ability to
system operations safety concerns • Eradicate when its safe for operations
Detected information A cyber incident affecting the ability to view
change the state of the physical process.
The abuse of internal native system compo- An incident affecting the physical properties
• Analyze the impact of any reliance on external vendors and IT
An example would be an oil refinery not • Apply lessons learned to the ICS Incident Response Plan
the state of the physical process. An ex- nents or protocols such as Human Machine or integrity of physical assets, or introducing • Regularly conduct ICS incident response tabletop exercises
Installation or execution of malicious An incident exfiltrating sensitive control being able to safely shut down its crude oil
ample would be a power generating facility Interface (HMI) commands, and ICS proto- a potential physical safety impact to plant • Examine the connectivity and isolation of legacy devices
software. An example would be an adver- system information that could be used to do distillation process or maintain pressure in
not being able to view the current system cols such as EthernetIP, ModbusTCP, DNP3, operators, workers, and/or on-site visitors, • Determine operational impacts
sary gaining physical or logical access to a harm. An example would be the identifica- a pipeline.
load or the current power grid operating 61850, OPC, etc. An example would be the contractors, etc. • Develop countermeasures
network, system, or data without authoriza- tion of ICS field device ladder logic, control
frequency to maintain 60Hz. use of DNP3 to send unauthenticated “open • Use indicator “hits” to scope infection
tion, or introducing a cyber containment system configuration, or historian database
breaker” commands to remote terminal unit • Compare production and baselined configs to detect tampering in controllers etc.
that could impact process control views or entries being copied off a network. This is
field devices to open electric circuit breakers • Present analysis and options (blocking C2 access, running ICS in manual mode,
controls. an indication of a follow-on (non-immediate
to cut power. removing remote access, etc.) to fight through the attack (contain/eradicate)
but potentially imminent) targeted ICS at-
tack. • Identify and apply lessons learned (e.g., correct gaps in evidence acquisition,
deploy additional ICS network visibility, detection capabilities, determine
whether threats are malware or human adversaries).

PERFORMING INDUSTRIAL INCIDENT RESPONSE TOP 3 ICS TABLETOP EXERCISES SCENARIO #2: SCENARIO #3:
TABLETOPS Physical Access to Cyber Access Event Ransomware on IT or ICS/OT networks
Regular Incident Response (IR) tabletop exercises are part of a mature ICS Security
Program and works to identify strong and weak points in ICS defense efforts. The

Remember,
Planning – Planning time will vary depending on team size, the scenario, resources, The Physical Security team notices a hole cut into the physical security perimeter ICS Operator Workstations in a control center(s) are infected with Ransomware and
scenarios designed to test ICS defense capabilities and cyber preparedness are – the fence surrounding a remote facility. The Physical Security team investigates are inoperable to view or control the industrial process. Alternatively, the IT business
and other factors, but it typically can take anywhere from a few days up to a month.
critical. Here are ICS IR Tabletops for consideration. and determines the physical attack could be a two-part attack. Physical access was network is inoperable due to a ransomware infection in the enterprise – critical ICS
Even a planning phase of just 2 to 5 days is enough to provide value in the outcome.
Spend time up front properly selecting realistic scenarios for your environment and gained, then attackers pivoted to a cyber-attack as a containment was introduced process application such as industrial billing, shipping logistics applications are
selecting the right teams. Include as many team players and observers as is practical. SCENARIO #1: into the control network at remote site. Traditional break-ins have been observed to inoperable.
Human Machine Interface Hi-Jack - On-screen Suspected Activity. be for monetary value, such as copper theft in Electric utilities. Some critical remote

“ICS Defense
ICS Teams – Include all teams that are practical to involve. Invite observers to listen ICS sites could be vulnerable to physical and a physical-cyber-attack. This also DISCUSSION: Does ICS rely on IT, and to what extent? Is it possible to island ICS from
to the discussions for training purposes. Start with the following: Human Machine Interface Operators notice the on-screen mouse moving and presents a safety concern for workers in remote facilities such as electric substations, IT in a safe cyber defensive position?
• Safety – Include the on-site safety and emergency response team. clicking on different control buttons on the HMI, which is not consistent with normal switching yards, oil and gas valve stations, fuel storage facilities such as marine
• Physical Security – Include the on-site facility physical security team. operations or a scheduled change or safety emergency. terminals, etc. TEAMS: IT, IT Security, ICS Security, Engineering, Operators, Safety.
• Compliance – Ensure that legal and regulatory compliance requirements are met.
• Cybersecurity – Since cybersecurity drives the scenario, participants must DISCUSSION: Which accounts, and individuals have access to HMIs for local or remote DISCUSSION: Physical security at remote sites could be the most vulnerable ICS PROTECTION: Email security (if IT is infected with the common email phishing vector),

Is Doable!”
understand the defenses and the Incident Response Plan, the technologies and access? facilities (substations, oil/gas storage facilities, valve stations, etc.). whitelisting on ICS endpoints, IT - ICS Network Segmentation (Purdue Network
the industrial operations process, protocols, critical assets, the network layout, Architecture).
etc. TEAMS: Engineering, Operators, ICS Security, Network Architects. TEAMS: Physical Security Teams, Engineering, Cybersecurity, Safety.
• Engineering – Include process control and field device technicians. DETECTION: ICS-specific endpoint protection, ICS NSM (lateral movement)
• Operators – These are the persons who control the process via remote and PROTECTION: Purdue Network Architecture, process control, operators having a PROTECTION: Security guards stationed at site(s) or security checks on rotation.
embedded HMIs, etc. process for reporting cyber events. RESPONSE: Is it possible or feasible to run the ICS process in manual mode from
• Management – Management and director-level stakeholders for all teams DETECTION: Physical door alarms, surveillance cameras, rotating security guards, etc. embedded HMIs on the plant floor in the event the primary HMIs are inoperable
involved need to have an awareness and understanding of ICS cybersecurity risk, DETECTION: Secure remote access event monitoring - External->Internal, Internal- due to Ransomware or another threat? It may be possible to respond by cutting or
impacts, protections, budget, resourcing, etc. >Internal - RDP, Multi-factor authentication, use of a jump box in ICS DMZ (Purdue RESPONSE: Roll trucks to site, law enforcement. limiting network segment communication for containment while fighting through the
Level 3, etc.). attack.
SAFETY: A concern for adversaries in dangerous life-threatening situations and
workers on-site in the event of a break-in. ICSPS_v1.0_0XXXXXXXX-21 - This poster was created by Dean C. Parsons.
RESPONSE: Disable remote access, run ICS on plant floor via embedded HMIs, ©2021 Dean C. Parsons. All Rights Reserved.
investigate NSM network traffic patterns, enable islanding from Internet, IT, etc.

SANS ICS RESOURCES ics.sans.org ics-community.sans.org/signup @SANSICS Free and open-source tools for ICS available at ControlThings.io SANS ICS Security SANS ICS