Operational Guide

OpGuide Name: Visit Manager Workcard Viewer - AWS

Syscode/Block Code: TOPEGDVMWC

Consumer Code DL

ia l
DR Tier: Sev 2

nt
Recovery Priority: RP1.2

i d e
n f
C o
 OpGuide v7.1 - 2023

Contents
Operational Guide ........................................................................................................................ 1

History ....................................................................................................................................... 4

OpGuide History ....................................................................................................................... 4

Purpose and Audience of this Document .......................................................................................... 5

System Profile ............................................................................................................................. 6

System Information................................................................................................................... 6

System Priority Level ................................................................................................................. 6

ia l
System Description ................................................................................................................... 6

Business Description ............................................................................................................... 6

nt
Business Impact ..................................................................................................................... 7

Business Process Workaround if Application is Down .................................................................... 7

d e
Technical Description .............................................................................................................. 7

Technical Workaround if Application is Down .............................................................................. 7

i
f
User Profile ................................................................................................................................. 8

n
System Availability....................................................................................................................... 8

o
Infrastructure .............................................................................................................................. 9

Infrastructure Components and Processes .................................................................................... 9

C
Infrastructure Details for Production Environment .......................................................................... 9

Monitoring ................................................................................................................................ 10

Operational Tools ....................................................................................................................... 11

Failover Automation ................................................................................................................... 12

Continuity Patrol Runbook – on Prem ......................................................................................... 12

AWS Failover .......................................................................................................................... 12

System Diagram ........................................................................................................................ 14

Deployment Architecture .......................................................................................................... 14

Interface Architecture .............................................................................................................. 14

 OpGuide v7.1 - 2023
Failover System Architecture .................................................................................................... 14

Physical Architecture ............................................................................................................... 14

Logical Drawing ...................................................................................................................... 14

System Contacts........................................................................................................................ 15

Call Tree Contacts List ............................................................................................................. 15

DR Core Recovery Team .......................................................................................................... 15

DR Extended Recovery Team .................................................................................................... 16

Business Contacts ................................................................................................................... 16

Log Files and Troubleshooting Checklist ........................................................................................ 17

l
Log Files ................................................................................................................................ 17

ia
Troubleshooting Checklist ......................................................................................................... 18

t
Non-Production Environment ....................................................................................................... 22

n
Known Issues & Impact .............................................................................................................. 25

Known Issues ......................................................................................................................... 25

i d e
Business Impact ..................................................................................................................... 25

Business Process Workaround for listed issues............................................................................. 25

f
Technical Workaround to minimize impact .................................................................................. 25

n
C o
 OpGuide v7.1 - 2023

History

OpGuide History

Date Version Editor Notes

6/15/2017 Rex McKee Create Op Plan Templates

8/18/2017 Bob Favre Updated for OpGuide Template

7/28/2021 RA Fraker Updates for Zulu

3/13/23 7.0 Kim Prindle/SRE Updates for AWS

4/17/23 7.1 Knowledge

Management
l
Updates to System Diagram section

ia
nt
i d e
n f
C o
 OpGuide v7.1 - 2023

Purpose and Audience of this Document

Purpose:

The OpGuide is to be used as a reference for configuration information, known issues

and recovery steps needed to mitigate issues and restore service during an incident.

Audience:

Operational Support Teams (e.g., Availability, Incident and Crisis Management, System
Monitoring, Network Monitoring).

ia l
nt
i d e
n f
C o
 OpGuide v7.1 - 2023

System Profile

System Information
System Aliases AMDS-DS, PPt

Business Function The Visit Manager application provides access to work

cards and work packages for the aircraft mechanics
worldwide 24 x 7. This application is used to print
aircraft visit work packages for Delta aircraft for Line
Maintenance, Base Maintenance, and Outside Service
Repairs

Vice President Portfolio Software Engineering

ia l
t
Facilities Usage Comment

Other (Named) Prod / DR

e n
Vendor hosted AWS

f i d
n
System Priority Level

o
Severity Level Sev 2

C Support
Style
Standard TechOps

System Description
Business Description
The AMDS-DS application provides access to work cards and work packages for the aircraft mechanics
worldwide 24 x 7. This application is used to print aircraft visit work packages for Delta aircraft for Line
Maintenance, Base Maintenance, and Outside Service Repairs.
 OpGuide v7.1 - 2023

Business Impact

The airline maintenance technicians would not have access to the maintenance instructions needed to
perform the maintenance on the aircraft.

Business Process Workaround if Application is Down

Airline maintenance technicians could view the aircraft maintenance manuals for maintenance
procedures located within the SkyView/OTIS application. But any customization that are on the work
cards would not be displayed in the manuals.

l
Technical Description

ia
Vendor hosted AWS

Technical Workaround if Application is Down

nt
Failover to DR region is the only workcaround

i d e
n f
C o
 OpGuide v7.1 - 2023

User Profile
Type of Users TechOps Aircraft Maintenance Technicians

System User Locations Worldwide

Number of Total Users ~10,000

Number of Concurrent Users ~2,000

System Availability
Operational Windows – Eastern Time

indicating times (e.g. 0100, 0900, 1300, and 1800).

Monday Tuesday Wednesday Thursday

ia l
Identify the primary business hours the system is in use. Please use the 24-hour Eastern Time clock when

Friday Saturday Sunday

0000

2359
0000

2359
0000

2359

nt
0000

2359
0000

2359
0000

2359
0000

2359

Maintenance Windows – Eastern Time

i d e
Identify the start and end times for any backup and maintenance activities for this system. Place a check

f
mark in the corresponding “yes” or “no” box indicating whether the system is available during those times.
Please use the 24-hour Eastern Time clock when indicating times.

Activity

No Mtc.

System
o
Monday

Yes
nTuesday

Yes
Wednesday

Yes
Thursday

Yes
Friday

Yes
Saturday

Yes
Sunday

Yes
available?

C No No No No No No No
 OpGuide v7.1 - 2023

Infrastructure

Infrastructure Components and Processes

Provide the Infrastructure Components and Processes for your system.

Infrastructure Details for Production Environment

Application Details Applicable Technology/Platform

(Yes/ No)
1. AWS Account Name No N/A – Vendor hosted AWS

2. ROSA Cluster Name No N/A – Vendor hosted AWS

3. Application Framework No N/A – Vendor hosted AWS

4. Application servers No Prod Server

Names

ia l DR Server Names

5. Serverless/Lambda Functions No

nt N/A – Vendor hosted AWS

e
6. Application type No N/A – Vendor hosted AWS

7. Application stage No N/A – Vendor hosted AWS

8. Operating system

9. Software/Language

f i d No

No
N/A – Vendor hosted AWS

N/A – Vendor hosted AWS

o
technologies used

11. UI session management

n
10. Tools and any Integration No

No
N/A – Vendor hosted AWS

N/A – Vendor hosted AWS

C
approach

12. Data store

13. Version control tool

14. Any Vendors, Version

No

No

No
N/A – Vendor hosted AWS

N/A – Vendor hosted AWS

N/A – Vendor hosted AWS

15. User Authentication Scheme No N/A – Vendor hosted AWS

16. High Availability No N/A – Vendor hosted AWS

17. Throughputs No N/A – Vendor hosted AWS

 OpGuide v7.1 - 2023

Monitoring

Identify and describe the method used for Monitoring (i.e. Operating Systems, Application Servers,
Network Monitoring, databases and other). Identify and describe the tools used to troubleshoot and
diagnose problems. (I.e. NetIQ, UIM, Dynatrace, Sumo, CloudWatch, X-ray, Travelport and other).

*Must have (at a min): Application, Infrastructure, and Logging Monitoring

What is monitored &

Monitoring Tools Alert Destination
alerted

1. OS Monitoring N/A – Vendor

hosted AWS
N/A – Vendor hosted
2. Application Synthetic
Tx Monitoring
AWS

3. Application Log
Monitoring
Example: Sumo Logic
Standards

ia l
Example: Reference Logging Example: PagerDuty, TCC
Availability, ServiceNow

t
N/A – Vendor hosted
4. Application Server
Monitoring
AWS

n
(availability)
N/A – Vendor hosted
5. Database Monitoring

e
AWS
N/A – Vendor hosted
6. Application Response

d
Time Monitoring
AWS

7. File System
Monitoring

8. Application Profiling
AWS

n f i
N/A – Vendor hosted

N/A – Vendor

o
hosted AWS
N/A – Vendor hosted
9. Transaction

C
Monitoring
AWS

N/A – Vendor hosted

10. Business Monitoring
AWS
11. Transactional N/A – Vendor
Throughput hosted AWS
Monitoring

12. Syslog/Logging N/A – Vendor

hosted AWS

13. Capacity data N/A – Vendor

hosted AWS
 OpGuide v7.1 - 2023

Operational Tools
Identify and describe the method used for software delivery (i.e. RET Process Model, OS/390, VM, Radia or
other). Identify and describe the tools used to troubleshoot and diagnose problems. (i.e. Cisco Viewer,
Load Manager, Netcool, Openview). Describe the tool’s connectivity to the system, how alerts are
generated, how the monitoring occurs, and which group performs monitoring.

Type Tool Description

Software Delivery N/A – Vendor hosted

AWS

System Monitoring N/A – Vendor hosted

AWS

ia l
nt
i d e
n f
C o
 OpGuide v7.1 - 2023

Failover Automation
Continuity Patrol Runbook – on Prem
Steps for executing the Go To Green orchestration: http://dpi948.delta.com/cobprod:/1%20-
%20Continuity%20of%20Business/Systems/Go%20To%20Green%20High%20Impact/ContinuityPatrolRunBook.do
c

[x] Stand alone

[] Limited Share Other Impacted Apps: __________________

[] Highly Shared Other Impacted Apps: __________________

Button Description:

N/A – Vendor hosted AWS

ia l
Execution/Testing Results [x] Fully tested [] Limited testing [] No testing

1.
Date
RFC Number
(Manual/Automated)

N/A – Vendor hosted

nt
Total Time - execution until
service is restored
Results

2.
AWS

i d e
n f
High Level Description of Execution steps of Failover Procedure (includes assignment groups and
Organization SME’s for support purposes)

1.

2.
Step#

C o Description

N/A – Vendor hosted

AWS
Standalone /Limit Shared
/Highly Shared
Assignment Group

AWS Failover
Steps for executing failover

Step# Description Standalone /Limit Shared Assignment Group

/Highly Shared

1. N/A – Vendor hosted AWS Example: Standalone Example: list assignment

group
 OpGuide v7.1 - 2023

AWS Failover
Steps for executing failover

Step# Description Standalone /Limit Shared Assignment Group

/Highly Shared

2.

ia l
nt
i d e
n f
C o
 OpGuide v7.1 - 2023

System Diagram
Provide the below Architecture Diagrams for your system. These diagrams Must Include Server/Device
Name Details. Provide Documentum links to these documents below, do not paste into this document.
Email [email protected] with the location/link to this guide.

Deployment Architecture
http://dpi948.delta.com/cobprod:/1%20-
%20Continuity%20of%20Business/Systems/Visit%20Manager%20Workcard%20Viewer-
AWS/Supplemental%20Documentation/PPt_CA_LA_Arch_Diagram.vsdx

Interface Architecture

http://dpi948.delta.com/cobprod:/1%20-

l
%20Continuity%20of%20Business/Systems/Visit%20Manager%20Workcard%20Viewer-
AWS/Supplemental%20Documentation/PPt_CA_LA_Arch_Diagram.vsdx

ia
Failover System Architecture

http://dpi948.delta.com/cobprod:/1%20-

nt
%20Continuity%20of%20Business/Systems/Visit%20Manager%20Workcard%20Viewer-
AWS/Supplemental%20Documentation/PPt_CA_LA_Arch_Diagram.vsdx

Physical Architecture

i d
http://dpi948.delta.com/cobprod:/1%20- e
f
%20Continuity%20of%20Business/Systems/Visit%20Manager%20Workcard%20Viewer-
AWS/Supplemental%20Documentation/PPt_CA_LA_Arch_Diagram.vsdx

Logical Drawing
n
o
http://dpi948.delta.com/cobprod:/1%20-
%20Continuity%20of%20Business/Systems/Visit%20Manager%20Workcard%20Viewer-

C
AWS/Supplemental%20Documentation/PPt_CA_LA_Arch_Diagram.vsdx
 OpGuide v7.1 - 2023

System Contacts

Call Tree Contacts List

Visit Manager Workcard Viewer - AWS escalation tree

Use Service Now for telephone numbers

Order Assignment Group Involvement Area Notes

to call

1 AMISUP TechOps support AMISUP primary – 661-793-3488

AMISUP secondary – 661-793-3489
AMISUP tertiary – 661-793-3490

ia l
DR Core Recovery Team

nt
The DR Core Recovery Team list is to be used in the event of a disaster. List the SME’s who are the most
knowledgeable about all components of the System and will begin the recovery efforts. The list must contain

Richmond (Axel) Racelis (Primary) Sr. Developer

e
a Primary and Secondary resource. This section should include individual names and contact information. Do
not refer to the Service Manager contact list for this section. Provide Office, Home, and Mobile numbers

Name

if d
Role Telephone Numbers

612-266-3138 - office

n
763-233-3643 - home
952-221-4749 - cell

o
Wanda Wirt (Secondary) Sr. Developer 612-266-3182 - Office
651-322-2331 - Home

C
651-398-1829 - Cell
 OpGuide v7.1 - 2023

DR Extended Recovery Team

The DR Extended Recovery Team list is to be used in the event of a disaster. These are Team members who
may be needed to assist with recovery efforts in the event of a disaster. This section should include backup
SME’s and Managers. Do not refer to the Service Manager contact list for this section.

Name Role Telephone Numbers

Nate Lorentz Sr. Analyst 612-266-3064 - office

651-324-4627 - cell
Dean Kraus IT Manager 612-266-3050 - Office
612-310-2998 - Cell
Bob Thoe Sr. System Engineer 612-266-3550 – office
612-417-2717 - Cell

Business Contacts

ia l
Business contacts include your primary business contact and business users who provide acceptance or

t
integration testing assistance.

Contact Name Role and Organization Telephone Numbers

Tuvayas Duckworth

James Griffie
Product Owner

e n
Manager – Tech Procedures
404-245-8971 - Cell

404-714-1326 – Office

d
678-857-9119 - Cell

Nazim Raza

Brian Winters

n f i Product Owner

Director, Line Mtc

404-714-5540 office
404-218-4041 cell
404-714-5942 office
404-583-1012 cell

C o
 OpGuide v7.1 - 2023

Log Files and Troubleshooting Checklist

Log Files
Application Log File Location Content Description &
Name Error. Patterns

(Split logs in this list by

component and service like (Please add what regex pattern to
front end, api, web, business look for errors and warnings)
layer, db layer or cache.)
N/A – Vendor hosted AWS

ia l
nt
i d e
n f
C o
 OpGuide v7.1 - 2023

Troubleshooting Checklist
Use this template to record the metrics you are monitoring and actions you will take when those
metrics degrade. Include expected behavior, observed symptoms, and steps to check / monitor
escalation path for technical assistance. ZAP guidance on building troubleshooting guides will be
coming soon. Examples of metrics to add to this list can be found below.

Layer/Category What to look for? Symptoms Escalation Runbook Steps

(metric and expected Path (including
behavior) Tool/Command)
Add additional layers
N/A – Vendor hosted
for SLI’s
AWS
• Latency
• Availability

l
• Error Rate
And actions to take

ia
when things fail
across the split by

t
components – what to
look for to

n
debug/troubleshoot

List upstream and

downstream
systems/dependencies

i d e
Ref: SLO+SLA - Delta
CCOE Documentation

n f
o
To each alert that N/A – Vendor hosted
comes out of system, AWS
what to look for to

C
troubleshoot

Add layers for cloud- N/A – Vendor hosted

native and modern AWS
containerized
parameters to reflect
the same.

Ref: SLI-AWS native

components - Delta
CCOE Documentation

SLI-ROSA-K8s - Delta
CCOE Documentation

Example SLIs N/A – Vendor hosted

AWS
 OpGuide v7.1 - 2023

Troubleshooting Checklist
Use this template to record the metrics you are monitoring and actions you will take when those
metrics degrade. Include expected behavior, observed symptoms, and steps to check / monitor
escalation path for technical assistance. ZAP guidance on building troubleshooting guides will be
coming soon. Examples of metrics to add to this list can be found below.

Layer/Category What to look for? Symptoms Escalation Runbook Steps

(metric and expected Path (including
behavior) Tool/Command)

OS Parameters - Memory N/A – Vendor

hosted AWS
- CPU%

- Disk space

-
Disk i/o

Traffic

ia l
- Errpt
messages(AIX)/
/var/adm/msg logs
in Linux

nt
Process (zombie/ hung,

d
consuming more cpu)

i e
f
Business Throughput Any sudden increase/ N/A – Vendor
decrease in business hosted AWS
throughout

Saturation

o
(How much of the
system's upper
n
Interoperability N/A – Vendor
hosted AWS

C
threshold are your
workloads utilizing)

Tx Response Time Sudden degradation in

response time of
N/A – Vendor
hosted AWS
monitored transactions

Availability of URL’s URL Availability N/A – Vendor

hosted AWS
(Include response
actions for each 5xx,
4xx, and 3xx error
codes)
 OpGuide v7.1 - 2023

Troubleshooting Checklist
Use this template to record the metrics you are monitoring and actions you will take when those
metrics degrade. Include expected behavior, observed symptoms, and steps to check / monitor
escalation path for technical assistance. ZAP guidance on building troubleshooting guides will be
coming soon. Examples of metrics to add to this list can be found below.

Layer/Category What to look for? Symptoms Escalation Runbook Steps

(metric and expected Path (including
behavior) Tool/Command)

Network bandwidth% Sudden change in N/A – Vendor

pattern of traffic-in and hosted AWS
traffic-out

Network Quality Load Balancer # of N/A – Vendor

l
sessions hosted AWS

CPU% of Load Balancer

Service Response
Packet drops

Firewall packet drops

nt
N/A – Vendor
ia
e
Time hosted AWS

d
Service Failures N/A – Vendor

i
hosted AWS

f
N/A – Vendor
hosted AWS

DB availability

DB performance

o n
DB alerts / traps

Load profile
N/A – Vendor
hosted AWS

N/A – Vendor
hosted AWS

C
(Include DB lock
actions, RW lock
actions, and query or
general response
degradation actions)

Authentication Issues N/A – Vendor

hosted AWS
 OpGuide v7.1 - 2023

Troubleshooting Checklist
Use this template to record the metrics you are monitoring and actions you will take when those
metrics degrade. Include expected behavior, observed symptoms, and steps to check / monitor
escalation path for technical assistance. ZAP guidance on building troubleshooting guides will be
coming soon. Examples of metrics to add to this list can be found below.

Layer/Category What to look for? Symptoms Escalation Runbook Steps

(metric and expected Path (including
behavior) Tool/Command)

Business Functionality Application error logs N/A – Vendor

failure hosted AWS

(Include specific
actions needed by

l
response code, or
potential DB record
entries for specific

ia
events)

Application fatal
errors

ITA Response Time

Application error logs

River Tx Response t
N/A – Vendor
hosted AWS

n
N/A – Vendor

e
hosted AWS

d
Sys Logs Any error messages N/A – Vendor

i
hosted AWS

Access Logs

n f
(Application access logs
and any anomalies
needing specific actions
N/A – Vendor
hosted AWS

Storage Issues

C o diagnosed or addressed)

Check for iops pattern &

transfer rate from
Storage console, look for
any change vis-à-vis
established baseline
N/A – Vendor
hosted AWS
 OpGuide v7.1 - 2023

Non-Production Environment
Provide the Infrastructure Non-Production components and processes details including Development
environment, System Integration Environment and Test Environment. Available for this system.

Detailed Non-Production Configuration – DB2

Supported Components Dev Configuration SI Configuration Test Configuration

N/A – Vendor hosted AWS

Detailed Non-Production Configuration – ORACLE - <Database Name>

Supported Components

N/A – Vendor hosted AWS

Dev Configuration

l
SI Configuration

ia
Test Configuration

nt
Detailed Non-Production Configuration – SQL - <Database Name>

Components

Server name(s)

d e
Dev Configuration

N/A – Vendor hosted AWS

i
SI Configuration Test Configuration

Server Location(s)

n f
N/A – Vendor hosted AWS

o
Detailed Non-Production Configuration – SYBASE - <Server Name>

Supported Components Dev Configuration SI Configuration Test Configuration

C
N/A – Vendor hosted AWS

Detailed Non-Production Configuration – MidTier AIX

Components Dev Configuration SI Configuration Test Configuration

Server Name(s) N/A – Vendor hosted AWS

Server Location(s) N/A – Vendor hosted AWS

 OpGuide v7.1 - 2023

Detailed Non-Production Configuration – MidTier HP

Components Dev Configuration SI Configuration Test Configuration

Server name(s) N/A – Vendor hosted AWS

Server Location(s) N/A – Vendor hosted AWS

Machine Configuration N/A – Vendor hosted AWS

Operating System N/A – Vendor hosted AWS

Detailed Non-Production Configuration – MidTier Linux

Components Dev Configuration SI Configuration Test Configuration

Server name(s) N/A – Vendor hosted AWS

ia l
t
Server Location(s) N/A – Vendor hosted AWS

n
Machine Configuration N/A – Vendor hosted AWS

Operating System N/A – Vendor hosted AWS

Components
if d
Dev Configuration
e
Detailed Non-Production Configuration – MidTier SUN

SI Configuration Test Configuration

Server name(s)

Server Location(s)

o n
N/A – Vendor hosted AWS

N/A – Vendor hosted AWS

C
Machine Configuration

Operating System
N/A – Vendor hosted AWS

N/A – Vendor hosted AWS

Detailed Non-Production Configuration – Windows Server

Components Dev Configuration SI Configuration Test Configuration

Server name(s) N/A – Vendor hosted AWS

Server Location(s) N/A – Vendor hosted AWS

Machine Configuration N/A – Vendor hosted AWS

Operating System N/A – Vendor hosted AWS

 OpGuide v7.1 - 2023

Detailed Non-Production Configuration – MQSeries OS390 Infra

Supported Components Dev Configuration SI Configuration Test Configuration

N/A – Vendor hosted AWS

Detailed Non-Production Configuration-MQ Series UNIX Infra/Client

Supported Components Dev Configuration SI Configuration Test Configuration

N/A – Vendor hosted AWS

Supported Components Dev Configuration l

Detailed Non-Production Configuration-MQ Series Windows Client

ia
SI Configuration Test Configuration

N/A – Vendor hosted AWS

nt
Components

i d e
Detailed Non-Production Configuration – System Name Application

Dev Configuration SI Configuration Test Configuration

Server name(s)

Server Location(s)

n f
N/A – Vendor hosted AWS

N/A – Vendor hosted AWS

C o
 OpGuide v7.1 - 2023

Known Issues & Impact

Known Issues
List all known issues in this table – for example Single Sign-on is not supported etc.

N/A – Vendor hosted AWS

Business Impact
What is the impact to Delta’s business in this scenario? Provide 1-2 paragraphs using only business
terms. Avoid acronyms Provide a 1-2 paragraph description of how the business is impacted

N/A – Vendor hosted AWS

Business Process Workaround for listed issues

N/A – Vendor hosted AWS

ia l
Technical Workaround to minimize impact

nt
e
N/A – Vendor hosted AWS

f i d
o n
C