AppServer Configuration

Phani Mohanty
23.02.2020
 Agenda

 Application Server Basics

 Usage of AppServer in Data Protector
 Walkthrough of AppServer configuration
 Walkthrough of AppServer certificates
 Data Protector-specific Customizations
 Impact of Application Server Failure on Data Protector
 Walkthrough of AppServer logs
 Log Files and settings for troubleshooting issues

2
 Application Server Basics
 What is an Application Server ?
 Also called an AppServer
 Program that handles all application operations between users and an organization's backend business applications or
databases
 Typically used for complex transaction-based applications
 Why do we need an Application Server?
 Provides useful services like automatic transaction management, authentication, authorization and lifecycle management.
 Provide REST based APIs.
 Why Wildfly ?
 Simplifies development of enterprise applications by providing a list of services out of the box:
 JDBC connection pool to manage access to relational databases
 Messaging broker compatible with JMS specifications
 Enables asynchronous and non-blocking APIs to maximize performance of critical resources in applications
 Open source, with Red Hat JBoss Enterprise Application Platform being built on it
 Seamless integration with Keycloak
 Easy deployment of services
 Rapid release cycles(3 months), so new features and vulnerability fixes are released.
3  Built-in industry security standards
 Application Server Basics (continued)
 Which AppServer does DP use?
 Usage started with DP 8.x and has changed over releases
 DP 8.1x and DP 9.x - JBOSS 7
 Uses Sun Java 7 as underlying runtime
 DP 10.x till 10.60 - Wildfly 10 (WIldfly Core 2.x)
 Uses Oracle Java 8 as underlying runtime (Oracle Java -> OpenJDK switch in 10.30)
 DP 10.60 - Wildfly 16 (Wildfly Core 8.x)
 Uses OpenJDK Java 8 as underlying runtime
 DP 10.70 - Wildfly 18 (WIldfly Core 10.x)
 Uses OpenJDK Java 8 as underlying runtime
 How to find which AppServer version is currently in use?
 Windows
 <DP_HOME_DIR>/AppServer/bin/standalone.bat --version
 Linux
4  /opt/omni/AppServer/bin/standalone.sh --version.
 Usage in Data Protector
 How does Data Protector use the AppServer?
 To host the web-based UI (dashboard, telemetry configuration, reporting, scheduler views etc)
 Executing schedules using Quartz
 For identity management through Keycloak
 To service REST APIs
 License management
 New Reporting
 Hosting VMwareGre Plugin
 What is a service and what are some related facts to keep in mind?
 WAR files act as services used for different purposes (number of WARs changed over versions)
 Authentication and Authorization of each service are important and are described later
 What is the location of AppServer binary files?
 <DP_HOME_DIR>/Appserver (Windows) or /opt/omni/AppServer (Linux)
 What is the location of AppServer configuration files?
 <DP_SDATA_DIR>/config/server/AppServer(Windows) or /etc/opt/omni/server/AppServer
 What is the location of AppServer data files?
 <DP_SDATA_DIR>/server/AppServer (Windows) or /var/opt/omni/server/AppServer(Linux)
 What is the log location of AppServer ?
5  <DP_DATA_DIR>/log/AppServer(Windows) or /var/opt/omni/log/AppServer
 Usage in Data Protector (continued)
 How to find the DP AppServer status ?
 Named as service/daemon "hpdp-as"
 Windows
 Launch services.msc and look for display name "Data Protector Application Server" to obtain status
 Linux
 ps -eaf |grep "/opt/omni/AppServer/standalone"
 How to stop and start AppServer in DP ?
 Windows
 Launch services.msc and look for display name "Data Protector Application Server". Right click ->
"stop" or "start"
 sc stop hpdp-as ---> stop AppServer using CLI
 sc start hpdp-as --> start AppServer using CLI
 Linux
 /etc/init.d/hpdp-as stop ----> stop AppServer using CLI
 /etc/init.d/hpdp-as start ----> start AppServer using CLI

6
 Usage in Data Protector (continued)
 How to verify through a web browser
 Users can try to access this URL from any browser - https://<CMNAME>:7116
 If you can see the text "Micro Focus Data Protector" with the DP logo, it means that the AppServer is
running and responding.
 How to connect to management console
 Access http://localhost:9990/error/index_win.html and follow the given instructions

7
 Usage in Data Protector (continued)
 If a management user is not present, one can be created by running "add-user.bat" or "add-user.sh" under
AppServer bin folder.

8
 Usage in Data Protector (WAR files)

9
 Walkthrough of AppServer configuration
 Important configuration file - standalone.xml
 Windows location
 <DP_SDATA_DIR>/config/server/AppServer/standalone.xml
 Linux location
 /etc/opt/omni/server/AppServer/standalone.xml

 Services(WAR files) can be found at

 Windows
 <DP_HOME_DIR>/bin/components/
 <DP_HOME_DIR>/bin (dp-scgui.ear)
 Linux
 /opt/omni/bin/components

10
 Walkthrough of AppServer certificates
 Self-signed certificates valid for 10 years created using "omnigencert.pl" during install/upgrade
 "Custom certificates" (created by DP but signed by a trusted certificate authority like Verisign / digicert)
 Server certificates and CA certificate "cacert.pem" can be found at following location:
 Windows : <DP_SDATA_DIR>/config/server/certificates
 Linux : /etc/opt/omni/server/certificates
 Multiple subject alternative names are added during certificate creation, allowing clients to connect to
AppServer using any of these -
 [ alt_names ]
 DNS.1 = <FQDN CM>
 DNS.2 = <Small Hostname>
 DNS.3 = localhost
 Certificate can be generated for additional alternative names using following command
 perl omnigencert.pl -server_id <FQDN of Cell Manager> -server_san "dns:<fqdn of
Cell Manager>" -user_id 2IWF111xxxx\administrator -store_password Q***Ec8F
 perl omnigencert.pl -server_id <FQDN of Cell Manager> -server_san "ip:<IP of
Cell Manager>" -user_id 2IWF111xxxx\administrator -store_password Q***Ec8F

11
 Walkthrough of AppServer certificates (continued)
 The Java keytool command can be used to see the certificate thumbprint.
 /opt/omni/jre/bin/keytool -list -keystore server/server.keystore -storepass XXXXXXX
Keystore type: jks
Keystore provider: SUN

Your keystore contains 2 entries

cn=ca <CM HOST>, o=micro focus, st=md, c=us, Feb 19, 2020, trustedCertEntry,
Certificate fingerprint (SHA1): 0B:1A:C0:06:75:07:01:B1:CB:79:37:D8:E0:CE:8E:53:41:8D:F9:12
cn=<CM HOST>, o=micro focus, st=md, c=us, Feb 19, 2020, PrivateKeyEntry,
Certificate fingerprint (SHA1): 46:3F:1F:2F:54:97:08:2E:20:E1:3D:12:FF:07:54:BB:C1:30:C1:DC

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard
format using "keytool -importkeystore -srckeystore server/server.keystore -destkeystore server/server.keystore -
deststoretype pkcs12".
12
 Data Protector-specific Customizations
Sections of standalone.xml customized for use by DP:
 ManagementRealm/UndertowRealm (https connector)
 DataSource Connection (JDBC driver connection to Postgres database)
 Port configuration (all ports used by DP)
 Security Domain (security provided to each WAR)
 Ciphers and protocols (connection protocols)
 Wildfly-Keycloak Adapter configuration (part of security)

13
 Data Protector-specific Customizations (continued)
 ManagementRealm/UndertowRealm
 Used for https connections where client’s certificate is validated by Wildfly.
 Certificate store and store password configured here, so that Wildfly can perform validation.
 Example below where certificate store is “C:/ProgramData/OmniBack/Config/server/certificates/server/server.keystore” and
store password is “Fz4e1NbyXgNeQ” -
<security-realm name="UndertowRealm">
<server-identities>
<ssl>
<keystore path="C:/ProgramData/OmniBack/Config/server/certificates/server/server.keystore" keystore-
password="Fz4e1NbyXgNeQ"/>
</ssl>
</server-identities>
</security-realm>
 DataSource Connection
 JDBC driver to connect to DP database
 Database name and its credentials needed

14
 Data Protector-specific Customizations (continued)

 Port configuration
 All AppServer ports used by DP listed below
 7116 for https and reserved; native management port 9999, management port 9990 can be used from other hosts

15
 Data Protector-specific Customizations (continued)
 Security Domain
 Sections relying on basic authentication and client-cert authentication listed below
 Basic authentication performed by DpLoginModule, by checking if login user is present in UserList file
 Client-cert authentication verified by JCE-Webservice-domain where key and certificate are tested using provided
keystore and trust store

16
 Data Protector-specific Customizations (continued)
 Ciphers and protocols
 Minimum TLS version and ciphers required for DP AppServer connections

17
 Data Protector-specific Customizations (continued)
 Wildfly Keycloak adapter configuration

 Some services authenticated by Keycloak

 Clients needing Keycloak clearance get redirected to Wildfly Keycloak adapter, which communicates with Keycloak
to check validity
 Keycloak realm called "DataProtector" created for validation, with client registered as "dp-gui"
 Keycloak realm's public key and authentication URL also configured

18
 Impact of Application Server Failure on Data Protector
 Licensing functionality broken - backups won't work
 CRS depends on AppServer to get license information and keeps retrying
 Result – CRS unable to respond to other requests
 MFC GUI is unusable because of Keycloak authentication
 Quartz schedules don't get triggered
 Telemetry service nonfunctional

19
 Walkthrough of AppServer logs
 Two log files related to AppServer -
 server.log (/var/opt/omni/log/AppServer on Linux, <OMNI_DATA>/log/AppServer on windows)
 DPServer.log (/var/opt/omni/log/AppServer on Linux, <OMNI_DATA>/log/AppServer on windows)
 Server.log contains Wildfly related information
 DPServer.log contains DP business logic
 Default logging is info for both log files
 Logging can be modified to debug for detailed logs for trouble shooting
 Logging changes can be done by editing standalone.xml file under logging section, followed by a reload
or restart
 CLI to change logging settings also available -
 Linux CM:
/opt/omni/bin/perl /opt/omni/sbin/omniasutil.pl -enableappserverdebug <-module moduleName> <-level levelName>
 Windows CM:
<OMNI>/bin/perl <OMNI>/bin/omniasutil.pl -enableappserverdebug <-module moduleName> <-level levelName>
 levelName can be info, debug, warn, error etc..
moduleName can be JBOSS or DP. DP for DPServer.log and JBOSS for server.log

20
 Walkthrough of AppServer logs (continued)
 If the following error is found in server.log during a fresh install, it can be ignored as it is related to
database connections by Wildfly. The password configuration acutally occurs towards end of the install
and the dummy password used in the beginning causes the error.

ERROR [AbstractDsDeployer$1:1492:MSC service thread 1-2] Exception during createSubject() for

java:jboss/datasources/HPJobControlEngineDS: PBOX00016: Access denied: authentication failed:
java.lang.SecurityException: PBOX00016: Access denied: authentication failed

 The following error can be ignored in the server.log file. This is because a token expired and was
renewed by DP automatically.

ERROR [BearerTokenRequestAuthenticator:89:default task-6] Failed to verify token:

org.keycloak.common.VerificationException: Token is not active

21
 Walkthrough of AppServer logs (continued)
There are some valid errors which can be seen -
 11:46:34,327 ERROR [org.keycloak.adapters.BearerTokenRequestAuthenticator] (default task-177)
Failed to verify token: org.keycloak.common.VerificationException: Failed to parse JWT
 Action : Please check the configuration file for Wildfly Keycloak adapter settings to see if the Cell
Manager name mentioned in the adapter and the webservice.properties file matches. This
generally occurs because of FQDN vs short hostname issues or if the CM host name changes etc.
 javax.ejb.EJBException: java.lang.IllegalStateException: JBAS011048: Failed to construct component
instance
 Action : This can happen for those services which are authenticated by the client-cert method.
There are internal clients like (dispatcher client, JCE client, Webservice client which send requests
to services with certificates . The CM CA certificate present in client.keystore and server.truststore
has to be in the JRE truststore as well (/opt/omni/jre/lib/security/cacerts). DP takes care of this as
part of the install/upgrade process. If it doesn’t happen automatically, this error can occur.
 javax.ejb.EJBException: java.lang.RuntimeException: javax.net.ssl.SSLException: hostname in
certificate didn't match: <skynet> != <skynet.citrusworld.com> OR <skynet.citrusworld.com>
 Action : This clearly indicates a problem in the certificate where it got created with the short host
name but the cell manager's name is in the FQDN format.
22
Thank You.