EdenNet 21 FP 2106

EdenNet Troubleshooting Guide

DN09245501
Issue: 1-2
EdenNet Troubleshooting Guide DN09245501 1-2 Disclaimer

The information in this document applies solely to the hardware/software product (“Product”) specified herein, and only as specified herein.

This document is intended for use by Nokia' customers (“You”) only, and it may not be used except for the purposes defined in the agreement
between You and Nokia (“Agreement”) under which this document is distributed. No part of this document may be used, copied, reproduced,
modified or transmitted in any form or means without the prior written permission of Nokia. If you have not entered into an Agreement
applicable to the Product, or if that Agreement has expired or has been terminated, You may not use this document in any manner and You
are obliged to return it to Nokia and destroy or delete any copies thereof.

The document has been prepared to be used by professional and properly trained personnel, and You assume full responsibility when using
it. Nokia welcome Your comments as part of the process of continuous development and improvement of the documentation.

This document and its contents are provided as a convenience to You. Any information or statements concerning the suitability, capacity,
fitness for purpose or performance of the Product are given solely on an “as is” and “as available” basis in this document, and Nokia reserves
the right to change any such information and statements without notice. Nokia has made all reasonable efforts to ensure that the content of
this document is adequate and free of material errors and omissions, and Nokia will correct errors that You identify in this document. But,
Nokia' total liability for any errors in the document is strictly limited to the correction of such error(s). Nokia does not warrant that the use of
the software in the Product will be uninterrupted or error-free.

N O WA RRA NT Y O F AN Y KI ND , EI T HER EXPR ES S OR I M P L I E D , I N C L U D I N G B U T N O T L I M I T E D TO A N Y

WARR ANT Y OF AVA IL ABI LI T Y, AC CU RAC Y, R EL I A B I L IT Y, T I T L E , N O N - I N F R I N G E M E N T, M E R C H A N TA B I L I TY
OR F IT NE SS FO R A PA RT ICU LAR PU RPO SE, I S M A D E IN R E L AT I O N TO T H E C O N T E N T O F T H I S D O C U M E N T.
IN NO EVEN T WI L L NOK IA B E LI ABLE F OR AN Y DA M A G E S , I N C L U D I N G B U T N O T L I M I T E D TO S P E C I A L ,
D IRE CT, IN D IRECT, I NCI DE NTAL OR C ON SEQ UE N T IA L OR A N Y L O S S E S , S U C H A S B U T N O T L I M I T E D TO LO SS
OF PRO F IT, REVE NU E, B US IN ESS IN T ER RU PT I ON , B U S I NE S S O P P O RT U N I T Y O R D ATA T H AT M AY A R I S E
FRO M T HE USE O F TH IS DO CU M EN T O R T HE IN F OR M AT IO N I N I T, E V E N I N T H E C A S E O F E R R O R S I N O R
OM IS SI O NS FRO M T HI S DOC UM EN T O R IT S CO NT E N T.

This document is Nokia’ proprietary and confidential information, which may not be distributed or disclosed to any third parties without the
prior written consent of Nokia.

Nokia is a registered trademark of Nokia Corporation. Other product names mentioned in this document may be trademarks of their
respective owners, and they are mentioned for identification purposes only.

Copyright © 2021 Nokia. All rights reserved.

Important Notice on Product Safety

This product may present safety risks due to laser, electricity, heat, and other sources of danger.
Only trained and qualified personnel may install, operate, maintain or otherwise handle this product and only after having carefully read the
safety information applicable to this product.
The safety information is provided in the Safety Information section in the “Legal, Safety and Environmental Information” part of this
document or documentation set.

Nokia is continually striving to reduce the adverse environmental effects of its products and services. We would like to encourage you
as our customers and users to join us in working towards a cleaner, safer environment. Please recycle product packaging and follow the
recommendations for power use and proper disposal of our products and their components.
If you should have questions regarding our Environmental Policy or any of the environmental services we offer, please contact us at Nokia for
any additional information.
EdenNet Troubleshooting Guide DN09245501 1-2 Table of Contents

Contents
1 Summary of changes...................................................................................................................................... 6

2 Overview of troubleshooting........................................................................................................................11

3 Changing the email configuration............................................................................................................... 12

4 Region or cells are not visible in EdenNet GUI......................................................................................... 14

5 EdenNet login fails........................................................................................................................................ 15

5.1 Restarting database services..................................................................................................................15
5.2 Restarting EdenNet services.................................................................................................................. 16
5.3 Restarting LDAP services....................................................................................................................... 16

6 Logs are not visible on module execution.................................................................................................18

7 Forgotten EdenNet GUI password...............................................................................................................19

8 Calendar events disappear...........................................................................................................................20

9 Limitation in EdenNet GUI due to KPI chart usage................................................................................... 21

10 Module throws Unable to check kpi names against server error.......................................................... 22

11 Metadata update fails.................................................................................................................................. 23

12 RabbitMQ server fails abruptly or rabbitmq service is unable to start................................................. 25

13 RabbitMQ server fails to start after the VM reboot................................................................................. 26

14 EdenNet installation failure due to failed dependencies........................................................................ 28

15 Errors in the module instance log............................................................................................................ 29

16 Failed to fetch feedback logs.................................................................................................................... 31

17 kpiApp fails after Python 3 migration....................................................................................................... 32

18 Module failure after upgrade......................................................................................................................33

19 Cells are skipped due to exception.......................................................................................................... 35

20 ANR 5G module does not retrieve existing X2 relations........................................................................ 36

EdenNet 21 FP 2106 © 2021 Nokia 3

EdenNet Troubleshooting Guide DN09245501 1-2 Table of Contents

21 Unable to import adapted module in EdenNet GUI................................................................................. 37

22 Tasks are not running in SON engine...................................................................................................... 39

23 Tier calculation is not happening in the system..................................................................................... 40

24 Workspace management error...................................................................................................................42

25 EdenNet services are not starting.............................................................................................................43

26 SDK method fails with SSLVerificationError............................................................................................ 44

27 get_rollback_changes() API is not working in NAdC environment........................................................45

28 CM data.........................................................................................................................................................46
28.1 CM data update fails.............................................................................................................................46
28.2 CM cache is not updated..................................................................................................................... 47
28.3 CM exports fail for Alcatel-Lucent vendor............................................................................................ 48

29 PM data......................................................................................................................................................... 50
29.1 PM data not retrieved from OSS.......................................................................................................... 50
29.2 KPI not retrieved in EdenNet due to changes in OSS......................................................................... 50
29.3 PM parsing takes a lot of time............................................................................................................. 52
29.4 KPI client failed after two hours without KPI data response................................................................ 54

30 Increase the memory limit for module execution of larger scope cells................................................ 55
30.1 For adapted modules............................................................................................................................ 55

31 Lightweight Directory Access Protocol (LDAP).......................................................................................57

31.1 Directory manager password lost and needs to be reset.....................................................................57
31.2 Reset password without admin access to GUI.....................................................................................58
31.3 Locked user...........................................................................................................................................58

32 Standard software commands................................................................................................................... 60

33 EdenNet list of services............................................................................................................................. 63

34 Shutting down the virtual machines......................................................................................................... 65

35 Starting up a virtual machine.................................................................................................................... 66

36 Stop and start EdenNet for system shutdown.........................................................................................67

36.1 Stopping EdenNet services...................................................................................................................67
36.2 Stopping EdenNet framework............................................................................................................... 67
36.3 Stopping databases in the DB server...................................................................................................68

EdenNet 21 FP 2106 © 2021 Nokia 4

EdenNet Troubleshooting Guide DN09245501 1-2 Table of Contents

36.4 Starting EdenNet................................................................................................................................... 68

37 Module reports are missing in GUI........................................................................................................... 70

38 Hotfix information is not available pop-up appears................................................................................ 71

39 Delay in displaying hotfix details window................................................................................................72

40 All modified MOs under a BSC/RNC are not seen in the Rollback tab when filters are selected...... 73

41 Modules fail due to presence of duplicate externals pointing to the same native cell........................ 74

42 Upgrade failure due to vault tokens not being generated...................................................................... 75

43 API operations on AC not working during Keycloak based AC integration to EdenNet...................... 77

44 API invocation is not working on KPI Supplier VM.................................................................................78

45 EdenNet logs................................................................................................................................................79
45.1 Log directory layout...............................................................................................................................79
45.2 Copying log files....................................................................................................................................82

46 Cognitive SON configuration files purging.............................................................................................. 83

47 Frequently Asked Questions (FAQ)...........................................................................................................84

EdenNet 21 FP 2106 © 2021 Nokia 5

EdenNet Troubleshooting Guide DN09245501 1-2 Summary of changes

1 Summary of changes

Release Change description

EdenNet 21 FP 2106 Added sections:

• Cognitive SON configuration files purging

• Limitation in EdenNet GUI due to KPI chart usage

EdenNet 21 FP 2105 Updated section:

• SDK method fails with SSLVerificationError

EdenNet 21 FP 2104 No change.

EdenNet 21 FP 2103 No change.

EdenNet 21 Added section:

• API invocation is not working on KPI Supplier VM

Updated sections:

• EdenNet list of services - updated the ckng-kong service.

• Module reports are missing in GUI - updated with IPv6 note.

EdenNet 20 FP 2011 No change.

EdenNet 20 FP 2010 Added section:

• API operations on AC not working during Keycloak based AC

integration to EdenNet

EdenNet 20 FP 2009 No change.

EdenNet 20 FP 2008 Added sections:

• Calendar events disappear

• Upgrade failure due to vault tokens not being generated

EdenNet 20 FP 2007 Added sections:

• Hotfix information is not available pop-up appears

• Delay in displaying hotfix details window

EdenNet 20 Updated sections:

• Standard software commands

• Starting EdenNet

Added sections:

• kpiApp fails after Python 3 migration

EdenNet 21 FP 2106 © 2021 Nokia 6

EdenNet Troubleshooting Guide DN09245501 1-2 Summary of changes

Release Change description

• All modified MOs under a BSC/RNC are not seen in the Roll-
back tab when filters are selected

EdenNet 19A FP 2004 Updated sections:

• Tier calculation is not happening in the system

• EdenNet services are not starting

EdenNet 19A FP 2003 Added section:

• Workspace management error

EdenNet 19A FP 2002 Added section:

• CM exports fail for Alcatel-Lucent vendor

EdenNet 19A FP 2001 Added section:

• Tier calculation is not happening in the system

EdenNet 19A FP 1912 Updated section:

• Unable to import adapted module in EdenNet GUI

EdenNet 19A FP 1911 Updated section:

• EdenNet list of services

EdenNet 19A Added section:

• EdenNet list of services

Updated sections:

• PM parsing takes a lot of time - Added a note to about check_

farm.
• Stopping EdenNet framework
• Tasks are not running in SON engine
• For non-root users:

– Changing the email configuration

– Region or cells are not visible in EdenNet GUI
– Restarting database services
– Restarting EdenNet services
– Restarting LDAP services
– Logs are not visible on module execution
– Forgotten EdenNet GUI password
– Module throws Unable to check kpi names against server
error

EdenNet 21 FP 2106 © 2021 Nokia 7

EdenNet Troubleshooting Guide DN09245501 1-2 Summary of changes

Release Change description

– Metadata update fails

– RabbitMQ server fails abruptly or rabbitmq service is unable
to start
– RabbitMQ server fails to start after the VM reboot
– Module failure after upgrade
– Cells are skipped due to exception
– Unable to import adapted module in EdenNet GUI
– CM data
– PM data
– Increase the memory limit for module execution of larger
scope cells
– Standard software commands
– Stop and start EdenNet for system shutdown
– Module reports are missing in GUI
– EdenNet logs

EdenNet 19 FP 1907 No change.

EdenNet 19 FP 1906 No change.

EdenNet 19 FP 1905 No change.

EdenNet 19 FP 1904 Updated section:

• Instances of Custom modules are changed to Adapted mod-

ules in the following section:

– Errors in the module instance log

Added section:

• ANR 5G module does not retrieve existing X2 relations

EdenNet 19 Updated sections:

• Restarting database services

• Restarting LDAP services
• Logs are not visible on module execution
• RabbitMQ server fails abruptly or rabbitmq service is unable to
start
• RabbitMQ server fails to start after the VM reboot
• Module failure after upgrade

Added section:

EdenNet 21 FP 2106 © 2021 Nokia 8

EdenNet Troubleshooting Guide DN09245501 1-2 Summary of changes

Release Change description

• Modules fail due to presence of duplicate externals pointing to

the same native cell

EdenNet 18 SP1 1901 Added sections:

• KPI client failed after two hours without KPI data response
• Module reports are missing in GUI

Updated section:

• Logs are not visible on module execution

EdenNet 18 SP1 1812 No change.

EdenNet 18 SP1 1811 Added section:

• Increase the memory limit for module execution of larger scope

cells

EdenNet 18 SP1 Updated sections:

• CM data update fails section is updated with the note on the AC

integration.
• RabbitMQ server fails abruptly or rabbitmq service is unable to
start is updated with a note.

Added sections:

• PM parsing takes a lot of time section is added.

• Tasks are not running in SON engine
• Starting EdenNet
• Unable to view module INI file after upgrade

EdenNet 18 Added sections:

• Log directory layout

• RabbitMQ server fails abruptly or rabbitmq service is unable to
start
• Module failure after upgrade
• Unable to import adapted module in EdenNet GUI
• Cells are skipped due to exception
• CM cache is not updated
• RabbitMQ server fails to start after the VM reboot

EdenNet 17 SP1 FP1 The following sections are moved from the Administering EdenNet
document:

• Standard software commands

• Shutting down the virtual machines

EdenNet 21 FP 2106 © 2021 Nokia 9

EdenNet Troubleshooting Guide DN09245501 1-2 Summary of changes

Release Change description

• Starting up a virtual machine

EdenNet 17 SP1 Added section:

• Failed to fetch feedback logs

EdenNet 17 FP1 No change.

EdenNet 17 Added sections:

• Logs are not visible on module execution

• Forgotten EdenNet GUI password
• Module throws Unable to check kpi names against server error
• Errors in the module script log
• PM data not retrieved from OSS
• Frequently Asked Questions (FAQ)

Updated section:

• Copying log files > Step 2 > Note is added related to log com-
mand that can be used to fetch the service and instance logs.

EdenNet 16 SP4 Added sections:

• Restarting LDAP services section is added.

• Metadata update fails section is added.
• Lightweight Directory Access Protocol (LDAP) section is added.
• Directory manager password lost and needs to be reset section
is added.
• Reset password without admin access to GUI section is added.
• Locked user section is added.

Updated section:

• EdenNet logs section is updated with LDAP related information.

EdenNet 16 SP3 This is a new document that provides solutions to be used to trou-
bleshoot errors for EdenNet.

Table 1: Summary of changes

EdenNet 21 FP 2106 © 2021 Nokia 10

EdenNet Troubleshooting Guide DN09245501 1-2 Overview of troubleshooting

2 Overview of troubleshooting
This document describes the troubleshooting steps to be followed when errors are encountered during
execution or installation of EdenNet.

EdenNet 21 FP 2106 © 2021 Nokia 11

EdenNet Troubleshooting Guide DN09245501 1-2 Changing the email configuration

3 Changing the email configuration

Problem

Module reports are not received in an email.

Possible cause

The email configuration parameters are not updated properly.

Note: The SMTP port must be configured as per customer policy.

Solution

To update the default email configuration parameters, do the following:

1. Create a configuration file with the following information (for example, filename is email.ini):

[params]

# Email server host address, e.g. smtp.office365.com

EMAIL_HOST =
EMAIL_PORT = 25

# User name and password to connect to the host

EMAIL_HOST_USER =
EMAIL_HOST_PASSWORD =
# Server email address
SERVER_EMAIL =
# Use transport layer security, default is True
EMAIL_USE_TLS = True
DEFAULT_FROM_EMAIL = [email protected]

Note: The SMTP port must be configured as per customer policy.

2. Log in to the application server as <installation_user>.

3. Switch the user to vson by entering:

sudo -i -u vson

4. Check the current configuration by entering:

enetcfg export -mp config/service/emailService

5. Update the new configuration by entering:

EdenNet 21 FP 2106 © 2021 Nokia 12

EdenNet Troubleshooting Guide DN09245501 1-2 Changing the email configuration

enetcfg import -mp config/service/emailService -p email.ini -b -push

6. Validate the changed values by entering:

enetcfg export -mp config/service/emailService

7. Restart the emailApp to apply the changes by entering:

enet restart emailApp

8. Verify the given email configuration changes in /home/vson/log/enet/emailApp.log file.

EdenNet 21 FP 2106 © 2021 Nokia 13

EdenNet Troubleshooting Guide DN09245501 1-2 Region or cells are not visible in EdenNet
GUI

4 Region or cells are not visible in EdenNet GUI

Problem

After EdenNet upgrade, the region is not visible in GUI.

Possible cause

Inconsistency of cells with Operations Support System (OSS).

Solution

1. Log in to the GUI Virtual Machine (VM) as <installation_user>.

2. Switch the user to vson by entering:

sudo -i -u vson

3. Restart tomcat by entering:

(enet) [vson@guinode ~]$ enet restart tomcat

EdenNet 21 FP 2106 © 2021 Nokia 14

EdenNet Troubleshooting Guide DN09245501 1-2 EdenNet login fails

5 EdenNet login fails

Problem

Attempt to log in to EdenNet GUI is failing.

Possible cause

The following EdenNet services may not be running:

• Database services
• EdenNet services
• LDAP services

Solution

If any of the following services are in failed state, try to restart that service and log in again.

• Database services. For more information, see Restarting database services.

• EdenNet services. For more information, see Restarting EdenNet services.
• LDAP services. For more information, see Restarting LDAP services.

5.1 Restarting database services

1. Log in to the database (db) server as <installation_user>.

2. Restart the database service by entering:

[user@DBnode ~]# sudo systemctl restart mysqld

3. Check the status of database service by entering:

[user@DBnode ~]# systemctl status mysqld

mysql.service - LSB: start and stop MySQL

Loaded: loaded (/etc/rc.d/init.d/mysql; bad; vendor preset:
disabled)
Active: active (running) since Wed 2019-03-13 02:39:24 EDT; 5 days
ago
Docs: man:systemd-sysv-generator(8)
CGroup: /system.slice/mysql.service
##6002 /bin/sh /usr/bin/mysqld_safe --datadir=/home/data/
mysql --p...
##6359 /usr/sbin/mysqld --basedir=/usr --datadir=/home/
data/mysql ...

EdenNet 21 FP 2106 © 2021 Nokia 15

EdenNet Troubleshooting Guide DN09245501 1-2 EdenNet login fails

Note:

Following are the commands to perform various database related operations:

• Stop the database service by entering:

[user@DBnode ~]# sudo systemctl stop mysqld

• Start the database service by entering:

[user@DBnode ~]# sudo systemctl start mysqld

5.2 Restarting EdenNet services

Note: Nokia recommends restarting an app (service) instead of restarting all the apps
(services).

1. Log in to the application server as <Installation_user> and switch the user to vson by entering:

sudo -i -u vson

2. Check the status of EdenNet services by entering:

(enet) [vson@appnode ~]$ enet status

3. Restart all the services by entering:

(enet) [vson@appnode ~]$ enet restart all

Note:

Following are the commands to perform various database related operations:

• Restart a service by entering:

(enet) [vson@appnode ~]$ enet restart <service name>

• Stop a service by entering:

(enet) [vson@appnode ~]$ enet stop <service name>

• Start a service by entering:

(enet) [vson@appnode ~]$ enet start <service name>

EdenNet 21 FP 2106 © 2021 Nokia 16

EdenNet Troubleshooting Guide DN09245501 1-2 EdenNet login fails

5.3 Restarting LDAP services

1. Log in to the Lightweight Directory Access Protocol (LDAP) server (same as database server) as
<Installation_user>.

2. Restart LDAP by entering:

sudo sh -c "systemctl restart dirsrv@*.service"

3. Check the status of LDAP service by entering:

systemctl status dirsrv@*.service

[email protected] - 389 Directory Server edennet. Loaded: loaded

(/usr/lib/systemd/system/[email protected]; enabled; vendor preset:
disabled) Active: active (running) since xxxxxxxxxxxxxx; xx ago
Process: xxxx ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/
dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS) Main PID: xxxx
(ns-slapd) Status: "slapd started: Ready to process requests" CGroup: /
system.slice/system-dirsrv.slice/[email protected] ##1618 /usr/
sbin/ns-slapd -D /etc/dirsrv/slapd-edennet -i /var/run/dirsrv/slapd-
edennet.pid

EdenNet 21 FP 2106 © 2021 Nokia 17

EdenNet Troubleshooting Guide DN09245501 1-2 Logs are not visible on module execution

6 Logs are not visible on module execution

Problem

EdenNet modules do not produce any log entries.

Possible causes

• Sufficient resources are not available in the environment.

• High CPU usage of GUI node.

Solution

1. Log in to the GUI node via SSH as <installation_user>.

2. Restart rsyslog daemon by entering:

sudo systemctl restart rsyslog

3. Check the status of rsyslog service by entering:

systemctl status rsyslog

The following status is visible:

rsyslog.service - System Logging Service Loaded: loaded (/usr/lib/

systemd/system/rsyslog.service; enabled; vendor preset: enabled) Active:
active (running) since Tue 2019-03-12 14:14:34 EET; 5 days ago Docs:
man:rsyslogd(8) http://www.rsyslog.com/doc/ Main PID: 27840 (rsyslogd)
CGroup: /system.slice/rsyslog.service ##27840 /usr/sbin/rsyslogd -n

EdenNet 21 FP 2106 © 2021 Nokia 18

EdenNet Troubleshooting Guide DN09245501 1-2 Forgotten EdenNet GUI password

7 Forgotten EdenNet GUI password

Problem

User has forgotten the EdenNet GUI password of the login user.

Solution

1. Log in to Central DB server as <installation_user>.

2. Create new_password.ldif file with the following content:

dn: uid=<enetuser>,ou=Users,ou=Accounts,dc=eden,dc=net
changetype: modify
replace: userPassword
userPassword: <new_password>

Note: Ensure to replace enetuser with admin username and new_password with the
desired password.

3. Reset the password by entering:

#ldapmodify -a -D "cn=directory manager" -w <directory server password>

-p 389 -h localhost -f new_password.ldif

4. To verify, log in to EdenNet GUI with the new password.

Using the new password you can log in to the EdenNet GUI.

EdenNet 21 FP 2106 © 2021 Nokia 19

EdenNet Troubleshooting Guide DN09245501 1-2 Calendar events disappear

8 Calendar events disappear

Problem

In the EdenNet GUI, the calendar events in Day and Week calendar tabs disappear during the follow-
ing scenarios:

• When the user navigates to Automated Site Creation → Site Map, and then to SON Modules →
Calendar, the calendar events in Day and Week calendar tabs that were previously present dis-
appear.
• When the user navigates to SON Modules → Calendar, Automated Site Creation → Site Map,
and then to SON Modules → Calendar, the Exceptions while rendering events dialog box ap-
pears:

When the user clicks on Retry or Quit, the calendar events in Day and Week calendar tabs that were
previously present disappear.

Possible Causes

Following are the possible causes:

• Calendar tab uses a third party library smartGWT component, which uses browser's cache to opti-
mize the widget creation.
• Site Map tab uses a third party library Openlayer 2, which uses tile manager to cache tiles for fast
reuse and to reduce loading time.
• Because of namespace conflict, the calendar cache object is overwritten by Openlayer's cache.
Hence, when the user navigates to Calendar tab, the smartGWT tries to refresh the view by
reusing the cache object and throws an error.

Solution

User must logout of the EdenNet GUI and then login to view the calendar events.

EdenNet 21 FP 2106 © 2021 Nokia 20

EdenNet Troubleshooting Guide DN09245501 1-2 Limitation in EdenNet GUI due to KPI
chart usage

9 Limitation in EdenNet GUI due to KPI chart usage

Problem

Active users get logged out from the EdenNet GUI during KPI chart access and a message appears in
Tomcat at /home/vson/opt/tomcat/logs/tomcat.log. Following is a sample message:

Heap dump file created [12522225824 bytes in 97.696 secs]

#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9 %p"

Possible Cause

Tomcat memory usage may aggravate when multiple users access KPI charts in EdenNet GUI concur-
rently. This usually happens when network size is more than 150K cells and KPI retrieval takes more
time due to modules load in parallel. If users log out from EdenNet GUI before the KPI charts are com-
pletely loaded, background processing is still on and that could lead to OutOfMemory in Tomcat.

Solution

Log in back to the EdenNet GUI.

EdenNet 21 FP 2106 © 2021 Nokia 21

EdenNet Troubleshooting Guide DN09245501 1-2 Module throws Unable to check kpi
names against server error

10 Module throws Unable to check kpi names against

server error
Problem

After installation or upgrade, the module throws a message Unable to check kpi names against server
error.

For example, the error might appear as:

Traceback (most recent call last):

File "/home/vson/enet/lib/python3.7/site-packages/enetclients/kpi.py",
line 87, in validate_kpi
if name not in self.kpi_names:
TypeError: argument of type 'NoneType' is not iterable
VM01 | 17-05-24 11:00:56-UTC[INFO] ERROR |2017-05-24 11:00:56 UTC|
kpi:90 |17517| Unable to check kpi names against server

Possible cause

This error occurs when after installation or upgrade kpiBrokerApp is not initialized properly.

Solution

1. Log in to the Central App server.

2. Switch to vson user by entering:

sudo -i -u vson

3. Restart <region>.pm.kpiBrokerApp service by entering:

#enet restart <region>.pm.kpiBrokerApp

4. Collect the symptoms by checking the module execution log in the GUI.

5. To verify, re-run the module and check the module execution log in the GUI.

The error will not be seen anymore.

EdenNet 21 FP 2106 © 2021 Nokia 22

EdenNet Troubleshooting Guide DN09245501 1-2 Metadata update fails

11 Metadata update fails

Problem

The metadata update fails.

Possible cause

This occurs due to socket timeout.

Solution

Modify the SSH timeout for metadata update by doing the following:

1. Log in to the central VM server as <Installation_ user>.

2. Switch the user to vson by entering:

sudo -i -u vson

3. Open the command line configuration utility by entering:

$ cfg

4. List the services by entering:

cfg> s

A list of services is displayed.

5. From the list, identify the number against which the <region_name>.meta_path service is listed
and enter:

cfg> s <the number against which the service <region_name>.meta_path is

listed>

For example,

cfg> s 25

vi of the configuration opens.

6. Add the following parameter or value, if missing:

meta_update_timeout = 300

7. Adjust the value for the meta_update_timeout parameter as required.

8. Save the changes and exit.

9. Restart the service by entering:

EdenNet 21 FP 2106 © 2021 Nokia 23

EdenNet Troubleshooting Guide DN09245501 1-2 Metadata update fails

enet restart <region_name>.cm.main

10. Update the metaservice by entering:

cmtask update metaservice <region_name>

11. Verify if the CM main logs for the metadata update is complete in /home/vson/log/enet/
<region_name>.cm.main.log file.

The metadata update complete message appears.

EdenNet 21 FP 2106 © 2021 Nokia 24

EdenNet Troubleshooting Guide DN09245501 1-2 RabbitMQ server fails abruptly or
rabbitmq service is unable to start

12 RabbitMQ server fails abruptly or rabbitmq service is

unable to start
Problem

RabbitMQ server fails abruptly or rabbitmq service is unable to start.

Possible cause

Improper file permissions.

Solution

1. Log in to Central Virtual Machine (VM) and check the status of rabbitmq service by entering:

# [user@centralAPP rabbitmq]# systemctl status rabbitmq-server

rabbitmq service is stopped and is unable to start as the files starting with rabbit@*.log in /
var/log/rabbitmq directory does not have rabbitmq ownership.

2. Check if /etc/logrotate.conf file has the correct line without any restrictions and is as
follows:

# create new (empty) log files after rotating old ones

create

3. If /etc/logrotate.conf file does not contain the above line, execute the following command to
add this line:

• # sudo sh -c "echo '# create new (empty) log files after rotating old
ones' >> /etc/logrotate.conf
• # sudo sh -c "echo create >> /etc/logrotate.conf"

If /etc/logrotate.conf file contains a different line, shut down the EdenNet services and
reboot all Virtual Machines (VM).

Note: Before performing upgrade, ensure that rabbitmq folder and rabbitmq*.log
files have the right permissions and ownership as follows:

• /var/log/rabbitmq directory should have 755 permissions and

rabbitmq:rabbitmq ownership.
• /var/log/rabbitmq/rabbit@*.log files should have 644 permissions and
rabbitmq:rabbitmq ownership.

EdenNet 21 FP 2106 © 2021 Nokia 25

EdenNet Troubleshooting Guide DN09245501 1-2 RabbitMQ server fails to start after the
VM reboot

13 RabbitMQ server fails to start after the VM reboot

Problem

RabbitMQ server fails to start after Virtual Machine (VM) reboot.

Possible cause

The port is already in use by other EdenNet process.

Solution

1. Log in to the Central VM and check the status of rabbitmq service by entering:

# [user@centralAPP rabbitmq]# systemctl status rabbitmq-server

If rabbitmq service is stopped and is unable to start due to port 61613 (fixed port for rabbitmq)
being used by any of the EdenNet service, then check the logs in /var/log/rabbitmq directory
which, in this case, contains address already in use error message:

rabbitmq error log file snippet:

Failed to start Ranch listener {acceptor,{0,0,0,0,0,0,0,0},61613}

in ranch_tcp:listen([{port,61613},{ip,{0,0,0,0,0,0,0,0}},inet6,
{backlog,128},{nodelay,true}]) for reason eaddrinuse (address already in
use)

2. Check the status of the port 61613:

a) Check the process id (pid) which uses the port 61613 by entering:

# [user@centralAPP rabbitmq]# sudo netstat -ap | grep -i 61613

b) Check the process which uses the port 61613 by entering:

# [user@centralAPP rabbitmq]# ps -ef | grep -i <pid>

3. Stop all the EdenNet services as vson user by entering:

# [vson@centralAPP rabbitmq]#enet stop all

4. Start the rabbitmq service as <installation_user> by entering:

# [user@centralAPP rabbitmq]# sudo systemctl start rabbitmq-server

Note: Start the rabbitmq service after all the EdenNet services are stopped.

5. Check the status of rabbitmq service by entering:

EdenNet 21 FP 2106 © 2021 Nokia 26

EdenNet Troubleshooting Guide DN09245501 1-2 RabbitMQ server fails to start after the
VM reboot

# [user@centralAPP rabbitmq]# systemctl status rabbitmq-server

6. Start all the EdenNet services after the rabbitmq service starts operating by entering:

# [vson@centralAPP rabbitmq]#enet start all

EdenNet 21 FP 2106 © 2021 Nokia 27

EdenNet Troubleshooting Guide DN09245501 1-2 EdenNet installation failure due to failed
dependencies

14 EdenNet installation failure due to failed

dependencies

Problem

EdenNet installation fails with the error:

error: Failed dependencies:

libXrender.so.1()(64bit) is needed by
java-1.8.0-openjdk-1:1.8.0.71-1.b15.el6_7.x86_64

Possible cause

Dependent RPMs are missing.

Solution

Install the required RPMs using the yum utility (for example, java-1.8.0-openjdk) and proceed
with the installation.

EdenNet 21 FP 2106 © 2021 Nokia 28

EdenNet Troubleshooting Guide DN09245501 1-2 Errors in the module instance log

15 Errors in the module instance log

Problem

Errors in the module instance log due to missing module input configuration files.

Possible cause

Cause 1: Module configuration file is missing or it is not added for the module.

Cause 2: Module configuration file is added, but it is not activated or it is the wrong file.

Solution

Note: The troubleshooting steps are the same for all the modules. Let us consider an exam-
ple of Parameter Consistency Check and Enforcement (PCE) module for troubleshooting.

1. Log in to EdenNet.

For more information, see Accessing EdenNet application in EdenNet User and Administration
Guide.

2. Click Status → Execution Status under the SON Modules tab.

3. Select a module from the left pane and check the instance log for the log details.

For example, the following errors are seen for PCE module:

Processing nsn LTE c2oss

No remote ip_address, skipping remote config file
National file is not available
Regional file is not available
Node Exclusion file is not available
National Configuration File not uploaded or activated. Please add and
activate National Configuration File first
Module exiting due to National Configuration File not uploaded or
activated.

Note: Additional details can be found under the Logs tab. For more details, check the log
file.

4. Click Administration → Modules.

The following module categories appear in the left pane:

• EdenNet modules: The modules that Nokia provides are available in this category.
• Adapted modules: The modules that users develop are available in this category.

EdenNet 21 FP 2106 © 2021 Nokia 29

EdenNet Troubleshooting Guide DN09245501 1-2 Errors in the module instance log

• Helper modules: These modules are mainly used for troubleshooting by Nokia support teams.
They are not categorized as Generally Available. General Availability implies that the release is
available to all customers.

5. Click PCEmodule from EdenNet modules.

6. Click Configure tab and then click Module Configuration.

The PCE Configuration Manager dialog box appears.

7. Select the particular category of module configuration and click Activate.

Note:

In case the file is already present, ensure that it is activated. In this example, the National
Configuration file for LTE is not activated.

EdenNet 21 FP 2106 © 2021 Nokia 30

EdenNet Troubleshooting Guide DN09245501 1-2 Failed to fetch feedback logs

16 Failed to fetch feedback logs

Problem

In the Provisioning Logs tab, when the user clicks the Logs link in the Feedback Logs window, an
error message Failed to fetch the logs appears.

Possible cause

One of the following services are not running:

• ScriptPlanMapperService
• database

Procedure

• Try restarting both the ScriptPlanMapperService and the database services.

EdenNet 21 FP 2106 © 2021 Nokia 31

EdenNet Troubleshooting Guide DN09245501 1-2 kpiApp fails after Python 3 migration

17 kpiApp fails after Python 3 migration

Problem

kpiApp fails after migration to Python 3.x and the following error message appears in
<region>.pm.kpiApp.log:

Traceback (most recent call last):

File "/apps/vson/enet/lib/python3.7/site-packages/enet_app_lib/
uwsgiServer.py", line 161, in <module>
application = initApp(entryPoint, configService=appConfigService,
db=db)
File "/apps/vson/enet/lib/python3.7/site-packages/enet_app_lib/
uwsgiServer.py", line 80, in initApp
return initMethod(**params)
File "/apps/vson/enet/lib/python3.7/site-packages/enetservice/app/kpiApp.
py", line 138, in init
return base_init(configService, region_config_service, db, pm_config,
region_name, supplier_mount)
File "/apps/vson/enet/lib/python3.7/site-packages/enetservice/app/kpiApp.
py", line 253, in base_init
equation_management.load_all_equations(oss_type)
File "/apps/vson/enet/lib/python3.7/site-packages/enetpm/equation_
management.py", line 100, in load_all_equations
equation_strings, result = _load_equations_using_query(query)
File "/apps/vson/enet/lib/python3.7/site-packages/enetpm/equation_
management.py", line 146, in _load_equations_using_query
compiled_equation = equationlib.unpack(packed_compiled_equation)
File "/apps/vson/enet/lib/python3.7/site-packages/enetpm/equationlib.py",
line 701, in unpack
return marshal.loads(base64.b64decode(equation))
ValueError: bad marshal data (unknown type code)

Solution

Procedure

• Enter the following command:

python3_migration_kpi_equations -t <hostname/IP of DB> -u <DB user> -p

<DB password>

EdenNet 21 FP 2106 © 2021 Nokia 32

EdenNet Troubleshooting Guide DN09245501 1-2 Module failure after upgrade

18 Module failure after upgrade

Problem

After upgrade, module starts failing with a message INTERNAL_SERVER_ERROR.

Possible cause

Keycloak service is not running.

Solution

1. Log in to Central App Server as <Installation_user>.

2. Check if the keycloak service is running by entering:

[user@server ~] # systemctl status keycloak

keycloak.service - Keycloak
Loaded: loaded (/etc/systemd/system/keycloak.service; enabled;
vendor preset: disabled)
Active: active (running) since Tue 2019-03-12 14:00:05 EET; 5 days
ago
Main PID: 21903 (sh)
CGroup: /system.slice/keycloak.service
##21903 /bin/sh -c /opt/keycloak/bin/standalone.sh -c
$WILDFLY_CONFIG -b $WILDFLY_BIND > /var/log/keycloak/keycloak.log 2>&1
##21904 /bin/sh /opt/keycloak/bin/standalone.sh -c
standalone.xml -b 0.0.0.0
##21957 java -D[Standalone] -server -Xms64m -Xmx512m
-XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.
preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -
Djava.awt.headless=true -Dorg.jboss.boot.lo...

3. Start the keycloak service by entering:

[user@server ~]# sudo systemctl start keycloak

4. Verify the status of the keycloak service by entering:

[ user@server ~]# systemctl status keycloak

keycloak.service - Keycloak
Loaded: loaded (/etc/systemd/system/keycloak.service; enabled;
vendor preset: disabled)
Active: active (running) since Tue 2019-03-12 14:00:05 EET; 5 days
ago
Main PID: 21903 (sh)

EdenNet 21 FP 2106 © 2021 Nokia 33

EdenNet Troubleshooting Guide DN09245501 1-2 Module failure after upgrade

CGroup: /system.slice/keycloak.service
##21903 /bin/sh -c /opt/keycloak/bin/standalone.sh -c
$WILDFLY_CONFIG -b $WILDFLY_BIND > /var/log/keycloak/keycloak.log 2>&1
##21904 /bin/sh /opt/keycloak/bin/standalone.sh -c
standalone.xml -b 0.0.0.0
##21957 java -D[Standalone] -server -Xms64m -Xmx512m
-XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.
preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -
Djava.awt.headless=true -Dorg.jboss.boot.lo...

EdenNet 21 FP 2106 © 2021 Nokia 34

EdenNet Troubleshooting Guide DN09245501 1-2 Cells are skipped due to exception

19 Cells are skipped due to exception

Problem

The following exception occurs in module log:

Unknown oss interface: unknown/external>. This may indicate inconsistencies

in the network which make correct cell building not possible.

Possible cause

Cell rehoming in network.

Solution

• In the Central CM app node, do a topology update manually by entering:

sudo -i -u vson

cmtask update topology all

EdenNet 21 FP 2106 © 2021 Nokia 35

EdenNet Troubleshooting Guide DN09245501 1-2 ANR 5G module does not retrieve
existing X2 relations

20 ANR 5G module does not retrieve existing X2

relations

Problem

The ANR 5G doesn’t retrieve existing X2 relations even though the X2 relation data is available in
NAdC environment.

Solution

• In the Central CM app node, do a topology update manually by entering:

sudo -i -u vson

cmtask update topology all

Note: The topology update can be verified by checking the topology_updated event in the
Events tab of EdenNet UI.

EdenNet 21 FP 2106 © 2021 Nokia 36

EdenNet Troubleshooting Guide DN09245501 1-2 Unable to import adapted module in
EdenNet GUI

21 Unable to import adapted module in EdenNet GUI

Problem 1

Command line import of the module succeeds but an error appears while importing the adapted mod-
ules via EdenNet GUI:

Unable to import module:

Traceback (most recent call last):
File "/home/vson/enet/bin/importscripttemplate", line 7, in <module>
from enet_tools.ImportScriptTemplate import Main
File "/home/vson/enet/lib/python3.7/site-packages/enet_tools/
ImportScriptTemplate.py", line 40, in <module>
from enetsdk import EnetDefines as Defs
File "/home/vson/enet/lib/python3.7/site-packages/enetsdk/_init_.py",
line 9, in <module>
import EnetConfig
File "/home/vson/enet/lib/python3.7/site-packages/enetsdk/EnetConfig.py",
line 7, in <module>
from sdklib.config_loader import get_conf, set_conf
File "/home/vson/enet/lib/python3.7/site-packages/sdklib/config_loader.
py", line 7, in <module>
from enetsdk_config.config_loader import get_conf, set_conf
File "/home/vson/enet/lib/python3.7/site-packages/enetsdk_config/config_
loader.py", line 109, in <module>
init()
File "/home/vson/enet/lib/python3.7/site-packages/enetsdk_config/config_
loader.py", line 67, in init
os.makedirs(path)
File "/home/vson/enet/lib/python3.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/.edennet'

Solution

1. Log in to Central App Server as <installation_user>.

2. Switch to vson user by entering:

sudo -i -u vson
3. Restart tomcat by entering:

#enet restart tomcat

EdenNet 21 FP 2106 © 2021 Nokia 37

EdenNet Troubleshooting Guide DN09245501 1-2 Unable to import adapted module in
EdenNet GUI

Problem 2

Even though all the required libraries for adapted modules are installed in the custom environment, im-
port of adapted module fails with the following error:

"ImportError ('No module named ...')"

Possible cause

The import of adapted module happens from the vson user's enet27 virtualenv on the GUI server VM,
which does not have the dependencies installed to parse and import the adapted module.

Solution

Install the dependencies of adapted modules into the vson user's enet27 virtualenv on GUI server
VM, using /home/vson/enet27/bin/pip command (if installing as a wheel package) or copying
the libraries directly under /home/vson/enet27/lib/python2.7/site-packages/ (if you have
the source packages).

EdenNet 21 FP 2106 © 2021 Nokia 38

EdenNet Troubleshooting Guide DN09245501 1-2 Tasks are not running in SON engine

22 Tasks are not running in SON engine

Problem

When a module such as load balancing module is executed and the module doesn’t report any outage
or compensation, it is possible that the tasks have not run in the SON engine.

To verify the problem:

• Go to /home/vson/log/enet directory and check the son engine log

(region_name.son.main.log).

The log will display:

Scheduling coc.root_task in 5 seconds or Scheduling loadbalance.root_task in

5 seconds.

The finished tasks will not be present in the log.

Possible cause

Data inconsistency.

Solution

1. Log in to the Central VM node as vson user and enter the following commands to flush the
pending tasks:

enet stop <region_name>.son.main

redis-cli s /var/run/enet/redis<region_name>-sonengine.sock -n 3
flushall

2. Start SON engine service by entering:

enet start <region_name>.son.main

Note: This problem can occur in load balancing modules.

EdenNet 21 FP 2106 © 2021 Nokia 39

EdenNet Troubleshooting Guide DN09245501 1-2 Tier calculation is not happening in the
system

23 Tier calculation is not happening in the system

Problem

• Modules reports such as ANR indicate greater than 10000 tier count values for all the available
source-target.
• Tier application logs have multiple errors such as “crippled...trying respawn”..

Solution

This solution is applicable for cases where Tier application is unable to complete the tier calculation
due to insufficient memory (RAM) only.

For other Tier application related issues, consult Nokia services team.

Changing Tier application configuration:

Tier count application loads all the valid internal and global external cells along with their antennas in
the memory during tier calculation.

Tier application runs on Central App VM. By default, the application is configured to run with 4 mules
(parallel processes) to perform tier calculation.

1. Log in to Central App VM as vson user and enter the following command to check how many
mules are configured for the tier application:

(enet) [vson@edenlab enet]$ cat /etc/enet/enetservice/registered/

tier.ini | grep "mules"

Based on the test results, the memory requirements that are observed for tier application are listed
in Table 2: Memory requirements for Tier application.

Total Memory Total Memory Total Memory

Total Number
Total Number of Requirement for Requirement for Requirement for
of cells (inter-
Antennas one mule config- 2 mules configu- 4 mules configu-
nal+external)
uration ration ration (default)

~200K ~450K ~30 GB ~60 GB ~ 120GB

~400K ~400K ~50 GB ~ 100 GB ~ 200 GB

Table 2: Memory requirements for Tier application

With the mentioned memory requirements, each mule would require that amount of RAM to be
made available for processing.

If the Central App resources are not sufficient to execute high number of mules for tier application,
then perform the following steps to reduce number of mules.

2. Update the mules configuration for the tier application by entering:

EdenNet 21 FP 2106 © 2021 Nokia 40

EdenNet Troubleshooting Guide DN09245501 1-2 Tier calculation is not happening in the
system

(enet) [vson@edenlab enet]$ vi /etc/enet/enetservice/registered/tier.ini

3. Edit the mules value from:

mules=4

to

mules=1

4. Save the file.

This will retrigger the tier application and the tier calculation start during the next periodic tier build-
ing cycle.

Ensure that above errors are not present anymore in the module or the tier application logs.

5. Stop the tier service by entering:

(enet) [vson@edenlab enet]$ enet stop tier

6. Find the process ID (pid) of the redis-tier.rdb file by entering:

(enet) [vson@edenlab enet]$ cat /var/run/enet/redis-tier.pid

7. Terminate the process by using the process ID found in step 6 by entering:

(enet) [vson@edenlab enet]$ kill -9 <pid from step 6>

8. Delete the redis-tier.rdb file by entering:

(enet) [vson@edenlab enet]$ rm -rf /home/vson/redis/redis-tier.rdb

9. Start the tier application by entering:

(enet) [vson@edenlab enet]$ enet start tier

EdenNet 21 FP 2106 © 2021 Nokia 41

EdenNet Troubleshooting Guide DN09245501 1-2 Workspace management error

24 Workspace management error

Problem

Workspace management throws the following error:

Failed to load workspace management

Solution

• Remove KPIs that are not monitored from the KPI charts

or
• Reduce the look back period configured in the KPI charts

EdenNet 21 FP 2106 © 2021 Nokia 42

EdenNet Troubleshooting Guide DN09245501 1-2 EdenNet services are not starting

25 EdenNet services are not starting

Problem

The EdenNet services are not starting and the following error message appears in the
appServersApp.log file available at /home/vson/log/enet:

"unavailable modifier requested: 0"

Solution

1. Log in as root user and stop all the EdenNet services by entering:

$ systemctl stop enet

All the EdenNet services stop.

2. Verify that all the EdenNet services have stopped by entering:

$ systemctl status enet

3. Log in as root user and start all the EdenNet services by entering:

$ systemctl start enet

EdenNet 21 FP 2106 © 2021 Nokia 43

EdenNet Troubleshooting Guide DN09245501 1-2 SDK method fails with
SSLVerificationError

26 SDK method fails with SSLVerificationError

Problem

SDK method fails with the following SSLVerificationError:

'[SSL: CERTIFICATE VERIFY FAILED]'

Solution

1. Log in to the GUI node as root user.

2. Go to /opt/nokia/certificate_mgmt/ and open ca_cert.crt .

3. Copy the contents of ca_cert.crt.

4. In the user system (SDK virtual environment), open cacert.pem in <python installation
folder>/Lib/site-packages/certify.

5. Append the contents of ca_cert.crt to cacert.pem under certify in the SDK virtual
environment

6. Reheck if the connection to server is through and rerun the module.

EdenNet 21 FP 2106 © 2021 Nokia 44

EdenNet Troubleshooting Guide DN09245501 1-2 get_rollback_changes() API is not
working in NAdC environment

27 get_rollback_changes() API is not working in NAdC

environment

Problem

The def get_rollback_ changes(self , rollback_filter) function returns an empty

value even though the changes are getting pushed successfully in cross-OSS NAdC environment.

Solution

This method is not supported in EdenNet Cross-OSS via NAdC.

Note: In direct integration, the API returns a list of CMChanges that contains operation, DN,
and attributes if EdenNet is connected to multiple OSS without NAdC

EdenNet 21 FP 2106 © 2021 Nokia 45

EdenNet Troubleshooting Guide DN09245501 1-2 CM data

28 CM data
This section describes the Configuration Management (CM) data troubleshooting scenarios.

28.1 CM data update fails

Problem

Configuration Management (CM) data update fails due to changes in Operations Support System
(OSS) configuration.

Possible cause

CM service of the region is not running.

Solution 1

To modify the CM interface once it is configured using cfg, do the following:

1. Log in to the application server as <installation_user>.

2. Switch the user to vson by entering:

sudo -i -u vson
3. List the interfaces by entering:
cfg> i

0) /region/testoss/cm/interface

1) /region/testoss/pm/interface
4. Select CM interface by typing i + the number.

for example:

cfg> i 0

vi of the configuration opens.

5. Edit and save the changes.
6. Restart the CM service of the region by entering:
(enet) [vson@appnode ~]$ enet restart <region>.cm.main

(enet) [vson@appnode ~]$ enet restart <region>.cm.cache

Note: In case of AC integration, restart the cell building app by entering:

enet restart <region_name>.cm.cell_building_app

Solution 2

To modify the CM interface once it is configured using enetcfg do the following:

EdenNet 21 FP 2106 © 2021 Nokia 46

EdenNet Troubleshooting Guide DN09245501 1-2 CM data

Note: The following example is for updating the IP address. You can add all the supported
CM interface parameters.

1. Create a configuration file with the following information (for example, filename is
cminterface.ini):

Example:

For NSN

[interface]

commIp = <central VM ip>

For Ericsson

[interface]

masterServerIp= <central VM ip>

2. Log in to the application server as <installation_user> and switch the user to vson by entering:

sudo -i -u vson
3. Check the current configuration by entering:

enetcfg export -mp region/<regionName>/cm/interface –b

4. Import the new configuration by entering:

enetcfg import -mp region/<regionName>/cm -p cminterface.ini -b –push

5. Validate the changed values by entering:

enetcfg export -mp region/<regionName>/cm/interface –b

6. Restart the CM service of the region by entering:
(enet) [vson@appnode ~]$ enet restart <region>.cm.main

(enet) [vson@appnode ~]$ enet restart <region>.cm.cache

Note: In case of AC integration, also restart the cell building app by entering:

enet restart <region_name>.cm.cell_building_app

28.2 CM cache is not updated

Problem

Configuration management (CM) cache is not updated with the latest CM data from Operations
Support System (OSS) for more than a day. The last cache update interval for each Managed Object
(MO) class can be verified in the ems_cache.cm_cache_schedule database table in central
database virtual machine (VM).

Possible cause

EdenNet 21 FP 2106 © 2021 Nokia 47

EdenNet Troubleshooting Guide DN09245501 1-2 CM data

The connection to OSS is lost for a long time, and EdenNet has not recovered after the connection
has been re-established.

Solution

1. Log in to Central App Server as <installation_user>.

2. Switch to vson user by entering:

sudo -i -u vson

3. Check if there are any cache update requests to CM Cache App by entering:

redis-cli s /var/run/enet/redis

{region_name}.cm.main-main.sock -n 3 zrevrangebyscore /region/

{region_name}

/queue_prio_set +inf -inf

Keep monitoring. If the same MO class is present or is added frequently, then proceed with the
next steps.

4. Stop CM main app by entering:

enet stop {region_name}

.cm.main

5. Wait until the CM cache is updated for all MOs.

Note: Check this by monitoring the update_time and failure_time columns in

ems_cache.cm_cache_schedule database table in central DB VM.

6. Start CM main app by entering:

enet start {region_name}

.cm.main

7. Verify that the CM main app is started properly by entering:

enet status{region_name}

.cm.main

Note: Do not execute any modules when CM main app is stopped.

EdenNet 21 FP 2106 © 2021 Nokia 48

EdenNet Troubleshooting Guide DN09245501 1-2 CM data

28.3 CM exports fail for Alcatel-Lucent vendor

Problem

Alcatel-Lucent vendor region specific CM exports fail with the following error:

Failed to process MO request - Duplicate parse request

Solution

This solution is applicable for Alcatel-Lucent direct integration cases. Alcatel-Lucent region specific
CM exports will fail with duplicate parse request errors.

Region specific CM cache logs will have the following error:

Failed to process MO request - Duplicate parse request

1. Log in to Central VM app as vson user and enter the following command:

(enet) [vson@edenlab enet]$ cfg

2. Type s within the cfg window.

All the available services are listed with its index numbers.

3. Choose the index number of Alcatel-Lucent region cm emsService

<region_name>.cm.emsService and enter the following:

s <above service index number>

4. Edit and add trigger_export_valid_duration = 18000 without quotes in a new line.

5. Save and exit the editor.

6. Enter quit and exit the cfg window.

7. Restart Alcatel-Lucent region CM main and cache services by entering:

(enet) [vson@edenlab enet]$ enet restart <region_name>.cm.main

(enet) [vson@edenlab enet]$ enet restart <region_name>.cm.cache

Note: <region_name> must be replaced with valid Alcatel-Lucent region name.

EdenNet 21 FP 2106 © 2021 Nokia 49

EdenNet Troubleshooting Guide DN09245501 1-2 PM data

29 PM data
This section describes the Performance Management (PM) data troubleshooting scenarios.

29.1 PM data not retrieved from OSS

Problem

Unable to retrieve Performance Management (PM) data from Operations Support System (OSS).

Possible Cause

Connectivity issue between kpi supplier app and OSS.

Solution

1. The PM data coming from the OSS first gets copied to /home/ vson /ftp/<region>/<re-
gion>. pm.CollectionApp /export/ folder in the PM-Ingestion Node.
2. Check if the _ files.json file timestamp is updated. This file contains the list of file names that
EdenNet fetches from the OSS.

If there are no files listed in the _ files.json file, then it indicates that EdenNet is not able to
fetch the files from OSS.

In this case, check the /home/ vson /log/ enet /<region>. pm.CollectionApp log file,
the following issues might appear:

• OSS user does not have permissions to transfer the file

• SFTP service in the OSS is not working
• The PM file folder in the OSS does not have read permission for the OSS user
3. Check for file parsing failures in the same log file.
4. Check the data under kpi.kpi_15mins table in the PM-DB node.

29.2 KPI not retrieved in EdenNet due to changes in OSS

Problem

KPI not retrieved in EdenNet due to changes in Operations Support System (OSS).

Posible cause

Improper Performance Management (PM) interface configuration.

Solution

Modify PM interface once it is configured.

EdenNet 21 FP 2106 © 2021 Nokia 50

EdenNet Troubleshooting Guide DN09245501 1-2 PM data

1. Create a configuration file with the following information (for example, the file name is
pminterface.ini):

[interface] DaysOfDaily = 180

DaysOfHourly = 28
exportDir = /home/sy4oss/XML
DaysOf15min = 3
ip = 5.194.31.191
user = test
password =Mediati0n$erver!
port = 22
2. Log in to the application server as <installation_user> and switch the user to vson by entering:

sudo -i -u vson
3. Check the current configuration by entering:

enetcfg export -mp region/<regionName>/pm –b

4. Import a new configuration by entering:

enetcfg import -mp region/<regionName>/pm -p pminterface.ini -b –push

5. Validate the changed values by entering:

enetcfg export -mp region/<regionName>/pm –b

6. Restart the pmcollectionApp service by entering:

(enet) [vson@appnode ~]$ enet stop <region>.pm.pmcollectionApp

(enet) [vson@appnode ~]$ enet start <region>.pm.pmcollectionApp

Note:

The PM collection connectivity can be checked using the pmtroubleshooter command

line tool. The pmtroubleshooter command uses the current configuration to connect to
the OSS (mediation server) and check for the PM files. If connection fails or if the PM files
are not found, then a warning is displayed.

For example, log in to the application server as <installation_user> and switch the user to
vson using the sudo -i -u vson command.

(enet) [vson@appnode01 ~]$ pmtroubleshooter

SSH: Using libssh2 for file transfers
INFO |2017-01-13_07:03:23
UTC|pm_troubleshooter:63|16618| Found the following
configured apps:
INFO |2017-01-13_07:03:23
UTC|pm_troubleshooter:66|16618| <region name>
INFO |2017-01-13_07:03:23
UTC|pm_troubleshooter:68|16618|/region/<region name>/pm/app/
pmcollectionApp/config
INFO |2017-01-13_07:03:23
UTC|pm_troubleshooter:79|16618| using normal file downloader
INFO |2017-01-13_07:03:23

EdenNet 21 FP 2106 © 2021 Nokia 51

EdenNet Troubleshooting Guide DN09245501 1-2 PM data

UTC|pm_troubleshooter:84|16618| attempting to connect to

remote server...
INFO |2017-01-13_07:03:24
UTC|pm_troubleshooter:99|16618| looks ok

29.3 PM parsing takes a lot of time

Problem

Performance Management (PM) parsing takes a lot of time and in pmcollectionapp.log you see
the re-spawning messages as the following one:

[uwsgi] | 2018-05-03 11:31:38[+0300] | - OOOPS mule 26 (pid: 15393)

crippled...trying respawn...
[uwsgi] | 2018-05-03 11:31:38[+0300] | - spawned uWSGI mule 26 (pid: 25027)

Possible cause

Insufficient memory or CPU in PM ingestion node.

Note: check_farm utility is used to verify if there is any slowness or issue with PM collec-
tion. If the queue sizes are huge (for example, if the parsing size is more than 10000), then
there is possible issue with PM collection (parsing or aggregation). To check the queue size,
run the following command on all KPI supplier nodes as vson user:

check_farm -r <region_name>

The sample output of check_farm utility is as follows:

check_farm -r <region_name>
Farmer: NADC_PM_worker_farm
Queue Sizes:
aggregate: 123131
aggregate_daily: 2350
combine: 0
commit: 2350
parse:1234561
parse_commit: 0
poll_parse: 0
prune: 0
prune_cells_cache: 0

If there is an issue with PM collection, contact Nokia Technical Support team.

Solution

To increase the memory limit of the mules, do the following:

1. Log in to the KPI supplier node as <installation_user>.

2. Switch the user to vson by entering:

EdenNet 21 FP 2106 © 2021 Nokia 52

EdenNet Troubleshooting Guide DN09245501 1-2 PM data

sudo -i -u vson
3. Open the command line configuration utility by entering:

(enet)[vson@localhost enet]$cfg
4. While in the cfg mode, enter a command and note down the number associated with
pmcollectionApp application in the output.

For example:

-----------------------------------------------------
Enet config editor: Registered apps list
-----------------------------------------------------
0) cron
1) mainApp
2) nadc.pm.kpiApp
3) nadc.pm.pmcollectionApp

5. Type the <number assigned to task app on the cfg> on the command line. For
example:

cfg> a 3
/region/nadc/pm/app/pmcollectionApp
[pmcollectionApp.init]
entrypoint = enetservice.app.pmcollectionApp:init
db_init = enetservice.app.pmcollectionApp:db_init
db_drop = enetservice.app.pmcollectionApp:db_drop
[pmcollectionApp.server]
workers=5
mules=20
[pmcollectionApp.config]
# interfaceConfig = import path to callable to create
parser parserType=PM
get_timeout = 1200
num_files_per_iteration=1000
cell_details_call_timeout=240
# Valid values are fractions in range: 0.5 - 0.9
# Smaller value indicates even distribution and higher values
indicate steeper
# distribution of mules among PM tasks; based on task priorities.
mule_distribution_factor=0.60
#memory limit per mule, default is 1GB
memory_limit=1024

Change the value of memory limit (in Mega Bytes). The default value is 1 GB.
6. Save the file by entering:

:wq!
7. Exit from cfg mode by entering:

cfg> q

EdenNet 21 FP 2106 © 2021 Nokia 53

EdenNet Troubleshooting Guide DN09245501 1-2 PM data

(enet) [vson@ctn1 root]$

8. Restart the pmcollectApp service by entering:

(enet) [vson@appnode ~]$ enet restart <region>.pm.pmcollectionApp

29.4 KPI client failed after two hours without KPI data response
Problem

Module has trace back error while caching KPI data. The KPI client fails after two hours without any
KPI data response. Following is the trace back error:

Traceback (most recent call last):

"/home/vson/enet/lib/python3.7/site-packages/streamlib/streamlib.py",
line 404, in _handle_async_response
raise HTTPError(client.errorMessage)
HTTPError: No status returned

Solution

Modify the KPI cache parameters in the GUI VM and Task VM servers.

1. Log in to the GUI VM server as <installation_user>.

2. Switch to vson user by entering:

sudo -i -u vson

3. Edit the /home/vson/enet/etc/kpicache.ini file and change the value of the following
parameter:

kpis_per_call = 2

Note: By default, the value of kpis_per_call = 4.

4. Save the changes and exit.

5. Repeat steps from 1 to 4 for the Task VM server.

6. Restart all services by entering:

$ enet restart all

Expected outcome

Module should not report trace back error.

EdenNet 21 FP 2106 © 2021 Nokia 54

EdenNet Troubleshooting Guide DN09245501 1-2 Increase the memory limit for module
execution of larger scope cells

30 Increase the memory limit for module execution of

larger scope cells
Problem

Increase the memory limit for module execution of larger scope cells.

Solution

Increase the memory upper limit for the EdenNet module execution instance:

1. Log in to the GUI/Task VM server (for 2VM configuration) or each Task VM server (for 5VM/8VM
configuration) as vson user.

2. Switch user to vson by entering:

sudo -i -u vson

3. Execute the following command:

$ cfg

4. List the apps by entering:

cfg> a

A list of apps are displayed.

5. From the list, identify the number against which the task app is listed.

6. Execute the following command:

cfg> a <the number against which the app is listed>

For example, cfg> a 45

vi of the configuration opens.

7. Edit the following parameter:

instance_memory_limit_mb = 4096

Adjust the value of the above parameter as required. The unit is in MiB. That is, the value must be
<number of GB>*1024.

8. Save the changes and exit.

9. Restart the service by entering:

enet restart task

EdenNet 21 FP 2106 © 2021 Nokia 55

EdenNet Troubleshooting Guide DN09245501 1-2 Increase the memory limit for module
execution of larger scope cells

30.1 For adapted modules

1. Log in to the GUI/Task VM server as vson user.

2. Switch user to vson by entering:

sudo -i -u vson

3. Execute the following command:

$ cfg

4. List the apps by entering:

cfg> a

A list of apps are displayed.

5. From the list, identify the number against which the custom_task app is listed.

6. Execute the following command:

cfg> a <the number against which the app is listed>

For example, cfg> a 9

vi of the configuration opens.

7. Edit the following parameter:

instance_memory_limit_mb = 4096

Adjust the value of the above parameter as required. The unit is in MiB. That is, the value must be
<number of GB>*1024.

8. Save the changes and exit.

9. Restart the service by entering:

enet restart custom_task

Note: Repeat the same steps in each VM for multiple task/custom task nodes. There
must not be any active module execution while performing the same.

EdenNet 21 FP 2106 © 2021 Nokia 56

EdenNet Troubleshooting Guide DN09245501 1-2 Lightweight Directory Access Protocol
(LDAP)

31 Lightweight Directory Access Protocol (LDAP)

This section describes the LDAP troubleshooting scenarios.

31.1 Directory manager password lost and needs to be reset

Problem

The directory manager password is lost and needs to be reset.

Solution

1. Edit the main server configuration file, dse.ldif:

Note: Before editing, ensure to shut down the service. If the server is running and the
dse.ldif file is edited, then the changes will be lost.

stop-dirsrv [edennet]

2. Generate the new password by entering:

/usr/bin/pwdhash newpassword

Step 2 prints out the hashed password string using the default directory manager password hash-
ing scheme for your instance (by default, SSHA), then enter:
3. Navigate to slapd-edennet by entering:

cd /etc/dirsrv/slapd-edennet

4. Edit the dse.ldif.

Note: The server must be shutdown, as performed in step 1.

5. Search for nsslapd-rootpw.

The following line is found:

nsslapd-rootpw: {SSHA}92ls0doP1i0VgQMm8jMjGw27AzVEzyLJS9sj02==

6. Replace the value with the value printed out by pwdhash.

7. Save the file and restart the server by entering:

start-dirsrv [edennet]

EdenNet 21 FP 2106 © 2021 Nokia 57

EdenNet Troubleshooting Guide DN09245501 1-2 Lightweight Directory Access Protocol
(LDAP)

8. Test the new password:

ldapsearch -x -D "cn=directory manager" -w newpassword

31.2 Reset password without admin access to GUI

Problem

If the server has a single administrator and the password is not known, then how to reset the pass-
word without admin access to GUI.

Solution

1. Log in to the database (DB) server with SSH (CENTRAL_VM_DB_SERVERS).

2. Create a new_password.ldif file with the following content:

dn: uid=<enetuser>,ou=Users,ou=Accounts,dc=eden,dc=net
changetype: modify
replace: userPassword
userPassword: <new_password>

3. Ensure to replace enetuser with admin username and new_password with the desired pass-
word.
4. Reset the password by entering:

ldapmodify -a -D "cn=directory manager" -w <directory server password>

-p 389 -h localhost -f new_password.ldif

5. Ensure to obtain directory server password from the installation engineer, or reset it. For more in-
formation, see Directory manager password lost and needs to be reset.

31.3 Locked user

Problem

User account gets locked.

Solution

To unlock the user account, do the following:

1. Log in to the database (DB) server with SSH (CENTRAL_VM_DB_SERVERS).

2. Create the unlock.ldif file with following content:

dn: uid=<enetuser>,ou=Users,ou=Accounts,dc=eden,dc=net
changetype: modify
delete: nsAccountLock
nsAccountLock: true

EdenNet 21 FP 2106 © 2021 Nokia 58

EdenNet Troubleshooting Guide DN09245501 1-2 Lightweight Directory Access Protocol
(LDAP)

3. Ensure to replace enetuser the with correct username.

4. Unlock the user by entering:

ldapmodify -a -D "cn=directory manager" -w <directory server

password> -p 389 -h localhost -f unlock.ldif

5. Ensure to obtain the directory server password from an installation engineer, or reset it. For more
information, see Directory manager password lost and needs to be reset.

EdenNet 21 FP 2106 © 2021 Nokia 59

EdenNet Troubleshooting Guide DN09245501 1-2 Standard software commands

32 Standard software commands

This section lists various operations and the standard software commands required for administering
EdenNet, such as:

• Verifying database connectivity

• Shutting down EdenNet
• Starting up EdenNet
• Shutting down EdenNet hardware
• Starting up EdenNet hardware
• Restarting Tomcat
• Rebooting EdenNet VM during High Availability (HA)
• Listing and verifying services

The commands in the Table 3: Standard software commands must be run on the relevant machine,
that is, use EdenNet commands for EdenNet and use the DB server for MySQL.

Operation Commands

Verifying the database connectivity To verify the database connectivity, enter the following
command from the App server:

mysql –h<hostname> -u<user> -p<password>

--ssl-mode=VERIFY_CA --ssl-ca=/opt/nokia/
mysql/certs/ca_cert.crt --ssl-cert=/opt/
nokia/mysql/certs/mysql_client.crt --ssl-
key=/opt/nokia/mysql/certs/mysql_client.
key

Shutting down EdenNet To stop the services, enter the following command as
vson user:

$ enet stop <name of the service>

To stop frameworks (such as, Redis, EdenNet, and so

on), enter the following command as <installation_user>:

sudo systemctl stop enet

Starting up EdenNet To start the complete framework, enter the following com-
mand as <installation_user>:

sudo systemctl start enet

To start EdenNet, enter the following command as vson

user:

enet start <service name>

EdenNet 21 FP 2106 © 2021 Nokia 60

EdenNet Troubleshooting Guide DN09245501 1-2 Standard software commands

Operation Commands

Shutting down the hardware To shut down EdenNet hardware, do the following:

1. Stop the services by entering:

vson$ enet stop all

2. Stop the framework by entering:

user# sudo systemctl stop enet

3. Shut down the system.

Starting up the hardware EdenNet starts automatically after restart or power off or
power on.

Restarting services To restart a service, enter the following command as

vson user:

vson$enet restart <service>

Rebooting EdenNet VM To reboot EdenNet VM, do the following:

1. Stop the services by entering:

vson$ enet stop all

2. Stop the framework by entering:

user# sudo systemctl stop enet

3. Reboot the system.

Listing and Verifying services To list and verify the EdenNet services, enter the follow-
ing command:

enet status

For example,

enet status

antenna_plan_app is ACTIVE
appServersApp is ACTIVE
cellPlanApp is ACTIVE
tomcat is ACTIVE
user_manager is ACTIVE
...

Note: Service name changes between the

servers and EdenNet versions.

EdenNet 21 FP 2106 © 2021 Nokia 61

EdenNet Troubleshooting Guide DN09245501 1-2 Standard software commands

Table 3: Standard software commands

Note: To verify the result, use the Listing and Verifying services using enet status com-
mand.

EdenNet 21 FP 2106 © 2021 Nokia 62

EdenNet Troubleshooting Guide DN09245501 1-2 EdenNet list of services

33 EdenNet list of services

This section lists EdenNet services and the nodes in which they are running:

Service Description Nodes

firewalld FirewallD is the Dynamic Fire- All the nodes

wall Manager used to configure
the network connections.

mysqld MySQL is the primary DB used All the DB nodes

for EdenNet.

cassandra_kong Cassandra is used as a primary CENTRAL_VM_DB_SERVERS

DB by Kong.

rpcbind It is a supporting process for re- • GUI_SERVERS

mote procedure calls (RPC). • TASK_SERVERS
• CUSTOM_TASK_
SERVERS
• CENTRAL_VM_SERVERS

nfslock It handles file locking for remote • GUI_SERVERS

and local NFS requests. • TASK_SERVERS
• CUSTOM_TASK_
SERVERS
• CENTRAL_VM_SERVERS

nfs NFS is a server/client based • GUI_SERVERS

networking protocol. • TASK_SERVERS
• CUSTOM_TASK_
SERVERS
• CENTRAL_VM_SERVERS

dirsrv.target It is an ldap server developed by CENTRAL_VM_DB_SERVERS

RHEL.

keycloak Keycloak is an authentication CENTRAL_VM_SERVERS

and authorization server.

rsyslog It is used for forwarding log All the nodes

messages in an IP network.

rabbitmq-server RabbitMQ is a messaging bro- CENTRAL_VM_SERVERS

ker; It provides a common plat-

EdenNet 21 FP 2106 © 2021 Nokia 63

EdenNet Troubleshooting Guide DN09245501 1-2 EdenNet list of services

Service Description Nodes

form to send and receive mes-

sages.

ckng-kong Kong is an API gateway and CENTRAL_VM_SERVERS

Platform. CSF's CKNG is the
wrapper for Kong.

Table 4: List of EdenNet services

You can use the following commands to:

• check the status of the services

• start, stop, or restart services or apps

Operation Commands

Check the status of the service # service <service_name> status

Start service # service <service_name> start

Stop service # service <service_name> stop

Restart service # service <service_name> restart

Start apps #enet start <app_name1> [ <app_name2> .... ]

Start all apps at once #enet start all

Stop apps #enet stop <app_name1> [ <app_name2> .... ]

Stop all apps at once #enet stop all

Restart apps #enet restart <app_name1> [ <app_name2> .... ]

Stop all apps at once #enet restart all

Table 5: Standard software commands

Note: To start, stop, or restart apps, you must switch to vson user by executing:

#sudo -i -u vson

EdenNet 21 FP 2106 © 2021 Nokia 64

EdenNet Troubleshooting Guide DN09245501 1-2 Shutting down the virtual machines

34 Shutting down the virtual machines

Note: Do the following procedure on all virtual machines (VM) in the system, except on the
vCenter VM.

1. Access the vSphere Web client.

2. From the vSphere Web Client Home page, click VMs and Templates.

3. Click your vCenter server, and click the Related Objects tab.

4. Click the Virtual Machines tab.

5. Right-click a VM that is not the vCenter VM, and then click Power → Shut Down Guest OS.

6. To confirm the shut down, click Yes.

7. Repeat steps 5 through 6 on all the other VMs, except the vCenter VM.

All the virtual machines are shut down successfully.

EdenNet 21 FP 2106 © 2021 Nokia 65

EdenNet Troubleshooting Guide DN09245501 1-2 Starting up a virtual machine

35 Starting up a virtual machine

To start a virtual machine (VM), do the following:

1. Access the vSphere Web client.

2. From the vSphere Web Client Home page, click VMs and Templates.

3. Select your vCenter server, and click the Related Objects tab.

4. Click the Virtual Machines tab.

5. Right-click a VM, and then click Power → Power On.

6. Repeat steps 4 through 5 for all the other remaining VMs.

All the VMs are started successfully.

EdenNet 21 FP 2106 © 2021 Nokia 66

EdenNet Troubleshooting Guide DN09245501 1-2 Stop and start EdenNet for system
shutdown

36 Stop and start EdenNet for system shutdown

This section covers the following topics:

• Stopping EdenNet services

• Stopping EdenNet framework
• Stopping databases in the DB server
• Starting EdenNet

36.1 Stopping EdenNet services

Note: Execute each of the commands described in this procedure on the following nodes:

• CM Framework
• GUI Server
• PM Ingestion

1. As a vson user, stop the services on each node by entering:

vson$ enet stop all

2. Verify the status of the services by entering:

vson$ enet status

All the services are stopped successfully.

36.2 Stopping EdenNet framework

Note: Execute the following commands in the following nodes:

• CM Framework
• GUI Server
• PM Ingestion

1. As a <installation_user>, stop the services by entering:

user# sudo systemctl stop enet

2. Validate that redis and nginx programs are stopped by entering:

(ps -ef | grep -e redis -e nginx)

EdenNet 21 FP 2106 © 2021 Nokia 67

EdenNet Troubleshooting Guide DN09245501 1-2 Stop and start EdenNet for system
shutdown

Note:

• If redis does not stop after a couple of minutes, enter one of the following command:

• redis-cli shutdown
Or

• sudo pkill -x redis-server

• If nginx does not stop, enter:

sudo pkill -3 -F /usr/local/nginx/logs/nginx.pid

The EdenNet framework is stopped.

36.3 Stopping databases in the DB server

1. As a <Installation_user> enter the following commands in both CM and PM databases:

sudo systemctl stop mysqld

2. Validate that mysql is stopped.

36.4 Starting EdenNet

Start or restart the virtual machines (VM):

Note: Ensure to follow the order, as described in the steps.

1. Start the CM and PM databases.

2. Wait for the databases to start.

3. Check that both DB VM's are powered on and validate that MySQL is running by connecting to it.
To connect locally, enter:

mysql –h<hostname> -u<user> -p<password> --ssl-mode=VERIFY_CA --ssl-

ca=/opt/nokia/mysql/certs/ca_cert.crt --ssl-cert=/opt/nokia/mysql/certs/
mysql_client.crt --ssl-key=/opt/nokia/mysql/certs/mysql_client.key

4. Power on the CM/Framework VM.

EdenNet should start automatically after the power-on. Validate by entering the following command
as vson user:

vson$ enet status

EdenNet 21 FP 2106 © 2021 Nokia 68

EdenNet Troubleshooting Guide DN09245501 1-2 Stop and start EdenNet for system
shutdown

Note: Services must have ACTIVE status. If they appear as INACTIVE, wait and check
again. If the status is still INACTIVE, run the following command:

enet start all

5. Execute step 4 in the GUI, PM ingestion, and task VMs.

EdenNet 21 FP 2106 © 2021 Nokia 69

EdenNet Troubleshooting Guide DN09245501 1-2 Module reports are missing in GUI

37 Module reports are missing in GUI

Problem

Module report is missing from the GUI and the Instance Outputs button is disabled.

Possible cause

Loss of NFS mounting on the Task VM where module is executed.

Solution 1 (Direct OSS integration)

1. Log in to the Task VM as <installation_user> and remount the shared directories by entering:

#/opt/nokia/backupandrestore/nfs_mount_script/remount_nfs_directories.sh
2. Verify the shared directories by entering:

#mount -l | grep nfs

Solution 2 (NAdC integration)

1. Log in to the Task VM as <installation_user> and remount the shared directories by entering:

#/opt/nokia/backupandrestore/nfs_mount_script/remount_nfs_directories.sh
2. Log in to the Central App server(s) as <installation_user> and enter:
#sudo mount -t nfs -o nosuid,nodev,noexec <GUI-IP>:/srv/nfs/mod-
ules/userOutputFiles /home/modules/eData/userOutputFiles

Note: If EdenNet is deployed in dual or standalone mode, then the GUI IPv6 address
must be enclosed with square brackets. For example, [2a00:abcd::0000]:/srv/nfs/mod-
ules/userOutputFiles.

3. Verify the shared directories by entering:

#mount -l | grep nfs

Result

The output must include the following line:

<GUI-IP>:/srv/nfs/modules/userOutputFiles on /
home/modules/eData/userOutputFiles type nfs
(rw,nosuid,nodev,noexec,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard
IP>,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=<GUI-IP>)

Note: If the GUI_IP is IPv6, then it is enclosed by square brackets.

EdenNet 21 FP 2106 © 2021 Nokia 70

EdenNet Troubleshooting Guide DN09245501 1-2 Hotfix information is not available pop-up
appears

38 Hotfix information is not available pop-up appears

Problem

In the EdenNet GUI, when the user clicks Help → Hotfix Details .

The Hotfix information is not available pop-up appears.

Possible causes

Following are the possible causes and the relevant error log messages are available in tomcat.log
file.

• The hotfix file /home/vson/tmp/log/hf_info.txt does not exist in the GUI VM.
• The vson or module user does not have read permission for /home/vson/tmp/log/
hf_info.txt hotfix file in UI VM.
• The hotfix file is empty.

Solution

• Verify if any EdenNet hotfixes are installed in the system via the hotfix framework which creates
hf_info.txt hotfix file under the /home/vson/tmp/log folder.
• Verify that the hf_info.txt hotfix file has read permissions for vson or module users.
• Verify that the hf_info.txt hotfix file is not empty and has data to view.

EdenNet 21 FP 2106 © 2021 Nokia 71

EdenNet Troubleshooting Guide DN09245501 1-2 Delay in displaying hotfix details window

39 Delay in displaying hotfix details window

Limitation

There is a delay (in seconds) in displaying the Hotfix Details window.

This delay occurs whenever the hf_info.txt hotfix file size is huge (> 1MB) due to multiple hotfixes
installed in the system.

Table 6: Lab performance statistics example provides some example statistics details from the lab
performance testing.

Total Time Taken to load

Hf_info.txt
Client Laptop OS Browser the hotfix file for viewing
file size
in EdenNet UI (Seconds)

Windows 10 Google Chrome 2MB 3.24s

Windows 10 Google Chrome 3MB 5.50s

Windows 10 Google Chrome 4MB 6.67s

Windows 10 Google Chrome 5MB 7.60

Table 6: Lab performance statistics example

EdenNet 21 FP 2106 © 2021 Nokia 72

EdenNet Troubleshooting Guide DN09245501 1-2 All modified MOs under a BSC/RNC are
not seen in the Rollback tab when filters
are selected

40 All modified MOs under a BSC/RNC are not seen in

the Rollback tab when filters are selected
Problem

If the user selects a BSC or RNC or any controller in the cells filter, some of the modified managed ob-
ject (MO) types are not listed in the Rollback tab.

Possible Causes

There is a limitation while trying to find managed objects types to rollback based on date and BSC/
RNC/MRBTS selection. Only controllers with less than or equal to 100 cells show all the MOs modi-
fied.

If the controllers have more than 100 cells, changes from some of the cells are not listed and some
MO types are not seen.

Solution

• Do not select whole BSC/RNC or any controller, if it has more than 100 cells.
• If the cells to be rolled back are known, and are less than 100, select the cells.

If changes of more than 100 cells must be seen, do not select any cell filters.

Perform the following:

1. In the Rollback Date field, enter the date as required.

2. Click Get Changes.

A list of all managed object types changed in the network since the specified date is displayed.
3. From the displayed list, select the required cell and click Rollback Selected Changes.

EdenNet 21 FP 2106 © 2021 Nokia 73

EdenNet Troubleshooting Guide DN09245501 1-2 Modules fail due to presence of duplicate
externals pointing to the same native cell

41 Modules fail due to presence of duplicate externals

pointing to the same native cell

Problem

Modules fail with the below exception due to the presence of duplicate externals pointing to the same
native cell:

<Cell x DN>
Illegal dependency. Skipping.
This object was earlier resolved as a source of <Cell y DN>, but now
appears as a source of <Cell z DN>
2019-02-08 12:46:38: Warning

Solution

Before provisioning a plan from EdenNet, operators must manually clear duplicate external represen-
tation on the OSS.

EdenNet 21 FP 2106 © 2021 Nokia 74

EdenNet Troubleshooting Guide DN09245501 1-2 Upgrade failure due to vault tokens not
being generated

42 Upgrade failure due to vault tokens not being

generated

Problem

Upgrade fails due to vault tokens not being generated.

Note: The failure does not occur during customer operation, but occurs when upgrades are
run for more than 100-150 times usually in Continuous Integration (CI).

Possible causes

• When Inode usage is checked at /var, if df-ih usage is above 100%, then vault or any service
which writes to /var will not work.
• The following error occurs in /var/log/messages

core: failed to create token: error="failed to

persist entry: open /var/vault/sys/token/id/_
h199d60a8869d6a821fb7ed588c3305e6e703d0f815255f422b6ab59e969bb646: no
space left on device"

Solution

CAUTION! If any other files are removed, the vaults can get sealed and cannot be retrieved.
Utmost care must be taken as this an extremely risky procedure. Before executing the steps
Contact Nokia Technical support.

1. Find the directories to be deleted by entering:

ls -d /var/vault/auth/*/accessor

There are multiple random folders under auth.

Example

ls -d /var/vault/auth/*/accessor

/var/vault/auth/fe9814d0-6720-ad0b-c4a3-97cc9673e812/accessor

Replace * with the <random folder> as shown in the example by entering:

find /var/vault/auth/*/accessor -mindepth 1 -delete

2. Find the folder which contains secret_id by entering:

ls -d /var/vault/auth/*/secret_id

There are multiple random folders under auth.

Example

EdenNet 21 FP 2106 © 2021 Nokia 75

EdenNet Troubleshooting Guide DN09245501 1-2 Upgrade failure due to vault tokens not
being generated

ls -d /var/vault/auth/*/secret_id

/var/vault/auth/fe9814d0-6720-ad0b-c4a3-97cc9673e812/secret_id

Replace * with the <random folder> as shown in the example by entering:

find /var/vault/auth/*/secret_id -mindepth 1 -delete

3. Find the folder which contains accessor by entering:

ls -d /var/vault/logical/<random_folder>/

There are four random folders under logical.

Example

ls -d /var/vault/logical/*

/var/vault/logical/616c2545-9719-ba67-1b10-74543d188eaa

/var/vault/logical/ba3ea69a-9d48-f9dc-002c-e135da7ada2a

/var/vault/logical/81851c1c-33ef-3cdc-3e2a-03ab2c200e82

/var/vault/logical/f7c3fdd0-8d8f-2828-8ff9-f63de466ea1c

Find the <random folder> which doesn’t have any of the following files:

• EdenNet
• pki
• packer
• role
• certs
• config
• _ca
• _urls
• _crl

Note: Ensure you delete the correct folder.

Replace the <random folder> value as shown in the example by entering:

find /var/vault/logical/<random_folder> -mindepth 1 -delete

EdenNet 21 FP 2106 © 2021 Nokia 76

EdenNet Troubleshooting Guide DN09245501 1-2 API operations on AC not working
during Keycloak based AC integration to
EdenNet

43 API operations on AC not working during Keycloak

based AC integration to EdenNet

Problem

API operations on AC are not working during Keycloak based AC integration to EdenNet.

Symptoms

HTTP/1.1 403 Forbidden

Expires: 0
Connection: keep-alive
Cache-Control: no-cache, no-store, must-revalidate
Pragma: no-cache
Content-Type: application/vnd.nokia-error-response+json
Content-Length: 84
Date: Wed, 23 Sep 2020 06:39:39 GMT
{"userMessage":"Not authorized","developerMessage":"Not authorized",
"errorCode":192}

Solution

• Enter the following command on APP VM of AC as root user:

[root@as ~]# /opt/jboss-eap/bin/jboss-cli.sh --commands='connect,/

system-
property=com.nokia.oss.cm.product:add(value="racbas.AuthorizationAdmin=disabled")
~]# systemctl restart jboss-eap

EdenNet 21 FP 2106 © 2021 Nokia 77

EdenNet Troubleshooting Guide DN09245501 1-2 API invocation is not working on KPI
Supplier VM

44 API invocation is not working on KPI Supplier VM

Problem

API invocation is not working on KPI Supplier VM

Symptoms

Check for the following symptoms:

2021/02/12 16:36:33 [error] 2018#2018: *4292 open() "/etc/nginx/html/

region/nadc/pm/app/pmcollectionApp" failed (13: Permission denied),
client: 127.0.0.1, server: , request: "GET //region/nadc/pm/app/
pmcollectionApp HTTP/1.1", host: "localhost:9600"
2021/02/12 16:36:33 [error] 2021#2021: *4295 open() "/etc/nginx/html/
region/nadc/pm/app/kpiApp" failed (13: Permission denied), client: 127.0.
0.1, server: , request: "GET //region/nadc/pm/app/kpiApp HTTP/1.1", host:
"localhost:9600" 2021/02/12 16:36:33 [error] 2018#2018: *4298 open() "/etc/
nginx/html/region/nadc/pm/app/nrtPmCollectionApp" failed (13: Permission
denied), client: 127.0.0.1, server: , request: "GET //region/nadc/pm/app/
nrtPmCollectionApp HTTP/1.1", host: "localhost:9600"

Solution

1. Log in as vson user.

2. Unregister PM Apps by entering:
$ enet bootstrap --timeout 180

$ enet restart all

EdenNet 21 FP 2106 © 2021 Nokia 78

EdenNet Troubleshooting Guide DN09245501 1-2 EdenNet logs

45 EdenNet logs
The EdenNet logs are located in the following locations:

• Module specific log files are located at /home/vson/logs in the application server.

For each module, a folder is created and when the user runs a module, the logs are stored in this
folder.

• EdenNet services related log files are located at /home/vson/log/enet in the application
server.

For each service in EdenNet, a log file is created in this path.

• Module instance log path (module output files) are located at /home/vson/eData/userOut-
putFiles/<user> folder in the application server.

• Installation logs are located at <home_of_installation_user>/ansible.log folder in the

control server.

• Internal LDAP server logs are available in the VM DB node in the /var/log/dirsrv/slapd-
edennet folder.

The important files are errors and access, which provides information for troubleshooting.

Another important log for troubleshooting Lightweight Directory Access Protocol (LDAP) in general
(including external server integrations) is the log file on the GUI server at /home/vson/log/
enet/user_manager.log.

45.1 Log directory layout

Table 7: EdenNet details of log locations lists the log location for EdenNet.

Log type Node Location

EdenNet uWSGI apps GUI_SERVERS /home/vson/log/enet

CENTRAL_VM_SERVERS

KPI_SUPPLIER_SERVERS

TASK_SERVERS

ASC GUI_SERVERS /home/vson/log/ automated-

sitecreation

Licensing service GUI_SERVERS /var/log/ edennet/ licens-

ingservice

EdenNet 21 FP 2106 © 2021 Nokia 79

EdenNet Troubleshooting Guide DN09245501 1-2 EdenNet logs

Log type Node Location

Event logs CENTRAL_VM_SERVERS /home/vson/log/logging

Too large events GUI_SERVERS /home/vson/log/ event_log-

ging
CENTRAL_VM_SERVERS

KPI_SUPPLIER_SERVERS

TASK_SERVERS

EMS validation report CENTRAL_VM_SERVERS /home/vson/log/ ems_valida-

tion_reports

Logrotate state GUI_SERVERS /home/vson/log/enet/ logro-

tation.state
CENTRAL_VM_SERVERS

KPI_SUPPLIER_SERVERS

TASK_SERVERS

sysBackup GUI_SERVERS /home/vson/logs/maint/backup

CENTRAL_VM_SERVERS

KPI_SUPPLIER_SERVERS

TASK_SERVERS

dbBackup GUI_SERVERS /home/vson/logs/maint/db-

Backups
CENTRAL_VM_SERVERS

KPI_SUPPLIER_SERVERS

TASK_SERVERS

Purging GUI_SERVERS /home/vson/logs/maint/purge

CENTRAL_VM_SERVERS

KPI_SUPPLIER_SERVERS

TASK_SERVERS

dbMaint GUI_SERVERS /home/vson/logs/maint/db-

Maint
CENTRAL_VM_SERVERS

KPI_SUPPLIER_SERVERS

TASK_SERVERS

sysmonitor GUI_SERVERS /home/vson/logs/syslogs

CENTRAL_VM_SERVERS

EdenNet 21 FP 2106 © 2021 Nokia 80

EdenNet Troubleshooting Guide DN09245501 1-2 EdenNet logs

Log type Node Location

KPI_SUPPLIER_SERVERS

TASK_SERVERS

SON Modules logsbase /home/vson/logs

dir

SON Modules GUI_SERVERS /home/vson/logs/ scriptIn-

stances

SON Modulesdebug/er- GUI_SERVERS /home/vson/logs/{moduleName}

ror

Ansible Installation server /var/log/ansible.log

NGINX GUI_SERVERS /home/vson/log/ nginx_error.

log
CENTRAL_VM_SERVERS

KPI_SUPPLIER_SERVERS

TASK_SERVERS

Redis GUI_SERVERS /home/vson/log/redis*.log/

home/custom/log/custom_re-
CENTRAL_VM_SERVERS
dis.log
KPI_SUPPLIER_SERVERS

TASK_SERVERS

Tomcat GUI_SERVERS /home/vson/opt/tomcat/logs

Tomcat GC GUI_SERVERS /home/vson/log/enet/tom-

cat-gc

RabbitMQ CENTRAL_VM_SERVERS /var/log/rabbitmq

MySQL error GUI_DB_SERVERSCENTRAL_ /home/data/mysql/{hostname}.

VM_DB_SERVERSKPI_SUP- err
PLIER_DB_SERVERSSELF-
MON_DB_SERVER

389 DS CENTRAL_VM_DB_SERVERS /var/log/dirsrv/slapd-eden-

net

Table 7: EdenNet details of log locations

Self Monitoring

Table 8: Self Monitoring new log directory layout explains the directory layout of self monitoring logs.

EdenNet 21 FP 2106 © 2021 Nokia 81

EdenNet Troubleshooting Guide DN09245501 1-2 EdenNet logs

Log type Node Location

Zabbix agent All /var/log/zabbix

Zabbix server SELFMON_SERVER /var/log/zabbix

Apache httpd SELFMON_SERVER /var/log/httpd

Table 8: Self Monitoring new log directory layout

45.2 Copying log files

You can copy all logs to a zip file.

1. Log in to the App server as a <installation_user> and switch the user to vson by entering:

sudo -i -u vson

2. Collect the available log files by entering:

(enet) [vson@appnode ~]$ edebug

These log files are used for troubleshooting.

Note:

• All the available service logs can be fetched by entering:

edebug
• Logs for an instance can be collected by entering:

edebug -t instance -i <instance_name> -days <number_of_days>

This command collects the instance log with the given instance name and also all the
service logs for the number of days specified (default will fetch complete log file avail-
able).
• Service logs can also be collected by specifying the number of days from the
present. Fetch the log by entering:
edebug -t service -days <number_of_days>

This command copies all the log files to a ZIP file under /home/vson/debug folder.
Copy the ZIP file and provide it to technical support engineer for further analysis.

EdenNet 21 FP 2106 © 2021 Nokia 82

EdenNet Troubleshooting Guide DN09245501 1-2 Cognitive SON configuration files purging

46 Cognitive SON configuration files purging

It is required to clear the old configuration files added into EdenNet through Cognitive SON. Purging
configuration files regularly ensures that the file system is not full and thus avoids VM failures due to
file system being full.

Users can:

• Edit the iniCleanUpCronShallRunAt parameter in the cronService.ini file of the GUI

node to change the time at which the job is run.
• Edit the userToPurgeIniFiles parameter in the cronService.ini file of the GUI node to
change the owner of the configuration files.
• Edit the numberOfDaysToRetainIniFiles parameter in the cronService.ini file of the
GUI node to change the number of days before which instances and configuration files are consid-
ered.

After the parameters are edited, restart the CRON APP in the GUI node.

EdenNet 21 FP 2106 © 2021 Nokia 83

EdenNet Troubleshooting Guide DN09245501 1-2 Frequently Asked Questions (FAQ)

47 Frequently Asked Questions (FAQ)

Table 9: List of FAQs provides the frequently asked questions and its answers relevant to EdenNet.

S.No. Question Answer

1 What Apps will be running in the Cen- • Cell plan

tral VM? • Event logging
• Main app
• CM cache
• CM main
• SON Engine

2 Which App is responsible to fetch the <region>. cm.cache

CM Export from the OSS?

3 Which App is responsible to maintain <region>. cm.cache

the CM Cache table in SON?

4 Name the App that is responsible to <region>. kpi.pmcollectionApp

fetch the PM Data from the OSS?

5 List the PM KPI tables created per Re- • <region>_15mins

gion. • <region>_Hour
• <region>_Daily

6 List the CM tables created per Region. • <region>_cache

• <region>_cells
• <region>_ neighbors
• <region>_ cell_relations
• <region>_changes
• <region>_ cache_changes
• <region>_schedule
• <region>_ sc_disambig
• <region>_band

7 What is the role of tomcat app? Provides UI functionality

8 In which VM tomcat App will be run- GUI VM

ning?

9 In which table will all the cells of a re- <region>_cells

gion be stored?

EdenNet 21 FP 2106 © 2021 Nokia 84

EdenNet Troubleshooting Guide DN09245501 1-2 Frequently Asked Questions (FAQ)

S.No. Question Answer

10 In which table are all the neighbor rela- <region>_neighbours

tions in a region stored?

11 In which table are the cell plan data cell_plan

stored?

12 In which table are all the cells of all the cell_details

regions stored?

13 In which table are the application ser- central_vm.app_servers

vices URL or details stored?

14 Which app service will delegate read- <region>. pm.kpiBrokerApp

ing the PM data from respective region
PM nodes?

15 Do we get a PM App per region in the Yes. It is named as

Central VM? If yes, name it.
<region>. pm.kpiBrokerApp

16 What are the SON modules that need • ANR

lat /long/azimuth of a cell? Name • MLB
them.
• CCO
• CAFD
• SC conflicts

17 What are the SON modules that do not • Parameter Consistency Enforcement
need lat /long/azimuth of a cell? (PCE)
• kpiChecker

18 What are the considerations that need The Cell_Id value should be of lcrid para-
to be taken care for creating cell plan meter of a LCEL
data of LTE cells?

19 What is the command used for import- edw target <cell plan csv file>
ing cell plan data?

20 In which location are the app server /home/ vson /log/ enet
logs located?

21 In which VM will the SON module in- GUI VM

puts, outputs, configurations, or logs be
written?

22 What is the path of the SON Module in- /home/ vson / eData /
puts, outputs, configurations, or logs?

EdenNet 21 FP 2106 © 2021 Nokia 85

EdenNet Troubleshooting Guide DN09245501 1-2 Frequently Asked Questions (FAQ)

S.No. Question Answer

23 Which is the mandatory App that is re- <region>. pm.kpiBrokerApp

quired to view KPIs and charts in UI?

24 Which App populates the data from <region>. cm.main

cache table to various CM tables?

25 What is the CM Database schema ems_cache

name?

26 List the tables in which the SON KPIs • <region>_15mins

are stored. • <region>_Hour
• <region>_Daily

27 Which backend service will contain the uwsgi / nginx

Rest services?

28 What is the command to view and cfg

change configuration apps, services,
and so on?

29 What is the command to create, re- rege

fresh, or delete the region?

30 Can one region represent multiple No

OSS?

31 What is the command to check status enet status

of Apps?

32 In the DB VMs, name the running SON No. In DB VM only mysql service runs. No
Apps. SON Apps will run.

33 What are the GUI VM apps or services • Provide UI capabilities

responsible for? • Running SON Modules
• Storing SON module inputs/outputs/logs

34 What are the different ways to create a • Import Cell list

cluster in the Topology UI? • Select it from Map

35 Which command is used to check if the cpm

KPI data is available in the PM DB?

36 In which VM and folder are the region In central VM at /home/ vson /ftp/
specific CM exported files present? <region>/

37 In which folder, the last fetched PM file /home/ vson /ftp/<region>/< re-
names will be written? gion.pm.collectionApp >/export/

EdenNet 21 FP 2106 © 2021 Nokia 86

EdenNet Troubleshooting Guide DN09245501 1-2 Frequently Asked Questions (FAQ)

S.No. Question Answer

38 What is the command to enforce topol- cmtask

ogy or metadata upload on a specific
Region?

39 In which App log can we see the CM In Central VM at <region>. cm.cache

export status?

40 If the PM files parsing is failed, the /home/ vson /ftp/<region>/< re-

failed files will be stored in which loca- gion.pm.collectionApp >/ failed_
tion? parses /

41 In which table, different frequency <region>_band

bands are stored?

42 How to fetch the PM Counter values kpi_checker can be used to fetch the
from UI? Which module to be used? counter values. If we know the KPIs, then
use the KPI charts in Topology

43 Which table stores the parameters of <region>_cache

the cell objects?

44 How to fetch the CM parameters of a Under Topology select CM and enter cell
specific cell from UI? name.

45 Which table stores the historical CM <region>_changes

changes pushed by SON?

46 In which table SC disambiguities will be <region>_ sc_disambig

stored?

47 In which table the site specific unique location

location is stored?

48 Does EdenNet support Self-Healing Yes, the supported Self-Healing use cases
use cases? are:

• MLB
• Sleeping Cell
• Alarm Based Outage Resolution

49 In which VM the SON Modules will be GUI App

running?

50 Which log shows the PM file fetch sta- region.pmcollectionApp.log

tus?

Table 9: List of FAQs

EdenNet 21 FP 2106 © 2021 Nokia 87