Epicor ERP

Performance and Diagnostic

Advanced - On Premise Course
10.2.600
Disclaimer
This document is for informational purposes only and is subject to change without notice. This document and its
contents, including the viewpoints, dates and functional content expressed herein are believed to be accurate as of its
date of publication. However, Epicor Software Corporation makes no guarantee, representations or warranties with
regard to the enclosed information and specifically disclaims any applicable implied warranties, such as fitness for a
particular purpose, merchantability, satisfactory quality or reasonable skill and care. As each user of Epicor software is
likely to be unique in their requirements in the use of such software and their business processes, users of this document
are always advised to discuss the content of this document with their Epicor account manager. All information contained
herein is subject to change without notice and changes to this document since printing and other important information
about the software product are made or published in release notes, and you are urged to obtain the current release
notes for the software product. We welcome user comments and reserve the right to revise this publication and/or
make improvements or changes to the products or programs described in this publication at any time, without notice.
The usage of any Epicor software shall be pursuant to an Epicor end user license agreement and the performance of
any consulting services by Epicor personnel shall be pursuant to Epicor's standard services terms and conditions. Usage
of the solution(s) described in this document with other Epicor software or third party products may require the purchase
of licenses for such other products. Where any software is expressed to be compliant with local laws or requirements
in this document, such compliance is not a warranty and is based solely on Epicor's current understanding of such laws
and requirements. All laws and requirements are subject to varying interpretations as well as to change and accordingly
Epicor cannot guarantee that the software will be compliant and up to date with such changes. All statements of
platform and product compatibility in this document shall be considered individually in relation to the products referred
to in the relevant statement, i.e., where any Epicor software is stated to be compatible with one product and also
stated to be compatible with another product, it should not be interpreted that such Epicor software is compatible
with both of the products running at the same time on the same platform or environment. Additionally platform or
product compatibility may require the application of Epicor or third-party updates, patches and/or service packs and
Epicor has no responsibility for compatibility issues which may be caused by updates, patches and/or service packs
released by third parties after the date of publication of this document. Epicor® is a registered trademark and/or
trademark of Epicor Software Corporation in the United States, certain other countries and/or the EU. All other
trademarks mentioned are the property of their respective owners. Copyright © Epicor Software Corporation 2020.
All rights reserved. Not for distribution or republication. Information in this document is subject to Epicor license
agreement(s).

EDE8508905
90521-10-9775-583101001
10.2.600
Revision: July 06, 2020 5:15 a.m.
Total pages: 90
course.ditaval
Performance and Diagnostic Advanced - On Premise Course Contents

Contents
Introduction............................................................................................................................6
Before You Begin....................................................................................................................7
Audience.........................................................................................................................................................7
Prerequisites....................................................................................................................................................7
Environment Setup..........................................................................................................................................7
Advanced Performance Tuning Resources............................................................................9
Hardware Sizing Guide....................................................................................................................................9
VMWare Best Practices Guides.........................................................................................................................9
Standard Metrics and Evaluation........................................................................................10
Review the Hardware Recommendation.........................................................................................................11
Determine Hardware Sizing....................................................................................................................11
Case Study: Hardware Sizing..................................................................................................................11
Standard Metrics............................................................................................................................................12
Workshop – SAN (I/O) Test Utility............................................................................................................13
Run the Diskspd Utility.....................................................................................................................14
Workshop - GetRowsKeepIdleTime Test..................................................................................................16
Review GetRowsKeepIdleTime Chart................................................................................................17
Workshop – Check Configuration...........................................................................................................18
Configure Performance and Diagnostic Tool (Network)....................................................................18
Run Config Check...........................................................................................................................19
Case Study: Configuration Check....................................................................................................20
Workshop – Network Performance.........................................................................................................21
Test the Network.............................................................................................................................21
Workshop – Customer Retrieval Baseline Test.........................................................................................22
Run the Customer Retrieval Test......................................................................................................22
Workshop – Sales Order Baseline Test.....................................................................................................23
Create the Sales Order Header.........................................................................................................23
Create the Spreadsheet...................................................................................................................24
Start the Tracing Log.......................................................................................................................25
Copy and Paste the Detail Lines.......................................................................................................25
Generate the Results.......................................................................................................................25
Workshop - Purchase Order Baseline Test...............................................................................................26
Create the Purchase Order Header...................................................................................................26
Create the Spreadsheet...................................................................................................................27
Start the Tracing Log.......................................................................................................................28
Copy and Paste the Detail Lines.......................................................................................................28
Generate the Results.......................................................................................................................28
Infrastructure Diagram...................................................................................................................................29
Instance Diagram...........................................................................................................................................30
The Performance Report................................................................................................................................31

Epicor ERP | 10.2.600 3

Contents Performance and Diagnostic Advanced - On Premise Course

Memory Tests........................................................................................................................33
Live Memory Inspection.................................................................................................................................33
Workshop: Live Memory Test..................................................................................................................34
Setup..............................................................................................................................................34
Run Live Memory Test.....................................................................................................................35
Memory Dump................................................................................................................................36
Get External DAC............................................................................................................................37
Memory Dump Compare.................................................................................................................37
Memory Leaks...............................................................................................................................................38
Workshop: Memory Dump File...............................................................................................................39
Create Memory Dump File...............................................................................................................39
Memory Dump Error........................................................................................................................40
Distribute the Load...............................................................................................................42
Application Servers........................................................................................................................................42
Task Agents...................................................................................................................................................43
Load Balancer................................................................................................................................................43
Security Groups.............................................................................................................................................43
System Agent................................................................................................................................................44
Workshop - New Application Server...............................................................................................................44
Add the Application Server.....................................................................................................................44
Select the Database................................................................................................................................46
Admin Console Settings..........................................................................................................................47
Reporting Services..................................................................................................................................48
Create the Task Agent............................................................................................................................50
Create a Task Agent Rule........................................................................................................................52
SQL Server Tuning................................................................................................................56
Workshop: Review Server Properties..............................................................................................................56
SQL Server Properties..............................................................................................................................56
Database Properties................................................................................................................................57
TempDB Files.................................................................................................................................................59
System Tips....................................................................................................................................................60
Microsoft Tools.....................................................................................................................61
Event Viewer..................................................................................................................................................61
SQL Performance Dashboard..........................................................................................................................62
Workshop: Install Performance Dashboard..............................................................................................62
Performance Analysis of Logs (PAL) Tool........................................................................................................63
Workshop - Use PAL Tool.......................................................................................................................64
Activate PowerShell.........................................................................................................................64
Gather Logs.....................................................................................................................................64
Generate Performance Analysis........................................................................................................65
SQL Server Trace Flags.........................................................................................................67
Workshop: Activate SQL Server Trace Flags....................................................................................................68
Add to Startup Parameters......................................................................................................................68
Add to Registry.......................................................................................................................................69

4 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Contents

Custom Trace Logs................................................................................................................71

Workshop: Customize Server Logs.................................................................................................................71
Workshop: Add Server Traces to Client Log....................................................................................................72
Workshop: Auto Capture Client Logs.............................................................................................................73
Deadlocks..............................................................................................................................75
Workshop: Detecting Deadlocks....................................................................................................................75
Activate the SQL Deadlock Graph...........................................................................................................76
Stop the SQL Deadlock Graph.................................................................................................................77
Review the SQL Deadlock Graph.............................................................................................................77
Locking and Blocking............................................................................................................79
Workshop: The Locks Log..............................................................................................................................79
Set up the sp_lock3.sql File.....................................................................................................................79
Configure the sp_lock3.sql File...............................................................................................................79
Create a SQL Job....................................................................................................................................80
Start the Locks Log.................................................................................................................................81
Review the Locks Log..............................................................................................................................82
Blocked Process Report Event.........................................................................................................................83
Workshop: Activate the Event.................................................................................................................83
Activate the Event...........................................................................................................................83
Run the Trace..................................................................................................................................84
Resolve Locking...............................................................................................................................85
Performance Tuning Strategy..............................................................................................87
Conclusion.............................................................................................................................88

Epicor ERP | 10.2.600 5

Introduction Performance and Diagnostic Advanced - On Premise Course

Introduction

The Performance and Diagnostic Advanced - On Premise course focuses on the techniques and tools available
to improve the performance of the Epicor ERP application. This course is intended as a follow-up to the Performance
and Diagnostics Basics - On Premise course.
These techniques and tools are typically used by partners, Epicor consultants, and technical support specialists.
The first section of this course explores the standard metrics for testing the performance of the Epicor ERP
application. It details these performance metrics and the tests you follow to evaluate each metric. This section
then describes how you create the Infrastructure Diagram and the Instance Diagram, two key illustrations that
can help Epicor professionals and customers discover sources of poor performance. This section then concludes
with a description of how to create a report on the performance evaluation results.
The course then explores more technical tools and techniques for evaluating performance. The Distribute the
Load section describes how you create multiple application servers for the same database; each application server
can then run specific tasks. The SQL Server Tuning section describes some ways you can tune the performance
® ®
of Microsoft SQL Server . The course also explains how you set up and use Microsoft tools like the Event Viewer
and SQL Performance Dashboard, activate trace flags, locate deadlocks, and identify locking and blocking. By
identifying deadlocks, locking and blocking, and other system issues, you can achieve significant performance
gains.
Upon successful completion of this course, you will be able to:
• Learn about the standard performance metrics recommended by Epicor and how to test a system against
these metrics.
• How to use the Performance and Diagnostic Tool to test memory use.
• Examine how to distribute the load between multiple application servers.
• Explore some techniques for improving the performance of Microsoft SQL Server.
• Review the performance tools available from Microsoft.
• Activate trace flags that improve how SQL Server interacts with the Epicor ERP application.
• Customize trace logs to track the specific information you need.
• Run tests to locate and correct deadlocks.
• Run tests to identify and correct causes of locking and blocking.

6 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Before You Begin

Before You Begin

Read this topic for information you should know in order to successfully complete this course.

Audience

Specific audiences will benefit from this course.

• System Administrators
• IT/Technical Staff
• Department Managers
• Epicor Consultants
• Epicor Partners
• Technical Support Specialists

Prerequisites

To complete the workshops in this course, the necessary modules must be licensed and operating in your training
environment. For more information on the modules available, contact your Epicor Customer Account Manager.
It is also important you understand the prerequisite knowledge contained in other valuable courses.
• Course - System Administration - On Premise
• Course - Performance and Diagnostic Basics - On Premise

Environment Setup

The environment setup steps and potential workshop constraints must be reviewed in order to successfully
complete the workshops in this course.
Your Epicor training environment, in which the Epicor demonstration database is found, enables you to experience
Epicor functionality in action but does not affect data in your live, production environment.
The following steps must be taken to successfully complete the workshops in this course.

1. Verify the following or ask your system administrator to verify for you:
• Your Epicor training icon (or web address if you are using Epicor Web Access) points to your
Epicor training environment with the Epicor demonstration database installed. Do not complete
the course workshops in your live, production environment.
Note It is recommended that multiple Epicor demonstration databases are installed. Contact
Support or Systems Consulting for billable assistance.

Epicor ERP | 10.2.600 7

Before You Begin Performance and Diagnostic Advanced - On Premise Course

• The Epicor demonstration database is at the same version as the Epicor application. The
demonstration database is installed from the Epicor Administration Console using the "Add Demo
Database" command under Database Server. See Epicor ERP installation guides for details. If you are an
Epicor Cloud ERP customer (and have licensed embedded education), the demonstration database is
installed for you.
• Your system administrator restored (refreshed) the Epicor demonstration database prior to
starting this course. The Epicor demonstration database comes standard with parts, customers, sales
orders, and so on, already defined. If the Epicor demonstration database is shared with multiple users
(that is, the database is located on a server and users access the same data, much like your live, production
environment) and is not periodically refreshed, unexpected results can occur. For example, if a course
workshop requires you to ship a sales order that came standard in the Epicor demonstration database,
but a different user already completed this workshop and the Epicor demonstration database was not
restored (refreshed), then you will not be able to ship the sales order. If you are an Epicor Cloud ERP
customer see section below.

2. Log in to the training environment using the credentials epicor/epicor.

If you are asked to log into your training environment with another User ID, for example, 'nancy' or 'bhoward',
on the initial logon, a message appears to inform you the password has expired. Perform the following steps
to set a new password:

1. To the Password Expired message, click Yes.

2. In the Current password field, enter the User ID of the user you are asked to log in as, for example,
'nancy'.

3. In the New password field, enter a new password, for example 'Train123'.
Important In Epicor ERP Cloud environment, the password must not contain user ID, must be
longer than 7 characters and include at least one uppercase letter.

4. In the Confirm new password field, re-enter the same password.

5. Click OK. The Change Password window closes and you are logged on with the new user ID.
Note Record the new password. This is important as this will be the password everyone uses
when they log on with this User ID, until the database is refreshed.

3. From the Main menu, select the company Epicor Education (EPIC06).

4. From the Main menu, select the Main site.

Epicor Cloud ERP Specific Information

Note If you are an Epicor Cloud ERP customer, then note the following about your Epicor-hosted education
company. All logins referenced in the course (such as manager, or epicor) should be changed to be the
<site ID>-. For example, if your site ID is 98315, then wherever you are instructed to use the login epicor,
instead use 98315-epicor. The password is 'Train18!'.

Note To refresh your Epicor training data, enter a support ticket in EpicCare and include your site ID.

8 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Advanced Performance Tuning Resources

Advanced Performance Tuning Resources

To further evaluate performance issues, leverage these key resources.

Hardware Sizing Guide

The Hardware Sizing Guide provides a practical approach to estimating the capacity you need for both the
Epicor ERP application and the database server.
The Hardware Sizing Guide matches the number of users against the database activity that occurs each day. It
next matches these two usage variables against a list of potential hardware configurations. You can then review
the various hardware and network items you could implement to improve application performance.
This guide also estimates the future load that may be required later as well. So even if the current network and
hardware configuration is adequate, you can still use this document to determine whether it makes sense to
upgrade the system now to prevent performance issues before they occur.
This document is intended as a tool to help identify potential system upgrades. Before making any significant
changes, Epicor representatives, network consultants, and customers should work together to determine the
best outcome. Epicor representatives and network consultants may also have further recommendations specific
to a customer organization that cannot be documented in this guide.
The Hardware Sizing Guide is available to download from EPICWeb. This guide is available under the Utilities
node on this site:
• https://epicweb.epicor.com/products/epicor-erp-10/downloads

VMWare Best Practices Guides

Virtual environments provide flexibility and can scale up or down more easily than physical environments. Through
this design approach, you can clone machines and add them to the load balancing pool to accommodate changes
in usage patterns.
If you are implementing VMWare in your environment, be sure to review the performance tuning documents
released by VMWare. These .pdf guides contain the information you need to improve the performance of your
virtual environment.
• http://www.vmware.com/files/pdf/solutions/SQL_Server_on_VMware-Best_Practices_Guide.pdf
• http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.5.pdf

Epicor ERP | 10.2.600 9

Standard Metrics and Evaluation Performance and Diagnostic Advanced - On Premise Course

Standard Metrics and Evaluation

The first step for improving performance is to identify areas of the system which do not run efficiently. Once
these areas are identified, an Epicor representative can recommend specific modifications that target these
inefficient areas.
Poor performance can typically slow down the entire Epicor ERP application, but possibly only a few sources are
the true causes for the issue. Epicor has created a standard evaluation process customers, partners, and Epicor
representatives must follow to accurately determine the sources of poor performance. Before Epicor can make
any performance recommendations, these first level tests must be run. Only after you have gathered this crucial
data should you contact an Epicor representative (technical support or consultant). The Epicor representatives
can then discuss potential solutions as well as conduct further performance tests.
To begin, review the Hardware Sizing and Configuration Guide to determine the recommended configuration
for your organization. You next evaluate your system against a series of standard metrics. These metrics spotlight
primary areas that may be the cause of poor performance. A key performance value from your system is compared
against a standard metric recommended by Epicor. For example, one of these metrics is CPU Speed. If a company's
CPU speed falls below the Epicor standard metric, the Epicor representative can determine what hardware may
be needed to optimize this area of the system.
When these standard metrics are determined, other areas of the system can be tested and evaluated. The
GetRowsKeepIdleTime test determines how the application performs when idle. This test can identify network
activity that may be interfering with Epicor calls between the client and server. Then the customer's hardware
configuration and instance makeup must be reviewed through diagrams. These diagrams give both the Epicor
representative and the customer a snapshot view of the network; these diagrams can also potentially locate
hardware issues. Next the system is evaluated through a series of baseline performance tests. Each test evaluates
a typical Epicor process and measures the results against an Epicor metric.
To complete the evaluation process, the Epicor representatives and the customer work together to complete the
Performance report. The Performance report encapsulates the areas responsible for poor performance and then
describes specific recommendations for improvement.

Evaluation Process Step by Step

The evaluation process:

1. Review the Hardware Sizing and Configuration Guide

2. Gather Standard Metrics

a. Storage Disk I/O Evaluation

b. Client Test

c. Server Test

d. Run the GetRowsKeepIdleTime Test

e. Network Test

f. Configuration Test

g. Customer Retrieval Baseline Test

h. Sales Order Baseline Test

10 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Standard Metrics and Evaluation

i. Purchase Order Baseline Test

3. Create the Infrastructure Diagram

4. Create the Instance Diagram

5. Complete the Performance Report

Review the Hardware Recommendation

Use the Hardware Sizing and Configuration Guide to determine the recommended hardware for the overall
system.

Determine Hardware Sizing

The Hardware Sizing and Configuration Guide is organized through different categories to determine a system's
optimal hardware configuration.
The guide is primary organized by load, which is the number of users who will use the Epicor ERP application.
The guide explores these main load categories:
• Up to 15 Users
• Up to 50 Users
• Up to 200 Users
• Up to 2,000 Users
• Up to 4,000 Users
The guide also has recommendations for SQL Server and virtual environments:
• Application Server/Hypervisor/Reporting Server
• SQL Server
• Virtual Appserver on VMWare
• Virtual Appserver on Hyper-V
• Virtual SQL Server on VMWare
• Virtual SQL Server on Hyper-V
This guide concludes with additional hardware recommendations, network bandwidth recommendations, and
other system information.

Case Study: Hardware Sizing

The following case study illustrates how to determine hardware sizing in a customer environment.
This organization has the following specifications:
• User Count -- 35 users (Total Office and MES users).
• Application Load Profile -- This manufacturing customer wants to run Material Requirements Planning
(MRP).
• Number of Epicor 10 Application Servers and Database Server Choice -- Epicor ERP version 10 using
Microsoft SQL Server 2012 database on a physical server configuration.

Epicor ERP | 10.2.600 11

Standard Metrics and Evaluation Performance and Diagnostic Advanced - On Premise Course

1. Review the recommendations for the ERP 10 - Hardware sizing guide (up to 50 user load) section (Table
3).

2. Notice this table contains two options for storage. You run the Diskspd test and determine you can use
Storage Option #2. This Diskspd test is described at the back of the Hardware Sizing Guide.

3. You next review the additional resources required to handle this customer's load. Check the ERP 10
Application usage Load Expectation (up to 50 user load) section (Table 4).

4. This customer runs MRP, so you review the MRP (Manufacturing customers) row. It recommends this
customer run MRP using the Net change option during less busy times during the work day.

5. Turn to the ERP 10-SQL Server section (Table 8). For storage, you decide to use the recommended option,
which is Option #1 - Fusion IO.

You have determined the hardware Epicor recommends for your system.

Standard Metrics

Epicor has developed a Standard Metrics table that identifies the benchmarks for optimal performance. Compare
these benchmarks against your system to locate areas of poor performance.
These performance benchmarks can identify key areas of improvement. Once these performance values are
determined, customers and Epicor representatives can investigate potential ways these performance values can
be tuned to achieve better results. This section first contains the Standard Metrics table and then a second example
table that illustrates a specific company's metrics compared against these standard metrics.

Epicor Standard Performance Metrics

Performance Metrics Epicor Component Impacted Standard Metrics (Expected

Result)
Storage Disk I/O Affects the SQL Server database Data drive. Throughput of 300 MB/Sec, with
less than 5ms or better latency
(Diskspd Test Utility) Performance of VMware machines as they are
stored on the shared NAS.

CPU Speed Tests the general speed of the application CPU speed <= 400 milliseconds
server.
(Config Check)

GetRowsKeepIdleTime An overall system performance test. How fast 15 milliseconds average, low
does this common, static API method run over performance variability throughout
(GetRowsKeepIdleTime
this infrastructure? This value can be compared the day; should always be less than
Chart)
across many machines. 30 milliseconds
Network Test Impact of the network on the client Server (Blue) < 0.5 seconds,
experience. Network (Green) < 0.4 seconds
(Network Diagnostics)

Configuration Check Reviews the performance of the Epicor ERP The Epicor application server passes
application server. the configuration tests
(Config Check)

Customer Retrieval Test This test measures database retrieval time by Observed time to move through
selecting and paging through customers. customers < 1 Second
(Baseline Test)

12 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Standard Metrics and Evaluation

Performance Metrics Epicor Component Impacted Standard Metrics (Expected

Result)
Sales Order Test This tests the performance of Sales Order Add 20 lines Sales Order < 36
Entry, a commonly used, important business seconds
(Baseline Test)
function. This test does not check
customizations or BPMs.
Purchase Order Test This tests the performance of Purchase Order Add 20 lines Purchase Order < 26
Entry, a commonly used, important business seconds
(Baseline Test)
function. This test does not check
customizations or BPMs.

The following sections describe how you test for these standard metrics.

Example Company's Metrics

Standard Test Expected Result Example Company

Storage Disk I/O Throughput > 300 Mb/Sec. 26 Mb/Sec
Storage Disk I/O Latency < 5 Milliseconds 37 Milliseconds*
CPU Speed <= 400 Milliseconds 720 Milliseconds*
(In CompanyA’s instance, VMs currently running
on Dell R610, Intel X5570 2.93 GHz were
tested.)

GetRowsKeepIdleTime < 30 Milliseconds 80 Milliseconds*

Network Test - Server < 500 Milliseconds N/A
Network Test - Client < 400 Milliseconds N/A
Configuration Check Pass Pass
Customer Retrieval Baseline Test <1 Second 1.2 Seconds*
Sales Order Baseline Test < 36 Seconds 75 Seconds*
Purchase Order Baseline Test < 26 Seconds 49 Seconds*

* The bold values indicate a performance metric value that exceeds the value of an Epicor standard metric.

Workshop – SAN (I/O) Test Utility

You can evaluate the efficiency of a Storage Area Network (SAN) by running a Microsoft subsystem benchmark
utility called Diskspd.
You download the Diskspd utility from Microsoft. Then you run a series of three tests through this utility.
The following tests are designed to test various aspects of an I/O disk subsystem i.e. bandwidth (Mega Bytes/second
i.e. MB/ Sec), Latency (milliseconds), performance of your I/O system with desired block size (64KB) and file size
and type of I/O – read or write and sequential v/s random writes. The parameters described in previous statement
have a great impact on IOPS and hence they are specified exactly as needed here for testing using Diskspd. On
the same machine you will get different IOPS number if you change any one parameter. So testing with Epicor
recommended parameters is highly recommended.

Epicor ERP | 10.2.600 13

Standard Metrics and Evaluation Performance and Diagnostic Advanced - On Premise Course

Run the Diskspd Utility

The following steps describe how you install, set up, and test I/O disk subsystem performance.

1. Download Diskspd utility from the link: https://gallery.technet.microsoft.com/DiskSpd-a-robust-storage-6c

d2f223
Diskspd utility is downloaded as a zipped file and can be extracted to your local drive as C:\Diskspd-v2.0.15\.

2. Diskspd doesn’t require installation; Diskspd.exe can be run via command prompt (with elevated permissions).
Based on the version of SQL Server installed, choose the correct path:
• ..\Diskspd-v2.0.15\x86fre (For 32Bit).
• ..\Diskspd-v2.0.15\amd64fre (For 64Bit).

3. Before you run each test:

a. Open up a Windows Command Prompt on the server where you have downloaded and unzipped the
Diskspd utility. For example Start > Run > cmd, this opens up the command window.

b. Change the directory to the location where Diskspd utility is extracted. For example, C:\Diskspd-v2.0.15\.

4. Run the three tests below using specified parameters:

a. 1: Database (.mdf) Drive Letter Test

This test evaluates 64K random writes.
In the command prompt, enter the following command:
diskspd -w100 -t8 -o8 -d900 -r -b64k -h -L -c80G c:\iotest.dat > c:\mdfio
testresult.txt
Note All the supplied parameters are case sensitive.

Parameter Description
-w100 100% Writes, No Read
-t8 8 Worker threads used against test file
-o8 8 outstanding IO requests
-d900 The test will last for 900 seconds
-r Random write test
-b64k 64kb block size per IO
-h Disabling software caching, only hardware caching
-L Capture Latency Information
-c80G Creates a workload file of 80GB
C:\iotest.dat This is the workload file of 80GB that will be
created, the drive should be same as the one which
has .mdf file
C:\mdfiotestresult.txt The results of Diskspd would be printed on
C:\mdfiotestresult.txt

14 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Standard Metrics and Evaluation

b. 2: Transaction Log (.ldf) Drive Letter Test

This test evaluates 64K sequential writes.
In the command prompt, enter the following command:
diskspd -w100 –t2 -o8 -d900 -b64k -h -L -c80G c:\iotest.dat > c:\ldfiotes
tresult.txt
Note All the supplied parameters are case sensitive.

Parameter Description
-w100 100% Writes, No Read
-t2 2 Worker threads used against test file
-o8 8 outstanding IO requests
-d900 The test will last for 900 seconds
-r parameter missing Sequential write test
-b64k 64kb block size per IO
-h Disabling software caching, only hardware caching
-L Capture Latency Information
-c80G Creates a workload file of 80GB
C:\iotest.dat This is the workload file of 80GB that will be
created, the drive should be same as the one which
has .ldf file
C:\ldfiotestresult.txt The results of Diskspd would be printed on
C:\ldfiotestresult.txt

c. 3: Tempdb Drive Letter Test

This test evaluates 64K random writes.
In the command prompt, enter the following command:
diskspd -w100 -t8 -o8 -d900 -r -b64k -h -L -c80G c:\iotest.dat > c:\tempd
biotestresult.txt
Note All the supplied parameters are case sensitive.

Parameter Description
-w100 100% Writes, No Read
-t8 8 Worker threads used against test file
-o8 8 outstanding IO requests
-d900 The test will last for 900 seconds
-r Random write test
-b64k 64kb block size per IO
-h Disabling software caching, only hardware caching
-L Capture Latency Information

Epicor ERP | 10.2.600 15

Standard Metrics and Evaluation Performance and Diagnostic Advanced - On Premise Course

Parameter Description
-c80G Creates a workload file of 80GB
C:\iotest.dat This is the workload file of 80GB that will be
created, the drive should be same as the one which
has .mdf file
C:\mdfiotestresult.txt The results of Diskspd would be printed on
C:\mdfiotestresult.txt

Use the results of each test to evaluate how your disk I/O subsystem works in comparison to similar subsystems.

Workshop - GetRowsKeepIdleTime Test

The GetRowsKeepIdleTime Test evaluates network traffic when users are not sending or requesting data from
the server. You run this test by tracking the GetRowsKeepIdleTime method.
The GetRowsKeepIdleTime method is used by the smart client. The client checks the application server to find
out if the report or process has finished its run. If the run is finished, the data is sent back to the client as output
for this method call. The client then uses this call to populate the results to the database or report. Because this
method is a system check regularly sent to the server that typically does not return data, you can also use this
method to measure network traffic that may impact performance.
The table displayed below illustrates how long it takes an optimized Epicor ERP application to complete a
GetRowsKeepIdleTime method run:

The following table illustrates a company experiencing performance issues. Note how much longer it takes this
Epicor instance to complete GetRowsKeepIdleTime methods during a day's activity.

16 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Standard Metrics and Evaluation

You can use this information to find out what other processes are running on the network during this time period.
Once you find out the reason for the increased traffic, you can reduce/eliminate the bottleneck by either changing
when the processes run or upgrading deficient hardware.

Review GetRowsKeepIdleTime Chart

Do the following evaluate performance for GetRowsKeepIdleTime method calls.

1. You first need to indicate you want to generate the GetRowsKeepIdleTime Chart. Within the Performance
and Diagnostic Tool, click Options > Settings.
The Settings window displays.

2. Click on the Server Diagnostics tab.

3. Select the Should Generate Chart check box.

4. Now indicate the Time Interval to Chart (minutes) option. This value defines the span of time you want
to evaluate at each point in the GetRowsKeepIdleTime Chart. For this workshop, keep the default one minute
value.

5. Click OK.
You return to the main window of the Performance and Diagnostic Tool.

6. From the Plug-Ins pane, select the Server Diagnostics link.

The main section of the window displays the server diagnostic items.

7. As described previously, click the Add Files... button to find and select all the server logs you generated.

8. Now clear (de-select) the Ignore GetRowsKeepIdleTime check box.

9. Click the Generate Results button.

10. Now click the GetRowsKeepIdleTime Chart tab.

Epicor ERP | 10.2.600 17

Standard Metrics and Evaluation Performance and Diagnostic Advanced - On Premise Course

The chart displays using the time interval you selected. The left side of the grid indicates how long it took
to run the GetRowsKeepIdleTime call, while the bottom of the grid indicates the time of the day when the
call occurred.

11. Optionally, you can click the Export to Excel button to save this graph as an .xslx spreadsheet.
You can then use Microsoft™ Excel™ to refine this graph as you need.

Workshop – Check Configuration

The Performance and Diagnostic Tool contains a utility to check the configuration of the application server. Use
this Config Check option to see what issues and potential issues you may have with the application server
configuration.
To do this, use the Config Check sheet to evaluate the configuration. This feature displays recommended actions
you can take to fix various issues.
The Config Check runs through a number of sources to determine its results. Current sources:
• web configuration files
• database settings
• application server settings
• application table values
As new versions and service packs are released, more checks will be added as needed to this diagnostic process.

Configure Performance and Diagnostic Tool (Network)

To gather the standard metric values from the network and the configuration, you will use the Performance and
Diagnostic Tool.

1. Within the Performance and Diagnostic Tool, click Options > Settings.
The Options window appears; the Config Check tab should display by default.

18 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Standard Metrics and Evaluation

2. If you access the application server through an http link, select the Connect using Http check box. You
then enter the link in the http:// field. For this workshop, do not select this check box.

3. By default you enter the net.tcp// connection.

You can get the correct URL that points to the application server from the .sysconfig file that runs this
instance of the Epicor ERP application. To find the .sysconfig file, right-click your Epicor ERP desktop icon;
from the context menu, select Open File Location. In your Epicor ERP application folder, the .sysconfig
files are located in the client\config folder.
Within the .sysconfig file, you can find this value in <appSettings><AppServerURL> configuration
setting.

4. Use the Connection Method drop-down list to indicate how this application server checks for authentication
certificates through Internet Information Services (IIS). When a user logs into the application, the selected
method checks whether the user can access the Epicor ERP application. Available options:
Tip You can also find the Connection Method in the .sysconfig file. Locate the <EndpointBindi
ng> value to see the method used the application server.

Available options:
• UsernameWindowsChannel -- Authenticate using an Epicor Username and Password. Only use this
method for application servers that run the Epicor Web Access (EWA) and mobile access client installations;
Windows checks for existing Epicor user accounts to authenticate logins.
• UsernameSSLChannel -- Use this option to authenticate using a Secure Sockets Layer (SSL) X509
certificate. Use this method for application servers that handle smart client installations when users reside
in different domains. By using an SSL certificate, users from these different domains can log into the
Epicor ERP application.
• Windows Authentication -- Select this option to authenticate using a Windows Username and Password.
You can select this method for application servers that handle smart client installations and Epicor Web
Access (EWA) installations where users access the application through the same domain. Any user with
a Windows Username and Password within this domain can successfully log into the Epicor application.
Note For information on how you set up each authentication method, review the Epicor 10
Architecture Guide.

5. Enter the User Id and Password for the Epicor user account used to access the application server.

6. Next enter the Client Directory for the folder that contains the client installation for the Epicor ERP
application. You can enter this path directly or click the Browse (...) button to find and select it. Enter the
path you used to locate the .sysconfig file.

7. Enter the Operation Timeout you wish to use; this value defines how long the Performance and Diagnostic
Tool waits until an incomplete operation is aborted by the application server. The default value is 300 seconds.

8. Click OK.

Run Config Check

When you link the Performance and Diagnostic Tool to the network, you can use it to verify the configuration is
set up correctly. This configuration check also informs you if other items, such as BPM methods, should be tested
for performance.

1. From the Plugins pane, select the Config Check link.

Epicor ERP | 10.2.600 19

Standard Metrics and Evaluation Performance and Diagnostic Advanced - On Premise Course

The main section of the window displays the check configuration items.

2. Now click the Check Configuration button.

3. The Config Check Summary sheet displays a list of rules run against which the application server was
checked.

4. The Result column displays the generated evaluation of this rule against your application server configuration.

5. Depending on the results, different instructional text displays in the Action Required column.
Available actions:
• EXISTS – Notifies you that various items, like customizations and BPM directives, are active in the current
system. These items should be evaluated for performance.
• FAIL – The application server configuration did not meet the rule requirements. The Action Required
column displays a recommended action you can follow.
• PASS – The configuration met or surpassed this rule requirements. No further action is needed.
• WARNING – Alerts you that potential performance issues may occur. Review these items to see if further
changes are needed.

6. The Config Check Details sheet displays the various rule keys run to evaluate each configuration rule.
Expand one of the rules to see the specific calls.

7. You can maximize the grids by clicking the Full screen button.
The interface now fills your screen, displaying more columns and information on the Config Check Summary
and Config Check Details grids. These grids keep any options you selected on the default view. To return
the Performance and Diagnostic Tool to the default view, click the Leave full screen mode option on the
title bar.

8. If you wish, you can save this log in .xls format by clicking the Export to Excel button.

Case Study: Configuration Check

The following case study illustrations the results from a configuration test.

RuleID RuleDescription Result ActionRequired

G000 Header Info EX Company Name [Example Company, Inc.]
G001 Check web PASS Web Configuration file [ E:\APPS\EPICOR10\server\config\web.config] passed.
configuration
file values
G002 CPU Speed Test Warning [ 4,683] milliseconds taken to do one million integer ADD and DIVIDE. Time <= 4000ms
PASS, Time >= 6000ms FAIL, else WARNING.
G003 List of User EXISTS Found customizations / verticalizations. Please review them for performance issue.
Interface
Customizations
G004 List of BPMs EXISTS Found BPMs - Method and/or Data Directive. Please review them for performance
issues.
G005 Power Settings PASS Power settings at High performance. Verify the output.
Check

20 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Standard Metrics and Evaluation

RuleID RuleDescription Result ActionRequired

G007 Application EX Company Name [Example Company, Inc.]
Setup
G008 AppServer FAIL Checked appserver
Settings [172.22.10.107::Epicor10::9401::6de0b61fee7bfd43:-2697cbaf:1367939e3fe:-77b4]
Please look at the details and take necessary action to fix the issues.
G009 System Info FAIL Checked machine of appserver [Epicor10] Please look at the details and take necessary
action to fix the issues.
SQL001 SQL 4GL Query PASS [ 1,882] milliseconds taken to finish 4GL SQL Query test. During the test the loop
Test executed [ 1,000] times. Each iteration of loop had 2 4GL SQL read queries. Time <=
3500ms PASS, Time >= 5500ms FAIL, else WARNING.
SQL002 Index PASS No further action required.
Non-Uniqueness
Check
SQL003 SQL Setup FAIL Check comment against each parameters in details tab.
Check
SQL004 Index PASS No further action required.
allow_row_locks
check
SQL005 Schema Holder FAIL If not fixed it can cause deadlock problems. You do not have the correct schema
Check holder.

Workshop – Network Performance

You next use the Performance and Diagnostic Tool to determine network performance. You can run multiple
tests to gauge the overall performance of the customer network, and compare these results against the network
standard metric.

Test the Network

In the Performance and Diagnostic Tool, you use the Network Diagnostics sheet to test network performance.
Important Before you can run this test, you need to configure the Performance and Diagnostic Tool so
it connects to your network. You establish this connection on the Settings > Options window. You did
this previously in this course when you set up the tool to run the Configuration Check.

1. Click on the Network Diagnostics option on the Plugins pane.

The Network Diagnostics interface displays.

2. Click the Run Test button three times.

During each network test, a data retrieval method runs ten times against the zDataTable in the Epicor ERP
database; the Performance and Diagnostic Tool evaluates the results of each call. When you finish running
these tests, three bar graphs appear on this sheet.

3. Click on each bar graph to view specific details of each network test.
The performance results for the selected test display in the fields above the graph. These results use the
Minutes: Seconds: Milliseconds format.
• Least - Displays the performance time for the shortest call. For example, .16 (16 milliseconds).

Epicor ERP | 10.2.600 21

Standard Metrics and Evaluation Performance and Diagnostic Advanced - On Premise Course

• Longest - Displays the performance time for the longest call. For example, 2.50 (2 seconds, 50
milliseconds).
• Average - Indicates the average performance time it took to finish each call. For example, .41 (41
milliseconds).
• Average Network Time - Indicates the average performance time it took each call to travel across the
network. For example, .14 (14 milliseconds).

4. You can also review the data performance for each call:
• Average MB Transferred - Displays how much data, in megabytes, was transferred on average during
each call. For example, 2.42 megabytes.
• Average MB/second Transferred - Displays how much data transferred on average during each second
of the method call. For example, 0.06 megabytes.

® ®
5. You can also view these test results in Microsoft Excel . To do this, click the Export to Excel button.

6. Save the .xls file to a directory you can access.

Now launch the .xls spreadsheet and review the network test results in Microsoft Excel.
Tip You can use the chart features in Excel to get a better view of the results. To do this, click the
Insert tab and select a bar chart option. Now right-click the Date area on the chart; from the context
menu, select Format Axis. The Format Axis window displays. Within the Axis Options pane, locate
the Axis Type section and select the Text axis radio button option. Each test now displays using
your selected bar chart format.

Expected Results
• Server Time (blue/lower bar) < 0.5 seconds
• Network Time (green/upper bar) < 0.4 seconds over a LAN
• Network Time (green/upper bar) < 7 seconds without compression over a WAN or < 1.5 seconds with
compression over a WAN.

Workshop – Customer Retrieval Baseline Test

You now run a series of baseline tests that measure processing performance against established Epicor ERP
application metrics. The Customer Retrieval baseline test measures database retrieval time by selecting and paging
through customers.

Run the Customer Retrieval Test

You will start the tracing log to capture the time required to select ten customers and page through them within
Customer Maintenance. After you complete this test, you write to the client trace log file and use the Performance
and Diagnostic Tool to analyze results.

1. Return to the Epicor ERP application; navigate to Customer Maintenance.

Menu Path: Financial Management > Accounts Receivable > Setup > Customer

2. Return to the Tracing Options Form and click the Clear Log button.

3. Now click the Enable Trace Logging check box and click the Apply button.

4. Return to Customer Maintenance; click the Customer... button to launch the search form.

22 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Standard Metrics and Evaluation

5. Now click the Search button to retrieve customer records.

6. Click the Select All button.

7. Click OK to pull in all the customer records into Customer Maintenance.

8. Using the Navigation toolbar, click the Right Arrow button nine times to display the next nine customers.

9. Exit Customer Maintenance.

10. Copy the Current Log File directory path.

11. Launch the Performance and Diagnostic Tool and from the Plugins pane, select the Client Trace Analysis
link.

12. Click the Clear Results button.

13. Click the Add Files... button.

The Select your UI Trace Files window displays.
Note You may need to close all instances of Epicor ERP to be able to add your log file.

14. Navigate to the directory location that stores the log file you just generated, such as
C:\ProgramData\Epicor\log.

15. Select the most recent log file. It should have today's date and a recent time stamp.

16. Click the Generate Diagnostics button to capture the performance results.

17. Click on the Results tab.

18. Group the results by Method Name.

19. Expand the GetByCustID method; you should have ten entries. Review the Execution Time for each method
call.

20. Click the Summary sheet.

21. Review the results for the GetByCustID method.

Expected Results:
Observed time to move between customers: < 1 second

Workshop – Sales Order Baseline Test

You run this standard metric test to measure database update performance. During this test, you create a sales
order, enter twenty detail lines in a spreadsheet, then copy and paste these lines from the spreadsheet into Sales
Order Entry.

Create the Sales Order Header

You first enter a sales order header and organize the Lines > List sheet to display columns in a specific order.

1. Within the Epicor ERP application, navigate to Sales Order Entry.

Epicor ERP | 10.2.600 23

Standard Metrics and Evaluation Performance and Diagnostic Advanced - On Premise Course

Menu Path: Sales Management > Order Management > General Operations > Order Entry
Tip The CRM menu path is: Customer Relationship Management > Order Management > General
Operations > Order Entry

2. To create a new sales order, click the New button.

3. For the Sold To ID value, enter DALTON.

4. Clear the Ready to Process check box (if it is selected).

5. Click Save to record the order.

6. Navigate to the Lines > List sheet. Reorganize the columns so they display in this order:

Line Open MfgJobType Part Rev Part Description Order Quantity

Create the Spreadsheet

You now create a spreadsheet and add the detail lines you need. You enter 20 detail lines.

1. Launch Microsoft Excel. Click Start > All Programs > Microsoft Office > Microsoft Office Excel
Tip You may see a dialog box asking you to verify the Microsoft product license. Just click OK to
close this window.

2. In your environment, you would now create a new spreadsheet that contains parts from your database.
However if you are in an Epicor University environment, you can load an existing spreadsheet. Click Open
and navigate to the C:\_PerformTune folder.

3. Open the BaseTest SalesOrder.xslx file.

4. In the spreadsheet, verify that twenty sales order detail lines display using the following values. Be sure these
Part IDs are the same:

Line Open Job Type Part Rev Part Quantity

Description
1 1 Manufactured AV-TA 12
2 1 Manufactured CA10 10
3 1 Manufactured CA12 12
4 1 Manufactured CA20 20
5 1 Manufactured DCD-300-OR 20
6 1 Manufactured DCD-400-KB 100
7 1 Manufactured DCD-600-CDL 600
8 1 Manufactured DSS-1030 10
9 1 Manufactured G1000 1,000
10 1 Manufactured MRP100-SK 100

24 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Standard Metrics and Evaluation

5. Save the spreadsheet to the desktop.

Start the Tracing Log

Now start the client tracing to capture the time it takes to enter the 20 detail lines.

1. Return to the Epicor ERP application.

2. Return to the Tracing Options Form.

3. Click the Clear Log button.

4. Now click the Enable Trace Logging check box.

5. Click the Apply button.

Copy and Paste the Detail Lines

Run the copy and paste test in Sales Order Entry.

1. Return to Excel and copy the twenty detail line from your spreadsheet into your clipboard. Do not select
the column headers.

2. Return to Sales Order Entry.

3. Right-click above the column headers in the Lines > List sheet; from the context menu, select Paste Insert.
Wait while the twenty order detail lines are loaded into the Sales Order Entry form.

4. Click Save.

5. Exit Sales Order Entry.

Generate the Results

Use the Performance and Diagnostic Tool to analyze the results.

1. Launch the Performance and Diagnostic Tool and from the Plugins pane, select the Client Trace Analysis
link.

2. Click the Clear Results button.

3. Click the Add Files... button.

The Select your UI Trace Files window displays.

4. Navigate to the directory location that stores the log file you just generated, such as
C:\ProgramData\Epicor\log.

5. Select the most recent log file. It should have today's date and a recent time stamp.

6. Click the Generate Diagnostics button to capture the performance results.

7. To determine the entire time for the process, navigate to the Results sheet.

Epicor ERP | 10.2.600 25

Standard Metrics and Evaluation Performance and Diagnostic Advanced - On Premise Course

8. Group by Method Name.

9. Expand the GetNewOrderDtl method node. You should have 10 method calls.

10. Navigate to the Summary sheet and review the TotalExecutionTime time for each method. Note each
method was called 10 times.

11. Click the Epsilon icon in the TotalExecutionTime column.

The Select Summaries window displays.

12. Select the Sum check box and click OK.

Expected Results
Total observed time for 10 lines: Less than 12 Seconds
This value is measured from the Results tab as difference between Start Time of the first method and the End
Time of the last method.

Workshop - Purchase Order Baseline Test

This test measures database update performance. During this test, you will create a purchase order that contains
twenty detail lines.
You will enter a purchase order header and then start the client tracing to capture the time it takes to enter the
20 detail lines. After you complete the test, you will write the client trace log file and use the Performance and
Diagnostic Tool to analyze the results.

Create the Purchase Order Header

You begin this performance test by creating a new purchase order.

1. Navigate to Purchase Order Entry:

Menu Path: Material Management > Purchase Management > General Operations > Purchase Order Entry

2. Click the New button.

3. For the Supplier ID value, enter ABCM.

4. Click Save.

5. Within Purchase Order Entry, navigate to the Lines > List > Inventory sheet.

6. Verify that the first seven column headers on this sheet match the column sequence in the following table.
If they do not, rearrange the columns to match this sequence.

Line Type Part Rev Description Our Quantity UOM

26 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Standard Metrics and Evaluation

Create the Spreadsheet

You now create a spreadsheet and add the detail lines you need. You enter 20 detail lines.

1. If it is not active, launch Microsoft Excel. Click Start > All Programs >Microsoft Office > Microsoft Office
Excel.
Tip You may see a dialog box asking you to verify the Microsoft product license. Just click OK to
close this window.

2. In your environment, you would now create a new spreadsheet that contains parts from your database.
However if you are in an Epicor University environment, you can load an existing spreadsheet. Click Open
and navigate to the C:\_PerformTune folder.

3. Open the BaseTest_PurchaseOrder.xslx file.

4. In the spreadsheet, verify that twenty purchase order detail lines display using the following values. Be sure
the Part IDs are the same.

Line Type Part Rev Description Our Quantity UOM

516X075B 10 EA
1032x075 20 EA
1032X100 20 EA
38X075B 10 EA
38X100B 10 EA
38X125B 10 EA
38X200 20 EA
516X75B 30 EA
516X100B 30 EA
516X150B 30 EA
516X200B 30 EA
8400S-042 30 EA
8400S-053 5 EA
8400S-060 5 EA
8400S-116 5 EA
8400S-610 A 5 EA
516FW 5 EA
516LN 30 EA
10w30Lube 10 EA
8mm-1.25HNUT 10 EA

5. Save the spreadsheet to the desktop.

Epicor ERP | 10.2.600 27

Standard Metrics and Evaluation Performance and Diagnostic Advanced - On Premise Course

Start the Tracing Log

Now start the client tracing to capture the time it takes to enter the 20 detail lines.

1. Return to the Epicor ERP application.

2. Return to the Tracing Options Form.

3. Click the Clear Log button.

4. Now click the Enable Trace Logging check box.

5. Click the Apply button.

Copy and Paste the Detail Lines

Run the copy and paste test in Purchase Order Entry.

1. Return to Excel and copy the twenty detail lines from your spreadsheet into your clipboard. Do not select
the column headers; only select the rows below the headers.

2. Return to Purchase Order Entry.

3. Right-click above the column header in the Lines > List sheet; from the context menu, select Paste Insert.
Wait while the twenty purchase order detail lines are added to the Purchase Order Entry form.

4. Click Save.

5. Exit Purchase Order Entry.

Generate the Results

Use the Performance and Diagnostic Tool to analyze the results.

1. Launch the Performance and Diagnostic Tool and from the Plug-Ins pane, select the Client Trace
Analysis link.

2. Click the Clear Results and the Clear Selected buttons.

3. Click the Add Files... button.

The Select your UI Trace Files window displays.

4. Navigate to the directory location that stores the log file you just generated, such as
C:\ProgramData\Epicor\log.

5. Select the most recent log file. It should have today's date and a recent time stamp.

6. Click the Generate Diagnostics button to capture the performance results.

7. To determine the entire time for the process, navigate to the Results sheet.

8. Group by Method Name.

28 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Standard Metrics and Evaluation

9. Expand the GetNewPODetail method node. You should have 20 method calls.

10. Navigate to the Summary sheet and review the TotalExecutionTime time for each method. Note each
method was called 20 times.

11. Click the Epsilon icon in the TotalExecutionTime column.

The Select Summaries window displays.

12. Select the Sum check box and click OK.

Expected Results
Total observed time for 20 lines: Less Than 51 Seconds
This value is measured from the Results tab as difference between Start Time of the first method and the End
Time of the last method.

Infrastructure Diagram

After you have gathered standard metrics and other performance information from the system, work with your
Epicor representative to create a diagram of your system's physical infrastructure.
This Infrastructure Diagram illustrates the computer hardware used to run the network. It displays the
specifications of each piece of computer hardware and it also illustrates the connections between the hardware
infrastructure. Be sure to include both the specifications for the physical boxes as well as the virtual machines
running over the network.
Minimum details to include on the physical infrastructure diagram:
• Server details like server, server name, and server processor.
• Other hardware specifications like RAM, disk, network, OS, VMware details, and so on.
• All servers participating in an Epicor instance including test system, pilot system, and other auxiliary systems.
• If the infrastructure is a VMware instance, document what virtual machines are running with which name on
which server.
• If the infrastructure is a VMWare instance, document virtual machine specifications that both are on or can
be powered on.
• Other technical details to document include Switches, SAN details, and F5 (if present).
• Infrastructure locations (be sure to mention if a large number of users connect over a WAN).
This diagram can be created using a number of different styles and graphics. You can create these diagrams
using Microsoft® Visio® or a similar 2D-object drawing application. The following illustration is provided as an
example; use a diagram style that works best for you.

Epicor ERP | 10.2.600 29

Standard Metrics and Evaluation Performance and Diagnostic Advanced - On Premise Course

Instance Diagram

You should also work with your Epicor representative to create a logical, or instance diagram that displays the
machines on which the Epicor ERP application is installed. This Instance Diagram needs to display the specifications
for each client and server machine.
Minimum details to include on the instance (logical) diagram:
• The physical machine and logical machine used for each Epicor component.
• The number of application server boxes and instances.
• Users and their locations; be sure to group these users together. For example, group together local users and
remote WAN users and indicate their locations.
• Other information like load balancing details, database details, database type, and machine names.
• Which component runs which part on local drives versus SAN drives.
This diagram can also be created using a number of different styles and graphics. You can create these diagrams
using Microsoft Visio or a similar 2D-object drawing application. The next illustration gives you an example;
develop a diagram style that works best for you.

30 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Standard Metrics and Evaluation

The Performance Report

Epicor representatives finish evaluating your system by creating a Performance report. This report summarizes
the information gathered through the standard metrics, additional performance tests, and the diagrams.
Although each performance report will be unique for each customer, the following list contains some suggested
sections the report should contain:
• Performance Metrics -- Create a table that compares the Epicor standard metrics against the customer's
performance metrics. This section provides a quick summary of the various performance issues and provides
some possible solutions to address these issues.
• Scalability Issues -- This section defines issues the customer organization may face to expand their use of
the Epicor ERP application. This section of the report encapsulates the information gathered through the
Infrastructure Diagram and the Instance Diagram.
• Specific Performance Issues -- Use this section to document specific performance issues. Any process, entry
program, report, and so on that has slow performance needs to be documented. This area of the report
contains a table that records how long it takes to run these processes under various conditions.
• Appendix -- Place the Infrastructure Diagram and the Instance Diagram in this section.

Epicor ERP | 10.2.600 31

Standard Metrics and Evaluation Performance and Diagnostic Advanced - On Premise Course

If you are in an Epicor University environment, a sample Performance report is available. Review this file to see
how to create this report. You locate this file by launching Windows Explorer and navigating to the
C:\_PerformTuning folder. Review this file:
• PerformanceReport_CompanyA.pdf

32 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Memory Tests

Memory Tests

The Epicor ERP application uses varying amounts of system memory each work day. One of the best ways to
evaluate application performance is through memory tests, as they can help you locate spikes in memory use.
You test memory use in a couple ways. You can use the Live Memory Inspection feature on the Performance and
Diagnostic Tool. Through this feature, you can select a process and generate both a Stack Trace and a Memory
Trace against it. The Performance and Diagnostic Tool contains a series of grids that help you analyze these trace
results. However if you are concerned about memory leaks, you should generate a memory dump file using the
Windows Task Manager. You can then evaluate the results contained in this file with your consultant or Epicor
Technical Support.

Live Memory Inspection

You run the Live Memory Inspection feature to both review how a .NET process runs during a specific date and
time and analyze memory crashes caused by a .NET process. This feature inspects .NET processes run by the
Epicor.exe, Epicor64.exe (client), and w3wp.exe (app server) executable programs.
Use these tests to determine why sometimes the Epicor ERP temporarily slows down. These tests are also useful
when you are experiencing unusually high CPU activity and high memory utilization. When application servers
require over 10 gigabytes (GB) to run, you should evaluate the reasons why your system needs so much memory.
To begin, you first select the process you want to review. You then decide if you will generate a stack trace, a
memory trace, or both traces:
• Stack Trace – Displays the method flow the .NET process ran, detailing a list that starts from the last method
to the first method called in each thread.
• Memory Trace – Records the objects stored in memory while the .NET process ran, displaying both a count
of how many objects of the same type ran as well as the size (in kilobytes) of each object.
If the stack trace and/or the memory trace does not return enough performance information or if a crash dump
you generated from the Microsoft operating system did not return enough results, you can also use this feature
to generate memory dump files. A memory dump file takes a snapshot of the memory required to run the
selected process. You can then review this file through the Performance and Diagnostic Tool or send this file to
Epicor Technical Support for further analysis.
Tip You can also create a memory dump file using the Windows Task Manager. The Memory Leaks section
of this course describes how you generate a memory dump file using the Windows Task Manager.

You typically run this memory inspection process in a test environment that mirrors your production environment.
Running this test temporarily halts the process you are inspecting, so this can disrupt the Epicor ERP application
in your production environment. Although you can run a basic stack trace without much disruption, most memory
tests temporarily freeze the application. Running these tests against a live environment is an extreme measure,
and you only should do this when you cannot determine the cause of performance issues in your test environment.
If Epicor Technical Support requires more information, they may also ask you run memory tests against your live
environment.

Epicor ERP | 10.2.600 33

Memory Tests Performance and Diagnostic Advanced - On Premise Course

Workshop: Live Memory Test

The next workshop explores how you run a live memory test using the Performance and Diagnostic Tool.

Setup

Before you can run a live memory inspection, you must define some options within the Performance and Diagnostic
Tool.

1. Click Options > Settings.

The Settings window displays.

2. Select the Live Memory Inspection tab.

3. The memory trace and stack trace use a color to categorize the calls made by the Epicor Framework (ICE)
and Epicor ERP application (ERP). You can then more easily see which system sent the call. If you like, you
can change these colors by selecting a different option on these drop-down lists:
• Erp Highlight Color – The default color is Dark Orange; use this drop-down list to select a different
color.
• Ice Highlight Color – The default color is Corn Flower Blue, use this drop-down list to select a different
color.

4. Indicate the Number of return rows (Memory Dump Compare) you want to display when comparing
memory dump results on the Post Mortem Memory Dump Compare sheet. By default, a maximum of
100 rows display in these additional results. Use this field to increase or decrease the number of rows that
return through this context menu option.

5. In the Pdb Path field, enter the path to a program database (PDB ) file that holds debugging and project
state information that allows incremental linking of a Debug configuration of your program.

34 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Memory Tests

6. Select the Using DebugDialog to capture a dump on First Chance Exception link to display a
documentation webpage from the Microsoft Developer Network (MSDN). This topic describes how to use
®
Microsoft's DebugDialog Tool to capture a memory dump for a First Change Exception (such as a crash
dump).
Tip You can also modify the Microsoft Registry Editor to generate a memory crash dump. To learn
how to set this up, review the Crash Diagnostics section in this guide.

7. When you finish setting up the Live Memory Inspection parameters, click OK.

You are now ready to run the memory test.

Run Live Memory Test

Do the following to run a live memory test. Use these controls to run a stack trace, a memory trace, or both trace
options.

1. On the Plugins list, select the Live Memory inspection option.

2. Select a Process ID for a process currently running on your server. This drop-down list displays all the current
w3wp processes with their application pool names as well as the names of any Epicor .exe processes. Typically
you select the application pool that runs the slow application server to gather the performance results you
need to review.
Tip You can also locate the Process ID, or PID, you need to inspect by analyzing the server trace log
or client trace log. Identity the process you wish to inspect and locate its PID value.

3. Now define what information you wish to see. Select the Memory Trace option to see details of the objects
stored in the memory of the process. This test generates a list of objects stored in memory while the .NET
process ran, displaying both a count of how many objects of the same type were activated as well as the
size (in kilobytes) of each object.
Only use this memory test option when you need, as the memory trace is more time consuming to run and
will cause the Epicor ERP application to freeze until it completes the test. Typically you should not run a
memory trace in your live environment.

4. Select the Stack Trace check box method to review the flow of the .NET process, detailing a list that starts
from the last method to the first method called in each thread. The default option, a stack trace runs quickly
(only 500-1000 milliseconds), so you can run this test in a live environment.

5. You can cause the stack trace to display more information about the objects and their values; this information
generates when their methods execute. Only select one of these options:

a. Just Epicor Code – The stack trace only gathers method calls made by the Epicor ERP application.

b. All Code – The stack trace gathers method calls made by both the Epicor ERP application and the
Windows operating system (Microsoft namespaces).

Important Because the objects in the stack are volatile and constantly changing, do not select these
options when you run a stack trace against a live environment.

Tip To gather this same information in a live environment, you should instead generate a memory
dump file by either clicking the Take Memory dump button or by using the Microsoft Registry
Editor. For information on using the Registry Editor feature, review the Crash Diagnostics section
later in this guide.

Epicor ERP | 10.2.600 35

Memory Tests Performance and Diagnostic Advanced - On Premise Course

6. To generate the memory trace and/or stack trace, click the PROCESS button.

7. You are warned this action will freeze the Epicor ERP application. If you run a memory trace, the application
freezes for about 30-50 seconds. If you only generate a stack trace without the Just Epicor Code or All
Code options, the application freezes for a shorter time. However when you run this stack trace test in a
live environment, users will notice their processes temporarily slow down. Click Yes to run the memory
inspection or No to exit the test.

You review the test results on the sheets available in the Performance and Diagnostic Tool:
• Stack Trace - Use this sheet to evaluate the results of the stack trace.
• Memory Trace - Likewise if you ran a memory trace, use this sheet to review how much memory the selected
process consumed during the inspection.
• MiscInfo - This sheet contains some additional information about the live memory test.
® ®
Tip Notice you can also display the memory trace and stack trace information in Microsoft Excel . This
program has more options for reviewing the memory inspection data.

Memory Dump

The Memory Dump feature is a separate task from the live memory test. When you create a memory dump file,
you record a snapshot of the memory and stack traces used by the selected process.
You can then review this memory dump file or pass it along to Epicor Technical Support for further analysis.
Specifically, Epicor Technical Support can help you determine the causes of memory leaks.
Be sure you have enough disk space to create these memory dump files. Each memory dump file will mirror the
size of the memory foot print (sometimes as much as 30 gigabytes (GB), so these files require a lot of disk drive
space.
The memory dump file you generate through this feature is different from the crash dump file you can generate
from the Windows operating system. You can set up Windows to automatically generate a crash dump file when
the operating system stops a process. The crash dump file is useful for determining what process caused the
exception that shut down the application. The memory dump file instead records what memory was required
while a specific process ran. It reduces how much you need to review the WinDebug tool (included with your
Windows oeperating system). Memory dump files contain less information and so are easier to review.
You should generate three dump files; generate one before the process starts, another file during the middle of
a process run, and then a third file at the end of the process run. You can then compare and contrast the memory
usage during different points during each part of the process cycle.
To generate a memory dump:

1. From the Process ID drop-down list, select the process you wish to review.

2. Click the Take Memory dump button. This generates a memory dump file before the process runs.

3. Now click the PROCESS button.

4. You are warned this action will freeze the Epicor ERP application. If you run a memory trace, the application
freezes for about 30-50 seconds. If you generate a stack trace without the Just Epicor Code or All Code
options, the application freezes for a shorter time. However when you run these tests in a live environment,
users will most likely notice their processes slow down. Click Yes to run the memory inspection or No to
exit the test.

5. Next while the process runs, click the Take Memory dump button.

36 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Memory Tests

6. Wait until the process ends. Now click the Take Memory dump button again.

You now have three memory dump files to review.

Get External DAC

If you will analyze a memory dump or a crash dump from a different computer, you need the mscordacwks (DAC)
file for the machine that generated the .dmp file. This item is the Discretionary Access Control file.
To access this file:

1. Log into the server machine that generated the .dmp file.

2. Launch the Performance and Diagnostic Tool.

3. Click Help > Download Diagnostic Resources.

The Download window displays.

4. Select the Get DAC for the current computer (x86 and x64) option.

5. Click the Download Selected Items button.

The Save As window appears, displaying the C:\ProgramData\Epicor\log path.

6. Click Save.
This downloads a .zip file that contains the x86 and x64 versions of the DAC.

7. Close the Downloads window.

8. Launch Windows Explorer and navigate to the C:\ProgramData\Epicor\log path.

9. Extract the .zip file into a folder.

Tip If Epicor Technical Support is helping you analyze memory dump files, you should also send this
DAC .zip file to support.

10. Now return to the Performance and Diagnostic Tool; navigate to the Process inspection tab.

11. For the Dump location, click the Browse (…) button to find and select directory path that contains the
external memory dump or crash dump file.

12. Notice the External DAC field (This field displays when you select the Post-Mortem analysis check box
on the Options window). Click the Browse (…) button next to this field and navigate to the folder where
you extracted the External DAC file.

You can use this DAC file to compare results between the memory dump you generated for the current system
against the memory dump generated on the external machine. If you want Epicor Technical Support to review
these files, you will also need to send this external DAC file with the memory dump files.

Memory Dump Compare

You next use the Memory Dump compare sheet to evaluate the results from two .dmp files:

1. Within the Performance and Diagnostic Tool, navigate to the Memory Dump compare > Diff sheet.

Epicor ERP | 10.2.600 37

Memory Tests Performance and Diagnostic Advanced - On Premise Course

2. Select the First Snapshot memory dump file you wish to use for the comparison. Click the Browse (…)
button next to this field to find and select this .dmp file.

3. Now select the Second Snapshot memory dump file you wish to use for this comparison. Click the Browse
(…) button next to this field to find and select this .dmp file.

4. If one of the snapshot .dmp files generated on an external machine, you add the External DAC file before
you run the analysis. Click the Browse (…) button to find and select the directory where you expanded this
.zip file; for information on downloading and unzipping this file, review the previous Get External DAC
topic.

5. Click the Analyze button.

The comparison results display on the Diff grid.

6. To see details on each type, right click the grid; from the context menu, select the show objects of this
type option.
If the object is simple, the trace displays its Type and Value; if the object is complex, the trace displays its
Property Name, Type, and Value.

7. To see the methods called during the memory dump, click the Roots tab.

8. Notice you can sort the methods in either Ascending or Descending order.

9.
Click on the Epsilon ( ) button to display the Summary window. You can summarize the method results
by Count, Minimum, and Maximum options. When you finish selecting the Summarize options, click OK.

10. Click the Filter button to hide rows you do not wish to display:
• (All) – The default option. This displays all the calls you captured through the stack trace.
• (Custom) – Displays the Custom Filter window. Use the controls on this window to set up a filter
statement.
• (Blanks) – Causes the stack trace results to only display rows that did not have a ManagedThreadID.
• (NonBlanks) – Causes the stack trace results to only display row that have a ManagedThreadID value.

11. Repeat these steps to load in other memory dump files for analysis.

Use the results to compare which methods consumed the most memory during the selected times recorded
within the two memory dump files.

Memory Leaks

The Epicor ERP application server is hosted within the w3wp.exe file. This file runs in Internet Information Services
(IIS).
You can launch the Windows Task Manager to see how much active memory the application servers are
consuming. During a typical work day, the Epicor ERP application processes a lot of data, so you may see memory
levels reach as high as these thresholds:
• Interactive Application Server – 20 Gigabytes
• Reporting Application Server – 30-35 Gigabytes

38 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Memory Tests

As long as w3wp.exe uses this much memory or less (Private Working Set memory), the Epicor ERP application
is consuming normal amounts of active memory. However if you are repeatedly seeing w3wp.exe take more
memory than these threshold levels over a period of several days, you may be experiencing a memory leak.
Increased memory consumption can occur on the application server when you have business activity queries
(BAQs) and trackers that return more than 500,000 records. Likewise if you run several reports and processes at
the same time or generate large reports, you can use up too much memory. These situations slow performance.
Epicor Technical Support can help determine the causes of memory leaks. To give them a snapshot of your
system’s memory consumption, create a memory dump file and then send it to Epicor Technical Support for
review.

Workshop: Memory Dump File

These exercises demonstrate how you create the memory dump file.

Create Memory Dump File

You use the Windows Task Manager to create the memory dump file.
Be aware this IS NOT a production friendly process. While you generate the memory dump file, the Epicor ERP
application will not run until this file is complete. You should only create a memory dump file as a last resort.
However if you are experiencing memory leaks, Epicor Technical Support will ask for this file to help locate the
issue.
This dump file can also be as large as the memory footprint it records, so it will take up a lot of space on your
C: drive. For example, a memory dump file that records 25 gigabytes of memory usage will use 25 gigabytes of
space. Be sure to move this file off your C: drive to avoid running out of hard drive space. To create the memory
dump file:

1. Launch Windows Task Manager.

2. Select the w3wp.exe process.

Epicor ERP | 10.2.600 39

Memory Tests Performance and Diagnostic Advanced - On Premise Course

3. Right-click this file; from the context menu, select Create Dump File.

4. The Windows Task Manager stops the Epicor ERP application and generates the dump file on your C: drive.
After the file is created, the Epicor ERP application will resume.

5. Using Windows Explorer, navigate to the memory dump file.

6. Save this memory dump file in a .zip file.

7. Send this .zip file to Epicor Technical Support. Be sure to detail how often you are experiencing the excessive
memory usage on your system.

Memory Dump Error

Because the Epicor ERP application is shut down while the memory dump file generates, Internet information
Services (IIS) may stop the memory dump. This occurs because the application pool does not respond to pings
during the dump, and so IIS recycles the application pool.
When IIS stops the memory dump process, you receive one of the following errors:
• The operation could not be completed. Only part of a ReadProcessMemory or WriteProcessMemory request
was completed.
• Could not create dump file *.dmp for process id*. GetLastError returns 0x8007012B.

40 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Memory Tests

Typically you receive this error when the memory dump file is very large. The process takes a long time to complete,
so eventually IIS checks, or pings, the application pool and notices the inactivity. To verify this situation is happening,
launch the Event Viewer. Navigate to the Windows Logs > System node to display the System Event Log. Look
for an error with the following values:

Source Event ID Message

WAS 5010 A process serving application pool... failed to respond to a ping.
The process id was '5423'.

To prevent this error, you can either temporarily shut off the Ping function or increase how long it takes IIS to
ping the application pool.

1. Launch the IIS Manager.

2. Use the tree view to display the Application Pools node.

3. Select the application pool.

4. From the Actions pane, select the Advanced Settings... option.

5. The Advanced Settings window displays. Do one of the following options:

a. To shut off the ping function, locate the Ping Enabled setting; from the drop-down list, select False.

b. To increase how long it takes IIS to ping the application pool, update these settings:

Setting Value
Ping Maximum Response Time (seconds) Increase to a value greater than 90.
Ping Period (seconds) Increase to a value greater than 30.

6. Run the memory dump as described previously.

7. After the memory dump file is generated, return to the IIS Manager and display the Advanced Settings
window.

8. Restore the Ping settings to their previous values.

Epicor ERP | 10.2.600 41

Distribute the Load Performance and Diagnostic Advanced - On Premise Course

Distribute the Load

You can improve how the Epicor ERP application performs by assigning different tasks to specific application
servers that can handle the load. This distributes, or scales out, the load more evenly across your system resources.
Note that typically you achieve better performance results by first improving your hardware. If you have the
resources to add more RAM, upgrade CPU processing, or increase disk capacity in your network or virtual
environment, you should do so. Also note that if you have less than two hundred users, the Epicor ERP application
can run efficiency through one server machine. However if you have more than 200 users, use VMWare (virtual
environment), or your organization requires special processing that requires significant system resources, consider
setting up multiple application servers to more evenly distribute the load. Not only does this improve performance,
but you also have spare capacity available to handle the load if a system component fails to run properly.
For example, you could assign a process that requires significant resources to run on a more powerful server, like
Material Requirements Planning, and then assign reports that generate less data to a server with fewer resources.
By distributing the load between these application servers, you reduce performance bottlenecks and match a
report/process with a machine best suited to run it. You could also assign users from different areas of the
company, such as the financials area and the shop floor area, to separate application servers and improve
performance. Moving heavy data processes like EDI or Epicor Service Connect to a separate application server
can improve efficiency as well.
You can scale out your system in many ways. Consider the data flow that occurs within your Epicor ERP application,
and develop a load distribution strategy that improves how this data flows between your users and the database.
The following section describes the components you can use to more efficiently distribute load across your system.
The accompanying workshop guides you through how to create a task agent rule to assign a specific task to an
application server.

Application Servers

An application server manages how a specific instance of the Epicor application runs. Through each application
server, you can configure licenses, companies, sessions, and users for a specific database.
To set up distributing the load, you first create application servers for each server machine available on your
system. You can set up multiple application servers to update the same database. For example, you create two
application servers for the same database, but these application servers are linked to different server machines
through their endpoint bindings. One application server is set up to run Epicor Web Access (EWA) on one server
machine, while another application server is set up to run a smart client through Net.TCP on a different server
machine. Likewise you could set up another application server that links to a machine which only handles SSRS
reporting tasks.
You add the application servers that will interact with your server machine through the Epicor Administration
Console. This management tool is located on your server installation. For information on how to add application
servers, review application help in the Epicor Administration Console.

42 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Distribute the Load

Task Agents

The task agent handles all scheduled tasks for an application server.
The task agent activates any program added though a recurring schedule. Users add programs to recurring
schedules through the Schedule drop-down lists available on programs throughout the Epicor application. You
create these schedules in the Epicor application using System Agent Maintenance, and you also use this program
to create task agent rules to distribute the load.
You can set up as many as three task agents to distribute tasks to the same Epicor database. This allows for
redundancy, as if one task agent fails another one can pick up the tasks and send them to the server. Each task
agent picks up any tasks that are currently available to process. If a task is processed by one agent, it is not
processed by another task agent.
You create task agents using the Task Agent Service Configuration program. You launch this program from
within the Epicor Administration Console. When you select the application server, the center pane on the Epicor
Administration Console displays the settings for the application server. Click the Task Agent Configuration
button to set up the task agent for the selected application server.
Note If the Task Agent stopped working and the Event Log Viewer has none or insufficient information
on an event, check the Application log in the Microsoft® Event Viewer® for an event where the source
is Task Agent Service.

Load Balancer

A load balancer is an optional hardware device that distributes data flow across multiple application servers. If
your organization needs to process large amounts of data through EDI, MRP, reports, and similar tasks that
require a lot of resources, consider incorporating load balancers into your system.
A load balancer can improve the performance of the Epicor application by distributing tasks between application
servers contained within an application pool. The load balancer takes the incoming tasks and sends them to an
application server with the available capacity to handle these tasks.
® ®
You can purchase load balancers from Kemp , F5 , and other manufacturers. Review the information from
these companies to learn how to best incorporate load balancers within your system.
However you can also use Application Request Routing (ARR) to load balance your system. This Internet
Information Services (IIS) extension causes a server farm to also run as a load balancer between application servers.
When ARR is installed, the server farm can now route incoming message calls to multiple application servers,
improving network performance. If you would like to use ARR to load balance your system, review the Epicor
Installation Guide.

Security Groups

Use security groups to organize your Epicor user base into separate functional areas. An optional feature, you
can use security groups to both control access to specific areas of the Epicor application and regulate data flow.
You create security groups through Security Group Maintenance. This program sets up the identifier and the
description for the security group. You then assign users to these security groups through User Account Security

Epicor ERP | 10.2.600 43

Distribute the Load Performance and Diagnostic Advanced - On Premise Course

Maintenance. Your users are now contained in specific security groups, and you can grant or prohibit access
to different areas of the Epicor application through the security programs.
You also use security groups to distribute the load. To do this, you set up task agent rules that link a security
group to a designated application server. Now when users within this security group enter data activity, this
designated application server handles the load.

System Agent

The System Agent regulates the processing between the Epicor application and the application servers. When
you first install the Epicor application or update your existing application, a system agent is automatically generated.
You manage the system agent through System Agent Maintenance. Use this program to modify settings on
the system agent, create rules to divide processing between different application servers, and define the default
schedules available to processes in ERP that can be run by automated schedules. Please note that there can only
be one system agent. You can delete it and create a new one, or just modify the existing one.
System agents are key for scaling out your system. Each system agent can have multiple task agent rules that
divide the system agent's processing between different application servers. You can set up task agents that
distribute the processing for specific tasks such as MRP or the Job Traveler report. You can also create task agent
rules that distribute processing for selected companies or security groups. By creating a series of task agent rules,
you can direct the data flow to the application server you designated to handle the specific load.

Workshop - New Application Server

During this workshop, you will create an additional application server for the same database. You will then set
up a task agent rule that causes the Production Detail report to generate through this alternate application server.

Add the Application Server

You first add the application server that will interact with your alternate server machine.

1. Launch the Epicor Administration Console:

a. Click the Windows + F buttons.

b. When the Charms bar displays, click the Apps button.

c. Select the Epicor Administration Console icon.

The Epicor Administration Console displays.

2. Use the tree view to expand the Server Management > localhost node.

3. Either from the Action menu or the Actions pane, select Add Application Server.
Tip You can also right-click the Epicor server to display this option on a context menu.

The Application Server - Create Site window displays.

4. You first enter the Application Name. This value is the name Internet Information Services (IIS) uses to
create the application, and this value is also added to the URL address which the client installation uses to

44 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Distribute the Load

connect to the application server. For example, if you enter Epicor10 in this field, the application server URL
will be net.tcp://<servername>/Epicor10.
When you change the value in this field, both the Web Site Directory and the Application Pool Name
fields update with this name change as well. This feature prevents a site that already exists from being
overwritten by the name change.

5. Enter the Deployment Directory that contains the Epicor server installation. For example:
\\EpicorServer\Epicor\Epicor10.1.100.
Important Be sure you are a member of the Administrators group on the server you enter in this
field.

6. Use the Deployment Version drop-down list to select your update version from the list of updates available
on your server. It is recommended that you select the latest update. If no updates are available, select the
Base option.
Note that when you click Deploy, the application server updates the Epicor ERP application to the selected
version. If prompted that all users will be disconnected while the system updates, verify all users have logged
out of the system and then click Yes to continue.

7. Now define the Web Site Directory on the server machine that will contain the application server. The
application server is installed in this location. For example: C:\Inetpub\wwwroot\Epicor10.

8. Define how this application server checks for authentication certificates through Internet Information Services
(IIS) by selecting an option from the Net TCP Binding Configuration drop-down list. When a user logs
into the application, the selected method checks whether the user can access the Epicor application.
Available options:
• UsernameWindowsChannel -- Authenticate using an Epicor Username and Password. Only use this
method for application servers that run the Epicor Web Access (EWA) and mobile access client installations;
Windows checks for existing Epicor user accounts to authenticate logins.
• UsernameSSLChannel -- Use this option to authenticate using a Secure Sockets Layer (SSL) X509
certificate. Use this method for application servers that handle smart client installations when users reside
in different domains. By using an SSL certificate, users from these different domains can log into the
Epicor ERP application.
• Windows Authentication -- Select this option to authenticate using a Windows Username and Password.
You can select this method for application servers that handle smart client installations and Epicor Web
Access (EWA) installations where users access the application through the same domain. Any user with
a Windows Username and Password within this domain can successfully log into the Epicor application.
Tip For information on how you set up each authentication method, review the Epicor 10
Architecture Guide.

9. If you have custom programs to incorporate with the Epicor ERP application, enter the Custom Directory
that contains these custom .dll files. You can enter this path directly or click the Browse (...) button to find
and select this path. After you click Deploy on this window, these custom .dll files are included in the Epicor
ERP application.
Tip As a best practice, you should always place custom programs in this separate Custom Directory.
Then the next time the application version is updated, these custom programs are not overwritten.
You can then modify these custom programs to work with the new version.

10. Select the Shared Assembly Location check box if you have a network load balanced environment.

Epicor ERP | 10.2.600 45

Distribute the Load Performance and Diagnostic Advanced - On Premise Course

For example, you may have the Epicor ERP application installed on multiple servers. You then must have a
central directory that contains all the server assemblies and Business Process Management (BPM) folders. If
your server environment is set up this way, activate this check box.

11. If you select the Shared Assembly Location check box, you also activate the Shared Directory field. Use
this field to define the directory which contains the server assemblies and BPM folders. You can enter this
path directory or click the Browse (...) button to find and select it.

12. By default the Application Pool Name uses the value you entered in the Application Name field. You
cannot change this value. This value defines the name of the application pool associated with the new
application server.
An application pool defines a group of related URLs that use the same process or set of processes. The new
application server must be placed in an application pool.

13. If you need to enter a specific user account for the Internet Information Services (IIS) application pool this
application server uses, select the Use Custom Account check box. Selecting this check box activates the
Application Pool Username and Application Pool Password fields. Enter the domain and the user
account in the Application Pool Username field; for example, enter MyDomain\UserName. To effectively
connect with the server, this account must be a valid domain account with access rights to the network. If
this account is not valid, you will not be able to stop and start the application pool.
If you do not select this check box, the application pool uses a default user account. If you use an SSRS
server, the connection uses the LocalSystem account. If you do not use an SSRS server, the connection
uses the ApplicationPoolIdentity account.

14. The Current Deployment section details the state of the application server. Review the information in
these fields:

a. Status Indicator - This icon indicates whether the application server is Not Installed (red) or Installed
(Green).

b. Installed On - Displays the date on which the application server was installed.

c. Server - Contains the name of the server on which the application server is installed.

d. Version - Displays the version number for the application server. Use this value to compare against the
current application server available from Epicor; if you have an older version, consider updating the
application server.

15. You next set up the Database Connection for this application server. Click this tab.

Select the Database

You use the settings on the Database Connection sheet to define how the current application server interacts
with the database.
To balance the load, be sure to select the same database used by the other application server(s).
You can modify the following settings:

1. Select the Server Name for the database SQL server instance that contains the SQL database you will use
with the new application server.

2. Now click the Authentication drop-down list and select either the Windows Authentication or SQL
Server Authentication option.

46 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Distribute the Load

a. If you select Windows Authentication, the User and Password default to your current login values and
these fields are disabled.

b. If you select SQL Server Authentication, enter the User and Password you use to log into SQL Server.

3. Now from the Database Name drop-down list, select the name of the SQL database you will link to this
application server. All the databases available under the selected SQL Server instance display on this drop-down
list.

4. To verify the application server can connect with this database, click Test Connection and click OK in the
confirmation message.

5. A dialog box displays indicating the test was successful. Click OK.

6. You next set up the Admin Console Settings for this application server. Click this tab.

Admin Console Settings

You use the Admin Console Settings sheet to modify the application server settings used by the Epicor
Administration Console.

1. For the Display Name, provide a display name that will identify the application server in the administration
console. Choose a name that helps you identify the purpose for the application server.

2. Enter the Epicor User Name. This value defines the user name for the Epicor user account that will access
the application server.

3. Now enter the Password for the user account used to access the application server. The password is stored
in an encrypted format.

4. Enter the Operation timeout value you want for the application server. This value determines the wait
time until an incomplete operation is aborted by the application server. The default value is 300 seconds.
Get the correct value from the <appSettings><OperationTimeOut> element of the .sysconfig
file that points to the application server. In your Epicor application installation, .sysconfig files are located
in the client\config folder.

5. Select or clear the Validate WCF Certificate checkbox to match the value set in the <appSettings><W
CFCertValidation> element of the .sysconfig file:
• Select checkbox if <WCFCertValidation value="True" />.
• Clear checkbox if <WCFCertValidation value="False" />.

6. For DNS Identity value, enter the expected DNS server name. There are two scenarios where you need to
enter a value in this field:
• UsernameSSLChannel Selected in Endpoint Binding - When authenticating using message-level or
transport-level Secure Sockets Layer (SSL) with X.509 certificates, WCF ensures that the certificate provided
during the SSL handshake contains a DNS or Common Name (CN) attribute equal to the value specified
in this field.
• Windows Selected in Endpoint Binding - When the service authenticates using message-level or
transport-level SSL with a Windows credential for authentication, and negotiates the credential, then
the negotiation passes the service principal name (SPN) so that the DNS name can be checked. The SPN
is in the form host/<dns name>.

Epicor ERP | 10.2.600 47

Distribute the Load Performance and Diagnostic Advanced - On Premise Course

Get the correct DNS server name from the <appSettings><DnsIdentity> element of the .sysconfig
file.

7. Now within the Epicor Application Launcher section, select one of the following options:

a. Do not allow access to user details - No method is used to launch the Epicor client. The default value,
the client is launched as normal.

b. Use Epicor Smart Client - If you select Use Epicor Smart Client, you need to click the Browse (...) button
to find and select the Epicor.exe file you will use to launch the Epicor client.

c. Use Epicor Web Access - If you use Epicor Web Access, select this option and click the drop-down list
to define the URL for the web access. This drop-down list contains the web access values defined in the
company configuration data for Epicor Web Access (set within the client).

8. You next set up the Reporting Services for this application server. Click this tab.

A status window displays progress through the application server setup steps. The new application server is added
to the tree view, as the application server appears under the selected Epicor server. The Epicor Administration
Console next connects to the application server. When the connection is complete, the center pane displays the
properties for the selected application server.

Reporting Services

Epicor reports generate through SQL Server Reporting Services (SSRS). You set up the SSRS server options through
the options on the Reporting Services tab.
You can modify the following settings:

1. Select the Configure SSRS check box to activate the SSRS fields. You can then define how this application
server interacts with SSRS.

2. Enter the SSRS Base URL for the SSRS Report Server. This value defines the Uniform Resource Locator (URL)
for the server, so enter the web site location that contains it. When you install SQL Server, you set up this
URL and so this value is typically http://<localhost>/ReportServer.
When you save, this URL location is validated. If the Application Server Setup program cannot find this
location, an error message displays.
Tip To find the value you need to enter in this field, go to the server machine and launch Reporting
Services Configuration Manager. From the tree view, click the Web Service URL icon. The value
you need displays in the Report Server Web Service URLs section. Copy this value into Notepad or a
similar text editor so you can later paste it into the Application Server window. For example:
http://HVW12AS09:80/ReportServer

3. Optionally, enter the SSRS Root Folder location. This directory defines the root folder location where you
will deploy the reports. For example, enter Epicor if you want the reports to deploy to the Epicor/Reports
folder. If you leave the field blank, this root folder will be the directory that contains the report server home
page file; the reports will deploy to the /Reports sub-folder in this directory.

4. Use the Server Name field to enter the name of the SQL Server instance for the Report Server. The Report
Server contains the SQL database you will use to generate SSRS reports.

5. Now click the Authentication drop-down list and select either the Windows Authentication or SQL
Server Authentication option.

48 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Distribute the Load

a. If you select Windows Authentication, the User and Password default to your current login values.
Tip To effectively connect with the server, this Windows user account must have access rights to
the SQL Server. If this user account does not have access rights to the SQL Server, the report data
cannot be generated and read from the report database.

b. If you select SQL Server Authentication, enter the User and Password you use to log into SQL Server.

6. Use the Database Name to either enter or select the SQL Server database that will contain the temporary
data used by SSRS to generate the report output. If the Create DB check box is selected, enter the database
name in this field. If the Create DB check box is clear (not selected), click the Down Arrow next to this
drop-down list; select the database you need from the list of options.
This database can be:
• The same database used by the Epicor application -- Although this set up is not recommended,
your report server database can be the same as your main database.
• A separate database on the SQL Server -- This set up method is most common, as the report data
then populates this separate database on the server.
• A database on a different SQL Server -- The report data from the Epicor application is sent to another
server dedicated to SSRS report processing. If you are a larger organization, you may set up your system
in this way to improve performance.
Important Do not select the system databases for the SSRS database, as these databases cannot
store temporary report data. The system databases include:
• ReportServer
• ReportServerTempDB
• model
• msdb
• master
• tempdb

7. If you are setting up SSRS for the first time, select the Create DB check box. When you select this option
and click OK, a new report database is generated using the name you entered in the Server Name field.
If you update the SSRS settings later on, you can select this database again or if needed, create a new
database.
When you save, this database is validated. If the Application Server Setup program cannot find this database
in the location you specified, an error message displays.
Note that when you register an existing application server, you cannot create a new report database. If you
wish to create a new reports database for an existing application server, you must reconfigure it. To do this,
right-click the application server and from the context menu, select Application Server Configuration.
You can then enter a new Server Name and select the Create DB check box; after you click OK the new
report database is generated.

8. When you finish defining your SSRS options, click the Test Connection button.
A message should display indicating that this application server is connected to SSRS. If you receive an error,
check your values to make sure they are accurate and then test the connection again.

9. Verify the Import Reports check box is selected. This indicates you are ready to deploy your reports. If you
are updating the application server and you do not need to deploy the SSRS reports, be sure to clear the
Import Reports check box.

Epicor ERP | 10.2.600 49

Distribute the Load Performance and Diagnostic Advanced - On Premise Course

Important While you can clear the Import Reports check box, do not clear the Configure SSRS check
box. If you clear this check box, it indicates you no longer use SSRS reporting, causing the application
server to reconfigure without the SSRS functions.

10. For the SSRS Location, select the directory that contains the SSRS ReportServer installation. Depending
on the SQL Server version you use, this location is similar to the following example directories. Your specific
directory path will be the name your system administrator assigned to the SQL Server instance during
installation.
• SQL Server 2014 -- C:\Program Files\Microsoft SQL Server\MSRS12.MSSQLSERVER\Reporting
Services\ReportServer
• SQL Server 2012 -- C:\Program Files\Microsoft SQL Server\MSRS11.MSSQLSERVER\Reporting
Services\ReportServer
• SQL Server 2008 R2 -- C:\Program Files\Microsoft SQL
Server\MSRS10_50.MSSQLSERVER\ReportingServices\ReportServer
• If the SSRS server is on a separate machine, enter the UNC path to the ReportServer directory. The current
user account must have permissions to write to this remote directory. Typically this directory is:
\\<RemotePCName>\C$\Program Files\Microsoft SQL Server\MSRS10_50.MSSQLSERVER\Reporting
Services\ReportServer
Important If you have multiple SQL Server versions installed, make sure you select the location that
matches the version used by the Epicor application.

11. When you finish entering the application server settings on the Application Server Settings, Database
Connection, Admin Console Settings, and Reporting Services sheets, click Deploy.
If you receive any errors, correct the set up information you entered and click Deploy again. When all errors
are resolved, the program will finish the setup process.
This program now takes the values you entered to configure the application server. Several values are written
to the web.config file, and these values mainly define the connection string (<connectionString>) setting
that links the application server to SQL Server. For example to ensure better performance, the SQL connection
pool is set to a minimum pool size (Min Pool Size) of 100 threads and a maximum pool size (Max Pool Size)
of 2,000 threads.
Tip If you need to review or update the web.config settings later, this file is located in the
...\epicor\<VersionNumber>\Server folder.

A status window displays progress through the application server setup steps. The new application server is added
to the tree view, as the application server appears under the selected Epicor server. The Epicor Administration
Console next connects to the application server. When the connection is complete, the center pane displays the
properties for the selected application server. The application server is installed and active on your server.

Create the Task Agent

Task agents handle all scheduled tasks for the application server.
The task agent activates any program added to a recurring schedule. Users add programs to recurring schedules
through the Schedule drop-down lists available on programs throughout the Epicor ERP application. You create
these schedules in the Epicor ERP application using System Agent Maintenance, and you also use this program
to create task agent rules to distribute the load. You will explore how to create task agent rules later in this
workshop.

50 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Distribute the Load

Tip To learn how to assign tasks to automatic schedules, review the System Agent Maintenance topics in
the application help or review the Automatic Data Processing chapter in the Epicor Implementation User
Guide.

You create a new task agent from the Task Agent Service Configuration program.

1. Within the Epicor Administration Console, click on an application server. The administration console connects
to the selected application server. When the connection is complete, the center pane displays the properties
for the selected application server.

2. Now click the Task Agent Configuration button.

The Task Agent Configuration window displays.

3. Click on File > New Task Agent.

The Add Task Agent window displays.

4. Enter the unique Name for this task agent. Enter a value that helps you easily identify the task agent later.

5. Verify the Enabled check box is selected.

6. Indicate the AppServer URL that connects the task agent to the application server (AppServer).
Tip This URL value displays in the center pane within the Epicor Administration Console.

7. Now select the Endpoint Binding type this task agent will use. This value defines how this application
server checks for authentication certificates through Internet Information Services (IIS). When a user logs
into the application, the selected method checks whether the user can access the Epicor application.
Available options:
• UsernameWindowsChannel -- Authenticate using an Epicor Username and Password. Only use this
method for application servers that run the Epicor Web Access (EWA) and mobile access client installations;
Windows checks for existing Epicor user accounts to authenticate logins.
• UsernameSSLChannel -- Use this option to authenticate using a Secure Sockets Layer (SSL) X509
certificate. Use this method for application servers that handle smart client installations when users reside
in different domains. By using an SSL certificate, users from these different domains can log into the
Epicor ERP application.
• Windows Authentication -- Select this option to authenticate using a Windows Username and Password.
You can select this method for application servers that handle smart client installations and Epicor Web
Access (EWA) installations where users access the application through the same domain. Any user with
a Windows Username and Password within this domain can successfully log into the Epicor application.

8. If you use UsernameWindowsChannel or UsernameSSLChannel endpoint binding, enter the User ID

and User Password for the account. Enter a valid ICE or Epicor user account identifier in these fields.
Important You should create a special ICE Manager account that has access to all companies and
has session impersonation rights. You create this account in User Account Security Maintenance.
You activate impersonation rights on the Options sheet; select the Allow System Impersonation
check box. You add companies to the user account through this program as well. For more information
on User Account Security Maintenance, review the Epicor ERP application help.
Typically this account should also be selected on the system agent in System Agent Maintenance.
If you use a different user account on the system agent, make sure it can access all companies and
sites, as this account runs processes like MRP and scheduling.

Epicor ERP | 10.2.600 51

Distribute the Load Performance and Diagnostic Advanced - On Premise Course

9. Use the Operation Timeout field to define how long, in seconds, it takes a server call to generate an error
and fail.

10. If an error occurs, the task agent will try to send the call back to the server. The Max Connection Attempts
value defines how many times the task agent will attempt to send the call again. Increase or decrease this
value as you need.

11. The Max Concurrent Tasks field defines how many calls the task agent can send to the application server
at the same time. Increase or decrease this value to reflect the capacity of your application server.

12. If you use the UsernameSslChannel option for the Endpoint Binding type, the Validate WCF Certificate
check box is active. This check box indicates whether the task agent service reviews the Secure Sockets Layer
(SSL) Certificate to make sure this certificate is valid. When you use this option, the task agent service must
be set up to run as a domain user, and this domain user account must be able to log into the Epicor ERP
application.
Important If you use a self-signed certificate, do not select this check box.

13. The DNS Endpoint Identify field specifies the expected Domain Name System (DNS) identity of the server.
If you are setting up a task agent that uses a Secure Sockets Layer (SSL) Certificate for endpoint binding,
enter the identity in this field.

14. When you finish defining the task agent properties, click Save.

The new task agent is created. By default, the task agent is enabled; a green icon indicates it is active.
You now have finished setting up the application server and task agent. You next launch the Epicor ERP application
and indicate how the application interacts with the available application servers.

Create a Task Agent Rule

After you have set up the new application server and security for your users, you are ready to create task agent
rules that distribute the load between the available application servers.

1. Launch the Epicor ERP application.

2. Login using epicor/epicor.

3. Navigate to System Agent Maintenance.

Menu Path: System Setup > System Maintenance > System Agent
Important This program is not available in Epicor Web Access.

4. The current system agent displays on the Detail sheet. Only one system agent is available.

5. Click on the Actions menu and select the Edit Task Agent Rules... option.
The Task Agent Rules window appears.

6. Click the New button.

7. Select the Company for which this task agent rule will generate tasks.
Only companies assigned to the current user account display on this drop-down list. The task agent rule will
then handle processing for the selected company.

52 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Distribute the Load

8. Optionally select the Security Group for the task agent rule.
Whenever a user assigned to the selected security group runs a report or process linked to this task agent
rule, the application server linked to this rule generates the system activity.

9. Next define the Rule Type option. This value indicates what tasks are handled by the task agent rule.
Available options:
• Specific Task - Indicates this task agent rule will only run against a specific process. After you select this
rule type, you next select the process from the Process Id drop-down list.
• All Tasks - Indicates all processes are run against this task agent rule. Any time a process is launched by
users within a company or a security group, this task agent rule handles the processing.
• Specific Report - Indicates this task agent rule will only run against a specific report. After you select
this rule type, you next select the report from the Process Id drop-down list.
• All Reports - Indicates all reports are run against this task agent rule. Any time a report is launched by
users within a company or a security group, this task agent rule handles the report generation.

10. If you select either the Specific Task or the Specific Report rule type, you next select the Process Id for the
item you want the task agent rule to run. Depending on the rule type, either reports or processes display
on this drop-down list.

11. Enter the Appserver URL for the application server that will run the activity for this task agent rule. This
value links the task agent rule to the application server's location.
Tip You can find this value by opening the system configuration (.sysconfig) file for the client
installation. Locate the <appSettings><AppServerURL> node and copy this value.

12. Use the Endpoint Binding drop-down list to indicate how this application server checks for authentication
certificates through Internet Information Services (IIS). Select the Endpoint Binding defined on the application
server; the same options display on this drop-down list.
• UsernameWindowsChannel - This NET.TCP binding authenticates transactions through an Epicor
Username and Password. Windows checks for existing Epicor user accounts to authenticate logins.
• UsernameSSLChannel - This NET.TCP binding authenticates transactions using a Secure Sockets Layer
(SSL) X509 certificate. Leverage this method for application servers that handle smart client installations
when users reside in different domains. By using an SSL certificate, users from these different domains
can log into the Epicor application.
Selecting this option causes the SSL Certificate Subject Name and DNS Endpoint Identity fields to
appear. You use these fields to enter the name of your SSL certificate and the identity of the server.
• Windows - This NET.TCP binding authenticates transactions using a Windows Username and Password.
Any user with a Windows Username and Password within this domain can successfully log into the Epicor
application.
• HttpBinaryUsernameSslChannel - This HTTP binding protocol authenticates using a Secure Sockets
Layer (SSL) X509 certificate. The data transfers between the client and server using Hypertext Transfer
Protocol (HTTP). Instead of the transport, the message which contains the data transfer is encrypted.
Because this binding does not use Hypertext Transfer Protocol Secure (HTTPS), it tends to be slower than
bindings which use HTTPS.
Use this method for application servers that handle smart client installations when users reside in different
domains. By using an SSL certificate, users from these different domains can log into the Epicor ERP
application.
Selecting this option causes the SSL Certificate Subject Name and DNS Endpoint Identity fields to appear.
You use these fields to enter the name of your SSL certificate and the identity of the server.

Epicor ERP | 10.2.600 53

Distribute the Load Performance and Diagnostic Advanced - On Premise Course

• HttpsBinaryUsernameChannel - This HTTPS binding authenticates transactions using an Epicor Username

and Password. The data transfers between the client and server using Hypertext Transfer Protocol Secure
(HTTPS). HTTPS encrypts the data transfer.
• HttpsBinaryWindowsChannel - This HTTPS binding authenticates transactions using a Windows
Username and Password. The data transfers between the client and server using Hypertext Transfer
Protocol Secure (HTTPS).
You can select this method for application servers that handle smart client installations and Epicor Web
Access (EWA) installations where users access the application through the same domain. Any user with
a Windows Username and Password within this domain can successfully log into the Epicor application.
• HttpsOffloadbinaryUserNameChannel - This HTTPS protocol binding is a configuration that offloads
encryption handling to an intermediary Application Request Router such as an F5.
The binding authenticates using an Epicor Username and Password token. The data transfers between
the client and server using Hypertext Transfer Protocol Secure (HTTPS). This protocol is configured to
move encryption handling to an intermediary Application Request Router like F5 or a similar router.
• HttpsOffloadBinaryAzureChannel - This HTTPS protocol binding is a configuration that offloads
encryption handling between Epicor ERP to an intermediary Application Request Router such as an F5.
The binding authenticates using a security token by specifying a valid authentication claim between
Epicor ERP and Azure AD. The data transfers between the client and server using Hypertext Transfer
Protocol Secure (HTTPS). This protocol is configured to move encryption handling to an intermediary
Application Request Router like F5 or a similar router.
Important When this binding is implemented, in order to avoid the AddressFilter mismatch error,
be sure to uncomment the AddressFilterModeAny node in web.config as shown below:

Uncomment this element when AddressFilter mismatch at the EndpointDispatc

her happens
<AddressFilterModeAny/>

• HttpsOffloadBinaryIdpChannel - This HTTPS protocol binding is a configuration that offloads encryption

handling between Epicor ERP to an intermediary Application Request Router such as an F5.
Important Epicor Identity Provider is a new Global Authentication Service that unifies various
identity and authentication mechanisms across ERP products. The service will be made available
for approved customers in upcoming releases of Epicor ERP. By default, this option is only available
internally to Epicor.

The binding authenticates using a security token by specifying a valid authentication claim between
Epicor ERP and Epicor Identity Provider deployment. The data transfers between the client and server
using Hypertext Transfer Protocol Secure (HTTPS). This protocol is configured to move encryption handling
to an intermediary Application Request Router like F5 or a similar router.
Important When this binding is implemented, in order to avoid the AddressFilter mismatch error,
be sure to uncomment the AddressFilterModeAny node in web.config as shown below:

Uncomment this element when AddressFilter mismatch at the EndpointDispatc

her happens
<AddressFilterModeAny/>

• HttpsBinaryAzureChannel - Use this protocol to enable authentication of ERP application users against
users in Microsoft Azure Active Directory (Azure AD).
This binding relies upon the user authenticating against Azure Active Directory and obtaining a token
to present to Epicor ERP. The data transfers between the client and server using Hypertext Transfer
Protocol Secure (HTTPS).

54 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Distribute the Load

• HttpsBinaryIdpChannel - Use this protocol to enable authentication of ERP application against Epicor
Identity Provider (IdP).
Important Epicor Identity Provider is a new Global Authentication Service that unifies various
identity and authentication mechanisms across ERP products. The service will be made available
for approved customers in upcoming releases of Epicor ERP. By default, this option is only available
internally to Epicor.

This binding relies upon the user authenticating against IdP and obtaining a token to present to Epicor
ERP. The data transfers between the client and server using Hypertext Transfer Protocol Secure (HTTPS).

13. If you select either the UsernameWindowsChannel, UsernameSSLChannel, or the

HttpsBinaryUsernameSslChannel endpoint binding option, enter the User ID and Password for the account
that can access this application server.

14. When you finish adding the task agent rule, click Save.

Continue to add the task agent rules you need. If you need to remove a task agent rule, highlight it on the grid
and click the Delete button.
Now the next time the system agent activates a schedule or a user launches a process or report, the tasks are
distributed to the application servers defined on the task agent rules.

Epicor ERP | 10.2.600 55

SQL Server Tuning Performance and Diagnostic Advanced - On Premise Course

SQL Server Tuning

® ®
This section of the course describes some performance tuning options and system tips for Microsoft SQL Server .

Workshop: Review Server Properties

This workshop explores some server properties you can check to make sure SQL Server is set up for good
performance.

SQL Server Properties

You can review some SQL Server properties to make sure SQL Server is running at optimal performance. These
properties display in Microsoft SQL Server Management Studio.

1. Launch Microsoft SQL Server Management Studio.

2. Connect to SQL Server.

3. From the Object Explorer, right click the SQL Server icon; from the context menu, select Properties.
The Server Properties window displays.

56 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course SQL Server Tuning

4. From the Select a Page group box, select the Advanced option.

5. Locate the Miscellaneous section.

6. Change the Blocked Process Threshold property to 5.

This indicates that this property will monitor the tasks running on the server; any process taking longer than
five seconds will generate a blocked process threshold event in the log.

Database Properties

You can review some database properties to make sure SQL Server is running at optimal performance. You view
these properties in Microsoft SQL Server Management Studio.

1. Return to Microsoft SQL Server Management Studio.

2. From the Object Explorer, expand the Databases node, right click on a database icon, and select Properties.
The Database Properties window displays.

Epicor ERP | 10.2.600 57

SQL Server Tuning Performance and Diagnostic Advanced - On Premise Course

3. From the Select a Page group box, select the Options icon.

4. Parameterization - This database property needs to be set to Forced. These causes literal values within
SELECT, INSERT, UPDATE, or DELETE statements to convert to a parameter when queries are compiled.

5. Recovery Model -- Verify the Recovery model property is set to Full. Now when you need to recover data,
the transactions are stored in two locations. Once you define this property, you need to schedule transaction
log backups; this reduces the transaction log (xxx.ldf) file size.

6. Database Size - Be sure to increase, or "pre-grow" the database to the size you estimate it will be in a
year. To do this, select the Files node. Enter this pre-growth value in the Initial Size column. The file growth
size should be by percentage; typically you should set this value to 10%.

7. Log Locations -- The database log (for example Epicor10.mdf) and the transaction log (for example,
Epicor10_log.ldf) should be located in separate storage locations. You enter these different locations by
selecting Files node and defining the path for each log.

58 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course SQL Server Tuning

TempDB Files

You should create one tempdb file for every two CPU cores available on your server. This increases concurrent
use of the server when the tempdb files are accessed.
These tempdb files should not be located on the same drive as your operating system. If possible, these files
should also be in a separate drive other than the Epicor ERP database.
Use the following script to add the correct number of tempdb files. You modify a value in this script to reflect
the number of CPU cores you have available:
USE [master]
GO
DECLARE @cpu_count int,
@file_count int,
@logical_name sysname,
@file_name nvarchar(520),
@physical_name nvarchar(520),
@size int,
@max_size int,
@growth int,
@alter_command nvarchar(max)

SELECT @physical_name = physical_name,

@size = size / 128,
@max_size = max_size / 128,
@growth = growth / 128
FROM tempdb.sys.database_files
WHERE name = 'tempdev'

SELECT @file_count = COUNT(*)

FROM tempdb.sys.database_files
WHERE type_desc = 'ROWS'

SELECT @cpu_count = cpu_count

FROM sys.dm_os_sys_info

WHILE @file_count < @cpu_count 0.5 -- Add * 0.25 here to add 1 file for every 4
cpus, * .5 for every 2 etc.
BEGIN
SELECT @logical_name = 'tempdev' + CAST(@file_count AS nvarchar)
SELECT @file_name = REPLACE(@physical_name, 'tempdb.mdf', @logical_name +
'.ndf')
SELECT @alter_command = 'ALTER DATABASE [tempdb] ADD FILE ( NAME =N''' +
@logical_name + ''', FILENAME =N''' + @file_name + ''', SIZE = ' + CAST(@size
AS nvarchar) + 'MB, MAXSIZE = ' + CAST(@max_size AS nvarchar) + 'MB, FILEGROWTH
= ' + CAST(@growth AS nvarchar) + 'MB )'
PRINT @alter_command
EXEC sp_executesql @alter_command
SELECT @file_count = @file_count + 1
END
After you execute this query, restart the MSSQLSERVER service to implement the changes.

Epicor ERP | 10.2.600 59

SQL Server Tuning Performance and Diagnostic Advanced - On Premise Course

System Tips

The following are some additional system troubleshooting tips:

• Schedule virus scans during off hours.
• Schedule the different types of SQL database backups at appropriate times during the work day. Schedule
Full database backups during off hours, Differential backups periodically throughout the day (typically once
an hour), and Transaction Log backups every 10-15 minutes.
• Use the SQL transaction replication functionality to set up a separate reporting database on a separate server.
Several Epicor tools, including AFR, EPM, and XLConnect, require transaction replication.
• Avoid running ad hoc queries and Access queries against the production SQL server.
• Be aware third party Business Intelligence (BI) tools that directly connect to the production SQL server can
slow performance.
• When you use a dedicated application server and SQL server and then have multiple applications running at
the same time, performance slows.
• Only use the programs you need when you need them; close them if they are no longer active.
• If external companies are not being used, delete them.
• To improve performance on client workstations, consider restricting the number of concurrent sessions users
can run.
• Performance slows when several large reports generate at the same time. To resolve this, move your report
generation processes to separate application servers. This distributes the load across your system.
• Schedule large reports, such as the Job Traveler, to automatically generate during off hours.
• Integrating data with a third party application can cause performance delays. If a database integration is "in
transaction" and can only complete when the entire loop to the third party application is complete, you could
experience locking and blocking. To avoid performance delays, do not use SQL triggers on Epicor database
tables. Instead use Service Connect to set up the integration.
• When you use Service Connect to set up the integration, use .NET references, Epicor ERP business objects,
and method calls.
• Avoid creating Business Process Management (BPM) methods that generate email messages synchronously.
When you generate email through this BPM option, each email needs to be sent before other functions can
run. If these synchronous methods run frequently, performance is reduced. Instead, set up BPM methods to
send email asynchronously. Other functions can then run while the email is in the system queue.
Note Some of these tips you have already explored through workshops in either the Performance Tuning
and Diagnostic Basics course or this advanced performance tuning course.

60 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Microsoft Tools

Microsoft Tools

This section of the course describes the performance tools available from Microsoft. These tools are either included
in your Windows operating system or can be downloaded from Microsoft.

Event Viewer

The Event Viewer is an administrative tool installed with your Windows operating system. Use this tool to review
exceptions that occur outside of the Epicor ERP application.
The application server tracing log you activate within the Epicor Administration Console records business logic
exceptions internal to the Epicor ERP application. For example, this log records an error when too many characters
are entered in a field, a record already exists, how long it takes a business object method to run, and so on.
Any errors that occur outside of the internal business logic is recorded in the Event Viewer. Any errors you are
not seeing in the application server log should display in this administrative tool. Examples of items not captured
by this log include framework exceptions, security exceptions, fatal errors, and similar items.
Be sure to launch the Event Viewer to catch other problems that may slow performance. When you use the Event
Viewer with the application server log, you have a complete picture of the application, server, and network issues
you may be experiencing.

To launch the Event Viewer:

1. Access the Control Panels.

Epicor ERP | 10.2.600 61

Microsoft Tools Performance and Diagnostic Advanced - On Premise Course

2. Select the System and Security link.

3. Now click the Administrative Tools link.

4. Click the Event Viewer shortcut icon. The Event Viewer displays.

You should consider adding this shortcut to your desktop so you can more easily launch this key tool when you
need it.

SQL Performance Dashboard

The SQL Performance Dashboard displays a visual report that shows the current performance of your SQL Server.
This key tool provides you with a snapshot of how well the CPU is utilized, current waiting requests, and other
key performance information.
You can use this tool to gauge how well your Epicor ERP application is running. Since the Performance Dashboard
displays the current state of the system, launch this tool to both determine if poor performance is occurring and
evaluate the performance results after you have made a change.

The dashboard contains graphics and hyperlinks. You can click on these items to drill down further into the
accumulated information. Use the Performance Dashboard with the other performance tuning tools to develop
a complete understanding of your system and application performance.

Workshop: Install Performance Dashboard

You download this tool and then set it up to monitor your SQL Server.
While you can install the Performance Dashboard on your SQL®Server machine, you can install this tool on other
machines as long as you have SQL Server Management Studio available on these machines. You will also need
access to SQL Server. You can then indicate from which SQL Server instance you want to pull the performance
data.

62 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Microsoft Tools

1. Use your internet browser to navigate to:

http://www.microsoft.com/en-us/download/confirmation.aspx?id=29063

2. Download the Performance Dashboard installer. Be sure to use the SQL Server 2012 version.

3. Double-click this program to launch the installer. Advance through the wizard to complete the installation.
®
4. Launch SQL Server Management Studio .

5. Connect to SQL Server.

6. From the tree view, expand the Databases node.

7. Right click the database you wish to review; from the context menu, select Reports > Custom Reports.
The Open window displays.

8. Navigate to the C:\Program Files (x86)\Microsoft SQL Server\110\Tools\Performance Dashboard\

folder; select the performance_dashboard_main.rdl file.

9. When the Run Custom Report warning message displays, click OK.

The SQL Performance Dashboard is connected to your SQL Server instance and is displaying performance data.
Now you can always launch the Performance Monitor by first right-clicking the database, selecting Reports from
the context menu, and then selecting the performance_dashboard_main report.

Performance Analysis of Logs (PAL) Tool

The Performance Analysis of Logs (PAL) tool is a shareware program that evaluates server logs to generate a
detailed HTML performance report. This report evaluates several areas of your system and displays alerts when
logs exceed specific performance thresholds.
This report graphically displays key performance counters, and then alerts you when the system test surpasses
the optimal performance level defined for the counter. These thresholds were originally defined by members of
the BizTalk Server and Microsoft Support teams, and so the counters use established performance thresholds to
determine the report results. Use this report tool to both automate performance log analysis and save
troubleshooting time.
A common mistake when evaluating poor performance is to assume the cause is an application process, when
instead the actual issue is the overall scalability of the system. If too many users access a system that does not
contain enough resources to handle them, slow performance can result. The PAL tool generates a report that
can help you accurately evaluate whether the issue is application performance or system scalability. Generally,
poor performance has one of these main causes:
• Application Performance – The overall Epicor ERP application runs slowly due to a configuration issue or
some other cause. To resolve this situation, the application can be optimized in various ways. If the performance
issue persists, further performance optimization may be required from Epicor.
• Program Performance – When a specific program is run, like Customer Shipment Entry, it runs slowly. Due
to the bogged down database access, this can cause other areas of the system to run slowly as well. To fix
this performance, evaluate the process to determine if small changes can resolve the issue or if the optimization
needs to be handled by Epicor.
• Scalability – The application is set up correctly, but the daily load on the system slows everything down. To
improve this situation, better system hardware may be required.

Epicor ERP | 10.2.600 63

Microsoft Tools Performance and Diagnostic Advanced - On Premise Course

The PAL tool generates an accurate report that can help you determine what specifically is causing the slow
performance. By evaluating the threshold alerts generated by this report, you can pinpoint the cause(s) of the
slow performance.

Download the PAL Tool

The PAL tool is available as a free download from the CodePlex website. To download this tool, navigate to the
following website: http://pal.codeplex.com/
PAL tool documentation is also available from Codeplex. To download sample reports and other documentation,
navigate to this website: https://pal.codeplex.com/releases/view/6759
The following workshop illustrates how you generate the logs you need for the PAL tool. It then shows you a
case study that configures and runs the PAL tool system report.
Important The Performance Analysis of Logs (PAL) Tool only runs on 64-bit machines. You will not be
able to install this tool on a 32-bit machine, so be sure to install the PAL tool in a 64-bit environment.
However, you can still upload system logs generated on 32-bit machines and analyze them through the
PAL tool.

Workshop - Use PAL Tool

The following series of workshops illustrate how to use the PAL tool.

Activate PowerShell

The PowerShell program needs to be active in order for the PAL tool to generate Perfmon analysis reports.
The PowerShell script that PAL runs is not signed, so you may need to run the following command to execute
these unsigned scripts.

1. From the Windows desktop, click Start > Run.

The Run window displays.

2. In the Open field, enter powershell and click OK.

The DOS shell window displays.

3. At the prompt, enter Set-ExecutionPolicy Unrestricted and press <Enter>.

4. When you are asked if you want to change the execution policy, enter Y for Yes.

You are now ready to run the PAL tool to generate the performance report.

Gather Logs

These instructions describe how you set up the logs you need to analyze through the Performance Analysis of
Logs (PAL) tool.
You run this log on the Application server(s) and the SQL server — one log instance on each server. To generate
the best results, run this log over a work day that experiences heavy network load.
To begin, you generate a .blg file through the Windows Performance Monitor (perfmon). This performance
tool is available on all Windows systems. Use it to record some key performance counters you later evaluate using
the Performance Analysis of Logs (PAL) tool.

1. From the Windows desktop, either click Start > Run or launch the PowerShell.

2. In either the Open field or the PowerShell prompt, enter perfmon.

64 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Microsoft Tools

The Performance Monitor window displays.

3. From the Tree View, expand the Data Collector Sets > User Defined folder.

4. Right-click this folder; from the context menu select New > Data Collector Set.
The Create new Data Collector Set window displays.

5. In the Name field, enter SystemOverview (or another name that helps you identify the log file.

6. Select the Create from a template (Recommended) radio button option.

7. Click Next.

8. You are asked which template you would like to use. Select the System Performance option.

9. Click Next.

10. Now indicate the folder where you would like the data to be saved. You can enter this Root directory
directly or click the Browse... button to find and select this folder.

11. Click Next.

12. You are next asked if you want to create the data collector set. Select the Save and close radio button
option.

13. Click Finish.

You return to the Windows Performance Monitor.

14. Right-click the SystemPerformance node; from the context menu, select Start.

The Windows Performance Monitor now captures data throughout the working day and stores it in a .blg file in
either C:\PerfLogs or C:\PerfLogs\<Username> (where <Username> is the name of the user logged into
Windows).
Tip The number of CPU cores affect how long the BULK INSERT statements run on target machines. You
should see high use of the CPU cores on the target MSSQL database, as BULK INSERT statements work
effectively when parallel cores executive (MDOP), improving the performance of the server.

Generate Performance Analysis

After you have collected the log data you need, stop the log and save the generated .blg file. You are now ready
to analysis the log through the PAL tool.
Do the following to generate the performance analysis. Be sure to run this process on a 64-bit environment.

1. Launch the PAL tool; you should be able to launch the program from the Start button. However, the default
install path is either C:\Program Files\PAL or C:Program Files(x86)\PAL.
The PAL Wizard window displays.

2. Click on the Counter Log button.

3. For the Counter Log Path, click the Browse (...) button to find and select the .blg file. For this example,
select the SamplePerfmonLog file and click Open.

4. Click Next.

Epicor ERP | 10.2.600 65

Microsoft Tools Performance and Diagnostic Advanced - On Premise Course

The Threshold File options display.

5. From the Threshold File Title drop-down list, select the title of the file you used to gather data through
the Windows Performance Monitor (Perfmon). If you used the SystemOverview.xml file, you would select
System Overview from this list.

6. Click Next.
The Questions options display.

7. Answer the various Questions that appear under the Questions list. These questions identify the various
aspects of the system. Answer the following questions:
• OS – Defines the operating system and architecture you use.
• PhysicalMemory – Indicate how much RAM memory is available is available on the server in gigabytes.
• UserVa - Defines the size of the user mode virtual address space in megabytes (MB). The PAL Wizard
uses this value if you are evaluating a 32-bit system, but it ignores this value on a 64-bit system.

8. Click Next.
The Output Options display.

9. Indicate how many time intervals (timeslices) you want to use on the report. These time intervals separate
the log into regular time units. Click the Analysis Interval drop-down list to select the interval you need;
select AUTO to divide the log into 30 timeslices.

10. Click Next.

The File Output options display.

11. For the Output Directory file, accept the default or click the Browse (...) button to specify where you
would like to place the PAL analysis report.

12. If you wish, enter the HTML Report File Name you need. Notice by default it uses the [LogFileName] as
a prefix and adds a [DateTimeStamp] suffix.

13. Click Next twice to display the Execute options.

14. Select the Execute: Execute what is currently in the queue radio button option.

15. Click Finish.

The PAL tool will then analyze the .blg file. Depending on the size of the file, it may take as long as 2-3 hours to
finish the analysis. When the process is complete, the .html file is available in the File Output location you specified.
You display this report in your internet browser. You click on links at the top of this .html report to locate the
performance analysis section you need.
Tip If you need more help evaluating these logs, you could also gather these logs and send them to Epicor.
Epicor technical support specialists can then evaluate the performance of the Epicor ERP application and
the overall system.

66 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course SQL Server Trace Flags

SQL Server Trace Flags

Epicor has identified a series of SQL Server trace flags that you should activate for all your Epicor ERP installations.
These trace flags define server characteristics and actions which will improve how SQL Server interacts with the
Epicor ERP application.
Activate these trace flags:

Trace Flag -T1117

By default, each database file automatically grows one file at a time. When all files in a file group are full, the
auto growth event will next increase the size for the first data file using the auto growth amount you have
defined. As rows are added, SQL Server will use this free space inside this first data file to expand the database.
Because only one file has available space, SQL Server will only add data to this file until it is full, causing the auto
grow process to launch again on the next file. This can slow performance.
When you activate trace flag -T1117, all files within a file group grow simultaneously using the auto growth
amount. For example, if your auto growth setting is 2 MB, all files within the file group will automatically grow
by 2 MB. This makes more space available, reducing the number of times the auto growth process activates.

Trace Flag -T1118

Use this trace flag to improve how SQL Server allocates full extents in the tempdb database. By default SQL Server
will allocate a page to the tempdb object, and each page can be a different size. If you move data in a large
environment, the multiple page sizes can slow performance. When you enable trace -t1118, you cause SQL Server
to generate uniform extent allocation sizes.

Trace Flag -T1222

This trace flag tracks deadlock information for display in the SQL Server log. You can then display these errors
within the Event Viewer. It returns key information using an XML format; each deadlock call returns:

1. The deadlock victim

2. The processes involved in the deadlock

3. The resources involved in the deadlock

You can use this information together with the deadlock graph to determine what processes are causing deadlocks.
To learn more about deadlocks, review the Deadlocks section later in this guide.

Trace Flag -T2453

When a table variable populates with many rows and then joins these rows with other tables, the query optimizer
may inefficiently join these tables. This can slow query performance. Activate this trace flag to cause SQL Server
to recompile a table variable when enough rows change. This updated recompile process causes SQL Server to
more efficiently join these tables, so the performance of framework base methods such as GetRows, GetByID,
GetList, and GetBySysRowID will improve.

Epicor ERP | 10.2.600 67

SQL Server Trace Flags Performance and Diagnostic Advanced - On Premise Course

Workshop: Activate SQL Server Trace Flags

The following workshops explore how you activate SQL Server trace flags.

Add to Startup Parameters

You can add SQL Server trace flags to the startup parameters for each instance of SQL Server. You do this through
the SQL Server Configuration Manager.

1. Launch SQL Server Configuration Manager. In some Windows installations, you can launch it by clicking
Start > Microsoft SQL Server > Configuration Tools > SQL Server Configuration Manager. You can
also launch the Search bar to find and select this program.
The SQL Server Configuration Manager displays.

2. From the tree view, select SQL Server Services.

Your SQL Server instances display.

3. Right-click the SQL Server instance you need to update (for example, MSSQLSERVER); from the context
menu, select Properties.
The Properties window displays.

68 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course SQL Server Trace Flags

4. Click the Startup Parameters tab.

5. In the Specify a startup parameter: field, enter the trace flag you want to activate. For this example, enter
-T1222.

6. Click Add.
The trace flag displays in the Existing parameters: field. Note that if you wish to remove the trace flag,
highlight it on this list and click the Remove button.

7. Continue to add or remove trace flags to this SQL Server instance. When you finish, click Apply and then
OK.

8. To activate the trace flag, you next must stop and restart the service. Right-click the SQL Server instance;
from the context menu, select Stop.

9. Now right-click the SQL Server instance; select Start.

The SQL Server log will now record deadlock information.

Add to Registry

Instead of activating SQL Server trace flags on each SQL Server instance, you can add them to the registry for all
SQL Server instances on your system.
If you have multiple SQL Server instances running, manually changing the startup parameters on each instance
can be time consuming. When you instead add these trace flags to the registry script, all SQL Server instances
now use these trace flags.
Be sure you test this script against a SQL server instance. When the test works as expected, you can extend this
script to run against all SQL Server instances by using a Central Management Server.

1. Launch SQL Server Management Studio.

2. Navigate to the SQL Server registry file.

3. You first indicate which trace flag activates. You add this flag to the @Parameters variable.
Example
declare @Parameters varchar(max)='.T1222',
@Argument_Number int,
@Argument varchar(max),
@Reg_Hive varchar(max),
@CMD varchar(max)

4. In case the parameter already exists, you next enter code to clean up the specified startup parameter. This
ensures the trace flag is reusable, so the code will not break if the parameter already exists.
Through the following code, you check the registry for a SQL instance that uses the sys.dm_server_registry
DMV. This will return the correct path for the startup parameters.
Example
if exists(select * from sys.dm_server_registry where value_name like 'S
QLArg%' and convert(varchar(max), value_data)=@Parameters)
begin
select
@Argument=value_name,@Reg_Hive=substring(registry_key,len('HKLM\')
+1,len(registry_key))
from sys.dm_server_registry where value_name like 'SQLArg%' and co

Epicor ERP | 10.2.600 69

SQL Server Trace Flags Performance and Diagnostic Advanced - On Premise Course

nvert(varchar(max),value_data)=@Parameters

set @CMD='master..xp_regdeletevalue
"HKEY_LOCAL_MACHINE",
"'+@Reg_Hive+'",
"'+@Argument+'"'
exec(@CMD)

end

5. You next leverage xp_regwrite to add the trace flag as a startup parameter.
Example
----------------------------------------------------{Add Parameter}
--select *from sys.dm_server_registry where value_name like 'SQLArg%'
select @Reg_Hive=substring(registry_key,len('HKLM\')+1,len(registry_key
)),@Argument_Number=max(convert(int,right(value_name,1)))+1
from sys.dm_server_registry
where value_name like 'SQLArg%'
group by substring(registry_key,len('HKLM\')+1,len(registry_key))

set @Argument= 'SQLArg'+convert(varchar(1),@Argument_Number)

select @Argument,@Reg_Hive

set @CMD='master..xp_regwrite
"HKEY_LOCAL_MACHINE",
"'+@Reg_Hive+'",
"'+@Argument+'",
"REG_SZ",
"'@Parameters+'"'
exec (@CMD)

6. Save the registry file.

7. For this update to run, your SQL Server instances need to refresh. You do this in SQL Server Management
Studio; stop and restart the SQL Server Service.

Now each time SQL Server instances launch throughout your system, the trace flag activates on them.

70 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Custom Trace Logs

Custom Trace Logs

The client and server logs have a series of default tracing options you can activate. However you can also customize
what the server log and the client log captures, displaying other operations or server activity you may want to
review.
To customize the server log, you can add profiles and tracing options by manually updating the AppServer.config
file. You can then capture additional data to help you evaluate a performance issue and/or track server activity.
You can record these same server operations or activity in the client trace log. You might want to activate the
profiles and trace options from a client machine instead to reduce the impact on performance. These tracing
options will only capture server activity caused by actions that occur on the client installation. To customize the
operations or server activity recorded on the client log, you add a setting on the client installation's .sysconfig
file.
Note For a list of the main traces and profiles you can activate, review the Performance Tuning Guide in
the application help. The Customize Logs section documents the profiles and traces that can help you
track and identify performance issues.

Workshop: Customize Server Logs

To add profiles and traces to the server log:

1. 1. Navigate to the AppServer.config.template file.

This file is located in this directory path:
• C:\Epicor\ERP10\<YourEpicorVersion>\Server\AppServer.config.template

2. Within your text editor, click File > Save As.

3. Save the template as the AppServer.config.

These default profile and tracking options are now available to use.

4. You can activate a profile or tracking option by removing the comments around it. Remove the comments
from the <add uri="trace://ice/fw/DynamicQuery" /> trace flag.
This trace captures business activity query (BAQ) execution information, which will help you analyze
performance issues.

5. You can also add additional profiles and traces. Place the profile or trace syntax within the <TraceFlags>
setting.

6. Save the AppServer.config.

7. Launch the Epicor Administration Console.

8. Expand the tree view to locate and select the application server.

9. Stop the application server.

Epicor ERP | 10.2.600 71

Custom Trace Logs Performance and Diagnostic Advanced - On Premise Course

10. Start the application server.

The server log now tracks query calls to the server. This example illustrates a log entry generated by the dynamic
query trace:
<Op Utc="2013-08-15T08:48:28.9994569Z" act="Ice:BO:DynamicQuery/DynamicQuerySvc
Contract/Execute" dur="20.9593" cli="fe80::35a2:7247:6cab:d0e9%11:53737" usr="m
anager" tid="17" pid="11176">
<BAQ QueryID="udfield" />
<BAQ Company="EPIC03" />
<BAQ SQLExecTime="12.2756" />
<BAQ DataFetchTime="3.0597" />
<BAQ PagingMethod="Simple" />
<BAQ PageNumber="1" />
<BAQ PageSize="2" />
<BAQ TotalTime="19.4385" />
<BAQ TotalRows="2" />
</Op>

Workshop: Add Server Traces to Client Log

You might want to track the custom profiles and trace options from a client machine instead to reduce the impact
on performance. These tracing options will then only capture server activity caused by actions that occur on the
client installation.
To customize the operations or server activity recorded on the client log, you add a setting on the client installation's
.sysconfig file.

1. Navigate to the client default.sysconfig file.

The client default.sysconfig file is located in this directory:
• C:\< YourEpicorInstallation>\Client\Config\default.sysconfig

2. Make a copy of the default.syconfig file. Name this copy using a file name that helps you easily identify it.
Important You should not make changes to the default.sysconfig file. By creating a copy of this
default file, you can always use the original default file to revert the client installation back to its
original launch settings.

3. After you copy the .sysconfig file, set up the client installation to launch with this file. Right-click the Epicor
ERP client icon; from the context menu, select Properties.

4. Add the following run time argument (also called a switch) to the Target field:
/config=<FileName>.sysconfig

5. Open your custom <FileName>.sysconfig file in Notepad or a similar text editor.

6. Add the <serverTraceFlagsSettings> node to the .sysconfig file. You can then place the server trace and
profile options within this .sysconfig setting. For example:
<serverTraceFlagsSettings>
<add uri="trace://ice/log" />
<add uri="profile://ice/fw/tableset" />
</serverTraceFlagsSettings>

72 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Custom Trace Logs

7. Save the custom .sysconfig file.

8. Now launch the Epicor ERP application.

9. You may activate the client trace log in a couple ways:

a. If you use the Modern Shell interface, click the Arrow at the button of the Epicor ERP window. From
the bottom toolbar, select the Tracing Options button.

b. If you use the Classic interface, from the Main Menu, click Options > Tracing Options.

The Tracing Options Form window displays.

10. To activate the client trace log, select the Enable Trace Logging check box.

11. Now select the Include Server Trace check box. This indicates the client trace log will use the custom server
profiles and traces you added to the .sysconfig file.

12. Click the Apply button.

Your tracing options run immediately. As long as the Tracing Options Form remains open, the log captures the
server trace options.
Note You can also activate this feature from User Account Security Maintenance. When you activate
the client log from the Tracing sheet, you can select the Include Server Trace check box. Then each time
a user logs in with this user account, the client log automatically generates with the server traces you added
to the <FileName>.sysconfig file.

Workshop: Auto Capture Client Logs

Besides capturing server logs, you can also set up your system to automatically place client (UI Trace) logs in the
Results folder or another folder you specify. You can then activate the client log on a user account and the log
files are placed in this directory.
You do this by first modifying some settings in the Epicor.exe.config file. You then activate the client log on the
user account.

1. Using Windows Explorer, navigate to your Epicor ERP client directory. For example:
• C:\Epicor\<YourEpicorVersion>\Client

2. Locate the Epicor.exe.config file.

3. Right-click this file; from the context menu, select Open With.... > Notepad.
The Epicor.exe.config file displays.

4. Remove the comments from the <add key="EpicorAppConfigFileName" value="EpicorApp.

config"/> setting. If you need, you can change the value option to a different file name by using
environment variables or a specific file name value.
This setting causes the client installation to generate the EpicorApp.config or <YourFileName>.config
file. This file uploads any client logs generated by the current client machine to a specific directory folder
location.

Epicor ERP | 10.2.600 73

Custom Trace Logs Performance and Diagnostic Advanced - On Premise Course

5. You next define that directory file location by removing the comments around the <add key="UITrace
FileDefaultDirectory" value="%appdata%\epicor\logs"/> setting. By default, the client
log files upload to the logs folder in application data directory (for example:
C:\Users\<ClientUserName>\AppData\epicor\log); however you can change this value to a shared folder to
set up a central location to capture your client (UI Trace) files.
Tip To make it easy to find these client logs, consider placing them in the same Results folder you
defined within the Performance and Diagnostic Tool. You set up this results path on the
Options>Settings program; the Results path is on the Log Capture tab.

6. Optionally, remove the comments from the <add key="DiagnosticThreadEnabled" value="fa

lse"/> setting and change this value to True. This setting records additional performance information
you may need.
When you activate this setting, it adds a tracePacket node to the client log. This node displays CPU
information and some memory performance counters. It adds a new performance tracePacket to the client
log every 15 seconds.

7. Save the Epicor.exe.config file.

You can now activate the client log on the user account and indicate log files generated from this account
upload to the UITraceFileDefaultDirectory you indicated in this file.

8. Log into the Epicor ERP application with your security manager account.

9. Navigate to User Account Security Maintenance.

Menu Path: System Setup > Security Maintenance > User Account Security Maintenance

10. Use the Detail sheet to find and select the user account.

11. Select the Enable Trace Logging check box.

This activates the client (UI Trace) log. The next time this user logs into the application, a client log
automatically captures the calls this user makes from this client installation.

12. Now from the Log Directory Scheme drop-down list, select the Default from Epicor.exe.config file
option.
This causes the application to automatically place the client logs generated by this user account in the
directory folder you specified.

13. Save the user account.

Client information now automatically generates through this user account. These logs are available in a central
directory file folder location. You can use the Client Trace Analysis features in the Performance and Diagnostic
Tool to review these tracing results; you can also send these files to Epicor Technical Support or your Epicor
consultant for further analysis.

74 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Deadlocks

Deadlocks

A deadlock occurs when two users or sessions have locks on separate business objects, and each business object
process tries to establish a lock on the business object in use by the other user/session.
SQL Server automatically detects and resolves deadlocks. If a deadlock occurs, one process ends (the victim),
while the other process runs (the culprit). The victim transaction is rolled back. While determining the victim is
relatively easy, what you need to discover is the culprit process that caused the deadlock. This culprit could be
another Epicor ERP application process running at nearly the same time or a third party application preventing
the victim process to initialize.
You can monitor when deadlocks occur by activating trace log -T1222. If you discover deadlocks are happening,
you should then use the SQL Server Deadlock Graph. While this graph runs, deadlock events are recorded in
the SQL Profiler trace log.

Workshop: Detecting Deadlocks

You previously activated Trace Flag -T1222 in your startup procedures. Now that this trace flag is running, the
system will track deadlock errors and display them in the Event Viewer.
Trace Flag T1222 returns deadlock information using an XML format. The deadlock messages return these
attributes:
• deadlock victim - Displays the physical memory address of the task the system selected as the victim. If the
deadlock is not resolved, this value displays a 0 (zero) value.
• executionstack - Displays the Transact-SQL code that ran when the deadlock occurred.
• priority - Indicates the importance of the deadlock in comparison to other deadlocks that may be occurring.
• logused - Displays the log space used by the task.
• owner id - indicates the identifier for the transaction that controlled the deadlock request.
• status - Indicates the state of the task; this attribute can have one of these values:
• pending - The task is waiting for a worker thread.
• runnable - The task is ready to run but is waiting for a quantum.
• running - The task is running on the scheduler.
• suspended - The task is current not running.
• done - The task has finished processing.
• spinloop - The task is waiting for a free spinlock.

• waitresource - Indicates the resource this task needs to complete its run.
• scheduleid - Displays the scheduler associated with the task.
• hostname - Displays the workstation which hosted the task.
• isolationlevel - Indicates the current level of isolation for the task.
• Xactid - Displays the ID for the transaction that controls the request.
• currentdb - Contains the ID for the database.
• lastbatchcompleted - Displays the last time a client process executed a batch.

Epicor ERP | 10.2.600 75

Deadlocks Performance and Diagnostic Advanced - On Premise Course

• clientoption1 and clientoption2 - A bitmask that displays information about options controlled by SET
statements such as SET NOCOUNT and SET XACTABORT.
• associatedObjectId - Displays the heap or b-tree identifier.
The system tracks deadlocks through a lock monitor thread that initiates a search through all the tasks. You
previously reviewed this property when you checked the SQL Server Properties; you verified the Blocked Process
Threshold property was set to five seconds. If the lock monitor detects frequent deadlocks, this interval can
become as low as 100 milliseconds. When a deadlock is discovered, the system assumes the next threads that
must wait for a lock are entering the deadlock cycle. These lock waits will immediately launch a deadlock search
instead of waiting for the next deadlock interval.
Once you have determined deadlocks are occurring on your system, you should use the Deadlock Graph to get
more details on the deadlock to help you determine how to resolve the issue.

Activate the SQL Deadlock Graph

You access the SQL Server Profiler to activate the SQL Deadlock Graph and run a deadlock trace against a server.
To do this, launch SQL Server Management Studio.

1. Click Start > All Programs > Microsoft SQL Server 2012 > SQL Server Management Studio.
Microsoft SQL Server Management Studio displays.

2. When the Connect to Server window displays, click Connect.

Tip Notice you can select a different server on this window. Use this feature to activate the deadlock
graph for different servers.

3. Now click Tools > SQL Server Profiler.

The SQL Server Profiler program displays.

4. Select the File > New Trace... option.

5. When the Connect to Server window displays, click Connect.

The Trace Properties window displays, and the General sheet is active.

6. In the Trace Name field, enter a meaningful name for the tracing file.

7. From the Use the template drop-down list, select Blank.

8. Click the Events Selection tab.

9. Expand the Locks node and select the following options:

• Deadlock graph
• Lock: Deadlock
• Lock: Deadlock Chain

10. Click the Run button.

The words "Trace Start" display in the SQL Server Profiler and the trace begins. You typically run this trace log
for a full day to capture the potential deadlocks that occur on your system.

76 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Deadlocks

Stop the SQL Deadlock Graph

1. When you are ready to stop the deadlock trace log, click File > Save As > Trace File.
The Save As window displays.

2. Enter the File Name that helps you locate the file.

3. Either save the file to the default folder, or click the Browse Folders button to find and select a different
folder.

4. Click Save.

5. Now click File > Stop Trace.

Review the SQL Deadlock Graph

You are now ready to review the SQL Deadlock Graph.

1. Return to the SQL Server Profiler.

2. Using Windows Explorer, navigate to the directory that contains your tracing file. This file has the .trc
extension. If you are in an Epicor University training environment, you can view an example SQL Deadlock
Graph. Navigate to the C:\_Performance\DeadlockSample.trc file.

3. Click Open.
The trace log displays in the SQL Server Profiler view window.

4. Now from the grid, select the Deadlock graph option.

The deadlock graph displays at the bottom of the SQL Server Profiler window. Use this graph to determine
which process was the victim and which was the culprit.

Epicor ERP | 10.2.600 77

Deadlocks Performance and Diagnostic Advanced - On Premise Course

5. The left bubble (it has an "x" through it) identifies the process that was the deadlock victim. Use these
values to determine which process was locked.

6. The Key Lock squares identify which culprit business objects were trying to establish a lock on the victim
business object. Both these culprit business objects requested an update to the same key on the victim
business object.

7. The right bubble identifies the business object that is the owner of the culprit business objects.

8. You can right-click in the deadlock graph to display additional information about the deadlocked transaction.

9. When you finish your analysis, close the SQL Server Profiler.

Tip To help you further analyze this data, you can also zip up this trace file and send it to Epicor Technical
Support.

78 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Locking and Blocking

Locking and Blocking

Locking and blocking occurs when two or more database connections try to access the same piece of data
simultaneously.
A piece of data is locked when a connection needs exclusive access to it. SQL Server prevents, or locks, this piece
of data so it cannot be updated by other connections. This data is then blocked when another connection attempts
to access this data. This data becomes available when the first connection releases it. Similar to deadlocks, you
need to discover what connection(s) could not access the data (the victims) and what connection blocked the
data (the culprit).
Excessive locking and blocking slows down database performance. You can use the sp_lock3.sql stored procedure
to determine if a system is experiencing excessive data locking and blocking. By running the sp_lock3.sql stored
procedure, you can determine whether excessive locks and blocks are occurring on your database.

Workshop: The Locks Log

This series of exercises demonstrate how you set up and review the Locks Log.

Set up the sp_lock3.sql File

You download the sp_lock3.sql file through the Performance and Diagnostic Tool.

1. Launch the Performance and Diagnostic Tool. Depending on your operating system, you launch this tool in
different ways:

a. If you are on Windows SQL Server 2008 R2 or Windows 7, click Start > All Programs > Epicor Software
> Epicor Administrative Tools > Performance and Diagnostic Tool.

b. If you are on Windows SQL Server 2012 or Windows 8, press the <Windows> + F button to display the
Charms bar; from the Apps screen, select Performance and Diagnostic Tool.

The Performance and Diagnostic Tool displays.

2. Click Help > Download Diagnostic Resources.

The Download Resources window displays.

3. Select the check box next to the DeadlockAnalysis.zip item.

4. Click OK.
The DeadlockAnalysis.zip file is downloaded to your computer.

5. Unzip the DeadlockAnalysis.zip file in a file location you need.

Configure the sp_lock3.sql File

You use SQL Server Management Studio to configure this stored procedure file.

1. Click Start > All Programs > Microsoft SQL Server 2008 > SQL Server Management Studio.

Epicor ERP | 10.2.600 79

Locking and Blocking Performance and Diagnostic Advanced - On Premise Course

Microsoft SQL Server Management Studio displays.

2. Connect Microsoft SQL Server Management Studio to the SQL instance you wish to monitor.

3. Open the sp_lock3.sql script and run it against the database you wish to test. To do this, first click File >
Open > File and navigate to the folder that contains the sp_lock3.sql file; select this file.

4. Click Open.

5. In the Tree View, expand the Databases node and select the database against which you want to test for
locks and blocks.

6. Now click in the sp_lock3.sql pane; this causes the procedure to be in focus.

7. Either click the Execute button or press <Ctrl> + E.

You should see a "Command(s) completed successfully." message. This adds the sp_lock3.sql procedure to
the selected database.

8. To test the procedure, click the New Query button.

9. Enter sp_lock3 @dbname=[DatabaseName], and click the Execute button.

Tip Make sure you enter a space between sp_lock3 and the database name. Notice you place the
name of the selected database between the brackets; for example:[MainDatabase]

The log will run and you should see a "Query executed successfully." message. Depending on the locking
situation, it may show some lock data. However if you receive a "Could not find stored procedure 'sp_lock3'"
error message, the SQL procedure is not installed correctly. Repeat these steps to re-install the script.

The sp_lock3.sql procedure is now configured to create a log against the selected database.

Create a SQL Job

Now within SQL Server Management Studio, you need to schedule a SQL Server Agent job that runs the
sp_lock3.sql script and then appends its output to a log file.

1. Using the Object Explorer, expand the SQL Server Agent node; right-click the Jobs folder and select the
New Job... option.
The New Job window displays. By default, the General node is selected.

2. Enter a Name for your log file. For example, enter <XXX>_LocksLog (Where XXX are your initials or another
identifying value).

3. Verify the Enabled check box is selected.

4. Now on the Tree View, select the Steps node.

5. Click the New button.

The New Job Step window appears.

6. Enter a Step name for this job step. Enter <XXX>_JobStep01 (Where XXX are your initials or another
identifying value).

80 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Locking and Blocking

7. Click the Database drop-down list and select the database against which you want to test the locks and
blocks. For example, select Demo.

8. Now in the Command field, enter the following: sp_lock3 @dbname=[DatabaseName]

Tip Replace the [DatabaseName] text with the actual name of your database. Make sure you have a
space between sp_lock3 and the database name command.

9. Return to the tree view and select the Advanced node.

10. Use the Output file field to define the directory path for the log file. You can select the directory path by
clicking the Browse (...) button.

11. For the File name, enter <XXX>LocksLog_Output.txt (Where XXX are your initials).

12. Click OK.

You return to the New Job Step window.

13. Select the Append output to existing file check box.

14. Close the Job Step window by clicking OK.

You return to the New Job window.

15. From the Tree View, select the Schedules node.

16. Click the New button.

The New Job Schedule window displays.

17. Enter the following values in this window:

• Name: <XXX>_LocksLogSchedule (Where XXX are your initials or another identifying value)
• Schedule type: From this drop-down list, verify the Recurring option is selected.
• Enabled: Verify this check box is selected.
• Occurs: From this drop-down list, select the Daily option.
• Occurs every: Click this radio button option and indicate this log runs every 1 minute(s).
• No end date: Verify this radio button option is selected.

18. To exit out of the jobs windows, click OK twice.

You return to SQL Server Management Studio.

Start the Locks Log

You now activate the log.

1. Launch SQL Server Configuration Manager. In some Windows installations, you can launch it by clicking
Start > Microsoft SQL Server > Configuration Tools > SQL Server Configuration Manager. You can
also launch the Search bar to find and select this program.
The SQL Server Configuration Manager displays.

2. In the tree view, click the SQL Server Services icon.

The right pane displays the SQL Server icons. Verify a green arrow appears next to the SQL Server instance
you want running. For example: SQL Server (SQL2012)

Epicor ERP | 10.2.600 81

Locking and Blocking Performance and Diagnostic Advanced - On Premise Course

3. Close the SQL Server Configuration Manager.

You return to Microsoft SQL Server Management Studio.

4. If the SQL Server Agent is not running, right-click this icon on the tree view. From the context menu, select
Start.

5. You are asked if you want to start this agent. Click Yes.

6. From the Object Explorer, expand the SQL Server Agent > Jobs node.

7. Right-click the <XXX>_LocksLog (Where XXX are your initials or another identifying value); from the context
menu, select Start Job at Step.
The Start Jobs window displays with both the Success icon and statuses listed in the grid.

8. Click the Close button.

The sp_lock3 output file is appended to the database. Locking and blocking data now records to this log every
minute. Track this data for a day and then return to this log to evaluate the results.

Review the Locks Log

This section describes how you review the locks log and resolve the locking issues.
The sp_lock3 procedure only generates output when it detects locking and blocking issues. If the locks log is
empty, locking and blocking is not a performance issue on the database.
However if this log contains some locking and blocking entries, use the log to review what database connection
is trying to access the data at the same time.

In the above example, COMPANY/JoeS from the USERCOMP84 machine is accessing the Epicor ERP database.
The access currently given to USERCOMP84 is not recommended (this connection is the culprit), as it can cause
locking and blocking. In this case, Epicor would recommend that only the COMPANY/EPICADMIN login from the
APPSERVER machine have access to this data.
If the log records locking and blocking issues while users run custom or modified reports using an ODBC
connection, consider creating a table view for the report. Since database tables can become locked while the
report runs, the table view can prevent locking and blocking situations. To create a table view:
®
1. Open the report in a third party application like Microsoft® SQL Server Report Builder, Excel, or another
ODBC source.

2. Create a view for each table that has ODBC access.

3. Enter a with(nolock) statement on the view.

4. Save the file.

When users access the table view, they cannot create a share lock on the data. This improves the performance
of the report.

82 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Locking and Blocking

Tip If you decided not to create a table view for the report, you should instead enter the with(nolock
) option in the SELECT statement for the table. You should also use this option to optimize performance
on Business Activity Queries (BAQs), Business Process Management (BPM) methods, and customizations.

Blocked Process Report Event

The Blocked Process Report Event captures the details you need to identify blocked processes.
Turn on this event when you suspect your SQL Server is experiencing locking and blocking issues. Run this event
during a period when you think deadlocks are occurring. When you have gathered the trace log data you need,
shut off the event. You will then have a log that contains deadlock information to review.

Workshop: Activate the Event

Do the following to activate and use the Blocked Process Event.

Activate the Event

You activate the Blocked Process Report Event by using the SP_CONFIGURE ‘blocked process threshold’ command
in SQL Server Management Studio. This command is typically set to 0, which indicates the event is turned off.
You set this value to a number higher than 0 to activate this event.
The ‘blocked process threshold’ command is an advanced SP_CONFIGURE command. Because of this, you must
first activate the ‘show advanced options’ command. After you activate this command, the ‘blocked process
threshold’ command is available to run. You activate both commands by entering the RECONFIGURE command.
To activate the Blocked Process Report Event:

1. Launch SQL Server Management Studio.

2. Enter your credentials for the Epicor ERP SQL Server instance.

3. Click File > New > Query with Current Connection. The middle pane displays a blank query window.

4. Enter the following code block:

SP_CONFIGURE’show advanced options’,1 ;
GO
RECONFIGURE
;
GO

5. The advanced options now activate. You next activate the Blocked Process Report Event. Enter the code
similar to the following:
SP_CONFIGURE’blocked process threshold’,10 ;
GO
RECONFIGURE
;
GO
Notice in the above example the number 10 value. This value indicates how often you want this event to
check for blocked processes, calculated in seconds. So in the above example, the event runs every 10 seconds.
If any blocked processes happen during each 10 second time span, the event captures them and the locking

Epicor ERP | 10.2.600 83

Locking and Blocking Performance and Diagnostic Advanced - On Premise Course

event details are saved to the trace log. Enter the duration you think will best help you capture the locking
and blocking information.
Tip If you need to run the trace for a day or more, consider setting this duration for a longer time
period like 1800. This causes the event to activate every 30 minutes. The threshold can be set from
0 to 86,400 seconds (24 hours).

6. Save the query.

Run the event for as long as you need. Performance is not affected, as the Blocked Process Report event
uses a system task for capturing deadlocks. This system task, REQUEST_FOR_DEADLOCK_SEARCH, checks
for deadlocks every five seconds. When it finds a deadlock, it terminates one of the sessions. The Blocked
Process Report event then captures the deadlock entry by recording its details in the trace log.

7. When you finish checking for deadlocks, return to SQL Server Management Studio.

8. Access the query.

9. Shut off this event by entering 0 for the event value:

SP_CONFIGURE’blocked process threshold’,0 ;
GO
RECONFIGURE
;
GO

Run the Trace

Now that the event is active, you can run the trace log.

1. Launch SQL Server Profiler.

2. Click on File > New Trace.

3. Connect to the Epicor ERP SQL Server instance.

The Trace Properties window displays.

4. Enter a Trace name for the trace file.

5. From the Use the template drop-down list, select TSQL_Locks.

6. Click the Events Selection tab.

7. Verify the blocked process report check box is selected.

8. Clear (deactivate) the check boxes for the rest of the events.

9. Click Run.
The trace log window displays. Let the trace run for as long as you need.

10. When you suspect enough time has elapsed, return to the trace log window.

11. Select a Blocked process report row.

84 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Locking and Blocking

The blocked process report event displays in the text box at the bottom of the window. If you are in an
Epicor University training environment, you can view an example Blocked Process Report. Click File > Open
and navigate to the C:\_Performance\Profiler_BlockedProcessReport.trc file.

The data within the <inputbuf> tags contains the locking information. Review this information to determine
the source of the locking. However if the query is too large to fit in the <inputbuf> tag, the data is truncated.
To get the full query code, launch SQL Server Management Studio and enter this query statement:
select * from ::fn_get_sql(<SqlHandle>)
Copy and paste the query. Replace the <SqlHandle> tags with any of the <SqlHandle> values from the blocked
process report. This returns the full query to analyze.
When you review these blocked process entries, locate the lockMode value. This field displays the lock type that
was received or requested for the deadlock event. Some example lock types include 0=NULL, 8=IX, 16=Rangel-S.
These lock types will help you figure out why the lock occurred. You can review the various lock types in SQL
Server Books Online.
Besides the event, the accompanying .xml data might be helpful, as the values for SPID, username, current state,
and so on may help you locate the cause of the locking and blocking.
Be aware that each time duration interval may contain different blocking events. Look for events that are locked
the longest by evaluating the Duration value. If these events happen frequently, you can then identify the code
that activated the event. You should be able to determine the cause of the locking and blocking.

Resolve Locking

Blocking and locking can happen because of a number of reasons. Perhaps too much data is being returned, or
an index isn’t available, or SQL Server didn’t lock the best process. The following list identifies some solutions for
locking and blocking. As you analyze the events in the trace log, determine what would be the best solution for
resolving the blocking. Some ideas:
• Rewrite the code causing the lock.
• Fine tune your transactions so they run as quickly as possible.
• Return the minimum data amount that you need to complete a process.
• Add any indexes that may be missing.
• Verify both column and index information is up to date.

Epicor ERP | 10.2.600 85

Locking and Blocking Performance and Diagnostic Advanced - On Premise Course

• Set the isolation level to the lowest setting that will work with your system.
If you would like more information on tracing locking and blocking, Microsoft has resources available on
technet.com and in SQL Server Books Online.

86 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Performance Tuning Strategy

Performance Tuning Strategy

Once you have determined some actual and potential causes for poor performance, you are ready to try some
options.
The main strategy to remember is always change one aspect of the system at a time. That way you can clearly
evaluate the benefits and costs for each change. When you try a performance option, do the following:
• Test how long it takes to run a process before you use a performance option. Then after you implement the
option, test the same process again. You should see a significant savings in performance time.
• Be sure to record each change and why you made it, as you then can review what you did later on. Write
comments in your scripts, customizations, web configuration file, and other locations to document the changes.
Remember you can gain a lot of performance by just doing a few things. Usually adding memory and spreading
the disk workload across as many disks as possible gives you the best performance gains. Always stop after you
have accomplished enough; the more tuning you do, the smaller the return on your investment and time.

Epicor ERP | 10.2.600 87

Conclusion Performance and Diagnostic Advanced - On Premise Course

Conclusion

Congratulations! You have completed the Performance Techniques course.

88 Epicor ERP | 10.2.600

Performance and Diagnostic Advanced - On Premise Course Index

Index
L M
live memory inspection, setup 34 memory test, setup 34

Epicor ERP | 10.2.600 89

 Additional information is available at the Education and
Documentation areas of the EPICweb Customer Portal. To access
this site, you need a Site ID and an EPICweb account. To create an
account, go to http://support.epicor.com.