Sas315 2014
Sas315 2014
Sas315 2014
®
SAS 9.4 Web Application Performance: Monitoring, Tuning, Scaling, and
Troubleshooting
Rob Sioss, SAS Institute Inc., Cary, NC
ABSTRACT
SAS® 9.4 introduces several new software products to better support SAS® web applications. These products
include SAS® Web Server, SAS® Web Application Server (with the availability of out-of-the-box clustering), and
SAS® Environment Manager. Even though these products have been tuned and tested for SAS 9.4 web applications,
advanced users might want to know the tools and techniques that they can use to further monitor, manage, tune, and
improve the performance of their environment. This paper discusses how customers can achieve that by exploring
the following concepts, activities, techniques, and tools:
using SAS Environment Manager to monitor run-time performance of middle-tier components
using additional tools to monitor middle-tier components (Apache server-status, Java VisualVM, Java command-
line tools, Java GC logging)
identifying the potential bottlenecks and tuning suggestions
identifying appropriate clustering strategy (single-server vs. multi-server for homogenous or heterogeneous
clustering)
suggesting the data to collect when analyzing performance (GC data, thread dumps, heapdumps, system
resource utilization information, log files)
discussing in-depth performance analysis tools (Thread Dump Analyzer, HPjmeter, Eclipse Memory Analyzer
(MAT), IBM Support Assistant tools: GC and Memory Visualizer, Memory Analyzer, Thread, and Monitor Dump
Analyzer)
INTRODUCTION
There are many factors that can impact the overall performance of web applications and therefore dictate the
resource requirements for various components of the environment. Some of the relevant factors include the number
of active concurrent users, the usage patterns of the different applications in the deployment, common workflows
within the applications, and dependencies on external resources. The out-of-the-box configuration of the SAS® web
applications is intended to be a decent starting point, but will often require tuning to match the demands of the
environment. While the opportunities to tune various components are seemingly endless, collecting the appropriate
monitoring data and following a few general guidelines can greatly improve the performance of the applications. As
the needs of the environment change over time (due to additional users, the adoption of different workflows and
usage patterns, and so on), the systems can continually be monitored, tuned, and scaled in order to maintain optimal
performance.
1
Additional capacity can be added for the applications that require it. Due to the implementation of out-of-the-box
clustering support for SAS 9.4, there are some key points to highlight regarding heterogeneous clustering:
1. Configuring a heterogeneous cluster requires that the web application server be configured with the multiple
server option selected at configuration time. Some products cause this option to be selected by default, but
for other products, the option must be selected manually by going through the custom path of prompts in the
SAS Deployment Wizard.
2. When configuring clustering with the multiple server option selected, each server instance can be clustered
independently. Each server instance is associated with a specific set of applications, and clustering in this
approach enables the addition of nodes for that set of applications independently from applications
designated for other server instances.
WHAT TO MONITOR
When analyzing the performance of the SAS web applications, there are some key resources to track. These
resources can frequently become the limiting factor to improved performance if not properly tuned for the needs of
the environment. This list is not an exhaustive list of all factors that impact performance, but they do represent the
items that most often require tuning in order to make improvements. In addition, the guidelines presented are not
hard and fast rules. They are general recommendations that apply to the majority of scenarios, but the needs of a
given deployment might dictate deviation from the suggestions. Steps for tuning these settings are discussed later in
the section “Tuning and Scaling the Environment.”
There are JDBC resources configured for the data sources that the SAS web applications need to access. In
some cases, the workload on the system my benefit from a larger pool of connections for a particular data
source. The most commonly used data source is the SharedServices data source. The presence of the
following error message in any of the SAS web application log files (found under Web/Logs/<server
instance>/) normally indicates that the size of the connection pool should be increased:
com.atomikos.jdbc.AtomikosSQLException: Connection pool exhausted - try
increasing 'maxPoolSize' and/or 'borrowConnectionTimeout' on the DataSourceBean.
Since there might be a number of JDBC resources configured across one or more SAS Web Application Server
instances, it is possible that the SAS Web Infrastructure Platform Data Server might need to be configured to
allow more total connections. The number of allowed connections on the data server should be greater than the
sum of the maxPoolSize values of all JDBC resources in all of the SAS Web Application Server instances. Other
processes (such as SAS® Environment Manager) will consume connections as well, so the limit needs to be
high enough to accommodate all incoming connections. The following error message in the SAS web application
log files normally indicates that the SAS Web Infrastructure Platform Data Server has run out of connections:
Could not get JDBC Connection; nested exception is
com.atomikos.jdbc.AtomikosSQLException: Failed to grow the connection pool
2
SAS® Web Server Thread Utilization
ThreadsPerChild (Windows) or MaxClients and ServerLimit (UNIX, Linux):
By default the SAS Web Server is configured to support the processing of 1,024 incoming requests at a given
time. If a message similar to the example below (captured from a Windows system) is logged to the SAS Web
Server’s error log (WebServer/logs/error.log), the limit likely needs to be increased:
[warn] Server ran out of threads to serve requests. Consider raising the
ThreadsPerChild setting
MONITORING TOOLS
There are a number of different monitoring tools that can be used to collect the suggested data. As with the list of
metrics to monitor, this is not an exhaustive list of monitoring tools, but it does include tools that can be used to
collect the data needed to diagnose the majority of performance issues that can occur for the SAS web applications.
3
Figure 1 – SAS Environment Manager Monitoring Data: SAS Web Application Server Resources, Including
Heap
The tomcatThreadPool Executor thread pool is considered a service of the SAS Web Application Server instance.
Clicking the thread pool service in the list of services associated with the instance will display the metrics for the pool.
4
Figure 2 – SAS Environment Manager Monitoring Data: SAS Web Application Server Executor Thread Pool
Resources
Lastly, each host platform system in the SAS Environment Manager inventory can be monitored for potential system
resource utilization issues.
5
Figure 3 – SAS Environment Manager Monitoring Data: Host Platform Operating System Resources
Oracle HotSpot Java Tools and Features (All Platforms except AIX)
The SAS 9.4 Web Application Server (along with other SAS products) use the SAS Private Java Runtime
Environment (JRE) to run Java code. One limitation of the JRE is that it does not include some of the tools that are
shipped with a full-fledged Java Development Kit (JDK). For some of the tools listed in this section, the full JDK must
be downloaded from Oracle directly. For more information about these tools, please refer to the Oracle Java SE
documentation. Links to some of the documentation are included in the "References" section.
Java VisualVM (Requires JDK Download, Java 7 Update 15 or Higher)
The Java VisualVM utility from Oracle is another powerful monitoring tool. Using VisualVM, you can obtain statistics
from a running Java process including heap utilization, PermGen utilization, overall thread count (and thread
execution stacks), and much more. The tool also provides the capability to capture thread dumps and heap dumps
for additional analysis. In addition to the base VisualVM product, there are a number of plugins that can be
downloaded that extend the functionality. One such plugin is an MBean viewer, which provides a tree view of the
various Java Management Extension (JMX) MBeans that are available for a Java process.
To connect to a SAS Web Application Server instance, you can create a JMX connection profile for the server. By
default, the SAS Web Application Server processes are configured to only listen for incoming JMX connections on
localhost (127.0.0.1), so it is simplest to run VisualVM from the same machine as the server instance being
monitored. To allow monitoring from additional hosts, the JMX listener must be modified to listen on an address that
is accessible from the client machine. Modify the following Listener element in the <server instance>/conf
server.xml file for each instance you wish to connect to remotely, modifying the bind attribute as needed (0.0.0.0
will listen on all addresses):
<Listener accessFile="${catalina.base}/conf/jmxremote.access" authenticate="true"
bind="0.0.0.0"
className="com.springsource.tcserver.serviceability.rmi.JmxSocketListener"
6
passwordFile="${catalina.base}/conf/jmxremote.password" port="${base.jmx.port}"
useSSL="false"/>
Figure 4 – Java VisualVM Monitoring Data: JVM Resources, Including CPU, Heap, Threads, and Classes
If desired, additional plugins can be downloaded and added to VisualVM to extend the functionality by going into the
Tools -> Plugins menu. The VisualVM-MBeans plugin provides a tree view of the available JMX MBeans.
7
Figure 5 – Java VisualVM Monitoring Data: MBean Viewer Plugin Showing tomcatThreadPool MBean Data
Verbose GC Logging
Another mechanism for monitoring the Java heap is to enable GC logging for the server instances. To do so, add the
following arguments to the JVM options for each server instance to be monitored (substituting a unique, valid path for
the log for each instance):
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -
Xloggc:/path/to/filename.log
The process for modifying the JVM options is detailed in SAS 9.4 Web Applications: Tuning for Performance and
Scalability, Chapter 4. The GC log data can be analyzed and displayed graphically using a tool such as HPjmeter, as
discussed below in the section "Advanced Troubleshooting and In-Depth Analysis."
Java Command Line Utilities
There are additional Java command line utilities that are shipped with the JDK that can be used to collect other types
of data from the running process. These utilities require the ability to attach to the process, so they are most useful in
UNIX or Linux environments and Windows environments when the server is executed from a script. When a server is
deployed on Windows as a service, these utilities will not work.
The jmap utility (requires JDK download, Java 7 Update 15 only) can be used to capture statistics about running Java
processes, including heap configuration and heap usage. To obtain this data, run the following command:
jmap -heap <pid>
In addition, jmap can be used to capture a heap dump, which is a snapshot of all Java objects in the heap of the
8
running process. This is useful for analyzing a suspected memory leak. To capture the heap dump, run the following
command (substituting a valid file path with the .hprof extension):
jmap -dump:format=b,file=/path/to/file.hprof <pid>
The jstack utility (requires JDK download, Java 7 Update 15 or higher) can be used to capture a thread dump, which
is a snapshot of the execution stack of all threads in the running process. This is useful for troubleshooting a variety
of performance issues, such as slowdowns, hangs, or unresponsiveness of a process. To capture the thread dump,
run the following (substituting a valid path to redirect the output to):
jstack -F <pid> > /path/to/threaddump.txt
Alternatively, a thread dump can be captured on UNIX or Linux by running the following command (which does not
require a JDK), and the output will be sent to the <server instance>/logs/server.log file:
kill -QUIT <pid>
The heap dumps and thread dumps can be analyzed using tools such as Eclipse Memory Analyzer (MAT) and
Thread Dump Analyzer (TDA), as discussed later in the section "Advanced Troubleshooting and In-Depth Analysis."
The GC log data can be analyzed and displayed graphically using a tool such as the IBM Garbage Collection and
Memory Visualizer (GCMV), as discussed later in the section "Advanced Troubleshooting and In-Depth Analysis."
The IBM JVM also has the ability of generating heap dumps and javacore files (thread dumps) from a running
process. To enable the generation of heap dumps on demand, add the following JVM option:
-Xdump:heap:file=/path/to/heapdump.phd:events=user
To enable the generation of javacores on demand, add the following JVM option:
–Xdump:java:file=/path/to/javacore.txt:events=user
If desired, these arguments can both be added together in order to produce both types of artifacts on demand. After
adding these arguments, run the following command to trigger the generation of the snapshots:
kill -QUIT <pid>
For additional information about these tools, please refer to the AIX User Guide for IBM SDK, Java Technology
Edition, Version 7 Release 1. A link to the documentation has been included in the "References" section below.
9
Allow from 127.0.0.1 subdomain.domain.com host.subdomain.domain.com
</Location>
To view the mod_status data, simply point a browser running from a system that is allowed access to the /server-
status location to the host and port of the SAS Web Server with a request path of /server-status:
http://hostname.domain.com:port/server-status. Details about the status of the different workers that point to SAS
Web Application Server instances can be found near the bottom of the server-status page as shown below in Figure
6. For additional information about the Apache mod_status data, please refer to the Apache mod_status module
documentation. A link to the documentation can be found in the "References" section below.
10
# WINDOWS SYSTEMS
<IfModule mpm_winnt_module>
ThreadsPerChild 1280
MaxRequestsPerChild 0
</IfModule>
If system resource utilization for the OS resources indicates that a resource is being exhausted, the consideration
must be made whether to upgrade the physical hardware to add more capacity for the strained resource, or to add
cluster nodes on additional hardware. The process of adding cluster nodes is documented in the SAS 9.4
Intelligence Platform: Middle-Tier Administration Guide.
11
Figure 7 – Eclipse Memory Analyzer Displaying Histogram of All com.sas.* Java Objects in Heap Dump
Figure 8 – IBM Memory Analyzer Displaying Histogram of All com.sas.* Java Objects in Heap Dump
12
Monitor Dump Analyzer (an IBM Support Assistant Tool) for IBM heap dumps. A sample screenshot of Thread Dump
Analyzer is shown below in Figure 9. A similar screenshot of IBM Thread Dump and Monitor Analyzer is shown in
Figure 10.
Figure 9 – Thread Dump Analyzer Viewing Thread and Monitor Details from a HotSpot JVM
13
Figure 10 – IBM Thread and Monitor Dump Analyzer Viewing Thread and Monitor Details from an IBM JVM
ANALYZING GC LOGS
As previously mentioned, one technique for monitoring the health of the Java heap over time is to enable GC logging
for the Java processes. Thankfully there are tools available to that can easily parse and display the data from these
logs graphically for analysis. Among the tools that provide this functionality are HPjmeter (for HotSpot JVMs) and
IBM GCMV (for IBM JVMs, but also includes support for HotSpot JVMs). Sample screenshots of HPjmeter (Figure
11) and IBM GCMV (Figure 12) are shown below.
14
Figure 11 – Sample GC Data Displayed in HPjmeter
15
Figure 12 – Sample GC Data Displayed in IBM GCMV
CONCLUSION
The SAS web application environment is a complex environment comprised of many different components, each of
which contains a number of resource dependencies that can be tuned or scaled up to improve performance. As a
result, the process of determining how to analyze the needs of the environment and pinpoint the potential areas of
improvement might seem overwhelming. However, by applying the information presented in this article, key
resources can be tracked in a variety of ways in order to identify potential areas of concern. Monitoring, tuning, and
scaling is an ongoing process, and by applying these concepts on a consistent basis administrators can keep their
SAS web applications running smoothly for their users.
REFERENCES
The Apache Software Foundation. Apache Module mod_status. Available at
http://httpd.apache.org/docs/2.2/mod/mod_status.html.
The Eclipse Foundation. Eclipse Memory Analyzer. Available at https://www.eclipse.org/mat/.
Hewlett-Packard Development Company. HPjmeter Downloads and Documentation. Available at
http://www.hp.com/go/hpjmeter.
IBM Corporation. "AIX problem determination" In AIX User Guide for IBM SDK, Java Technology Edition, Version 7
Release 1. Available at
http://pic.dhe.ibm.com/infocenter/java7sdk/v7r0/index.jsp?topic=%2Fcom.ibm.java.aix.71.doc%2Fdiag%2Fproblem_d
etermination%2Faix.html.
16
IBM Corporation. IBM Monitoring and Diagnostic Tools for Java. Available at
https://www.ibm.com/developerworks/java/jdk/tools/#tools.
IBM Corporation. IBM Support Assistant Workbench. Available at http://www-
01.ibm.com/software/support/isa/workbench.html.
IBM Corporation. IBM Thread and Monitor Dump Analyzer for Java. Available at
https://www.ibm.com/developerworks/community/alphaworks/tech/jca.
Oracle. Java VisualVM. Available at http://docs.oracle.com/javase/7/docs/technotes/guides/visualvm/.
Oracle. JDK Tools and Utilities. Available at http://docs.oracle.com/javase/7/docs/technotes/tools/.
Oracle. Thread Dump Analyzer. Available at https://java.net/projects/tda.
RECOMMENDED READING
SAS Institute Inc. 2013. SAS Environment Manager 2.1: User's Guide. 2d edition. Cary, NC: SAS Institute Inc.
SAS Institute Inc. 2013. SAS 9.4 Intelligence Platform: Middle-Tier Administration Guide. 2d edition. Cary, NC: SAS
Institute Inc.
SAS Institute Inc. 2013. SAS 9.4 Intelligence Platform: Overview. Cary, NC: SAS Institute Inc.
SAS Institute Inc. 2013. SAS 9.4 Intelligence Platform: System Administration Guide. 2d edition. Cary, NC: SAS
Institute Inc.
SAS Institute Inc. 2014. SAS 9.4 Web Applications: Tuning for Performance and Scalability. Cary, NC: SAS Institute
Inc.
CONTACT INFORMATION
Your comments and questions are valued and encouraged. Contact the author at:
Rob Sioss
SAS Institute
SAS Campus Drive
Cary, NC 27513
(919) 531-1297
Fax: (919) 677-4444
rob.sioss@sas.com
http://www.sas.com
SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS
Institute Inc. in the USA and other countries. ® indicates USA registration.
Other brand and product names are trademarks of their respective companies.
17