DM Perf Tuning2 PDF

Copyright IBM Corporation 2010 Trademarks
Monitoring and tuning InfoSphere Master Data Management

Server, Part 2: Monitor the DB2 layer and learn about different
monitoring tools
Page 1 of 19
Monitoring and tuning InfoSphere Master Data
Management Server, Part 2: Monitor the DB2 layer and
learn about different monitoring tools
Jesse Chen (jfchen@us.ibm.com)
Senior Performance Engineer
IBM
Yongli An (yongli@ca.ibm.com)
MDM Performance Manager
IBM
Adam Muise (amuise@ca.ibm.com)
IBM InfoSphere Evangelist
IBM
25 October 2010
This article series provides guidelines on how to effectively monitor and properly tune each
layer of the infrastructure in IBM InfoSphere Master Data Management Server by focusing on
an IBM "Blue stack." The blue stack includes MDM Server, WebSphere Application Server,
DB2 on AIX, and AIX operating-system-level resources. Part 1 described for each layer a
list of key parameters and variables that affect performance. It also offers detailed instructions
on how to monitor, and it provides general recommendations on how to tune each layer.
This Part 2 of the series includes practical examples of monitoring MDM Server, along with a
summary of all the recommended tools for your reference.
View more content in this series
Introduction
This is Part 2 of an article series on how to effectively monitor and tune the IBM InfoSphere Master
Data Management Server (MDM Server) stack for overall optimal MDM Server performance. Part
1 covered important guidelines on how to effectively monitor, tune, and optimize MDM Server and
the WebSphere Application Server layer for optimal performance, with brief discussions about
best practices for a list of key areas and features that affect performance in these two layers. This
Part 2 moves on to the lower parts of the stack to cover guidelines and recommendations on how
to effectively monitor and tune the DB2 layer. This article also describes tools that are commonly
used to monitor the operating system level resources.
developerWorks ibm.com/developerWorks/
monitoring tools
Page 2 of 19
Monitoring MDM Server database and tuning DB2 on AIX
Just as important as monitoring and tuning other layers in the MDM Server stack, monitoring
and tuning the database layer is critical for optimal performance of the MDM system. In general,
common DB2 best practices apply. There is not really anything specific or special about tuning
the MDM database. The most important practice is to monitor regularly and to plan tuning and
optimization proactively to avoid a negative impact to your production systems. This section covers
some basic recommendations on what to monitor and how to monitor the DB2 database in your
MDM system. This section also describes several known tuning tips that are specific to the MDM
Server database.
There are a variety of tools you can use to monitor and analyze DB2 performance. For example,
you might traditionally rely on snapshot and event monitors for performance monitoring and
evaluation. This approach is still supported and working in recent DB2 versions. However, starting
in DB2 Version 9.7, table functions are available as a light-weight alternative to the traditional
system monitor. These new table functions are planned to be updated and enhanced with new
releases of DB2, while the snapshot interface will not be. For details, see Resources for links to
the product manual and the DB2 information center. There are also various commercial tools that
leverage the same underlying monitoring technology in DB2 for end users to achieve the same
monitoring purpose.
This section describes the approach that works for old and new DB2 versions. You can either
follow the steps to do exactly the same, or learn the techniques and ideas and apply them when
using different tools. Alternatively, the key information above is also available from the reports such
as DBSUMMARY, CURRENTSQ, CURRENTAPPS, CONNECTION, and LOCKWAIT, which are
generated by the MONREPORT module in DB2 Version 9.7 or later.
Database monitoring
Although this section focuses on monitoring the database itself, that is, what's going on inside
the database, it's also very important to monitor the status of the MDM database server in terms
of memory, CPU, and input/output activities. In particular, it's critical to have a close eye on the
utilization of the input/output system that supports the MDM database.
You need to understand exactly how well the database performs and what's going on inside
the MDM Server database as result of the MDM workload running in the system. There are two
primary tools, each serving a different purpose, with which you can access database monitor
information: the snapshot monitor and event monitors. The snapshot monitor enables you to
capture a picture of the state of the database activity at a specific moment in time, while event
monitors log data only as specified database events occur.
The snapshot monitor tool is a powerful but simple monitoring tool that incurs very little overhead.
Snapshots provide important information not only about the database overall, but also about the
applications connected to the database at a specific time. To effectively monitor the system, you
can take regular snapshot data at different times of a day and over an extended period of time
(covering days and months), or you can target specific workload intervals that are known to be
ibm.com/developerWorks/ developerWorks
monitoring tools
Page 3 of 19
performance sensitive. A well-conceived monitoring plan, especially during performance testing or
during important production workloads, can help implementers foresee and plan for many potential
database performance challenges.
Most of the monitor counters are cumulative, reflecting the total count of a particular activity since
the last snapshot monitor reset within your session from which you are issuing your db2 command.
If no snapshot monitor reset is issued, the data reflects a running count since the first db2
connection was established. Typically the data collected without a reset is not very useful because
the counters span a longer period of time, including drastically different workloads along peaks
and valleys in load. Focusing in on a particular workload or time period with snapshot collection
reveals the unique characteristics of the workload and enables more decisive tuning decisions. For
a deeper understanding of a database workload during optimization or problem resolution, try the
low-impact investigative technique of taking snapshots in intervals, with monitor resets between
each capture. For example, to monitor a one-hour period for a production workload, a script can be
easily constructed to do the logic of thread 1 and thread 2 as shown in the thread in Listing 1.
Thread 1 in Listing 1 is for taking multiple interval snapshots of 60 seconds each with a reset
monitor at the beginning of each interval snapshot.
Listing 1. Thread 1 pseudo code
db2 -v update monitor switches using bufferpool on sort on lock on statement on \
table on uow on
db2 -v get monitor switches
count = 0
while (count < 10)
sleep 300 ## not being monitored
db2 -v reset monitor all ## clear the counters
sleep 60 ## collect data for only 60-second interval
db2 -v get snapshot for all on <db_name> 2>&1 > snapshot_${count}_withReset.out
end while
db2 -v update monitor switches using bufferpool off sort off lock off \
statement off table off uow off
db2 -v terminate
Thread 2 in Listing 2 is for taking multiple snapshots over the period without issuing the reset
monitor command after the initial reset monitor command.
Listing 2. Thread 2 pseudo code
db2 -v update monitor switches using bufferpool on sort on lock on statement on table \
on uow on
db2 -v get monitor switches
count = 0
while (count < 10) loop
sleep 360 ## taking a snapshot every 6 minutes
db2 -v get snapshot for all on <db_name> 2>&1 > snapshot_${count}_NoReset.out
end while
db2 -v update monitor switches using bufferpool off sort off lock off \
statement off table off uow off
db2 -v terminate
When analyzing the snapshot data, there are certain metrics that require special attention. From
the database snapshot, note the counters in Listing 3 and monitor the ratio among these counters.
monitoring tools
Page 4 of 19
Buffer pool hit ratio is very important to monitor. It can be calculated based on the following
counters or reported with the latest new table functions.
Listing 3. Buffer pool hit ratio counters
Buffer pool data logical reads =
Buffer pool data physical reads =
Buffer pool index logical reads =
Buffer pool index physical reads =
Obviously, with availability of physical memory on the database server, a high buffer pool hit ratio
is always better for your database performance. The counters in Listing 4 can give you a good idea
of the SQL distribution from your workload. In particular, the ratio between select statement and
update/inserts/delete statements is useful to tell the split between data inquiries and updates.
Listing 4. SQL distribution
Commit statements attempted =
Rollback statements attempted =
Dynamic statements attempted =
Static statements attempted =
Failed statement operations =
Select SQL statements executed =
Xquery statements executed =
Update/Insert/Delete statements executed =
The counters in Listing 5 are important to tell on average, how many rows are deleted, inserted,
updated, selected, and read. The ratios among them are key characteristics of your workload.
Any dramatic changes in the ratios require some explanation and could indicate some potential
issues. In particular, the ratio between rows read and selected can tell if all SQL statements are
using indexes efficiently. If this ratio is extremely high, then it may cause performance issues. More
investigation is needed to isolate the SQL statement(s), which causes the high read/select ratio.
Consequently some optimization might be needed. For example, a better index might be provided.
Rows deleted =
Rows inserted =
Rows updated =
Rows selected =
Rows read =
From dynamic SQL snapshot data as the initial step based on the counters in Listing 6, you should
be able to list out the top SQL statements in terms of total cost, frequency, and cost per invocation.
If some of the top SQL statements are considered unexpectedly expensive, further action is
required to analyze the SQL access plan to see if anything can be improved.
Number of executions =
Total execution time (sec.microsec)=
Statement text =
monitoring tools
Page 5 of 19
From the table space snapshot data, the used page objects can be used to tell the total size of
the index, tables, and the total database size. It's always good to know the growth trend of each
object and the overall database to help you plan ahead for disk space requirements and database
tuning. This is where you can also tell which tables are the most active ones based on Rows Read
and Rows Written, as well as the size of each active table and their individual growth patterns, as
shown in Listing 7.
Table Schema =
Table Name =
Data Object Pages =
Index Object Pages =
Rows Read =
Rows Written =
Event monitors are used to collect information about the database and about any connected
applications when specified events occur. Events that the event monitor can capture include
database activity, such as connections, deadlocks, statements, or transactions. You can define an
event monitor by the type of event or events you want it to monitor. For example, a deadlock event
monitor can be used to monitor deadlocks. When a deadlock occurs, it collects information about
the applications involved, the SQL statements participating in the deadlock, and the locks causing
high contention.
One common usage of event monitor in the MDM Server environment is deadlock event
monitoring to understand and resolve excessive deadlocks that sometimes happen in an
MDM Server database. By default, all databases have an event monitor defined named
DB2DETAILDEADLOCK, which records detailed information about deadlock events. The
DB2DETAILDEADLOCK event monitor starts automatically when the database starts.
After formatting the event monitor files, typically you should be able to find all the SQL statements
participating in the deadlocks. By correlating the SQL statements back to the MDM Server
application layer, you should be able to understand the application flows leading to the deadlocks.
It could be due to the customized logic, a particular use case, or the data itself (quality and
distribution). Once you know the cause, make a decision accordingly to resolve the issue. Another
major factor that affects database performance is the input/output capacity. Similarly, closely
monitor the disks utilization where the database containers reside. If monitoring shows unbalanced
input/output usage across disks with a bottleneck associated with a set of hot disks, make a
database layout adjustment to maximize the input/output and to reduce the impact of the limiting
hot disks.
Database tuning
Most of the areas where data tuning happens correspond to the key areas mentioned earlier for
monitoring purposes. This section discusses specific tuning recommendations.
DB2 buffer pool tuning is one of the easiest tuning steps that potentially can give a quick
performance boost. With the buffer pool runtime statistics gathered from the monitoring tasks, you
monitoring tools
Page 6 of 19
can see whether any performance gain can be achieved from buffer pool tuning. In general, it's
desirable and recommended to have a high data hit ratio (80% or higher) and to have very high
index hit ratio (90% or higher).
For a large database, it's possible that increasing the buffer pool size can have minimal effect on
the buffer pool hit ratio. The number of data pages can be so large that the statistical chances of a
hit are not improved by increasing its size. Instead, you might find that tuning the index buffer pool
hit ratio achieves the desired result. This can be achieved using either of two methods:
Split the data and indices into two different buffer pools and tune them separately.
Use one buffer pool, but increase its size until the index hit ratio stops increasing.
The first method is often more effective, but because it requires indices and data to reside in
different table spaces, it may not be an option for existing databases. It also requires tuning two
buffer pools instead of one, which can mean slightly more work, particularly when memory is
constrained.
When multiple buffer pools are in use, the buffer pool snapshot data can be used to calculate the
buffer pool hit ratio of each pool to indicate the effectiveness of each buffer pool. If needed, reduce
the size of the buffer pool that cannot get help from the extra memory so that the available memory
can be allocated to other low-hit-ratio buffer pools that can get noticeable improvement from some
extra memory allocation.
Another important factor that affects SQL statement performance is database statistics. Having up-
to-date statistics is always necessary to ensure optimal SQL performance. Recent DB2 releases
support automatic runstats as part of the autonomic features. When enabled, automatic statistics
collection helps improve database performance by ensuring that you have up-to-date table
statistics. If this is not the desired option in a given environment, you can instead use DB2 runstats
or reorgchk commands explicitly in a scheduled maintenance window.
The DB2 runstats command updates statistics that profile the physical characteristics of a
database table, along with its associated indexes. Dynamic SQL statements automatically use
these updated statistics. Use runstats on a regular basis as part of regular database maintenance.
You can use runstats either by using a cron job or by manually using a script. Along with runstats,
the DB2 reorgchk command should also be run to confirm whether a reorganization of certain
tables or indexes is warranted.
Listing 8 shows commands that dynamically determine MDM Server database tables that runstats
can run on. A ksh file actually runs the runstats commands.
Listing 8. runstats commands
db2 connect to MDMDB user <uid> using <password>
db2 -x select tabname from syscat.tables where type='T' and tabschema='SCHEMA1' \
| sed -e "s/ *.$//" | sed -e "s/^/db2 -v runstats on table SCHEMA1./" \
| sed -e "s/$/ with distribution and detailed indexes all/" > .\my_runstats.ksh
The my_runstats.ksh file should contain lines similar to those shown in Listing 9.
monitoring tools
Page 7 of 19
Listing 9. runstats script
db2 -v runstats on table SCHEMA1.CONTACT with distribution and detailed indexes all
db2 -v runstats on table SCHEMA1.CONTRACT with distribution and detailed indexes all
db2 -v runstats on table SCHEMA1.ADDRESS with distribution and detailed indexes all
db2 -v runstats on table SCHEMA1.IDENTIFIER with distribution and detailed indexes all
db2 -v runstats on table SCHEMA1.ADDRESSGROUP with distribution and detailed indexes all
Each line is a runstats command on a table or index. By running the my_runstats.ksh program,
performance statistics on all tables of the MDMDB will be profiled. To shorten the duration for
updating statistics, it is recommended that you split the ksh file into several files so that you can
run multiple scripts concurrently to update different sets of tables in parallel to utilize the CPU
power on the DB2 server. To use the DB2 reorgchk command to check the statistics of all the
tables, and to determine if reorganization of certain tables or indexes is required, study the output
file that result from issuing the commands in Listing 10.
Listing 10. reorgchk commands
db2 -v connect to MDMDB user <uid> using <password>
db2 -v reorgchk current statistics on table all
In some cases, the reorgchk command can also be used to update the database statistics with
one simple command, as shown in Listing 11. However, it does not easily offer the flexibility of
running multiple concurrent scripts to update statistics against different sets of tables in the same
database.
Listing 11. reorgchk commands
db2 -v connect to MDMDB user <uid> using <password>
db2 -v reorgchk current statistics on table all ##old copy of the statistics
db2 -v reorgchk update statistics on table all ## update statistics
For slow transaction issues, the generic steps include:
Determining the specific MDM Server transaction
Running the particular transaction to further gather SQL statements involved
Confirming the most poorly performing SQL statements
The identified problematic SQL statements should be explained to show the access plan for any
needed, specific tuning. Tuning can include adding extra indexes to yield a more efficient access
plan, which makes tables volatile to encourage index usage and avoid table scans if possible. In
some cases, the out-of-the-box SQL should replaced by a customized SQL based on the specific
business requirements. Using the pluggable inquiry feature satisfies the performance requirement.
All of these actions are typical performance-tuning exercises completed during the performance-
testing phase of an MDM implementation project. The performance-testing phase is a critical part
of the MDM project to validate whether predefined performance goals are met with everything
specific to the implementation. Plan this phase with sufficient time and resources.
One of the most important steps to optimizing SQL statements is to explain the statements
of interest and to study their access plans. The first objective is to eliminate any costly table
monitoring tools
Page 8 of 19
scan operations. The next objective is to further reduce the SQL cost by optimizing the access
plan. The EXFMT format for the explain output file is recommended, because it provides the
needed information to help such SQL optimization. If the OOTB SQL cannot provide the needed
performance because of the combination of business customization and data, you can rewrite the
SQL to meet your specific business needs using the pluggable inquiry feature in MDM Server. See
Resources for tips on how to write customized SQL statements for optimal performance.
A few deadlocks once in a while typically are not concerning. However, it can be a problem when
the frequency or percentage of deadlocks is high. See Resources for a Technote about applying
some lock-avoidance registry settings in DB2. Applying the registry variables or performing
other database tuning that leads to overall better SQL performance will help reduce or eliminate
deadlocks.
Note that for DB2 for z/OS, it's not always acceptable to apply the lock-avoidance settings,
because the settings affect all the applications that share the same DB2 system.
For more DB2 tuning best practices, see Resources.
Optimizing the default isolation level in JDBC
Out of the box, MDM Version 8.0 and later defaults to the JDBC isolation level of
TRANSACTION_REPEATABLE_READ or, in DB2 terms: read stability (RS). While this isolation
level is fine from a functional standpoint, lock contention in an actively used database can be
reduced by using the TRANSACTION_READ_COMMITTED isolation level, which is also known
as cursor stability (CS). If you are migrating to MDM, this point is especially important, because
this step was not previously required. MDM defaults to the more optimistic isolation level of CS.
One simple and effective way to change the default isolation level in MDM to the more optimistic
CS is to add a custom directive on the jdbc/DWLCustomer data source in WebSphere Application
Server. Complete the following steps in the WebSphere Application Server administrative console.
1. Click Resources > JDBC provider.
2. Click Configured_JDBC_Provider > DataSource.
3. Click the name of the data source (DWLCustomer) for which you want to customize the
default isolation level.
4. Under Additional properties, click Custom properties > New.
5. Specify webSphereDefaultIsolationLevel in the Name field, and a value of 2 in the Value
field. The default value is 4. See Table 1 for isolation levels.
6. Click OK, and save the configuration.
7. Synchronize your changes across nodes.
8. Stop and restart the application server.
Table 1. Isolation levels
Possible values JDBC isolation level DB2 isolation level
8 TRANSACTION_SERIALIZABLE Repeatable read (RR)
4 TRANSACTION_REPEATABLE_READ Read stability (RS)
monitoring tools
Page 9 of 19
2 TRANSACTION_READ_COMMITTED Cursor stability (CS)
1 TRANSACTION_READ_UNCOMMITTED Uncommitted read (UR)
Exploring recommended monitoring tools
MDM Server supports multiple operating system platforms, databases, and application servers.
This section describes performance monitoring tools that work on a subset of these platforms.
Used effectively, these tools will help you narrow down performance issues and help pinpoint root
causes.
UNIX CPU performance monitoring tools
Listings 12 and 13 show CPU utilization using different tools for UNIX systems.
Listing 12. AIX CPU utilization
# sar -u 5 100
AIX othello 1 5 000C30AD4C00 06/17/05
12:38:09 %usr %sys %wio %idle
12:38:24 12 1 1 85
12:38:39 26 2 0 72
12:38:54 9 2 1 89
Listing 13. Solaris CPU utilization
# sar 5 100
SunOS cage 5.9 Generic_118558-05 sun4u 07/12/2005
15:31:07 %usr %sys %wio %idle
15:31:12 0 0 0 100
15:31:17 1 0 0 99
15:31:22 0 0 0 100
15:31:27 0 0 0 100
The mpstat command is another tool to monitor CPU utilization on Solaris. The mpstat command
reports CPU utilization on a per-CPU basis, and it reports extended statistics, including context
switches, interrupts, and so on. The mpstat command is shown in Listing 14.
Listing 14. mpstat command
mpstat 5 5 will take 5 samples every 5 seconds.
mpstat 5 100
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
2 14 0 26 5 1 60 2 3 9 0 170 1 1 0 99
3 15 0 22 28 24 62 1 2 10 0 142 1 1 0 99
34 19 0 24 193 14 65 1 2 11 0 153 1 1 0 99
35 17 0 26 5 1 67 2 3 11 0 152 1 1 0 99
66 17 0 25 89 1 69 2 2 10 0 154 1 1 0 99
67 15 0 23 5 1 68 2 2 10 0 149 1 1 0 99
98 14 0 23 5 1 66 2 2 10 0 148 1 1 0 99
99 14 0 23 5 1 101 2 2 10 0 149 1 0 0 99
130 14 0 26 5 1 66 2 2 11 0 137 1 0 0 99
131 14 0 22 5 1 67 2 2 11 0 140 1 0 0 99
162 15 0 22 5 1 67 2 2 10 0 144 1 0 0 99
monitoring tools
Page 10 of 19
UNIX input/output performance monitoring tools
Listings 15 and 16 are sample outputs of the iostat command on AIX and Solaris. The reports are
different for each operating system. On AIX, the iostat command with the -a option gives input/
output statistics on a per adapter basis, which can be very useful.
Listing 15. AIX iostat output
AIX
# iostat -a
System configuration: lcpu=4 disk=12
tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 0.0 0.2 0.5 99.2 0.1
Adapter: Kbps tps Kb_read Kb_wrtn
scsi0 5.2 1.2 196495 2154064
Paths/Disk: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk1_Path0 0.0 0.0 0.0 0 0
hdisk2_Path0 0.0 0.0 0.0 7397 4
hdisk3_Path0 0.0 0.0 0.0 2199 28
hdisk0_Path0 0.7 5.2 1.2 186899 2154032
Adapter: Kbps tps Kb_read Kb_wrtn
fcs0 11.8 1.3 110301 5166084
Paths/Disk: % tm_act Kbps tps Kb_read Kb_wrtn
vpath1_hdisk17 0.0 0.7 0.2 22729 282512
vpath1_hdisk13 0.0 0.7 0.2 17239 281396
vpath0_hdisk16 0.0 2.3 0.2 7217 1006120
vpath0_hdisk12 0.0 2.3 0.2 8758 1010056
vpath3_hdisk19 0.0 2.3 0.2 22024 1006064
vpath2_hdisk14 0.0 0.6 0.2 8980 281792
vpath3_hdisk15 0.0 2.3 0.2 14352 1016480
vpath2_hdisk18 0.0 0.6 0.2 9002 281664
The Solaris iostat command has many options that will help monitoring input/output devices.
Listing 16 shows the -x option, which gives extended statistics but there are many more arguments
available.
Listing 16. Solaris iostat output
Solaris
# iostat -x
extended device statistics
device r/s w/s kr/s kw/s wait actv svc_t %w %b
sd15 0.0 0.0 0.0 0.0 0.0 0.0 2.4 0 0
sd16 0.0 0.0 0.0 0.0 0.0 0.0 2.4 0 0
sd17 3.7 0.6 9.2 5.1 0.0 0.0 1.3 0 0
sd45 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
sd46 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
sd47 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
sd75 0.0 0.0 0.0 0.0 0.0 0.0 1.4 0 0
sd76 0.0 0.0 0.0 0.0 0.0 0.0 1.6 0 0
sd105 0.0 0.0 0.0 0.0 0.0 0.0 1.3 0 0
sd106 0.0 0.0 0.0 0.0 0.0 0.0 1.3 0 0
sd107 0.0 0.0 0.0 0.0 0.0 0.0 1.3 0 0
sd136 0.0 3.1 0.0 12.6 0.0 0.0 0.6 0 0
sd137 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0 0
sd139 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
sd140 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
monitoring tools
Page 11 of 19
sd2315 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0 0
sd2316 0.0 0.0 2.9 0.0 0.0 0.0 13.8 0 0
sd2317 0.3 3.7 2.1 26.5 0.0 0.0 1.4 0 0
sd2442 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0 0
sd2443 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0 0
UNIX network performance monitoring tools
Listings 17 and 18 are sample outputs of the netstat command on AIX and Solaris.
Listing 17. AIX netstat output
AIX (snapshot every 5 seconds on the en0 adapter)
# netstat -I en0 5
input (en0) output input (Total) output
packets errs packets errs colls packets errs packets errs colls
260984 0 128140 0 0 514023 0 405638 0 0
342 0 331 0 0 342 0 331 0 0
837 0 828 0 0 837 0 828 0 0
671 0 670 0 0 673 0 672 0 0
Listing 18. Solaris netstat output
Solaris (snapshot every 5 seconds on the ce8 adapter)
# netstat -I ce8 5
input ce8 output input (Total) output
packets errs packets errs colls packets errs packets errs colls
39248977 0 39931746 0 0 39739399 0 40422168 0 0
7 0 0 0 0 7 0 0 0 0
6 0 0 0 0 6 0 0 0 0
7 0 31 0 0 7 0 31 0 0
8 0 0 0 0 8 0 0 0 0
9 0 0 0 0 9 0 0 0 0
5 0 0 0 0 5 0 0 0 0
UNIX memory usage monitoring tools
Listings 19 and 20 are sample outputs of the vmstat command on AIX and Solaris.
Listing 19. AIX memory usage
(snapshot every 5 seconds)
# vmstat 5
System Configuration: lcpu=4 mem=14336MB
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
1 1 772314 2075422 0 0 0 0 0 0 575 810 242 0 1 99 0
0 0 772156 2075580 0 0 0 0 0 0 579 579 214 0 1 99 0
0 0 772414 2075321 0 0 0 0 0 0 571 1034 198 0 0 99 0
0 0 772006 2075730 0 0 0 0 0 0 560 1060 163 0 0 99 0
0 0 772006 2075730 0 0 0 0 0 0 562 521 165 0 1 99 0
Listing 20. Solaris memory usage
(snapshot every 5 seconds)
# vmstat 5
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s1 s1 s1 s4 in sy cs us sy id
0 0 0 39372952 28946544 127 272 9 2 2 0 0 0 0 4 0 803 2498 1152 1 1 99
0 0 0 43181064 29508744 0 1 0 0 0 0 0 0 0 0 0 384 559 328 0 0 100
0 0 0 43179664 29508112 232 536 16 0 0 0 0 0 0 8 0 616 2286 361 1 1 99
0 0 0 43181064 29508744 0 0 0 0 0 0 0 0 0 0 0 379 774 326 0 0 100
0 0 0 43179648 29508096 232 536 16 0 0 0 0 0 0 8 0 618 2304 366 1 1 99
monitoring tools
Page 12 of 19
Automating resource monitoring on UNIX using nmon
It's useful to automate the resource utilization collection during a performance test or in a
production environment. IBM offers a free tool called nmon that enables you to do just that. The
nmon tool runs on the AIX and Linux platforms. The tool is designed to collect and analyze (with
the nmon analyzer Excel spreadsheet) AIX performance data, including the following:
CPU utilization
Memory use
Kernel statistics
Run queue information
Disks input/output rates
Transfers
Read/write ratios
Free space on file systems
Disk adapters
Network input/output rates
Paging space
Paging rates
CPU and AIX specification
Top processors
IBM HTTP web cache
User-defined disk groups
Machine details and resources
Asynchronous input/output
Workload manager
ESS disks
NFS
Dynamic LPAR changes
To start the nmon process and log entries to a file, enter nmon -f -s 15 -c 120. This command logs
performance statistics every 15 seconds for 120 intervals, or (15*120)/60 = 30 minutes. The log
is in a file called <hostname>_date_time.nmon in the running directory. You might want to run this
with the nohup option if you need to log out of the system.
When the tool finishes, run the sort command sort -A mymachine_311201_1030.nmon >
res_usage_test1.csv to sort and generate a comma-delimited file for the Excel nmon analyzer to
use. Open the Excel spreadsheet called nmon analyzer v301.xls, and load the .csv file, as shown
in Figure 1.
monitoring tools
Page 13 of 19
Figure 1. Excel nmon processing tool
After analysis, save the output file as a regular Excel file. In the output file, resource utilization is
listed under each tab. The first sheet is the summary sheet. Figure 2 shows an example of the
generated output.
Figure 2. Sample of processed nmon output
(View a larger version of Figure 2.)
WebSphere JVM monitoring tools: IBM Pattern Modeling and Analysis Tool for
Java Garbage Collector
IBM Pattern Modeling and Analysis Tool for Java Garbage Collector is a tool that parses verbose
GC trace, analyzes Java heap usage, and recommends key configurations based on pattern
modeling of Java heap usage. See Resources for more details. Figure 3 shows a sample
screenshot of the tool in action.
monitoring tools
Page 14 of 19
Figure 3. IBM Java Pattern and Analysis Tool for GC
WebSphere JVM HeapAnalyzer
HeapAnalyzer is a graphical tool to discover possible Java heap leaks. See Resources for more
details. Figure 4 shows a screenshot of the tool.
Figure 4. IBM HeapAnalyzer
monitoring tools
Page 15 of 19
MDM Server built-in traces
MDM Server enables logging at different levels. You can also turn on tracing for a particular
component (functional wise). To troubleshoot performance issues, one of the most-used trace
facilities is the PerformanceTracking parameter. Complete the following steps to use the
PerformanceTracking parameter.
1. Enable performance monitor from the configelement table, as shown in Listing 21.
Listing 21. Enabling performance monitor
db2 -v update configelement set value='-1' where name in \
('/IBM/DWLCommonServices/PerformanceTracking/level')
db2 -v update configelement set value='true' where name like \
'/IBM/DWLCommonServices/PerformanceTracking/%enabled'
2. Configure log4J in properties.jar for the performancemonitor.log, as shown in Listing 22.
Listing 22. Configuring log4J
log4j.appender.performanceLog=org.apache.log4j.RollingFileAppender
log4j.appender.performanceLog.Encoding=UTF-8
log4j.appender.performanceLog.Threshold=ALL
log4j.appender.performanceLog.layout.ConversionPattern=%d : %m%n
log4j.appender.performanceLog.layout=org.apache.log4j.PatternLayout
log4j.appender.performanceLog.File=../../../logs/MDM80IT6/performancemonitor.log
log4j.logger.com.dwl.base.performance.internal.PerformanceMonitorLog=INFO,\
performanceLog
log4j.appender.performanceLog.MaxBackupIndex=2
log4j.appender.performanceLog.MaxFileSize=100MB
3. Restart WebSphere Application Server.
4. Run the warmup of the transactions.
5. Clean the performance monitor files, by entering code such as bash-3.00# cat /dev/null >
performancemonitor.log.
6. Run the transaction being debugged several times (usually 100 times).
7. Stop the test.
8. Verify that the file starts with code as shown in Listing 23.
Listing 23. Success message for file
"21194878586562 processTx : processTx(String,Object) @ DWLRequestHandler : :\
1961194878586562 : 24785000 : : SUCCESS"
Conclusion
IBM InfoSphere Master Data Management Server helps companies gain control over business
information by enabling them to manage and maintain a complete and accurate view of their
master data. InfoSphere MDM Server enables companies to extract maximum value from master
data by centralizing multiple data domains and providing a comprehensive set of pre-built business
services that support a full range of master data management functionality. Highly flexible in
monitoring tools
Page 16 of 19
nature, MDM Server offers a wide range of customization capabilities that enable companies to
develop their own services that meet their customers' demands. Due to the highly customizable
nature, you need to understand the impact that each tier has on overall performance. This article
series contains conceptual and practical information to help you understand InfoSphere MDM
Server and the ways that you can monitor and tune it to meet your performance goals and
business needs. It provides valuable reference information that you can refer to when you develop,
analyze, and troubleshoot performance issues in your MDM system.
Acknowledgments
We would like to thank David Borean, Lena Woolf, Bill Xu, Steve Reese, and Berni Schiefer for
their input and suggestions for this article series.
monitoring tools
Page 17 of 19
Resources
Learn
Go to Part 1 of the series for details about the goals and how to begin your approach to
optimization.
Explore DB2 performance tuning scenarios (developerWorks May 2008).
Refer to the IBM RedbookNew Mainframe: z/OS Basics for an introduction to z/OS basics.
Continue tuning and monitoring with Best practices: Tuning and monitoring database system
performance (developerWorks, various dates).
Get the IBM Pattern Modeling and Analysis Tool for Java Garbage Collector for a tool that
parses verbose GC trace, analyzes Java heap usage, and recommends key configurations
based on pattern modeling of Java heap usage.
See HeapAnalyzer for a graphical tool to discover possible Java heap leaks.
Review the Redbook WebSphere Application Server V6 Scalability and Performance
Handbook for more details.
Check out DB2 UDB for z/OS Version 8 Performance Topics for more about performance.
Browse the MDM Server Infocenter for background information.
Read the IBM Redpaper WebSphere Customer Center: Understanding Performance for more
details.
Learn to load a large volume of Master Data Management data quickly using RDP MDM
Server maintenance services batch (developerWorks Aug 2009) .
Look at MDM Server Support Technote: MDM Server Performance Must-Gather information.
Walk through DB2 tuning tips for OLTP applications (developerWorks Jan 2007) for more
tips.
Learn more about Information Management at the developerWorks Information Management
zone. Find technical documentation, how-to articles, education, downloads, product
information, and more.
Stay current with developerWorks technical events and webcasts.
Follow developerWorks on Twitter.
Get products and technologies
Build your next development project with IBM trial software, available for download directly
from developerWorks.
Discuss
Participate in the discussion forum for this content.
Check out the developerWorks blogs and get involved in the developerWorks community.
monitoring tools
Page 18 of 19
About the authors
Jesse Chen
Jesse F Chen is a senior software engineer in the Information Management
group with a focus on software performance and scalability. He has 12 years of IT
experience in software development and services. His recent projects include FileNet
and DB2 integration and optimization, Master Data Management, and WebSphere
Product Center. Jesse is also a frequent contributor and speaker at multiple ECM
conferences on topics of performance, scalability, and capacity planning. He earned
a Bachelor's degree in Electrical Engineering and Computer Science from the
University of California, at Berkeley, and a M.S. degree in Management in Information
and Systems from New York University.
Yongli An
Yongli An is an experienced performance engineer focusing on Master Data
Management products and solutions. He is also experienced in DB2 database server
and WebSphere performance tuning and benchmarking. He is an IBM Certified
Application Developer and Database Administrator - DB2 for Linux, UNIX, and
Windows. He joined IBM in 1998. He holds a bachelor degree in Computer Science
and Engineering and a Masters degree in Computer Science. Currently Yongli is the
manager of the MDM performance and benchmarks team, focusing on Master Data
Management Server performance and benchmarks, and helping customers achieve
optimal performance for their MDM systems.
Adam Muise
Adam Muise is a Senior Consultant for IBM InfoSphere products in the Information
Management Technology Ecosystem. Here he enables IBM Business Partners to
sell and implement InfoSphere software, with a focus on Master Data Management
solutions. In his previous role at IBM, he was a founding member of the MDM
Performance and Benchmarks team. Before joining IBM in 2006, Adam worked as a
Senior Consultant for Performance and High Availability at Hewlett Packard and as a
Performance Specialist at CIBC. Adam holds an honors degree from York University
in Information Technology and Digital Humanities.
Copyright IBM Corporation 2010
(www.ibm.com/legal/copytrade.shtml)
Trademarks
(www.ibm.com/developerworks/ibm/trademarks/)
monitoring tools
Page 19 of 19

DM Perf Tuning2 PDF

Uploaded by

Copyright:

Available Formats

DM Perf Tuning2 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

DM Perf Tuning2 PDF

Uploaded by

Copyright:

Available Formats

What are some recommendations for monitoring the DB2 database in an MDM system?

What tools can be used to monitor DB2 performance?

Copyright IBM Corporation 2010 Trademarks

Monitoring and tuning InfoSphere Master Data Management

You might also like

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.