IBM Tivoli Monitoring V6.2.

3
Troubleshooting guidelines in the missing or incorrect
historical data scenario

2012 IBM Corporation

This presentation has steps for troubleshooting the missing or incorrect historical data
scenario. In this scenario, the Tivoli Monitoring product cannot correctly manage shortterm historical data with unexpected outcomes on the Tivoli Enterprise Portal interface.

troubleshooting_missing_or_incorrect_historical_data.ppt

Page 1 of 18

Assumptions

Before you proceed, the module designer assumes that you have these skills and

knowledge:

Tivoli Monitoring general architecture

The troubleshooting approach that is based on the RAS1 logs
Completion of the module, IBM Tivoli Monitoring V6.2.3, Historical data collection data
flow, which describes how to configure the historical data collection

Troubleshooting guidelines in the missing or incorrect historical data scenario

2012 IBM Corporation

To complete this module, you need to know the general architecture of the IBM Tivoli
Monitoring products and be familiar with the troubleshooting approach based on the RAS1
tracing and logging.

troubleshooting_missing_or_incorrect_historical_data.ppt

Page 2 of 18

Objectives

When you complete this module, you can troubleshoot issues with the historical data
collection

Note: This module does not cover the issues that are related to warehouse management

Troubleshooting guidelines in the missing or incorrect historical data scenario

2012 IBM Corporation

When you complete this module, you can troubleshoot the scenario where the monitoring
product cannot collect and store historical data and has unexpected outcomes in the Tivoli
Enterprise Portal interface.

troubleshooting_missing_or_incorrect_historical_data.ppt

Page 3 of 18

Product architecture

Troubleshooting guidelines in the missing or incorrect historical data scenario

2012 IBM Corporation

You might know the Tivoli Monitoring architecture and its components. The core of the
architecture is the Tivoli Enterprise Monitoring Server component, which can be HUB, or
optionally (for scalability reasons in large environments), remote. The Tivoli Enterprise
Management Server manages the communication with the agents and owns the tables
where the performance data (coming from the agents) are stored. It is also responsible for
the situation and policy management.
Tivoli Enterprise Portal Server is the presentation component of the collected data to the
user. Its configuration data (that is, workspaces definitions, user permissions) are stored in
a database that can be based on DB2, Oracle, or SQLSERVER. The agents that support
most of the known platforms are for the data collection.
There is also a component that is devoted to the historical data collection. It includes a
warehouse proxy agent and a summarization and pruning agent.

troubleshooting_missing_or_incorrect_historical_data.ppt

Page 4 of 18

Configuration files and their locations

AGENTS:
On Windows:

<install path>\TMAITM6\logs\<hostname>_pc_<hextimestamp>-nn
<install path>\TMAITM6\logs\/<hostname>.LGx (x=1,2) (Operation Log)
On UNIX or Linux:<install path>/logs/<hostname>_pc_<hextimestamp>-nn
<install path>/logs/<hostname>.LGx (x=1,2) (Operation Log)
On z/OS:
Rklvlog/Rkpdlog

Tivoli Enterprise Portal Server:

On Windows:
<install path>\CNPS\KFWENV
On UNIX or Linux: <install path>/config/cq.ini
Tivoli Enterprise Monitoring Server:
On Windows: <install path>\CMS\KBBENV
On UNIX or Linux: <install path>/config/<tems name>_ms_<tems name>.config
On z/OS:
hlq.mlq.RKANPAR(KDSENV)
Tivoli Enterprise Managed Agent (TEMA):
On Windows:
<install path>\TMAITM6\<pc>env
On UNIX or Linux: <install path>/config/<pc>.ini
On z/OS:
hlq.mlq.RKANPAR(K<pc>ENV)
5

Troubleshooting guidelines in the missing or incorrect historical data scenario

2012 IBM Corporation

This slide shows the physical location of the Tivoli Monitoring configuration files on
supported OS platforms.
Details are provided for the three main components: Tivoli Enterprise Portal Server, Tivoli
Enterprise Monitoring Server, and the agent.

troubleshooting_missing_or_incorrect_historical_data.ppt

Page 5 of 18

RAS1 log files and their locations

AGENTS:
On Windows:
<install path>\TMAITM6\logs\<hostname>_pc_<hextimestamp>-nn
On UNIX or Linux: <install path>/logs/<hostname>_pc_<hextimestamp>-nn
On zOS:
Rklvlog/Rkpdlog
Tivoli Enterprise Monitoring Server and Tivoli Enterprise Portal Server:
On Windows:
<install path>\logs\<hostname>_pc_<hextimestamp>-nn
On UNIX or Linux: <install path>/logs/<hostname>_pc_<hextimestamp>-nn
On zOS:
Rklvlog/Rkpdlog (Tivoli Enterprise Monitoring Server only)
Tivoli Enterprise Portal Browser:
On Windows:
C:\Documents and Settings\Administrator\ApplicationData\IBM\Java\
Deployment\log\Plugin142.trace
On UNIX or Linux: None
Tivoli Enterprise Portal Desktop:
On Windows:
<install path>\cnp\logs\kcjras1.log
On UNIX or Linux: <installpath>/logs/kcjras1.log
6

Troubleshooting guidelines in the missing or incorrect historical data scenario

2012 IBM Corporation

This slide shows the physical location of the RAS1 log files on supported OS platforms.
Details are provided for the three main components: Tivoli Enterprise Portal Server, Tivoli
Enterprise Monitoring Server, and the agent.

troubleshooting_missing_or_incorrect_historical_data.ppt

Page 6 of 18

Troubleshooting strategy for the error scenario

The history data collection allows you to collect and store data in short-term history files (on
Tivoli Enterprise Monitoring Server or on agent)
You can see this data on specific views of the Tivoli Enterprise Portal console
The data that is collected in the first 24 hours is stored in short-term history files
The older data is recorded in the Data Warehouse component (not in the scope of this
module)
For issues with historical data collection, perform these checks:
The Historical data collection is configured and started successfully (stage 1)
The Historical data are produced and available for further processing (stage 2)

Troubleshooting guidelines in the missing or incorrect historical data scenario

2012 IBM Corporation

The history data collection feature allows you to collect and store data in short-term history
files on the Tivoli Enterprise Monitoring Server or on the agent. Then, you can retrieve
such data and view it in a specific view of the Tivoli Enterprise Portal interface. The data
that is collected in the first 24 hours is stored in short-term history files; data older than 24
hours are recorded in the Data Warehouse component. This second component is not in
the scope of this module.
When you encounter issues with the historical data collection, you must check to see
whether the historical data collection is properly configured and started, which is stage 1.
Then, you ensure that the historical data is collected and stored so that you can retrieve
and use it, which is stage 2.

troubleshooting_missing_or_incorrect_historical_data.ppt

Page 7 of 18

Trace settings

Tivoli Enterprise Portal Server:

KBB_RAS1=ERROR (UNIT:ctsql IN,ER)

Tivoli Enterprise Monitoring Server:

KBB_RAS1=ERROR (UNIT:kpxreqds ALL) (UNIT:kpxrpcrq ALL)
(UNIT:kfasphst ALL) (UNIT: kpxhsloc ALL) (UNIT:krabhsco ALL)
AGENT:
KBB_RAS1=ERROR (UNIT:kra ALL)

Troubleshooting guidelines in the missing or incorrect historical data scenario

2012 IBM Corporation

The error scenario involves all of the three main components: the Tivoli Enterprise Portal
Server, which places the data request; the Tivoli Enterprise Management Server, which
manages and drives the request; and the agent, which performs the data collection. Each
of these components must be instrumented with specific traces, as shown in this slide.
The traces that are activated on Tivoli Enterprise Management Server allow you to track
the interactions between the Tivoli Enterprise Management Server and the persistent data
files.

troubleshooting_missing_or_incorrect_historical_data.ppt

Page 8 of 18

Step 1: Data collection request completes

Tivoli Enterprise Monitoring Server: Uadvisor situation started correctly

1090922163004125KRAIRA000 Starting UADVISOR_KNT_WTLOGCLDSK

<1179648077,2263877552> for KNT.WTLOGCLDSK.

Tivoli Enterprise Monitoring Server and TEMA establish a connection

(2009.266 01.30.04-145C:kpxreqds.cpp,474,"Update") Request

<1179648077> to node Primary:LAB232157:NT now has status 1

Tivoli Enterprise Monitoring Server requests to start collection

(2009.266 01.30.04-1C94:kpxrpcrq.cpp,365,"PrintSelf") RPC

request <1179648077> to node Primary:LAB232157:NT address

ip:#9.168.114.50[38213]

(2009.266 01.30.04-1C94:kpxrpcrq.cpp,173,"requestStart")

StartAgent instruction UTF8: 0, len:

31[HIST()LSTDATE(1090922163004000)]

(2009.266 01.30.04-1C94:kpxreqds.cpp,474,"Update") Request

<1179648077> to node Primary:LAB232157:NT now has status 3

(2009.266 01.30.04-1C94:kpxrpcrq.cpp,269,"requestStart") RPC

StartAgent completed

Troubleshooting guidelines in the missing or incorrect historical data scenario

2012 IBM Corporation

During step 1, the data collection request is started, and after processing, comes to a
positive end. The operational logs (.lg0/lg1) are useful to verify that the configured
historical collections started. Then, the historical collection is started for an attribute group,
and a Uadvisor situation on the attribute group is started.
This log is useful because you can see which historical collections are active. There is a
data collection behind each historical data collection start, like the online collection. Tivoli
Enterprise Management Server establishes a connection to the agent. After the
connection to the agent completes, the collection on the connected agent can begin.

troubleshooting_missing_or_incorrect_historical_data.ppt

Page 9 of 18

Step 2: Historical data is collected at agent side

TEMA: Collection that is associated to Uadvisor situation is started
(2009.266 01.30.04-2110:kraafmgr.cpp,642,"Start") Start received

UADVISOR_KNT_WTLOGCLDSK <1179648077,0> on *.WTLOGCLDSK

(2009.266 01.30.04-2110:kraafmgr.cpp,646,"Start") Value of

instruction <HIST()LSTDATE(1090922163004000)>

(2009.266 01.30.04-2110:kraafmgr.cpp,674,"Start") Start complete

UADVISOR_KNT_WTLOGCLDSK <1179648077,2263877552> on *.WTLOGCLDSK,

status = 0

TEMA: Collection historical data are written on persistent data store file
(2009.266 01.35.04-242C:kraahbin.cpp,204,"WriteMetaFile") Metafile

name is <D:\ibm\ITM\TMAITM6\logs\WTLOGCLDSK.hdr>

(2009.266 01.35.04-242C:kraahbin.cpp,279,"WriteMetaFile") Exit

(2009.266 01.35.04-242C:kraahbin.cpp,308,"WriteRow") _buffer_size:

384. Application data size: 356

(2009.266 01.35.04-242C:kraahbin.cpp,349,"WriteRow") Using timestamp

of 1090922163504

(2009.266 01.35.04-242C:kraahbin.cpp,356,"WriteRow") *INFO: History

Record TimeZone Offset +25200

(2009.266 01.35.04-242C:kraahbin.cpp,459,"WriteRow") Wrote 4 rows

history data, UADVISOR_KNT_WTLOGCLDSK KNT.WTLOGCLDSK,

<1179648077,2263877552>.

10

Troubleshooting guidelines in the missing or incorrect historical data scenario

2012 IBM Corporation

During this step, the collection that is associated to the Uadvisor situation is started on the
agent side. After the collection on the agent is completed and the data is available, the
next step is to write it on a physical file to keep it permanently. This action is done by
following the attribute group metafile that drives the historical data store task. As a result,
some rows (in this case, four rows) are written on the short history file. The Wrote n Rows
string is the key point to look at to ensure that your historical collection is completed and
produced some data. In this specific case, four rows of data are stored on file
WTLOGCLDSK.

troubleshooting_missing_or_incorrect_historical_data.ppt

Page 10 of 18

Step 3: Tivoli Enterprise Portal Server issues the historical request

Tivoli Enterprise Portal Server: SQL query is created

(2009.266 02.04.16
1F50:ctsqlstatement.cpp,212,"SQLStatement::SQLStatement")

HUB_LAB232157(60): SELECT WTLOGCLDSK.WRITETIME,

WTLOGCLDSK.ORIGINNODE, WTLOGCLDSK.INSTCNAME,

....................................................................

WTLOGCLDSK.DSKPCTFRSP, WTLOGCLDSK.TIMESTAMP FROM KNT.WTLOGCLDSK AT(

'HUB_LAB232157' ) HISTORY() WHERE ( (WTLOGCLDSK.WRITETIME >= ?) AND

(WTLOGCLDSK.WRITETIME <= ?) ) AND ( (WTLOGCLDSK.ORIGINNODE = ?) AND

SYSTEM.PARMA("TIMEOUT","600",3) )

Tivoli Enterprise Portal Server: SQL query variables are resolved with values
(2009.266 02.04.16-1F50:ctsqlstatement.cpp,237,"SQLStatement::SQL

Statement") HUB_LAB232157(60): Values: '1090922160417000'

'1090922170417000' 'Primary:LAB232157:NT'

11

Troubleshooting guidelines in the missing or incorrect historical data scenario

2012 IBM Corporation

In this step, you issue a request for retrieving a set of historical data. The mechanism is
similar for the real-time data. First, an SQL query gets created at the Tivoli Enterprise
Portal Server level. The HISTORY () clause is the relevant part, which helps distinguish a
query for historical data from a query for online data, where the HISTORY() clause is not
present. Then, this SQL query is resolved with values to determine the interval range of
interest for the data collection.

troubleshooting_missing_or_incorrect_historical_data.ppt

Page 11 of 18

Step 4: Tivoli Enterprise Monitoring Server starts history read data

request

Tivoli Enterprise Monitoring Server manages the history read request (rpc call)
(2009.266 02.04.16-2E6C:kpxreqds.cpp,474,"Update") Request

<1873809319> to node Primary:LAB232157:NT now has status 1

(2009.266 02.04.16-2C44:kpxrpcrq.cpp,365,"PrintSelf") RPC request

<1873809319> to node Primary:LAB232157:NT address

ip.pipe:#9.168.114.50[10111]

(2009.266 02.04.16-2C44:kpxrpcrq.cpp,173,"requestStart")

StartAgent instruction UTF8: 0, len:

220[HISTREAD(TABLE(WTLOGCLDSK)APPL(KNT)COLS(TIMESTAMP:164,DSKPCTF

RSP:160,DSKQUELEN:156,PCTDSKTIME:152,PCTDSKWRTM:148,PCTDSKRDTM:14

4,TOTALSIZE:140,PCFREE:136,PCUSED:132,FREEMGBTES:128,INSTCNAME:64

,ORIGINNODE:0,WRITETIME:180))]

(2009.266 02.04.16-2C44:kpxreqds.cpp,474,"Update") Request

<1873809319> to node Primary:LAB232157:NT now has status 3

(2009.266 02.04.16-2C44:kpxrpcrq.cpp,269,"requestStart") RPC

StartAgent completed.

12

Troubleshooting guidelines in the missing or incorrect historical data scenario

2012 IBM Corporation

In this part of the flow, the historical data request gets issued as an RPC call from Tivoli
Enterprise Management Server to the target agent. As in a normal request of real-time
data, you can track the current state by looking at the status values. The HISTREAD
clause means that the table of data of interest is read from a file.

troubleshooting_missing_or_incorrect_historical_data.ppt

Page 12 of 18

Step 5: The agent processes the request

TEMA: Request of history data read is received
(2009.266 02.04.16-2F1C:kraafmgr.cpp,642,"Start") Start received

<1873809319,0> on *.READHIST

(2009.266 02.04.16-2F1C:kraafmgr.cpp,646,"Start") Value of

instruction

<HISTREAD(TABLE(WTLOGCLDSK)APPL(KNT)COLS(TIMESTAMP:164,DSKPCTFRSP

:160,DSKQUELEN:156,PCTDSKTIME:152,PCTDSKWRTM:148,PCTDSKRDTM:144,T

OTALSIZE:140,PCFREE:136,PCUSED:132,FREEMGBTES:128,INSTCNAME:64,OR

IGINNODE:0,WRITETIME:180))>

TEMA: Source file and metafile are located

(2009.266 02.04.16-2F1C:krarhagt.cpp,195,"SetFilenames") Setting

Datafile <D:\ibm\ITM\TMAITM6\logs\WTLOGCLDSK>,

Metafile <D:\ibm\ITM\TMAITM6\logs\WTLOGCLDSK.hdr>

(2009.266 02.04.16-2F1C:kraafmgr.cpp,674,"Start") Start complete

<1873809319,1229981629> on *.READHIST, status = 0

(2009.266 02.04.16-1B04:krabhsco.cpp,629,"KRA_OpenHistoryFiles")

Opened source file D:\ibm\ITM\TMAITM6\logs\WTLOGCLDSK

TEMA: Data is returned

(2009.266 02.04.16-1B04:kraadspt.cpp,254,"sendDataToProxy")

Sending 20 rows for KPX.READHIST, <1873809319,1229981629>.

13

Troubleshooting guidelines in the missing or incorrect historical data scenario

2012 IBM Corporation

The request for data-read of the historical data gets to the agent side. The agent locates
the header file (with extension hdr) containing the data definitions of the files that it must
read. Then, it opens the source file where the historical data is physically located. Finally,
it returns it as several rows to Tivoli Enterprise Monitoring Server.

troubleshooting_missing_or_incorrect_historical_data.ppt

Page 13 of 18

Step 6: Tivoli Enterprise Monitoring Server forwards the data to Tivoli

Enterprise Portal Server

Tivoli Enterprise Monitoring Server receives the data

(2009.266 02.04.16-20C4:kpxrpcrq.cpp,743,"IRA_NCS_Sample") Rcvd 20
rows sz 196 tbl *.READHIST req <1873809319,1229981629> node
<Primary:LAB232157:NT>
(2009.266 02.04.16-20C4:kpxreqds.cpp,474,"Update") Request

<1873809319> to node Primary:LAB232157:NT now has status 7

Tivoli Enterprise Portal Server gets the data

(2009.266 02.04.16
29A8:ctsqlaccesssql1.cpp,1001,"CTSQLEvaluatorSQL1_i::AccessElement:
:pullSequenceWithTimeout") HUB_LAB232157(60): Rows returned: 20

14

Troubleshooting guidelines in the missing or incorrect historical data scenario

2012 IBM Corporation

During this step, Tivoli Enterprise Monitoring Server receives the 20 rows that the agent
sent and marks the request with status=7, which means that the data arrived correctly.
Then, Tivoli Enterprise Management Server passes back the 20 rows to Tivoli Enterprise
Portal Server, which closes the initial SQL query for historical data.

troubleshooting_missing_or_incorrect_historical_data.ppt

Page 14 of 18

Step 7: Workspaces are filled with data

Recording Time

15

Troubleshooting guidelines in the missing or incorrect historical data scenario

2012 IBM Corporation

On this slide, you see the four rows of data returned. The Tivoli Enterprise Portal view
shows the 20 rows of data returned, circled on the slide. You can easily determine that
there is historical data available for a view by looking at the hourglass symbol on the upper
left corner of the window. Another good indication is the Recording Time column that is
specific to the historical data tables.

troubleshooting_missing_or_incorrect_historical_data.ppt

Page 15 of 18

Summary

Now that you completed this module, you can troubleshoot issues with the historical data
collection with incorrect or empty outcomes in the related view on the Tivoli Enterprise Portal

16

Troubleshooting guidelines in the missing or incorrect historical data scenario

2012 IBM Corporation

Now that you completed this module, you can troubleshoot the scenario where the
monitoring product cannot collect and store historical data.

troubleshooting_missing_or_incorrect_historical_data.ppt

Page 16 of 18

Feedback

Your feedback is valuable

You can help improve the quality of IBM Education Assistant content to better meet your
needs by providing feedback.
Did you find this module useful?
Did it help you solve a problem or answer a question?
Do you have suggestions for improvements?

Click to send email feedback:

mailto:iea@us.ibm.com?subject=Feedback_about_troubleshooting_missing_or_incorrect_historical_data.pp
t

This module is also available in PDF format at: ../troubleshooting_missing_or_incorrect_historical_data.pdf

17

Troubleshooting guidelines in the missing or incorrect historical data scenario

2012 IBM Corporation

You can help improve the quality of IBM Education Assistant content by providing
feedback.

troubleshooting_missing_or_incorrect_historical_data.ppt

Page 17 of 18

Trademarks, disclaimer, and copyright information

IBM, the IBM logo, ibm.com, DB2, Tivoli, and z/OS are trademarks or registered trademarks of International Business Machines Corp., registered in
many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of other IBM trademarks
is available on the web at "Copyright and trademark information" at http://www.ibm.com/legal/copytrade.shtml
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.
Windows is a registered trademark of Microsoft Corporation in the United States, other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Other company, product, or service names may be trademarks or service marks of others.
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE
MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED
"AS IS" WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBMS CURRENT
PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR
ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING ANY WARRANTIES OR
REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICENSORS), OR ALTERING THE TERMS AND CONDITIONS OF ANY AGREEMENT
OR LICENSE GOVERNING THE USE OF IBM PRODUCTS OR SOFTWARE.
Copyright International Business Machines Corporation 2012. All rights reserved.

18

troubleshooting_missing_or_incorrect_historical_data.ppt

2012 IBM Corporation

Page 18 of 18