11g Mediator - Diagnosing Resequencer Issues PDF
11g Mediator - Diagnosing Resequencer Issues PDF
11g Mediator - Diagnosing Resequencer Issues PDF
Leave a Comment
In a previous blog post, we saw a few useful tips to help us quickly monitor the health of resequencer
components in a soa system at runtime. In this blog post, let us explore some tips to diagnose mediator
resequencer issues. During the diagnosis we will also learn some key points to consider for Integration
systems that run Mediator Resequencer composites.
Please refer to the Resequencer White paper for a review of the basic concepts of resequencing and
the interplay of various subsystems involved in the execution of Resequencer Mediator composites.
Context
In this blog post we will refer to the AIA Communications O2C Pre-Built Integration pack (aka O2C PIPs) as
an example for understanding some issues that can arise at runtime with resequencer systems and how
we can diagnose the cause of such issues. The O2C PIP uses resequencing-enabled flows. One such is
the UpdateSalesOrder flow between OSM and Siebel. It is used to process the OSM status of Sales
Orders in proper time sequence within the Siebel system.
data:text/html;charset=utf-8,%3Ch1%20class%3D%22entry-title%22%20style%3D%22color%3A%20rgb(49%2C%2049%2C%2049)%3B%20font-family%3A%
1/6
12/30/2014
Enabling this logger just for a few minutes will suffice and one can see messages such as below in soa
servers diagnostic logs, once every lease refresh cycle. The default refresh cycle is 60s apart.
[APP: soa-infra] [SRC_METHOD: renewContainerIdLease] Renew container id
[34DB0F60899911E39F24117FE503A156] at database time :2014-01-31 06:11:18.913
It implies, the server which logged the above message is running with a containerId of
34DB0F60899911E39F24117FE503A156 !
2/6
12/30/2014
average Elapsed Time and other performance indicators for the locker query.
Huge data volume due to no proper purging strategy for Mediator tables is a common reason for
deteriorated Locker query performance. Regular data purging, partitioning, statistics gathering and
creation of required indexes on MEDIATOR_GROUP_STATUS will usually ensure good performance of
locker query.
Note that there is only one Resequencer Locker thread running per server at runtime. Any database issue
that impacts the locker thread will impair all the Mediator Composites that use the same resequencing
strategy. The mediator resequencer uses database for storage, retrieval of messages to implement the
reordering and sequencing logic. Hence, the proper and timely maintenance of SOA database goes a
long way in ensuring a good performance.
3/6
12/30/2014
at jrockit/proxy/sun/misc/Unsafe.park(Unsafe.java:616)[inli
ned] from a group that
message
at java/util/concurrent/locks/LockSupport.parkNanos(LockSup
por
t.jav
a:196
)[i
nlined]
has
been
locked
by
the
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Ab
Locker.
at java/util/concurrent/<strong>LinkedBlockingQueue.poll</strong>(LinkedBlockingQueue.
at oracle/tip/mediator/common/listener/<strong>AbstractWork
er.rWorker
un</str
ong>(AbstractWor
Busy
Thread:
at oracle/integration/platform/blocks/executor/WorkManagerExecutor$1.run(WorkManagerEx
at weblogic/work/j2ee/J2EEWorkManager$WorkWithListener.run(
2EEWor
kM
anage
r.ja
va:184)
ItJ
should
be
noted
that
all
at weblogic/work/DaemonWorkThread.run(DaemonWorkThread.java
:30) processing of the
further
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
message until the next
-- end of trace
.
transaction boundary
happens in the context of
at oracle/tip/mediator/service/BaseActionHandler.requestProcess(BaseActionHandler.java
this worker thread. For
at oracle/tip/mediator/service/OneWayActionHandler.process(OneWayActionHandler.java:47
example,
the
diagram
at oracle/tip/mediator/service/ActionProcessor.onMessage(Ac
tionPro
ces
sor.java:64)[opti
below
shows
the
O2C
at oracle/tip/mediator/dispatch/MessageDispatcher.executeCase(MessageDis
patcher.java:1
at oracle/tip/mediator/dispatch/InitialMessageDispatcher.pr
UpdateSalesOrder
ocessCase(InitialMessageDis
at oracle/tip/mediator/dispatch/InitialMessageDispatcher.pr
ocessCase
s(In
itia
Integration
flow,
from
alMessageDi
at oracle/tip/mediator/dispatch/InitialMessageDispatcher.pr
o
c
e
s
s
N
o
r
m
a
l
C
a
s
e
s
(InitialMes
threads perspective. Here,
at oracle/tip/mediator/dispatch/resequencer/ResequencerMessageDispatcher.processCases(
the BPEL ABCS
at oracle/tip/mediator/dispatch/InitialMessageDispatcher.dispatch(InitialMessageDispat
processing,
the
calls
to
AIA
at oracle/tip/mediator/dispatch/resequencer/ResequencerMess
ageHandle
r.h
andl
eM
essage(Re
SessionPoolManager,
at oracle/tip/mediator/resequencer/<strong>ResequencerDBWor
ker.handleMessag<as
/strong>e(
at oracle/tip/mediator/resequencer/ResequencerDBWorker.proc
ess
(Rthe
eseSynchronous
quencerDBWorker.jav
well
as
at oracle/tip/mediator/common/listener/AbstractWorker.run(A
b
s
t
r
a
c
t
W
or
ker.java:81)
invoke to the
Siebel
at oracle/integration/platform/blocks/executor/WorkManagerExecutor$1.run(WorkManagerEx
Webservice, all happen in
at weblogic/work/j2ee/J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:184)
the
resequencer
worker
at weblogic/work/DaemonWorkThread.run(DaemonWorkThread.java
:30
)
thread.
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
Now consider an example thread stack as shown below seen in the server thread dump. It shows a worker
thread seen to be engaged in http communication with an external system.
Stuck Worker Thread:
"Workmanager: , Version: 0, Scheduled=false, Started=false, Wai
ti
me: 0remains
ms
Ift
this
thread
at the
data:text/html;charset=utf-8,%3Ch1%20class%3D%22entry-title%22%20style%3D%22color%3A%20rgb(49%2C%2049%2C%2049)%3B%20font-family%3A%
4/6
12/30/2014
" id=299 idx=0x174 tid=72518 prio=10 alive, in native, daemonsame position across
at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDe
scrip
tor;[B
III)I(Na
tive Met
thread
dumps
spanning
few
at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:32)[inlined]
minutes, it would indicate
at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStr
that
the
worker
thread
at java/net/<strong>SocketInputStream.read</strong>(SocketInp
utS
tre
am.jav
a:129is
)[optimize
blocked
on
external
at HTTPClient/BufferedInputStream.fillBuff(BufferedInputStrea
m.java
:20
6)
at HTTPClient/BufferedInputStream.read(BufferedInputStream.ja
va:126)[oapplication.
ptimized]If
Webservice
at HTTPClient/StreamDemultiplexor.read(StreamDemultiplexor.ja
va:3
56)[opt
imized]
such
external
system
^-- Holding lock: HTTPClient/StreamDemultiplexor@0x1758a7ae0[recursive]
issues block a significant
at HTTPClient/RespInputStream.read(RespInputStream.java:151)[optimized]
number of worker threads
.
from the pool of available
.
at oraclele/tip/mediator/dispatch/resequencer/ResequencerMess
ageDithreads,
spatcheitr.
processCases(
worker
will
at oracle/tip/mediator/dispatch/InitialMessageDispatcher.disp
a
t
c
h
(
I
n
i
t
i
a
l
M
e
impact the overall ssageDispatch
at oracle/tip/mediator/dispatch/resequencer/ResequencerMessageHandler.handleMessage(Rese
throughput of the system.
at oracle/tip/mediator/resequencer/<strong>ResequencerDBWorker.handleMessage</strong>(Re
There
will
be
fewer
workers
at oracle/tip/mediator/resequencer/ResequencerDBWorker.proces
s(Res
equ
en
cerDB
Worker.java:
available
to
process
all
the
at oracle/tip/mediator/common/listener/AbstractWorker.run(Abs
tractWo
rk
er.java
:8
1)
groups
that
are
being
at oracle/integration/platform/blocks/executor/WorkManagerExe
cutor$
1.r
un(
WorkManagerExec
at weblogic/work/j2ee/J2EEWorkManager$WorkWithListener.run(J2
EEWor
kMa
nag
er.java:184)
locked
by
the
Locker
at weblogic/work/DaemonWorkThread.run(DaemonWorkThread.java:3
0
)
thread, across all
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
composites that use
-- end of trace
Note that JTA timeout will not abort these busy threads. Such threads may eventually return after the JTA
transaction has rolled back, or in some cases depending on how sockets are handled by the external
system, may not return at all.
For such integration flows, it is advisable to configure HTTP Connect and Read timeouts for Webservice
calls at the composites Reference properties. Figure below shows a screenshot of the properties. This will
ensure that worker threads are not held up due to external issues and affect processing of other
components that rely on worker threads.
data:text/html;charset=utf-8,%3Ch1%20class%3D%22entry-title%22%20style%3D%22color%3A%20rgb(49%2C%2049%2C%2049)%3B%20font-family%3A%
5/6
12/30/2014
Conclusion
We have explored into how problems at various different layers can manifest at the resequencer in an
Integration system and how the cause of these issues can be diagnosed.
We have seen
Useful pointers in diagnosing resequencer issues and where to look for relevant information
How a good SOA database maintenance strategy is important for resequencer health
How timeout considerations play a role in resequencer performance
data:text/html;charset=utf-8,%3Ch1%20class%3D%22entry-title%22%20style%3D%22color%3A%20rgb(49%2C%2049%2C%2049)%3B%20font-family%3A%
6/6
Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.
Alternative Proxies: