Internal Concurrent Manager Status
Internal Concurrent Manager Status
Internal Concurrent Manager Status
Shankar Govindan On most sites the using Oracle Applications the common practice of an Oracle Apps DBA is to check the Unix Process of the Internal Concurrent Manager using the command: ps ef | grep CPMGR or ps ef | grep FNDCPMBR
applmgr 24988 24984 applmgr 20050 20040 applmgr 19900 19889 applmgr 22629 22624 applmgr 24757 24753 0 0 0 0 0 Dec 26 ? Dec 16 ? Dec 16 ? Dec 16 ? Dec 26 ? 5:16 FNDLIBR FND CPMGR FNDCPMBR sysmgr="" 3:30 FNDLIBR FND CPMGR FNDCPMBR sysmgr="" 4:53 FNDLIBR FND CPMGR FNDCPMBR sysmgr="" 0:56 FNDLIBR FND CPMGR FNDCPMBR sysmgr="" 5:04 FNDLIBR FND CPMGR FNDCPMBR sysmgr="" sleep=30 logfile=/Ora_base/thor/rpt/co sleep=30 logfile=/Ora_base/thor/rpt/co sleep=30 logfile=/Ora_base/thor/rpt/co sleep=30 logfile=/Ora_base/thor/applmg sleep=30 logfile=/Ora_base/thor/rpt/co
Well, either way the result of the output will be the same, because the search string exists in the output. If you have a single instance, then you would get just one output and that would help to know that the Internal Concurrent Manager Process is up (Note: Not necessarily running or active). In case you have different releases of Oracle Apps on the same server and if the output of the command above is as shown, then it would be difficult to find out which background process belongs to which Application. The goal of this Paper is to find out not only if the background process is alive, but also to find out if the Internal Manager is really active and to find the same across releases. Note different releases have different ways of addressing this problem, which you would see as we go along. According to Oracle Notes: Internal Concurrent Manager's function is only to run 'queue control' requests, which are requests to startup or shutdown other managers. It is responsible for startup and shutdown of the whole concurrent processing facility, and it also monitors the other managers periodically, and restarts them if they should go down. It can also take over the Conflict Resolution manager's job, and resolve incompatibilities. If the ICM itself should go down, requests will continue to run normally, except for 'queue control' requests. If the ICM should go down, you can restart it with 'startmgr'. You do not need to kill the other managers first. From the above note it is clear that, if the Internal Concurrent Manager is not active, then the background process would still exist and might become a zombie. The Activation of the other managers and failed managers would not happen and the normal symptom would be the Conflict Resolution Manager piling up all the requests. An example of which would be when the Internal Manager goes down with an error : APP-01167 Internal Concurrent Manager has encountered an error. This error would be in the ORACLE_SID.mgr file in the directory $APPLCSF/$APPLLOG , where all the Manager information log goes into, provided you have set the environment correctly in the APPLSYS.env file. This error is logged when an Internal Manager encounters various errors and deactivates the Internal Concurrent Manager and sets its Active process to zero. You can check if the the Active process is zero by logging in to the Application and navigate to, Concurrent Manager Administor
You would see that the Internal Managers Active Process would be zero and the Target Process would be one. The background CPMGR related process would be running, so this does not reflect the correct status of the Internal Concurrent Manager. You can also check which process belongs to which database by picking the OS Process ID ORACLE_SID.mgr file in the directory $APPLCSF/$APPLLOG , where the OS Process ID gets logged when the Internal Manager comes up but it is more work, example logfile output, ======================================================================== Starting dev3 internal concurrent manager -- shell process ID 24984 logfile=/Ora_base/thor/rpt/common/logdev3/dev3.mgr PRINTER=ioo341 mailto=applmgr restart=N diag=N sleep=30 pmon=20 (default) quesiz=1 (default) +---------------------------------------------------------------------------+ ps -ef | grep 24984
applmgr 24988 24984 0 Dec 26 ? 5:20 FNDLIBR FND CPMGR FNDCPMBR sysmgr="" sleep=30 ogfile=/Ora_base/thor/rpt/co
From the above command we know that the Internal Manager Process exists and is running for Database DEV3, but does not show dev3 anywhere at the backend when you grep the PPID or the PID. You can then verify the Internal Manager from the front-end application to check if it is running. This shows that it is imperative that we not only check the Internal Manager Background process, but also to make sure that the Actual Process of the Internal Manager is not set to zero and running. From the above ps ef | grep CPMGR command we got an output of Five Internal Manager Background process and we have to now figure out which process belongs to which database, which is essentially our goal.
Internal Manager and then do a grep on the Process ID. Then use the Unix pstree ID command to trace the process and pick the parent ID and its corresponding ORACLE_SID. The better way to do that would be to log on to the Application and pick the same OS Process ID of the Internal Concurrent Manager that gets logged in the $APPLCSF/$APPLLOG/ORACLE_SID.mgr file from the database by querying the table FND_CONCURRENT_PROCESSES. The table registers the OS Process ID in the field OS_PROCESS_ID and we can also check the Running_Processes to find out if the Actual Process is not equal to zero. Once the OS Process Id and the active Process is checked we can then pick the corresponding ORACLE_PORCESS_ID from the same table and compare it with the SID in the dynamic view v$session. The FND_CONCURENT_QUEUE would identify if the process is the Internal Manager Process, and hence it would check comparing the APPLICATION_IDs. The paddr of the v$session for the SID would then be compared with the addr of V$Process and the corresponding SPID can be retrieved. This SPID would be the spawned parent ID that would have the ORACLE_SID attached to it. The script below exactly does the same: ********************************** CUT HERE ********************************************** #!/bin/ksh #=========================================================================== # Filename: icmtest107 # Author : Shankar Govindan. # Created : 12/25/2001 # Good for : Release 107 Server Partition Mode only # Description : This script is used for checking out the status of Internal Manager in Oracle APPS, # It picks up the OS_PROCESS_ID from FND_CONCURRENT_PROCESSES, spools # them and then looks for the actual OS process with its corresponding ORACLE_SID. # The grep looks for SPID and LOCAL = NO because of Oracle version 7x. # Note : I have used a password file with system/manager which is in my /dba/etc dir. # # 1st Argument: status # 2nd Argument: ORACLE_SID (optional) # #=========================================================================== if ( [ "$1" != "status" ] ) then echo "Usage: $0 status " echo " Eg.: $0 status OSID" exit fi CMD=$1 OSID=$2 ORACLE_SID=`echo $ORACLE_SID` TEMPDIR=`pwd` GROUP=`groups | cut -d" " -f1` function _icmtestcmd { echo "===========================================================================" echo "checking Internal concurrent Manager status for Database ${ORACLE_SID} " date
echo "===========================================================================" echo PWD_FILE=/dba/etc/.${GROUP}_${ORACLE_SID}.pwd SYSTEM_PWD=`grep -i "^system/" $PWD_FILE` case $CMD in status) if [ "$SYSTEM_PWD" = "" ] ; then echo "\nERROR: userid=system does not exist in ${PWD_FILE}" return 1 fi # echo "** Sending status for Oracle database=${ORACLE_SID}" (echo ${SYSTEM_PWD}; echo set pages 0 pause off verify off feedback off termout off; echo "select p .spid from v\$process p, v\$session s, apps.fnd_concurrent_processes f, apps.fnd_concurrent_queues q where p.addr = s.paddr and s.sid = f.oracle_process_id and f.concurrent_queue_id = q.concurrent_q ueue_id and q.concurrent_queue_name = 'Internal Manager' and q.running_processes = 1 group by p.spi d order by 1 ;") | sqlplus -s > $TEMPDIR/icmppid ps -ef | grep -i `cat $TEMPDIR/icmppid` |grep NO | while read LINE do MATCH=`echo $LINE | awk '{print $9}'` if [ "$MATCH" = "oracle${ORACLE_SID}" ] ; then echo "For Database=$ORACLE_SID , Internal Concurrent Manager is UP and RUNNING !!" echo return 1 echo else echo "Database=$ORACLE_SID Internal Concurrent Manager is DOWN" echo fi done return 0 ;; *) echo "\nERROR: Invalid option=$CMD (in function _icmtestcmd)" return 2 ;; esac } # #MAIN # if [ "$CMD" = "status" ]; then _icmtestcmd $CMD $OSID else echo "Check the Command again " wait return 1 fi ********************************** CUT HERE **********************************************
In Release 11, the setup is similar to 10.7 NCA without the server partition mode, the old scripts would hold good, but it does not the check the active process from the database to compare and hence the script of 10.7 NCA server Partition Mode would be a better one. Although the concept for getting the information is the same, the stored information in the AOL tables has changed from 10.7 to Rel 11. The FND_CONCURRENT_QUEUES table does not store the CONCURRENT_QUEUE_NAME = Internal Manager but as FNDICM, so we pick the information from the view FND_CONCURENT_QUEUE_VL.USER_CONCURRENT_QUEUE_NAME = Internal Concurrent Manager and all the other info from the same view. The rest of the information are retrieved just like the above script used for 10.7 NCA Server Partition Mode. In the Rel 11 we use Net 8 Adapter and according to Oracle, The BEQUEATH adapter is essentially a true Net8 adapter though it has one unique distinction in that it does not connect to Oracle kernel via a Listener process, it connects directly to the kernel. Oracle 8 applications use the BEQUEATH adapter by default though you can specify a TNS alias for BEQUEATH. The BEQUEATH adapter has been around since Oracle 7.3.3. However, there were problems using 7.3.3. and Net 2.3.3 to connect to a local Oracle 8 kernel. These were addressed in the 7.3.4/2.3.4 release. In 10.7 Sever Partition Mode script we grep not only for the Process ID, but also grep for LOCAL=NO, since that identifies the connectivity for the process. In Release 11 we grep LOCAL = YES because of the above explanation for Net 8 adapter for Oracle 8 Kernel, the connection would look like:
oracle 22637 22633 0 Dec 16 ? 0:04 oracledev4 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
The script used for Release 11 Internal Manager status check: ************************************* Cut Here ********************************************* #!/bin/ksh #=========================================================================== # Filename: icmtest_rel11 # Author : Shankar Govindan. # Created : 01/04/2002 # Good for : Release 11 only # Description : This script is used for checking out the status of Internal Manager in Oracle APPS, # It picks up the OS_PROCESS_ID from FND_CONCURRENT_PROCESSES, spools them # and then looks for the actual OS process with its corresponding ORACLE_SID. # The grep looks for SPID and LOCAL = YES because of REL 11 SqlNet beq default mode. # Note : I have used a password file with system/manager. # # 1st Argument: status # 2nd Argument: ORACLE_SID (optional) # #=========================================================================== if ( [ "$1" != "status" ] ) then echo "Usage: $0 status " echo " Eg.: $0 status " exit fi CMD=$1 OSID=$2 ORACLE_SID=`echo $ORACLE_SID` TEMPDIR=`pwd`
GROUP=`groups | cut -d" " -f1` function icmtestcmd_rel11 { echo "===========================================================================" echo "checking Internal concurrent Manager status for Database ${ORACLE_SID} " date echo "===========================================================================" echo PWD_FILE=/dba/etc/.${GROUP}_${ORACLE_SID}.pwd SYSTEM_PWD=`grep -i "^system/" $PWD_FILE` case $CMD in status) if [ "$SYSTEM_PWD" = "" ] ; then echo "\nERROR: userid=system does not exist in ${PWD_FILE}" return 1 fi # echo "** Sending status for Oracle database=${ORACLE_SID}" (echo ${SYSTEM_PWD}; echo set pages 0 pause off verify off feedback off termout off; echo "select p .spid from v\$process p, v\$session s, apps.fnd_concurrent_processes f, apps.fnd_concurrent_queues_ vl q where p.addr = s.paddr and s.audsid = f.session_id and f.concurrent_queue_id = q.concurrent_qu eue_id and f.queue_application_id = q.application_id and q.user_concurrent_queue_name = 'Internal M anager' and q.manager_type = f.manager_type and q.running_processes = 1 group by p.spid order by 1 ;") | sqlplus -s > $TEMPDIR/icmppid ps -ef | grep -i `cat $TEMPDIR/icmppid` | grep YES | while read LINE do MATCH=`echo $LINE | awk '{print $9}'` if [ "$MATCH" = "oracle${ORACLE_SID}" ] ; then echo "For Database=$ORACLE_SID , Internal Concurrent Manager is UP and RUNNING !!" echo return 1 echo else echo "Database=$ORACLE_SID Internal Cocnurrent Manager is DOWN" echo fi done return 0 ;; *) echo "\nERROR: Invalid option=$CMD (in function _icmtestcmd)" return 2 ;; esac } # #MAIN # if [ "$CMD" = "status" ]; then icmtestcmd_rel11 $CMD $OSID else echo " Check your command " wait return 1
fi ************************************* Cut Here ********************************************* Although I have installed Oracle Apps 11I CRM, the present site where I wrote and tested these scripts does not have the same yet and couldnt test the same on 11i. Presume it should work since it is built on the same server partition setup and the FND_COCNURRENT_PROCESSES and FND_CONCURRENT_QUEUES table would exist. The above script calls customized functions and hence can be combined into a single script for different releases by a simple check on some input variable. For example you could check the ORACLE_SIDs you have from ORATAB file and check if the ORACLE_SID belongs to which release from a RELTAB file or input the release along with the command and check for the same. If you have a lot of Application Databases on a single server like we had then you can use the above scripts to loop around using the ORATAB file and check for Internal Manager Status of every ORACLE_SID that is registered in the ORATAB using the same concept of dbstart and dbshut scripts of oracle like: ONLYSID= if [ "$SID" != "" ] then grep -v ^# $ORATAB | grep "^${SID}:" >/dev/null if [ $? != 0 ] then echo "ERROR: Oracle SID=$SID not exists on this system (oratab)" exit 1 fi ONLYSID=" | grep ^${SID}:" fi eval "grep -v ^# $ORATAB $ONLYSID" | while read LINE do ORACLE_SID=`echo $LINE | awk -F":" '{print $1}'` if [ "$CMD" == "status" ] ; then _cmctlcmd $CMD $ORACLE_SID else echo Wrong command wait fi done
Shankar Govindan works as a Sr. Oracle DBA at CNF , Portland, Oregon. Shankar Govindan is Oracle Certified 7, 8 and 8I, you can contact him at shankargovindan@yahoo.com. The above scripts were tested on Sun Solaris 2.6 and Oracle Apps 10.7 NCA and Release 11. Note: The above info as usual is of my individual tests and opinions and has nothing to do with the company I work for or represent. __________________________________________________________________________________