Oracle Middleware 12c In-

place Upgrade procedure

Project: EOL 2021

TSOE- CCOE

Classification: Internal
 Document History

Version Date Changed By Reviewed By Change Description

1.0 31/03/2021 HFA025 Created by Houssam Fatnaci.

1.1 07/04/2021 HFA025 Updates/Corrections

1.2 08/04/2021 HFA025 Enhancements

1.3 10/05//2021 HFA025 Splitting installation and upgrade

1.4 17/05/2021 HFA025 OSB upgrade

1.5 17/05/2021 HFA025 Add Restore procedure

Classification: Internal
 Table of Contents
1 INTRODUCTION.......................................................................................................4
2 IBM DC’ ORACLE MW IN-PLACE UPGRADE PROCEDURE.........................6
2.1 Weblogic.....................................................................................................6

2.2 Oracle Service Bus...................................................................................10

2.3 BPEL........................................................................................................25

3 AZURE’ ORACLE MW IN-PLACE UPGRADE PROCEDURE.......................26

3.1 Weblogic...................................................................................................26

3.2 Oracle Service Bus...................................................................................30

3.3 BPEL........................................................................................................45

Classification: Internal
 1

Classification: Internal
 2
Introduction
EOL (End Of Life) 2021 is continuation of previous M&I (Maintenance and Improve)
program launched back in 2018 to address Software technical debt removal from
Maersk’ IBM and Azure managed estate. Other objectives are to align existing systems
and applications to TSOE Engineering standards which are embodied in Reference
Architectures, Blueprints and Deployment patterns of various software stack that TSOE
Engineering and Operations support, this include but not limited:

 Oracle Fusion Middleware

 IBM MQ
 Oracle Database
 RHEL and Windows OS
 MS SQL Server

Objectives:

Part of EOL 2021 is to remediate Oracle Fusion Middleware products, the guideline has
been set to target latest 12c stable release 12.2.1.4 which has Extended EoS August
2025.
Oracle Fusion Middleware product includes: Weblogic, OHS, SOA Suite, BPEL, OSB, ESS.

Note: At the time of writing this document, some of Oracle Fusion Middleware products
-but not all- have been released on the new 14c version but it is at its first GA release
14.1.1 with no Extended EoS date! which will be only announced for The Terminal (or
Long Term Support) Release, and it is only at that point of time TSOE and Platforms
teams will be tasked to kick a new migration plan from 12c to 14c.

TSOE used to perform upgrade to new version by doing new builds, ie new VMs having
the target MW version, also known as green-field upgrade. Experience in previous M&I
program has shown that this practice had led to drastic ramifications in term of time
delivery thus affecting project milestones due to various factors:
 IBM Forecast Capacity 3 months lead time to allocate required capacity to spin up new VMs
 Ops onboarding
 Application migration from existing version to new version

An alternative solution has emerged to avoid greed-field upgrade by performing in-place

upgrade when it is applicable and if it is supported by Oracle.
Oracle provides instructions and steps how to perform such upgrade which relies on
upgrade wizard utility.

The utility and steps have been incorporated in our MW puppet framework -when
possible- which is used by TSOE Engineering to build new MW solutions and being used
by MW OPs for MW configuration Management

Classification: Internal
 The purpose of this document is to list the required steps to perform the in-
place upgrade automation via puppet framework + manual steps when required
in IBM and Azure Infra.

Classification: Internal
 3 IBM DC’ ORACLE
MW IN-PLACE UPGRADE
PROCEDURE
Each oracle fusion Middleware product will have its own section, SOA products tends to
be more complex as it involves DB

3.1 Weblogic

2.1.1 OFMW 12214 Installation (as root user)

The new OFMW 12214(weblogic) binaries installation can be executed online,

ie without affecting the existing installation /u01/oracle/product/fmw1221
as the 12214 will be installed in a dedicated path:
/u01/oracle/product/fmw12214; thus avoiding any downtime of the application
during the installation.

a. Git pull master repo “puppet_automation_ibm”

cd /u01/puppet/environment/<proj>_<env>
git config --global http.proxy http://10.0.11.1:9400
git remote set-url origin https://github.com/Maersk-Global/puppet_automation_ibm.git
git pull (you will be asked to enter your username and pwd for github)

b. Copy script /u01/puppet/environments/<proj>_<env>/bootstrap/ibm/

install_12214.sh to /tmp :
cd /tmp
cp /u01/puppet/environments/${PROJECT}_${ENVIRONMENT}/bootstrap/ibm/install_12214.sh /tmp
chmod u+x install_12214.sh

c. Install on Admin node

  /install_12214.sh -r admin -p <project> -x <prefix> -e <environment> -f <platform> -v 12214

     e.g :  ./install_12214.sh -r admin -p edi -e dev1 -f  wls -v 12214

Note: Don’t supply -x prefix if the domain name doesn’t have one!

d. Install on Managed nodes

Note: the installation can be executed in parallel in all nodes of
the domain

./install_12214.sh -r managed -p <project> -x <prefix> -e <environment> -f <platform> -v 12214

Classification: Internal
 2.1.2 Upgrade Pre-checks

 Ensure that there is only the domain folder under:

/u01/oracle/config/domains (admin node)
/u02/oracle/config/domains (managed node)
any other folders must be moved to /u02/oracle/backup or removed
if deemed unnecessary

 Check the domain size folder not exceeding 2GB or 3GB max as
this can be problematic when taking a backup of the domain, so
any means to clean up unwanted files should be done prior to
upgrade

 Ensure that in the node yaml file:

/u01/puppet/environments/${PROJECT}_$
{ENVIRONMENT}/hieradata/nodes/${PROJECT}_${ENVIRONMENT}/
$hostname.crb.apmoller.net.yaml
jdk, patches, source repo entries are removed(take backup of
the file before update):

wls_jdk_version:
wls_jdk_full_version:
jdk_package:
crypto_extension_file:

opatch_instances:
'27342434':
....

wls_source: &wls_source "/u02/oracle/software"

 If the domain has USI Realm configuration(regardless if it is

enabled or disabled): USI configuration must be removed from
the domain config.xml before we can proceed to next step or the
upgrade will fail:
Note: This is only applicable on the Admin node, not Managed
nodes

 Take backup of the config.xml file before then

remove the USI realm definition from the weblogic domain
config file, the usi entry in the
config.xml(/u01/oracle/config/domains/<domain_name>/config)
is:

  <realm>

      <sec:authentication-provider
xmlns:sspi="http://sspi.security.apmoller.net"
xsi:type="sspi:msl-authenticatorType">
        <sec:name>MSLAuthenticator</sec:name>

Classification: Internal
         <sec:control-flag>SUFFICIENT</sec:control-flag>
      </sec:authentication-provider>
………….
      <sec:name>USI</sec:name>
      <sec:delegate-m-bean-
authorization>false</sec:delegate-m-bean-authorization>
    </realm>

 and ensure default realm is set to myrealm instead of USI

<default-realm>myrealm</default-realm>

2.1.3 Domain Upgrade Process

a. Server Stop procedure

Login as Oracle user and execute below commands one by one

stopms  To stop Managed server

stopadm  To stop Admin server
stopnm  To stop Nodemanager

Note:
For HA environment with weblogic cluster set up, the stopping of
servers must be done one at the time to avoid full downtime.
Always start with VM hosting the admin server and once upgrade is
done on that VM, move to other VMs hosting the managed servers

b. Copy script
/u01/puppet/environments/<proj>_<env>/bootstrap/ibm/wls_upgrade_ib
m.sh to /tmp :
cd /tmp
cp /u01/puppet/environments/${PROJECT}_${ENVIRONMENT}/bootstrap/ibm/wls_upgrade_ibm.sh /tmp
chmod u+x wls_upgrade_ibm.sh

c. Upgrade Weblogic Admin Server (as root)

  /wls_upgrade_ibm.sh -r admin -p <project> -x <prefix> -e <environment> -f <platform> -v 12214

     e.g :  ./wls_upgrade_ibm.sh -r admin -p edi -e dev1 -f  wls -v 12214

This will upgrade the Node hosting admin server and any managed server(s)
coexisting with admin as well but only admin server will come up
automatically upon upgrade completion, so you need to go to start up these
managed servers yourselves manually (ie startms cmd)

If the domain is USI-secured:

After the upgrade is completed on admin node and before moving to managed
node (HA set up), the new config.xml of 12214 domain should be updated by
adding back the USI entry we removed in the pre-checks section and

Classification: Internal
 resetting default-realm to USI with admin server restart then move to the
managed node by node.

d. Upgrade Weblogic Managed Server (as root)

Note: Execute only in VMs where managed server is not running

alongside admin server in the same VM, i.e. managed server running
in a dedicated VM. So, this is not applicable to standalone domain
and only HA ones

./wls_upgrade_ibm.sh -r managed -p <project> -x <prefix> -e <environment> -f <platform> -v 12214

Startup properties files to be copied from backup to new upgrade

domain or the managed servers will fail to come up
cd /u02/oracle/backups/<backup domain>/servers/<project>_wls_pp_ms2/data/nodemanager
cp * /u02/oracle/config/domains/<project>_wls_pp/servers/<project>_wls_pp_ms2/data/nodemanager/

Note: Check the managed server name running on the VM you are
upgrading: ms2, ms3 or msxxx and adapt the above cmd accordingly

2.1.4 Start Weblogic Admin, Managed server & Nodemanager(As oracle user)

     Node manager and admin server will be started automatically by the
upgrade, you only need to start the managed server:
startms    -- To start managed server

2.1.5 For USI-secured Domain:

After the upgrade is completed (step 2.1.4), the new config.xml of 12214
domain should be updated by adding the USI entry we removed in the pre-
checks section and resetting default-realm to USI with full domain restart
node by node.

2.1.6 Update ownership of applications artefacts (if needed)

For projects using CI/CD pipeline, applications artefacts are owned by

devops or other users than oracle, e.g for EDDI2:

During the upgrade, the puppet fmk will reset all the permissions under
/u01 and /u02 to oracle user, which means applications artefacts ownership
will change from devops to oracle, this will cause the CD pipeline to fail
if it tries to redeploy the same existing artefacts.
So, it is required in this case to change ownership of these artefacts from
oracle to devops user as post upgrade task and before kicking the
deployment, e.g :

Classification: Internal
 cd /u01/oracle/application/config/
chown -R devops:oinstall EDDI2*

For new artefacts that will be pushed by the CD pipeline there should be no
issue as the pipeline will place new artefacts as devops user.

2.1.7 Test the application and confirm - Application support

2.1.8 Redeploy the Application (CI/CD or manual) and Test again
2.1.9

2.1.10 Restore Procedure

 Ensure the folder domains folder is empty:
- Admin node to: /u01/oracle/config/domains/
- Managed Node to :/u02/oracle/config/domains/
 All node manager, admin or managed servers’ processes are
shutdown
 Copy the backup from /u02/oracle/backups/<domain_name>_$date
- Admin node to: /u01/oracle/config/domains/<domain_name>
- Managed Node to :/u02/oracle/config/domains/<domain_name>
 Update the /home/oracle/.bash_profile to point to fmw1221
instead of fmw12214 , the source the file
 Simply restart your domain by starting first node
manager(s),admin server and managed server(s)

3.2 Oracle Service Bus

2.2.1 OFMW/OSB 12214 Installation (as root user)

The new OFMW + OSB 12214 binaries installation can be executed online, ie
without affecting the existing installation /u01/oracle/product/fmw1221 as
the 12214 will be installed in a dedicated path:
/u01/oracle/product/fmw12214; thus avoiding any downtime of the application
during the installation.

a. Git pull master repo “puppet_automation”

cd /u01/puppet/environment/<proj>_<env>
git config --global http.proxy http://10.0.11.1:9400
git remote set-url origin https://github.com/Maersk-Global/puppet_automation.git
git pull (you will be asked to enter your username and pwd for github)

b. Copy script /u01/puppet/environments/<proj>_<env>/bootstrap/ibm/

install_12214.sh to /tmp :
cd /tmp
cp /u01/puppet/environments/${PROJECT}_${ENVIRONMENT}/bootstrap/ibm/install_12214.sh /tmp
chmod u+x install_12214.sh

c. Install on Admin node

  /install_12214.sh -r admin -p <project> -x <prefix> -e <environment> -f <platform> -v 12214

e.g :  ./install_12214.sh -r admin -p esb -x 1 -e int2 -f osb -v 12214

Classification: Internal
 d. Install on Managed nodes
Note: the installation can be executed in parallel in all nodes of
the domain

./install_12214.sh -r managed -p <project> -e <environment> -f <platform> -v 12214

e.g :  ./install_12214.sh -r managed -p esb -x 1 -e int2 -f osb -v 12214

2.2.2 Upgrade Pre-checks

 Ensure that there is only the domain folder under:

/u01/oracle/config/domains (admin node)
/u02/oracle/config/domains (managed node)
any other folders must be moved to /u02/oracle/backup or removed
if deemed unnecessary

 Check the domain size folder not exceeding 2GB or 3GB max as
this can be problematic when taking a backup of the domain, so
any means to clean up unwanted files should be done prior to
upgrade

 Ensure that in the node yaml file:

/u01/puppet/environments/${PROJECT}_$
{ENVIRONMENT}/hieradata/nodes/${PROJECT}_${ENVIRONMENT}/
$hostname.crb.apmoller.net.yaml
jdk, patches, source repo entries are removed(take backup of
the file before update):

wls_jdk_version:
wls_jdk_full_version:
jdk_package:
crypto_extension_file:

opatch_instances:
'27342434':
....

wls_source: &wls_source "/u02/oracle/software"

 If the domain has USI Realm configuration(regardless if it is

enabled or disabled): USI configuration must be removed from
the domain config.xml before we can proceed to next step or the
upgrade will fail:
Note: This is only applicable on the Admin node, not Managed
nodes

 Take backup of the config.xml file before then

remove the USI realm definition from the weblogic domain
config file, the usi entry in the
config.xml(/u01/oracle/config/domains/<domain_name>/config)
is:

  <realm>

      <sec:authentication-provider
xmlns:sspi="http://sspi.security.apmoller.net"

Classification: Internal
 xsi:type="sspi:msl-authenticatorType">
        <sec:name>MSLAuthenticator</sec:name>
        <sec:control-flag>SUFFICIENT</sec:control-flag>
      </sec:authentication-provider>
………….
      <sec:name>USI</sec:name>
      <sec:delegate-m-bean-
authorization>false</sec:delegate-m-bean-authorization>
    </realm>

 and ensure default realm is set to myrealm instead of USI

<default-realm>myrealm</default-realm>

 The following pre-checks have been results of issues and

learned lessons faced in previous upgrade tests as it turns
out some envs are missing some data/config that cause the
upgrade to fail:

Issue Description Solution

ID
1 Missing temporary Ask DB Ops to check if it exist and create it
tablespace in OSB PDBs: if not there
<OSBPDB>_IAS_TEMP

2 Missing files on For some reason, when installing 12214 the

/u01/oracle/product/fmw1 folder in question isn’t always the same!
2214/soa/common/sql check the size by “du -s” if it doesn’t match
73908, then you need to copy from a valid
installation
3 Missing right on Ask the DBA to check if below rules are there,
XXX_SOAINFRA schema if not add them:
GRANT EXECUTE ON DBMS_LOCK TO XXXX_SOAINFRA;
GRANT CREATE JOB TO XXXX_SOAINFRA;
GRANT CREATE EXTERNAL JOB TO XXXX_SOAINFRA;
GRANT SELECT ON GV_$SESSION TO XXXX_SOAINFRA
4 XXX_SOAINFRA is INVALID Check if it is in INVALID by:
SELECT * FROM SCHEMA_VERSION_REGISTRY WHERE
owner= ‘XXX_SOAINFRA’
If it is invalid, update it:
UPDATE SCHEMA_VERSION_REGISTRY SET STATUS =
'VALID' WHERE OWNER='xxx_SOAINFRA';
5 In domain config.xml, Add the below to config.xml by targeting the
the app-deployment app to ms1(update accordingly to fit the ms1 of
"Service Bus Domain your domain)
Singleton Marker <app-deployment>
<name>Service Bus Domain Singleton Marker
Application" is not
Application</name>
target to any server <target>esb_osb_2_int2_ms1</target>
<module-type>ear</module-type>
<source-

Classification: Internal
 path>/u01/oracle/product/fmw1221/osb/lib/apps/domainsinglet
onmarker.ear</source-path>
<deployment-order>31</deployment-order>
<security-dd-model>DDOnly</security-dd-model>
<staging-mode>nostage</staging-mode>
<plan-staging-mode xsi:nil="true"></plan-staging-mode>
<cache-in-app-directory>false</cache-in-app-directory>
</app-deployment>

2.2.3 Domain Upgrade Process (Perform with oracle user)

a. Prerequisites:
-> putty (configure it with X11-Forward)

->Xming installed
-> sysdba user on OSB PDBs
-> oracle user pwd to ssh to VM hosting the admin server
-> a guaranteed restore point has been captured online by DB Ops
on OSB PDBs before the upgrade kicks in (step d)

b. Upgrade Readiness check:

cd /u01/oracle/product/fmw12214/oracle_common/upgrade/bin/
./ua -readiness

Classification: Internal
Classification: Internal
 Enter the sysdba user/pwd ensure you enter as shown below :

Classification: Internal
 c. Stop all domain processes

Classification: Internal
 d. Schema upgrade (GUI)

cd /u01/oracle/product/fmw12214/oracle_common/upgrade/bin/
./ua

Classification: Internal
Classification: Internal
Classification: Internal
Classification: Internal
 e. Domain Reconfiguration(as root):

cd /tmp
cp /u01/puppet/environments/${PROJECT}_${ENVIRONMENT}/bootstrap/ibm/osb_upgrade_ibm.sh .
chmod u+x osb_upgrade_ibm.sh
./osb_upgrade_ibm.sh -r admin -p <project> -x <prefix> -e <env> -f osb -v 12214
     e.g: ./ osb_upgrade_ibm.sh -r admin -p esb -x 1 -e int2 -f osb -v 12214

f. Domain Components upgrade (GUI) (as oracle)

cd /u01/oracle/product/fmw12214/oracle_common/upgrade/bin/
./ua

Classification: Internal
Classification: Internal
Classification: Internal
 g. If the domain is USI-secured

After the upgrade is completed on admin node and before moving

to managed node (HA set up), the new config.xml of 12214 domain
should be updated by adding back the USI entry we removed in
the pre-checks section and resetting default-realm to USI with
admin server restart then move to the managed node by node.

h. Pack the domain on Admin Node

->required only for HA env so the domain can be unpacked on
managed nodes:

puppet apply --test --debug -e "include maersk::oracle::weblogic::packdomain"

i. Upgrade on managed nodes (as root)

->should be executed in parallel in managed nodes to reduce
downtime :
Do NOT run on the node having admin server
Prerequisites:
-> Admin server is running on admin
-> Ensure below folder is empty on every managed node
:/u02/oracle/config/applications/<domain_name>/

cd /tmp
cp /u01/puppet/environments/${PROJECT}_${ENVIRONMENT}/bootstrap/ibm/osb_upgrade_ibm.sh .
chmod u+x osb_upgrade_ibm.sh
./osb_upgrade_ibm.sh -r managed -p <project> -x <prefix> -e <env> -f osb -v 12214

e.g : ./osb_upgrade_ibm.sh -r managed -p esb -x 1 -e int2 -f osb -v 12214

Post-upgrade(as oracle):
-> start node manager: startnm
->ensure the below startup scripts are pointing to correct managed
server name running on the node you are upgrading:
/u02/oracle/scripts/stopms.py
/u02/oracle/scripts/startms.py

i. Start up the domain

 start node manager by startnm if not already done

 Start admin server by startadm if not already up
 In every managed node, start managed server by startms

Classification: Internal
 First start up of managed node will fail but it will create the server
folder but it will be missing the startup properties.
copy those from the backup domain:

/u02/oracle/backups/<backedup _domain>>/servers/<ms_name>/data/nodemanager

to:
/u01/oracle/config/domains/ <domain_folder>/servers/<ms_name>/data/nodemanager

then restart the node manager and start the managed server it
again

2.2.4 Restore Procedure

 Ensure the below folders are empty:

- Admin node to: /u01/oracle/config/domains/ &
/u01/oracle/config/applications/<domain_name>/
- Managed Node to :/u02/oracle/config/domains/ &
/u02/oracle/config/applications/<domain_name>/
 All node manager, admin or managed servers’ processes are
shutdown
 Copy the backups from :
/u02/oracle/backups/<domain_name>_$date and
/u02/oracle/backups/<domain_name>_app_$date respectively
to
- Admin node: /u01/oracle/config/domains/<domain_name> &
/u01/oracle/config/applications/<domain_name>
- Managed Node to :/u02/oracle/config/domains/<domain_name>
& /u02/oracle/config/applications/<domain_name>/
 Update the /home/oracle/.bash_profile to point to fmw1221
instead of fmw12214 , then source the file
 Ask DBA to restore the PDBs from the restore point taken
before the upgrade
 Simply restart your domain by starting first node
manager(s),admin server and managed server(s)

3.3 BPEL

Classification: Internal
 4 AZURE’ ORACLE
MW IN-PLACE UPGRADE
PROCEDURE

4.1 Weblogic

3.1.1 OFMW 12214 Installation (as root user)

The new OFMW 12214(weblogic) binaries installation can be executed online,

ie without affecting the existing installation /u01/oracle/product/fmw1221
as the 12214 will be installed in a dedicated path:
/u01/oracle/product/fmw12214; thus avoiding any downtime of the application
during the installation.

e. Git pull master repo “puppet_automation”

cd /u01/puppet/environment/<proj>_<env>
git config --global http.proxy http://10.0.11.1:9400
git remote set-url origin https://github.com/Maersk-Global/puppet_automation.git
git pull (you will be asked to enter your username and pwd for github)

f. Copy script /u01/puppet/environments/<proj>_<env>/bootstrap/azure/

install_12214.sh to /tmp :
cd /tmp
cp /u01/puppet/environments/${PROJECT}_${ENVIRONMENT}/bootstrap/azure/install_12214.sh /tmp
chmod u+x install_12214.sh

g. Install on Admin node

  /install_12214.sh -r admin -p <project> -x <prefix> -e <environment> -f wls -v 12214

     e.g :  ./install_12214.sh -r admin -p esb -e dev1 -f wls -v 12214

Note: Don’t supply x prefix if the domain name doesn’t have one!

h. Install on Managed nodes

Note: the installation can be executed in parallel in all nodes of
the domain

./install_12214.sh -r managed -p <project> -x <prefix> -e <environment> -f <platform> -v 12214

3.1.2 Upgrade Pre-checks

Classification: Internal
  Ensure that there is only the domain folder under:
/u01/oracle/config/domains (admin node)
/u02/oracle/config/domains (managed node)
any other folders must be moved to /u02/oracle/backup or removed
if deemed unnecessary

 Check the domain size folder not exceeding 2GB or 3GB max as
this can be problematic when taking a backup of the domain, so
any means to clean up unwanted files should be done prior to
upgrade

 Ensure that in the node yaml file:

/u01/puppet/environments/${PROJECT}_$
{ENVIRONMENT}/hieradata/nodes/${PROJECT}_${ENVIRONMENT}/
$hostname.crb.apmoller.net.yaml
jdk, patches, source repo entries are removed(take backup of
the file before update):

wls_jdk_version:
wls_jdk_full_version:
jdk_package:
crypto_extension_file:

opatch_instances:
'27342434':
....

wls_source: &wls_source "/u02/oracle/software"

 If the domain has USI Realm configuration(regardless if it is

enabled or disabled): USI configuration must be removed from
the domain config.xml before we can proceed to next step or the
upgrade will fail:
Note: This is only applicable on the Admin node, not Managed
nodes

 Take backup of the config.xml file before then

remove the USI realm definition from the weblogic domain
config file, the usi entry in the
config.xml(/u01/oracle/config/domains/<domain_name>/config)
is:

  <realm>

      <sec:authentication-provider
xmlns:sspi="http://sspi.security.apmoller.net"
xsi:type="sspi:msl-authenticatorType">
        <sec:name>MSLAuthenticator</sec:name>
        <sec:control-flag>SUFFICIENT</sec:control-flag>
      </sec:authentication-provider>
………….
      <sec:name>USI</sec:name>
      <sec:delegate-m-bean-
authorization>false</sec:delegate-m-bean-authorization>

Classification: Internal
     </realm>

 and ensure default realm is set to myrealm instead of USI

<default-realm>myrealm</default-realm>

3.1.3 Domain Upgrade Process

e. Server Stop procedure

Login as Oracle user and execute below commands one by one

stopms  To stop Managed server

stopadm  To stop Admin server
stopnm  To stop Nodemanager

Note:
For HA environment with weblogic cluster set up, the stopping of
servers must be done one at the time to avoid full downtime.
Always start with VM hosting the admin server and once upgrade is
done on that VM, move to other VMs hosting the managed servers

f. Copy script
/u01/puppet/environments/<proj>_<env>/bootstrap/azure/wls_upgrade_
azure.sh to /tmp:

cd /tmp
cp /u01/puppet/environments/${PROJECT}_${ENVIRONMENT}/bootstrap/azure/wls_upgrade_azure.sh /tmp
chmod u+x wls_upgrade_azure.sh

g. Upgrade Weblogic Admin Server (as root)

  /wls_upgrade_azure.sh -r admin -p <project> -x <prefix> -e <environment> -f <platform> -v 12214

     e.g :  ./wls_upgrade_azure.sh -r admin -p edi -e dev1 -f  wls -v 12214

This will upgrade the Node hosting admin server and any managed server(s)
coexisting with admin as well but only admin server will come up
automatically upon upgrade completion, so you need to go to start up these
managed servers yourselves manually (ie startms cmd)

If the domain is USI-secured:

After the upgrade is completed on admin node and before moving to managed
node (HA set up), the new config.xml of 12214 domain should be updated by
adding back the USI entry we removed in the pre-checks section and
resetting default-realm to USI with admin server restart then move to the
managed node by node.

h. Upgrade Weblogic Managed Server (as root)

Note: Execute only in VMs where managed server is not running

alongside admin server in the same VM, i.e. managed server running

Classification: Internal
 in a dedicated VM. So, this is not applicable to standalone domain
and only HA ones

./wls_upgrade_azure.sh -r managed -p <project> -x <prefix> -e <environment> -f <platform> -v

12214

Startup properties files to be copied from backup to new upgrade

domain or the managed servers will fail to come up (as oracle)
cd /u02/oracle/backups/<backup domain>/servers/<project>_wls_pp_ms2/data/nodemanager
cp * /u02/oracle/config/domains/<project>_wls_pp/servers/<project>_wls_pp_ms2/data/nodemanager/

Note: Check the managed server name running on the VM you are
upgrading: ms2, ms3 or msxxx and adapt the above cmd accordingly

3.1.4 Start Weblogic Admin, Managed server & Nodemanager(As oracle user)

     Node manager and admin server will be started automatically by the
upgrade, you only need to start the managed server:
startms    -- To start managed server

3.1.5 Update ownership of applications artefacts (if needed)

For projects using CI/CD pipeline, applications artefacts are owned by

devops or other users than oracle, e.g for EDDI2:

During the upgrade, the puppet fmk will reset all the permissions under
/u01 and /u02 to oracle user, which means applications artefacts ownership
will change from devops to oracle, this will cause the CD pipeline to fail
if it tries to redeploy the same existing artefacts.
So, it is required in this case to change ownership of these artefacts from
oracle to devops user as post upgrade task and before kicking the
deployment, e.g :

cd /u01/oracle/application/config/
chown -R devops:oinstall EDDI2*

For new artefacts that will be pushed by the CD pipeline there should be no
issue as the pipeline will place new artefacts as devops user.

3.1.6 Test the application and confirm - Application support

3.1.7 Redeploy the Application (CI/CD or Manual)and Test

3.1.8 Restore Procedure

 Ensure the folder domains folder is empty:
- Admin node to: /u01/oracle/config/domains/
- Managed Node to :/u02/oracle/config/domains/

Classification: Internal
  All node manager, admin or managed servers’ processes are
shutdown
 Copy the backup from /u02/oracle/backups/<domain_name>_$date
- Admin node to: /u01/oracle/config/domains/<domain_name>
- Managed Node to :/u02/oracle/config/domains/<domain_name>
 Update the /home/oracle/.bash_profile to point to fmw1221
instead of fmw12214 , the source the file
 Simply restart your domain by starting first node
manager(s),admin server and managed server(s)

4.2 Oracle Service Bus

3.2.1 OFMW/OSB 12214 Installation (as root user)

The new OFMW + OSB 12214 binaries installation can be executed online, ie
without affecting the existing installation /u01/oracle/product/fmw1221 as
the 12214 will be installed in a dedicated path:
/u01/oracle/product/fmw12214; thus avoiding any downtime of the application
during the installation.

i. Git pull master repo “puppet_automation”

cd /u01/puppet/environment/<proj>_<env>
git config --global http.proxy http://10.0.11.1:9400
git remote set-url origin https://github.com/Maersk-Global/puppet_automation.git
git pull (you will be asked to enter your username and pwd for github)

j. Copy script /u01/puppet/environments/<proj>_<env>/bootstrap/azure/

install_12214.sh to /tmp :
cd /tmp
cp /u01/puppet/environments/${PROJECT}_${ENVIRONMENT}/bootstrap/azure/install_12214.sh /tmp
chmod u+x install_12214.sh

k. Install on Admin node

  /install_12214.sh -r admin -p <project> -x <prefix> -e <environment> -f <platform> -v 12214

e.g :  ./install_12214.sh -r admin -p esb -x 1 -e int2 -f osb -v 12214

l. Install on Managed nodes

Note: the installation can be executed in parallel in all nodes of
the domain

./install_12214.sh -r managed -p <project> -e <environment> -f <platform> -v 12214

e.g :  ./install_12214.sh -r managed -p esb -x 1 -e int2 -f osb -v 12214

3.2.2 Upgrade Pre-checks

 Ensure that there is only the domain folder under:

/u01/oracle/config/domains (admin node)

Classification: Internal
 /u02/oracle/config/domains (managed node)
any other folders must be moved to /u02/oracle/backup or removed
if deemed unnecessary

 Check the domain size folder not exceeding 2GB or 3GB max as
this can be problematic when taking a backup of the domain, so
any means to clean up unwanted files should be done prior to
upgrade

 Ensure that in the node yaml file:

/u01/puppet/environments/${PROJECT}_$
{ENVIRONMENT}/hieradata/nodes/${PROJECT}_${ENVIRONMENT}/
$hostname.crb.apmoller.net.yaml
jdk, patches, source repo entries are removed(take backup of
the file before update):

wls_jdk_version:
wls_jdk_full_version:
jdk_package:
crypto_extension_file:

opatch_instances:
'27342434':
....

wls_source: &wls_source "/u02/oracle/software"

 If the domain has USI Realm configuration(regardless if it is

enabled or disabled): USI configuration must be removed from
the domain config.xml before we can proceed to next step or the
upgrade will fail:
Note: This is only applicable on the Admin node, not Managed
nodes

 Take backup of the config.xml file before then

remove the USI realm definition from the weblogic domain
config file, the usi entry in the
config.xml(/u01/oracle/config/domains/<domain_name>/config)
is:

  <realm>

      <sec:authentication-provider
xmlns:sspi="http://sspi.security.apmoller.net"
xsi:type="sspi:msl-authenticatorType">
        <sec:name>MSLAuthenticator</sec:name>
        <sec:control-flag>SUFFICIENT</sec:control-flag>
      </sec:authentication-provider>
………….
      <sec:name>USI</sec:name>
      <sec:delegate-m-bean-
authorization>false</sec:delegate-m-bean-authorization>
    </realm>

 and ensure default realm is set to myrealm instead of USI

Classification: Internal
 <default-realm>myrealm</default-realm>

 The following pre-checks have been results of issues and

learned lessons faced in previous upgrade tests as it turns
out some envs are missing some data/config that cause the
upgrade to fail:

Issue Description Solution

ID
1 Missing temporary Ask DB Ops to check if it exist and create it
tablespace in OSB PDBs: if not there
<OSBPDB>_IAS_TEMP

2 Missing files on For some reason, when installing 12214 the

/u01/oracle/product/fmw1 folder in question isn’t always the same!
2214/soa/common/sql check the size by “du -s” if it doesn’t match
73908, then you need to copy from a valid
installation
3 Missing right on Ask the DBA to check if below rules are there,
XXX_SOAINFRA schema if not add them:
GRANT EXECUTE ON DBMS_LOCK TO XXXX_SOAINFRA;
GRANT CREATE JOB TO XXXX_SOAINFRA;
GRANT CREATE EXTERNAL JOB TO XXXX_SOAINFRA;
GRANT SELECT ON GV_$SESSION TO XXXX_SOAINFRA
4 XXX_SOAINFRA is INVALID Check if it is in INVALID by:
SELECT * FROM SCHEMA_VERSION_REGISTRY WHERE
owner= ‘XXX_SOAINFRA’
If it is invalid, update it:
UPDATE SCHEMA_VERSION_REGISTRY SET STATUS =
'VALID' WHERE OWNER='xxx_SOAINFRA';
5 In domain config.xml, Add the below to config.xml by targeting the
the app-deployment app to ms1(update accordingly to fit the ms1 of
"Service Bus Domain your domain)
Singleton Marker <app-deployment>
<name>Service Bus Domain Singleton Marker
Application" is not Application</name>
target to any server <target>esb_osb_2_int2_ms1</target>
<module-type>ear</module-type>
<source-
path>/u01/oracle/product/fmw1221/osb/lib/apps/domainsinglet
onmarker.ear</source-path>
<deployment-order>31</deployment-order>
<security-dd-model>DDOnly</security-dd-model>
<staging-mode>nostage</staging-mode>
<plan-staging-mode xsi:nil="true"></plan-staging-mode>
<cache-in-app-directory>false</cache-in-app-directory>
</app-deployment>

Classification: Internal
 3.2.3 Domain Upgrade Process (Perform with oracle user)

a. Prerequisites:
-> putty (configure it with X11-Forward)

->Xming installed
-> sysdba user on OSB PDBs
-> oracle user pwd to ssh to VM hosting the admin server
-> a guaranteed restore point has been captured online by DB Ops
on OSB PDBs before the upgrade kicks in (step d)

Classification: Internal
 b. Upgrade Readiness check:

cd /u01/oracle/product/fmw12214/oracle_common/upgrade/bin/
./ua -readiness

Classification: Internal
 Enter the sysdba user/pwd ensure you enter as shown below :

Classification: Internal
Classification: Internal
 c. Stop all domain processes

d. Schema upgrade (GUI) (as oracle)

cd /u01/oracle/product/fmw12214/oracle_common/upgrade/bin/
./ua

Classification: Internal
Classification: Internal
Classification: Internal
Classification: Internal
Classification: Internal
 e. Domain Reconfiguration (as root):

cd /tmp
cp /u01/puppet/environments/${PROJECT}_${ENVIRONMENT}/bootstrap/azure/osb_upgrade_azure.sh .
chmod u+x osb_upgrade_azure.sh
./osb_upgrade_azure.sh -r admin -p <project> -x <prefix> -e <env> -f osb -v 12214
     e.g: ./ osb_upgrade_azure.sh -r admin -p esb -x 1 -e int2 -f osb -v 12214

f. Domain Components upgrade (GUI) (as oracle)

cd /u01/oracle/product/fmw12214/oracle_common/upgrade/bin/
./ua

Classification: Internal
Classification: Internal
Classification: Internal
 g. If the domain is USI-secured

After the upgrade is completed on admin node and before moving

to managed node (HA set up), the new config.xml of 12214 domain
should be updated by adding back the USI entry we removed in
the pre-checks section and resetting default-realm to USI with
admin server restart then move to the managed node by node.

h. Pack the domain on Admin Node

->required only for HA env so the domain can be unpacked on
managed nodes:

puppet apply --test --debug -e "include maersk::oracle::weblogic::packdomain"

i. Upgrade on managed nodes (as root)

 should be executed in parallel in managed nodes to reduce
downtime ):
Do NOT run on the node having admin server
Prerequisites:
-> Admin server is running on admin
-> Ensure below folder is empty on every managed node
:/u02/oracle/config/applications/<domain_name>/

cd /tmp
cp /u01/puppet/environments/${PROJECT}_${ENVIRONMENT}/bootstrap/azure/osb_upgrade_azure.sh .
chmod u+x osb_upgrade_azure.sh
./osb_upgrade_azure.sh -r managed -p <project> -x <prefix> -e <env> -f osb -v 12214

e.g : ./osb_upgrade_azure.sh -r managed -p esb -x 1 -e int2 -f osb -v 12214

Post-upgrade (as oracle):

-> start node manager: startnm
->ensure the below startup scripts are pointing to correct managed
server name running on the node you are upgrading:
/u02/oracle/scripts/stopms.py
/u02/oracle/scripts/startms.py

i. Start up the domain

 start node manager by startnm if not already done

 Start admin server by startadm if not already up
 In every managed node, start managed server by startms

Classification: Internal
 First start up of managed node will fail but it will create the server
folder but it will be missing the startup properties.
copy those from the backup domain:

/u02/oracle/backups/<backedup _domain>>/servers/<ms_name>/data/nodemanager

to:
/u01/oracle/config/domains/ <domain_folder>/servers/<ms_name>/data/nodemanager

then restart the node manager and start the managed server it
again

3.2.4 Restore Procedure

 Ensure the below folders are empty:

- Admin node to: /u01/oracle/config/domains/ &
/u01/oracle/config/applications/<domain_name>/
- Managed Node to :/u02/oracle/config/domains/ &
/u02/oracle/config/applications/<domain_name>/
 All node manager, admin or managed servers’ processes are
shutdown
 Copy the backups from :
/u02/oracle/backups/<domain_name>_$date and
/u02/oracle/backups/<domain_name>_app_$date respectively
to
- Admin node: /u01/oracle/config/domains/<domain_name> &
/u01/oracle/config/applications/<domain_name>
- Managed Node to :/u02/oracle/config/domains/<domain_name>
& /u02/oracle/config/applications/<domain_name>/
 Update the /home/oracle/.bash_profile to point to fmw1221
instead of fmw12214 , then source the file
 Ask DBA to restore the PDBs from the restore point taken
before the upgrade
 Simply restart your domain by starting first node
manager(s),admin server and managed server(s)

4.3 BPEL

Classification: Internal