Friday, April 8, 2022

Apply R12.2 January 2022 CPU patches

Oracle E-Business Suite Release 12.2 Critical Patch Update Availability Document (January 2022) (Doc ID 2815550.1) gives al details on applying January 2022 CPU patches. The document is not easy to follow. An easy way is to run the latest ETCC script checkMTpatch.sh from patch 17537119. It will list all patches needed for Technology Stack Components.

Download and copy all .zip files to a shard location /u46/ora/R122patch_PGRND/CPU_JAN22
$ ZIPfolder=/u46/ora/R122patch_PGRND/CPU_JAN22
$ echo $ZIPfolder
$ cd $ZIPfolder

Back up database and EBS file systems on all nodes. 
Shutdown all EBS services. 

1. Oracle Fusion Middleware 10.1.2.3 ORACLE_HOME (Table 3)

26825525 (CPUOct2017. Applied)
32922089 (For patching Oracle Forms))

$ vi /etc/oraInst.loc
$ echo $FILE_EDITION
run
$ echo $ORACLE_HOME
/aebsu07/app/AEBSGOLD/fs1/EBSapps/10.1.2
$ which opatch
$ cd /aetnas46/oracmprod/R122patch_IFSPGRND/CPU_JAN22
$ unzip p32922089_101232_LINUX.zip
$ cd 32922089
$ opatch apply
... ...
Removing patch 26825525...
Applying patch 32922089...
OPatch succeeded.

$ opatch lsinventory
44 patches.

Do not see file orion-web.xml asked in post-patch instruction. Skip it by assuming it does not apply to EBS.

2. Oracle Fusion Middleware 11.1.1.9 - Oracle Common Home (Table 3)
Patch 31985571 (A superset of patch 30358663. Already applied)
Patch 26933408 (for patching JDBC)

$ export ORACLE_HOME=$FMW_HOME/oracle_common
$ export PATH=$ORACLE_HOME/OPatch:$PATH
$ echo $ORACLE_HOME
$ which opatch
$ cd /aetnas46/oracmprod/R122patch_IFSPGRND/CPU_JAN22
$ unzip p26933408_111190_Generic.zip
$ cd 26933408
$ opatch apply
... ...
Patching component oracle.jrf.thirdparty.jee, 11.1.1.9.0...
Verifying the update...
Patch 26933408 successfully applied

$ opatch lsinventory
Interim patches (14) : ...

3. Oracle Fusion Middleware 11.1.1.9 - Web Tier Home (4 patches, Table 3)

32287205 (OSS BUNDLE PATCH 11.1.1.9.210420)
33311587 (OHS 11.1.1.9.0 SPU FOR OctCPU2021)
23716938 (OPMN patch. Already applied)
32928416 (OPMN patch)
33144848 (OHS PLUGIN 11.1.1.9.0 SPU FOR APRCPU2021)

$ echo $IAS_ORACLE_HOME
/aebsu07/app/AEBSGOLD/fs1/FMW_Home/webtier
$ export ORACLE_HOME=$IAS_ORACLE_HOME
$ echo $ORACLE_HOME
/aebsu07/app/AEBSGOLD/fs1/FMW_Home/webtier
$ export PATH=$IAS_ORACLE_HOME/OPatch:$PATH
$ which opatch
~/AEBSGOLD/fs1/FMW_Home/webtier/OPatch/opatch
$ cd ..
$ unzip p33311587_111190_Linux-x86-64.zip
$ cd 33311587

$ opatch apply -jre $ORACLE_HOME/jdk/jre
... ...
Backing up files...
Rolling back interim patch '31047338' from OH '/aebsu07/app/AEBSGOLD/fs1/FMW_Home/webtier'
Patching component oracle.ohs2, 11.1.1.9.0...
RollbackSession removing interim patch '31047338' from inventory
OPatch back to application of the patch '33311587' after auto-rollback.
Patching component oracle.ohs2, 11.1.1.9.0...
Verifying the update...
Patch 33311587 successfully applied

$ cd ..
$ unzip p32287205_111190_Linux-x86-64.zip
$ cd 32287205
$ opatch apply
... ...
Backing up files...
Rolling back interim patch '31304503' from OH '/aebsu07/app/AEBSGOLD/fs1/FMW_Home/webtier'
Patching component oracle.ldap.rsf, 11.1.1.9.0...
Patching component oracle.rdbms.rsf, 11.1.0.7.0...
RollbackSession removing interim patch '31304503' from inventory
OPatch back to application of the patch '32287205' after auto-rollback.
Patching component oracle.ldap.rsf, 11.1.1.9.0...
Patching component oracle.rdbms.rsf, 11.1.0.7.0...
Verifying the update...
Patch 32287205 successfully applied

$ cd ..
$ unzip p33144848_111190_Linux-x86-64.zip
$ cd 33144848
$ opatch apply -jre $ORACLE_HOME/jdk/jre
... ...
Backing up files...
Patching component oracle.wlsplugins, 11.1.1.9.0...
Verifying the update...
Patch 33144848 successfully applied

$ cd ..
$ unzip p32928416_111190_Linux-x86-64.zip
$ cd 32928416
$ opatch apply
... ...
Backing up files...
Patching component oracle.opmn, 11.1.1.9.0...
Verifying the update...
Patch 32928416 successfully applied

$ opatch lsinventory

Interim patches (17) : ... ...

4. Oracle WebLogic Server 10.3.6.0 (Table 2)

On 2/15/2022,  patches 33796519 and 33845432 were added to Doc ID 2815550.1 (Table 3) for Log4j vulnerability. 

Start a new OS session (to get the correct ORACLR_HOME)
$ cd $ZIPfolder/WLS_CPU

a) Get WLS batch files ready:  6 .zip files (Do this once for all instances) 
Patch 33845432 replaces patch 32898996 to patch Smart Update Tool (bsu) and BUG33681492 - CVE-2021-4104, CVE-2022-23305 AND CVE-2022-23307:
$ unzip -o p33845432_R12_GENERIC.zip
Archive:  p33845432_R12_GENERIC.zip
  inflating: Readme.txt
 extracting: p33800106_1036_Generic.zip

Totally, 5 patches to be applied by bsu:
p31241365_1036_Linux-x86-64.zip [CW7X]
p33494814_1036_Generic.zip [HYG5]
p13964737_10360220118_Generic.zip  [7GCA]
p13845626_10360220118_Generic.zip  [IJC1]
p33796519_10360220118_Generic.zip [TNS1]

$ unzip -o p31241365_1036_Linux-x86-64.zip
Archive:  p31241365_1036_Linux-x86-64.zip
  inflating: CW7X.jar
  inflating: patch-catalog_27153.xml
  inflating: README.txt

$ unzip -o p33796519_10360220118_Generic.zip
Archive:  p33796519_10360220118_Generic.zip
  inflating: README.txt
  inflating: patch-catalog_27832.xml
 extracting: TNS1.jar

But three of them are wrapped by other patch numbers:

$ unzip p33528262_R12_GENERIC.zip
Archive:  p33528262_R12_GENERIC.zip
 extracting: p33494814_1036_Generic.zip  
  inflating: Readme.txt

Patch 33494814: WLS PATCH SET UPDATE (PSU) 10.3.6.0.220118  [HYG5]
$ unzip -o p33494814_1036_Generic.zip  
Archive:  p33494814_1036_Generic.zip
 extracting: HYG5.jar
  inflating: patch-catalog_27794.xml
  inflating: README.html
  inflating: README.txt

Patch 33570839 (Consolidated Patch) has two patches. See
Oracle E-Business Suite Release 12.2: Consolidated List of Patches and Technology Bug Fixes (Doc ID 1594274.1)

$ unzip p33570839_R12_LINUX.zip          
             <= It includes a lot of .zip files. Two of them are needed
cp etcc-bundle/LINUX_X86-64/wls/10.3.6.0.220118/p13964737_10360220118_Generic.zip .
cp etcc-bundle/LINUX_X86-64/wls/10.3.6.0.220118/p13845626_10360220118_Generic.zip .

$ unzip -o p13964737_10360220118_Generic.zip
Archive:  p13964737_10360220118_Generic.zip
  inflating: README.txt
  inflating: 7GCA.jar
  inflating: patch-catalog_27820.xml

$ unzip -o p13845626_10360220118_Generic.zip
Archive:  p13845626_10360220118_Generic.zip
  inflating: README.txt
  inflating: patch-catalog_27818.xml
  inflating: IJC1.jar

In other instances, just copy the files:
$ cp -p p33800106_1036_Generic.zip $FMW_HOME/utils/bsu
$ cp *.jar $FMW_HOME/utils/bsu/cache_dir        <= 5 .jar files.
$ cp *.xml $FMW_HOME/utils/bsu/cache_dir
NOTES: .xml file is necessary to avoid error: Encountered unrecognized patch ID: xxxx

$ ls -altr  $FMW_HOME/utils/bsu/cache_dir/*.jar

b) Fist, apply patch 33800106 (SMART UPDATE TOOL ENHANCEMENT V5) to bsu
NOTE: After this parch was applied, bsu runs much faster, reducing time from 20 minutes to one minute or so for applying one patch.

$ cd $FMW_HOME/utils/bsu
$ unzip p33800106_1036_Generic.zip
Archive:  p33800106_1036_Generic.zip
  inflating: bsu_update.bat
  inflating: bsu_update.sh
   creating: bsu_update/GA/
   creating: bsu_update/GA/modules/
  inflating: bsu_update/GA/modules/com.bea.cie.comdev_6.1.3.0.jar
  inflating: bsu_update/GA/modules/com.bea.cie.patch-client_3.3.0.0.jar
  inflating: bsu_update/GA/modules/com.bea.cie.patch-common_3.3.0.0.jar
  inflating: bsu_update/GA/modules/com.bea.core.apache.log4j_1.2.13.jar
   creating: bsu_update/GA/modules/features/
  inflating: bsu_update/GA/modules/features/com.bea.cie.patch-client_3.3.0.0.jar
  inflating: bsu_update/GA/modules/features/com.bea.cie.patch-client_3.3.0.0.xml
   creating: bsu_update/Patch/
   creating: bsu_update/Patch/modules/
  inflating: bsu_update/Patch/modules/com.bea.cie.comdev_6.1.3.1.jar
  inflating: bsu_update/Patch/modules/com.bea.cie.patch-client_3.3.0.0.jar
  inflating: bsu_update/Patch/modules/com.bea.cie.patch-common_3.3.0.0.jar
  inflating: bsu_update/Patch/modules/com.bea.core.apache.log4j_1.2.13.jar
   creating: bsu_update/Patch/modules/features/
  inflating: bsu_update/Patch/modules/features/com.bea.cie.patch-client_3.3.0.0.jar
  inflating: bsu_update/Patch/modules/features/com.bea.cie.patch-client_3.3.0.0.xml
  inflating: README.txt

$ chmod +x bsu_update.sh
$ bsu.sh -version
Oracle Smart Update. Version: 3.3.0.0
$ ./bsu_update.sh install        <== very quick.
Installing Smart Update V5...
Updating bsu modules
Update was successful.

c)  Apply 5 WLS patches using bsu.sh
$ ls -altr $FMW_HOME/utils/bsu/cache_dir
$ cd $FMW_HOME/utils/bsu
$ echo $ORACLE_HOME
$RUN_BASE/EBSapps/10.1.2
$ echo $FILE_EDITION
run

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=CW7X -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected
Installing Patch ID: CW7X..
Result: Success

Optional:
$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=HYG5 -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
Conflict(s) detected - resolve conflict condition and execute patch installation again
Conflict condition details follow:
Patch HYG5 is mutually exclusive and cannot coexist with patch(es): YVDZ,1YWL

$ ./bsu.sh -remove -patchlist=YVDZ,1YWL -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected
Removing Patch ID: YVDZ..
Result: Success
Removing Patch ID: 1YWL..
Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=HYG5 -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected
Installing Patch ID: HYG5..
Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=IJC1,7GCA -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected
Installing Patch ID: IJC1.
Result: Success
Installing Patch ID: 7GCA..
Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=TNS1 -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..............
No conflict(s) detected
Installing Patch ID: TNS1..
Result: Success

$ ./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -view -status=applied
ProductName:       WebLogic Server
ProductVersion:    10.3 MP6
Components:        WebLogic Server/Core Application Server,WebLogic Server/
                   Administration Console,WebLogic Server/Configuration Wizard and
                   Upgrade Framework,WebLogic Server/Web 2.0 HTTP Pub-Sub Server
                   WebLogic Server/WebLogic SCA,WebLogic Server/WebLogic JDBC
                   Drivers,WebLogic Server/Third Party JDBC Drivers,WebLogic Server/
                   WebLogic Server Clients,WebLogic Server/WebLogic Web Server
                   Plugins,WebLogic Server/UDDI and Xquery Support,WebLogic
                   Server/Evaluation Database,WebLogic Server/Workshop Code
                   Completion Support
BEAHome:           $FMW_HOME
ProductHome:       $FMW_HOME/wlserver_10.3
PatchSystemDir:   $FMW_HOME/utils/bsu
PatchDir:               $FMW_HOME/patch_wls1036
Profile:                  Default
DownloadDir:       $FMW_HOME/utils/bsu/cache_dir
JavaVersion:         1.6.0_29
JavaVendor:          Sun

Patch ID:          32I2
Patch ID:          S9MN (14272383)
Patch ID:          TTEM (14621810)
Patch ID:          7FC9 (17893334)
Patch ID:          S8H6 (20474010)
Patch ID:          DI8E (22323006)
Patch ID:          DN1F (26795917)
Patch ID:          CW7X (31241365)
Patch ID:          HYG5 (33494814)
Patch ID:          IJC1 (13845626)
Patch ID:          7GCA (13964737)
Patch ID:          TNS1 (33796519)

Run checkMTpatch.sh again to confirm all recommended patches are applied.

5. Apply EBS patches (Table 1 & 2)
Patch 33487428 CPU PATCH FOR JAN 2022 
Notes: its adgrants.sql is older that $APPL_TOP/admin/adgrants.sql
Patch 33207251:R12.FND.C   (included in 33487428)
Patch 33168664:R12.FWK.C
Patch 32980025:R12.AK.C
Patch 33286000:R12.OAM.C
Patch 29631318:R12.OWF.C  (already applied)
Patch 32564885:R12.OWF.C
Patch 33625264:R12.OWF.C (replaces 33559216)
Notes: skip below two WF patches, since we did not apply Jan 2021 CPU patch
Patch 32117360:R12.OWF.C
Patch 32460404:R12.OWF.C

Copy and unzip 6 patch files (from $ZIPfolder/EBS_CPU) to all nodes:
$ cd $PATCH_TOP
$ cp -p $ZIPfolder/EBS_CPU/*.zip .
$ ls -altr

unzip p33487428_12.2.0_R12_LINUX.zip
unzip p33168664_R12.FWK.C_R12_GENERIC.zip
unzip p32980025_R12.AK.C_R12_GENERIC.zip
unzip p33286000_R12.OAM.C_R12_GENERIC.zip
unzip p32564885_R12.OWF.C_R12_GENERIC.zip
unzip p33625264_R12.OWF.C_R12_GENERIC.zip

$ echo $ORACLE_HOME
$ echo $FILE_EDITION
run

$ adop phase=apply apply_mode=downtime patches=33487428 workers=8 patchtop=$NE_BASE/EBSapps/patch          
                                                                    <== takes 30 minutes on each node

Post-patch steps (from README):
a) Run autoConfig in RUN file system on all Apps-tie nodes
$ ./adautocfg.sh     

b) $ perl $AD_TOP/bin/admkappsutil.pl

c) Run autoConfig on database node
$ adconfig.sh contextfile=$ORACLE_HOME/.../appsutil/$CONTEXT_NAME.xml

Optionally, start Apps services on all nodes for a check-out.

Apply other 5 patches:
$ adop phase=apply apply_mode=downtime patches=33168664,32980025 workers=8 patchtop=/aebsu07/app/AEBSGOLD/fs_ne/EBSapps/patch

$ adop phase=apply apply_mode=downtime patches=33286000,32564885,33625264 workers=8 patchtop=/aebsu07/app/AEBSGOLD/fs_ne/EBSapps/patch

Confirm all 8 patches were applied on all nodes.
SQL> SELECT adb.bug_number, aas.name appl_top_name, adb.creation_date, adb.language,
decode(ad_patch.is_patch_applied('R12',aas.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status
FROM ad_bugs adb,
(select distinct appltop_id appl_top_id, node_name name from ad_adop_sessions 
           where node_name in (select node_name from ADOP_VALID_NODES)
           ) aas
where adb.bug_number in (
'33487428',
'33207251',
'33168664',
'32980025',
'33286000',
'29631318',
'32564885',
'33625264'
) order by creation_date desc, adb.bug_number,aas.name,adb.language ;

6. Start Apps services

7. Run FS_CLONE and make sure it completes successfully.
$ adop phase=fs_clone


No comments: