Friday, August 11, 2023

Apply R12.2 July 2023 CPU patches

Oracle E-Business Suite Release 12.2 Critical Patch Update Availability Document (July 2023) (Doc ID 2953580.1) is the document for July 2023 CPU patches.

Check to find ATG_PF.C.delta.9 is the latest ATG (Application Technology Family) patch in my instances.

SQL> select * from ad_bugs where bug_number in (31856779, 30399994, 33527666);

30399994 Patch R12.ATG_PF.C.delta.9 (Application Technology Family)
31856779 Patch R12.ATG_PF.C.delta.10 
33527666 Patch R12.ATG_PF.C.Delta.11 
                 (Released on Jan 9 2023. See Doc ID 2876714.1 on how to apply it)

1. Pre-requisites: R12.AD.C.Delta.14 and R12.TXK.C.Delta.14 patches

Follow steps in Apply AD-TXK Delta 14 RUPs to apply them. After all are done, 8 patches were applied:

SQL> SELECT adb.bug_number, aas.name appl_top_name, adb.creation_date, adb.language,
decode(ad_patch.is_patch_applied('R12',aas.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status
FROM ad_bugs adb,
(select distinct appltop_id appl_top_id, node_name name from ad_adop_sessions 
           where node_name in (select node_name from ADOP_VALID_NODES) ) aas
where adb.bug_number in (
'33600809',  -- R12.AD.delta.C.14
'34668508',
'34669333',
'34681299',
'35280947',
'33602997',  -- R12.TXK.delta.C.14
'34708635',
'34654260'
) order by creation_date desc, adb.bug_number,aas.name,adb.language;

2. Apply July 2023 CPU patch 35385938. It takes about 30 minutes in the multi-node.

First of all, download all EBS patches to a shared location /a46/CPUs/CPU_July2023

$ echo $FILE_EDITION
run

$ adop phase=apply apply_mode=downtime patches=35385938 patchtop=/a46/CPUs/CPU_July2023/CPU

3. Run autoconfig
$ perl $AD_TOP/bin/admkappsutil.pl
$ cp -p $INST_TOP/admin/out/appsutil.zip $APPLPTMP

Autoconfig on db node first
Autoconfig on all apps nodes

NOTES: After CPU patching, custom .tmp file in $FND_TOP/admin/template/custom may be out-of-date and needs an update if existing to avoid autoconfig fails on apps nodes.

4. Post patches:  It turns out that it needs to apply only 5 post patches in my instances.

$ adop phase=apply apply_mode=downtime patches=34212478,34556525,34291981,35181823,35355008 patchtop=/a46/CPUs/CPU_July2023/posts

After that, verify all 11 patches were applied:

SQL> SELECT adb.bug_number, aas.name appl_top_name, adb.creation_date, adb.language,
decode(ad_patch.is_patch_applied('R12',aas.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status
FROM ad_bugs adb,
(select distinct appltop_id appl_top_id, node_name name from ad_adop_sessions 
           where node_name in (select node_name from ADOP_VALID_NODES) ) aas
where adb.bug_number in (
'35181823',    -- post patch   (Table 1)
'35355008',    -- post patch   (Table 1)
-- '33207251',  -- Post-CPU EBS Security Patches  (Table 2)
-- '32980025',  -- Post-CPU EBS Security Patches
-- '33286000',  -- Post-CPU EBS Security Patches
'34212478',     -- Post-CPU EBS Security Patches
'34556525',     -- Post-CPU EBS Security Patches
'34291981'      -- Post-CPU EBS Security Patches
-- '33625264',  -- Post-CPU EBS Security Patches
-- '33862025',  -- needed for AD.C.Delta.13 and TXK.Delta.13. Comes after CPU 35385938
-- '33568131'   -- needed for AD.C.Delta.13 and TXK.Delta.13. Comes after CPU 35385938
) order by creation_date desc, adb.bug_number,aas.name,adb.language;

Optional: start apps services to verify all work.

6. Run ETCC script checkMTpatch.sh to get the list of patches asked by ETCC (patch 17537119). But, for WebLogic, Table 4 of Doc ID 2953580.1 asks to apply Oracle WebLogic Server 10.3.6.0 patch 34900723 (WLS PSU 10.3.6.0.230418). 

Download directly patch file p34900723_1036_Generic.zip and copy it to $FMW_HOME/utils/bsu/cache_dir.

$ cd $FMW_HOME/utils/bsu/cache_dir

$ unzip -o p34900723_1036_Generic.zip
Archive:  p34900723_1036_Generic.zip
 extracting: P8S7.jar
  inflating: patch-catalog_27964.xml
  inflating: README.html
  inflating: README.txt

$ cd $FMW_HOME/utils/bsu
$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=P8S7 -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
Conflict(s) detected - resolve conflict condition and execute patch installation again
Conflict condition details follow:
Patch P8S7 is mutually exclusive and cannot coexist with patch(es): HYG5,TNS1,IJC1,7GCA

$ ./bsu.sh -remove -patchlist=IJC1,7GCA,TNS1 -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected

Removing Patch ID: IJC1.
Result: Success

Removing Patch ID: 7GCA..
Result: Success

Removing Patch ID: TNS1..
Result: Success

$ ./bsu.sh -remove -patchlist=HYG5 -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected

Removing Patch ID: HYG5..
Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=P8S7 -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected

Installing Patch ID: P8S7..
Result: Success

-- verify it
$ ./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -status=applied -verbose -view | grep P8S7
Patch ID:          P8S7
PatchContainer:    P8S7.jar

6. Run ETCC script checkMTpatch.sh to identify missing technology patches. 

Starting Application Tier Technology Codelevel Checker
Version: 120.0.12020000.65

Bugfix XML file version: 120.0.12020000.61
This file will be used for identifying missing bugfixes.

Mapping XML file version: 120.0.12020000.49
This file will be used for mapping bugfixes to patches.
... ...
The default patch recommendations to install these missing bugfixes are:
-------------------------------------------------------------------------------
Oracle Fusion Middleware (FMW) - Web Tier 11.1.1.9.0
-------------------------------------------------------------------------------
  Patch 35540062
    - Filename: p35540062_111190_Linux-x86-64.zip
  Patch 34067016
    - Filename: p34067016_11119210420OSS_Linux-x86-64.zip

-------------------------------------------------------------------------------
Oracle Fusion Middleware (FMW) - oracle_common 11.1.1.9.0
-------------------------------------------------------------------------------
  Patch 34330735
    - Filename: p34330735_111190_Generic.zip

-------------------------------------------------------------------------------
Oracle WebLogic Server (WLS) 10.3.6.0.230418
-------------------------------------------------------------------------------
  Patch 13845626 [SU Patch [AMGE]]
    - Filename: p13845626_10360230418_Generic.zip
  Patch 34856303 [SU Patch [ENUU]]
    - Filename: p34856303_1036_Generic.zip
  Patch 31042881 [SU Patch [7BIA]]
    - Filename: p31042881_1036_Generic.zip

7. Continue to apply WLS patches (after 3 patches files were downloaded and copied to $FMW_HOME/utils/bsu/cache_dir)

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=AMGE -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected

Installing Patch ID: AMGE.
Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=ENUU -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
Conflict(s) detected - resolve conflict condition and execute patch installation again
Conflict condition details follow:
Patch ENUU is mutually exclusive and cannot coexist with patch(es): DN1F

$ ./bsu.sh -remove -patchlist=DN1F -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected

Removing Patch ID: DN1F.
Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=ENUU -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected

Installing Patch ID: ENUU.
Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=7BIA -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected

Installing Patch ID: 7BIA.
Result: Success

8. Apply patches from ETCC list to FMW homes.
Apply 35540062 and 34067016 to Oracle Fusion Middleware (FMW) - Web Tier
Note: 35540062 (OHS Security Patch Update JulCPU2023 Patch) will roll back 34867865 (JanCPU2023 Patch) if it was applied before.

$ echo $IAS_ORACLE_HOME
$ export ORACLE_HOME=$IAS_ORACLE_HOME
$ echo $ORACLE_HOME          <== $FMW_Home/webtier
$ export PATH=$IAS_ORACLE_HOME/OPatch:$PATH
$ which opatch

$ echo $FILE_EDITION
run

$ opatch lsinventory | grep 31304503

$ opatch lsinventory
Interim patches (17)     <== depending on patching history

$ cd /a46/CPUs/CPU_July2023/tech

$ unzip p34067016_11119210420OSS_Linux-x86-64.zip
$ cd 34067016
$ opatch apply
... ...
Patching component oracle.ldap.rsf, 11.1.1.9.0...
Patching component oracle.rdbms.rsf, 11.1.0.7.0...

Verifying the update...
Patch 34067016 successfully applied

$ cd ..
$ rm -rf 34067016

$ cd 35540062
$ opatch apply

Patching component oracle.ohs2, 11.1.1.9.0...
RollbackSession removing interim patch '33311587' from inventory
OPatch back to application of the patch '35540062' after auto-rollback.
Patching component oracle.ohs2, 11.1.1.9.0...
Verifying the update...
Patch 35540062 successfully applied

$ cd ..
$ rm -rf 35540062

$ opatch lsinventory
Interim patches (18) 
.. ...

9. Apply patch 34330735 to Oracle Fusion Middleware (FMW) - Common

$ export ORACLE_HOME=$FMW_HOME/oracle_common
$ export PATH=$ORACLE_HOME/OPatch:$PATH

$ echo $ORACLE_HOME
.../FMW_Home/oracle_common
$ which opatch
.../FMW_Home/oracle_common/OPatch/opatch

$ cd /a46/CPUs/CPU_July2023/tech
$ opatch lsinventory
Interim patches (14)

$ unzip p34330735_111190_Generic.zip
$ cd 34330735/oui
$ ls
$ opatch apply
... ...
RollbackSession removing interim patch '31985571' from inventory
OPatch back to application of the patch '34330735' after auto-rollback.
Patching component oracle.jrf.adfrt, 11.1.1.9.0...

Verifying the update...
Patch 34330735 successfully applied

cd ..
rm -rp 34330735

$ opatch lsinventory
Interim patches (14)

By now ETCC script shows all patches were applied.

10. Upgrade JDK to 1.7.0_391

See Upgrade JDK 7 in EBS R12.2 for detail on upgrading JDK. Table 3 in CPU release document 2953580.1 shows JDK 1.7.0_391 is the latest JDK for EBS. Use the link in Doc ID 2957260.1 (Oracle Critical Patch Update (CPU) Jul 2023 for Oracle Java SE) to download patch 35374159 files  p35374159_170391_LINUX.zip & p35374159_170391_Linux-x86-64.zip and save them to /a46/CPUs/CPU_July2023/jdk. 

$ unzip p35374159_170391_Linux-x86-64.zip
$ tar vzxfp jdk-7u391-linux-x64.tar.gz
$ mv jdk1.7.0_391 jdk_391_64

$ unzip p35374159_170391_LINUX.zip
$ tar vzxfp jdk-7u391-linux-i586.tar.gz
$ mv jdk1.7.0_391 jdk_391_32

$ echo $FILE_EDITION
run

JDKfolder=/a46/CPUs/CPU_July2023/jdk

cd $COMMON_TOP/util
tar -cvzf jdk64_BK.tar.gz jdk64
tar -cvzf jdk32_BK.tar.gz jdk32
rm -fr jdk64
rm -fr jdk32
cp -rp $JDKfolder/jdk_391_64 jdk64
cp -rp $JDKfolder/jdk_391_32 jdk32

cd $FMW_HOME/webtier
tar -cvzf jdk64_BK.tar.gz jdk
rm -fr jdk
cp -rp $JDKfolder/jdk_391_64 jdk

cd $ORACLE_HOME
ls
tar -cvzf jdk32_BK.tar.gz jdk
ls -al
rm -fr jdk
cp -rp $JDKfolder/jdk_391_32 jdk

$ADJVAPRG -version
$AFJVAPRG -version

cd $ORACLE_HOME/forms/lib
make -f ins_forms.mk sharedlib install
cd $ORACLE_HOME/reports/lib
make -f ins_reports.mk install

11. Upgrade JRE (Java Run-time Environment)

See 4) in Enable JWS with EBS R12.2 for JRE upgrade. Doc ID 2957260.1 also gives a link to download JRE 8 Update 381 (patch 35374151). I used file p35374151_180381_WINNT.zip (for Windows 32-bit).

$ cd /a46/CPUs/CPU_July2023/jre
$ unzip p35374151_180381_WINNT.zip
Archive:  p35374151_180381_WINNT.zip
  inflating: jre-8u381-windows-i586.exe
  inflating: jre-8u381-windows-i586.tar.gz
  inflating: readme.txt

$ cp jre-8u381-windows-i586.exe $COMMON_TOP/webapps/oacore/util/javaplugin/j2se18381.exe
$ cd $COMMON_TOP/webapps/oacore/util/javaplugin
$ ls -al
$ echo $FILE_EDITION
run

$ $FND_TOP/bin/txkSetPlugin.sh 18381
 
$ grep sun $CONTEXT_FILE
         <sun_plugin_ver oa_var="s_sun_plugin_ver">1.8.0_381</sun_plugin_ver>
         <sun_plugin_type oa_var="s_sun_plugin_type">jdk</sun_plugin_type>
         <sun_clsid oa_var="s_sun_clsid">CAFEEFAC-0018-0000-0381-ABCDEFFEDCBA</sun_clsid>

12. Optional: Re-generate and sign JAR files.
$ adadmin   (1 => 4 => Yes)
13. Start Apps services for test
$ adstrtal.sh apps/xxxx
14. adop phase=fs_clone  (sync RUN file system to PATCH file system)

Saturday, August 5, 2023

vulnerabilities by Qualys report

Qualys Scan reports huge number of vulnerabilities in our EBS R12.2.10 servers where JDK was upgraded to 1.7.0_331. Vulnerability issue became a priority task for us due to corporate security requirement.

$ $ADJVAPRG -version
java version "1.7.0_331"
Java(TM) SE Runtime Environment (build 1.7.0_331-b06)
Java HotSpot(TM) Server VM (build 24.331-b06, mixed mode)

$ $AFJVAPRG -version
java version "1.7.0_331"
Java(TM) SE Runtime Environment (build 1.7.0_331-b06)
Java HotSpot(TM) 64-Bit Server VM (build 24.331-b06, mixed mode)

Some vulnerabilities can be addressed by applying latest CPU patches, upgrading JDK and JRE (see details). 

Vulnerabilities also reported:

$COMMON_TOP/clone/FMW/t2pjdk/bin/java 1.7.0_85-b15 Enhanced
$COMMON_TOP/clone/FMW/t2pjdk/jre/bin/java 1.7.0_85-b15 Enhanced
(It is safe to delete above folders because they will be recreated by an adpreclone.pl run)
$COMMON_TOP/clone/jre/bin/java 1.7.0_331-b06 Enhanced  
(It is not in JDK folders and so a JDK upgrade will not fix it. But it may be recreated by an adpreclone.pl run)
$COMMON_TOP/adopclone_nodeName/FMW/t2pjdk/jre/bin/java 1.7.0_85-b15 Enhanced
$COMMON_TOP/adopclone_nodeName/FMW/t2pjdk/bin/java 1.7.0_85-b15 Enhanced
$COMMON_TOP/adopclone_nodeName/jre/bin/java 1.7.0_85-b15 Enhanced
($COMMON_TOP/adopclone_nodeName may get created/updated from ADOP run, such as fs_clone. nodeName is the EBS host, where ADOP (or adpreclone.pl?) ran. )
$APPL_TOP/msc/12.0.0/bin/SNO/scp/12.2/sno/installer_jre/bin/java 1.6.0_12-b04 Enhanced
$APPL_TOP/msc/12.0.0/bin/PS/scp/12.2/ps/jre/bin/java 1.8.0_92-b14 Enhanced

Oracle support confirmed that they can be deleted and some of them will be recreated next time adpreclone.pl is run. I wrote a script to clean them on RUN file system (and on PATCH file system or run FS_CLONE). 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# delete Java files for vulnerabilities by Qualys report

curr=`pwd`
echo "current path: $curr"
cd $COMMON_TOP/clone/FMW/t2pjdk/bin
pwd
ls -al java
rm -f java

cd $COMMON_TOP/clone/FMW/t2pjdk/jre/bin
pwd
ls -al java
rm -f java

cd $COMMON_TOP/clone/jre/bin
pwd
ls -al java
rm -f java

cd $APPL_TOP/msc/12.0.0/bin/SNO/scp/12.2/sno/installer_jre/bin
pwd
ls -al java
rm -f java

cd $APPL_TOP/msc/12.0.0/bin/PS/scp/12.2/ps/jre/bin
pwd
ls -al java
rm -f java

# loop folder(s) $COMMON_TOP/adopclone_*
for x in `ls -ald $COMMON_TOP/adopclone_* |egrep "^drw" |awk '{print $9}'`
do
echo Under path: $x
cd $x/jre/bin
pwd
ls -al java
rm -f java
cd $x/FMW/t2pjdk/jre/bin
pwd
ls -al java
rm -f java
cd $x/FMW/t2pjdk/bin
pwd
ls -al java
rm -f java
done

cd $curr
exit