Friday, May 16, 2025

Apply EBS January 2025 CPU patches

The first and important step in applying EBS CPU patches is to identify which patches are required for each EBS components. After step 1 below, the patching process is the same for different CPU releases. I used my shell scripts to perform two steps for January 2025 CPU, which make the process much easier and quicker.

1. Download patch files.
Oracle E-Business Suite Release 12.2 Critical Patch Update Availability Document (January 2025) (Doc ID 3061170.1) is the document for applying January 2025 CPU patches. Run checkers and read the CPU document to decide which patches are required. Then, download and save them to a shared location /path/to/Jan2025_CPU that will be used for all nodes and different instances (Dev, QA and Prod, etc). This folder has 9 sub-folders for different purposes:

$ cd /path/to/Jan2025_CPU
$ ls -d *
EBS ECPUC EJCPUC ETCC FMW_Comm FMW_Ora JDK JRE WLS

- EBS  hold 3 patches required by ECPUC.sql. We decided to apply only 3 patches because other modules are not used in our EBS instances.

$ cd EBS
$ ls -trd * | egrep -i '36560216|37078855|37237361'
p37237361_12.2.0_R12_LINUX.zip
p37078855_R12.ATG_PF.C_R12_GENERIC.zip
p36560216_R12.PO.D_R12_GENERIC.zip
36560216
37237361
37078855

# grant permission once for ADOP to create multiple sub-folders under backup folder:
$ chmod 777 36560216/backup
$ chmod 777 37237361/backup
$ chmod 777 37078855/backup

- ECPUC  hold ECPUC.sql (from patch p35583866) for checking which EBS patches are required.

$ grep '$Header' *.sql
REM $Header: ECPUC.sql 120.0.12020000.10 2025/03/06 10:04:57 spullach noship $

- EJCPUC  hold ejcpuc.sh from patch p37171025 

$ grep '$Header' ejcpuc.sh
rcs="$Header: ejcpuc.sh,v 1.3 2025/01/21 04:30:00 egravers Exp egravers $"

Run ejcpuc.sh to confirm JDK upgrade is needed:
$ ./ejcpuc.sh
###############################################################
Checking Apptier Java 7 for CPU 2025.01 on Platform Linux_x64 - need 1.7.0_451
2025-05-xx 11:41:30 EDT on node_name.domain.com
###############################################################
2025.01    action    Your Version   bitness   Java Location
----------  -------- ---------------- -------- ------------------
1.7.0_451 UPDATE 1.7.0_391 32-bit $RUN_BASE/EBSapps/10.1.2/jdk/bin/java
1.7.0_451 UPDATE 1.7.0_391 32-bit $RUN_BASE/EBSapps/comn/util/jdk32/bin/java
1.7.0_451 UPDATE 1.7.0_391 64-bit $RUN_BASE/EBSapps/comn/util/jdk64/bin/java1
1.7.0_451 UPDATE 1.7.0_391 64-bit $RUN_BASE/FMW_Home/webtier/jdk/bin/java
Follow 1530033.1 to update the JDK(s). Your application tier JDK 7 is lower than the 1.7.0_451 update released in CPU 2025.01.

- ETCC  hold ETCC scripts from patch p17537119
Make sure apps and database use the scripts from the same p17537119 release/download.
$ grep '$Header' *.sh
checkDBpatch.sh:# $Header: checkDBpatch.sh 120.133 2025/01/30 17:56:04 chrhill noship $
checkMTpatch.sh:# $Header: checkMTpatch.sh 120.0.12020000.70 2025/01/30 18:28:36 chrhill noship $

- FMW_Comm  holds 3 patch files that required by ETCC
$ ls *.*
ocm.rsp
p33974106_111190_Generic.zip
p33960746_111190_Generic.zip
p34714760_111190_Generic.zip
33974106
33960746
34714760
apply_EBStechPatches.sh

Also two custom files were copied here:
apply_EBStechPatches.sh
ocm.rep 

- FMW_Ora  holds patch files for Oracle Home of forms and report (if required).
No required patches by January 2025 CPU release for my instances.

- JDK  holds files needed for JDK upgrades.
From Doc ID 3061170.1, Go to Table 3: Security Patches for Technology Stack Components With a New Patch in This CPU.

Document ID 3066051.1 (Oracle Critical Patch Update (CPU) Jan 2025 for Oracle Java SE) has a link (on the low end) to download JDK 7 Update 451 Restricted: Patch 37308812 for JDK 1.7.0_451.
p37308812_170_451_Linux-x86-64.zip
p37308812_170_451_LINUX.zip 

Then unzip them to get two JDK files:
jdk-7u451-linux-i586.tar.gz 
jdk-7u451-linux-x64.tar.gz

Also copy the custom script here.
JDK_upgrade1_7_xxx.sh

- JRE  holds files for JRE upgrade on the server.
Document ID 3066051.1 (Oracle Critical Patch Update (CPU) Jan 2025 for Oracle Java SE) also has a link for JRE 8 Update 441: Patch 37308802. Download  p37308802_180441_WINNT.zip (for Microsoft Windows 32-bit), and unzip to see two files:
jre-8u441-windows-i586.zip
jre-8u441-windows-i586.exe

- WLS  holds 3 WLS patch files required by ETCC and CPU Doc ID 3061170.1
13845626/p13845626_10360231017_Generic.zip
35476084/p35476084_1036_Linux-x86-64.zip
35586779/p35586779_1036_Generic.zip

2. Apply database patches

Run ETCC checker to find the required patches, and then apply them. Run ejcpuc.sh to make sure Java versions in database are up to date.

3. Backups
Back up apps filesystems, hold scheduled jobs if necessary. Also create a GRP in database.

4. Apply EBS patches
$ adop - status        <= make sure fs_clone worked successfully.
$ vi /etc/oraInst.loc
$ stop app services   (and check passwordless ssh works if multi-nodes are used)

$ cd /path/to/Jan2025_CPU/ECPUC
$ sqlplus apps/
SQL> @ECPUC.sql

$ adop phase=apply apply_mode=downtime patches=37237361 patchtop=/path/to/Jan2025_CPU/EBS

$ adop phase=apply apply_mode=downtime patches=37078855,36560216 patchtop=/path/to/Jan2025_CPU/EBS

After their successful completion, confirm January 2025 CPU was applied:
SQL> col CPU format a9
SQL> select max(CODELEVEL) "CPU" 
             from ad_trackable_entities where abbreviation in ('ebscpu');
CPU
----------
2025.01

$ perl $AD_TOP/bin/admkappsutil.pl
Starting the generation of appsutil.zip
Log file located at $INST_TOP/admin/log/MakeAppsUtil_11260913.log
output located at $INST_TOP/admin/out/appsutil.zip
MakeAppsUtil completed successfully.

$ cp -p $INST_TOP/admin/out/appsutil.zip $APPLPTMP
$ chmod 777 $APPLPTMP/appsutil.zip

DBA runs the below steps on database server.
$ cp /path/to/shared/utl_dir/appsutil.zip $ORACLE_HOME/
$ cd $ORACLE_HOME 
$ unzip -o appsutil.zip
Set env and then run autoconfig on the database server.

5. Apply technology patches (on each node)
$ cd /path/to/Jan2025_CPU/FMW_Comm
$ ./apply_EBStechPatches.sh 'appsPWD'

6. Upgrade JDK
$ cd /path/to/Jan2025_CPU/JDK
$ ./JDK_upgrade1_7_xxx.sh

$ cd ../ejcpuc
$ ./ejcpuc.sh

7. Upgrade JRE

$ cd /path/to/Jan2025_CPU/JRE
$ cp jre-8u441-windows-i586.exe $COMMON_TOP/webapps/oacore/util/javaplugin/j2se18441.exe

$ cd $COMMON_TOP/webapps/oacore/util/javaplugin
$ $FND_TOP/bin/txkSetPlugin.sh 18441

$ grep sun_plugin_ver $CONTEXT_FILE
$ grep s_forms_launch_method $CONTEXT_FILE

8. Run adadmin to sign JAR files (optional) and run antoconfig on each node.

Optional: verify the digital signature of a Jar file (and new timestamp):
$ jarsigner -verify -verbose -certs $AD_TOP/java/jar/adxlib.jar
$ ls -altr $JAVA_TOP/oracle/apps/fnd/jar

9. Start apps services.

10. Test EBS site.

Tuesday, May 6, 2025

Script for applying EBS technology patches

A key step in applying EBS CPU patches is to decide which technology patches are required by both ETCC checker and quarterly CPU Release document for WebLogic and Oracle Homes of Fusion Middleware. After the list of required patches for each component is decided, I write a shell script to run the patching process and make it more consistent and much quicker cross nodes and different instances (Dev, QA, Prod).

Pre-steps: place technology patch files to a location shared by many servers. For January 2025 CPU patches (Doc ID 3061170.1), 3 patches are needed and copied to each sub-folder WLS and FMW_Comm under location /path/to/Jan2025_CPU:
 
- FMW_Comm
Holds 3 Fusion Middleware patch files and opatch response file ocm.rsp. Each .zip file is unzipped to its own sub-folder here.
$ cd /path/to/Jan2025_CPU/FMW_Comm
$ ls *.*
ocm.rsp
p33974106_111190_Generic.zip
p33960746_111190_Generic.zip
p34714760_111190_Generic.zip

Do unzip and chmod just one time.
$ unzip p33974106_111190_Generic.zip
$ unzip p33960746_111190_Generic.zip
$ unzip p34714760_111190_Generic.zip

Open READ permission for other OS accounts to run the script to patch instances on other servers: (Or, remove the sub-folder and then unzip the .zip file again before each opatch run.)
$ chmod -R +r 33974106
$ chmod -R +r 33960746
$ chmod -R +r 34714760

$ ls -l | egrep "^drw" |awk '{print $9}'
33974106
33960746
34714760

- WLS 
Holds 3 WLS patch zip files, and each file is saved in its own sub-folder:
13845626/p13845626_10360231017_Generic.zip
35476084/p35476084_1036_Linux-x86-64.zip
35586779/p35586779_1036_Generic.zip

============= script apply_EBStechPatches.sh ============
# apply tech patches for January 2025 EBS CPU technology patches. 
# it can be modified for any other R12.2 EBS CPU releases.
# 1. Make sure same patches are required cross different nodes and instances!
# 2. Assume all patch files are saved in sub-folders of a shared location: FMW_Comm and WLS
#
# Specify the path to the shared location where patch files are saved:
patchFileLoc=/path/to/Jan2025_CPU

# Run/call the script, for example: ./apply_EBStechPatches.sh 'appsPWD$%^&_!'
appsPWD=$1
if [ -z "$appsPWD" ]; then
  echo "ETCC needs password to run the script. Exit ..."
  exit;
else
echo "Run ETCC checker:"
cd $patchFileLoc/ETCC
{ echo $appsPWD } | checkMTpatch.sh

echo "Apply WLS patches:"
# Because it may required to remove WLS patch before installing a new WLS patch,
# to edit this section is needed for each CPU release. Can not make it truly automatic!
echo $FMW_HOME
cd $FMW_HOME/utils/bsu/cache_dir
echo $PWD
cp -f $patchFileLoc/WLS/13845626/p13845626_10360231017_Generic.zip .
cp -f $patchFileLoc/WLS/35476084/p35476084_1036_Linux-x86-64.zip .
cp -f $patchFileLoc/WLS/35586779/p35586779_1036_Generic.zip .
ls -altr
unzip -o p35586779_1036_Generic.zip
unzip -o p35476084_1036_Linux-x86-64.zip
unzip -o p13845626_10360231017_Generic.zip
cd ..
echo $PWD
./bsu.sh -remove -patchlist=AMGE -prod_dir=$FMW_HOME/wlserver_10.3
./bsu.sh -remove -patchlist=P8S7 -prod_dir=$FMW_HOME/wlserver_10.3
./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=E7HI -prod_dir=
$FMW_HOME/wlserver_10.3
./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=WY44 -prod_dir=
$FMW_HOME/wlserver_10.3
./bsu.sh -remove -patchlist=CW7X -prod_dir=$FMW_HOME/wlserver_10.3
./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=KMHV -prod_dir=
$FMW_HOME/wlserver_10.3
echo "confirm 3 patches are installed"
./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -status=applied -verbose -view | egrep -i 'KMHV|WY44|E7HI'

echo "Apply patches to ORACLE_COMMON: Truly automatic."
export ORACLE_HOME=$FMW_HOME/oracle_common
export PATH=$ORACLE_HOME/OPatch:$PATH
echo "Oracle Home: $ORACLE_HOME"
which opatch
ps -ef | grep $LOGNAME

cd $patchFileLoc/FMW_Comm
for x in ls -l | egrep "^drw" |awk '{print $9}'
do
echo "apply patch $x"
cd $x
opatch apply -silent -ocmrf $patchFileLoc/FMW_Comm/ocm.rsp   # silent!
echo "confirm patch $x is applied:"
opatch lsinventory | egrep -i "$x"
cd ..
echo
done

# Add easily a section here if patches are required for Oracle Home of Forms and Reports

echo "ETCC to confirm no more patches are needed:"
cd $patchFileLoc/ETCC
{ echo $appsPWD } | checkMTpatch.sh
exit

fi
============== End ================

Friday, May 2, 2025

Run opatch in silent mode in EBS R12.2

Sometimes, you want to apply patches to Oracle Home in a shell script. To do that, you do not want to run opatch interactively. Here is how to apply patches to an Oracle Home of Fusion Middleware in silent mode.

$ export ORACLE_HOME=$FMW_HOME/oracle_common
$ export PATH=$ORACLE_HOME/OPatch:$PATH
$ which opatch    <= make sure opatch is from oracle_common/OPatch
$ echo $ORACLE_HOME
$RUN_BASE/FMW_Home/oracle_common

First, run emocmrsp to create a response file ocm.rsp:
$ cd $ORACLE_HOME/OPatch/ocm/bin
$ ./emocmrsp
The ORACLE_HOME does not contain java.
The ORACLE_HOME does not contain a valid JDK/JRE.
Redefine JAVA_HOME to refer to a JDK/JRE 1.2.2 or greater.

$ export JAVA_HOME=$FMW_HOME/webtier/jdk   <= I chose this 64-bit JDK
$ echo $JAVA_HOME
$RUN_BASE/FMW_Home/webtier/jdk
$ ./emocmrsp
OCM Installation Response Generator 10.3.7.0.0 - Production
Copyright (c) 2005, 2012, Oracle and/or its affiliates.  All rights reserved.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
The OCM configuration response file (ocm.rsp) was successfully created.

$ ls -al
-rwxr-x---. 1 userID Group 9063 Jul 21  2022 emocmrsp
-rw-rw-r--. 1 userID Group oaa  622 May 1 13:08 ocm.rsp

File ocm.rsp is created in $ORACLE_HOME/OPatch/ocm/bin by emocmrsp. It is a binary file, and can be copied to a shared location /path/to/sharedLocation for patching in other nodes or instances. 

Now, apply a patch to Oracle Common home without answering any opatch questions. For example, apply patch 33974106.

$ ps -ef | grep $LOGNAME     <= make sure EBS services are shutdown
$ cd /path/to/33974106

$ opatch apply -silent -ocmrf /path/to/sharedLocation/ocm.rsp
Oracle Interim Patch Installer version 11.1.0.12.9
Copyright (c) 2025, Oracle Corporation.  All rights reserved.
Oracle Home       : $RUN_BASE/FMW_Home/oracle_common
Central Inventory : /u03/app/oraInventoryDEVEBS
from                     : $RUN_BASE/FMW_Home/oracle_common/oraInst.loc
OPatch version    : 11.1.0.12.9
OUI version         : 11.1.0.11.0
Log file location : $RUN_BASE/FMW_Home/oracle_common/cfgtoollogs/opatch/33974106_May_1_2025_13_02_04/apply2025-05-1_13-02-04PM_1.log
OPatch detects the Middleware Home as "$RUN_BASE/FMW_Home"
Applying interim patch '33974106' to OH '$RUN_BASE/FMW_Home/oracle_common'
Verifying environment and performing prerequisite checks...
All checks passed.
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '$RUN_BASE/FMW_Home/oracle_common')
Is the local system ready for patching? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
Backing up files...
Patching component oracle.jrf.thirdparty.jee, 11.1.1.9.0...
Verifying the update...
Patch 33974106 successfully applied
Log file location: $RUN_BASE/FMW_Home/oracle_common/cfgtoollogs/opatch/33974106_May_1_2025_13_02_04/apply2025-05-1_13-02-04PM_1.log
OPatch succeeded.