Friday, May 16, 2025

Apply EBS January 2025 CPU patches

The first and important step in applying EBS CPU patches is to identify which patches are required for each EBS components. After step 1 below, the patching steps are the same for recent CPU releases. I used my shell scripts to perform two key steps for January 2025 CPU, which make the patching process much quicker and more consistent across nodes and instances. Posts for previous CPU patchings, such as October 2024 CPU Patches, have more details.

1. Download patch files.
Oracle E-Business Suite Release 12.2 Critical Patch Update Availability Document (January 2025) (Doc ID 3061170.1) is the document for applying January 2025 CPU patches. Run checkers and read the CPU document to decide which patches are required. Then, download and save them to a shared location /path/to/Jan2025_CPU that will be used for all nodes and different instances (Dev, QA and Prod, etc). This folder is organized to 10 sub-folders for different components:

$ cd /path/to/Jan2025_CPU
$ ls -d *
EBS ECPUC EJCPUC ETCC FMW_Comm FMW_Ora JDK JRE WLS

- EBS  hold patches required by ECPUC.sql.  Warning: ECPUC.sql only lists patches that are required or recommended by a EBS CPU patch but does NOT list prerequisites for each patch.
We decided to apply only 3 patches because other modules are not used in our EBS instances.

$ cd EBS
$ ls -trd * | egrep -i '36560216|37078855|37237361'
p37237361_12.2.0_R12_LINUX.zip
p37078855_R12.ATG_PF.C_R12_GENERIC.zip
p36560216_R12.PO.D_R12_GENERIC.zip
36560216
37237361
37078855

Grant permission once for ADOP to create multiple sub-folders under backup folder:
$ chmod 666 36560216/backup
$ chmod 666 37237361/backup
$ chmod 666 37078855/backup

- ECPUC  hold ECPUC.sql (from patch p35583866) for checking which EBS patches are required.

$ grep '$Header' *.sql
REM $Header: ECPUC.sql 120.0.12020000.10 2025/03/06 10:04:57 spullach noship $

- EJCPUC  hold ejcpuc.sh from patch p37171025 

$ grep '$Header' ejcpuc.sh
rcs="$Header: ejcpuc.sh,v 1.3 2025/01/21 04:30:00 egravers Exp egravers $"

Run ejcpuc.sh to confirm JDK upgrade is needed:
$ ./ejcpuc.sh
###############################################################
Checking Apptier Java 7 for CPU 2025.01 on Platform Linux_x64 - need 1.7.0_451
2025-05-xx 11:41:30 EDT on node_name.domain.com
###############################################################
2025.01    action    Your Version   bitness   Java Location
----------  -------- ---------------- -------- ------------------
1.7.0_451 UPDATE 1.7.0_391 32-bit $RUN_BASE/EBSapps/10.1.2/jdk/bin/java
1.7.0_451 UPDATE 1.7.0_391 32-bit $RUN_BASE/EBSapps/comn/util/jdk32/bin/java
1.7.0_451 UPDATE 1.7.0_391 64-bit $RUN_BASE/EBSapps/comn/util/jdk64/bin/java1
1.7.0_451 UPDATE 1.7.0_391 64-bit $RUN_BASE/FMW_Home/webtier/jdk/bin/java
Follow 1530033.1 to update the JDK(s). Your application tier JDK 7 is lower than the 1.7.0_451 update released in CPU 2025.01.

- ETCC  hold ETCC scripts from patch p17537119
Make sure both EBS apps and database use the scripts from the same p17537119 release/download.
$ grep '$Header' *.sh
checkDBpatch.sh:# $Header: checkDBpatch.sh 120.133 2025/01/30 17:56:04 chrhill noship $
checkMTpatch.sh:# $Header: checkMTpatch.sh 120.0.12020000.70 2025/01/30 18:28:36 chrhill noship $

- FMW_Ora  holds patch files for Oracle Home of forms and report (if required).
No required patches by January 2025 CPU release for my instances.
Custom file ocm.rep (see run Opatch in silent) was copied here for patching script.
$ ls *.*
ocm.rsp

- FMW_Comm  holds patch files for Oracle Common. 3 patch files that required by ETCC
$ ls *.*
p33974106_111190_Generic.zip
p33960746_111190_Generic.zip
p34714760_111190_Generic.zip
33974106
33960746
34714760
apply_EBStechPatches.sh

In addition, one custom file was copied here: apply_EBStechPatches.sh (see script for applying technology patches)

- FMW_Web  holds patch files for IAS_ORACLE_HOME (or, $FMW_HOME/webtier).
No patch is required by January 2025 CPU release for my instances.

- JDK  holds files needed for JDK upgrades.
Since Java 7 is officially out of Oracle support, we have to follow Oracle EBS CPU documents (such as Doc ID 3061170.1) & go to Table 3: Security Patches for Technology Stack Components With a New Patch in This CPU. Then, get a link to Oracle Java SE:
Doc ID 3066051.1 (Oracle Critical Patch Update (CPU) Jan 2025 for Oracle Java SE) has a link (on the low end) to download JDK 7 Update 451 Restricted: Patch 37308812 for JDK 1.7.0_451.
p37308812_170_451_Linux-x86-64.zip
p37308812_170_451_LINUX.zip 

Then unzip them to get two JDK files:
jdk-7u451-linux-i586.tar.gz 
jdk-7u451-linux-x64.tar.gz

Also copy/modify the custom script here.
JDK_upgrade1_7_xxx.sh  (see script for updating JDK)

- JRE  holds files for JRE upgrade on the server.
Document ID 3066051.1 (Oracle Critical Patch Update (CPU) Jan 2025 for Oracle Java SE) also has a link for JRE 8 Update 441: Patch 37308802. Download  p37308802_180441_WINNT.zip (for Microsoft Windows 32-bit), and unzip to see two files:
jre-8u441-windows-i586.zip
jre-8u441-windows-i586.exe

- WLS  holds 3 WLS patch files required by ETCC and CPU Doc ID 3061170.1
13845626/p13845626_10360231017_Generic.zip
35476084/p35476084_1036_Linux-x86-64.zip
35586779/p35586779_1036_Generic.zip

2. Apply database patches

Run ETCC checker to find the required patches, and then apply them. Run ejcpuc.sh to make sure Java versions in database are up to date.

3. Backups
Back up apps filesystems. Optional: hold scheduled concurrent jobs if necessary & create a GRP in database.

4. Apply EBS patches (in downtime mode on my instances)
$ adop - status      <= It's better to make sure fs_clone worked successfully before.
$ vi /etc/oraInst.loc
$ stop app services   
Also, check passwordless ssh works in multi-nodes instance for ADOP working on allnodes mode.

$ cd /path/to/Jan2025_CPU/ECPUC
$ sqlplus apps/
SQL> @ECPUC.sql

$ adop phase=apply apply_mode=downtime patches=37237361 patchtop=/path/to/Jan2025_CPU/EBS

$ adop phase=apply apply_mode=downtime patches=37078855,36560216 patchtop=/path/to/Jan2025_CPU/EBS

After their successful completion, confirm January 2025 CPU was applied:
SQL> col CPU format a9
SQL> select max(CODELEVEL) "CPU" 
             from ad_trackable_entities where abbreviation in ('ebscpu');
CPU
----------
2025.01

$ perl $AD_TOP/bin/admkappsutil.pl
Starting the generation of appsutil.zip
Log file located at $INST_TOP/admin/log/MakeAppsUtil_11260913.log
output located at $INST_TOP/admin/out/appsutil.zip
MakeAppsUtil completed successfully.

$ cp -p $INST_TOP/admin/out/appsutil.zip $APPLPTMP
$ chmod 666 $APPLPTMP/appsutil.zip

DBA runs the below steps on database server.
$ cp /path/to/shared/utl_dir/appsutil.zip $ORACLE_HOME/
$ cd $ORACLE_HOME 
$ unzip -o appsutil.zip
Set env and then run autoconfig on the database server.

5. Apply technology patches (on each node)
$ cd /path/to/Jan2025_CPU/FMW_Comm  # if the shell script resides there
$ ./apply_EBStechPatches.sh 'appsPWD'

6. Upgrade JDK (on each node)
$ cd /path/to/Jan2025_CPU/JDK   # if the shell script resides there
$ ./JDK_upgrade1_7_xxx.sh

Verify the result:
$ cd ../EJCPUC
$ ./ejcpuc.sh

7. Upgrade JRE

$ cd /path/to/Jan2025_CPU/JRE
$ cp jre-8u441-windows-i586.exe $COMMON_TOP/webapps/oacore/util/javaplugin/j2se18441.exe

$ cd $COMMON_TOP/webapps/oacore/util/javaplugin
$ $FND_TOP/bin/txkSetPlugin.sh 18441

Verify the result:
$ grep sun_plugin_ver $CONTEXT_FILE
$ grep s_forms_launch_method $CONTEXT_FILE

8. Run adadmin to sign JAR files (optional) and run antoconfig on each node.

Optional: verify the digital signature of a Jar file (and new timestamp):
$ jarsigner -verify -verbose -certs $AD_TOP/java/jar/adxlib.jar
$ ls -altr $JAVA_TOP/oracle/apps/fnd/jar

9. Start apps services.

10. Test EBS website.
 
11. adop phase=fs_clone

Tuesday, May 6, 2025

Script for applying EBS technology patches

A key step in applying EBS CPU patches is to decide which technology patches are required by both ETCC checker and quarterly CPU Release document for WebLogic and Oracle Homes of Fusion Middleware. After the list of required patches for each component is decided, I write a shell script to run the patching process and make it more consistent and much quicker cross nodes and different instances (Dev, QA, & Prod).

Pre-steps: place technology patch files to a location shared by many servers. For January 2025 CPU patches (Doc ID 3061170.1), 3 patches are needed and copied to each dedicated sub-folder WLS and FMW_Comm under location /path/to/Jan2025_CPU:
 
- FMW_Comm
Holds 3 Fusion Middleware patch files. Each .zip file is unzipped to its own sub-folder here.
$ cd /path/to/Jan2025_CPU/FMW_Comm
$ ls *.*
ocm.rsp
p33974106_111190_Generic.zip
p33960746_111190_Generic.zip
p34714760_111190_Generic.zip

Do unzip and chmod just one time.
$ unzip p33974106_111190_Generic.zip
$ unzip p33960746_111190_Generic.zip
$ unzip p34714760_111190_Generic.zip

Open READ permission for other OS accounts to run the script to patch instances on other servers: (Or, remove the sub-folder and then unzip the .zip file again before each opatch run.)
$ chmod -R +r 33974106
$ chmod -R +r 33960746
$ chmod -R +r 34714760

$ ls -l | egrep "^drw" |awk '{print $9}'
33974106
33960746
34714760

- WLS 
Holds 3 WLS patch zip files, and each file is saved in its own sub-folder:
13845626/p13845626_10360231017_Generic.zip
35476084/p35476084_1036_Linux-x86-64.zip
35586779/p35586779_1036_Generic.zip

- FMW_Ora
Only holds opatch response file ocm.rsp. No Forms & Reports patch is required by January 2025 CPU patches.

-FMW_Web
Empty. No IAS_ORACLE_HOME patch is required by January 2025 CPU patches.

============= script apply_EBStechPatches.sh ============
# apply tech patches for January 2025 EBS CPU technology patches. 
# it can be easily modified for other R12.2 EBS CPU releases.
# 1. Make sure same patches are required cross different nodes and instances!
# 2. Assume all patch files are saved in 4 sub-folders of a shared location: 
# FMW_Ora, FMW_Comm, FMW_Web, WLS
# 3. Assume ocm.rsp for for running opatch silently exists in folder $patchFileLoc/FMW_Ora
#
# Call the script, for example: ./apply_EBStechPatches.sh 'appsPWD$%^&_!'
#
###  Specify the path to the shared location where patch files are saved:
patchFileLoc=/path/to/Jan2025_CPU
appsPWD=$1
if [ -z "$appsPWD" ]; then
 echo "ETCC needs password to run the script. Exit ..."
 exit 1
fi
runcount=`ps -ef | grep ${LOGNAME:0:7} | grep tnslsnr | wc -l`
if [ $runcount -gt 1 ]
then
 echo "EBS services may still running. Please stop them first. (count: $runcount)"
 exit 1
fi

## Optional
# echo "Run ETCC script:" 
# cd $patchFileLoc/ETCC
# echo $appsPWD | checkMTpatch.sh

echo "Apply WLS patches:"
if [ -f p*.zip ]; then   # if no .zip files in this folder, move to patch next ORACLE_HOME
 zipcount=`ls -al $patchFileLoc/WLS/*.zip | wc -l`
 echo "Apply $zipcount patch(es) to WebLogic..."
 echo $FMW_HOME
 cd $FMW_HOME/utils/bsu/cache_dir
 echo $PWD

 # assume all patch .zip files were copied to folder $patchFileLoc/WLS.
 #
 ### Because removal of conflict WLS patches may required, it is difficult to automate this part.
 ### Editing is needed for each CPU release! ###
 ### Below lines are for January 2025 CPU patching.
 cp -f $patchFileLoc/WLS/13845626/p13845626_10360231017_Generic.zip .
 cp -f $patchFileLoc/WLS/35476084/p35476084_1036_Linux-x86-64.zip .
 cp -f $patchFileLoc/WLS/35586779/p35586779_1036_Generic.zip .
 ls -altr
 unzip -o p35586779_1036_Generic.zip
 unzip -o p35476084_1036_Linux-x86-64.zip
 unzip -o p13845626_10360231017_Generic.zip
 cd ..
 echo $PWD
 ./bsu.sh -remove -patchlist=AMGE -prod_dir=$FMW_HOME/wlserver_10.3
 ./bsu.sh -remove -patchlist=P8S7 -prod_dir=$FMW_HOME/wlserver_10.3
 ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=E7HI -prod_dir=$FMW_HOME/wlserver_10.3
 ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=WY44 -prod_dir=$FMW_HOME/wlserver_10.3
 ./bsu.sh -remove -patchlist=CW7X -prod_dir=$FMW_HOME/wlserver_10.3
 ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=KMHV -prod_dir=$FMW_HOME/wlserver_10.3
 ./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -status=applied -verbose -view | egrep -i 'KMHV|WY44|E7HI'
 ### stop editing ###
else
 echo "No zip files in this folder. No WLS patching is needed for this release."
fi

# Apply patches to 3 Oracle_HOMEs
# Assume ocm.rsp for opatch exists in folder $patchFileLoc/FMW_Ora

echo "Apply patches to Forms & Reports"
cd $patchFileLoc/FMW_Ora       # assume all ORACLE_HOME .zip files were unzipped in this folder.
foldercount=`find * -prune -type d | wc -l`
if [ $foldercount -gt 0 ]; then
 export ORACLE_HOME=$RUN_BASE/EBSapps/10.1.2
 export PATH=$ORACLE_HOME/OPatch:$PATH
 echo "apply $foldercount patch(es) to Oracle Home: $ORACLE_HOME"
 which opatch
 for x in `ls -l | egrep "^drw" |awk '{print $9}'`
 do
  echo "apply patch $x"
  cd $x
  opatch apply -silent -ocmrf $patchFileLoc/FMW_Ora/ocm.rsp
  cd ..
  opatch lsinventory | egrep -i "$x"
  echo
 done
else
 echo "No patch is needed"
fi

echo "Apply patches to ORACLE_WEBTIER:"
cd $patchFileLoc/FMW_Web      # assume all IAS_ORACLE_HOME .zip files were unzipped in this folder.
foldercount=`find * -prune -type d | wc -l`
if [ $foldercount -gt 0 ]; then
 export ORACLE_HOME=$IAS_ORACLE_HOME   # Or, $FMW_HOME/webtier
 export PATH=$ORACLE_HOME/OPatch:$PATH
 echo "apply $foldercount patch(es) to Oracle Home: $ORACLE_HOME"
 which opatch
 for x in `ls -l | egrep "^drw" |awk '{print $9}'`
 do
  echo "apply patch $x"
  cd $x
  opatch apply -silent -ocmrf $patchFileLoc/FMW_Ora/ocm.rsp
  cd ..
  opatch lsinventory | egrep -i "$x"
  echo
 done
else
 echo "No patch is needed"
fi

echo "Apply patches to ORACLE_COMMON:"
cd $patchFileLoc/FMW_Comm      # assume all ORACLE_COMMON .zip files were unzipped in this folder.
foldercount=`find * -prune -type d | wc -l`
if [ $foldercount -gt 0 ]; then
 export ORACLE_HOME=$FMW_HOME/oracle_common
 export PATH=$ORACLE_HOME/OPatch:$PATH
 echo "apply $foldercount patch(es) to Oracle Home: $ORACLE_HOME"
 which opatch
 for x in `ls -l | egrep "^drw" |awk '{print $9}'`
 do
  echo "apply patch $x"
  cd $x
  opatch apply -silent -ocmrf $patchFileLoc/FMW_Ora/ocm.rsp
  cd ..
  opatch lsinventory | egrep -i "$x"
 echo
 done
else
 echo "No patch is needed"
fi

echo "ETCC to confirm no more patches are needed:"
cd $patchFileLoc/ETCC
echo $appsPWD | checkMTpatch.sh
exit 0
============== End ================

Friday, May 2, 2025

Run opatch in silent mode in EBS R12.2

Sometimes, you want to apply patches to Oracle Home in a shell script. To do that, you do not want to run opatch interactively. Here is how to apply patches to an Oracle Home of Fusion Middleware in silent mode.

$ export ORACLE_HOME=$FMW_HOME/oracle_common
$ export PATH=$ORACLE_HOME/OPatch:$PATH
$ which opatch    <= make sure opatch is from oracle_common/OPatch
$ echo $ORACLE_HOME
$RUN_BASE/FMW_Home/oracle_common

First, run emocmrsp to create a response file ocm.rsp:
$ cd $ORACLE_HOME/OPatch/ocm/bin
$ ./emocmrsp
The ORACLE_HOME does not contain java.
The ORACLE_HOME does not contain a valid JDK/JRE.
Redefine JAVA_HOME to refer to a JDK/JRE 1.2.2 or greater.

It needs proper JAVA_HOME:
$ export JAVA_HOME=$FMW_HOME/webtier/jdk   <= I chose this 64-bit JDK
$ echo $JAVA_HOME
$RUN_BASE/FMW_Home/webtier/jdk
$ ./emocmrsp
OCM Installation Response Generator 10.3.7.0.0 - Production
Copyright (c) 2005, 2012, Oracle and/or its affiliates.  All rights reserved.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
The OCM configuration response file (ocm.rsp) was successfully created.

$ ls -al
-rwxr-x---. 1 userID Group 9063 Jul 21  2022 emocmrsp
-rw-rw-r--. 1 userID Group oaa  622 May 1 13:08 ocm.rsp

File ocm.rsp is created in $ORACLE_HOME/OPatch/ocm/bin by emocmrsp. It is a binary file, and can be copied to a shared location /path/to/sharedLocation for patching in other nodes or instances. 

Now, apply a patch to Oracle Common home without answering any opatch questions. For example, apply patch 33974106.

$ ps -ef | grep $LOGNAME     <= make sure EBS services are shutdown
$ cd /path/to/33974106

$ opatch apply -silent -ocmrf /path/to/sharedLocation/ocm.rsp
Oracle Interim Patch Installer version 11.1.0.12.9
Copyright (c) 2025, Oracle Corporation.  All rights reserved.
Oracle Home       : $RUN_BASE/FMW_Home/oracle_common
Central Inventory : /u03/app/oraInventoryDEVEBS
from                     : $RUN_BASE/FMW_Home/oracle_common/oraInst.loc
OPatch version    : 11.1.0.12.9
OUI version         : 11.1.0.11.0
Log file location : $RUN_BASE/FMW_Home/oracle_common/cfgtoollogs/opatch/33974106_May_1_2025_13_02_04/apply2025-05-1_13-02-04PM_1.log
OPatch detects the Middleware Home as "$RUN_BASE/FMW_Home"
Applying interim patch '33974106' to OH '$RUN_BASE/FMW_Home/oracle_common'
Verifying environment and performing prerequisite checks...
All checks passed.
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '$RUN_BASE/FMW_Home/oracle_common')
Is the local system ready for patching? [y|n]
Y (auto-answered by -silent)
User Responded with: Y
Backing up files...
Patching component oracle.jrf.thirdparty.jee, 11.1.1.9.0...
Verifying the update...
Patch 33974106 successfully applied
Log file location: $RUN_BASE/FMW_Home/oracle_common/cfgtoollogs/opatch/33974106_May_1_2025_13_02_04/apply2025-05-1_13-02-04PM_1.log
OPatch succeeded.

Note if a patching asks more questions, the silent mode will fail, e.g.
... ...
applying interim patch '38059622' to OH '$FMW_HOME/oracle_common'
Patch [ 38059622 ] conflict with patch(es) [  33960746 ] in the Oracle Home.
To resolve patch conflicts please contact Oracle Support Services.
If you continue, patch(es) [  33960746 ] will be rolled back and the new Patch  [ 38059622 ] will be installed.
Do you want to proceed? [y|n]
N (auto-answered by -silent)
User Responded with: N
Log file location: $FMW_HOME/oracle_common/cfgtoollogs/opatch/38059622_xxxx_1.log
OPatch failed with error code 61

In that case, we have to apply it interactively.