Tuesday, December 2, 2025

October 2025 CPU patchset and CVE-2025-61882

Oracle emailed Security Alert CVE-2025-61882 out on Oct 5, 2025 and published Doc ID 3106344.1 (Security Alert CVE-2025-61882 Patch Availability Document for Oracle E-Business Suite) for it. The document first strongly recommends a set of patches 38501230, 38501349, 38501757 as hotfixes. Then, Oracle modified the document and recommended October 2025 CPU patch set or 6 hotfix patches 38507994, 38518258, 38523311, 38523302, and 38510732, 38501757.

I applied October 2025 CPU patch and found it includes all individual patches mentioned above. So, it is a good and clean way to fix CVE-2025-61882 vulnerabilities. Below are EBS patches I applied to my instances

38298685 12.2.0  Oct 2025 CPU
38261405 R12.FWK.C  Oracle Applications Framework
37450688 R12.OWF.C  Oracle Workflow
38180394 R12.FND.C  (prerequisite: OCT 2020 CPU: 31643029:12.2.0)
38510732 R12.XDO.C  BI Publisher (formerly XML Publisher)

Note ECPUC.sql only lists patches that are required or recommended by a CPU patch release but does NOT list prerequisites by each patch. For example, October 2025 CPU patch 38298685 requires R12.AD.C.DELTA.15 & R12.TXK.C.DELTA.15 (or higher) and so I had first to apply below two patches in instances that had AD and TXK Delta 14:

36119925 (R12.AD.C.DELTA.16) Built: JUL-16-2024
36117775 (R12.TXK.C.DELTA.16) Built: JUL-16-2024

Run new adgrants.sql as SYSDBA using 
SQL> @/path/to/adgrants.sql <APPS schema name>
$ adop phase=apply apply_mode=downtime patches=36119925,36303698,36989014,37988551 merge=yes patchtop=/path/to/Oct2025_CPU/AD workers=16 wait_on_failed_job=yes
$ adop phase=apply apply_mode=downtime patches=36117775,36641685,37500697 merge=yes patchtop=/path/to/Oct2025_CPU/TXK workers=16 wait_on_failed_job=yes

Detailed steps in applying patches in October 2025 CPU release are almost the same as in applying January 2025 patches.

To get the list of CVEs addressed by each CPU patchset, go to https://www.oracle.com/security-alerts and then click on the individual CPU release.




Tuesday, November 25, 2025

ADOP and adstrtal.sh may skip reading $HOME/.profile

When user applMgr on a primary node of RHEL8 runs ADOP to apply patches to slave/remote nodes or runs adstrtal.sh in allnodes mode to start EBS services on remote nodes, it does not read applMgr's .profile on remote nodes but get some OS settings from /etc/login.defs of the server instead. That makes some custom or important behaviors, such permission on file $CONTEXT_FILE or on files in $APPLCSF/out, inconsistent in a multi-node instance.

Two ways to avoid the problem:

1. Add line(s) to custom env file
$INST_TOP/appl/admin/custom$CONTEXT_NAME.env on remote nodes, e.g.
umask 0022
echo "this is for debug"

2. Comment out line(s), such as "# UMASK   077", in /etc/login.defs to force/make ADOP and adstrtal.sh get OS env settings from applMgr's .profile on remote nodes.

Similarly software tools, such as ftp & scp, also read OS env variables from file /etc/login.defs of server level. 

Sunday, October 19, 2025

How to retrieve WebLogic password in R12.2

WebLogic is used as part of Oracle EBS R12.2. "weblogic" account is used to start EBS services and is used to log onto WebLogic Admin console (at hostname.domain.com:s_wls_adminport). Its password is encrypted and saved on the file system. Here are two steps I ran in my R12.2 environments to decrypt it.

1. Identify WebLogic Domain path and boot.properties file

$ echo $EBS_DOMAIN_HOME
$RUN_BASE/FMW_Home/user_projects/domains/EBS_domain
$ cd $EBS_DOMAIN_HOME/servers/AdminServer/security
$ ls 
boot.properties
$ more boot.properties
username={AES}mE0mATL4+Lv/gLcIuuuuuuuuu=
password={AES}2kGMi4fcZ7FwYXWIxxxxxxxxx=

2. Run wlst.sh to decrypt the password

$ cd $FMW_HOME/wlserver_10.3/common/bin
$ ls
commEnv.sh       config.sh  security    startDerby.sh   stopDerby.sh  upgrade.sh     wlsifconfig.sh
config_builder.sh  pack.sh    setPatchEnv.sh  startManagedWebLogic.sh  unpack.sh     wlscontrol.sh  wlst.sh
$ sh wlst.sh
CLASSPATH=/u01/app/... ......
Initializing WebLogic Scripting Tool (WLST) ...
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
wls:/offline> domain = '$EBS_DOMAIN_HOME'   # <= replace the env variable with the real path
wls:/offline> service = weblogic.security.internal.SerializedSystemIni.getEncryptionService(domain)
wls:/offline> encryption = weblogic.security.internal.encryption.ClearOrEncryptedService(service)
wls:/offline> print "Weblogic server password is: %s" %encryption.decrypt("{AES}2kGMi4fcZ7FwYXWIxxxxxxxxx=")
Weblogic server password is: webLogicPWD
wls:/offline> ^C

If you get Java error, most likely you entered a wrong path or a wrong encrypted password/string.

My WebLogic version is 10.3.6.0.210119. It can be found by two lines:
$ . $FMW_HOME/wlserver_10.3/server/bin/setWLSEnv.sh
... ...
Your environment has been set.
$  java weblogic.version
... ...
WebLogic Server 10.3.6.0.210119 PSU Patch for ...
... ...
If you want to change weblogic password, please read How to change weblogic password

Saturday, September 6, 2025

RHEL7 to RHEL8 upgrade on EBS server

Oracle EBS did not support in-place OS upgrade until April 30, 2024 when they updated Oracle document 1330701.1 (Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.2) for Linux x86-64):

An in-place OS upgrade from Oracle Linux 7 to 8 using the Leapp utility is certified for Oracle E-Business Suite Release 12.2. While you were previously required to install Oracle Linux 8 on a separate target environment before cloning or re-mounting file systems to it, you can now choose to preserve your current Oracle Linux 7 environment and perform an in-place upgrade to Oracle Linux 8 using Leapp. You should review the Operating System upgrades using Leapp section below.

If server team allows the server name unchanged as we did in RHEL 5, 6, and 7 upgrades, it is an easy task for EBS Apps Admin to bring EBS up on the new RHEL 8 server. But, if server team has to change the server name (e.g. migrate the host to a cloud), we have to clone EBS apps to the new server and to make other areas (such as in network, F5, security/firewall, FTP/email, custom codes and PPM) also work with cloned EBS instances.

RHEL 8 in-place upgrade steps on our R12.2.10 servers:

1. Check current RHEL version
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)
2. Apply two patches to EBS: 
- WebLogic patch 32931976. Download the .zip file to get patch file p31090393_1036_Linux-x86-64.zip (which has no relation with patch 31090393). Without this patch, users get a blank page without error message when viewing output or log of concurrent requests. Unzip the file to cache_dir folder and then use bsu.sh to apply EG54. 

$ cd $FMW_HOME/utils/bsu/
$ ./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -status=applied -verbose -view | egrep -i 'EG54'

- Patch 33471111. Follow special steps in Doc ID 13307011.1 to apply it by using folder prereq from p33471111_111190_Generic.zip to replace both $FMW_HOME/webtier/inventory/Scripts/prereq and $PATCH_BASE/FMW_Home/webtier/inventory/Scripts/prereq

3. Verify apps (login page & forms) works and back up all file systems.
4. Shutdown all EBS services.
5. Linux Admin upgrades RHEL to RHEL 8 by using Leapp.
6. Verify it is RHEL8 now
$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.10 (Ootpa)

- Kernel Settings: /etc/sysctl.conf
Restore sysctl.conf from the backup that was taken prior to the upgrade.
- Make sure to include the options parameter in the resolve.conf.
$ cat /etc/resolv.conf | grep -i options
options timeout:1 attempts:5 rotate
- If the /etc/sysconfig/networking/profiles/default/network file exists, remove it.
- Values in /etc/security/limits.conf
Restore /etc/security/limits.conf from the backup
- Make sure /etc/oraInst.loc has proper permission.

7. Verify all rpm packages required by Oracle doc 1330701.1 exist on the server. Use YUM to install them, such as
$ yum install libXp-devel-1.0.3-3.el8.i686

Some important packages installed on my servers:
$ rpm -qa | grep libXp
libXp-devel-1.0.3-3.el8.i686
libXp-1.0.3-3.el8.x86_64
libXp-1.0.3-3.el8.i686
libXp-devel-1.0.3-3.el8.x86_64
libXpm-3.5.12-11.el8.x86_64

$ rpm -qa | grep motif
motif-devel-2.3.4-20.el8.x86_64
motif-2.3.4-20.el8.x86_64
motif-devel-2.3.4-20.el8.i686
motif-2.3.4-20.el8.i686
Notes: opemmotif21 is not installed

$ rpm -qa | grep redhat-lsb
redhat-lsb-desktop-4.1-47.el8.x86_64
redhat-lsb-printing-4.1-47.el8.x86_64
redhat-lsb-submod-security-4.1-47.el8.x86_64
redhat-lsb-cxx-4.1-47.el8.x86_64
redhat-lsb-submod-multimedia-4.1-47.el8.x86_64
redhat-lsb-languages-4.1-47.el8.x86_64
redhat-lsb-core-4.1-47.el8.x86_64
redhat-lsb-4.1-47.el8.x86_64

$ rpm -qa | grep xorg-x11     (Optional?)
xorg-x11-fonts-misc-7.5-19.el8.noarch
xorg-x11-xbitmaps-1.1.1-13.el8.noarch
xorg-x11-server-utils-7.7-27.el8.x86_64
xorg-x11-apps-7.7-21.el8.x86_64            <= xclock
xorg-x11-utils-7.5-28.el8.x86_64
xorg-x11-proto-devel-2020.1-3.el8.noarch
xorg-x11-xauth-1.0.9-12.el8.x86_64
xorg-x11-font-utils-7.5-41.el8.x86_64
xorg-x11-xinit-1.3.4-18.el8.x86_64

$ rpm -qa | grep libXrender
libXrender-devel-0.9.10-7.el8.x86_64
libXrender-0.9.10-7.el8.x86_64
libXrender-0.9.10-7.el8.i686

8. Run AutoConfig 
9. Start all EBS  services
10. After RHEL8 upgrade, some ports may not be opened on the server. If Apache services start without error, but EBS login webpage does not load up on browsers, the port number may become not open for TCP or may be blocked by a firewall after OS upgrade. In a case, "wget https://sitename.domain.com:4470" returned "connected" on the Linux server itself but got "failed: No route to host." from any remote server. After I asked Linux Admin to fix the issue on port 4470, the login page https://sitename.domain.com:4470 works on browsers.
Please check my post on port or https://runcloud.io/blog/check-tcp-port-linux,

11. Listener issue in opening concurrent job log. 
It was also due to port problem. When trying to open concurrent request log, get message an error occurred while attempting to establish an Applications File Server connection with the node FNDFS_nodeName. There may be a network configuration problem, or the TNS listener on node FNDFS_nodeName may not be running. Please contact your system administrator.

"tnsping FNDFS_nodeName" should work and tell the listener port on the server where FNDFS_nodeName is an entry of tnsnames.ora file. Most likely, the listener port is blocked on the server.

12. SELinux may be enabled in RHEL8. Check file /etc/selinux/config. That may make ssh passwordless login fails. "-Z" option in "ls -alZ" command is available only when selinux is enabled.

13. Make sure applMgr account has the right umask value in $HOME/.profile. This is very important if other OS users need to view output file (in $APPLCSF) of concurrent jobs or to submit concurrent jobs from a shell script.
$ umask 0022

14. Kill switches
Minimum Security Baseline (MSB), also known as a Minimum Baseline Security Standard (MBSS), is a set of foundational security guidelines and requirements that organizations must meet to protect their information systems and data, ensuring a basic level of security. 
MSB says $HOME directories should be 700 permissions (drwx------.). If an EBS is installed under /u01/app ($HOME) directory, the proper permission on app is 750 (drwxr-x---.) and its owner is applMgr. In that case, Server Admin has to add a kill switch so that Chef will not remove those permissions. A Kill Switch is one line (such as cis_rhel8_01_02_03=false) added to a company custom file (usually, it is /etc/xxxxx.info) to prevent something is uninstalled or is removed by Chef.

15. X11 is removed on RHEL8 unless an Exception (such as Archer exception. See https://www.archerirm.com) is filed, and a kill switch is added.
In RHEL8, folder lib under /usr/X11R6 may still exist, but /usr/X11R6/lib does not have any .so files or is empty. openmofit21 is replaced by another package in RHEL8. Without it, EBS Forms still works. If Forms Server failed to start with error FRM-92101, see Doc ID 3052209.1. Also check logs in $EBS_DOMAIN_HOME/servers/forms_server1/log/*.out

16. X-Windows is disabled in RHEL8 by default per Security Requirements as it does not meet the MSB as set in the CIS standards. An exception is needed to make X-Window (such xclock) work on laptop/PC.

17. Server tools in our RHEL8 servers are installed under /opt.
$ ls -d */
BESClient
Tivoli
tivoli
Symantec
appdynamics
chef
Tanium
splunkforwarder
CrowdStrike

Each of them is a folder. Use "ps -ef | grep chef" (e.g.) to check if the tool is running on the server.

Saturday, June 21, 2025

How to check a port is open on the firewall

If the port used by a website is not open in the firewall of the web server, the webpage will not be accessible. "nc" can be used to verify if a network port is open and available on a server. Below result indicates port 7042 is open on server serverName.

$ nc -zv serverName 7042
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 167.xx.xx.50:7042.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.

If nothing is listening on port 7042, "nc" gives a different result and does not tell if port is the problem. 
$ netstat -tuanp | grep 7042
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)

$ nc -zv serverName 7042      <= run it on another server
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connection refused.

So make sure the Apps service is working correctly and running before using "nc" to check port's status. When the port is used on the server,  "nc" message No route to host tells us that the port is blocked by firewall of the server. 
$ netstat -tuanp | grep 7043
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6       0      0 167.xx.xx.50:7043       :::*                 LISTEN      2915615/java
tcp6       0      0 167.xx.xx.50:52210      167.xx.xx.50:7043       ESTABLISHED 2922169/java

$ nc -zv serverName 7043        <= run it on another server
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: No route to host.

wget may give similar failure message if WLS webservice is listening on port 7043:
$ wget serverName.domain.com:7043/console
... ...
Connection ... ... failed: No route to host

root is able to check and see the list of open ports on the firewall of the server:
$ echo $USER
applmgr
$ sudo su -       
[sudo] password for applmgr:        
Hostname:  serverName.domain.com   <= ONLY if sudo root is granted to applmgr
OS:  Red Hat Enterprise Linux release 8.10 (Ootpa)
Arch:  x86_64
[root@serverName ~]# cd /etc/firewalld
[root@serverName ~]# which firewall-cmd
/usr/bin/firewall-cmd
[root@serverName firewalld]# firewall-cmd --list-all
internal (active)
target: default
icmp-block-inversion: no
interfaces: ens192
sources:
services: cockpit dhcpv6-client mdns samba-client ssh
ports: ... 443/tcp 7042/tcp 7051/tcp 7052/tcp 8004/tcp 4484/tcp ... 757/udp 17430/udp ...
protocols:
forward: no
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
[root@serverName firewalld]#

Troubleshooting case. In a multi-node EBS instance, ADOP fs_clone failed on 2nd node with message "The admin server for the patch file system is not running". I started WebLogic services in patch filesystem of Primary node serverName successfully:
$ sh adadminsrvctl.sh start forcepatchfs
... ...
adadminsrvctl.sh: exiting with status 0
... ...
$ grep s_wls_adminport $CONTEXT_FILE
<wls_adminport oa_var="s_wls_adminport" oa_type="PORT" base="7001" step="1" range="-1" label="WLS Admin Server Port">7043</wls_adminport>

But, the problem was console webpage serverName.domain.com:7043/console did not work and showed generic errors on browser. Too many Java errors and messages in WebLogic logfiles under folder $FMW_HOME/user_projects/domains/EBS_domain/servers/AdminServer/logs
made logs not useful as well. Finally, I found the root cause is port 7043 was not open on the server. After it was added to the firewall by Linux Admin, the WebLogic website became accessible from browsers and then fs_clone worked.