ERP on DB: March 2022

Friday, March 11, 2022

R12.2 site is down with error "Failure of Web Server bridge"

EBS R12.2 site is not accessible and login page gives error:

Failure of Web Server bridge:

No backend server available for connection: timed out after 10 seconds or idempotent set to OFF or method not idempotent.

The message does not tell much about the root cause. Most likely it happens when a large data set was pulled by EBS users from database, which might make oacore crash. We took a few actions to address the problem.

1. Update NodeManager parameters via s_append_nm_jvmargs in ${CONTEXT_FILE} (by default, it is not set). For details, see Doc ID 2300621.1 (Node Manager Consumes High Virtual Memory).

$ grep s_append_nm_jvmargs $CONTEXT_FILE

<append_nm_jvmargs oa_var="s_append_nm_jvmargs">-XX:PermSize=256m -XX:MaxPermSize=512m -Xms1024m -Xmx1024m</append_nm_jvmargs>

2. Increase memory for oacore. Use the following steps to customize the managed server configuration via the WebLogic Server Administration Console. See Doc ID 2104351.1 ("java.lang.OutOfMemoryError, msg=Java heap space" Error While Uploading In APCC).

$ grep s_wls_adminport $CONTEXT_FILE

1). Log on to the WebLogic Server Administration Console as weblogic (http://[host.domain.com]:s_wls_adminport/console).

2). Click on Environment => Servers (link).

3). Click on oacore_server1 - the managed server whose configuration needs to be updated.

4). Under Configuration (tab) => Server Start (tab) => You can see the Arguments Section.

5). Click on Lock and Edit button in the 'Change Center' panel.

6). Update the Heap space from

-XX:PermSize=128m -XX:MaxPermSize=384m -Xms512m -Xmx512m -

-Djava.security.policy=/.../server/lib/weblogic.policy

to new set:

-XX:PermSize=512m -XX:MaxPermSize=512m -Xms2048m -Xmx4096m -Djava.security.policy=/.../server/lib/weblogic.policy

7). Click the 'Save' button to save the configuration changes.

8). Once the customizations are complete and saved, click the 'Activate Changes' button in the 'Change Center' panel to activate the changes.

Do the same for oacore_server2 if it is a multi-node instance.

(Same steps can also be done by running script $ perl $AD_TOP/patch/115/bin/adProvisionEBS.pl)

3. Set the Inactive Connection Timeout parameter to one hour. Steps are in Doc ID 1940996.1 (Oracle E-Business Suite 12.2 Data Source Connection Pool Diagnostics)

Navigate to Services (tree link) => Data Sources (link) => EBSDataSource (page link) => Connection Pool (tab) => Advanced (Expand arrow on the bottom).

Click on the Lock and Edit Button.

Update the Inactive Connection Timeout to the desired value in seconds: 3600 (for one hour).

Then, click on Save, and the 'Activate Changes' button.

4. Steps to ensure the timeouts are in sync:

Under 'Domain Structure', click on 'Deployments'

Go to the Next page until you see the 'oacore'

Click on the '+' sign next to the word 'oacore'

Click on the module '/OA_HTML'

Click on the 'Configuration' tab

Set 'Session Timeout (in seconds)' to 1800

Click 'Save'

Under 'Domain Structure', click on 'Deployments'

Go to the Next page until you see the 'oacore' and click on it

Click on the 'Configuration' tab

Set 'Session Timeout (in seconds)' to 1800

Click 'Save'

Click 'Release Configuration'

Set the 'ICX Session Timeout' profile option value at the site level to 30. (Notes: later business users requested to change this profile option to 120, which made them out of sync. But EBS site worked fine after that).

Restart services via adstpall.sh / adstrtal.sh

5. Check oacore connections

When oacore went down, it had a high number of Active Counts in the Console. Navigate to to Services (tree Link) => Data Sources (tree Link) => EBSDataSource (page link) => Monitoring (tab), It got

Server ActiveConnectionsCurrentCount

oacore_server1 218

oacore_server2 119

- Doc ID 1940996.1 (Oracle E-Business Suite 12.2 Data Source Connection Pool Diagnostics) gives details on how to collect data on connection leaks.

Log in to the WebLogic Console. Click on Services (tree link) => Data Sources (tree link) => EBSDataSource (page link) => Monitoring (tab)...

- Doc ID 1905593.1 (Managing Configuration of Oracle HTTP Server and Web Application Services in Oracle E-Business Suite Release 12.2) give a way to add additional oacore managed servers depending upon the user load. 1 oacore JVM can handle up to 150 users. (See 4.4.1 Adding a new managed server)

6. Perform the following to enable debug and help isolate the problem:

- Login to the WLS console

- Environment => Servers => Click on oacore_server<x> (link) for which debug will be enabled => Debug (tab)

- Use Edit and Lock to open the configuration

- Open the Weblogic => Servlet and select line for DebugHttp

- Click on Enable, and then Activate Changes (no bounce is required)

Now the debug is enabled and will log the HTTP request in the $EBS_DOMAIN_HOME/servers/oacore_server<x>/logs/oacore_server<x>.log file

When the dump occurs in server log is enabled this should allow to track back the request(s) just before the dump occurred. Remember to disable it after troubleshooting period.

7. Collecting Fusion Middleware Log Files ( Doc ID 1362900.1 ) (for Oracle Support)

Run this script as the owner of the applications file system to gather the log files for Fusion Middleware components such as NodeManager, AdminServer, forms, oacore and oafm services:

zip -r /tmp/`uname -n`_`date +%m%d%y.%H%M`_FMW.zip $IAS_ORACLE_HOME/../wlserver_10.3/common/nodemanager $EBS_DOMAIN_HOME/servers/oa*/logs/*out* $EBS_DOMAIN_HOME/servers/oa*/logs/*log* $EBS_DOMAIN_HOME/servers/forms*/logs/*out* $EBS_DOMAIN_HOME/servers/forms*/logs/*log* $EBS_DOMAIN_HOME/servers/AdminServer/logs/*out* $EBS_DOMAIN_HOME/servers/AdminServer/logs/*log* $EBS_DOMAIN_HOME/sysman/log/* $EBS_DOMAIN_HOME/servers/oac*/adr/diag/ofm/EBS_domain*/oac*/incident/* $EBS_DOMAIN_HOME/servers/forms_s*/adr/diag/ofm/EBS_domain*/forms_s*/incident/*

Oracle Support recommended some patches for "potentially" fixing the problem. But we did not apply them. Poor database performance might also cause users' session timeouts and hit this error.

8. How to bring the site up quickly?

When the problem happens and EBS site is down, the priority is to bring EBS site up and available. First of all, check if oacore is running or not. Use below command to find the PID of oacore :

$ ps -ef | grep $LOGNAME | grep oacore | grep -v grep | awk '{print $2}'

(or, $ ps -fu $LOGNAME | grep oacore | grep -v grep | awk '{print $2}' )

To find the true problem, also check WLS console is up or not: http://[master_node.domain.com]:wls_adminport/console

If WLS console is available, click on Servers to check the status of oacore process, and recycle/start it under the Control tab. Or, run

$ ./admanagedsrvctl.sh status oacore_server1

You are running admanagedsrvctl.sh version 120.14.12020000.12

Enter the WebLogic Admin password:

oacore_server1 is shutdown

Server specific logs are located at $FMW_HOME/user_projects/domains/EBS_domain_$TWO_TASK/servers/oacore_server1/logs

admanagedsrvctl.sh: exiting with status 0

I did experience that after " ./admanagedsrvctl.sh stop oacore_server1 " ran successfully, " ./admanagedsrvctl.sh status oacore_server1 " showed oacore_server1 was still running. I had to use "Force Shutdown" in the Console to stop it.

If console is not available, most likely Admin Server is down.

$ adadminsrvctl.sh status

You are running adadminsrvctl.sh version 120.10.12020000.11

Enter the WebLogic Admin password:

Enter the APPS Schema password:

The AdminServer is not running

AdminServer logs are located at $FMW_HOME/user_projects/domains/EBS_domain_$TWO_TASK/servers/AdminServer/logs

adadminsrvctl.sh: exiting with status 0

adadminsrvctl.sh: check the logfile $LOG_HOME/appl/admin/log/adadminsrvctl.txt for more information ...

In this case, oacore does not start:

$ ./admanagedsrvctl.sh start oacore_server1

You are running admanagedsrvctl.sh version 120.14.12020000.12

Enter the WebLogic Admin password:

Calling txkChkEBSDependecies.pl to perform dependency checks for oacore_server1

*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS

*** Log File = $LOG_HOME/appl/rgf/TXK/txkChkEBSDependecies_.../txkChkEBSDependecies_....log

Perl script txkChkEBSDependecies.pl got executed successfully

Starting oacore_server1...

Server specific logs are located at

$FMW_HOME/user_projects/domains/EBS_domain_${TWO_TASK}/servers/oacore_server1/logs

admanagedsrvctl.sh: exiting with status 1

admanagedsrvctl.sh: check the logfile $LOG_HOME/appl/admin/log/adoacorectl.txt for more information ...

In another crash, admanagedsrvctl.sh did not take the password and hung, and even adstrtal.sh did the same. We had to kill all OS processes. Log file oacore_server1.out in $FMW_HOME/user_projects/domains/EBS_domain_${TWO_TASK}/servers/oacore_server1/logs had message:

<Error> <ServletContext-/OA_HTML> <BEA-000000> <chain failed

javax.servlet.ServletException: java.lang.OutOfMemoryError: GC overhead limit exceeded

at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:342)

at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)

at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)

at oracle.apps.fnd.security.csrf.GuardFilter.doFilter(GuardFilter.java:311)

at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)

Truncated. see log file for complete stacktrace

Caused By: java.lang.OutOfMemoryError: GC overhead limit exceeded

Memory setting can also be seen in the Console: Environment => Servers => Click on oacore_server1 => Monitoring => Performance

I saw warning "threadpool has stuck threads" in the Console, and also messages in log file oacore_server1.out from time to time. I assume they can be ignored:

<Error> <Net> <BEA-000903> <Failed to communicate with proxy: /80. Will try connection site_name.domain.com/443 now.java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

Truncated. see log file for complete stacktrace

and FORCE_SUSPENDING FORCE_SHUTTING_DOWN

Also see Java error in log:

Forcibly releasing inactive/harvested connection "weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection@21edc9b" back into the data source connection pool "EBSDataSource", currently reserved by: java.lang.Exception

at weblogic.jdbc.common.internal.ConnectionEnv.setup(ConnectionEnv.java:367)

at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:379)

at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:345)

at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:469)

at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)

at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:125)

at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:469)

at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:553)

at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:513)

at sun.reflect.GeneratedMethodAccessor106.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at oracle.apps.fnd.security.DBConnObjWLSDSPool.getConnection(DBConnObjWLSDSPool.java:390)

at oracle.apps.fnd.security.AppsConnectionManagerWLSDS.dbConnect(AppsConnectionManagerWLSDS.java:223)

at oracle.apps.fnd.security.AppsConnectionManagerWLSDS.localAppsConnect(AppsConnectionManagerWLSDS.java:193)

at oracle.apps.fnd.security.AppsConnectionManagerWLSDS.makeGuestConnection(AppsConnectionManagerWLSDS.java:68)

at oracle.apps.fnd.security.DBConnObjDS.getLabelledConnection(DBConnObjDS.java:157)

at oracle.apps.fnd.security.DBConnObj.<init>(DBConnObj.java:233)

at oracle.apps.fnd.security.DBConnObjDS.<init>(DBConnObjDS.java:106) 3-27724233841

Unbalanced oacore connection number

By default, Active Connections Count is not balanced in R12.2 when multiple oacore services are used:

In the Console, navigate to to Services (tree link) -> Data Sources (tree link) -> EBSDataSource (page link) -> Monitoring (tab), I got

Server ActiveConnectionsAverageCount ActiveConnectionsCurrentCount CurrentCapacity LeakedConnectionCount

oacore_server1 40 42 48 1292

oacore_server2 13 13 18 310

The Active Count shall match from another navigation in identifying active user / oacore counts:

Click on Environments => Servers => oacore => Monitoring (tab) => JDBC (tab) that will only show oacore. In fact it will only show the oacore you selected. Lets say you chose oacore_server1 from the oacore cluster. It will not show oacore_server_2

Query showed unbalanced user sessions as well:

SQL> select a.node_name, 'Number of user sessions : ' || count(distinct b.session_id) How_many_user_sessions

from apps.fnd_nodes a,apps.icx_sessions b

where disabled_flag != 'Y' and PSEUDO_FLAG = 'N'

and (last_connect + decode(apps.FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT'),

NULL,limit_time, 0,limit_time,apps.FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT')/60)/24) > sysdate

and a.node_id=b.node_id and counter < limit_connects group by a.node_name

NODE_NAME HOW_MANY_USER_SESSIONS

------------------ --------------------------------------

EBSAPP1P Number of user sessions : 144

EBSAPP2P Number of user sessions : 53

In normal situations, there are two oacore servers, the $CONTEXT_FILE, apps.conf and mod_wl_ohs.conf should include all oacore servers setup including port number.

Here is an example of typical configuration to fix this:

1. $CONTEXT_FILE:

<oacore_server_ports oa_var="s_oacore_server_ports">oacore_server1:7201,oacore_server2:7202</oacore_server_ports>

<oacore_nodes oa_var="s_oacore_nodes">[HOSTNAME]:7201,[HOSTNAME]:7202</oacore_nodes>

2. $IAS_ORACLE_HOME/instances/EBS_web_[SID]_OHS1/config/OHS/EBS_web_[SID]/apps.conf:

######################

# for oacore

######################

ProxyPass balancer://oacorecluster_oamedia

ProxyPassReverse balancer://oacorecluster_oamedia

</Location>

BalancerMember http://[HOSTNAME]:7202/OA_HTML/media

BalancerMember http://[HOSTNAME]:7201/OA_HTML/media

</Proxy>

ProxyPass balancer://oacorecluster_oajava

ProxyPassReverse balancer://oacorecluster_oajava

</Location>

BalancerMember http://[HOSTNAME]:7202/OA_HTML/classes

BalancerMember http://[HOSTNAME]:7201/OA_HTML/classes

</Proxy>

ProxyPass balancer://oacorecluster_oacgi/OA_HTML/txkFNDWRR.pl

ProxyPassReverse balancer://oacorecluster_oacgi/OA_HTML/txkFNDWRR.pl

</Location>

BalancerMember http://[HOSTNAME]:7202

BalancerMember http://[HOSTNAME]:7201

</Proxy>

3. $IAS_ORACLE_HOME/instances/EBS_web_[SID]_OHS1/config/OHS/EBS_web_[SID]/mod_wl_ohs.conf:

SetHandler weblogic-handler

WebLogicCluster [HOSTNAME]:7202,[HOSTNAME]:7201

WLTempDir ${ORACLE_INSTANCE}/tmp

</Location>

This oacore issue was fixed after manually modified files: mod_wl_ohs.conf and apps.conf to add the lost oacore setup. Doc ID 2224190.1

Seems AutoConfig does not update mod_wl_ohs.conf and apps.conf.

Below query "may" tell number of users' connections:

SQL> select count(distinct user_id) "USERS" from icx_sessions

where last_connect > sysdate - 1/24 and user_id != '-1';

USERS

-------------

Reference (do not understand yet)

Data Source Active Connections Current Count exceeds Maximum Capacity of the Pool (Doc ID 1372488.1)

Wednesday, March 9, 2022

FND_OAM_CONTEXT_FILES misses a row of context file

I got below error from using ADOP to apply patches:

*******FATAL ERROR*******

PROGRAM : ($AD_TOP/patch/115/bin/txkADOPEvalSrvStatus.pl)

TIME : Thu Feb 24 12:26:26 2022

FUNCTION: TXK::XML::load_doc [ Level 1 ]

MESSAGES:

error = Cannot open XML file for load

errorno = No such file or directory

file = $NE_BASE/EBSapps/log/adop/.../apply/node_name/TXK_EVAL_apply_.../ctx_files/$CONTEXT_NAME.xml

STACK TRACE

at $AU_TOP/perl/TXK/Error.pm line 168

TXK::Error::abort('TXK::Error', 'HASH(0x3d523b0)') called at $AU_TOP/perl/TXK/XML.pm line 280

TXK::XML::load_doc('TXK::XML=HASH(0x3d49840)', 'HASH(0x3d1c698)') called at $AU_TOP/perl/TXK/XML.pm line 266

TXK::XML::loadDocument('TXK::XML=HASH(0x3d49840)', 'HASH(0x3d1c698)') called at $AD_TOP/patch/115/bin/txkADOPEvalSrvStatus.pl line 1238

... ...

The error and logs do not tell what is the problem. But when I ran ADOP validation, it gave me direction to find the root cause.

$ adop -validate

... ...

[UNEXPECTED] Could not find patch context file from database

[UNEXPECTED]Remote action failed.

... ...

"adop phase=fs_clone" can hit the same error:

Verifying existence of context files in database.

[UNEXPECTED] Could not find patch context file from database

UNEXPECTED]Patch edition context file not found in database for host node_NAME

1. What was the problem?

Most likely "EXEC FND_CONC_CLONE.SETUP_CLEAN;" was executed during a clone or somewhere, which deleted rows in tables:

SQL> select * from fnd_nodes;

SQL> select * from fnd_oam_context_files;

I used query to confirm the row for PATCH context file of the newly added node was not in table FND_OAM_CONTEXT_FILES:

SQL> select distinct(PATH) from FND_OAM_CONTEXT_FILES

where NAME not in ('TEMPLATE','METADATA','config.txt') and CTX_TYPE='A'

and (status is null or upper(status) in ('S','F')) and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = 'patch';

2. Fix: run AutoConfig on PATCH file system

a) Disable a trigger

SQL> conn system/systemPWD

Connected.

SQL> alter trigger ebs_logon disable;

Trigger altered.

b) Set PATCH env and run autoConfig

$ . /<EBS_HOME_BASE>/EBSapps.env patch

$ echo $TWO_TASK

EBSDEV_patch

$ sqlplus apps/appsPWD -- modify tnsnames.ora if it failed

SQL> show user

USER is "APPS"

SQL> exit

$ cd $ADMIN_SCRIPTS_HOME

$ echo $FILE_EDITION

patch

$ ./adautocfg.sh

Enter the APPS user password:

The log file for this session is located at: $INST_TOP/admin/log/MMDDHHMI/adconfig.log

... ...

AutoConfig completed successfully.

c). Enable the trigger

SQL> conn system/systemPWD

Connected.

SQL> alter trigger ebs_logon enable;

Trigger altered.

d). Confirm two rows in FND_OAM_CONTEXT_FILES for patch filesystem now

SQL> connect apps/appsPWD

SQL> select distinct(PATH) from FND_OAM_CONTEXT_FILES

where NAME not in ('TEMPLATE','METADATA','config.txt') and CTX_TYPE='A'

and (status is null or upper(status) in ('S','F'))

and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = 'patch';

AutoConfig also inserted one row to table applsys.FND_APPL_TOPS.

3. Run ADOP again from the beginning. All shall work.

Start a new OS session

$ echo $FILE_EDITION

run

$ adop -validate

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

.. ...

adop exiting with status = 0 (Success)

Notes:

1. After AutoConfig on patch filesystem, it may bring a wrong login settings to the site. You will have to run AutoConfig in RUN filesystem to overwrite them.

2. If this problem was from "adop phase=fs_clone", run it again by "adop phase=fs_clone force=yes" to start it from the beginning to avoid the Apache failure (because some wrong info was saved in adop from previous run).

Unable to initialize SSL environment, nzos call nzosSetCredential returned 28791

Another possible way to fix the problem:

$ . ./EBSapps.env patch

$ echo $FILE_EDITION

patch

$ echo $PATCH_BASE

<EBS_HOME_BASE>/fs2

$ $ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer \

action=upload \ contextfile=/path on fs2/to/$CONTEXT_FILE

logfile=/path/to/patchupload.log

REFERENCE: Doc ID 2090393.1 (Error "Use of uninitialized value $result in split" In File txkADOPValidations.error When Running fs_clone)

Monday, March 7, 2022

Clear BNE / WebADI cache in R12.2

In R12.2, URL link https://[hostname.domain]:portnumber/OA_HTML/BneAdminServlet does not work by default. It works for user with Desktop Integration Manager responsibility.

After logged in under Desktop Integration Manager responsibility, click on a link to open any Oracle Application Framework (FWK) page. Then, try:

https://[hostname.domain]:portnumber/OA_HTML/BneAdminServlet

It provides a link to clear BNE cache.

The URL may give error in the first try:

Desktop Integration Manager is not a valid responsibility for the current user. Please contact your System Administrator.

Clear the middle tier cache to resolve this issue, because middle tier has not picked the change in assigned responsibility yet:

a) Navigate to Functional administrator > Core Service > Caching Framework > Global Configuration

b) Click on Clear All Cache

c) Try the link again.

Thursday, March 3, 2022

Upgrade JDK 7 in EBS R12.2

EBS installation brings in low version of JDK. It is necessary to upgrade JDK to address security vulnerabilities and to meet the need of product support. The easiest way to find the latest JDK version that works with EBS is Oracle release document for quarterly CPU patches. Currently. EBS R12.2 is certified only with JDK 1.7.

EBS R12.2 uses both 64-bit and 32-bit JDK. Use below lines to find current JDK version in an EBS environment:

$ echo $CLASSPATH

$COMMON_TOP/util/jdk32/lib/dt.jar:/... ... etc

$ $ADJVAPRG -version

java version "1.7.0_85"

Java(TM) SE Runtime Environment (build 1.7.0_85-b15)

Java HotSpot(TM) Server VM (build 24.85-b06, mixed mode)

$ echo $AF_CLASSPATH

$COMMON_TOP/util/jdk64/lib/dt.jar:/... ... etc

$ $AFJVAPRG -version

java version "1.7.0_85"

Java(TM) SE Runtime Environment (build 1.7.0_85-b15)

Java HotSpot(TM) 64-Bit Server VM (build 24.85-b06, mixed mode)

The CLASSPATH and AF_CLASSPATH environment variable must contain the necessary Java 7.0 libraries, which should include:

[COMMON_TOP]/util/<Java7>/lib/dt.jar

[COMMON_TOP]/util/<Java7>/lib/tools.jar

[COMMON_TOP]/util/<Java7>/jre/lib/rt.jar

1.7.0_85 is the default version from R12.2 upgrade, and Weblogic console does give warning:

Warning: This server is not secure. Upgrade to Java 1.7.0_191 or higher.

Follow steps in Scenario 3.5.2 of Doc ID 1530033.1 (Using the Latest JDK 7.0 Update with Oracle E-Business Suite Release 12.2) to upgrade it. Currently, JDK 7 Update 331 is recommended by the most recent R12.2 CPU patch (January 2022, Doc ID 2815550.1)

1. First of all, use link in Doc ID 1439822.1 to download patch 33518566 (Oracle JDK 7 Update 331) for both 64 & 32 bit, and then save two files p33518566_170331_Linux-x86-64.zip & p33518566_170331_LINUX.zip to a shared location /u46/path/to/JDK331. After unzip, two files jdk-7u331-linux-x64.tar.gz & jdk-7u331-linux-i586.tar.gz provide two folders under /u46/path/to/JDK331 for staging:

$ tar vzxfp jdk-7u331-linux-x64.tar.gz

$ mv jdk1.7.0_331 jdk1.7.0_331_64

$ tar vzxfp jdk-7u331-linux-i586.tar.gz

$ mv jdk1.7.0_331 jdk1.7.0_331_32

UPDATES in February 2023:

Oracle ended support for JDK 7 after July 2022 (See NOTE:2809708.1 - Oracle Java SE 7 End of Extended Support). So, It is important to read CPU release documents to find where to download the right JDK file for upgrading JDK embedded in E-Business Suites. For example, January 2023 CPU document 2897309.1 ( Oracle E-Business Suite Release 12.2 Critical Patch Update Availability Document (January 2023)) points to Doc ID 2897309.1 (Oracle Critical Patch Update (CPU) Oct 2022 for Oracle Java SE), which has a link to download JDK 1.7.0_361 (patch 34569003) files p34569003_170361_LINUX.zip and p34569003_170361_Linux-x86-64.zip.

2. Confirmed below patches were applied to $ORACLE_HOME:

OracleAS 10.1.2 Patch 16271876

OracleAS 10.1.2 Patch 17907988

OracleAS 10.1.2 Forms Patch 17653437

OracleAS 10.1.2 Patch 16241466

OracleAS 10.1.2 Patch 17645157

Then, shutdown Apps services

$ { echo apps ; echo appsPWD; echo wlsPWD ; } | adstpall.sh @ -mode=allnodes -nopromptmsg

$ ps -ef | grep $LOGNAME

$ echo $FILE_EDITION

run

Make sure there is no open online patching cycle (which is started by "adop phase=prepare"):

$ adop -status

... ...

Node Name Node Type Phase Status Started Finished Elapsed

--------------- ------------------ --------------- ----------------- ---------------- -------------- ------------

node1 master APPLY ACTIVE xxxxx xxxxx xxxxx

CLEANUP NOT STARTED

node2 slave APPLY ACTIVE xxxxx xxxxx xxxxx

CLEANUP NOT STARTED

I upgraded JDK folders in RUN file system directly. I did not use ADOP patching cycle for that, and only ran FS_CLONE after JDK folders are replaced in RUN file system on all nodes.

$ JDKfolder=/u46/path/to/JDK331

$ cd $JDKfolder

$ ls

jdk1.7.0_331_32 jdk-7u331-linux-i586.tar.gz p33518566_170331_Linux-x86-64.zip readme.txt

jdk1.7.0_331_64 jdk-7u331-linux-x64.tar.gz p33518566_170331_LINUX.zip

3. Install New JDK 7.0 and Replace the Existing JDK Home Contents

$ echo $COMMON_TOP/util

$RUN_BASE/EBSapps/comn/util

$ cd $COMMON_TOP/util

$ mv jdk64 jdk64_OLD_2022 <== Or, tar it up by "tar -cvzf jdk64_OLD.tar.gz jdk64"

<== then delete old folder jdk64

$ mv jdk32 jdk32_OLD_2022

$ cp -rp $JDKfolder/jdk1.7.0_331_64 jdk64

$ cp -rp $JDKfolder/jdk1.7.0_331_32 jdk32

4. Upgrade Java (64 bit) in Oracle Fusion Middleware 11g Web Tier

$ echo $FMW_HOME/webtier

$RUN_BASE/FMW_Home/webtier

$ cd $FMW_HOME/webtier

$ mv jdk jdk_OLD_2022

$ cp -rp $JDKfolder/jdk1.7.0_331_64 jdk

5. Upgrade Java (32 bit) in OracleAS 10.1.2 Oracle_Home

$ echo $ORACLE_HOME

$RUN_BASE/EBSapps/10.1.2

$ cd $ORACLE_HOME

$ mv jdk jdk_OLD_2022

$ cp -rp $JDKfolder/jdk1.7.0_331_32 jdk

6. Rebuild Forms and Reports Executables (For UNIX/Linux Only)

Make sure ORACLE_HOME points to 10.1.2 Oracle Home before relinking forms and reports executables.

$ cd $ORACLE_HOME/forms/lib

$ make -f ins_forms.mk sharedlib install

... ...

/usr/bin/ld: warning: -z lazyload ignored.

$ cd $ORACLE_HOME/reports/lib

$ make -f ins_reports.mk install

7. Repeat the steps in other nodes

8. Run autoconfig on all apps-tier nodes

$ cd $ADMIN_SCRIPTS_HOME

$ ./adautocfg.sh

9. Verify and start Apps services

$ $ADJVAPRG -version

java version "1.7.0_331"

Java(TM) SE Runtime Environment (build 1.7.0_331-b06)

Java HotSpot(TM) Server VM (build 24.331-b06, mixed mode)

$ $AFJVAPRG -version

java version "1.7.0_331"

Java(TM) SE Runtime Environment (build 1.7.0_331-b06)

Java HotSpot(TM) 64-Bit Server VM (build 24.331-b06, mixed mode)

$ { echo apps ; echo appsPWD ; echo wlsPWD ; } | adstrtal.sh @ -mode=allnodes -nopromptmsg

A JSP file can be used as a test.

$ cd $OA_HTML

$ vi JDKtest.jsp

$ more JDKtest.jsp

The JDK version is: <%= System.getProperty("java.version") %>

$ perl -x $FND_TOP/patch/115/bin/ojspCompile.pl --compile -s 'JDKtest.jsp'

Access the JSP file from a browser, using the URL:

http://[web server]:[port]/OA_HTML/JDKtest.jsp

10. If no other changes, sync patch file system by fs_clone:

$ adop phase=fs_clone

Additional NOTES:

A) If downtime is not allowed, the upgrade has to be in PATCH file system first. The upgrade steps are in the Oracle document.

a) Check that there is no open online patching cycle

$ adop -status

$ adop phase=prepare

b) Source the EBS environment with the patch file system

$ cd <EBS_BASE_HOME>

/* <EBS_BASE_HOME> is where fs1, fs2, and others are installed. */

$ . EBSapps.env PATCH

$ echo $FILE_EDITION

c) Complete sames steps 3 - 7 above.

d) Run AutoConfig on PATCH file system on all nodes

e) Cut over to the newly installed JDK 7.0 patch file system

$ adop phase=cutover

B) I used the same way to upgrade old JDK in an Oracle 11g Client by replacing the jdk folder under $ORACLE_HOME to address vulnerability found by Qualys scan report.

C) JDK is also installed by rpm on Linux server level. I am not sure EBS needs it or not. Here is from one of my EBS servers:

$ rpm -qa | grep jdk

copy-jdk-configs-3.3-11.el7_9.noarch

java-1.8.0-openjdk-1.8.0.382.b05-1.el7_9.x86_64

java-1.8.0-openjdk-headless-1.8.0.382.b05-1.el7_9.x86_64

jdk1.7-1.7.0_391-fcs.x86_64

$ which java

/bin/java

$ java -version

openjdk version "1.8.0_382" <== It is the OS Java

OpenJDK Runtime Environment (build 1.8.0_382-b05)

OpenJDK 64-Bit Server VM (build 25.382-b05, mixed mode)

$ cd /usr/java <== It is for apps. I guess it may not be used

$ ls -altr

lrwxrwxrwx 1 root root 16 Feb 19 2020 default -> /usr/java/latest

drwxr-xr-x 8 root root 4096 Aug 3 09:27 jdk1.7.0_391-amd64

lrwxrwxrwx 1 root root 28 Aug 3 09:27 latest -> /usr/java/jdk1.7.0_391-amd64