Friday, March 18, 2022

Weblogic, Fusion Middleware and Forms version in R12.2

ETCC script from patch 17537119 is the easiest way to find R12.2 component versions. But, there are other ways to do that. Two documents are useful to find the version of WebLogic and Fusion Middleware in R12.2:

. How To Find the Full WebLogic Server Version and Full Patch Level (Doc ID 1051959.1)
. How To Find The Version Of Oracle Fusion Middleware Used By E-Business Suite 12.2 (Doc ID 2073977.1)

1. WebLogic Server version (in R12.2.10)

$ . $FMW_HOME/wlserver_10.3/server/bin/setWLSEnv.sh
... ...
Your environment has been set.

$  java weblogic.version
WebLogic Server Temporary Patch for BUG13845626 Tue Jan 05 07:49:28 UTC 2021
WebLogic Server 10.3.6.0.210119 PSU Patch for BUG32052267 Mon Nov 23 07:28:31 UTC 2020
WebLogic Server Temporary Patch for BUG13964737 Fri Dec 20 11:32:08 IST 2013
WebLogic Server Temporary Patch for BUG20474010 Sun Mar 01 17:22:18 IST 2015
WebLogic Server Temporary Patch for ${CRS} Mon Jul 30 16:45:20 EDT 2012
WebLogic Server Temporary Patch for ${CRS} Mon Jul 30 16:45:20 EDT 2012
WebLogic Server 10.3.6.0  Tue Nov 15 08:52:36 PST 2011 1441050
Use 'weblogic.version -verbose' to get subsystem information
Use 'weblogic.utils.Versions' to get version information for all modules

$ which java
$COMMON_TOP/util/jdk64/jre/bin/java    <= WebLogic uses 64-bit Java
$ echo $ORACLE_HOME                        <= WebLogic may not use it !
$RUN_BASE/EBSapps/10.1.2

NOTES: script setDomainEnv.sh may do the same:
$ . $EBS_DOMAIN_HOME/bin/setDomainEnv.sh
$ java weblogic.version
... ...

$ echo $EBS_DOMAIN_HOME
$RUN_BASE/FMW_Home/user_projects/domains/EBS_domain

2. Fusion Middleware
Run ETCC script to find Fusion Middleware version, or check its opatch Inventory:

Fusion Middleware is in $IAS_ORACLE_HOME (or, $FMW_HOME/webtier)
$ echo $IAS_ORACLE_HOME
$ export ORACLE_HOME=$IAS_ORACLE_HOME
$ echo $ORACLE_HOME
$FMW_HOME/webtier              <= e.g. under .../fs1/FMW_Home folder

$ export PATH=$IAS_ORACLE_HOME/OPatch:$PATH
$ opatch lsinventory | more

Installed Top-level Products (1):

Oracle WebTier and Utilities CD                                      11.1.1.9.0
There are 1 products installed in this Oracle Home.

3. Forms and Reports version
Forms is in 10.1.2 Oracle home. Use below line to find its version:

$ echo $ORACLE_HOME
$RUN_BASE/EBSapps/10.1.2       <= e.g. under .../fs1/EBSapps folder
$ which java
$COMMON_TOP/util/jdk32/jre/bin/java   <= 32-bit Java

$ ${ORACLE_HOME}/bin/frmcmp_batch help=y |grep 'Forms 10.1 (Form Compiler) Version' |awk '{ print "Oracle Forms and Reports Product version : " $6 }'
Oracle Forms and Reports Product version : 10.1.2.3.0

One reference may help on other components:
How to Check Oracle Fusion Middleware 11g and WebLogic Component Versions? (Doc ID 1151602.1)

Friday, March 11, 2022

R12.2 site is down with error "Failure of Web Server bridge"

EBS R12.2 site is not accessible and login page gives error: 

Failure of Web Server bridge:

No backend server available for connection: timed out after 10 seconds or idempotent set to OFF or method not idempotent.

The message does not tell much about the root cause. Most likely it happens when a large data set was pulled by EBS users from database, which might make oacore crash. We took a few actions to address the problem.

1. Update NodeManager parameters via s_append_nm_jvmargs in ${CONTEXT_FILE} (by default, it is not set).  For details, see Doc ID 2300621.1 (Node Manager Consumes High Virtual Memory).

$ grep s_append_nm_jvmargs $CONTEXT_FILE
         <append_nm_jvmargs oa_var="s_append_nm_jvmargs">-XX:PermSize=256m -XX:MaxPermSize=512m -Xms1024m -Xmx1024m</append_nm_jvmargs>

2. Increase memory for oacore. Use the following steps to customize the managed server configuration via the WebLogic Server Administration Console. See Doc ID 2104351.1 ("java.lang.OutOfMemoryError, msg=Java heap space" Error While Uploading In APCC).

$ grep s_wls_adminport $CONTEXT_FILE

1). Log on to the WebLogic Server Administration Console as weblogic (http://[host.domain.com]:s_wls_adminport/console). 
2). Click on Environment => Servers (link).  
3). Click on oacore_server1 - the managed server whose configuration needs to be updated. 
4). Under Configuration (tab) => Server Start  (tab) => You can see the Arguments Section.
5). Click on Lock and Edit button in the 'Change Center' panel.
6). Update the Heap space from
    -XX:PermSize=128m -XX:MaxPermSize=384m -Xms512m -Xmx512m -
-Djava.security.policy=/.../server/lib/weblogic.policy
to new set:
  -XX:PermSize=512m -XX:MaxPermSize=512m -Xms2048m -Xmx4096m -Djava.security.policy=/.../server/lib/weblogic.policy
7). Click the 'Save' button to save the configuration changes.
8). Once the customizations are complete and saved, click the 'Activate Changes' button in the 'Change Center' panel to activate the changes.

Do the same for oacore_server2 if it is a multi-node instance.
(Same steps can also be done by running script  $ perl $AD_TOP/patch/115/bin/adProvisionEBS.pl)

3. Set the Inactive Connection Timeout parameter to one hour. Steps are in Doc ID 1940996.1 (Oracle E-Business Suite 12.2 Data Source Connection Pool Diagnostics)

Login to the WebLogic Server console
Navigate to Services (tree link) => Data Sources (link) => EBSDataSource (page link) => Connection Pool (tab) => Advanced (Expand arrow on the bottom).
Click on the Lock and Edit Button.
Update the Inactive Connection Timeout to the desired value in seconds: 3600 (for one hour).
Then, click on Save, and the 'Activate Changes' button.

4. Steps to ensure the timeouts are in sync:

Login to WebLogic Admin Console, and click on 'Lock & Edit'
Under 'Domain Structure', click on 'Deployments'
Go to the Next page until you see the 'oacore'
Click on the '+' sign next to the word 'oacore'
Click on the module '/OA_HTML'
Click on the 'Configuration' tab
Set 'Session Timeout (in seconds)' to 1800
Click 'Save'
Under 'Domain Structure', click on 'Deployments'
Go to the Next page until you see the 'oacore' and click on it
Click on the 'Configuration' tab
Set 'Session Timeout (in seconds)' to 1800
Click 'Save'
Click 'Release Configuration'

Set the 'ICX Session Timeout' profile option value at the site level to 30. (Notes: later business users requested to change this profile option to 120, which made them out of sync. But EBS site worked fine after that). 
Restart services via adstpall.sh / adstrtal.sh

5. Check oacore connections 
When oacore went down, it had a high number of Active Counts in the Console. Navigate to to Services (tree Link) => Data Sources (tree Link) => EBSDataSource (page link) => Monitoring (tab), It got

Server  ActiveConnectionsCurrentCount
oacore_server1 218
oacore_server2 119

- Doc ID 1940996.1 (Oracle E-Business Suite 12.2 Data Source Connection Pool Diagnostics) gives details on how to collect data on connection leaks.
Log in to the WebLogic Console. Click on Services (tree link) => Data Sources (tree link) => EBSDataSource (page link) => Monitoring (tab)...

- Doc ID 1905593.1 (Managing Configuration of Oracle HTTP Server and Web Application Services in Oracle E-Business Suite Release 12.2) give a way to add additional oacore managed servers depending upon the user load. 1 oacore JVM can handle up to 150 users. (See 4.4.1 Adding a new managed server)

6. Perform the following to enable debug and help isolate the problem:
- Login to the WLS console 
- Environment => Servers => Click on oacore_server<x> (link) for which debug will be enabled => Debug (tab)
- Use Edit and Lock to open the configuration
- Open the Weblogic => Servlet and select line for DebugHttp
- Click on Enable, and then Activate Changes (no bounce is required)

Now the debug is enabled and will log the HTTP request in the $EBS_DOMAIN_HOME/servers/oacore_server<x>/logs/oacore_server<x>.log file

When the dump occurs in server log is enabled this should allow to track back the request(s) just before the dump occurred. Remember to disable it after troubleshooting period.

7. Collecting Fusion Middleware Log Files ( Doc ID 1362900.1 ) (for Oracle Support)

Run this script as the owner of the applications file system to gather the log files for Fusion Middleware components such as NodeManager, AdminServer, forms, oacore and oafm services:
zip -r /tmp/`uname -n`_`date +%m%d%y.%H%M`_FMW.zip $IAS_ORACLE_HOME/../wlserver_10.3/common/nodemanager $EBS_DOMAIN_HOME/servers/oa*/logs/*out* $EBS_DOMAIN_HOME/servers/oa*/logs/*log* $EBS_DOMAIN_HOME/servers/forms*/logs/*out* $EBS_DOMAIN_HOME/servers/forms*/logs/*log* $EBS_DOMAIN_HOME/servers/AdminServer/logs/*out* $EBS_DOMAIN_HOME/servers/AdminServer/logs/*log* $EBS_DOMAIN_HOME/sysman/log/* $EBS_DOMAIN_HOME/servers/oac*/adr/diag/ofm/EBS_domain*/oac*/incident/* $EBS_DOMAIN_HOME/servers/forms_s*/adr/diag/ofm/EBS_domain*/forms_s*/incident/*

Oracle Support recommended some patches for "potentially" fixing the problem. But we did not apply them. Poor database performance might also cause users' session timeouts and hit this error.

8. How to bring the site up quickly?

When the problem happens and EBS site is down, the priority is to bring EBS site up and available. First of all, check if oacore is running or not. Use below command to find the PID of oacore :

$ ps -ef | grep $LOGNAME | grep oacore | grep -v grep | awk '{print $2}'
(or, $ ps -fu $LOGNAME | grep oacore | grep -v grep | awk '{print $2}' )

To find the true problem, also check WLS console is up or not: http://[master_node.domain.com]:wls_adminport/console

If WLS console is available, click on Servers to check the status of oacore process, and recycle/start it under the Control tab. Or, run

$ ./admanagedsrvctl.sh status oacore_server1
You are running admanagedsrvctl.sh version 120.14.12020000.12
Enter the WebLogic Admin password:
oacore_server1 is shutdown
Server specific logs are located at $FMW_HOME/user_projects/domains/EBS_domain_$TWO_TASK/servers/oacore_server1/logs
admanagedsrvctl.sh: exiting with status 0

I did experience that after " ./admanagedsrvctl.sh stop oacore_server1 " ran successfully, " ./admanagedsrvctl.sh status oacore_server1 " showed oacore_server1 was still running. I had to use "Force Shutdown" in the Console to stop it.

If console is not available, most likely Admin Server is down. 
$ adadminsrvctl.sh status
You are running adadminsrvctl.sh version 120.10.12020000.11
Enter the WebLogic Admin password:
Enter the APPS Schema password:
 The AdminServer is not running
AdminServer logs are located at $FMW_HOME/user_projects/domains/EBS_domain_$TWO_TASK/servers/AdminServer/logs
adadminsrvctl.sh: exiting with status 0
adadminsrvctl.sh: check the logfile $LOG_HOME/appl/admin/log/adadminsrvctl.txt for more information ...

In this case, oacore does not start:
$ ./admanagedsrvctl.sh start oacore_server1
You are running admanagedsrvctl.sh version 120.14.12020000.12
Enter the WebLogic Admin password:
Calling txkChkEBSDependecies.pl to perform dependency checks for oacore_server1
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = $LOG_HOME/appl/rgf/TXK/txkChkEBSDependecies_.../txkChkEBSDependecies_....log 
Perl script txkChkEBSDependecies.pl got executed successfully
Starting oacore_server1...
Server specific logs are located at 
$FMW_HOME/user_projects/domains/EBS_domain_${TWO_TASK}/servers/oacore_server1/logs
admanagedsrvctl.sh: exiting with status 1
admanagedsrvctl.sh: check the logfile $LOG_HOME/appl/admin/log/adoacorectl.txt for more information ...  

In another crash, admanagedsrvctl.sh did not take the password and hung, and even adstrtal.sh did the same. We had to kill all OS processes. Log file oacore_server1.out in $FMW_HOME/user_projects/domains/EBS_domain_${TWO_TASK}/servers/oacore_server1/logs had message:

<Error> <ServletContext-/OA_HTML> <BEA-000000> <chain failed
javax.servlet.ServletException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:342)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at oracle.apps.fnd.security.csrf.GuardFilter.doFilter(GuardFilter.java:311)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
Truncated. see log file for complete stacktrace
Caused By: java.lang.OutOfMemoryError: GC overhead limit exceeded

Memory setting can also be seen in the Console: Environment => Servers => Click on oacore_server1 => Monitoring => Performance

I saw warning "threadpool has stuck threads" in the Console, and also messages in log file oacore_server1.out from time to time. I assume they can be ignored:

<Error> <Net> <BEA-000903> <Failed to communicate with proxy: /80. Will try connection site_name.domain.com/443 now.java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
Truncated. see log file for complete stacktrace
and FORCE_SUSPENDING FORCE_SHUTTING_DOWN

Also see Java error in log:
Forcibly releasing inactive/harvested connection "weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection@21edc9b" back into the data source connection pool "EBSDataSource", currently reserved by: java.lang.Exception
        at weblogic.jdbc.common.internal.ConnectionEnv.setup(ConnectionEnv.java:367)
        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:379)
        at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:345)
        at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:469)
        at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
        at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:125)
        at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:469)
        at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:553)
        at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:513)
        at sun.reflect.GeneratedMethodAccessor106.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at oracle.apps.fnd.security.DBConnObjWLSDSPool.getConnection(DBConnObjWLSDSPool.java:390)
        at oracle.apps.fnd.security.AppsConnectionManagerWLSDS.dbConnect(AppsConnectionManagerWLSDS.java:223)
        at oracle.apps.fnd.security.AppsConnectionManagerWLSDS.localAppsConnect(AppsConnectionManagerWLSDS.java:193)
        at oracle.apps.fnd.security.AppsConnectionManagerWLSDS.makeGuestConnection(AppsConnectionManagerWLSDS.java:68)
        at oracle.apps.fnd.security.DBConnObjDS.getLabelledConnection(DBConnObjDS.java:157)
        at oracle.apps.fnd.security.DBConnObj.<init>(DBConnObj.java:233)
        at oracle.apps.fnd.security.DBConnObjDS.<init>(DBConnObjDS.java:106)    3-27724233841

Unbalanced oacore connection number

By default, Active Connections Count is not balanced in R12.2 when multiple oacore services are used:

In the Console, navigate to to Services (tree link) -> Data Sources (tree link) -> EBSDataSource (page link) -> Monitoring (tab), I got
Server ActiveConnectionsAverageCount ActiveConnectionsCurrentCount CurrentCapacity LeakedConnectionCount 
oacore_server1     40 42 48 1292
oacore_server2     13 13 18 310

The Active Count shall match from another navigation in identifying active user / oacore counts:
Click on Environments => Servers => oacore => Monitoring (tab) => JDBC (tab) that will only show oacore. In fact it will only show the oacore you selected. Lets say you chose oacore_server1 from the oacore cluster. It will not show oacore_server_2

Query showed unbalanced user sessions as well:
SQL> select a.node_name, 'Number of user sessions : ' || count(distinct b.session_id) How_many_user_sessions 
from apps.fnd_nodes a,apps.icx_sessions b
where disabled_flag != 'Y' and PSEUDO_FLAG = 'N'
and (last_connect + decode(apps.FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT'),
NULL,limit_time, 0,limit_time,apps.FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT')/60)/24) > sysdate
and a.node_id=b.node_id and counter < limit_connects group by a.node_name
/

NODE_NAME  HOW_MANY_USER_SESSIONS
------------------ --------------------------------------
EBSAPP1P       Number of user sessions : 144
EBSAPP2P       Number of user sessions : 53

In normal situations, there are two oacore servers, the $CONTEXT_FILE, apps.conf and mod_wl_ohs.conf should include all oacore servers setup including port number.

Here is an example of typical configuration to fix this:

1. $CONTEXT_FILE:

<oacore_server_ports oa_var="s_oacore_server_ports">oacore_server1:7201,oacore_server2:7202</oacore_server_ports>
<oacore_nodes oa_var="s_oacore_nodes">[HOSTNAME]:7201,[HOSTNAME]:7202</oacore_nodes>

2. $IAS_ORACLE_HOME/instances/EBS_web_[SID]_OHS1/config/OHS/EBS_web_[SID]/apps.conf:

######################
# for oacore
######################
<Location /OA_MEDIA>
ProxyPass balancer://oacorecluster_oamedia
ProxyPassReverse balancer://oacorecluster_oamedia
</Location>
<Proxy balancer://oacorecluster_oamedia>
BalancerMember http://[HOSTNAME]:7202/OA_HTML/media
BalancerMember http://[HOSTNAME]:7201/OA_HTML/media
</Proxy>

<Location /OA_JAVA>
ProxyPass balancer://oacorecluster_oajava
ProxyPassReverse balancer://oacorecluster_oajava
</Location>
<Proxy balancer://oacorecluster_oajava>
BalancerMember http://[HOSTNAME]:7202/OA_HTML/classes
BalancerMember http://[HOSTNAME]:7201/OA_HTML/classes
</Proxy>

<Location /OA_CGI/FNDWRR.exe>
ProxyPass balancer://oacorecluster_oacgi/OA_HTML/txkFNDWRR.pl
ProxyPassReverse balancer://oacorecluster_oacgi/OA_HTML/txkFNDWRR.pl
</Location>
<Proxy balancer://oacorecluster_oacgi/OA_HTML/txkFNDWRR.pl>
BalancerMember http://[HOSTNAME]:7202
BalancerMember http://[HOSTNAME]:7201
</Proxy>

3. $IAS_ORACLE_HOME/instances/EBS_web_[SID]_OHS1/config/OHS/EBS_web_[SID]/mod_wl_ohs.conf:

<Location /OA_HTML>
SetHandler weblogic-handler
WebLogicCluster [HOSTNAME]:7202,[HOSTNAME]:7201
WLTempDir ${ORACLE_INSTANCE}/tmp
</Location>

This oacore issue was fixed after manually modified files: mod_wl_ohs.conf and apps.conf to add the lost oacore setup. Doc ID 2224190.1

Seems AutoConfig does not update mod_wl_ohs.conf and apps.conf.

Below query "may" tell number of users' connections:
SQL> select count(distinct user_id) "USERS" from icx_sessions
where  last_connect > sysdate - 1/24 and user_id != '-1';
     USERS
-------------
        80

Reference (do not understand yet)
Data Source Active Connections Current Count exceeds Maximum Capacity of the Pool (Doc ID 1372488.1)

Wednesday, March 9, 2022

FND_OAM_CONTEXT_FILES misses a row of context file

I got below error from using ADOP to apply patches:

*******FATAL ERROR*******
PROGRAM : ($AD_TOP/patch/115/bin/txkADOPEvalSrvStatus.pl)
TIME    : Thu Feb 24 12:26:26 2022
FUNCTION: TXK::XML::load_doc [ Level 1 ]
MESSAGES:
error = Cannot open XML file for load
errorno = No such file or directory
file = $NE_BASE/EBSapps/log/adop/.../apply/node_name/TXK_EVAL_apply_.../ctx_files/$CONTEXT_NAME.xml

STACK TRACE
 at $AU_TOP/perl/TXK/Error.pm line 168
        TXK::Error::abort('TXK::Error', 'HASH(0x3d523b0)') called at $AU_TOP/perl/TXK/XML.pm line 280
        TXK::XML::load_doc('TXK::XML=HASH(0x3d49840)', 'HASH(0x3d1c698)') called at $AU_TOP/perl/TXK/XML.pm line 266
        TXK::XML::loadDocument('TXK::XML=HASH(0x3d49840)', 'HASH(0x3d1c698)') called at $AD_TOP/patch/115/bin/txkADOPEvalSrvStatus.pl line 1238
... ...        

The error and logs do not tell what is the problem. But when I ran ADOP validation, it gave me direction to find the root cause.

$ adop -validate
... ... 
    [UNEXPECTED] Could not find patch context file from database
    [UNEXPECTED]Remote action failed.
... ...
"adop phase=fs_clone" can hit the same error:

   Verifying existence of context files in database.
       [UNEXPECTED] Could not find patch context file from database
       UNEXPECTED]Patch edition context file not found in database for host node_NAME

1. What was the problem?
Most likely "EXEC FND_CONC_CLONE.SETUP_CLEAN;" was executed during a clone or somewhere, which deleted rows in tables:
SQL> select * from fnd_nodes;
SQL> select * from fnd_oam_context_files;

I used query to confirm the row for PATCH context file of the newly added node was not in table FND_OAM_CONTEXT_FILES:

SQL> select distinct(PATH) from FND_OAM_CONTEXT_FILES 
where NAME not in ('TEMPLATE','METADATA','config.txt') and CTX_TYPE='A' 
and (status is null or upper(status) in ('S','F')) and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = 'patch';

2. Fix: run AutoConfig on PATCH file system
a) Disable a trigger
SQL> conn system/systemPWD
Connected.
SQL> alter trigger ebs_logon disable;
Trigger altered.

b) Set PATCH env and run autoConfig
$ . /<EBS_HOME_BASE>/EBSapps.env patch

$ echo $TWO_TASK
EBSDEV_patch
$ sqlplus apps/appsPWD   -- modify tnsnames.ora if it failed
SQL> show user
USER is "APPS"
SQL> exit

$ cd $ADMIN_SCRIPTS_HOME
$ echo $FILE_EDITION
patch
$ ./adautocfg.sh
Enter the APPS user password:

The log file for this session is located at: $INST_TOP/admin/log/MMDDHHMI/adconfig.log
... ...
AutoConfig completed successfully.

c). Enable the trigger
SQL> conn system/systemPWD
Connected.
SQL> alter trigger ebs_logon enable;
Trigger altered.

d). Confirm two rows in FND_OAM_CONTEXT_FILES for patch filesystem now
SQL> connect apps/appsPWD
SQL> select distinct(PATH) from FND_OAM_CONTEXT_FILES 
where NAME not in ('TEMPLATE','METADATA','config.txt') and CTX_TYPE='A' 
and (status is null or upper(status) in ('S','F')) 
and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = 'patch';

AutoConfig also inserted one row to table applsys.FND_APPL_TOPS.

3. Run ADOP again from the beginning. All shall work.
Start a new OS session 
$ echo $FILE_EDITION
run
$  adop -validate
Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:
.. ...
adop exiting with status = 0 (Success)

Notes: 
1. After AutoConfig on patch filesystem, it may bring a wrong login settings to the site. You will have to run AutoConfig in RUN filesystem to overwrite them.
2. If this problem was from "adop phase=fs_clone", run it again by "adop phase=fs_clone force=yes" to start it from the beginning to avoid the Apache failure (because some wrong info was saved in adop from previous run).
Unable to initialize SSL environment, nzos call nzosSetCredential returned 28791

Another possible way to fix the problem:

$ . ./EBSapps.env patch
$ echo $FILE_EDITION
patch
$ echo $PATCH_BASE
<EBS_HOME_BASE>/fs2
$ $ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer \
        action=upload \ contextfile=/path on fs2/to/$CONTEXT_FILE
        logfile=/path/to/patchupload.log

REFERENCE:  Doc ID 2090393.1 (Error "Use of uninitialized value $result in split" In File txkADOPValidations.error When Running fs_clone)

Monday, March 7, 2022

Clear BNE / WebADI cache in R12.2

In R12.2, URL link https://[hostname.domain]:portnumber/OA_HTML/BneAdminServlet does not work by default.  It works for user with Desktop Integration Manager responsibility.

After logged in under Desktop Integration Manager responsibility, click on a link to open any Oracle Application Framework (FWK) page. Then, try:

https://[hostname.domain]:portnumber/OA_HTML/BneAdminServlet 

It provides a link to clear BNE cache.

The URL may give error in the first try:

Desktop Integration Manager is not a valid responsibility for the current user. Please contact your System Administrator.

Clear the middle tier cache to resolve this issue, because middle tier has not picked the change in assigned responsibility yet:
a) Navigate to Functional administrator > Core Service > Caching Framework > Global Configuration
b) Click on Clear All Cache 
c) Try the link again.