Wednesday, August 24, 2022

Re-create three AZ tables of iSetup

After database Export and Import for 19c database upgrade, three AZ tables for EBS iSetup becomes missing in 19c database:
SQL> select * from dba_tables where owner = 'AZ' and table_name in
 ('AZ_REQUESTS','AZ_REPORTER_DATA','AZ_DIFF_RESULTS'); 
no rows selected

Even iSetup is not used, Business user still wants to get them back. Oracle Doc ID 832459.1 (How To Cleanup Invalid Oracle iSetup (AZ) Tables And Recreate) provides steps on re-creating them.

$ appPWD='PWDappsXXX'
$ sysPWD='PWDsysXXX'

1.  Make sure account XDB is OPEN

SQL> SELECT status FROM dba_registry WHERE comp_id = 'XDB';

STATUS
-----------
VALID

SQL> SELECT username, account_status FROM dba_users WHERE username = 'XDB';

USERNAME  ACCOUNT_STATUS
----------------- ---------------
XDB               LOCKED

SQL> conn system/$sysPWD
Connected.
SQL> ALTER USER xdb IDENTIFIED BY xdb ACCOUNT UNLOCK;
ALTER USER xdb IDENTIFIED BY xdb ACCOUNT UNLOCK
*
ERROR at line 1:
ORA-01031: insufficient privileges

Fix: sysdba needs to unlock XDB account on the root$ container

SQL> SELECT username, account_status FROM dba_users WHERE username = 'XDB';

USERNAME  ACCOUNT_STATUS
----------------- ---------------
XDB               OPEN

2. Get the script ready
SQL> show user
USER is "SYSTEM"
SQL> set pagesize 200
SQL> column OWNER format a15
SQL> column TYPE_NAME format a30
SQL> SELECT owner, type_name FROM all_types WHERE type_name LIKE 'AZ%';

OWNER           TYPE_NAME 
-------------- ------------------------------    
APPS_NE         AZ_RPTDATA_H  
APPS_NE         AZ_RPTDATA_V_COLL   
APPS_NE         AZ_RPTDATA_V_TAGS  
APPS_NE         AZ_RPTDIFF_H     
APPS_NE         AZ_RPTDIFF_V_COLL         
APPS_NE         AZ_RPTDIFF_V_TAGS   
APPS_NE         AZ_SELSETS_EXT    
APPS_NE         AZ_SELSETS_H     
APPS_NE         AZ_SELSETS_H_COLL      
APPS_NE         AZ_SELSETS_V652_T   
APPS_NE         AZ_SELSETS_V_COLL 

11 rows selected.

3. Run "drop type" script as SYSTEM
SQL> show user
USER is "SYSTEM"
SQL> SELECT 'drop type ' || owner || '.' || type_name || ' force;' FROM all_types WHERE type_name LIKE 'AZ%';

Then run below lines to drop those types forcefully as SYSTEM:

drop type APPS_NE.AZ_RPTDATA_H force; 
drop type APPS_NE.AZ_RPTDATA_V_COLL force;
drop type APPS_NE.AZ_RPTDATA_V_TAGS force;
drop type APPS_NE.AZ_RPTDIFF_H force;      
drop type APPS_NE.AZ_RPTDIFF_V_COLL force;   
drop type APPS_NE.AZ_RPTDIFF_V_TAGS force; 
drop type APPS_NE.AZ_SELSETS_EXT force;    
drop type APPS_NE.AZ_SELSETS_H force;      
drop type APPS_NE.AZ_SELSETS_H_COLL force;  
drop type APPS_NE.AZ_SELSETS_V652_T force; 
drop type APPS_NE.AZ_SELSETS_V_COLL force;  

4. Reset AZ password if it is lost. Then, login as AZ to drop three tables if they exist.

$ FNDCPASS apps/$appPWD 0 Y system/$sysPWD ORACLE AZ azPWD

$ sqlplus az/azPWD
SQL> DROP TABLE AZ_REQUESTS;
ERROR at line 1:
ORA-00942: table or view does not exist
SQL> DROP TABLE AZ_REPORTER_DATA;
SQL> DROP TABLE AZ_DIFF_RESULTS;

5. Run 3 XDF scripts on OS level to create 3 tables and 6 indexes:

$AZ_TOP/patch/115/xdf/az_diff_results.xdf
$AZ_TOP/patch/115/xdf/az_reporter_data.xdf
$AZ_TOP/patch/115/xdf/az_requests.xdf

Use "tnsping $TWO_TASK" to get the database connection info to replace part of below lines.

$ cd $AZ_TOP/patch/115/xdf

$ adjava "-mx128m" "-nojit" "oracle.apps.fnd.odf2.FndXdfCmp" \
"az" "azPWD" "apps" "$appPWD" \
"thin" "(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=<database-host>)(PORT=<database-port>))(CONNECT_DATA=(SID=<database-SID>)))" \
"table" "az_diff_results.xdf" "$FND_TOP/patch/115/xdf/xsl"
 
$ adjava "-mx128m" "-nojit" "oracle.apps.fnd.odf2.FndXdfCmp" \
"az" "azPWD" "apps" "$appPWD" \
"thin" "(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=<database-host>)(PORT=<database-port>))(CONNECT_DATA=(SID=<database-SID>)))" \
"table" "az_reporter_data.xdf" "$FND_TOP/patch/115/xdf/xsl"

$ adjava "-mx128m" "-nojit" "oracle.apps.fnd.odf2.FndXdfCmp" \
"az" "azPWD" "apps" "$appPWD" \
"thin" "(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=<database-host>)(PORT=<database-port>))(CONNECT_DATA=(SID=<database-SID>)))" \
"table" "az_requests.xdf" "$FND_TOP/patch/115/xdf/xsl"

SQL> desc AZ_DIFF_RESULTS
 Name                                      Null?    Type
 ----------------------------------------- -------- ----------------------------
 NAME                                              VARCHAR2(240)
 DISPLAY_NAME                            VARCHAR2(240)
 REQUEST_ID                                NOT NULL NUMBER(15)
 SOURCE                                        NOT NULL VARCHAR2(240)
 TYPE                                               NUMBER(2)
 ID                                                    NOT NULL NUMBER
 PARENT_ID                                      NUMBER
 HASHCODE_DETAILS                     VARCHAR2(2000)
 DEPTH                                              NUMBER
 IS_DIFFERENT                                 VARCHAR2(1)
 IS_TRANSFORMED                          VARCHAR2(1)
 SHOW_ONLY_DIFF                          VARCHAR2(1)
 PARAM1                                             VARCHAR2(240)
 PARAM2                                             VARCHAR2(240)
 PARAM3                                             VARCHAR2(240)
 PARAM4                                             VARCHAR2(240)
 PARAM5                                             VARCHAR2(240)
 DETAIL_LOG_STATUS                    VARCHAR2(255)
 DETAIL_LOG_MSG                         VARCHAR2(2000)

6. Run 3 scripts to restore the iSetup XMLDB Schemas

$ sqlplus apps/$appPWD @$AZ_TOP/patch/115/sql/azhDiffSchema.sql $sysPWD
... ...
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Connected.
declare
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [qmtcolcb_nomatch], [ID],
[annotation], [], [], [], [], [], [], [], [], []
ORA-06512: at "XDB.DBMS_XMLSCHEMA_INT", line 122
ORA-06512: at "XDB.DBMS_XMLSCHEMA", line 106
ORA-06512: at line 5

Fix:  Database has been started with the oracle user having the wrong LIBPATH set. After 19c upgrade, some of the old 12c paths mixed in with new 19c.  The fix requires a database bounce. 
See Doc ID 1292089.1 (Primary Note for Oracle XML Database (XDB) Install / Deinstall)

$ sqlplus apps/$appPWD @$AZ_TOP/patch/115/sql/azhRptSchema.sql $sysPWD

$ sqlplus apps/$appPWD @$AZ_TOP/patch/115/sql/azhSelSetSchema.sql $sysPWD
... ...
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Connected.
declare
*
ERROR at line 1:
ORA-20101: Exception occurred in Selection Set Schema Registration procedure
:ORA-64479: element 'AZ_SELSETS_V' is missing 'SQLType' annotation; schema
cannot be registered by common users
ORA-06512: at line 92

Fix:  Apply patch 31791566
adop phase=apply apply_mode=downtime patches=31791566 patchtop=/.../fs_ne/EBSapps/patch

7. Restore iSetup specific XMLTYPE columns
sqlplus apps/$appPWD @$AZ_TOP/patch/115/sql/azhAddDiffCol.sql az azPWD
sqlplus apps/$appPWD @$AZ_TOP/patch/115/sql/azhAddDriver.sql az azPWD
sqlplus apps/$appPWD @$AZ_TOP/patch/115/sql/azhAddRptCol.sql az azPWD
sqlplus apps/$appPWD @$AZ_TOP/patch/115/sql/azhAddSelSet.sql az azPWD

8. For 12.2 only. All steps have to be done outside of a patching cycle (i.e. prepare has not been run)

SQL> show user
USER is "APPS"
SQL> set serveroutput on
SQL> exec ad_zd_table.patch('AZ','AZ_REQUESTS');
PL/SQL procedure successfully completed.

SQL> exec ad_zd_table.patch('AZ','AZ_REPORTER_DATA');
PL/SQL procedure successfully completed.

SQL> exec ad_zd_table.patch('AZ','AZ_DIFF_RESULTS');
PL/SQL procedure successfully completed.

SQL> desc AZ_DIFF_RESULTS
 Name                                      Null?    Type
 ----------------------------------------- -------- ----------------------------
 NAME                                              VARCHAR2(240)
 DISPLAY_NAME                            VARCHAR2(240)
 REQUEST_ID                                NOT NULL NUMBER(15)
 SOURCE                                        NOT NULL VARCHAR2(240)
 TYPE                                               NUMBER(2)
 ID                                                    NOT NULL NUMBER
 PARENT_ID                                      NUMBER
 HASHCODE_DETAILS                     VARCHAR2(2000)
 DEPTH                                              NUMBER
 IS_DIFFERENT                                 VARCHAR2(1)
 IS_TRANSFORMED                          VARCHAR2(1)
 SHOW_ONLY_DIFF                          VARCHAR2(1)
 PARAM1                                             VARCHAR2(240)
 PARAM2                                             VARCHAR2(240)
 PARAM3                                             VARCHAR2(240)
 PARAM4                                             VARCHAR2(240)
 PARAM5                                             VARCHAR2(240)
 DETAIL_LOG_STATUS                    VARCHAR2(255)
 DETAIL_LOG_MSG                          VARCHAR2(2000)
 ATTR_DIFF                                        SYS.XMLTYPE(XMLSchema "http:
                                                            //isetup.oracle.com/2006/dif
                                                            fresultdata.xsd" Element "H"
                                                            )

SQL> SELECT owner, type_name FROM all_types WHERE type_name LIKE 'AZ%';
OWNER           TYPE_NAME
--------------- ------------------------------
SYSTEM          AZ_RPTDATA_H
SYSTEM          AZ_RPTDATA_V_COLL
SYSTEM          AZ_RPTDATA_V_TAGS
SYSTEM          AZ_RPTDIFF_H
SYSTEM          AZ_RPTDIFF_V_COLL
SYSTEM          AZ_RPTDIFF_V_TAGS
SYSTEM          AZ_SELSETS_EXT
SYSTEM          AZ_SELSETS_H
SYSTEM          AZ_SELSETS_H_COLL
SYSTEM          AZ_SELSETS_V
SYSTEM          AZ_SELSETS_V_COLL

11 rows selected.

9. Provide appropriate grants

$ sqlplus apps/$appPWD  @$AZ_TOP/patch/115/sql/azR12grants.sql $sysPWD
... ...
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Connected.

Grant succeeded.
Grant succeeded.
Grant succeeded.
Grant succeeded.
Grant succeeded.
Grant succeeded.
Commit complete.

iSetup tables have column of data-type "SYS.XMLTYPE". If it becomes invalid, EBS may hit below message. That may require table re-creation also.
PLS-00905: object AZ.AZ_<xxxxx> is invalid 

Saturday, July 30, 2022

Apply R12.2 April 2022 CPU patch

We decided to apply April 2022 CPU to EBS. To make patching consistent in both db side and EBS side, DBA applied 2022 April database PSU together with database 19c upgrade. The document for applying April 2022 CPU is 2856621.1. I followed below steps and order to apply it. As always, first of all identify which patches are required and download all patch files to a central location, such as /s46/path/to/CPU_April22

1. Confirm only patch 30399994 was applied:
SQL> select * from ad_bugs where bug_number in (31856779, 30399994);

30399994 Patch R12.ATG_PF.C.delta.9 (Application Technology Family)
31856779 Patch R12.ATG_PF.C.delta.10

2. Identify and decide 9 EBS patches will be applied in this time:
CPU patch:
33782739 RELEASE 12.2: CPU PATCH FOR APR 2022

Post Patching:
33908208:R12.FWK.C (12.2.10 CONSOLIDATED PATCH FOR APR CPU 2022 FIXES FOR FWK)
33207251:R12.FND.C (Application Object Library. Applied as part of 33782739)
32980025:R12.AK.C (Application Framework)
33286000:R12.OAM.C (Oracle Applications Manager)
33625264:R12.OWF.C (Workflow)

Oracle support recommended when troubleshooting oacore (optional):
34005885:R12.FWK.C ORACLE APPL FRAMEWORK (FWK) RELEASE 12.2.10 BUNDLE 9
33659063:R12.FWK.C Image Patch for 12.2.10 Post Consolidated Patch Product OAF
33973100:R12.FND.C - CONNECTION LEAK _PAGES._JSP._FND.__CLOSE._JSPSERVICE 

Patch for a bug when re-creating AZ tables (Doc ID 2816670.1):
p31791566_R12.AZ.C_R12_GENERIC.zip

$ cd $PATCH_TOP

$ cp /s46/path/to/CPU_April22/EBS/*.zip .      (9 zip files)

unzip p33782739_12.2.0_R12_LINUX.zip

unzip p33908208_R12.FWK.C_R12_GENERIC.zip
unzip p32980025_R12.AK.C_R12_GENERIC.zip
unzip p33286000_R12.OAM.C_R12_GENERIC.zip
unzip p33625264_R12.OWF.C_R12_GENERIC.zip

unzip p33659063_R12.FWK.C_R12_GENERIC.zip
unzip p33973100_R12.FND.C_R12_GENERIC.zip
unzip p34005885_R12.FWK.C_R12_GENERIC.zip

unzip p31791566_R12.AZ.C_R12_GENERIC.zip

$ echo $FILE_EDITION
run 

Note: adgrants.sql from patch 33782739 is an older version than the one in $APPL_TOP/admin

$ adop phase=apply apply_mode=downtime patches=33782739 patchtop=$NE_BASE/EBSapps/patch
                                                                                      <== it may take one hour
Post-patch steps (from README):
a) Run autoConfig in RUN file system on all Apps-tie nodes
$ ./adautocfg.sh     
b) $ perl $AD_TOP/bin/admkappsutil.pl
c) Run autoConfig on database node       (Note: I skipped this step)
$ adconfig.sh contextfile=$ORACLE_HOME/.../appsutil/$CONTEXT_NAME.xml

$ adop phase=apply apply_mode=downtime patches=33908208,34005885,33659063,33973100,32980025,33286000,33625264  patchtop=$NE_BASE/EBSapps/patch

$ adop phase=apply apply_mode=downtime patches=31791566 patchtop=$NE_BASE/EBSapps/patch

-- Verify all 10 patches were applied to all nodes
SQL> SELECT adb.bug_number, aas.name appl_top_name, adb.creation_date, adb.language,
decode(ad_patch.is_patch_applied('R12',aas.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status
FROM ad_bugs adb,
(select distinct appltop_id appl_top_id, node_name name from ad_adop_sessions 
           where node_name in (select node_name from ADOP_VALID_NODES) ) aas
where adb.bug_number in (
'33782739',
'33908208',
'34005885',
'33659063',
'33973100',
'33207251',
'32980025',
'33286000',
'33625264',
'31791566'
) order by creation_date desc, adb.bug_number,aas.name,adb.language;

3. Run latest ETCC (patch 17537119). This will help in identifying and confirming what patches will be applied.

4. Apply patches for Oracle WebLogic Server (WLS) 10.3.6.0

$ cd $FMW_HOME/utils/bsu/cache_dir                       
$ cp /s46/path/to/CPU_April22/WLS/*.zip .                   (7 .zip files)

$ mv p33800106_1036_Generic.zip ../.
$ cd ..
$ ls
$ unzip p33800106_1036_Generic.zip      (from 33845432)
$ chmod +x bsu_update.sh
$ bsu.sh -version
$ ./bsu_update.sh install           <== it runs very quick.
Installing Smart Update V5...
Updating bsu modules
Update was successful.
$ bsu.sh -version

$ cd cache_dir

$ unzip -o p33791826_1036_Generic.zip  (33946345 WLS PATCH SET UPDATE 10.3.6.0.220419)
$ unzip -o p13845626_10360220419_Generic.zip  (33812339 CONSOLIDATED FMW FIXES)
$ unzip -o p13964737_10360220419_Generic.zip  (zip file from 33812339)
$ unzip -o p33560682_1036_Generic.zip                (from 33812339)
$ unzip -o p31241365_1036_Linux-x86-64.zip      (from 33812339)
$ unzip -o p31042881_1036_Generic.zip                (from 33812339)

$ cd ..
$ ./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -status=applied -verbose -view | egrep -i "T64V|UCI6|N3YF|9UNH|CW7X|7BIA"

-- it detects conflicts    
$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=T64V -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
Conflict(s) detected - resolve conflict condition and execute patch installation again
Conflict condition details follow:
Patch T64V is mutually exclusive and cannot coexist with patch(es): YVDZ,1YWL,HJT5

./bsu.sh -remove -patchlist=YVDZ -prod_dir=$FMW_HOME/wlserver_10.3

./bsu.sh -remove -patchlist=HJT5 -prod_dir=$FMW_HOME/wlserver_10.3
   
./bsu.sh -remove -patchlist=1YWL -prod_dir=$FMW_HOME/wlserver_10.3

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=T64V -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected
Installing Patch ID: T64V..
Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=UCI6 -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected
Installing Patch ID: UCI6.
Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=N3YF -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected
Installing Patch ID: N3YF..
Result: Success

-- it detects a conflict
$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=9UNH -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
Conflict(s) detected - resolve conflict condition and execute patch installation again
Conflict condition details follow:
Patch 9UNH is mutually exclusive and cannot coexist with patch(es): DN1F

$ ./bsu.sh -remove -patchlist=DN1F -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts..
No conflict(s) detected
Removing Patch ID: DN1F..
Result: Success

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=9UNH -prod_dir=$FMW_HOME/wlserver_10.3

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=CW7X -prod_dir=$FMW_HOME/wlserver_10.3 

$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=7BIA -prod_dir=$FMW_HOME/wlserver_10.3 

$ ./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -status=applied -verbose -view | egrep -i "T64V|UCI6|N3YF|9UNH|CW7X|7BIA"
Patch ID:               T64V
PatchContainer:    T64V.jar
Patch ID:               UCI6
PatchContainer:    UCI6.jar
                   33791826 , Patch ID :T64V) in the environment
Patch ID:               N3YF
PatchContainer:    N3YF.jar
                   U (Patch Number:33791826 , Patch ID :T64V ) in the environme
Patch ID:               9UNH
PatchContainer:    9UNH.jar
Patch ID:               CW7X
PatchContainer:    CW7X.jar
Patch ID:               7BIA
PatchContainer:    7BIA.jar

5. apply 4 patches for Oracle Fusion Middleware (FMW) 11.1.1.9 OHS - Web Tier

$ echo $IAS_ORACLE_HOME  
$ export ORACLE_HOME=$IAS_ORACLE_HOME
$ echo $ORACLE_HOME
$RUN_BASE/FMW_Home/webtier
$ export PATH=$IAS_ORACLE_HOME/OPatch:$PATH
$ which opatch
$RUN_BASE/FMW_Home/webtier/OPatch/opatch

cd /s46/path/to/CPU_April22/OH          (7 zip files)

Note: when all .zip files are saved in a shared location, only unzip them once for all nodes/instances
$ unzip p33963904_111190_Linux-x86-64.zip  
$ cd 33963904

$ opatch apply -jre $ORACLE_HOME/jdk/jre
Oracle Interim Patch Installer version 11.1.0.12.9
Copyright (c) 2022, Oracle Corporation.  All rights reserved.
... ...
Patching component oracle.ohs2, 11.1.1.9.0...
RollbackSession removing interim patch '31047338' from inventory
OPatch back to application of the patch '33963904' after auto-rollback.

Patching component oracle.ohs2, 11.1.1.9.0...
Verifying the update...
Patch 33963904 successfully applied

$ cd ..
$ unzip p32287205_111190_Linux-x86-64.zip
$ cd 32287205
$ opatch apply
... ...
Patching component oracle.ldap.rsf, 11.1.1.9.0...
Patching component oracle.rdbms.rsf, 11.1.0.7.0...
RollbackSession removing interim patch '31304503' from inventory
OPatch back to application of the patch '32287205' after auto-rollback.
Patching component oracle.ldap.rsf, 11.1.1.9.0...
Patching component oracle.rdbms.rsf, 11.1.0.7.0...
Verifying the update...
Patch 32287205 successfully applied

$ cd ..
$ unzip p32928416_111190_Linux-x86-64.zip
$ cd 32928416
$ opatch apply
... ...
Patching component oracle.opmn, 11.1.1.9.0...
Verifying the update...
Patch 32928416 successfully applied

$ cd ..
$ unzip p33144848_111190_Linux-x86-64.zip
$ cd 33144848
$ opatch apply -jre $ORACLE_HOME/jdk/jre
... ...
Patching component oracle.wlsplugins, 11.1.1.9.0...
Verifying the update...
Patch 33144848 successfully applied

$ opatch lsinventory
Interim patches (17) :

6. apply 2 patches for Oracle Fusion Middleware (FMW) - Oracle Common

$ export ORACLE_HOME=$FMW_HOME/oracle_common
$ export PATH=$ORACLE_HOME/OPatch:$PATH

$ echo $ORACLE_HOME
$FMW_HOME/oracle_common
$ which opatch
$FMW_HOME/oracle_common/OPatch/opatch

$ cd ..
$ unzip p31985571_111190_Generic.zip
$ cd 31985571/oui
$ opatch apply
... ...
Patching component oracle.jrf.adfrt, 11.1.1.9.0...
RollbackSession removing interim patch '30368663' from inventory
OPatch back to application of the patch '31985571' after auto-rollback.
Patching component oracle.jrf.adfrt, 11.1.1.9.0...
Verifying the update...
Patch 31985571 successfully applied

$ cd ../..
$ unzip p26933408_111190_Generic.zip
$ cd 26933408
$ opatch apply
... ...
Patching component oracle.jrf.thirdparty.jee, 11.1.1.9.0...
Verifying the update...
Patch 26933408 successfully applied

$ opatch lsinventory
Interim patches (14) : ...

7. apply 1 patch for Oracle Fusion Middleware 10.1.2.3 - Forms and Reports

Start a new OS session
$ echo $FILE_EDITION
run
$ echo $ORACLE_HOME
$RUN_BASE/EBSapps/10.1.2
$ which opatch
$RUN_BASE/EBSapps/10.1.2/OPatch/opatch

$ cd /path/to/CPU_April22/OH
$ unzip p32922089_101232_LINUX.zip
$ cd 32922089
$ opatch apply
... ...
Removing patch 26825525...
Applying patch 32922089...
OPatch succeeded.

$ opatch lsinventory
44 patches.

8. Run ETCC script again to confirm all required technology patches were applied.

9. Testing and and after signed-off, run FS_CLONE.
$ cd $ADMIN_SCRIPTS_HOME
$ ./adautocfg.sh
$ ./adstrtal.sh
Do a backup on file systems and then run fs_clone. 
$ adop phase=fs_clone

Additional notes:
An Oracle document shows Log4j vulnerabilities are mitigated by April 2022 CPU patches. Seems Oracle collected all remediations into this patch set. Here is the document (as of July 30 2022):

CVE-2021-4104/CVE-2022-23302/CVE-2022-23305/CVE-2022-23307 Advisory for Oracle E-Business Suite (Apache Log4j 1.x Vulnerabilities) (Doc ID 2858304.1)

SYMPTOMS
These vulnerabilities have various conditions of exploitability. For example, to be maliciously exploited vulnerability CVE-2022-23307 requires that the attacker be able to create a malicious log entry and either the attacker or a customer run the Chainsaw component on the targeted system. Because of these various conditions of exploitability and the way the Log4j components are used in Oracle products, the reported CVSS Base Scores for these vulnerabilities do not necessarily reflect the criticality of the issues in the context of their use in an Oracle product. These non-default configurations of Log4j are not used by Oracle E-Business Suite.

CVE-2021-4104 (CVSS 7.5) deserialization of untrusted data in JMSAppender
Vulnerable if: log4j configuration uses JMSAppender (not used by default)
CVE-2022-23302 (CVSS 8.8) deserialization of untrusted data in JMSSink
Vulnerable if: log4j configuration uses JMSSink (not used by default)
CVE-2022-23305 (CVSS 9.8) SQL injection in JDBCAppender
Vulnerable if: log4j configuration uses JDBCAppender (not used by default)
CVE-2022-23307 (CVSS 9.8) Chainsaw GUI logviewer (for log4j's XML output)
Vulnerable if: using chainsaw

PATCHES
The following patches are available to mitigate the Log4j related vulnerabilities. If you find Log4j in other components not listed here, file a service request (SR) against the respective components.

Core Oracle E-Business Suite Functionality
Apply the following patches:

For Oracle E-Business Suite Release 12.2.x with WLS 10.3.6 and FMW 11.1.1.9:
o Apply EBS April 2022 CPU Patch 33782739.
o Apply WLS 10.3.6 overlay Patch 33796519, which is included in Patch 33570839, on top of the 10.3.6 January 2022 PSU. This patch is included in Patch 33946345 EBS REHOSTED: 33791826 WLS PATCH SET UPDATE 10.3.6.0.220419.
o Apply WLS 10.3.6 Smart Update V5 Patch 33845432,  which is also included in Patch 33570839 EBS RELEASE 12.2 CONSOLIDATED FMW FIXES FOR JAN 2022 and subsequent Critical Patch Updates (CPUs).
o Apply FMW 11.1.1.9 Patch 33960746 (to oracle_common).

The following occurrences of log4j jar files are modified by the aforementioned patches:
$FMW_HOME/modules/com.bea.core.apache.log4j_1.2.13.jar
$FMW_HOME/modules/com.bea.core.apache.log4j_1.1.0.0_1-2-15.jar
$FMW_HOME/wlserver_10.3/server/lib/consoleapp/APP-INF/lib/log4j-1.2.17-16.jar
$FMW_HOME/wlserver_10.3/server/lib/consoleapp/APP-INF/lib/log4j-1.2.8.jar
$FMW_HOME/wlserver_10.3/server/lib/wlstestclient.ear/webapp.war/log4j-1.2.8.jar
$FMW_HOME/patch_wls1036/backup/backup.jar/wlstestclient.ear/webapp.war/log4j-1.2.8.jar
$FMW_HOME/oracle_common/sysman/archives/fmwctrl/app/em.ear
$FMW_HOME/oracle_common/sysman/archives/applications/11_1_1_0_0_emcore.war
$FMW_HOME/oracle_common/sysman/jlib/log4j-core.jar

Any occurrences of log4j jar files in these directories can be manually removed:
All Log4j libraries under the following directories:
o /clone (such as …/EBSapps/comn/clone/)
o /adopclone (such as …/EBSapps/comn/adopclone_xxxxx/)
o /inventory (such as …/EBSapps/10.1.2/ccr/inventory/)
o /patch stage (such as .../bsu/cache_dir/ and .../.patch_storage/)
o $FMW_HOME/user_projects/domains/EBS_domain/servers/AdminServer/tmp/_WL_user/emcore/28c293/WEB-INF/lib/log4j-core.jar
Shut down the WLS server before removing this file. After a while, the library might come back again which is the patched library.
o Run the following commands to stub out the log4j.jar under the 10.1.2 Oracle home:
     $ cd <ORACLE_HOME>
     $ perl -e print "" > ./sysman/admin/emdrep/lib/log4j.jar
o Run the following commands to stub out the EAR file under the 10.1.2 Oracle home for Oracle E-Business Suite Release 12.2 and 10.1.3 Oracle home for Oracle E-Business Suite Release 12.1:
     $ cd <ORACLE_HOME>
     $ perl -e print "" > ./oc4j/j2ee/oc4j_applications/applications/isqlplus.ear

Notes in Dec 2023: even it does not contain any Java class after patching, file $FMW_HOME/wlserver_10.3/server/lib/consoleapp/APP-INF/lib/log4j-1.2.8.jar can NOT be deleted. Oracle Support says it is a place holder. Run unzip to see .jar file's contents:
$ cd $FMW_HOME/wlserver_10.3/server/lib/consoleapp/APP-INF/lib
$ ls -al log4j-1.2.8.jar       <= only 523 bits in size
$ unzip log4j-1.2.8.jar      <= it only holds two files

Our analysis of the following occurrences of log4j v1.x indicates that these can be considered false positives. These instances are false positives since they do not contain the vulnerable classes.

The following Log4j jar files are not impacted by the CVEs listed in this note since they do not contain the vulnerable classes:
10.1.2/sysman/jlib/log4j-core.jar
10.1.2/OPatch/ocm/ocm.zip/core.jar/payload.zip/log4j-core.jar
10.1.2/diagnostics/lib/ojdl-log4j.jar
$FMW_HOME/Oracle_EBS-app1/ccr/lib/log4j-core.jar
$FMW_HOME/Oracle_EBS-app1/oui/jlib/lib/log4j-core.jar
$FMW_HOME/Oracle_OAMWebGate1/oui/jlib/jlib/log4j-core.jar
$FMW_HOME/webtier/OPatch/ocm/lib/log4j-core.jar
$FMW_HOME/webtier/OPatch/ocm/ocm.zip/log4j-core.jar
$FMW_HOME/webtier/OPatch/ocm/ocm.zip/core.jar/payload.zip/log4j-core.jar
$FMW_HOME/webtier/ccr/lib/log4j-core.jar
$FMW_HOME/webtier/oui/jlib/jlib/log4j-core.jar
$FMW_HOME/oracle_common/OPatch/ocm/lib/log4j-core.jar
$FMW_HOME/oracle_common/OPatch/ocm/ocm.zip/log4j-core.jar
$FMW_HOME/oracle_common/OPatch/ocm/ocm.zip/core.jar/payload.zip/log4j-core.jar
$FMW_HOME/oracle_common/modules/oracle.odl_11.1.1/ojdl-log4j.jar
DBhome/sysman/jlib/ocm/log4j-core.jar
DBhome/oui/jlib/jlib/log4j-core.jar
DBhome/oc4j/j2ee/home/applications/ascontrol.ear/ascontrol.war/log4j-core.jar
DBHome/ccr/lib/log4j-core.jar
DBHome/javavm/lib/dbhadoop12101.jar

Vulnerability NOTES: 

In another Oracle document ID 2827804.1 (CVE-2021-44228/CVE-2021-45046/CVE-2021-44832/CVE-2021-45105 Advisory for Oracle E-Business Suite (Apache log4j Vulnerabilities), the Core Oracle E-Business Suite Functionality section asks to apply EBS patch 33672402:R12.TXK.C (in early 2022) to mitigate the vulnerability in Oracle E-Business Suite 12.2 associated with 
CVE-2021-44228, 
CVE-2021-45046, 
CVE-2021-44832, 
CVE-2021-45105

It patches file $COMMON_TOP/java/lib/log4j_core.jar.

UPDATE: Oracle provides a Map of CVE to CPU Advisory or Security Alert to show a huge long list of CVEs addressed by CPU patches. Click on the CPU name to get the list addressed by an individual CPU.

Wednesday, July 27, 2022

Add OACORE servers in R12.2

Oracle Doc ID 1905593.1 (Managing Configuration of Oracle HTTP Server and Web Application Services in Oracle E-Business Suite Release 12.2) gives steps on adding oacore process to the instance. See "4.4.1 Adding a new managed server".

Addition of managed servers needs to be done on the RUN file system when there is no active ADOP cycle. During the next adop prepare, the Configuration Change Detector identifies that the addition has been made and the managed servers are automatically synced up from the run file system to the patch file system. The synchronization also gets done when fs_clone is executed.

Most likely, adding CPU and memory to the server are needed to support additional oacore.

1. Backup $CONTEXT_FILE
2. Backup two conf files (in below path or similar. Many conf files resides in this folder)
$ cd  $IAS_ORACLE_HOME/instances/EBS_web_OHS1/config/OHS/EBS_web
mod_wl_ohs.conf
apps.conf

3. Check ADOP session
$ hostname
master_node.domain.com
$ echo $TWO_TASK
EBSDEV

$ adop -status
Enter the APPS password:
Connected.
==============================================================
ADOP (C.Delta.12)
Session Id: 5
Command: status
Output: $NE_BASE/EBSapps/log/adop/5/20220X08_115941/adzdshowstatus.out
===============================================================
Node Name       Node Type  Phase           Status          Started              Finished             Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
2nd_node            slave    FS_CLONE        COMPLETED       2022/02/17 23:05:12  2022/02/18 00:01:37  0:56:25
3rd_node            slave     FS_CLONE        COMPLETED       2022/02/17 23:05:12  2022/02/17 23:59:02  0:53:50
naster_node      master    FS_CLONE        COMPLETED       2022/02/17 23:05:12  2022/02/17 23:39:21  0:34:09

4. Identify port numbers (and how many OACORE server(s) on each node)
$ grep -i oacore_server_ports $RUN_BASE/inst/apps/$CONTEXT_NAME/appl/admin/$CONTEXT_NAME.xml
<oacore_server_ports oa_var="s_oacore_server_ports">oacore_server1:7206</oacore_server_ports>

$ grep -i oacore_server_ports $PATCH_BASE/inst/apps/$CONTEXT_NAME/appl/admin/EBSDEV*.xml
<oacore_server_ports oa_var="s_oacore_server_ports">oacore_server1:7207</oacore_server_ports>

Port 7206 is used for RUN file system and 7207 is used for PATCH file system. Port 7208 is free and is chosen for the new oacore. Also oacore_server1 is running on master node and oacore_server2 is running on 2nd node. Name oacore_server3 as the new server.

5.  Add a new managed server. This will create a managed server and add a new entry to the context file for starting and stopping the new managed server via the adstrtal and adstpall scripts:

$ echo $FILE_EDITION
run
$ echo $APPL_TOP
/u02/app/EBSPROD/fs1/EBSapps/appl    

$ echo $CONTEXT_FILE
$ perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
        ebs-create-managedserver -contextfile=$CONTEXT_FILE \
        -managedsrvname=oacore_server3 -servicetype=oacore \
        -managedsrvport=7208 -logfile=$APPLRGF/TXK/addMS_oacoreserver3_2022.log
Enter the APPS Schema password:
Enter the WebLogic AdminServer password:

The logfile for this session is located at $APPLRGF/TXK/addMS_oacoreserver3_2022.log
ManagedServer oacore_server3 created successfully.

Note: use the real path and file name for $CONTEXT_FILE in above line

$  egrep -i "Error|Failed|Warning|ORA-" $APPLRGF/TXK/addMS_oacoreserver3_2022.log
Warning: An insecure protocol was used to connect to the
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
Warning: An insecure protocol was used to connect to the

$ grep -i s_oacore_nodes $CONTEXT_FILE
       <oacore_nodes oa_var="s_oacore_nodes">master_node.domain.com:7206</oacore_nodes>

$ grep -i oacore_server_ports $CONTEXT_FILE
  <oacore_server_ports oa_var="s_oacore_server_ports">oacore_server1:7206,oacore_server3:7208</oacore_server_ports>

By the way, it does not add anything to PATCH file system.
$ grep -i oacore_server_ports $PATCH_BASE/inst/apps/$CONTEXT_NAME/appl/admin/EBSDEV*.xml
<oacore_server_ports oa_var="s_oacore_server_ports">oacore_server1:7207</oacore_server_ports>

Now, starting oacore_server3 shall work:
sh $ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh status oacore_server3
sh $ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh start oacore_server3
sh $ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh status oacore_server3
You are running admanagedsrvctl.sh version 120.14.12020000.12
Enter the WebLogic Admin password:
oacore_server3 is running.

6. Add details of the newly added managed servers into the OHS configuration files mod_wl_ohs.conf and apps.conf on the current node:

$ echo $CONTEXT_FILE
$ perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl \
        -contextfile=$CONTEXT_FILE \
        -configoption=addMS -oacore=node_name.domain.com:7208

Log file shall say mod_wl_ohs.conf and apps.conf were generated successfully.
$ egrep -i "Error|Failed|Warning|ORA-" $APPLRGF/TXK/txkSetAppsConf_*.log

$ cd $ADMIN_SCRIPTS_HOME
$ adapcctl.sh stop
$ adopmnctl.sh status
$ adapcctl.sh start start
$ adopmnctl.sh status

Recycle all services:
$ adstpall.sh apps/appsPWD
$ adstrtal.sh apps/appsPWD
Then, log onto Weblogic console to see two oacore servers are running.

7. Repeat steps on 2nd web/forms node. Skip this on node dedicated for concurrent jobs. 

Before adding more oacore
$ grep -i oacore_server_ports $CONTEXT_FILE
   <oacore_server_ports oa_var="s_oacore_server_ports">oacore_server2:7206</oacore_server_ports>

After adding one more oacore
$ grep -i s_oacore_nodes $CONTEXT_FILE
      <oacore_nodes oa_var="s_oacore_nodes">master_node.domain.com:7206</oacore_nodes>
Note: it points to master_node on the 2nd node config. This could be the reason for unbalanced number of connections.

$ grep -i oacore_server_ports $CONTEXT_FILE
  <oacore_server_ports oa_var="s_oacore_server_ports">oacore_server2:7206,oacore_server4:7208</oacore_server_ports>

Again, verify it does not do anything on PATCH file system.
$ grep -i oacore_server_ports $PATCH_BASE/inst/apps/$CONTEXT_NAME/appl/admin/EBSDEV*.xml
<oacore_server_ports oa_var="s_oacore_server_ports">oacore_server2:7207</oacore_server_ports>

8.  Summary on oacore name and port:
Node1   RUN  oacore_server1:7206,   oacore_server3:7208
Node2   RUN  oacore_server2:7206,   oacore_server4:7208
After an adop CUTOVER or FS_CLONE, the change will be copied to PATCH file system. Names will be the same but use different ports:
Node1   PATCH  oacore_server1:7207,   oacore_server3:7209
Node2   PATCH  oacore_server2:7207,   oacore_server4:7209

The original idea for adding more oacore servers to our R12.2.10 instance was to address the unbalance of connection loadings between oacore_server1 and oacore_server2. But adding more oacore servers still does not resolve the issue. oacore servers on master node still get much more connections. 
The benefit is that if one OACORE is down or frozen, we can recycle it while the 2nd one is still running. That could reduce downtime on end-users.

UPDATES in 2023:
After we opened a SR with Oracle Support on OACORE connection distribution uneven, Oracle engineer provided an action plan for us to add entries to apps.conf and mod_wl_ohs.conf files which will allow for load balancing of traffic to all OACOREs.

1. Take a backup of your EBS environment
2. Connect to master_node and add oacore_server2 and oacore_server4 to the configuration:

a. Open a terminal window and connect to master node
b. Switch to the apps user
c. Source the apps environment
d. Run the commands below:

$ echo $FILE_EDITION
run
$ echo $APPL_TOP
/u02/app/EBSPROD/fs2/EBSapps/appl     <== a CUSOVER was done after oacore was added 

$ perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl \
-contextfile=<CONTEXT_FILE> \
-configoption=addMS \
-oacore=2nd_node.domain.com:7207

$ perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl \
-contextfile=<CONTEXT_FILE> \
-configoption=addMS \
-oacore=2nd_node.domain.com:7209

e. Restart OHS by running adapcctl.sh

3. Connect to 2nd_node and add oacore_server3 to the configuration:

a. Open a terminal window and connect to 2nd_node
b. Switch to the apps user
c. Source the apps environment
d. Run the below command:

$ perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl \
-contextfile=<CONTEXT_FILE> \
-configoption=addMS \
-oacore=master_node.domain.com:7209

e. Restart OHS by running adapcctl.sh

4. Direct your load balancer team to stop sending web requests to your CONCURRENT MANAGER ONLY node.
5. Retest for the issue

The txkSetAppsConf.pl command should be run after adding apps tier nodes to your EBS environment. This is mentioned in the excerpt below from Doc ID 1383621.1 :
"If any of the managed servers from the newly added node are desired to be part of the cluster configuration on the current node, run the following command to add details of these managed servers into the OHS configuration files mod_wl_ohs.conf and apps.conf on the current node:..."

Before performing action plan from the SR (on master_node), entries in two .conf files: 
$ cd $IAS_ORACLE_HOME/instances/EBS_web_OHS1/config/OHS/EBS_web
$ grep 7209 apps.conf
BalancerMember http://master_node.domain.com:7209/OA_HTML/media
BalancerMember http://master_node.domain.com:7209/OA_HTML/classes
BalancerMember http://master_node.domain.com:7209

$ grep 7209 mod_wl_ohs.conf
WebLogicCluster master_node.domain.com:7207,master_node.domain.com:7209

$ grep 7207 apps.conf
BalancerMember http://master_node.domain.com:7207/OA_HTML/media
BalancerMember http://master_node.domain.com:7207/OA_HTML/classes
BalancerMember http://master_node.domain.com:7207

$ grep 7207 mod_wl_ohs.conf
WebLogicCluster master_node.domain.com:7207,master_node.domain.com:7209

After performing action plan from the SR (on master_node), all entries are balanced with two nodes.

$ grep 7209 apps.conf
   BalancerMember http://master_node.domain.com:7209/OA_HTML/media
   BalancerMember http://2nd_node.domain.com:7209/OA_HTML/media
   BalancerMember http://master_node.domain.com:7209/OA_HTML/classes
   BalancerMember http://2nd_node.domain.com:7209/OA_HTML/classes
   BalancerMember http://master_node.domain.com:7209
   BalancerMember http://2nd_node.domain.com:7209

$ grep 7209 mod_wl_ohs.conf
WebLogicCluster master_node.domain.com:7207,master_node.domain.com:7209,2nd_node.domain.com:7207,2nd_node.domain.com:7209

$ grep 7207 apps.conf
   BalancerMember http://master_node.domain.com:7207/OA_HTML/media
   BalancerMember http://2nd_node.domain.com:7207/OA_HTML/media
   BalancerMember http://master_node.domain.com:7207/OA_HTML/classes
   BalancerMember http://2nd_node.domain.com:7207/OA_HTML/classes
   BalancerMember http://master_node.domain.com:7207
   BalancerMember http://2nd_node.domain.com:7207

$ grep 7207 mod_wl_ohs.conf
WebLogicCluster master_node.domain.com:7207,master_node.domain.com:7209,2nd_node.domain.com:7207,2nd_node.domain.com:7209

Very similar entries in .conf files in 2nd_node, after action plan was performed on 2nd_node. 

Wednesday, June 29, 2022

FS_CLONE failed with SSL cert file and ADOP troubleshooting

When run " adop phase=FS_CLONE " to sync RUN file system on /fs1 to PATCH file system on /fs2 on a concurrent node in which Web/Forms services are not enabled, it failed with generic error (getting from "adopscanlog -latest=yes"):

FUNCTION: main::runFSCloneApply [ Level 1 ]
ERRORMSG: $COMMON_TOP/adopclone_nodeNamw/bin/adclone.pl did not go through successfully.

Log file $NE_BASE/EBSapps/log/adop/6/xxx /fs_clone/xxm4p/TXK_SYNC_create/fsclone_apply/ohsT2PApply/CLONE2022-10-14_16-38-02_839829939.log points out an error on /fs2. Seems it had problem in starting Apache service on /fs2.

Error Message  :1  
PLUGIN][OHS] - ERROR - Oct 14, 2022 16:38:27 - CLONE-26009   OHS T2P failed.  
[PLUGIN][OHS] - CAUSE - Oct 14, 2022 16:38:27 - CLONE-26009   Unable to start OS component. [PLUGIN][OHS] - ACTION - Oct 14, 2022 16:38:27 - CLONE-26009   Check clone log and error file and ohs log file $PATCH_BASE/FMW_Home/webtier/instances/EBS_web_OHS3/diagnostics/logs/OHS/EBS_web/console~OHS~1.log for root cause.

 After I copied good file cwallet.sso to right directories which is required for enabling ssl, FS_CLONE worked. Most likely, the cwallet.sso expired and did not get updated. 

So, fs_clone process tests starting Apache service on /fs2 even Web services are not enabled on RUN file system /fs1 of the node.

When looking further, found a message in a log file $PATCH_BASE/inst/apps/EBSDEV_xxm4p/logs/appl/rgf/TXK/txkSetAppsConf_10141635.log on /fs2
ERROR: The value <$PATCH_BASE/FMW_Home/webtier/instances/EBS_web_OHS3> for s_ohs_instance_loc in <CONTEXT_FILE on /fs2>

This message is a false statement and a misleading one.

Troubleshoot ADOP error

In another FS_CLONE failure on the primary node with same error 
FUNCTION: main::runFSCloneApply

To troubleshoot it, I ran "adopscanlog" to get the list of errors. Note "adopscanlog -latest=yes" may not list important errors at the very beginning. Seems they list errors by timestamp order.

Log file  $NE_BASE/EBSapps/log/adop/6/20221201_155422/fs_clone/nodeName/TXK_SYNC_create/fsclone_stage/FSCloneStageAppsTier_12011556.log has errors

ERROR: Number of servers are not in sync between Run and Patch Context.
ERROR: Managed Servers would be in-sync at apply phase

The Error matches the one in Doc ID 1634239.1 ("Number of servers are not in sync between Run and Patch Context" While fs_clone of patching (WLS 17495356) Cycle). I ran SQL statements from the document and found the difference. So, the fix is to make them match.

SQL> SELECT
extractValue(XMLType(TEXT),'//oa_service_name[@oa_var="s_adminservername"]'),
extractValue(XMLType(TEXT),'//oacore_server_ports'),
extractValue(XMLType(TEXT),'//forms_server_ports'),
extractValue(XMLType(TEXT),'//oafm_server_ports'),
extractValue(XMLType(TEXT),'//forms-c4ws_server_ports'),
extractValue(XMLType(TEXT),'//oaea_server_ports')
from fnd_oam_context_files
where name not in ('TEMPLATE','METADATA')
and (status is null or status !='H')
and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type')='patch'
and CTX_TYPE = 'A';
--------------------------------------------------------------------------------
AdminServer
null  (blank)
null
null
null
null

SQL> SELECT
extractValue(XMLType(TEXT),'//oa_service_name[@oa_var="s_adminservername"]'),
extractValue(XMLType(TEXT),'//oacore_server_ports'),
extractValue(XMLType(TEXT),'//forms_server_ports'),
extractValue(XMLType(TEXT),'//oafm_server_ports'),
extractValue(XMLType(TEXT),'//forms-c4ws_server_ports'),
extractValue(XMLType(TEXT),'//oaea_server_ports')
from fnd_oam_context_files
where name not in ('TEMPLATE','METADATA')
and (status is null or status !='H')
and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type')='run'
and CTX_TYPE = 'A';
--------------------------------------------------------------------------------
AdminServer
oacore_server1:7205
forms_server1:7405
oafm_server1:7605
forms-c4ws_server1:7805
null

AdminServer
oacore_server1:7205
forms_server1:7405
oafm_server1:7605
forms-c4ws_server1:7805
null

Also, log $NE_BASE/EBSapps/log/adop/6/20221201_155422/fs_clone/nodeName/TXK_SYNC_create/fsclone_apply/FSCloneApplyAppsTier_12011608.log has error at its end:

configProperty id = Server2
Count for NodeIterator nextNode = 3
ERROR: Managed Server's are not in sync between file system context and DB context
ERROR: Update Moveplan Fail

START: Inside exitClone....
Updating status INCOMPLETE for ApplywlsTechStack
START: Updating status INCOMPLETE for action ApplywlsTechStack
END: Updated status INCOMPLETE for action ApplywlsTechStack

The error matches the description in Doc ID 2296036.1 (EBS R12.2: FS_CLONE Failing With ERROR: Update Moveplan Fail). But the Machine list under " 'Domain Structure' => click on the Environments => 'Machines' " link in WebLogic Admin Console is correct. So this document did not help. Note: only web/forms nodes are listed under Machine.

Log file $PATCH_BASE/inst/apps/EBSDEV_nodeName/logs/appl/rgf/TXK/txkSetAppsConf_08241646.log on /fs2 gives a different error without detail. I believe it is irrelevant. 

Error in getting Context Value for s_ohs_instance_loc
Config Option : standaloneohsconfig

Tuesday, June 28, 2022

Abort an ADOP patching cycle

When trying to run through the adop cycle before database 19c upgrade, it failed on PREPARE:
$ adop phase=prepare
... ...
Initializing.
    Run Edition context  : $CONTEXT_FILE
    Patch edition context: ... ...
    Patch file system free space: 50.28 GB

Validating system setup.
    Node registry is valid.
    Log: $APPLRGF/TXK/verifyssh.log
    Output: $APPLRGF/TXK/out.xml
    Remote execution is operational.
... ...
Running prepare phase on node(s): ... ...
    [UNEXPECTED]adop phase=prepare on all the nodes returned failure.
    [UNEXPECTED]Unable to continue processing.

The log file does not tell any specific reason. The SQL line prior to the error tells it may fail on checking the status of a CUSTOM TOP.

$ vi txkADOPPreparePhaseSynchronize.log
... ...
==========================
Inside getPatchStatus()...
==========================
SQL Command: SELECT status||',' FROM ad_adop_session_patches 
WHERE  --... ...
bug_number = 'ADSPLICE_docXX'

patch_status = Y
Updated patch_status = Y
EXIT STATUS: 1

=============================
Inside evalADPATCHStatus()...
=============================
message_status: ERROR
Adsplice action did not go through successfully.
*******FATAL ERROR*******
PROGRAM : ($AD_TOP/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Wed Jul 20 14:00:36 2022
FUNCTION: main::execADSPLICE [ Level 1 ]
ERRORMSG: Adsplice action did not go through successfully.

I ran below statement to get an idea what could be the problem, then tried to de-register the bad CUSTOM TOP docXX (as I did not know what was bad):
SQL> select * FROM ad_adop_session_patches where bug_number like 'ADSPLICE%'
order by adop_session_id desc, end_date desc;

$ perl $AD_TOP/bin/adDeregisterCustomProd.pl

Enter the APPS username: apps
Enter the APPS password:
Enter the Custom Application to De-Register: docXX

Enter the Application Id to De-Register: 20268
Script adjava oracle.apps.ad.splice.adCustProdSplicer mode=uninstall options=inputpair logfile=customderegister.log
.. ...
Performing Validations for De-Register

AD Custom Product Splicer error:
Patching cycle in progress - run this utility from patch file system
You can only run it from run file system when not patching

AD Custom Product Splicer is completed with errors.
Please see the below log file for more details.

I believe I also tried to run "adop phase=fs_clone" at this stage, and it did not do anything and failed very quickly. I did not want and had no confidence to de-register a CUSTOM TOP from patch file system. So, to run through the ADOP patch cycle, the only choice is to abort the patch session first in below three steps:

$ adop phase=abort
.. ...
Validating system setup.
    Node registry is valid.
    Log: $APPLRGF/TXK/verifyssh.log
    Output: $APPLRGF/TXK/out.xml
    Remote execution is operational.

Checking for existing adop sessions.
    Continuing with existing session [Session ID: 7].
        Session Id            :   7
        Prepare phase status  :  FAILED
        Apply phase status    :    NOT COMPLETED
        Cutover  phase status :  NOT COMPLETED
        Abort phase status    :    NOT COMPLETED
        Session status        :       FAILED
The above session will be aborted. Do you want to continue [Y/N]? Y
===================================================================
ADOP (C.Delta.12)
Session ID: 7
Node: node1Name
Phase: abort
Log: $NE_BASE/EBSapps/log/adop/7/20220X21_170727/adop.log
=====================================================================

Verifying existence of context files in database.

Checking if adop can continue with available nodes in the configuration.
    Log: $NE_BASE/EBSapps/log/adop/7/20220721_170727/abort/node1Name
        txkADOPEvalSrvStatus.pl returned SUCCESS

Creating list of nodes list where abort needs to be run.
    The abort phase needs to be run on node: node1Name
    The abort phase needs to be run on node: node2Name
    The abort phase needs to be run on node: node3Name

Running abort phase on node(s): [node1Name,node2Name and node3Name].
    Output: $NE_BASE/EBSapps/log/adop/7/20220721_170727/abort/remote_execution_result_level1.xml
    Log: $NE_BASE/EBSapps/log/adop/7/20220721_170727/abort/node1Name
        txkADOPEvalSrvStatus.pl returned SUCCESS

Generating node-specific status report.
    Output: $NE_BASE/EBSapps/log/adop/7/20220721_170727/adzdnodestat.out

Summary report for current adop session:
    Node node1Name: Completed successfully
       - Abort status:      Completed successfully
    Node node2Name: Completed successfully
       - Abort status:      Completed successfully
    Node node3Name: Completed successfully
       - Abort status:      Completed successfully
    For more details, run the command: adop -status -detail
adop exiting with status = 0 (Success)

$ adop phase=cleanup cleanup_mode=full
... ...
The cleanup phase completed successfully.
adop exiting with status = 0 (Success)

$ adop -status

Enter the APPS password:
Connected.
==============================================================
ADOP (C.Delta.12)
Session Id: 7
Command: status
Output: $NE_BASE/EBSapps/log/adop/7/20220X21_172115/adzdshowstatus.out
===============================================================
Node Name       Node Type  Phase           Status                            Started              Finished             Elapsed
--------------- ---------- --------------- ----------------------------------- ------------------- -------------------- ------------
node1Name       master  PREPARE     SESSION ABORTED 2022/0X/20 13:38:49  2022/0X/21 12:54:19  23:15:30
                                       APPLY               SESSION ABORTED
                                       FINALIZE         SESSION ABORTED
                                       CUTOVER        SESSION ABORTED
                                       CLEANUP        COMPLETED       2022/0X/21 17:13:58  2022/0X/21 17:14:48  0:00:50
node2Name       slave    PREPARE      SESSION ABORTED 2022/0X/20 13:38:49  2022/0X/21 12:55:01  23:16:12
                                       APPLY              SESSION ABORTED
                                       FINALIZE        SESSION ABORTED
                                       CUTOVER       SESSION ABORTED
                                       CLEANUP       COMPLETED       2022/0X/21 17:13:58  2022/0X/21 17:14:48  0:00:50
node3Name       slave    PREPARE      SESSION ABORTED 2022/0X/20 13:38:49  2022/0X/21 12:55:00  23:16:11
                                       APPLY              SESSION ABORTED
                                       FINALIZE        SESSION ABORTED
                                       CUTOVER       SESSION ABORTED
                                       CLEANUP       COMPLETED       2022/0X/21 17:13:58  2022/0X/21 17:14:48  0:00:50
File System Synchronization Type: Full

INFORMATION: Patching cycle aborted, so fs_clone will run automatically on node2Name,node3Name nodes in prepare phase of next patching cycle.
adop exiting with status = 0 (Success)

$ adop phase=fs_clone
... ...
The fs_clone phase completed successfully.
adop exiting with status = 0 (Success)

When I ran "adop -status" again, it did not mention "fs_clone" but gave the same status as in previous one.

Then, I stop Apps services and de-register the custom top on all nodes. Now, it worked:
$ ./adstpall.sh apps/appsPWD
$ perl $AD_TOP/bin/adDeregisterCustomProd.pl   (all three nodes)

$ adstrtal.sh apps/appsPWD

After that, the patch cycle ran through successfully on the current run file system:
$ adop phase=prepare
$ adop phase=actualize_all
$ adop phase=finalize finalize_mode=full
$ adop phase=cutover
On the new run file system:
$ adop phase=cleanup cleanup_mode=full 

To re-create the CUSTOM TOP (on all three nodes. First, make sure three config files in $APPL_TOP/admin are good on all nodes):

$ ./adstpall.sh apps/appsPWD
$ adsplice 


In another case, CUTOVER phase failed with below error. I had no idea on how to fix it and ran ABORT phase to get out from it.
$NE_BASE/EBSapps/log/adop/5/20210228_114950/cutover/validate/node1Name/ADOPValidations_detailed.log: 
-------------------------------------------------------------------------------------------------------------------
Lines #(51-55):
Checking the value of s_active_webport...

ERROR: The value of s_active_webport are different on RUN & PATCH Context files.
The Value present in RUN Context file = 4484
The Value present in PATCH Context file = 8041

I think if I ran "adop phase=fs_clone" BEFORE running PREPARE phase, it might screen out the issue or avoid the issue and so I did not have to run ABORT (which is time consuming).