Tuesday, March 9, 2021

Enable R12.2 online patching (ADOP)

After R12.2.0 is configured, the next step is to enable online patch (adop), one of the major changes in R12.2. Followings are steps to do that.

1). Apply online patching Readiness and GSCC Report patch (Doc ID: 1531121.1)

a) Get the R12.2 env
$ cd /u02/app/${TWO_TASK}/fs1/EBSapps/appl
$ . APPS${CONTEXT_NAME}.env
$ echo $FILE_EDITION
run
Confirm $AD_TOP/patch/115/sql/adzddtsfix.sql and folder $FND_TOP/perl/GSCC do not exist
b) $ unzip p31026891_R12.AD.C_R12_GENERIC.zip    (download file "C" for R12.2.0)
(If apply this patch to a R12.1 instance before R12.2 upgrade, download file p31026891_R12.AD.B_R12_GENERIC.zip)
c) Confirm the same version of adgrants.sql as in this patch was executed before.

d) Enable maintenance mode
SQL> @$RUN_BASE/EBSapps/appl/ad/12.0.0/patch/115/sql/adsetmmd.sql ENABLE
SQL> select fnd_profile.value('APPS_MAINTENANCE_MODE') from dual;
--------------------------------------------------------------------------------
MAINT

e) apply Readiness patch
$ echo $FILE_EDITION
run
$ echo $ORACLE_HOME
/u02/app/$TWO_TASK/fs1/EBSapps/10.1.2
$ cd 31026891
$ ls
$ adpatch
No error in log file in folder $APPL_TOP/admin/${TWO_TASK}/log

SQL> select bug_number, to_char(creation_date,'DD-MON-RRRR HH24:MI:SS') date_applied 
from apps.ad_bugs where bug_number = '31026891';
BUG_NUMBER             DATE_APPLIED
------------------------------ --------------------
31026891                       DD-MON-2021 15:34:18

2). Run the reports (See Upgrade Guide p.122 and README in Section 5)
$ cd $LOG_HOME/appl
$ ls
admin  oam  rgf
$ mkdir op
$ cd op
$ sqlplus system @$AD_TOP/sql/ADZDPSUM.sql      <= It only queries the data dictionary
$ mv adzdpsum.txt adzdpsum_pre_dbprep.txt
$ sqlplus system @$AD_TOP/sql/ADZDPMAN.sql     <= Lists all violations that need a fix 
$ mv adzdpman.txt adzdpman_pre_dbprep.txt
$ sqlplus system @$AD_TOP/sql/ADZDPAUT.sql       <= Generates an information-only report
$ mv adzdpaut.txt adzdpaut_pre_dbprep.txt

$ sqlplus apps  @$AD_TOP/patch/115/sql/adzddtsfix.sql
exec dbms_utility.invalidate(36166,NULL,0);
exec dbms_utility.invalidate(1015,NULL,0);
exec sys.utl_recomp.recomp_parallel
exit;
$ ls
adzddtsfixout.sql  adzdpaut_pre_dbprep.txt  adzdpman_pre_dbprep.txt  adzdpsum_pre_dbprep.txt

$ more adzddtsfixout.sql
exec dbms_utility.invalidate(36166,NULL,0);
exec dbms_utility.invalidate(1015,NULL,0);
exec sys.utl_recomp.recomp_parallel       / <== See Upgrade Guide p.126
exit;
$ sqlplus apps @adzddtsfixout.sql
PL/SQL procedure successfully completed.
PL/SQL procedure successfully completed.
PL/SQL procedure successfully completed.

$ sqlplus apps @$AD_TOP/sql/ADZDDBCC.sql         <= List database violations
$ mv adzddbcc.lst adzddbcc_pre_dbprep.txt

Note: needs to specify entries in ADZDPCUST.sql before running it. (I skipped it)

$ ls $LOG_HOME/appl/op
adzdpsum_pre_dbprep.txt        <= send it to Developer
adzdpman_pre_dbprep.txt        <= send it to Developer (Specially, Section 5 & 8)
adzdpaut_pre_dbprep.txt       
adzddbcc_pre_dbprep.txt         
    <= send it to DBA who has to address all issues in it, 
           except SECTION-37 ("Unfixed violations can be ignored")

If APPS_DDL shows up in SECTION-47, follow Doc ID 2735375.1 to install it to avoid errors:
AutoPatch error: adpmrp: Error while installing apps_ddl packages.
AutoPatch - aidafoGetFileDbVersion: INFO:  ORA-01403: no data found

In my cases, Developer did not correct many findings in above reports.

3). Run ETCC scripts again to make sure no new technology patches are needed.
Oracle updates/release ETCC (Patch 17537119) from time to time. During the R12.2 upgrade, download and keep a same version to run a few times on both database and apps. That way will potentially save your time from keeping applying other supporting patches.

4). Gather SYS schema stats. Run it as SYSDBA
SQL> begin
dbms_stats.gather_schema_stats(
'SYS',
options=>'GATHER STALE',
estimate_percent =>
DBMS_STATS.AUTO_SAMPLE_SIZE,
method_opt => 'FOR ALL COLUMNS SIZE AUTO',
cascade => TRUE);
end;

5). Check database free space:
$ export HOSTNAME=node_name   (<= without domain name. Maybe an optional step)
$ perl $AD_TOP/bin/adzdreport.pl apps
Enter the APPS Password:
    Online Patching Diagnostic Reports Main Menu
    --------------------------------------------
    1.  Run edition reports
    2.  Patch edition reports
    3.  Other generic reports
    4.  Exit
    Enter your choice [4]: 3

    Other Generic Reports Sub Menu
    ------------------------------
    1.  Editions summary
    2.  Editioned objects summary
    3.  Free space in important tablespaces
    4.  … … … 
    Enter your choice [16]: 3

Database tablespace info will be saved in file $APPL_TOP/admin/$TWO_TASK/out/adzdshowts.out. 

Tablespaces MUST be extended to avoid patch job fails:
SYSTEM Tablespace: has a minimum of 25 GB of free space
APPS_TS_SEED Tablespace: has a minimum of 5 GB of free space

6) AD_ZD_PREP has to be valid before moving forward

SQL> Select owner,object_name,object_type,status from dba_objects where object_name like 'AD_ZD%' and status ='INVALID';
OWNER    OBJECT_NAME           OBJECT_TYPE       STATUS
------------- ----------------------------- ------------------------ ------------
APPS         AD_ZD_PREP                PACKAGE BODY  INVALID
 
SQL> alter package AD_ZD_PREP compile body;
SQL> show error;
Errors for PACKAGE BODY AD_ZD_PREP:
LINE/COL ERROR
-------- -----------------------------------------------------------------
463/3    PL/SQL: SQL Statement ignored
463/19   PL/SQL: ORA-00942: table or view does not exist
467/5    PL/SQL: SQL Statement ignored
467/21   PL/SQL: ORA-00942: table or view does not exist
487/5    PL/SQL: Statement ignored
487/5    PLS-00201: identifier 'SYS.XDB_MIGRATESCHEMA' must be declared
501/5    PL/SQL: SQL Statement ignored
501/21   PL/SQL: ORA-00942: table or view does not exist

Fix: Follow Doc ID 2066607.1 (Patch 13543062 Fails With Error: ORA-06508: PL/SQL: could not find APPS.AD_ZD_PREPAD_ZD_PREP Due To Missing Package 'xdb_migrateschema' ). Actually, this could be avoid if a database upgrade post-step (Doc ID 1524398.1) was executed.

a). conn / as sysdba
@?/rdbms/admin/dbmsxdbschmig.sql
@?/rdbms/admin/prvtxdbschmig.plb

SQL> set line 130
SQL> set pages 50
SQL> col OWNER for a25;
SQL> col OBJECT_NAME for a30
SQL> col OBJECT_TYPE for a30;
SQL> select owner, object_name, object_type from dba_objects 
where object_name like upper ('xdb_migrateschema');
OWNER               OBJECT_NAME                    OBJECT_TYPE
---------------------- ---------------------------------- -----------------
SYS                      XDB_MIGRATESCHEMA     PACKAGE
SYS                      XDB_MIGRATESCHEMA     PACKAGE BODY

b). Run adgrants.sql.

c). Recompile the package AD_ZD_PREP and ensure it compiles successfully.
SQL> alter package AD_ZD_PREP compile body;
Package body altered.

SQL> select owner,object_name,status from dba_objects 
where object_name like 'AD_ZD%' and status ='INVALID';
no rows selected

7). Check database to avoid errors from applying patch 13543062

$ cd $LOG_HOME/appl/op
$ sqlplus apps @$AD_TOP/sql/ADZDEXRPT.sql
Start Time: XX-02-2021 14:44:31
PL/SQL procedure successfully completed.
DOC>
DOC>  Online Patch Enablement - Execution Status Report of SQL jobs
DOC>
DOC>  Report Script: $AD_TOP/sql/ADZDEXRPT.sql
DOC>  Result Spooled Into: adzdexrpt.txt
DOC>
DOC>  Currently running Workers (database sessions) for SQL jobs execution.
DOC>
DOC>#
no rows selected
DOC>
DOC>  ***********************************************
DOC>  SECTION 1
DOC>  ***********************************************
DOC>  Count of Current DDL Jobs per Phase and Status
DOC>
DOC>  Phases:
DOC>    1.  DROP_UNUSED_OBJECT : Remove obsolete objects from XLA schema.
DOC>    2.  COPY_TYPE          : Create EBS Types in APPS_NE schema, stop Advanced Queues.
DOC>    3.  COMPILE_TYPE   : Recompile APPS_NE schema.
DOC>    4.  COPY_EVOLVED_TYPE  : Create evolved types in APPS_NE schema.
DOC>    5.  FIX_COLUMN                   : Fix TYPE reference to APPS_NE.<type> at table and AQ level.
DOC>    6.  FIX_TYPE              : Additional check, Fix TYPE reference and then drop TYPE from source schema.
DOC>    7.  FIX_PUBLIC_SYNONYM : Fix public synonyms.
DOC>    8.  RECREATE_AQ_OBJECT : Re-create internal AQ objects after TYPE Ref fix.
DOC>    9.  DROP_OBJECT     : Drop objects like CTXSYS.<package>, CTXSYS.<synonym> and others if any.
DOC>    10. ENABLE_EDITIONING  : Enable registered schemas for Online Patching.
DOC>    11. UPGRADE_TABLE      : Create Editioning View cover layer over EBS data model.
DOC>    12. UPGRADE_SEED       : Upgrade seed data tables to support edition based storage.
DOC>    13. COLLECT_STATS      : Gather Statistics on Seed Data Tables after upgrade.
DOC>    14. UPGRADE_MVIEW   : Convert Materialized Views with expand query.
DOC>    15. CUTOVER            : Perform maintenance operations on columns and indexes after adop APPLY phase
DOC>                                           and before actually changing Patch-Edition as Default-Edition.
DOC>    16. ACTUALIZE_ALL     : Actualize all editioned objects into the current edition.
DOC>    17. CLEANUP                   : Cleanup task after adop CUTOVER.
DOC>
DOC>#
Phase                              Count Status
------------------------------ --------- -----------------------
UPGRADE_SEED         2605   N - Not Executed
COLLECT_STATS         2605  N - Not Executed
CLEANUP                            1  N - Not Executed

3 rows selected.
Elapsed: 00:00:00.02
DOC>
DOC>  ************************************
DOC>  SECTION 2
DOC>  ************************************
DOC>  Failed Jobs with UNEXPECTED error, For complete error stack and DDL,
DOC>  Please check AD_ZD_DDL_HANDLER table for corresponding DDL_ID.
DOC>
DOC>#
no rows selected
Elapsed: 00:00:00.01
DOC>
DOC>  *************************************
DOC>  SECTION 3
DOC>  *************************************
DOC>  Failed Jobs with ERRORS, For complete error stack and DDL,
DOC>  Please check AD_ZD_DDL_HANDLER table for corresponding DDL_ID
DOC>
DOC>#
no rows selected
Elapsed: 00:00:00.01
DOC>
DOC>
DOC>  ************************************
DOC>  SECTION 4
DOC>  ************************************
DOC>  Running Jobs , Please check AD_ZD_DDL_HANDLER table for
DOC>  corresponding DDL_ID
DOC>
DOC>#
no rows selected
Elapsed: 00:00:00.00
DOC>
DOC>  ************************************
DOC>  SECTION 5
DOC>  ************************************
DOC>  Failed Jobs with Warnings, For complete error stack and DDL,
DOC>  Please check AD_ZD_DDL_HANDLER table for corresponding DDL_ID
DOC>
DOC>#
no rows selected
Elapsed: 00:00:00.01
DOC>
DOC>  ************************************
DOC>  SECTION 6
DOC>  ************************************
DOC>  E-Business Suite INVALID object-count: Object-Type wise and Total
DOC>
DOC>#
no rows selected
Elapsed: 00:00:00.00
Invalid Object Type                Count
------------------------------ ---------
PROCEDURE                             23
VIEW                                            5
TRIGGER                                     3
PACKAGE BODY                      45
PACKAGE                                    1
                               ---------
Total Invalid Objects (EBS):      77
5 rows selected.
Elapsed: 00:00:00.13
DOC>
DOC>  *****************************************
DOC>  SECTION 7
DOC>  *****************************************
DOC>  List of E-Business Suite INVALID objects
DOC>
DOC>#
Owner                Type                           Name
-------------------- ------------------------------ ---------------------------
CUSTOM          XXXXXXX           XXXXXXXXX
etc … … … 
APPS                 PACKAGE                               AK$RCV_CONFIRM_RECEIPT$178
APPS                 PACKAGE BODY                   AK$RCV_CONFIRM_RECEIPT$178
APPS                 PACKAGE BODY                   AP_PAYMENT_EVENT_WF_PKG
APPS                 PACKAGE BODY                   CZ_IMP_SINGLE
APPS                 PACKAGE BODY                   DOCSAVI_UA_DQ
APPS                 PACKAGE BODY                   HZ_STAGE_MAP_TRANSFORM
APPS                 PACKAGE BODY                   M4R_7B1_WSM_IN
APPS                 PACKAGE BODY                   OKS_COVERAGE_MIGRATION
APPS                 PACKAGE BODY                   OKS_RULE_MIGRATE
APPS                 PACKAGE BODY                   PA_RBS_MAPPING
APPS                 PACKAGE BODY                   PON_AUC_INTERFACE_TABLE_PKG
APPS                 PACKAGE BODY                   PON_CP_INTRFAC_TO_TRANSACTION
APPS                 PACKAGE BODY                   PON_OPEN_INTERFACE_PVT
APPS                 PACKAGE BODY                   PON_SLM_UTIL_PKG
APPS                 PACKAGE BODY                   POS_AP_INVOICES_PKG
APPS                 PACKAGE BODY                   POS_DATA_SECURITY
APPS                 PACKAGE BODY                   POS_SUPP_APPR
APPS                 PACKAGE BODY                   POS_WC_VIEW_CANCEL_PVT
APPS                 PACKAGE BODY                   PO_IMPORT_PACKAGE
APPS                 PACKAGE BODY                   PQP_AAT_INS
APPS                 PACKAGE BODY                   PQP_AAT_SHD
APPS                 PACKAGE BODY                   PQP_AAT_UPD
APPS                 PACKAGE BODY                   PSP_ENC_CREATE_LINES
APPS                 PACKAGE BODY                   PSP_SUM_TRANS
APPS                 PACKAGE BODY                   QA_RESULTS_INTERFACE_PKG
APPS                 PACKAGE BODY                   QLTTRAMB
APPS                 PACKAGE BODY                   UMX_LOGIN_HELP_PVT
APPS                 PACKAGE BODY                   UMX_NOTIFICATION_UTIL
APPS                 PACKAGE BODY                   UMX_REG_REQUESTS_PVT
APPS                 PACKAGE BODY                   UTL_RECOMP
APPS                 PACKAGE BODY                   WF_REPORT
APPS                 PACKAGE BODY                   WSH_WF_LOAD_TEND
APPS                 VIEW                                       OKS_COVTIMES_HIST_V
77 rows selected.
Elapsed: 00:00:00.12
DOC>
DOC>  ***********************************
DOC>  SECTION 8
DOC>  ***********************************
DOC>  Time taken by each PHASE
DOC>
DOC>#
Phase                    ROW_COUNT START_TIME        END_TIME        ELAPSED_TIME_MIN
------------------------ ----------- -------------------- -------------------- ------------
UPGRADE_SEED        2605
COLLECT_STATS       2605
CLEANUP                          1
3 rows selected.
Elapsed: 00:00:00.02
End Time: xx-02-2021 14:44:31
… … 

Run below script again to confirm DBA's fixes  (Recommended)
$ sqlplus apps @$AD_TOP/sql/ADZDDBCC.sql

8) Pre-steps for apply patch 13543062

SQL> set pages 200 lines 300
SQL> select owner,object_name,object_type,status from dba_objects
           where object_name='HR_PA_MAINTN_JOB_HIST_BRI' and owner='APPS';

OWNER   OBJECT_NAME                                 OBJECT_TYPE    STATUS
------------ ---------------------------------------------- ----------------------- -------
APPS        HR_PA_MAINTN_JOB_HIST_BRI   TRIGGER             VALID

SQL> select SERVICE_ID,NAME from dba_services where name='ebs_patch';
SERVICE_ID NAME
---------- ----------------------------------------------------------------
         8 ebs_patch
If it does not exist, follow Doc ID 2579880.1 to re-create it.

SQL> ALTER SYSTEM SET "_disable_actualization_for_grant"=true;      (if not set)
System altered.

SQL> alter system set max_dump_file_size=unlimited;
to avoid Patch 1354362 hangs on script ADZDWRKR.sql (Doc ID 1946877.1)

Find the version of Human Resources (HR) on my system (Doc ID 269286.1):
SQL> SELECT substr(fpi.application_id,1,6) APP_ID,
substr(fat.application_name,1,40) APPLICATION, substr(l.meaning,1,9) STATUS,
substr(decode(fpi.patch_level,null,'11i.'||fa.application_short_name||'.?',fpi.patch_level),1,12) PATCH
FROM fnd_product_installations fpi,
fnd_application_tl fat, fnd_application fa, fnd_lookups l
WHERE (fpi.application_id between 800 and 850 OR fpi.application_id in (178,275,712,777))
AND fpi.application_id = fat.application_id
AND fpi.application_id = fa.application_id
AND l.lookup_type = 'FND_PRODUCT_STATUS'
AND l.lookup_code = fpi.status
ORDER BY fat.application_id;

APP_ID APPLICATION                              STATUS    PATCH
------ ---------------------------------------- --------- ------------
178    Oracle iProcurement                      Installed   R12.ICX.D
275    Projects                                          Shared      R12.PA.C
712    Project Manufacturing                   Not insta   R12.PJM.C
777    Project Contracts                           Not insta   R12.OKE.H
800    Human Resources                         Shared       R12.PER.C
801    Payroll                                           Shared       R12.PAY.C
802    FastFormula                                  Shared       R12.FF.C
803    DateTrack                                      Shared      R12.DT.C
804    SSP                                                Not insta   R12.SSP.C
805    Advanced Benefits                        Not insta   R12.BEN.C
808    Time and Labor                             Not insta   R12.HXT.C
809    Time and Labor Engine                 Not insta  11i.HXC.C
810    Learning Management                  Not insta   R12.OTA.C
821    iRecruitment                                 Not insta    R12.IRC.C
14 rows selected.

SQL> set lines 180
set pages 50
col LANGUAGE_CODE for a15
col NLS_LANGUAGE for a20
col ISO_LANGUAGE for a15
col INSTALLED_FLAG for a15
SQL> SELECT LANGUAGE_CODE,NLS_LANGUAGE, ISO_LANGUAGE,INSTALLED_FLAG
FROM APPS.FND_LANGUAGES
WHERE INSTALLED_FLAG IN ('B','I');
LANGUAGE_CODE   NLS_LANGUAGE   ISO_LANGUAGE   INSTALLED_FLAG
--------------------------- -------------------------- ------------------------ ---------------
US                                 AMERICAN              EN                            B

SQL> select owner,object_name,status from dba_objects
           where object_name like 'AD_%' and status ='INVALID';
no rows selected

SQL> select fnd_profile.value('APPS_MAINTENANCE_MODE') from dual;
FND_PROFILE.VALUE('APPS_MAINTENANCE_MODE')
--------------------------------------------------------------------------------
MAINT

9) Apply patch 13543062

$ echo $FILE_EDITION
run
$ unzip p13543062_R12.AD.C_R12_GENERIC.zip      (only 6k in size)
$ cd 13543062
$ ls
$ adpatch options=hotpatch,forceapply
                               <== It already exists and so "forceapply"
                               <== Upgrade Guide p.126. It spends mostly on running SQL files
… etc … 
AutoPatch is complete.
AutoPatch may have written informational messages to the file
$APPL_TOP/admin/$TWO_TASK/log/13543062.lgi
Errors and warnings are listed in the log file
$APPL_TOP/admin/$TWO_TASK/log/13543062.log

$ egrep -i "error|fail|ORA-|Failed-|rc-" $APPL_TOP/admin/$TWO_TASK/log/13543062.log

Note: query "select * from ad_bugs where bug_number = '13543062';" will show a earlier creation_date.

If patch failed and not able to restart Online Enablement patch 13543062 using adpatch, try solution in Doc ID 1908956.1.  To implement the solution please perform following
a) In the terminal
export ENABLE_ADPATCH=YES
b) Run adaptch from the same terminal.
Note : The above solution is applicable only for Online enablement patch 13543062

10) Monitor the Online Patching enablement patch application: 
The enablement patch application may take more than one hour to finish. You can monitor its progress at any time by running the DDL Status Report (ADZDSHOWDDLS.sql) as follows:
SQL> show user
USER is "APPS"
SQL> @$AD_TOP/sql/ADZDSHOWDDLS.sql
DOC>
DOC>  Count of Current DDL Jobs per Phase and Status
DOC>
DOC>  Phases:
DOC>    1.  DROP_UNUSED_OBJECT : Remove obsolete objects from XLA schema.
DOC>    2.  COPY_TYPE          : Create EBS Types in APPS_NE schema, stop Advanced Queues.
DOC>    3.  COMPILE_TYPE       : Recompile APPS_NE schema.
DOC>    4.  COPY_EVOLVED_TYPE  : Create evolved types in APPS_NE schema.
DOC>    5.  FIX_COLUMN         : Fix TYPE reference to APPS_NE.<type> at table and AQ level.
DOC>    6.  FIX_TYPE           : Additional check, Fix TYPE reference and then drop TYPE
DOC>                             from source schema.
DOC>    7.  FIX_PUBLIC_SYNONYM : Fix public synonyms.
DOC>    8.  RECREATE_AQ_OBJECT : Re-create internal AQ objects after TYPE Ref fix.
DOC>    9.  DROP_OBJECT        : Drop objects like CTXSYS.<package>, CTXSYS.<synonym> and others if any.
DOC>    10. ENABLE_EDITIONING  : Enable registered schemas for Online Patching.
DOC>    11. UPGRADE_TABLE      : Create Editioning View cover layer over EBS data model.
DOC>    12. UPGRADE_SEED       : Upgrade seed data tables to support edition based storage.
DOC>    13. COLLECT_STATS      : Gather Statistics on Seed Data Tables after upgrade.
DOC>    14. UPGRADE_MVIEW      : Convert Materialized Views with expand query.
DOC>    15. CUTOVER            : Perform maintenance operations on columns and indexes after adop APPLY phase
DOC>                             and before actually changing Patch-Edition as Default-Edition.
DOC>    16. ACTUALIZE_ALL      : Actualize all editioned objects into the current edition.
DOC>    17. CLEANUP            : Cleanup task after adop CUTOVER.
DOC>
DOC>
DOC>  Results spooled to: ADZDSHOWDDLS.out
DOC>#
Phase                                      Count Status
----------------------------------- -------  --------------------
DROP_UNUSED_OBJECT         8 S - Successfully Executed
COPY_TYPE                              64 S - Successfully Executed
COMPILE_TYPE                         1 S - Successfully Executed
FIX_COLUMN                             3 S - Successfully Executed
FIX_TYPE                                  33 S - Successfully Executed
FIX_PUBLIC_SYNONYM    8680 S - Successfully Executed
RECREATE_AQ_OBJECT       16  S - Successfully Executed
DROP_OBJECT                        11  S - Successfully Executed
ENABLE_EDITIONING        173  S - Successfully Executed
UPGRADE_TABLE           17974   S - Successfully Executed
UPGRADE_SEED                2605  S - Successfully Executed
COLLECT_STATS               2605  S - Successfully Executed
UPGRADE_MVIEW              113  S - Successfully Executed
                                                     1  E - Error
CLEANUP                                  1  N - Not Executed

15 rows selected.
DOC>
DOC>  Currently running phase is:
DOC>
DOC>#
no rows selected

SQL> @$AD_TOP/sql/ADZDDDLERROR.sql
==============================
=           Errors in ad_zd_ddl_handler table
==============================
ORA-22818: subquery expressions not allowed here        <= not sure what it indicates.

11) Post patching

After the patch application completed, check invalid objects and re-run readiness reports: 

SQL> show user
USER is "APPS"
SQL> exec sys.utl_recomp.recomp_parallel;
PL/SQL procedure successfully completed.
SQL>  set lines 333 pages 123;
col owner for a27;
col object_name for a45;
col object_typr for a27;
col status for a9;
SQL> select owner,object_type,count(*) from dba_objects
where status <> 'VALID' and owner in ('APPS', 'PUBLIC')
group by owner,object_type order by owner,object_type;  
OWNER           OBJECT_TYPE             COUNT(*)
------------------- ---------------------------- ----------
APPS                 PACKAGE                    1
APPS                 PACKAGE BODY        39
APPS                 PROCEDURE               2
APPS                 VIEW                            3
PUBLIC            SYNONYM                  4181

Check out invalid synonyms:  
SQL> col table_owner for a15;
SQL> select table_owner, count(*) from dba_synonyms where synonym_name in
(select object_name from dba_objects
where status = 'INVALID' and owner in ('APPS', 'PUBLIC')) group by table_owner;

In R12.2, synonyms owned by PUBLIC can be dropped since PUBLIC shall not own synonym in R12.2. 

Re-run reports for comparison:

$ cd $LOG_HOME/appl/op
$ sqlplus apps @$AD_TOP/sql/ADZDEXRPT.sql          -- Online patching enablement report
$ mv adzdexrpt.txt adzdexrpt_post_dbprep.txt
NOTES: 
Any errors listed in this report must be fixed. Failure to comply may result in unexpecting failures during future patching. But ignore error on MView MRP_COMPANY_USERS_SN if Oracle MRP/Planning is not used due to an internal bugs.

$ sqlplus system @$AD_TOP/sql/ADZDPSUM.sql        -- Readiness reports
$ mv adzdpsum.txt adzdpsum_post_dbprep.txt
$ sqlplus system @$AD_TOP/sql/ADZDPMAN.sql
$ mv adzdpman.txt adzdpman_post_dbprep.txt
$ sqlplus system @$AD_TOP/sql/ADZDPAUT.sql
$ mv adzdpaut.txt adzdpaut_post_dbprep.txt

$ sqlplus apps @$AD_TOP/sql/ADZDDBCC.sql           -- Database compliance checker report
$ mv adzddbcc.lst adzddbcc_post_dbprep.lst
NOTES: 
Full Compliance (Full) - These checks indicate whether an object can be patched using Online Patching. Objects which do not meet full compliance may have limitations in how they can be patched, or may need to be patched using downtime patching. Full compliance also requires that all minimal compliance checks are passed. Custom code that will only be patched using downtime patching does not need to meet the full compliance level. 

Warning: Do not attempt to operate the system if there are P1 minimal compliance violations. Custom code should pass the minimal compliance checks before being used in a Release 12.2 system. P1 violations must be fixed before using the system or object

Check all reports to see if any fixes are necessary in Dev and QA environments. That will help to run same fixes quickly during upgrade's go-live. 

$ du -h --max-depth=1
32G     ./fs1
29G     ./fs2
32K     ./fs_ne

Do a backup on database and 3 R12.2 folders: fs1, fs2, fs_ne
-rw-r--r--  1 user group 12942534633 Feb 11 13:12 fs1_bkp_0211.tar.gz
-rw-r--r--  1 user group 11625901541 Feb 11 13:12 fs2_bkp_0211.tar.gz
-rw-r--r--  1 user group               4402 Feb 11 12:34 fs_ne_bkp_0211.tar.gz

By now,  R12.2.0 upgrade is completed and Online Patching is enabled. All Oracle E-Business Suite patches from this point forward will be performed using AD Online Patching (adop).

Next, follow Doc ID 1617461.1 to apply the Latest AD and TXK Release Update Packs (delta.12) to Oracle E-Business Suite Release 12.2.

Sunday, February 28, 2021

Apply CUP & merged patches, and run Rapid Install to configure R12.2.0

After Rapid Installation worked,  the next is to follow steps in Section 3 of Doc ID 1320300.1 to apply R12.2 Consolidated Upgrade Patch (CUP) & merged patches, and then configure R12.2.0 upgrade using Rapid Install (see EBS Upgrade Guide Release 12.0 and 12.1 to 12.2. Part No. E73540-09).

1. Pre-steps: 

a) download 2 patches to and merge them
$ cd /u01/app/patchR122/AD_source
$ ls
p10117518_R12_LINUX.zip                     
p30370733_R12.AD.C_R12_LINUX.zip   <== It includes patches 21670164 and 21833257
$ unzip p10117518_R12_LINUX.zip
$ unzip p30370733_R12.AD.C_R12_LINUX.zip
$ cd ..
$ admrgpch -s /path/to/AD_source -d /path/to/10117518_CUP11 -merge_name 10117518_cup11 -admode

Executing the merge of the patch drivers
 -- Processing patch: /path/to/AD_source/10117518
 -- Processing file: /path/to/AD_source/10117518/u10117518.drv
 -- Done processing file: /path/to/AD_source/10117518/u10117518.drv
 -- Done processing patch: /path/to/AD_source/10117518
 -- Processing patch: /path/to/AD_source/30370733
 -- Processing file: /path/to/AD_source/30370733/u30370733.drv
 -- Done processing file: /path/to/AD_source/30370733/u30370733.drv
 -- Done processing patch: /path/to/AD_source/30370733
Copying files...
5% complete. Copied 47 files of 925...
10% complete. Copied 93 files of 925...
15% complete. Copied 139 files of 925...
20% complete. Copied 185 files of 925...
25% complete. Copied 232 files of 925...
30% complete. Copied 278 files of 925...
35% complete. Copied 324 files of 925...
40% complete. Copied 370 files of 925...
45% complete. Copied 417 files of 925...
50% complete. Copied 463 files of 925...
55% complete. Copied 509 files of 925...
60% complete. Copied 555 files of 925...
65% complete. Copied 602 files of 925...
70% complete. Copied 648 files of 925...
75% complete. Copied 694 files of 925...
80% complete. Copied 740 files of 925...
85% complete. Copied 787 files of 925...
90% complete. Copied 833 files of 925...
95% complete. Copied 879 files of 925...
100% complete. Copied 925 files of 925...
Character-set converting files...
  2 unified drivers merged.
Patch merge completed successfully
Please check the log file at ./admrgpch.log.

$ cd 10117518_CUP11     <= created by admrgpch. It can be zipped up and used by other instances
$ ls 
$ find . -name adgrants.sql
./admin/adgrants.sql
$ grep Header ./admin/adgrants.sql
REM $Header: adgrants.sql 120.67.12020000.69 2020/03/20 00:56:38 jwsmith ship $

$ cp -p ./admin/adgrants.sql /to/path/for/DBA
$ ls -al /to/path/for/DBA/adgrants.sql
-rw-r--r--  1 user group  108545 Xxx 20 12:09 adgrants.sql

b) DBA work

- runs adgrants.sql (It is from patch 30370733:R12.AD.C above).
sqlplus / as sysdba @adgrants.sql APPS

- Check nothing is enabled or the view dba_audit_policies does not exist:
SQL> select distinct enabled from dba_audit_policies;

- make sure database Profile options are good to avoid below error from installing merged patch:
ORA-28003: password verification for the specified password failed
ORA-20001: Password contains the username. (GHG)
ORA-28007: the password cannot be reused

SQL> select profile,resource_name,limit from dba_profiles
where profile in ('DEFAULT','TRUSTED_ID_NO_EXPIRE')
and  resource_type = 'PASSWORD'  order by 1,2;

c) Apply patch 6767273 asked by README of 10117518 (if it has not been applied)

d) Apply two AD patches

$ cd /path/to/$TWO_TASK/fs1/EBSapps/appl
$ . ./APPS*.env
$ which adadmin
~/$TWO_TASK/fs1/EBSapps/appl/ad/12.0.0/bin/adadmin
$ adadmin 
to enable maintenance mode. Or, run SQL statement to do that. Confirm it:

SQL> select fnd_profile.value('APPS_MAINTENANCE_MODE') from dual;
FND_PROFILE.VALUE('APPS_MAINTENANCE_MODE')
--------------------------------------------------------------------------------
MAINT

$ which adpatch
~/$TWO_TASK/fs1/EBSapps/appl/ad/12.0.0/bin/adpatch

$ cd /path/to/10117518_CUP11
$ adpatch logfile=10117518.log
… … Your default directory is '/u02/app/$TWO_TASK/fs1/EBSapps/appl'
... ...
No of records processed =415053 Updating Snapshot Tables...Start time:Mon XXX 28 2021 13:09:13
Done Updating Snapshot Tables for the above rows...End Time:Mon XXX 28 2021 13:09:14
... ...
No of records processed =654892
   Updating Snapshot Tables...Start time:Mon Jun 28 2021 13:09:43
      Done Updating Snapshot Tables for the above rows...End Time:Mon XXX 28 2021 13:09:43
Preseeded snapshot import completed successfully.
... ...
The default directory is [/path/to/10117518_CUP11] :
Please enter the name of your AutoPatch driver file : u_10117518_cup11.drv

It takes about 15 minutes. The log file is saved in $APPL_TOP/admin/$TWO_TASK/log. 
 
Confirm two database accounts APPS_NE and GHG are created, and 5 patches were applied.

SQL> select * from dba_users where username in ('GHG', 'APPS_NE');
SQL> select bug_number, to_char(creation_date,'DD-MON-RRRR HH24:MI:SS') date_applied
from apps.ad_bugs
where bug_number in ('6767273', '21670164','21833257', '10117518', '30370733' );

Post-step: compile APPS schema

$ adadmin
  a. Run adadmin => Compile/Reload Applications Database Entities menu 
                            => Compile APPS schema.
  b. Enter "No" when prompt for Run Invoker Rights processing in incremental mode [No] ?

SQL> set lines 333 pages 123;
 col owner for a20;
 col object_name for a45;
 col object_type for a20;
SQL> SELECT owner, substr(object_name,1,30) object_name, object_type, status
          FROM dba_objects where status = 'INVALID'
        ORDER BY owner, object_type, object_name;

2. Preinstall 3 patches
Apply the pre-upgrade patches that were released after the Consolidated Upgrade Patch 30399970. See Doc ID 1448102.2. In January 2021, the document added patch 31745734.  We did not hit problem in upgrading R12.1.3 to R12.2 without applying patch 31745734, prior to January 2021.

$ echo $ORACLE_HOME
/u02/app/$TWO_TASK/fs1/EBSapps/10.1.2
$ echo $FILE_EDITION
run

$ ls -al $AU_TOP/patch/115/driver
-rwxr-xr-x 1  user group      233 Nov 24  2012 augenmsg.drvx
-rwxr-xr-x 1  user group      755 Nov 24  2012 ausstats.drvx
-rw-r--r-- 1 user group 83457868 Nov 25  2012 u10124646.drv
-rw-r--r-- 1 user group 14305818 Nov 25  2012 u10201000.drv

$ cd /path/to/preinstall
$ unzip p30399970_12.2.0_R12_LINUX.zip
$ cd 30399970
$ ls
$ adpatch preinstall=y
(Or, $ adpatch preinstall=y defaultsfile=$APPL_TOP/admin/$TWO_TASK/adalldefaults.txt logfile=30399970.log driver=u30399970.drv )

Take default answer to all questions. It does not ask for db passwords. 
It does make, relink, etc. New files are created in folder: 
$ ls -al $APPL_TOP/admin/$TWO_TASK

$ cd ../
$ unzip p31219447_R12.PA.C_R12_GENERIC.zip
$ cd 31219447
$ ls
$ adpatch preinstall=y

$ cd ..
$ unzip p31745734_R12.FND.C_R12_GENERIC.zip
$ cd 31745734
$ adpatch preinstall=y logfile=u31745734.log driver=u31745734.drv
$ egrep -i "Error|Failed|ORA-" ... /log/u31745734.log

By now, two patches are saved to $APPL_TOP/admin/$TWO_TASK/preinstall
$ ls -al $APPL_TOP/admin/$TWO_TASK/preinstall

-rw-r--r-- 1 user group 342037 Jan 27 15:46 u30399970.drv
-rw-r--r-- 1 user group   1323   Jan 27 15:48 u31219447.drv
-rw-r--r-- 1 user group   1292   Jan 27 16:54 u31745734.drv

3. Merge 4 patches 

$ echo  $ORA_NLS10
$RUN_BASE/EBSapps/10.1.2/nls/data/9idata
$ cd $AU_TOP/patch/115/driver
$ ls -al $AU_TOP/patch/115/driver
-rwxr-xr-x 1  user group      233 Nov 24  2012 augenmsg.drvx
-rwxr-xr-x 1  user group      755 Nov 24  2012 ausstats.drvx
-rw-r--r-- 1 user group 83457868 Nov 25  2012 u10124646.drv
-rw-r--r-- 1 user group 14305818 Nov 25  2012 u10201000.drv

$ admrgpch -d . -preinstall -master u10124646.drv
Executing the merge of the patch drivers
 -- Processing file: $APPL_TOP/admin/$TWO_TASK/preinstall/u30399970.drv
 -- Done processing file: $APPL_TOP/admin/$TWO_TASK/preinstall/u30399970.drv
 -- Processing file: $APPL_TOP/admin/$TWO_TASK/preinstall/u31219447.drv
 -- Done processing file: $APPL_TOP/admin/$TWO_TASK/preinstall/u31219447.drv
 -- Processing file: $APPL_TOP/admin/$TWO_TASK/preinstall/u31745734.drv
 -- Done processing file: $APPL_TOP/admin/$TWO_TASK/preinstall/u31745734.drv
 -- Processing file: u10124646.drv
 -- Done processing file: u10124646.drv
4 unified drivers merged.
Patch merge completed successfully
Please check the log file at ./admrgpch.log.

$ ls -altr    <== to see new file u_merged.drv
-rwxr-xr-x 1 user group      755      Nov 24  2012 ausstats.drvx
-rw-r--r-- 1 user group 83457868   Nov 25  2012 u10124646.drv
-rw-r--r-- 1 user group 14305818   Nov 25  2012 u10201000.drv
-rwxr-xr-x 1 user group      242      Jan  27  11:30 augenmsg.drvx
-rw-rw-r-- 1 user group 83855365 Jan  27  11:38 u_merged.drv
-rw-rw-r-- 1 user group     1595      Jan 27  11:38 admrgpch.log

4. Apply the merged patch (with 10124646). Note: it may take more than 4 hours.

Database pre-steps by SYSTEM (in my instance to avoid error during patching):
a. SQL> drop index AP.AP_LIABILITY_BALANCE_N4;   
    If it exists, drop it to avoid error on aplbupgind.sql

b SQL> SELECT RULE_ID , LEVEL_ID , LEVEL_VALUE , LEVEL_VALUE_APPLICATION_ID , COUNT(*) 
FROM APPLSYS.FND_FORM_CUSTOM_SCOPES 
GROUP BY RULE_ID , LEVEL_ID , LEVEL_VALUE , LEVEL_VALUE_APPLICATION_ID 
HAVING COUNT(*) > 1;
If duplicate row exists, delete it to avoid error "Index FND_FORM_CUSTOM_SCOPES_U1 does not exist in APPLSYS" (Doc ID 1591975.1)

c. SQL> GRANT DROP ANY PROCEDURE TO APPS;     
    To avoid privilege error from gldrpzfa.sql, gldrpzsa.sql

d. Confirmed Service Contracts (OKS) is not being used. See Doc ID 741800.1 about errors on OKSCVTIM.sql,  OKSCTIMZ.sql, OKSRTIME.sql on table OKC.OKC_TIMEVALUES_B.

e. SQL> select fnd_profile.value('APPS_MAINTENANCE_MODE') from dual;
FND_PROFILE.VALUE('APPS_MAINTENANCE_MODE')
------------------------------------------------------
MAINT

Apply the merged patch:
$ echo $FILE_EDITION
run
$ cd $AU_TOP/patch/115/driver
$ adpatch options=nocopyportion,nogenerateportion logfile=merge_10124646.log driver=u_merged.drv

The default directory is [$AU_TOP/patch/115/driver]:

AutoPatch will not execute the following actions in your patch driver file
because you requested this using the keyword argument 'options='.
  libout
  copy
  forcecopy
  jcopy
  genform
  libin
  link
  genrep
  genrpll
  genfpll
  genmenu
  genmesg
  genwfmsg
  makedir
  gengpll
  genogd
Do you want to run AutoPatch with these actions turned off [Yes] ?   <= Enter to accept Yes

It takes totally 3 hours or more than 4 hours (depending on the database size). It may spend long time on adobjcmp.sql (twice) and on other sql scripts.  After more than 2 hours (running db scripts), I had to use adctrl to skip (#8) OKS scripts 3 times.

Logs are in $APPL_TOP/admin/$TWO_TASK/log. Files *_preenv.lst & *postenv.lst in  $APPL_TOP/admin/$TWO_TASK/log list invalid objects.

By now, steps in Section 3 of Doc ID 1320300.1 are completed. Below query returns 6 rows:

SQL> select * from ad_bugs
where bug_number in ('10117518','30370733','30399970','31219447','31745734','10124646')
order by creation_date desc;

Disable maintenance mode by adadmin, or run
SQL> @$RUN_BASE/EBSapps/appl/ad/12.0.0/patch/115/sql/adsetmmd.sql DISABLE

$ cd $TWO_TASK
$ du -sh fs*
31G     fs1
28G     fs2
32K     fs_ne 

Backup apps file system and database (Recommended!)

5. Prepare and Run Rapid Install to configure and finish the 12.2.0 Upgrade 
(page 111 (4-49), Oracle® EBS Upgrade Guide Release 12.0 and 12.1 to 12.2. Part No. E73540-09)

$ cd $TWO_TASK/fs1/EBSapps/appl
$ . ./APPS*.env

$ grep s_dbhost $CONTEXT_FILE
         <dbhost oa_var="s_dbhost">db_nodename</dbhost>

a) Clean up old node information. 
SQL> exec fnd_conc_clone.setup_clean;
SQL> select * from fnd_nodes;

Notes: This is a very important and necessary step. Without it, it may hit error on finding adop_valid_nodes when applying R12.AD.C.Delta.12 patch (30628681) later and other strange errors. If so, it is better to start the upgrade again from the beginning.

b) Create appsuil.zip file for db node
$ cd $AD_TOP/bin
$ perl admkappsutil.pl
Copy $INST_TOP/admin/out/appsutil.zip to database node

c) Run autoconfig on db node
$ perl adbldxml.pl jtop=$ORACLE_HOME/jdk
Starting context file generation for db tier..
Enter the value for Display Variable: ebsw1d:0.0
The context file has been created at: $ORACLE_HOME/appsutil/<CONTEXT_FILE>.xml
$ ./adconfig.sh 
Enter the full path to the Context file:

d) Make sure FAILED_LOGIN_ATTEMPTS is 'UNLIMITED' for database profile used by APPS account.
SQL> select profile, resource_name, limit  from dba_profiles where profile in (
select profile from dba_users where username = 'APPS')
and resource_type = 'PASSWORD'
order by resource_name;

Notes: Without this setting, Rapid Install on completing the process of configuring R12.2.0 will fail with "Database Availability" on the last GUI screen because APPS account will be locked. If this happens, do not click on "Retry" after APPS account is unlocked as it will skip some important steps, such as creating entire folder user_projects under $FMW_HOME. You have to launch Rapid Install to run configuration from beginning until it completes successfully.

e) Drop table ADX_PRE_AUTOCONFIG from APPS schema (which will be created by autoconfig in APPLSYS schema)
$ cd $AD_TOP/patch/115/sql
$ sqlplus apps/appsPWD 
SQL> select owner, object_type from dba_objects 
  where object_name in ('ADX_PRE_AUTOCONFIG') ;
SQL> @txkDropAdxPreAutoConfig.sql
SQL> select * from dba_objects where object_name in ('ADX_PRE_AUTOCONFIG') ;
no rows selected

f) Make sure database accounts APPS and SYSTEM are not locked, and verify other info:
SQL> select SERVICE_ID,NAME from dba_services where name='ebs_patch';
SERVICE_ID NAME
---------- -----------------
         8 ebs_patch
SQL> select fnd_web_sec.validate_login('GUEST','ORACLE') valid from dual;
VALID
------------------------------------------------------
Y
SQL> select fnd_profile.value('APPS_MAINTENANCE_MODE') from dual;
FND_PROFILE.VALUE('APPS_MAINTENANCE_MODE')
------------------------------------------------------
NORMAL


g) run Rapid Install to configure R12.2.
$ echo $FILE_EDITION
run
$ echo $RUN_BASE
/u02/app/$TWO_TASK/fs1
$ echo $ORACLE_HOME
$RUN_BASE/EBSapps/10.1.2
$ export DISPLAY=123.45.678:0.0

$ cd /path/to/rapidSTARTCD/startCD/Disk1/rapidwiz
$ ./rapidwiz

choose Upgrade to Oracle E-Business Suite Release 12.2.0
            "Configure Updated Release 12.2.0 Instance" => use $CONTEXT_FILE
enter password for apps, applsyspub, guest and WebLogic Admin
System check marks on GUI shall all be green.

It stays at 0% for a log time. By the end, a final GUI shall show all green check marks and say "This instance passed all of the post-install tests". It writes some error message from autoconfig and others to log files at $INST_TOP/logs. The log file size on /sf1 shall be about 190k. If the log size is much smaller, most likely it skipped some steps due to a problem. The end of it shall have:
… … 
riwTDBup.sh exiting with status 0
Apps ORACLE_HOME connection test has succeeded
Environment File
-----------------
   ADOVARS environment file passes instantiated variables test:
      File = $RUN_BASE/EBSapps/appl/admin/adovars.env
   APPSCONFIG passes instantiated variables test:
      File =
$RUN_BASE/EBSapps/appl/admin/adconfig.txt
DBC File
---------
   DBC file passes instantiated variables test: File = $FND_TOP/secure/${TWO_TASK}.dbc
OS Password is null.  SSH is disabled.
WLS Domain Validation
----------------------
EBS Domain validation Successful for RUN filesystem.
-------------------------------------------------
EBS Domain validation Successful for PATCH filesystem.
-------------------------------------------------
DoInstallPanel - Summary Text
The Rapid Install Wizard will now run AutoConfig to configure this node:


The end of log file on /sf2 shall have:
… … 
AutoConfig completed successfully.
FINISHED INSTALL PHASE : APPLTOP : Web MMM 6 12:23:34 EDT 2021
… …  
stty: standard input: Inappropriate ioctl for device
ConfigAppNode :Please refer to the remaining logs on applications node at - $INST_TOP/logs/xxxx1143.log

$ grep s_dbhost $CONTEXT_FILE
         <dbhost oa_var="s_dbhost">db_nodename</dbhost>          (without "domain.com")

$ grep -i APPLPTMP $CONTEXT_FILE
         <APPLPTMP oa_var="s_applptmp" osd="UNIX">/usr/tmp</APPLPTMP>
SQL> select value from v$parameter where name='utl_file_dir';
VALUE
------------------------------------------------------------------------
/path/to/EBSDEV/utl_dir

Ensure that the $APPLPTMP environment variable is set to the first database directory
defined for PL/SQL file I/O. See Doc ID 2525754.1 for more on UTL_FILE_DIR. Concurrent job uses $APPLPTMP (and this can be done later).
$ vi $CONTEXT_FILE   (to make s_applptmp use /path/to/EBSDEV/utl_dir)

At this stage, adadmin and adautocfg.sh all work.  I did run adautocfg.sh to confirm no error.
$ cd $ADMIN_SCRIPTS_HOME
$ ./adautocfg.sh
… …  AutoConfig completed successfully.

If AutoConfig failed because of table APPLSYS.ADX_PRE_AUTOCONFIG, drop it. When autoconfig is re-run, table ADX_PRE_AUTOCONFIG will be created with the correct number of columns (Doc ID 1912332.1).
SQL> drop table APPLSYS.ADX_PRE_AUTOCONFIG;

$ cd $FMW_HOME
$ ls -al user_projects        <= to confirm it exists. If not, Configuration ran wrong.
$ cd 
$ cd $TWO_TASK
$ du -h --max-depth=1    ( or, $ du -sh fs* )
33G     ./apps
3.0G    ./inst
32G     ./fs1
29G     ./fs2
32K     ./fs_ne

After Rapid Install configuration completed, do not run ADSplice for CUSTOM TOPs now. If run it, ADOP later may have problem on FS_CLONE. EBS Upgrade Guide page 130 says:
Do not run adsplice when you are at the Oracle E-Business Suite Release 12.2.0 level. Before running adsplice, you must upgrade to R12.AD.C.Delta.5 and R12.TXK.C.Delta.5 Release update packs or higher for AD and TXK. (They will be applied by using adop).

Tuesday, January 26, 2021

Apply patches required by checkMTpatch.sh (ETCC)

After rapidwiz completed successfully, the next step is to apply a list of patches reported by checkMTpatch.sh from patch 17537119 (see page 97 of  R12.2 Upgrade Guide: Release 12.0 and 12.1 to 12.2, Part No. E73540-10). More details are in Section 4 of Doc ID 1594274.1 (R12.2: Consolidated List of Patches and Technology Bug Fixes). 

The purpose is to patch and address issues in software installed by Rapid Install that was released a while ago. Oracle keeps updating patch 17537119, which may report a different list of patches depending on when patch 17537119 was downloaded.

1. Generate ETCC report
Add ". /u02/app/EBSDEV/fs1/EBSapps/appl/APPS*.env" to OS .profile, and start a new OS session. Or,
$ cd ../fs1/EBSapps/appl
$ . ./APPS*.env

$ strings -a  checkMTpatch.sh |grep '$Header'
# $Header: checkMTpatch.sh 120.0.12020000.42 2020/08/18 04:16:12 chrhill noship $

$ ./checkMTpatch.sh
It needs APPS password, and creates a log file with the list of patches in it.

2. Apply patches: patching in 5 areas

1) Oracle Forms and Reports 10.1.2.3.0  (3 patches)
  Patch 26825525
  Patch 25342269
  Patch 27491934

$ echo $FILE_EDITION
run
$ echo $ORACLE_HOME
/u02/app/$TWO_TASK/fs1/EBSapps/10.1.2
$ which opatch
~/$TWO_TASK/fs1/EBSapps/10.1.2/OPatch/opatch

$ unzip p26825525_101232_LINUX.zip
$ cd 26825525
$ opatch apply
Oracle Interim Patch Installer version 1.0.0.0.64 … ...
Rolling back patch 8557019 ...
Rolling back patch 18620223 ...
Rolling back patch 19434967 ...
Applying patch 26825525...

$ cd ..
$ unzip p25342269_101232_Generic.zip
$ cd 25342269
$ opatch apply

$ cd ..
$ unzip p27491934_101232_LINUX.zip
$ cd 27491934
$ opatch apply

$ opatch lsinventory
It lists 44 patches in my instance.

2) Oracle Fusion Middleware (FMW) - Web Tier 11.1.1.9.0  (4 patches)
  Patch 22288381
  Patch 31047338 (OHS 11.1.1.9.0 SPU FOR AprCPU2020, Doc ID 2650675.1)
  Patch 31304503
  Patch 23716938

$ echo $IAS_ORACLE_HOME
$ export ORACLE_HOME=$IAS_ORACLE_HOME
$ echo $ORACLE_HOME
/u02/app/$TWO_TASK/fs1/FMW_Home/webtier
$ export PATH=$IAS_ORACLE_HOME/OPatch:$PATH
$ echo $PATH
$ which opatch
~/$TWO_TASK/fs1/FMW_Home/webtier/OPatch/opatch
$ opatch lsinventory
Oracle Interim Patch Installer version 11.1.0.12.9 … ...

$ cd ..
$ unzip p22288381_111190_Generic.zip
$ cd 22288381
$ opatch apply
Rolling back interim patch '20922780' ...
Patch 22288381 successfully applied

$ cd ..
$ unzip p31047338_111190_Linux-x86-64.zip
$ cd 31047338
$ opatch apply
Patching component oracle.ohs2, 11.1.1.9.0...
Patch 31047338 successfully applied

$ cd ..
$ unzip p31304503_111190_Linux-x86-64.zip
$ cd 31304503
$ opatch apply
Patch 31304503: Optional component(s) missing : [ oracle.idm.oid, 11.1.1.9.0 ] …  … 
Is the local system ready for patching? [y|n]
Y
User Responded with: Y
Backing up files...
Patching component oracle.ldap.rsf, 11.1.1.9.0...
Patching component oracle.rdbms.rsf, 11.1.0.7.0...
Patch 31304503 successfully applied

$ cd ..
$ unzip p23716938_111190_Generic.zip
$ cd 23716938
$ opatch apply
Patching component oracle.opmn, 11.1.1.9.0...
Patch 23716938 successfully applied
  
3) RSF within FMW Web tier 11.1.0.7.0 (fs1/FMW_Home/webtier, 1 patch)
-------------------------------------------------
$ echo $IAS_ORACLE_HOME
$ export ORACLE_HOME=$IAS_ORACLE_HOME
$ export PATH=$IAS_ORACLE_HOME/OPatch:$PATH
$ echo $ORACLE_HOME
/u02/app/$TWO_TASK/fs1/FMW_Home/webtier   (<= same as above 2)! )
$ which opatch
~/$TWO_TASK/fs1/FMW_Home/webtier/OPatch/opatch

$ cd ..
$ unzip p22290164_111070_Linux-x86-64.zip   (<= it is a large patch)
$ cd 22290164
$ opatch apply

OPatch detects the Middleware Home as "/u02/app/$TWO_TASK/fs1/FMW_Home"
Applying interim patch '22290164' to OH '/u02/app/$TWO_TASK/fs1/FMW_Home/webtier'
Verifying environment and performing prerequisite checks...
Patch 22290164: Optional component(s) missing : [ oracle.network.listener, 11.1.0.7.0 ] , … … 
… … 
All checks passed.

Is the local system ready for patching? [y|n]
Y
User Responded with: Y
Backing up files...
Patching component oracle.rdbms.rsf, 11.1.0.7.0...
Patching component oracle.network.rsf, 11.1.0.7.0...
Patching component oracle.xdk.parser.java, 11.1.0.7.0...
Patching component oracle.xdk, 11.1.0.7.0...
Patching component oracle.xdk.rsf, 11.1.0.7.0...
Patching component oracle.rdbms.rsf, 11.1.0.7.0...
Patching component oracle.network.rsf, 11.1.0.7.0...
Patching component oracle.dbjava.ic, 11.1.0.7.0...
Patching component oracle.dbjava.jdbc, 11.1.0.7.0...
Verifying the update...
Patch 22290164 successfully applied
Log file location: /u02/app/$TWO_TASK/fs1/FMW_Home/webtier/cfgtoollogs/opatch/22290164_x/apply2021_x.log
OPatch succeeded.

$ opatch lsinventory
It lists 15 patches in my instance.

4) Oracle Fusion Middleware (FMW) - oracle_common 11.1.1.9.0  (12 patches)
---------------------------------------------------------
$ echo $FMW_HOME
$ export ORACLE_HOME=$FMW_HOME/oracle_common
$ export PATH=$ORACLE_HOME/OPatch:$PATH
$ echo $ORACLE_HOME
/u02/app/$TWO_TASK/fs1/FMW_Home/oracle_common
$ which opatch
~/$TWO_TASK/fs1/FMW_Home/oracle_common/OPatch/opatch

$ unzip p9905685_111190_Generic.zip
$ cd 9905685/oui
$ opatch apply
Patching component oracle.soa.mgmt, 11.1.1.9.0...

$ unzip p17428617_111190_Generic.zip
$ cd 17428617
$ opatch apply
Patching component oracle.jrf.thirdparty.jee, 11.1.1.9.0...

$ unzip p21366277_111190_Generic.zip
$ cd 21366277
$ opatch apply
Patching component oracle.as.common.clone, 11.1.1.9.0...

$ unzip p21628307_111190_Generic.zip
$ cd 21628307
$ opatch apply
Patching component oracle.jrf.thirdparty.jee, 11.1.1.9.0...

$ unzip p24606362_111190_Generic.zip
$ cd 24606362
$ opatch apply
Patching component oracle.as.common.clone, 11.1.1.9.0...

$ unzip p24843064_111190_Generic.zip
$ cd 24843064/oui
$ opatch apply
Patching component oracle.jrf.adfrt, 11.1.1.9.0...

$ unzip p13544814_111190_Generic.zip
$ cd 13544814
$ opatch apply
Patching component oracle.jrf.thirdparty.jee, 11.1.1.9.0...

$ unzip p16089637_111190_Generic.zip
$ cd 16089637
$ opatch apply
Patching component oracle.jrf.thirdparty.jee, 11.1.1.9.0...

$ unzip p20141119_111190_Generic.zip
$ cd 20141119
$ opatch apply
Patching component oracle.jrf.thirdparty.jee, 11.1.1.9.0...

$ unzip p30368663_111190_Generic.zip
$ cd 30368663/oui
$ opatch apply
Patching component oracle.jrf.adfrt, 11.1.1.9.0...

$ unzip p27212806_111190_Generic.zip
$ cd 27212806
$ opatch apply
Patching component oracle.as.common.clone, 11.1.1.9.0...

$ unzip p28708563_111190_Generic.zip
$ cd 28708563
$ opatch apply
Rolling back interim patch '20484781' … 
OPatch back to application of the patch '28708563' after auto-rollback.
Patching component oracle.as.common.clone, 11.1.1.9.0...

$ opatch lsinventory
It lists 13 patches in my instance.

Exit from this OS session

5) Oracle WebLogic Server (WLS) 10.3.6.0.7
-------------------------------------------------------------------------------
  Patch 29633432 [SU Patch [MXLE]: WLS PSU 10.3.6.0.190716]
    - Filename: p29633432_1036_Generic.zip

Since Oracle site says 29633432 was originally replaced by patch 30109677, and the most recent replacement for this patch is 32052267 (WLS PATCH SET UPDATE 10.3.6.0.210119), I decided to apply patch 32052267, instead. (Notes on March 2021: Doc 2737201.1 (Table 2) says WLS patch 32052267 is part of R12.2 Jan2021 CPU patch. So, applying it was a good choice as it will save time when applying Jan2021 CPU ).

Start a new OS session (to get the right ORACLE_HOME)
$ echo $TWO_TASK
$ echo $ORACLE_HOME
/u02/app/$TWO_TASK/fs1/EBSapps/10.1.2
$ ls -al $FMW_HOME/wlserver_10.3
$ echo $FILE_EDITION
run

$ cd $FMW_HOME/utils/bsu
$ cp -p bsu.sh bsh.sh_orig
$ vi bsu.sh       <== to increase Java cache size
$ diff bsu.sh bsu.sh_orig
< MEM_ARGS="-Xms4096m -Xmx4096m -XX:+UseParallelGC"
---
> MEM_ARGS="-Xms512m -Xmx1024m -XX:+UseParallelGC"

Download p32052267_1036_Generic.zip to WLS folder and apply it:
$ cd $FMW_HOME/utils/bsu/cache_dir
$ unzip p32052267_1036_Generic.zip
Archive:  p32052267_1036_Generic.zip
  inflating: 1YWL.jar
  inflating: patch-catalog_27373.xml
  inflating: README.html
  inflating: README.txt

$ cd $FMW_HOME/utils/bsu
$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=1YWL -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts............                                      <= this takes 15 minutes or so.
Conflict(s) detected - resolve conflict condition and execute patch installation again
Conflict condition details follow:
Patch 1YWL is mutually exclusive and cannot coexist with patch(es): YHJK,2GYW,VKXF,FCX7,BLTG,Y5AP,CW9T,1LRI

$  ./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -remove -verbose -patchlist=YHJK
Checking for conflicts..........                                 <= this takes about 15 minutes.
No conflict(s) detected
Starting removal of Patch ID: YHJK
--- ---
Result: Success

$ ./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -remove -verbose -patchlist=2GYW
$ ./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -remove -verbose -patchlist=VKXF

$ ./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -remove -verbose -patchlist=FCX7
Checking for conflicts..........       
Conflict(s) detected - resolve conflict condition and execute patch removal again
Conflict condition details follow:
The selected patch cannot be removed until the following patch(es) are removed first: 1LRI,CW9T,BLTG

./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -remove -verbose -patchlist=BLTG
./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -remove -verbose -patchlist=CW9T
./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -remove -verbose -patchlist=1LRI
./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -remove -verbose -patchlist=FCX7
./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -remove -verbose -patchlist=Y5AP

- Apply it again (after all removals)
$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=1YWL -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts............
No conflict(s) detected
Installing Patch ID: 1YWL..
Result: Success

- To check the applied WLS patches:
$ ./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -view -status=applied
ProductName:       WebLogic Server
ProductVersion:    10.3 MP6
Components:        WebLogic Server/Core Application Server,WebLogic Server/Administration Console,WebLogic Server/Configuration Wizard and Upgrade Framework,WebLogic Server/Web 2.0 HTTP Pub-Sub Server,WebLogic Server/WebLogic SCA,WebLogic Server/WebLogic JDBC Drivers,WebLogic Server/Third Party JDBC Drivers,WebLogic Server/WebLogic Server Clients,WebLogic Server/WebLogic Web Server Plugins,WebLogic Server/UDDI and Xquery Support,WebLogic Server/Evaluation Database,WebLogic Server/Workshop Code Completion Support

BEAHome:            /u02/app/$TWO_TASK/fs1/FMW_Home
ProductHome:       /u02/app/$TWO_TASK/fs1/FMW_Home/wlserver_10.3
PatchSystemDir:   /u02/app/$TWO_TASK/fs1/FMW_Home/utils/bsu
PatchDir:              /u02/app/$TWO_TASK/fs1/FMW_Home/patch_wls1036
Profile:                 Default
DownloadDir:      /u02/app/$TWO_TASK/fs1/FMW_Home/utils/bsu/cache_dir
JavaVersion:       1.6.0_29
JavaVendor:        Sun

Patch ID:          32I2
Patch ID:          S9MN (14272383)
Patch ID:          TTEM (14621810)
Patch ID:          CIH8 (17431995,17495356)
Patch ID:          7FC9 (17893334)
Patch ID:          S8H6 (20474010)
Patch ID:          YVDZ (13964737)
Patch ID:          1YWL (32052267)


- To determine the application of WebLogic Server PSU
$ . $FMW_HOME/wlserver_10.3/server/bin/setWLSEnv.sh
$ java weblogic.version
WebLogic Server 10.3.6.0.210119 PSU Patch for BUG32052267 Mon Nov 23 07:28:31 UTC 2020
WebLogic Server Temporary Patch for BUG13964737 Fri Dec 20 11:32:08 IST 2013
WebLogic Server Temporary Patch for BUG20474010 Sun Mar 01 17:22:18 IST 2015
WebLogic Server Temporary Patch for BUG17495356 Tue Sep 24 00:23:38 CDT 2013
WebLogic Server Temporary Patch for BUG17495356 Tue Sep 24 00:23:38 CDT 2013
WebLogic Server Temporary Patch for BUG17495356 Tue Sep 24 00:23:38 CDT 2013
WebLogic Server Temporary Patch for BUG17495356 Tue Sep 24 00:23:38 CDT 2013
WebLogic Server Temporary Patch for BUG17495356 Tue Sep 24 00:23:38 CDT 2013
WebLogic Server Temporary Patch for ${CRS} Mon Jul 30 16:45:20 EDT 2012
WebLogic Server Temporary Patch for ${CRS} Mon Jul 30 16:45:20 EDT 2012
WebLogic Server 10.3.6.0  Tue Nov 15 08:52:36 PST 2011 1441050
Use 'weblogic.version -verbose' to get subsystem information
Use 'weblogic.utils.Versions' to get version information for all modules

- To re-run ETCC
$ ./checkMTpatch.sh

Using context file from currently set applications environment:
/u02/app/$TWO_TASK/fs1/inst/apps/<CONTEXT_NAME>/appl/admin/<CONTEXT_NAME>.xml
Starting Application Tier Technology Codelevel Checker
Version: 120.0.12020000.42.
Tue Jan 26 17:35:43 EST 2021
Log file for this session: /u02/app/temp/log/checkMTpatch_463.log
Bugfix XML file version: 120.0.12020000.48
This file will be used for identifying missing bugfixes.
Mapping XML file version: 120.0.12020000.34
This file will be used for mapping bugfixes to patches.
Checking for prerequisite bugfixes in File Edition: run
Connecting to database.
Database connection successful.
The installed AD.B.8 codelevel does not support storing the results in the database.
======================================
Oracle Forms and Reports
======================================
Now examining product Oracle Forms and Reports.
Oracle Home = /u02/app/$TWO_TASK/fs1/EBSapps/10.1.2.
Product version = 10.1.2.3.0.
Checking required bugfixes for Oracle Forms and Reports 10.1.2.3.0.
All required bugfixes are present for Oracle Forms and Reports.
 DST ts home /u02/app/$TWO_TASK/fs1/EBSapps/10.1.2.
Installed DST version under $ORACLE_HOME/oracore/zoneinfo is DST 18.
Checking required bugfixes for RSF within Forms 10.1.0.5.0.
All required bugfixes are present for RSF within Forms.
======================================
Oracle Fusion Middleware (FMW) - Web Tier
======================================
Now examining product Oracle Fusion Middleware (FMW) - Web Tier.
Oracle Home = /u02/app/$TWO_TASK/fs1/FMW_Home/webtier.
Product Version = 11.1.1.9.0
Checking required bugfixes for FMW - Web Tier 11.1.1.9.0.
All required bugfixes are present for FMW - Web Tier.
 DST ts home /u02/app/$TWO_TASK/fs1/FMW_Home/webtier.
Installed DST version under $ORACLE_HOME/oracore/zoneinfo is DST 18.
Checking required bugfixes for RSF within FMW Web tier 11.1.0.7.0.
All required bugfixes are present for RSF within FMW Web tier.
========================================
Oracle Fusion Middleware (FMW) - oracle_common
========================================
Now examining product Oracle Fusion Middleware (FMW) - oracle_common.
Oracle Home = /u02/app/$TWO_TASK/fs1/FMW_Home/oracle_common.
Product Version = 11.1.1.9.0
Checking required bugfixes for FMW - oracle common 11.1.1.9.0.
All required bugfixes are present for FMW - oracle common.
=======================================
Oracle WebLogic Server (WLS)
=======================================
Now examining product Oracle WebLogic Server (WLS).
Oracle Home = /u02/app/$TWO_TASK/fs1/FMW_Home/wlserver_10.3.
Product Version = 10.3.6.0.210119
Unable to extract list of patches from Oracle WebLogic Server (WLS) for version 10.3.6.0.210119.
Verify XML file is correct and retry.

Seems WLS PSU patch 32052267 is too new for current version of ETCC to recognize it! 

6. Notes in March 2021

After ran a newer checkMTpatch.sh of patch 17537119, it says 3 more patches are missed. So, I applied 3 more WLS patches.

$ strings -a  checkMTpatch.sh |grep '$Header'
# $Header: checkMTpatch.sh 120.0.12020000.43 2021/02/11 20:32:12 chrhill noship $
$ ./checkMTpatch.sh
… … 
===============================================
Oracle WebLogic Server (WLS)
===============================================
Now examining product Oracle WebLogic Server (WLS).
Oracle Home = $FMW_Home/wlserver_10.3.
Product Version = 10.3.6.0.210119
Note that for Oracle WebLogic Server, patches rather than bugfixes are verified.
Checking required patches for Oracle WebLogic Server (WLS) 10.3.6.0.210119.
  Missing Patch ID: 13845626
  Missing Patch ID: 22323006
  Missing Patch ID: 26795917
The above list shows missing patches for Oracle WebLogic Server.


Download & unzip 3 patch files to $FMW_HOME/utils/bsu/cache_dir folder and then applied them.
p13845626_10360210119_Generic.zip (HJT5.jar)
p22323006_1036_Generic.zip (DI8E.jar) 
p26795917_1036_Generic.zip (DN1F.jar)

$ echo $ORACLE_HOME
/u02/app/$TWO_TASK/fs1/EBSapps/10.1.2
$ echo $FILE_EDITION
run
$ cd $FMW_HOME/utils/bsu
$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=HJT5 -prod_dir=$FMW_HOME/wlserver_10.3
$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=DI8E -prod_dir=$FMW_HOME/wlserver_10.3
$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=DN1F -prod_dir=$FMW_HOME/wlserver_10.3
Checking for conflicts.............
Conflict(s) detected - resolve conflict condition and execute patch installation again
Conflict condition details follow:
Patch DN1F is mutually exclusive and cannot coexist with patch(es): CIH8

$ ./bsu.sh -prod_dir=$FMW_HOME/wlserver_10.3 -remove -verbose -patchlist=CIH8
$ ./bsu.sh -install -patch_download_dir=$FMW_HOME/utils/bsu/cache_dir -patchlist=DN1F -prod_dir=$FMW_HOME/wlserver_10.3

After all 3 WLS patches are applied successfully, no more patches is needed by ETCC:
$ ./checkMTpatch.sh
… …
================================
Oracle WebLogic Server (WLS)
================================
Now examining product Oracle WebLogic Server (WLS).
Oracle Home = $FMW_HOME/wlserver_10.3.
Product Version = 10.3.6.0.210119
Note that for Oracle WebLogic Server, patches rather than bugfixes are verified.
Checking required patches for Oracle WebLogic Server (WLS) 10.3.6.0.210119.
All required patches are present for Oracle WebLogic Server (WLS).


$ . $FMW_HOME/wlserver_10.3/server/bin/setWLSEnv.sh
$  java weblogic.version
WebLogic Server Temporary Patch for BUG13845626 Tue Jan 05 07:49:28 UTC 2021
WebLogic Server 10.3.6.0.210119 PSU Patch for BUG32052267 Mon Nov 23 07:28:31 UTC 2020
WebLogic Server Temporary Patch for BUG13964737 Fri Dec 20 11:32:08 IST 2013
WebLogic Server Temporary Patch for BUG20474010 Sun Mar 01 17:22:18 IST 2015
WebLogic Server Temporary Patch for ${CRS} Mon Jul 30 16:45:20 EDT 2012
WebLogic Server Temporary Patch for ${CRS} Mon Jul 30 16:45:20 EDT 2012
WebLogic Server 10.3.6.0  Tue Nov 15 08:52:36 PST 2011 1441050
Use 'weblogic.version -verbose' to get subsystem information
Use 'weblogic.utils.Versions' to get version information for all modules

NOTE: in this stage, under the new R12.2 environment, adadmin works and $ADMIN_SCRIPTS_HOME will point to R12.2 scripts (but do not run any of them).

----------------
Oracle E-Business Suite 12.2 Patching Technology Components Guide (Doc ID 1355068.1)
R12.2 Pre-install patches report (Doc ID 1448102.2) ?