Tuesday, April 25, 2023

Upgrade AD and TXK from Delta 12 to Delta 14 RUPs in R12.2

AD.C.Delta.14 (patch 33600809) and TXK.C.Delta.14 (patch 33602997) are pre-requisite for January 2023 CPU patches (see Doc ID 2916871.1). We have to apply them before that CPU patch set can be applied to R12.2 instance.

Oracle document ID 1617461.1 (Applying the Latest AD and TXK Release Update Packs to Oracle E-Business Suite Release 12.2) gives details on how to apply the latest Oracle Applications DBA (AD) and Oracle E-Business Suite Technology Stack (TXK) release update packs (RUPs) to EBS R12.2. I performed below steps to upgrade them to delta 14 from delta 12 in my instances where existing package levels are:
R12.AD.C.Delta.12
R12.TXK.C.Delta.12
R12.ATG_PF.C.delta.9 (30399994)

1. DBA runs the latest version of ETCC (via Patch 17537119), and ensure all required database objects by the Delta 14 RUP installation process will be found. One of recent database PSU patches will help to achieve this.

The following optimizer parameter should always be set to a value of TRUE:
_disable_actualization_for_grant=true

2. Pre-step: download all .zip files and unzip them to $PATCH_TOP (or an alternation folder) of all nodes. Then, DBA runs $PATCH_TOP/34669333/admin/adgrants.sql as SYSDBA

AD critical patch 34669333 has the latest adgrants.sql, as of now. Please read the instruction in the file before run it.

$ grep Header $PATCH_TOP/34669333/admin/adgrants.sql
REM $Header: adgrants.sql 120.67.12020000.123 2022/09/30 21:13:17 jwsmith ship $

3. Verify new db account ebs_system. 
adgrants.sql creates new database account ebs_system. Unlock it and change its password to be the same as SYSTEM's password (if it has not been done yet). 
$ sqlplus system/systemPWD
SQL> alter user ebs_system account unlock;
SQL> alter user ebs_system identified by systemPWD;
SQL> conn ebs_system/systemPWD

4. Run the validation script
$ perl $PATCH_TOP/33600809/ad/bin/adValidateEbssystemSchema.pl

Enter the APPS password:
Enter the SYSTEM password:
Validating APPS credentials...validated successfully
Validating SYSTEM credentials...validated successfully
Validating EBS_SYSTEM user
ebssys entity is not registered
All required validations completed successfully

5. Optional: Validate adop, and stop all services (if use downtime mode to apply the patch)
$ adop -validate
$ adop -status

$ cd $ADMIN_SCRIPTS_HOME
$ ./adadminsrvctl.sh stop
$ ./adnodemgrctl.sh stop

$ ps -ef | grep $LOGNAME

6. Apply AD.C.Delta.14 patch, plus 3 critical patches, by running below line on primary mode. I used downtime mode to apply them.

$ adop phase=apply apply_mode=downtime patches=33600809,34668508,34669333,34681299 merge=yes patchtop=$PATCH_TOP

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

If patching gets error, most likely there is a problem with something else. Confirm 4 patches were applied to all nodes:
SQL> SELECT adb.bug_number, aas.name appl_top_name, adb.creation_date, adb.language,
decode(ad_patch.is_patch_applied('R12',aas.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status
FROM ad_bugs adb,
(select distinct appltop_id appl_top_id, node_name name from ad_adop_sessions 
           where node_name in (select node_name from ADOP_VALID_NODES) ) aas
where adb.bug_number in (
'33600809',
'34668508',
'34669333',
'34681299'
) order by creation_date desc, adb.bug_number,aas.name,adb.language;

UPDATE in July 2023: patch 35280947 was added as a AD critical patch in the document. When they were applied to an instance with one single node, the adop log is

$ adop phase=apply apply_mode=downtime patches=33600809,34668508,34669333,34681299,35280947 merge=yes patchtop=$PATCH_TOP
... ...
Copying files...
5% complete. Copied 26 files of 519...
10% complete. Copied 52 files of 519...
15% complete. Copied 78 files of 519...
20% complete. Copied 104 files of 519...
25% complete. Copied 130 files of 519...
30% complete. Copied 156 files of 519...
35% complete. Copied 182 files of 519...
40% complete. Copied 208 files of 519...
45% complete. Copied 234 files of 519...
50% complete. Copied 260 files of 519...
55% complete. Copied 286 files of 519...
60% complete. Copied 312 files of 519...
65% complete. Copied 338 files of 519...
70% complete. Copied 364 files of 519...
75% complete. Copied 390 files of 519...
80% complete. Copied 416 files of 519...
85% complete. Copied 442 files of 519...
90% complete. Copied 468 files of 519...
95% complete. Copied 494 files of 519...
100% complete. Copied 519 files of 519...

Character-set converting files...
  5 unified drivers merged.
Patch merge completed successfully

Please check the log file at $ADOP_LOG_HOME/8/20230804_163029/apply/$EBS_HOSTNAME/admrgpch.log.

Applying patch ADOP_MRG_20230804_1691181079:u_ad_3366500462.drv.
    Log: $ADOP_LOG_HOME/8/20230804_163029/apply/$EBS_HOSTNAME/ADOP_MRG_20230804_1691181079/log/u_ad_3366500462.log

Running finalize actions for the patches being applied.
    Log: @ADZDSHOWLOG.sql "2023/08/04 16:36:47"

Running cutover actions for the patches being applied.
    Creating workers to process cutover DDL in parallel
    Log: $ADOP_LOG_HOME/8/20230804_163029/apply/$EBS_HOSTNAME/log/cutover.log
    Performing database cutover in Quick mode

Generating post apply reports.

Generating log report.
    Output: $ADOP_LOG_HOME/8/20230804_163029/apply/$EBS_HOSTNAME/adzdshowlog.out

The apply phase completed successfully.
adop exiting with status = 0 (Success)

7. Apply TXK.C.Delta.14 patch, plus 2 critical patches, by below line on primary mode. Note it now asks for EBS_SYSTEM password.

$ adop phase=apply apply_mode=downtime patches=33602997,34708635,34654260 merge=yes patchtop=$PATCH_TOP

Enter the APPS password:
Enter the EBS_SYSTEM password:
Enter the WLSADMIN password:

Confirm 3 patches were applied successfully:
SQL> SELECT adb.bug_number, aas.name appl_top_name, adb.creation_date, adb.language,
decode(ad_patch.is_patch_applied('R12',aas.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status
FROM ad_bugs adb,
(select distinct appltop_id appl_top_id, node_name name from ad_adop_sessions 
           where node_name in (select node_name from ADOP_VALID_NODES) ) aas
where adb.bug_number in (
'33602997',
'34708635',
'34654260'
) order by creation_date desc, adb.bug_number,aas.name,adb.language;

8. Post steps
$ perl $AD_TOP/bin/admkappsutil.pl
$ cp -p $INST_TOP/admin/out/appsutil.zip $APPLPTMP

$ Autoconfig on db node
$ Autoconfig on all apps nodes

9. Start Apps services
$ ./adstrtal.sh apps/aapsPWD -mode=allnodes

10. After the instance was verified and checked out, run FS_CLONE to sync the file systems.

NOTES: adgrants.sql from patch 34669333 makes important changes in database. During above patching, it is copied to $APPL_TOP/admin as the current one. Below is its log by SYSDBA: 

Current user is SYS
------------------------------------------------------------
--- adgrants.sql started at 2023-04-20 11:15:11 ---
------------------------------------------------------------
Removing logs from prior runs of adgrants.sql
-
Start granting from SYS to EBS_SYSTEM
-
End granting from SYS to EBS_SYSTEM
Completed granting and checking privileges
Generating list of ERRORS and WARNINGS to print out
PRINT_ERROR
---------------------
:current_user

CURRENT_USER
-----------------------------------------------------------------------------------------
SYS user ERRORS and WARNINGS will report at end of script
to print out
CURRENT_USER
-----------------------------------------------------------------------------------------
SYS user ERRORS and WARNINGS will report at end of script
Creating PL/SQL profiler objects.
---------------------------------------------------------------
--- profload.sql started at 2023-04-25 11:15:14 ---

In 12.2c and beyond, the Oracle-supplied profload.sql script is a verification script, not an installation script.
Testing for correct installation
SYS.DBMS_PROFILER successfully loaded.
-----------------------------------------------------
--- profload.sql completed at 2023-04-25 11:15:16 ---
--------------------------------------------------
--- proftab.sql started at 2023-04-25 11:15:16 ---
-----------------------------------------------------
--- profltab.sql completed at 2023-04-25 11:15:18 ---

Installing Hierarchical Profiler.
-
Loading Stylesheets if missing
Begin creating the AD_ZD_SYS package
End creating the AD_ZD_SYS package
Executing PURGE DBA_RECYCLEBIN.
The following ERRORS and WARNINGS have been encountered during this adgrants session:
Grants given by this script have been written to the ad_zd_logs table.
You can run $AD_TOP/sql/ADZDSHOWLOG.sql to produce a report showing these grants.

Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.18.0.0.0
Started at 2023-04-25 11:15:14 ---

In 12.2c and beyond, the Oracle-supplied profload.sql script is a verification script, not an installation script.
Testing for correct installation
SYS.DBMS_PROFILER successfully loaded.
-----------------------------------------------------
--- profload.sql completed at 2023-04-25 11:15:16 ---
--------------------------------------------------
--- proftab.sql started at 2023-04-25 11:15:16 ---
-----------------------------------------------------
--- profltab.sql completed at 2023-04-25 11:15:18 ---

Installing Hierarchical Profiler.

HPTAB_SCRIPT
---------------------------
?/rdbms/admin/nothing.sql
-
Loading Stylesheets if missing
PACKAGE_NAME
------------------------
AD_ZD_SYS
Begin creating the AD_ZD_SYS package
End creating the AD_ZD_SYS package
Executing PURGE DBA_RECYCLEBIN.
The following ERRORS and WARNINGS have been encountered during this adgrants session:
Grants given by this script have been written to the ad_zd_logs table.
You can run $AD_TOP/sql/ADZDSHOWLOG.sql to produce a report showing these grants.