ERP on DB: March 2021

Monday, March 22, 2021

12.2.10 patch 30399999 failed on POS_SUPPLIER_SEARCH_INDEX.sql

After all pre-steps were completed and conditions met the requirement, I downloaded zip files for 12.2.10 patch 30399999 and unzipped to folder /path/to/R12210_Patches. Command line to apply it:

$ adop phase=apply apply_mode=downtime patches=30399999 patchtop=/path/to/R12210_Patches

ADOP failed and exited with error after ran for hours:

You should check the file

$NE_BASE/EBSapps/log/adop/.../apply/node_name/30399999/log/u30399999.log

for errors.

[UNEXPECTED]Error occurred executing "adpatch workers=16 options=hotpatch console=no interactive=no defaultsfile=/path/to/R12210_Patches//30399999 driver=u30399999.drv logfile=u30399999.log"

[UNEXPECTED]Refer to the log files for more information.

[UNEXPECTED]Apply phase has failed.

[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.

adop exiting with status = 1 (Fail)

Check adwork001.log file and see the real error:

sqlplus -s APPS/***** @$AD_TOP/patch/115/sql/adsqlwrapper.sql '$POS_TOP/patch/115/sql/POS_SUPPLIER_SEARCH_INDEX.sql '

Connected.

PL/SQL procedure successfully completed.

DECLARE

ERROR at line 1:

ORA-20000: possearchindex.sql(500): ORA-20000: Exception at

POS_SUPPLIER_SEARCH_INDEX_PKG.create_index(1800): ORA-20000: Exception at

POS_SUPPLIER_SEARCH_INDEX_PKG.create_index(1800): ORA-29855: error occurred in

the execution of ODCIINDEXCREATE routine

ORA-20000: Oracle Text error:

DRG-10754: Size (in bytes) must be between 1024 and 52428800

ORA-06512: at line 56

Time when worker failed: Fri Sep 24 2021 13:56:21

SQL> select index_name, table_name, index_type, status, domidx_opstatus

from all_indexes where domidx_opstatus != 'VALID';

Control

Worker Code Context Filename Status

------ -------- ----------------- -------------------------- --------------

1 Done AutoPatch R120 POS_SUPPLIER_SEARCH_INDEX. FAILED

At this stage, we do not want to make any changes to the instance. The only option is to skip this SQL code and then to re-start the adop session.

Fortunately adctrl still shows the workers' status, while ADOP session already exited by the error. I choose option #8 to skip the failed worker.

After that, I ran below line to re-start from where it failed and it completed:

$ adop phase=apply apply_mode=downtime patches=30399999 patchtop=/path/to/R12210_Patches restart=yes

$ vi u30399999.log

... ...

Worker process 15 started.

Worker process 16 started.

Checking if all jobs have their actual and symbolic arguments in sync....

Done.

Reading jobs from FND_INSTALL_PROCESSES table ...

Fixed: file POS_SUPPLIER_SEARCH_INDEX.sql on worker 1 for product pos username POS.

Time is: Fri Sep 24 2021 16:22:18

Done reading jobs from FND_INSTALL_PROCESSES table ...

Telling workers to read 'todo' restart file.

Done.

Completed: file POS_SUPPLIER_SEARCH_INDEX.sql on worker 1 for product pos username POS.

Time is: Fri Sep 24 2021 16:22:18

Connecting to APPS......Connected successfully.

Even u30399999.log says file POS_SUPPLIER_SEARCH_INDEX.sql was completed. But the worker log shows it was skipped. Index table still shows a failure as well.

$ vi adwork001.log

... ...

Ready to run jobs.

Setting parallel context: PARALLEL AUTOPATCH AT R120

Restarting job that failed - will skip a single step.

Time when worker restarted job: Fri Sep 24 2021 16:22:18

Skipping step POS_SUPPLIER_SEARCH_INDEX.sql.

Step skipped at time: Fri Sep 24 2021 16:22:18

Time when worker completed job: Fri Sep 24 2021 16:22:18

SQL> select index_name, table_name, index_type, status, domidx_opstatus

from all_indexes where domidx_opstatus != 'VALID';

INDEX_NAME TABLE_NAME INDEX_TYPE STATUS DOMIDX_OPSTATUS

--------------------------------------------- -------------------------------------------- ------------------ ------------ ----------------------------

POS_SUPPLIER_SEARCH_INDEX POS_SUPPLIER_ENTITY_DATA DOMAIN VALID FAILED

I believe our instance does not use this index and just ignored it. I do not know exactly what causes the failure on creating the index. I believe it relates to the 0 value on two database init parameters:

SQL> show parameter PGA_AGGREGATE_TARGET;

NAME TYPE VALUE

----------------------------- ----------- ------------------------------

pga_aggregate_target big integer 0 (Note: 0 means unlimited )

SQL> show parameter SGA_TARGET;

NAME TYPE VALUE

------------------------------ ----------- ------------------------------

sga_target big integer 0

Sunday, March 21, 2021

Run ADSplice in R12.2

ADSplice is the only way to create custom tops in EBS R12.2. Steps of running it is in Doc ID 1577707.1 (Creating a Custom Application in Oracle E-Business Suite Release 12.2). If it is a multi-node instance, you have to run it on each node.

1. Before running ADSplice, make sure latest AD and TXK for R12.2 patches were applied already:

SQL> select abbreviation ,codelevel from ad_trackable_entities

where abbreviation in( 'ad','txk') order by abbreviation;

ABBREVIATION CODELEVEL

--------------------- ---------

ad C.12

txk C.12

2. For a regular/normal custom top, it is not difficult to follow the Oracle document for creating or modifying it. Three files are required by ADSplice in folder $APPL_TOP/admin for one custom top $XXFS_TOP, for example

xxefprod.txt
xxefterr.txt
newprods.txt

3. Then run adsplice. It will ask for SYSTEM and APPS password. It also give option to run AutoConfig. So, it is better to shut down apps services before running adsplice.

$ echo $FILE_EDITION

run

$ cd $APPL_TOP/admin

$ adsplice

ADSplice will create folder structure for $XXFS_TOP under $APPL_TOP.

For multiple nodes, run adsplice on all nodes. After 1st node, adsplice runs much faster because it knows all database part was completed.

4. Most custom top has custom codes/files. Migrate custom codes/files to the correct new folder, and then compile them or link them to right executable files.

5. After ADSplice worked, add entries for all your custom files to the custom synchronization driver file located at $APPL_TOP_NE/ad/custom/adop_sync.drv. I used below line which works well in copying folder xxfs in <s_current_base> (source) to <s_other_base> (target)

rsync -a --delete %s_current_base%/EBSapps/appl/xxfs %s_other_base%/EBSapps/appl

The option -a is very powerful, it will copy the directory and all its sub-directories recursively, and preserve symbolic links, modification times, group, ownership, and permissions. It is a combination of a bunch of other options:

-r Recurse into directories

-l Copy symlinks as symlinks

-p Preserve file permissions

-t Preserve modification times

-g Preserve group

-o Preserve owner

-D Preserve device and special files

-- delete option will delete files and sub-directories in target folder but not in source folder.

6. Run FS_CLONE

ADSplice creates folders only in RUN Edition file system. You must run "adop phase=fs_clone". This ensures that all custom products are synchronized/copied to PATCH file system. Doc ID 2167878.1 indicates FS_CLONE does use adop_sync.drv.

I ran into a error with "adop phase=prepare". ADOP does not tell what is the specific problem at all!

=============================

Inside evalADPATCHStatus()...

=============================

message_status: ERROR

Adsplice action did not go through successfully.

*******FATAL ERROR*******

PROGRAM : ($AD_TOP/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)

TIME : Wed Jul 20 14:00:36 2022

FUNCTION: main::execADSPLICE [ Level 1 ]

ERRORMSG: Adsplice action did not go through successfully.

Before the error, log file txkADOPPreparePhaseSynchronize.log posts a SQL statement similar to below lines to check table AD_ADOP_SESSION_PATCHES:

SELECT * FROM ad_adop_session_patches

WHERE bug_number like 'ADSPLICE%' AND

ORDER BY TO_CHAR(end_date,'YYYY.MM.DD:HH24:MI:SS') DESC

Without running fs_clone, column PATCH_FILE_SYSTEM_BASE is empty for one CUSTOM TOP. I think that causes ADOP PREPARE failed.

After "adop phase=prepare" failed, "adop phase=fs_clone" will not work. You have to below steps in order:

$ adop phase=abort

$ adop phase=cleanup cleanup_mode=full

$ adop phase=fs_clone

Notes: Similarly, anytime after a change was made in RUN file system, e.g. a patch applied in HOTPATCH mode, you have to run "adop phase=fs_clone" to sync the change to PATCH file system.

7. A useful query to verify if a new custom top in R12.2 was created by ADSplice

SQL> set linesize 300

set pagesize 300

col APPL_ID format 99999;

col app_short_name format a15

col basepath format a 15

col basepath format a15

col application_name format a35

col inst_status format a10

col product_version a20

col product_version format a20

col patchset format a25

col oracle_id format 99999

SQL> SELECT fav.application_id APPL_ID, fav.application_short_name app_short_name, fav.basepath, fav.application_name,

decode(fpi.status,'I','Installed','S','Shared','N','No', fpi.status) inst_status, fpi.product_version,

nvl(fpi.patch_level, 'Not Available') patchset, fpi.oracle_id

FROM apps.fnd_application_vl fav, apps.fnd_product_installations fpi

WHERE fav.application_id = fpi.application_id

and (fav.application_short_name like 'XX%' )

ORDER BY app_short_name asc;

- TROUBLESHOOTING and lessons learned on ADSplice:

1. We had a custom top that was created in 9i with mismatch between SHORT_NAME and BASEPATH (the name of custom top):

SQL> select ORACLE_USERNAME,ORACLE_ID, dba.USERNAME

from fnd_oracle_userid fnd, dba_users dba

where fnd.oracle_username = dba.username and lower(oracle_username) in ('xxfs') ;

ORACLE_USERNAME ORACLE_ID USERNAME

------------------------------ ---------------- ----------------

XXFS 20024 XXFS

SQL> select application_id,application_short_name short_name,

last_update_date, basepath, product_code

from fnd_application

where application_short_name like '%XXFS%' or application_id = 20024;

APPLICATION_ID SHORT_NAME LAST_UPDATE BASEPATH PRODUCT_CODE

------------------------ -------------------- -------------------- --------------- ------------------------

20024 XXFFS 16-DEC-97 XXFS_TOP XXFFS

We tried to keep it as the old way to minimize the re-write of custom codes, by putting below lines in file xxefprod.txt

# Product specific Product ID's

xxfs 20024

END_OF_PRODUCT_ABBREVIATIONS -999

# ##################################################

# Oracle Support Diagnostic Tools

# ##################################################

# application id, abbreviation, shortname, prefix

20024 aefs XXFFS APP

# multiple product installations for msob, "controlled release", shared only

# optional fourth field is "stub product". default is No

No No No No

# multilingual, has NLADD.sql

Yes Yes

# install oracle id, default ORACLE username, default ORACLE password

20024 XXFS XXFS

Note: In above line, XXFS is not necessarily the current db password. If it does not exist in database, adsplice will create it.

ADSplice does not complain it and created $XXFS_TOP. But when I later applied patches, "adop phase=prepare" failed with error:

Custom product XXFS entry is NOT there in FND_APPLICATIONS table

ERROR: The following custom products have not been correctly added: XXFS

We also tried to created two custom tops $XXFFS_TOP and $XXFS_TOP (with a new application id assigned, such as 30034), while they share same existing database account XXFS. ADSplice also created two without complain. But "adop phase=prepare" failed again without giving a specific error:

message_status: ERROR

Adsplice action did not go through successfully.

*******FATAL ERROR*******

PROGRAM : ($AD_TOP/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)

TIME : Wed Xxx 5 11:52:14 2021

FUNCTION: main::execADSPLICE [ Level 1 ]

ERRORMSG: Adsplice action did not go through successfully.

I had to de-register $XXFFS_TOP first and then de-register $XXFS_TOP by using

$ perl $AD_TOP/bin/adDeregisterCustomProd.pl

After de-register them, ADOP worked in the full patching cycle.

Before that, I tried the suggestion in Doc ID 1932207.1 (Adop Prepare ERRORMSG: Adsplice Action Did Not Go Through Successfully After Add new Custom Application in R12.2). But I was not sure (do not remember) it helped or not.

SQL> UPDATE FND_ORACLE_USERID SET ENABLED_FLAG = 'N'

WHERE ORACLE_USERNAME in ('XXFS') ; -- from "(null)"

2. "perl $AD_TOP/bin/adDeregisterCustomProd.pl" will delete data from tables FND_OAM_CONTEXT_CUSTOM, FNS_APPLICATION. But it will not delete the row in table FND_PRODUCT_INSTALLATION.

3. If the password of Oracle database account XXEF was changed not by FNDCPASS, ADSplice will give ORA-01017 error. The fix is to use below line to change its password:

$ FNDCPASS apps/apps_PWD 0 Y system/'systemPWD' ORACLE xxef xxef_PWD

4. "adop phase=abort,cleanup cleanup_mode=full" failed with error message:

AutoPatch - aidafoGetFileDbVersion: INFO: ORA-01403: no data found

AutoPatch error:

Unable to get APPS_DDL package version from db <XXFS> <APPS_DDL> <PACKAGE>

AutoPatch error:

adpmrp: Error while installing apps_ddl packages.

Run below query to see its status. The fix is to run an UPDATE on the row.

SQL> select oracle_username, read_only_flag from fnd_oracle_userid

where read_only_flag <> 'A';

ORACLE_USERNAME READ_ONLY_FLAG

------------------------------ ---------------------------

APPLSYS E

APPS U

XXFS D

APPLSYSPUB C

APPS_NE Z

SQL> update fnd_oracle_userid set read_only_flag='A'

where ORACLE_USERNAME='XXFS';

Then "adop phase=abort" worked. (Note: In my another instance, it has the same status, but ADOP does not fail).

5. After a custom top is created, if it is used by a Responsibility or concurrent jobs, run autoconfig and bounce apps services to avoid error/message:

APP-FND-00362: Routine afpbep cannot execute request &REQUEST for program &PROGRAM, because the environment variable &BASEPATH is not set for the application to which the concurrent program executable &EXECUTABLE belongs.

Shut down the concurrent managers. Set the basepath environment variable for the application. Restart the concurrent managers. (VARIABLE=)

Routine AFPEOT cannot construct the name of an executable file for your concurrent request 430346.

Check that the file name components are correct and valid on your system. Check that the environment for the person who started the concurrent manager

6. R12.2 GUI forms will not create a custom top, because navigation Application => Register gives message:

Forms Registration for Custom Application or Schema is now blocked. Please consult ADSplice documentation for the current standards for deploying Customizations in EBS

Doc ID 2272551.1 says this is intended functionality. Application Forms cannot be used to add or remove applications. This is restricted in 12.2 and going forward.

7. If a custom top is not necessary but an env variable is needed as a pointer to a folder under $APPL_TOP, modify file $FND_TOP/fndenv.env by adding lines:

#Begin Customizations

AEUT_TOP=${APPL_TOP}/xxedi/12.0.0; export XXEDI_TOP

#End Customizations

With the use of adsplice, I am not sure if modifying Forms file is necessary in R12.2

$INST_TOP/ora/10.1.2/forms/server/default.env:

#Begin Customizations

XXEDI_TOP=/path/to/appl/xxedi/12.0.0

#End Customizations

Friday, March 19, 2021

Use adop to apply AD & TXK patches, and R12.2.10 patch

Steps in Doc ID 1617461.1 (Applying the Latest AD and TXK Release Update Packs to Oracle E-Business Suite Release 12.2) using ADOP and to complete R12.2.10 upgrade.

1). Checked database parameters for R12.2 (Doc ID: 396009.1) in 12c database:

SQL> select SERVICE_ID,NAME from dba_services where name='ebs_patch';

SERVICE_ID NAME

---------- ------------------------

8 ebs_patch

SQL> show parameter recyclebin

NAME TYPE VALUE

-------------------------------- ----------- ------------

recyclebin string OFF

SQL> show parameter _SYSTEM_TRIG_ENABLED

NAME TYPE VALUE

----------------------------- ----------- ------------

_system_trig_enabled boolean TRUE

SQL> show parameter optimizer_adaptive_features

NAME TYPE VALUE

------------------------------------ ----------- --------------

optimizer_adaptive_features boolean FALSE

SQL> show parameter _disable_actualization_for_grant

NAME TYPE VALUE

------------------------------------------ ----------- ----------------

_disable_actualization_for_grant boolean TRUE

SQL> select fnd_profile.value('APPS_MAINTENANCE_MODE') from dual;

FND_PROFILE.VALUE('APPS_MAINTENANCE_MODE')

--------------------------------------------------------------------------

NORMAL

SQL> set lines 333 pages 123;

col ABBREVIATION for a12;

col NAME for a45;

col codelevel for a12;

SQL> select ABBREVIATION, NAME, codelevel FROM AD_TRACKABLE_ENTITIES

where abbreviation in ('txk','ad');

ABBREVIATION NAME CODELEVEL

------------ --------------------------------------------- ------------

ad Applications DBA C

txk Oracle Applications Technology Stack C

SQL> select node_name from apps.fnd_nodes;

NODE_NAME

------------------------------

db_nodename

AUTHENTICATION

apps_nodename

SQL> select node_name from apps.TXK_TCC_RESULTS;

NODE_NAME

------------------------------------------------------------------------

db_nodename.domain_name.com

<== Should have db_nodename only

<== Not sure, but July 2020 db PSU 31113348 may fix the issue

$ grep s_dbhost $CONTEXT_FILE

<dbhost oa_var="s_dbhost">db_nodename</dbhost> <== mis-match

2). run checkDBpatch.sh

3). run checkMTpatch.sh

$ . $APPL_TOP/APPS$CONTEXT_NAME.env <= Source the env file

$ echo $FILE_EDITION

run

$ echo $RUN_BASE

/u02/app/$TWO_TASK/fs1

$ ./checkMTpatch.sh

Notes: without 2) and 3), later ADOP may report error even two scripts were previously executed:

Validating system setup.

Node registry is valid.

[ERROR] ETCC not run in the database node db_nodename

The EBS Technology Codelevel Checker needs to be run on the database node.

It is available as Patch 17537119.

[WARNING] MTCC has not been run. Apply Patch 17537119 and run checkMTpatch.sh.

Encountered the above errors when performing database validations.

Resolve the above errors and restart adop.

If the issue is persistent, check a solution in Doc ID 2191678.1

SQL> select node_name from apps.TXK_TCC_RESULTS;

4). Start WebLogic Admin Server

$ sh $ADMIN_SCRIPTS_HOME/adadminsrvctl.sh start

NMProcess: <Mn 7, 2021 4:47:14 PM> <INFO> <Loading domains file: $FMW_Home/wlserver_10.3/common/nodemanager/nmHome1/nodemanager.domains>

NMProcess: Mn 07, 2021 4:47:14 PM weblogic.nodemanager.server.NMServerConfig initDomainsMap

NMProcess: INFO: Loading domains file: $FMW_Home/wlserver_10.3/common/nodemanager/nmHome1/nodemanager.domains

NMProcess: <Mn 7, 2021 4:47:14 PM> <INFO> <Loaded node manager configuration properties from '$FMW_Home/wlserver_10.3/common/nodemanager/nmHome1/nodemanager.properties'>

NMProcess: Mn 07, 2021 4:47:14 PM weblogic.nodemanager.server.NMServer <init>

NMProcess: INFO: Loaded node manager configuration properties from '$FMW_Home/wlserver_10.3/common/nodemanager/nmHome1/nodemanager.properties'

NMProcess: Node manager v10.3

NMProcess:

NMProcess: Configuration settings:

... ...

adadminsrvctl.sh: exiting with status 0

adadminsrvctl.sh: check the logfile $LOG_HOME/appl/admin/log/adadminsrvctl.txt for more information ...

$ grep wls_admin $CONTEXT_FILE

5). Test WebLogic console works by login as "weblogic".

http://server_name.domain.com:s_wls_adminport/console

(After logged in, see "Warning: This server is not secure. Upgrade to Java 1.7.0_191 or higher.")

Below line shall also work:

$ wget http://server_name.domain.com:s_wls_adminport/console

… …

100%[===============================>] 3,162 --.-K/s in 0s

2021-Mn-12 10:25:54 (392 MB/s) - ‘console’ saved [3162/3162]

$ ps -ef | grep -i fs1 | wc -l

shall be 5 new sessions

Notes: at this moment, environment variables PATCH_TOP and NE_BASE are not defined yet, and ADOP status gives error because it has no history yet:

$ adop -status

Enter the APPS username: apps

Enter the APPS password:

where adop_session_id=;

ERROR at line 6:

ORA-06550: line 6, column 23:

PL/SQL: ORA-00936: missing expression

ORA-06550: line 4, column 1:

PL/SQL: SQL Statement ignored

adop exiting with status = 1 (Fail)

6). Download/copy patches' .zip files to

$ cd /path/to/$TWO_TASK/fs_ne/EBSapps/patch

$ ls

$ unzip p30628681_R12.AD.C_R12_LINUX.zip

7). Run 30628681/admin/adgrants.sql as SYSDBA.

(confirmed the same version was executed before and so skipped this step)

8). Apply patch 30628681 (Do not merge it with any other patches)

$ echo $RUN_BASE

/u02/app/$TWO_TASK/fs1

$ echo $FILE_EDITION

run

$ pwd

/path/to/$TWO_TASK/fs_ne/EBSapps/patch

$ adop phase=apply patches=30628681 hotpatch=yes patchtop=/path/to/$TWO_TASK/fs_ne/EBSapps/patch

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

Please wait. Validating credentials...

Enter the RUN file system context file name [$RUN_BASE/inst/apps/$CONTEXT_NAME/appl/admin/$CONTEXT_NAME.xml]:

[STATEMENT] [END 2021/0X/12 11:53:01] Performing verification of parameters

[STATEMENT] [START 2021/0X/12 11:53:01] Checking for the required ENV setup

[STATEMENT] [END 2021/0X/12 11:53:01] Checking for the required ENV setup

************* Start of session *************

version: 12.2.0

started at: Fri XXX 12 2021 11:53:02

… …

[STATEMENT] ADPATCH Log directory: $NE_BASE/EBSapps/log/adop/2/apply_20210X12_115025/$CONTEXT_NAME/30628681/log

[STATEMENT] Running: adpatch workers=24 options=hotpatch flags=autoskip console=no interactive=no defaultsfile=$APPL_TOP/admin/$TWO_TASK/adalldefaults.txt patchtop=$NE_BASE/EBSapps/patch/30628681 driver=u30628681.drv logfile=u30628681.log stdin=yes

Successfully created out directory <== stay here for about 10 minutes

If it takes hours, something was wrong.

[STATEMENT] [START 2021/09/14 13:32:57] Running finalize since in hotpatch mode

[STATEMENT] [END 2021/09/14 13:36:22] Running finalize since in hotpatch mode

[STATEMENT] [START 2021/09/14 13:36:23] Performing Automatic DB cutover

… …

[STATEMENT] adop phase=apply - Completed Successfully

[STATEMENT] Log file: $NE_BASE/EBSapps/log/adop/2/adop_20210X12_115025.log

adop exiting with status = 0 (Success)

Check logs to confirm it had no issue. It takes about 15 minutes.

Notes: Found ADOP does not like SYSTEM password has special char "!" at the end, with error:

ERROR: Invalid system user credentials.

Warning: Your system may have a limit on the number of failed login attempts before the account is locked out.

9). Use hotpatch mode to apply Patch 30735865 (R12.TXK.C.Delta.12) plus any additional critical patches (which is patch 31904550:R12.TXK.C for addressing fs_clone issue. See Section 4.2).

$ unzip p30735865_R12.TXK.C_R12_GENERIC.zip

$ unzip p31904550_R12.TXK.C_R12_GENERIC.zip

$ echo $RUN_BASE

/u02/app/$TWO_TASK/fs1

$ adop phase=apply patches=30735865,31904550 hotpatch=yes merge=yes patchtop=/path/to/$TWO_TASK/fs_ne/EBSapps/patch

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

Validating credentials.

Initializing.

Run Edition context : $CONTEXT_FILE

Patch edition context: /path/to/fs2/inst/apps/…/$CONTEXT_NAME.xml

Patch file system free space: 89.36 GB

Validating system setup.

Node registry is valid.

[WARNING] MTCC has not been run. Apply Patch 17537119 and run checkMTpatch.sh.

Checking for existing adop sessions.

Application tier services are down.

Continuing with the existing session [Session ID: 2].

==============================================

ADOP (C.Delta.12)

Session ID: 2

Node: node_name

Phase: apply

Log: $NE_BASE/EBSapps/log/adop/2/20210X15_094233/adop.log

==============================================

Starting patch merge.

Merging patches where applicable.

Log: $NE_BASE/EBSapps/log/adop/2/20210X15_094233/apply/node_name/admrgpch.log

Applying patch ADOP_MRG_20210X15_1613400209:u_1809065289.drv.

Log: $NE_BASE/EBSapps/log/adop/2/20210X15_094233/apply/node_name/ADOP_MRG_20210X15_1613400209/log/u_1809065289.log

New product CMI spliced. Use FNDCPASS to change the password.

New product YMS spliced. Use FNDCPASS to change the password.

<= it takes about 15 minutes after this line totally

Running finalize actions for the patches being applied.

Log: @ADZDSHOWLOG.sql "2021/0X/15 09:54:40"

Running cutover actions for the patches being applied.

Creating workers to process cutover DDL in parallel

Log: $NE_BASE/EBSapps/log/adop/2/20210X15_094233/apply/node_name/log/cutover.log

Performing database cutover in Quick mode

Generating post apply reports.

Generating log report.

Output: $NE_BASE/EBSapps/log/adop/2/20210X15_094233/apply/node_name/adzdshowlog.out

The apply phase completed successfully.

adop exiting with status = 0 (Success)

$ egrep -i 'fail|error|ora-' u_xxx.log

$ egrep -i 'fail|error|ora-'

$NE_BASE/EBSapps/log/adop/2/20210X15_094233/apply/node_name/adzdshowlog.out

It reports ORA- errors on material view MRP_COMPANY_USERS_SN. This error can be ignored per Doc ID 1957323.1.

Notes: if ETCC version was old and some databases patches were not applied, ADOP will give warnings, e.g.

[WARNING] ETCC: The following required database fixes have not been applied to node db_nodename:

21322448

22331786

31404014

31550110

Refer to My Oracle Support Knowledge Document 1594274.1 for instructions.

10). Confirm all worked

$ adop -status

Enter the APPS password:

Connected.

==================================================

ADOP (C.Delta.12)

Session Id: 2

Command: status

Output: $NE_BASE/EBSapps/log/adop/2/20210X15_100245/adzdshowstatus.out

=============================================================

Node Name Node Type Phase Status Started Finished Elapsed

--------------- ---------- --------------- --------------- -------------------- -------------------- ------------

$EBS_HOSTNAME master APPLY ACTIVE 2021/0X/12 11:53:31 2021/0X/15 09:56:28 70:02:57

FINALIZE NOT STARTED

CLEANUP NOT STARTED

File System Synchronization Type: None

adop exiting with status = 0 (Success)

Use query to confirm 3 patches were applied:

SQL> select ad_patch.is_patch_applied('R12',-1,30735865) from dual;

SQL> select ad_patch.is_patch_applied('R12',-1,31904550) from dual;

SQL> select ad_patch.is_patch_applied('R12',-1,30628681) from dual;

SQL> select ADOP_SESSION_ID, BUG_NUMBER, STATUS, APPLIED_FILE_SYSTEM_BASE,

ADPATCH_OPTIONS, round((end_date-start_date)*24*60, 1) EXEC_TIME, AUTOCONFIG_STATUS, DRIVER_FILE_NAME, NODE_NAME,

END_DATE, CLONE_STATUS

from ad_adop_session_patches

order by end_date desc;

30735865

31904550

30628681

SQL> set lines 333 pages 123;

col ABBREVIATION for a12;

col NAME for a45;

col codelevel for a12;

SQL> select ABBREVIATION, NAME, codelevel FROM AD_TRACKABLE_ENTITIES where abbreviation in ('txk','ad','fnd','fwk','atg_pf','icx');

ABBREVIATION NAME CODELEVEL

------------ --------------------------------------------- ------------

ad Applications DBA C.12

atg_pf Applications Technology C

fnd Oracle Application Object Library C

fwk Applications Framework C

icx Oracle iProcurement D

txk Oracle Applications Technology Stack C.12

6 rows selected.

SQL> select release_name from apps.fnd_product_groups;

RELEASE_NAME

------------------------------

12.2.0

SQL> select node_name from apps.fnd_nodes;

NODE_NAME

------------------------------

db_nodename

AUTHENTICATION

apps_nodename

SQL> col EDITION_NAME for a15;

col PARENT_EDITION_NAME for a20;

col USABLE for a10;

SQL> select * from dba_editions;

EDITION_NAME PARENT_EDITION_NAME USABLE

----------------------- ------------------------------------- ----------

ORA$BASE YES

11). Activate the new env file

Now, in a new session, PATCH_TOP and NE_BASE are setup, and new env file EBSapps.env is created under /path/to/$TWO_TASK. Add below line to .profile (of Linux applmgr account), and then start a new OS session:

. /u02/app/$TWO_TASK/EBSapps.env RUN

12). Run autoconfig to update Database Tier to latest code (Section 5 of Doc ID 1617461.1)

$ ./checkMTpatch.sh

No new patches are asked by the script.

Stop Weblogic Server Admin Server:

$ sh $ADMIN_SCRIPTS_HOME/adadminsrvctl.sh stop

You are running adadminsrvctl.sh version 120.10.12020000.2

Enter the WebLogic Admin password:

Enter the APPS Schema password:

Stopping WLS Admin Server...

Refer $LOG_HOME/appl/admin/log/adadminsrvctl.txt for details

adadminsrvctl.sh: exiting with status 0

$ ps -ef | grep $LOGNAME

Stop WebLogic Server Node Manager service:

$ sh $ADMIN_SCRIPTS_HOME/adnodemgrctl.sh stop

You are running adnodemgrctl.sh version 120.11.12020000.4

Enter the WebLogic Admin password:

adnodemgrctl.sh: exiting with status 0

adnodemgrctl.sh: check the logfile $LOG_HOME/appl/admin/log/adnodemgrctl.txt for more information ...

$ ls -lrt $AD_TOP/bin/admkappsutil.pl

-rwxr-xr-x 1 user group 7232 Nov 24 2012 /u02/app/$TWO_TASK/fs1/EBSapps/appl/ad/12.0.0/bin/admkappsutil.pl

$ perl $AD_TOP/bin/admkappsutil.pl

Starting the generation of appsutil.zip

Log file located at $INST_TOP/admin/log/MakeAppsUtil_02151153.log

output located at $INST_TOP/admin/out/appsutil.zip

MakeAppsUtil completed successfully.

DBA to run adautocfg.sh on <RDBMS ORACLE_HOME> after unzip appsutil.zip.

a) Source the environment for RDBMS ORACLE_HOME

b) Uncompress appsutil.zip, under <RDBMS ORACLE_HOME>.

c) Run AutoConfig on <RDBMS ORACLE_HOME>

$ sh <RDBMS_ORACLE_HOME>/appsutil/scripts/<CONTEXT_NAME>/adautocfg.sh

13) Run autoconfig on Apps node, After autoconfig completed on database node:

$ echo $FILE_EDITION

run

$ cd $ADMIN_SCRIPTS_HOME

$ ./adautocfg.sh

… …

AutoConfig completed successfully.

SQL> select node_name from apps.fnd_nodes;

NODE_NAME

------------------------------

db_nodename

AUTHENTICATION

apps_nodename

14) Run ADSplice to create CUSTOM TOPs. Steps are in Doc ID 1577707.1 (Creating a Custom Application in Oracle E-Business Suite Release 12.2).

SQL> select * from FND_OAM_CONTEXT_CUSTOM;

no row returned

$ cd $APPL_TOP/admin

$ copy .txt files to here (from an instance where adsplice worked)

$ adsplice

... ...

If you added new products, use FNDCPASS to change their default passwords.

AD Splicer is complete.

Errors and warnings are listed in the log file $APPL_TOP/admin/$TWO_TASK/log/adsplice.log

and in other log files in the same directory.

New entries shall show up in below queries:

SQL> select * from FND_OAM_CONTEXT_CUSTOM;

SQL> select fav.application_id APPL_ID, fav.application_short_name app_short_name, fav.basepath, fav.application_name,

decode(fpi.status,'I','Installed','S','Shared','N','No', fpi.status) inst_status, fpi.product_version,

nvl(fpi.patch_level, 'Not Available') patchset, fpi.oracle_id

from apps.fnd_application_vl fav, apps.fnd_product_installations fpi

where fav.application_id = fpi.application_id

order by app_short_name asc;

Backup Apps file system & database (Recommended)

15). Applied RUP (Release Update Pack - R12.2.10) patch 30399999 (See Doc ID 2666934.1)

a) download 11 .zip files from patch 30399999 to folder /path/to/R12_2_10

b) run "$ unzip unzip p30399999_R12_LINUX_Xof11.zip" on each of 11 zip files

c) copy software to $PATCH_TOP

$ cd $PATCH_TOP

$ cp -rp /path/to/R12_2_10/30399999 .

$ du -sk 30399999

12981020 30399999

Notes: patching process unzipped some files in the installation software folder. After it is applied:

$ du -sk 30399999

21489484 30399999

d) confirm nothing is running on Apps node.

$ ps -ef | grep $LOGNAME

e) Apply patch 30399999

$ echo $FILE_EDITION

run

$ adop phase=apply apply_mode=downtime patches=30399999 patchtop=/path/to/PATCH_TOP

<== It takes almost 7 hours

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

Validating credentials.

Initializing.

Run Edition context : $CONTEXT_FILE

Patch edition context: $CONTEXT_FILE on /fs2/

Patch file system free space: 76.62 GB

Validating system setup.

Node registry is valid.

Checking for existing adop sessions.

Application tier services are down.

Continuing with the existing session [Session ID: 2].

========================================================

ADOP (C.Delta.12)

Session ID: 2

Node: node_name

Phase: apply

Log: $NE_BASE/EBSapps/log/adop/2/20210X16_155144/adop.log

========================================================

Applying patch 30399999.

Log: $NE_BASE/EBSapps/log/adop/2/20210X16_155144/apply/node_name/30399999/log/u30399999.log

<== It takes upto 7 hours. adctrl can be used to check workers.

After about two hour, it runs adutlrcmp.sql for about 30 minutes.

After about 4 hours, it re-runs adutlrcmp.sql for about 20 minutes

Running finalize actions for the patches being applied.

Log: @ADZDSHOWLOG.sql "2021/0X/16 20:55:04"

Running cutover actions for the patches being applied.

Creating workers to process cutover DDL in parallel

Log: $NE_BASE/EBSapps/log/adop/2/20210X16_155144/apply/node_name/log/cutover.log

Performing database cutover in Quick mode

Generating post apply reports.

Generating log report.

Output: $NE_BASE/EBSapps/log/adop/2/20210X16_155144/apply/node_name/adzdshowlog.out

The apply phase completed successfully.

adop exiting with status = 0 (Success)

$ adopscanlog -latest=yes <== its message is not easy to read!

SQL> select release_name from apps.fnd_product_groups;

RELEASE_NAME

------------------------------

12.2.10

SQL> set lines 333 pages 123;

col owner for a27

col object_name for a45;

col object_typr for a27;

SQL> select owner,object_type,count(*) from dba_objects where status <> 'VALID'

group by owner,object_type order by owner, object_type;

OWNER OBJECT_TYPE COUNT(*)

--------------------------- ----------------------------- ----------

APPS PACKAGE 1

APPS PACKAGE BODY 19

APPS PROCEDURE 2

APPS VIEW 7

APPS_NE TYPE BODY 2

PUBLIC SYNONYM 4181

... ... ...

SQL> select * from dba_editions;

return one line.

Verify and confirm below 4 important patches were applied:

SQL> SELECT adb.bug_number, adb.creation_date,

ad_patch.is_patch_applied('R12',-1, adb.bug_number) status

FROM ad_bugs adb

WHERE adb.bug_number in ('30628681', '30735865', '30399994', '30399999');

BUG_NUMBER CREATION_DATE STATUS

--------------------- ----------------------- -----------

30399994 16-FEB-21 EXPLICIT

30399999 16-FEB-21 EXPLICIT

30628681 12-FEB-21 EXPLICIT

30735865 15-FEB-21 EXPLICIT

30399994 Patch R12.ATG_PF.C.delta.9 (Application Technology Family)

30399999 R12.2.10

30628681 Patch R12.AD.C.delta.12 (Application DBA)

30735865 Patch R12.TXK.C.delta.12 (Application Technology Stack)

16). Start R12.2 services

$ sh $ADMIN_SCRIPTS_HOME/adstrtal.sh apps/apps_PWD

It stayed long and may post errors in some instances from running "adadminsrvctl.sh start -nopromptmsg":

Executing service control script: $INST_TOP/admin/scripts/adadminsrvctl.sh start -nopromptmsg

Timeout specified in context file: -1 second(s)

script returned:

************************

You are running adadminsrvctl.sh version 120.10.12020000.11

Starting WLS Admin Server...

Refer $INST_TOP/logs/appl/admin/log/adadminsrvctl.txt for details

AdminServer logs are located at $FMW_HOME/user_projects/domains/EBS_domain_${TWO_TASK}/servers/AdminServer/logs

adadminsrvctl.sh: exiting with status 0

adadminsrvctl.sh: check the logfile $INST_TOP/logs/appl/admin/log/adadminsrvctl.txt for more information ...

.end std out.

java.rmi.NoSuchObjectException: The object identified by: '299' could not be found. Either it was has not been exported or it has been collected by the distributed garbage collector.

at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:237)

at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:223)

at javax.management.remote.rmi.RMIConnectionImpl_1036_WLStub.getMBeanCount(Unknown Source)

at weblogic.management.remote.common.RMIConnectionWrapper$11.run(ClientProviderBase.java:782)

… … …

.end err out.

During services startup, CPU usage is high on the server. Seems adadminsrvctl.sh did some configuration in the first execution. In its 2nd run, the Java errors did not show up again. And the errors do not show always up in my other instances. The Java errors match with Doc ID 2335354.1. But there are many folders called nodemanager. I do not know yet which file causes the problem.

$ find . -type d -name nodemanager

After adstrtal.sh finished, I was able to log onto EBS R12.2 site. The first page is Secure Configuration Console. Click on Unlock and choose Ignore .., then click on Proceed.

At this stage, after site http URL is added to Java Security tab, EBS Forms and concurrent jobs work only in IE browser. MS Edge does not open the Forms because default R12.2 JRE version 1.6 is lower than the JRE version on my PC.

There are two methods to access the Secure Configuration Console home page later (See https://docs.oracle.com/cd/E26401_01/doc.122/e22952/T156458T663583.htm).

. Using the Functional Administrator responsibility:

On the Oracle E-Business Suite home page, select the Functional Administrator responsibility in the Navigator pane. Then, on the Functional Administrator page, select the Configuration Manager tab.

. Using the OAM Security Dashboard:

On the Oracle E-Business Suite home page, select System Administrator in the Navigator pane. Then select Oracle Applications Manager, then OAM Security Dashboard. On the dashboard, under the Configuration Management section is a link to the Secure Configuration Console.

WLS Admin console at http://nodename.domian.com:s_wls_adminport/console is not accessible by error:

The Server is not able to service this request: [Socket:000445]Connection rejected, filter blocked Socket, weblogic.security.net.FilterException: [Security:090220] rule 2

The error is also in AdminServer.log under $FMW_HOME/user_projects/domains/EBS_domain_$TWO_TASK/servers/AdminServer/logs. The fix is replace one line in file config.xml under $FMW_HOME/user_projects/domains/EBS_domain_$TWO_TASK/config

<connection-filter-rule>0.0.0.0/0 * * deny</connection-filter-rule>

<connection-filter-rule>0.0.0.0/0 * * allow</connection-filter-rule> (<= good one)

After shutdown Admin Server services and re-start them, Admin Server console worked.

$ ps -ef | grep $LOGNAME | wc -l

17). Post actions when Apps services are running (See Doc ID 2666934.1 section 8.1)

Notes: ADOP cleanup and fs_clone steps here can be deferred and run after Post steps in which more patches are applied.

$ adop -status

Enter the APPS password:

Connected.

==============================================================

ADOP (C.Delta.12)

Session Id: 2

Command: status

Output: $NE_BASE/EBSapps/log/adop/2/20210217_114609/adzdshowstatus.out

====================================================================

Node Name Node Type Phase Status Started Finished Elapsed

--------------- ---------- --------------- --------------- -------------------- -------------------- ------------

node_name master APPLY ACTIVE 2021/02/12 11:53:31 2021/02/16 21:15:16 105:21:45

FINALIZE NOT STARTED

CLEANUP NOT STARTED

$ adop phase=cleanup (See Doc ID 2666934.1 section 8.1)

<== it takes 5 minutes

The cleanup phase completed successfully.

adop exiting with status = 0 (Success)

Run FS_CLONE. This action will copy the new run edition code and configuration to the other file system, to ensure that both file systems are in sync before applying patches using the regular adop cycle on the other file system.

$ echo $FILE_EDITION

run

$ adop phase=fs_clone

<= it takes 40 minutes or so

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

Validating credentials.

Initializing.

Run Edition context : $CONTEXT_FILE

Patch edition context: %CONTEXT_FILE on /fs2/

Patch file system free space: 62.92 GB

Validating system setup.

Node registry is valid.

Checking for existing adop sessions.

No pending session exists.

Starting new adop session.

===================================================

ADOP (C.Delta.12)

Session ID: 3

Node: $EBS_HOSTNAME

Phase: fs_clone

Log: $NE_BASE/EBSapps/log/adop/3/20210218_182108/adop.log

==================================================

Validating configuration on node: [$EBS_HOSTNAME].

Log: $NE_BASE/EBSapps/log/adop/3/20210218_182108/fs_clone/validate/$EBS_HOSTNAME

[WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.

Checking for pending cleanup actions.

No pending cleanup actions found.

Blocking managed server ports.

Log: $NE_BASE/EBSapps/log/adop/3/20210218_182108/fs_clone/$EBS_HOSTNAME/txkCloneAcquirePort.log

Performing CLONE steps.

Log: $NE_BASE/EBSapps/log/adop/3/20210218_182108/fs_clone/$EBS_HOSTNAME

Beginning application tier FSCloneStage - Thu Feb 18 18:23:54 2021

$COMMON_TOP/util/jdk32/bin/java -Xmx600M -Doracle.jdbc.autoCommitSpecCompliant=false -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath $FMW_HOME/webtier/lib/xmlparserv2.jar: … oracle.apps.ad.clone.FSCloneStageAppsTier -e $CONTEXT_FILE -targ -stage $COMMON_TOP/adopclone_$EBS_HOSTNAME -tmp /tmp -component appsTier -nopromptmsg

Log file located at $INST_TOP/admin/log/clone/FSCloneStageAppsTier_02181823.log

Completed FSCloneStage...

Thu Feb 18 18:30:49 2021

Beginning application tier FSCloneApply - Thu Feb 18 18:33:21 2021

$COMMON_TOP/adopclone_$EBS_HOSTNAME/FMW/t2pjdk/bin/java -Xmx600M -Doracle.jdbc.autoCommitSpecCompliant=false -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath $FMW_HOME/adopclone_$EBS_HOSTNAME/jlib/xmlparserv2.jar: … oracle.apps.ad.clone.FSCloneApplyAppsTier -e $CONTEXT_FILE ……

Log file located at $INST_TOP/admin/log/clone/FSCloneApplyAppsTier_02181833.log

Target System Fusion Middleware Home set to $PATCH_BASE/FMW_Home

Target System Web Oracle Home set to $PATCH_BASE/FMW_Home/webtier

Target System Appl TOP set to $PATCH_BASE/EBSapps/appl

Target System COMMON TOP set to $PATCH_BASE/EBSapps/comn

Target System Instance Top set to $PATCH_BASE/inst/apps/$CONTEXT_NAME

Report file located at $PATCH_BASE/inst/apps/$CONTEXT_NAME/temp/portpool.lst

The new APPL_TOP context file has been created : $CONTEXT_FILE on /fs2/

contextfile=$CONTEXT_FILE on /fs2/

Completed FSCloneApply...

Thu Feb 18 18:58:25 2021

Resetting FARM name...

runDomainName: EBS_domain_$TWO_TASK

patchDomainName:

targets_xml_loc: $FMW_HOME/user_projects/domains//sysman/state/targets.xml

$FMW_HOME/user_projects/domains//sysman/state/targets.xml file not found or it does not have read/write permissions.

Releasing managed server ports.

Log: $NE_BASE/EBSapps/log/adop/3/20210218_182108/fs_clone/$EBS_HOSTNAME/txkCloneAcquirePort.log

Synchronizing snapshots.

Stopping services on patch file system.

Stopping admin server.

You are running adadminsrvctl.sh version 120.10.12020000.11

Stopping WLS Admin Server...

Refer $PATCH_BASE/inst/apps/$CONTEXT_NAME/logs/appl/admin/log/adadminsrvctl.txt for details

AdminServer logs are located at $FMW_HOME/user_projects/domains/EBS_domain/servers/AdminServer/logs

adadminsrvctl.sh: exiting with status 0

adadminsrvctl.sh: check the logfile $PATCH_BASE/inst/apps/$CONTEXT_NAME/logs/appl/admin/log/adadminsrvctl.txt for more information ...

Stopping node manager.

You are running adnodemgrctl.sh version 120.11.12020000.12

The Node Manager is already shutdown

NodeManager log is located at $PATCH_BASE/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1

adnodemgrctl.sh: exiting with status 2

adnodemgrctl.sh: check the logfile $PATCH_BASE/inst/apps/$CONTEXT_NAME/logs/appl/admin/log/adnodemgrctl.txt for more information ...

Generating log report.

Output: $NE_BASE/EBSapps/log/adop/3/20210218_182108/fs_clone/$EBS_HOSTNAME/adzdshowlog.out

The fs_clone phase completed successfully.

adop exiting with status = 0 (Success)

$ adop -status

Enter the APPS password:

Connected.

=================================

ADOP (C.Delta.12)

Session Id: 3

Command: status

Output: $NE_BASE/EBSapps/log/adop/3/20210219_134510/adzdshowstatus.out

=========================================

Node Name Node Type Phase Status Started Finished Elapsed

--------------- ---------- --------------- --------------- -------------------- -------------------- ------------

$EBS_HOSTNAME master FS_CLONE COMPLETED 2021/02/18 18:21:59 2021/02/18 18:58:36 0:36:37

18) Post-upgrade steps & Additional critical patches (Doc Sections 9 & 10)

Apply patches 30399996 (12.2.10 Online Help. And, may need more product specific patches)

$ echo $APPL_TOP

/u02/app/$TWO_TASK/fs1/EBSapps/appl

$ echo $FILE_EDITION

run

$ cd $PATCH_TOP

$ unzip p30399996_R12_GENERIC.zip

$ adop phase=apply patches=30399996 hotpatch=yes

… … <= it takes 2 hours or more

Patch file system free space: 48.54 GB

Validating system setup.

Node registry is valid.

Checking for existing adop sessions.

[INFO] ICM is not down

[INFO] Connection to http://node_name.domain.com:8041 successful

[WARNING] You should only specify hotpatch mode when directed to by the patch readme.

No pending session exists.

Starting new adop session.

===========================================================

ADOP (C.Delta.12)

Session ID: 4

Node: $EBS_HOSTNAME

Phase: apply

Log: $NE_BASE/EBSapps/log/adop/4/20210219_141932/adop.log

===========================================================

Applying patch 30399996.

Log: $NE_BASE/EBSapps/log/adop/4/20210219_141932/apply/$EBS_HOSTNAME/30399996/log/u30399996.log

Running finalize actions for the patches being applied.

Log: @ADZDSHOWLOG.sql "2021/02/19 15:47:20"

Running cutover actions for the patches being applied.

Creating workers to process cutover DDL in parallel

Log: $NE_BASE/EBSapps/log/adop/4/20210219_141932/apply/$EBS_HOSTNAME/log/cutover.log

Performing database cutover in Quick mode

Generating post apply reports.

Generating log report.

Output: $NE_BASE/EBSapps/log/adop/4/20210219_141932/apply/$EBS_HOSTNAME/adzdshowlog.out

The apply phase completed successfully.

adop exiting with status = 0 (Success)

Apply Patch 31948857 (additional critical patch. Section 10)

$ cd $PATCH_TOP

$ unzip p31948857_R12.FND.C_R12_GENERIC.zip

$ echo $FILE_EDITION

run

$ adop phase=apply patches=31948857 hotpatch=yes

<= it takes less than 10 minutes

$ adop -status

Enter the APPS password:

Connected.

==================================================

ADOP (C.Delta.12)

Session Id: 4

Command: status

Output: $NE_BASE/EBSapps/log/adop/4/20210222_105721/adzdshowstatus.out

==================================================

Node Name Node Type Phase Status Started Finished Elapsed

-------------------------- ----------------- ----------------- -------------------- -------------------- ------------

$EBS_HOSTNAME master APPLY ACTIVE 2021/02/19 14:20:27 2021/02/19 16:41:32 2:21:05

CLEANUP NOT STARTED

Update the snapshot manually with the steps below. It is safe to run it (see Doc ID 1320300.1 Section 1.9):

1) Start adadmin after source the RUN FS env.

2) Select "2. Maintain Applications Files menu" in "AD Administration Main Menu".

3) In "Maintain Applications Files", select "4. Maintain snapshot information".

4) Select "2. Update current view snapshot" in the "Maintain Snapshot Information".

5) Select "1. Update Complete APPL_TOP" in the "Maintain Current View Snapshot Information".

$ adadmin <== it takes 20 to 30 minutes

… …

Note: Your APPL_TOP has 16 files missing. Some of these files are referenced in your file drivers but are missing in the APPL_TOP.

Cleaning up temporary files...

***** File-system snapshot utility completed successfully ******

Recording Adadmin action :ADADMIN_UPDATE_CURRENT_VIEW

Tokens:UPDATE_CURR_VW=1

Review the messages above, then press [Return] to continue.

Confirm all 6 patches were applied:

SQL> SELECT adb.bug_number, adb.creation_date, ad_patch.is_patch_applied('R12',-1, adb.bug_number) status

FROM ad_bugs adb

WHERE adb.bug_number in

('30628681', '30735865', '30399994', '30399999', '30399996', '31948857');

Confirm EBS R12.2 services are still up and running. I can log onto the site to submit concurrent job by IE (Internet Explorer version 1909).

$ ps -ef | grep $LOGNAME | wc -l

It is good time to check the diskspace usage and do a full backup.

$ cd $TWO_TASK

$ du -h --max-depth=1

33G ./apps

3.0G ./inst

43G ./fs1

36G ./fs2

22G ./fs_ne

138G .

From R12.2 upgrade installation, JRE version is 1.6 and JWS is not enabled by default:

$ cat $FORMS_WEB_CONFIG_FILE|grep sun_plugin_version| cut -c 1-35

sun_plugin_version=1.6.0_27

$ grep s_forms_launch_method $CONTEXT_FILE

<config_option type="techstack" oa_var="s_forms_launch_method">browser</config_option>

R12.2 comes with JDK 1.7.0_85. If JDK upgrade is necessary, follow Doc ID 1530033.1

- Using the Latest JDK 7.0 Update with Oracle E-Business Suite Release 12.2.

$ $ADJVAPRG -version

java version "1.7.0_85"

Java(TM) SE Runtime Environment (build 1.7.0_85-b15)

Java HotSpot(TM) Client VM (build 24.85-b06, mixed mode)

$ $AFJVAPRG -version

java version "1.7.0_85"

Java(TM) SE Runtime Environment (build 1.7.0_85-b15)

Java HotSpot(TM) 64-Bit Server VM (build 24.85-b06, mixed mode)

By now, EBS is successfully upgraded to R12.2.10.

Tuesday, March 9, 2021

Enable R12.2 online patching (ADOP)

After R12.2.0 is configured, the next step is to enable online patch (adop), one of the major changes in R12.2. Followings are steps to do that.

1). Apply online patching Readiness and GSCC Report patch (Doc ID: 1531121.1)

a) Get the R12.2 env

$ cd /u02/app/${TWO_TASK}/fs1/EBSapps/appl

$ . APPS${CONTEXT_NAME}.env

$ echo $FILE_EDITION

run

Confirm $AD_TOP/patch/115/sql/adzddtsfix.sql and folder $FND_TOP/perl/GSCC do not exist

b) $ unzip p31026891_R12.AD.C_R12_GENERIC.zip (download file "C" for R12.2.0)

(If apply this patch to a R12.1 instance before R12.2 upgrade, download file p31026891_R12.AD.B_R12_GENERIC.zip)

c) Confirm the same version of adgrants.sql as in this patch was executed before.

d) Enable maintenance mode

SQL> @$RUN_BASE/EBSapps/appl/ad/12.0.0/patch/115/sql/adsetmmd.sql ENABLE

SQL> select fnd_profile.value('APPS_MAINTENANCE_MODE') from dual;

--------------------------------------------------------------------------------

MAINT

e) apply Readiness patch

$ echo $FILE_EDITION

run

$ echo $ORACLE_HOME

/u02/app/$TWO_TASK/fs1/EBSapps/10.1.2

$ cd 31026891

$ ls

$ adpatch

No error in log file in folder $APPL_TOP/admin/${TWO_TASK}/log

SQL> select bug_number, to_char(creation_date,'DD-MON-RRRR HH24:MI:SS') date_applied

from apps.ad_bugs where bug_number = '31026891';

BUG_NUMBER DATE_APPLIED

------------------------------ --------------------

31026891 DD-MON-2021 15:34:18

2). Run the reports (See Upgrade Guide p.122 and README in Section 5)

$ cd $LOG_HOME/appl

$ ls

admin oam rgf

$ mkdir op

$ cd op

$ sqlplus system @$AD_TOP/sql/ADZDPSUM.sql <= It only queries the data dictionary

$ mv adzdpsum.txt adzdpsum_pre_dbprep.txt

$ sqlplus system @$AD_TOP/sql/ADZDPMAN.sql <= Lists all violations that need a fix

$ mv adzdpman.txt adzdpman_pre_dbprep.txt

$ sqlplus system @$AD_TOP/sql/ADZDPAUT.sql <= Generates an information-only report

$ mv adzdpaut.txt adzdpaut_pre_dbprep.txt

$ sqlplus apps @$AD_TOP/patch/115/sql/adzddtsfix.sql

exec dbms_utility.invalidate(36166,NULL,0);

exec dbms_utility.invalidate(1015,NULL,0);

exec sys.utl_recomp.recomp_parallel

exit;

$ ls

adzddtsfixout.sql adzdpaut_pre_dbprep.txt adzdpman_pre_dbprep.txt adzdpsum_pre_dbprep.txt

$ more adzddtsfixout.sql

exec dbms_utility.invalidate(36166,NULL,0);

exec dbms_utility.invalidate(1015,NULL,0);

exec sys.utl_recomp.recomp_parallel / <== See Upgrade Guide p.126

exit;

$ sqlplus apps @adzddtsfixout.sql

PL/SQL procedure successfully completed.

$ sqlplus apps @$AD_TOP/sql/ADZDDBCC.sql <= List database violations

$ mv adzddbcc.lst adzddbcc_pre_dbprep.txt

Note: needs to specify entries in ADZDPCUST.sql before running it. (I skipped it)

$ ls $LOG_HOME/appl/op

adzdpsum_pre_dbprep.txt <= send it to Developer

adzdpman_pre_dbprep.txt <= send it to Developer (Specially, Section 5 & 8)

adzdpaut_pre_dbprep.txt

adzddbcc_pre_dbprep.txt

<= send it to DBA who has to address all issues in it,

except SECTION-37 ("Unfixed violations can be ignored")

If APPS_DDL shows up in SECTION-47, follow Doc ID 2735375.1 to install it to avoid errors:

AutoPatch error: adpmrp: Error while installing apps_ddl packages.

AutoPatch - aidafoGetFileDbVersion: INFO: ORA-01403: no data found

In my cases, Developer did not correct many findings in above reports.

3). Run ETCC scripts again to make sure no new technology patches are needed.

Oracle updates/release ETCC (Patch 17537119) from time to time. During the R12.2 upgrade, download and keep a same version to run a few times on both database and apps. That way will potentially save your time from keeping applying other supporting patches.

4). Gather SYS schema stats. Run it as SYSDBA

SQL> begin

dbms_stats.gather_schema_stats(

'SYS',

options=>'GATHER STALE',

estimate_percent =>

DBMS_STATS.AUTO_SAMPLE_SIZE,

method_opt => 'FOR ALL COLUMNS SIZE AUTO',

cascade => TRUE);

end;

5). Check database free space:

$ export HOSTNAME=node_name (<= without domain name. Maybe an optional step)

$ perl $AD_TOP/bin/adzdreport.pl apps

Enter the APPS Password:

Online Patching Diagnostic Reports Main Menu

--------------------------------------------

1. Run edition reports

2. Patch edition reports

3. Other generic reports

4. Exit

Enter your choice [4]: 3

Other Generic Reports Sub Menu

------------------------------

1. Editions summary

2. Editioned objects summary

3. Free space in important tablespaces

4. … … …

Enter your choice [16]: 3

Database tablespace info will be saved in file $APPL_TOP/admin/$TWO_TASK/out/adzdshowts.out.

Tablespaces MUST be extended to avoid patch job fails:

SYSTEM Tablespace: has a minimum of 25 GB of free space

APPS_TS_SEED Tablespace: has a minimum of 5 GB of free space

6) AD_ZD_PREP has to be valid before moving forward

SQL> Select owner,object_name,object_type,status from dba_objects where object_name like 'AD_ZD%' and status ='INVALID';

OWNER OBJECT_NAME OBJECT_TYPE STATUS

------------- ----------------------------- ------------------------ ------------

APPS AD_ZD_PREP PACKAGE BODY INVALID

SQL> alter package AD_ZD_PREP compile body;

SQL> show error;

Errors for PACKAGE BODY AD_ZD_PREP:

LINE/COL ERROR

-------- -----------------------------------------------------------------

463/3 PL/SQL: SQL Statement ignored

463/19 PL/SQL: ORA-00942: table or view does not exist

467/5 PL/SQL: SQL Statement ignored

467/21 PL/SQL: ORA-00942: table or view does not exist

487/5 PL/SQL: Statement ignored

487/5 PLS-00201: identifier 'SYS.XDB_MIGRATESCHEMA' must be declared

501/5 PL/SQL: SQL Statement ignored

501/21 PL/SQL: ORA-00942: table or view does not exist

Fix: Follow Doc ID 2066607.1 (Patch 13543062 Fails With Error: ORA-06508: PL/SQL: could not find APPS.AD_ZD_PREPAD_ZD_PREP Due To Missing Package 'xdb_migrateschema' ). Actually, this could be avoid if a database upgrade post-step (Doc ID 1524398.1) was executed.

a). conn / as sysdba

@?/rdbms/admin/dbmsxdbschmig.sql

@?/rdbms/admin/prvtxdbschmig.plb

SQL> set line 130

SQL> set pages 50

SQL> col OWNER for a25;

SQL> col OBJECT_NAME for a30

SQL> col OBJECT_TYPE for a30;

SQL> select owner, object_name, object_type from dba_objects

where object_name like upper ('xdb_migrateschema');

OWNER OBJECT_NAME OBJECT_TYPE

---------------------- ---------------------------------- -----------------

SYS XDB_MIGRATESCHEMA PACKAGE

SYS XDB_MIGRATESCHEMA PACKAGE BODY

b). Run adgrants.sql.

c). Recompile the package AD_ZD_PREP and ensure it compiles successfully.

SQL> alter package AD_ZD_PREP compile body;

Package body altered.

SQL> select owner,object_name,status from dba_objects

where object_name like 'AD_ZD%' and status ='INVALID';

no rows selected

7). Check database to avoid errors from applying patch 13543062

$ cd $LOG_HOME/appl/op

$ sqlplus apps @$AD_TOP/sql/ADZDEXRPT.sql

Start Time: XX-02-2021 14:44:31

PL/SQL procedure successfully completed.

DOC>

DOC> Online Patch Enablement - Execution Status Report of SQL jobs

DOC>

DOC> Report Script: $AD_TOP/sql/ADZDEXRPT.sql

DOC> Result Spooled Into: adzdexrpt.txt

DOC>

DOC> Currently running Workers (database sessions) for SQL jobs execution.

DOC>

DOC>#

no rows selected

DOC>

DOC> ***********************************************

DOC> SECTION 1

DOC> ***********************************************

DOC> Count of Current DDL Jobs per Phase and Status

DOC>

DOC> Phases:

DOC> 1. DROP_UNUSED_OBJECT : Remove obsolete objects from XLA schema.

DOC> 2. COPY_TYPE : Create EBS Types in APPS_NE schema, stop Advanced Queues.

DOC> 3. COMPILE_TYPE : Recompile APPS_NE schema.

DOC> 4. COPY_EVOLVED_TYPE : Create evolved types in APPS_NE schema.

DOC> 5. FIX_COLUMN : Fix TYPE reference to APPS_NE.<type> at table and AQ level.

DOC> 6. FIX_TYPE : Additional check, Fix TYPE reference and then drop TYPE from source schema.

DOC> 7. FIX_PUBLIC_SYNONYM : Fix public synonyms.

DOC> 8. RECREATE_AQ_OBJECT : Re-create internal AQ objects after TYPE Ref fix.

DOC> 9. DROP_OBJECT : Drop objects like CTXSYS.<package>, CTXSYS.<synonym> and others if any.

DOC> 10. ENABLE_EDITIONING : Enable registered schemas for Online Patching.

DOC> 11. UPGRADE_TABLE : Create Editioning View cover layer over EBS data model.

DOC> 12. UPGRADE_SEED : Upgrade seed data tables to support edition based storage.

DOC> 13. COLLECT_STATS : Gather Statistics on Seed Data Tables after upgrade.

DOC> 14. UPGRADE_MVIEW : Convert Materialized Views with expand query.

DOC> 15. CUTOVER : Perform maintenance operations on columns and indexes after adop APPLY phase

DOC> and before actually changing Patch-Edition as Default-Edition.

DOC> 16. ACTUALIZE_ALL : Actualize all editioned objects into the current edition.

DOC> 17. CLEANUP : Cleanup task after adop CUTOVER.

DOC>

DOC>#

Phase Count Status

------------------------------ --------- -----------------------

UPGRADE_SEED 2605 N - Not Executed

COLLECT_STATS 2605 N - Not Executed

CLEANUP 1 N - Not Executed

3 rows selected.

Elapsed: 00:00:00.02

DOC>

DOC> ************************************

DOC> SECTION 2

DOC> ************************************

DOC> Failed Jobs with UNEXPECTED error, For complete error stack and DDL,

DOC> Please check AD_ZD_DDL_HANDLER table for corresponding DDL_ID.

DOC>

DOC>#

no rows selected

Elapsed: 00:00:00.01

DOC>

DOC> *************************************

DOC> SECTION 3

DOC> *************************************

DOC> Failed Jobs with ERRORS, For complete error stack and DDL,

DOC> Please check AD_ZD_DDL_HANDLER table for corresponding DDL_ID

DOC>

DOC>#

no rows selected

Elapsed: 00:00:00.01

DOC>

DOC> ************************************

DOC> SECTION 4

DOC> ************************************

DOC> Running Jobs , Please check AD_ZD_DDL_HANDLER table for

DOC> corresponding DDL_ID

DOC>

DOC>#

no rows selected

Elapsed: 00:00:00.00

DOC>

DOC> ************************************

DOC> SECTION 5

DOC> ************************************

DOC> Failed Jobs with Warnings, For complete error stack and DDL,

DOC> Please check AD_ZD_DDL_HANDLER table for corresponding DDL_ID

DOC>

DOC>#

no rows selected

Elapsed: 00:00:00.01

DOC>

DOC> ************************************

DOC> SECTION 6

DOC> ************************************

DOC> E-Business Suite INVALID object-count: Object-Type wise and Total

DOC>

DOC>#

no rows selected

Elapsed: 00:00:00.00

Invalid Object Type Count

------------------------------ ---------

PROCEDURE 23

TRIGGER 3

PACKAGE BODY 45

PACKAGE 1

---------

Total Invalid Objects (EBS): 77

5 rows selected.

Elapsed: 00:00:00.13

DOC>

DOC> *****************************************

DOC> SECTION 7

DOC> *****************************************

DOC> List of E-Business Suite INVALID objects

DOC>

DOC>#

Owner Type Name

-------------------- ------------------------------ ---------------------------

CUSTOM XXXXXXX XXXXXXXXX

etc … … …

APPS PACKAGE AK$RCV_CONFIRM_RECEIPT$178

APPS PACKAGE BODY AEAP_AID_TABLE_HANDLER_PKG

APPS PACKAGE BODY AEAP_PAYMENT_PROCESSOR

APPS PACKAGE BODY AEAP_WEB_RPT_DELINQUENT

APPS PACKAGE BODY AEGL_IFS_TO_FSAH_DRILL_PKG

APPS PACKAGE BODY AEPO_CHNPO_CANCEL_PKG

APPS PACKAGE BODY AEPO_CHNPO_PO_IMPORT_PKG

APPS PACKAGE BODY AEWEB_USERPASS

APPS PACKAGE BODY AE_AP_IMPORT_INVOICES_PKG

APPS PACKAGE BODY AK$RCV_CONFIRM_RECEIPT$178

APPS PACKAGE BODY AP_PAYMENT_EVENT_WF_PKG

APPS PACKAGE BODY CZ_IMP_SINGLE

APPS PACKAGE BODY DOCSAVI_UA_DQ

APPS PACKAGE BODY HZ_STAGE_MAP_TRANSFORM

APPS PACKAGE BODY M4R_7B1_WSM_IN

APPS PACKAGE BODY OKS_COVERAGE_MIGRATION

APPS PACKAGE BODY OKS_RULE_MIGRATE

APPS PACKAGE BODY PA_RBS_MAPPING

APPS PACKAGE BODY PON_AUC_INTERFACE_TABLE_PKG

APPS PACKAGE BODY PON_CP_INTRFAC_TO_TRANSACTION

APPS PACKAGE BODY PON_OPEN_INTERFACE_PVT

APPS PACKAGE BODY PON_SLM_UTIL_PKG

APPS PACKAGE BODY POS_AP_INVOICES_PKG

APPS PACKAGE BODY POS_DATA_SECURITY

APPS PACKAGE BODY POS_SUPP_APPR

APPS PACKAGE BODY POS_WC_VIEW_CANCEL_PVT

APPS PACKAGE BODY PO_IMPORT_PACKAGE

APPS PACKAGE BODY PQP_AAT_INS

APPS PACKAGE BODY PQP_AAT_SHD

APPS PACKAGE BODY PQP_AAT_UPD

APPS PACKAGE BODY PSP_ENC_CREATE_LINES

APPS PACKAGE BODY PSP_SUM_TRANS

APPS PACKAGE BODY QA_RESULTS_INTERFACE_PKG

APPS PACKAGE BODY QLTTRAMB

APPS PACKAGE BODY UMX_LOGIN_HELP_PVT

APPS PACKAGE BODY UMX_NOTIFICATION_UTIL

APPS PACKAGE BODY UMX_REG_REQUESTS_PVT

APPS PACKAGE BODY UTL_RECOMP

APPS PACKAGE BODY WF_REPORT

APPS PACKAGE BODY WSH_WF_LOAD_TEND

APPS VIEW OKS_COVTIMES_HIST_V

77 rows selected.

Elapsed: 00:00:00.12

DOC>

DOC> ***********************************

DOC> SECTION 8

DOC> ***********************************

DOC> Time taken by each PHASE

DOC>

DOC>#

Phase ROW_COUNT START_TIME END_TIME ELAPSED_TIME_MIN

------------------------ ----------- -------------------- -------------------- ------------

UPGRADE_SEED 2605

COLLECT_STATS 2605

CLEANUP 1

3 rows selected.

Elapsed: 00:00:00.02

End Time: xx-02-2021 14:44:31

… …

Run below script again to confirm DBA's fixes (Recommended)

$ sqlplus apps @$AD_TOP/sql/ADZDDBCC.sql

8) Pre-steps for apply patch 13543062

SQL> set pages 200 lines 300

SQL> select owner,object_name,object_type,status from dba_objects

where object_name='HR_PA_MAINTN_JOB_HIST_BRI' and owner='APPS';

OWNER OBJECT_NAME OBJECT_TYPE STATUS

------------ ---------------------------------------------- ----------------------- -------

APPS HR_PA_MAINTN_JOB_HIST_BRI TRIGGER VALID

SQL> select SERVICE_ID,NAME from dba_services where name='ebs_patch';

SERVICE_ID NAME

---------- ----------------------------------------------------------------

8 ebs_patch

If it does not exist, follow Doc ID 2579880.1 to re-create it.

SQL> ALTER SYSTEM SET "_disable_actualization_for_grant"=true; (if not set)

System altered.

SQL> alter system set max_dump_file_size=unlimited;

to avoid Patch 1354362 hangs on script ADZDWRKR.sql (Doc ID 1946877.1)

Find the version of Human Resources (HR) on my system (Doc ID 269286.1):

SQL> SELECT substr(fpi.application_id,1,6) APP_ID,

substr(fat.application_name,1,40) APPLICATION, substr(l.meaning,1,9) STATUS,

substr(decode(fpi.patch_level,null,'11i.'||fa.application_short_name||'.?',fpi.patch_level),1,12) PATCH

FROM fnd_product_installations fpi,

fnd_application_tl fat, fnd_application fa, fnd_lookups l

WHERE (fpi.application_id between 800 and 850 OR fpi.application_id in (178,275,712,777))

AND fpi.application_id = fat.application_id

AND fpi.application_id = fa.application_id

AND l.lookup_type = 'FND_PRODUCT_STATUS'

AND l.lookup_code = fpi.status

ORDER BY fat.application_id;

APP_ID APPLICATION STATUS PATCH

------ ---------------------------------------- --------- ------------

178 Oracle iProcurement Installed R12.ICX.D

275 Projects Shared R12.PA.C

712 Project Manufacturing Not insta R12.PJM.C

777 Project Contracts Not insta R12.OKE.H

800 Human Resources Shared R12.PER.C

801 Payroll Shared R12.PAY.C

802 FastFormula Shared R12.FF.C

803 DateTrack Shared R12.DT.C

804 SSP Not insta R12.SSP.C

805 Advanced Benefits Not insta R12.BEN.C

808 Time and Labor Not insta R12.HXT.C

809 Time and Labor Engine Not insta 11i.HXC.C

810 Learning Management Not insta R12.OTA.C

821 iRecruitment Not insta R12.IRC.C

14 rows selected.

SQL> set lines 180

set pages 50

col LANGUAGE_CODE for a15

col NLS_LANGUAGE for a20

col ISO_LANGUAGE for a15

col INSTALLED_FLAG for a15

SQL> SELECT LANGUAGE_CODE,NLS_LANGUAGE, ISO_LANGUAGE,INSTALLED_FLAG

FROM APPS.FND_LANGUAGES

WHERE INSTALLED_FLAG IN ('B','I');

LANGUAGE_CODE NLS_LANGUAGE ISO_LANGUAGE INSTALLED_FLAG

--------------------------- -------------------------- ------------------------ ---------------

US AMERICAN EN B

SQL> select owner,object_name,status from dba_objects

where object_name like 'AD_%' and status ='INVALID';

no rows selected

SQL> select fnd_profile.value('APPS_MAINTENANCE_MODE') from dual;

FND_PROFILE.VALUE('APPS_MAINTENANCE_MODE')

--------------------------------------------------------------------------------

MAINT

9) Apply patch 13543062

$ echo $FILE_EDITION

run

$ unzip p13543062_R12.AD.C_R12_GENERIC.zip (only 6k in size)

$ cd 13543062

$ ls

$ adpatch options=hotpatch,forceapply

<== It already exists and so "forceapply"

<== Upgrade Guide p.126. It spends mostly on running SQL files

… etc …

AutoPatch is complete.

AutoPatch may have written informational messages to the file

$APPL_TOP/admin/$TWO_TASK/log/13543062.lgi

Errors and warnings are listed in the log file

$APPL_TOP/admin/$TWO_TASK/log/13543062.log

$ egrep -i "error|fail|ORA-|Failed-|rc-" $APPL_TOP/admin/$TWO_TASK/log/13543062.log

Note: query "select * from ad_bugs where bug_number = '13543062';" will show a earlier creation_date.

If patch failed and not able to restart Online Enablement patch 13543062 using adpatch, try solution in Doc ID 1908956.1. To implement the solution please perform following

a) In the terminal

export ENABLE_ADPATCH=YES

b) Run adaptch from the same terminal.

Note : The above solution is applicable only for Online enablement patch 13543062

10) Monitor the Online Patching enablement patch application:

The enablement patch application may take more than one hour to finish. You can monitor its progress at any time by running the DDL Status Report (ADZDSHOWDDLS.sql) as follows:

SQL> show user

USER is "APPS"

SQL> @$AD_TOP/sql/ADZDSHOWDDLS.sql

DOC>

DOC> Count of Current DDL Jobs per Phase and Status

DOC>

DOC> Phases:

DOC> 1. DROP_UNUSED_OBJECT : Remove obsolete objects from XLA schema.

DOC> 2. COPY_TYPE : Create EBS Types in APPS_NE schema, stop Advanced Queues.

DOC> 3. COMPILE_TYPE : Recompile APPS_NE schema.

DOC> 4. COPY_EVOLVED_TYPE : Create evolved types in APPS_NE schema.

DOC> 5. FIX_COLUMN : Fix TYPE reference to APPS_NE.<type> at table and AQ level.

DOC> 6. FIX_TYPE : Additional check, Fix TYPE reference and then drop TYPE

DOC> from source schema.