Sunday, March 21, 2021

Run ADSplice in R12.2

ADSplice is the only way to create custom tops in EBS R12.2. Steps of running it is in Doc ID 1577707.1 (Creating a Custom Application in Oracle E-Business Suite Release 12.2). If it is a multi-node instance, you have to run it on each node.  

1. Before running ADSplice, make sure latest AD and TXK for R12.2 patches were applied already:
SQL> select abbreviation ,codelevel from ad_trackable_entities 
where abbreviation in( 'ad','txk') order by abbreviation;

ABBREVIATION CODELEVEL
--------------------- ---------
ad                         C.12
txk                        C.12

2. For a regular/normal custom top, it is not difficult to follow the Oracle document for creating or modifying it. Three files are required by ADSplice in folder $APPL_TOP/admin for one custom top $XXFS_TOP, for example
xxefprod.txt
xxefterr.txt
newprods.txt

3. Then run adsplice. It will ask for SYSTEM and APPS password. It also give option to run AutoConfig. So, it is better to shut down apps services before running adsplice.
 
$ echo $FILE_EDITION
run
$ cd $APPL_TOP/admin
$ adsplice

ADSplice will create folder structure for $XXFS_TOP under $APPL_TOP. 

For multiple nodes, run adsplice on all nodes. After 1st node, adsplice runs much faster because it knows all database part was completed.

4. Most custom top has custom codes/files. Migrate custom codes/files to the correct new folder, and then compile them or link them to right executable files.

5. After ADSplice worked, add entries for all your custom files to the custom synchronization driver file located at $APPL_TOP_NE/ad/custom/adop_sync.drv. I used below line which works well in copying folder xxfs in <s_current_base> (source) to <s_other_base> (target)

rsync -a --delete %s_current_base%/EBSapps/appl/xxfs %s_other_base%/EBSapps/appl

The option -a is very powerful, it will copy the directory and all its sub-directories recursively, and preserve symbolic links, modification times, group, ownership, and permissions. It is a combination of a bunch of other options:
-r  Recurse into directories  
-l  Copy symlinks as symlinks
-p  Preserve file permissions
-t  Preserve modification times
-g  Preserve group
-o  Preserve owner
-D  Preserve device and special files

-- delete option will delete files and sub-directories in target folder but not in source folder.

6. Run FS_CLONE
ADSplice creates folders only in RUN Edition file system. You must run "adop phase=fs_clone". This ensures that all custom products are synchronized/copied to PATCH file system. Doc ID 2167878.1 indicates FS_CLONE does use adop_sync.drv.

I ran into a error with "adop phase=prepare". ADOP does not tell what is the specific problem at all!
=============================
Inside evalADPATCHStatus()...
=============================
message_status: ERROR
Adsplice action did not go through successfully.
*******FATAL ERROR*******
PROGRAM : ($AD_TOP/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Wed Jul 20 14:00:36 2022
FUNCTION: main::execADSPLICE [ Level 1 ]
ERRORMSG: Adsplice action did not go through successfully. 

Before the error, log file txkADOPPreparePhaseSynchronize.log posts a SQL statement similar to below lines to check table AD_ADOP_SESSION_PATCHES:

SELECT * FROM ad_adop_session_patches 
WHERE bug_number like 'ADSPLICE%' AND 
ORDER BY TO_CHAR(end_date,'YYYY.MM.DD:HH24:MI:SS') DESC

Without running fs_clone, column PATCH_FILE_SYSTEM_BASE is empty for one CUSTOM TOP. I think that causes ADOP PREPARE failed.

After "adop phase=prepare" failed, "adop phase=fs_clone" will not work. You have to below steps in order:
$ adop phase=abort
$ adop phase=cleanup cleanup_mode=full
$ adop phase=fs_clone

Notes: Similarly, anytime after a change was made in RUN file system, e.g. a patch applied in HOTPATCH mode, you have to run "adop phase=fs_clone" to sync the change to PATCH file system.

7. A useful query to verify if a new custom top in R12.2 was created by ADSplice 
SQL> set linesize 300
set pagesize 300
col APPL_ID format 99999;
col app_short_name format a15
col basepath format a 15
col basepath format a15
col application_name format a35
col inst_status format a10
col product_version a20
col product_version format a20
col patchset format a25
col oracle_id format 99999

SQL> SELECT fav.application_id APPL_ID, fav.application_short_name app_short_name, fav.basepath, fav.application_name,
decode(fpi.status,'I','Installed','S','Shared','N','No', fpi.status) inst_status,  fpi.product_version,
nvl(fpi.patch_level, 'Not Available') patchset, fpi.oracle_id
FROM apps.fnd_application_vl fav, apps.fnd_product_installations fpi
WHERE fav.application_id = fpi.application_id
and (fav.application_short_name like 'XX%' )
ORDER BY app_short_name asc;

- TROUBLESHOOTING and lessons learned on ADSplice:

1. We had a custom top that was created in 9i with mismatch between SHORT_NAME and BASEPATH (the name of custom top):
SQL> select ORACLE_USERNAME,ORACLE_ID, dba.USERNAME
   from fnd_oracle_userid fnd, dba_users dba
 where fnd.oracle_username = dba.username and lower(oracle_username) in ('xxfs') ;

ORACLE_USERNAME ORACLE_ID USERNAME 
------------------------------ ----------------  ---------------- 
XXFS                               20024             XXFS             

SQL> select application_id,application_short_name short_name, 
last_update_date, basepath, product_code
   from fnd_application
 where application_short_name like '%XXFS%' or application_id = 20024;  

APPLICATION_ID SHORT_NAME LAST_UPDATE BASEPATH PRODUCT_CODE
------------------------ -------------------- -------------------- --------------- ------------------------
                     20024                XXFFS    16-DEC-97       XXFS_TOP  XXFFS

We tried to keep it as the old way to minimize the re-write of custom codes, by putting below lines in file xxefprod.txt

# Product specific Product ID's
xxfs     20024

END_OF_PRODUCT_ABBREVIATIONS   -999
#
# ##################################################
# Oracle Support Diagnostic Tools
# ##################################################
#
# application id, abbreviation, shortname, prefix
20024 aefs XXFFS APP
# multiple product installations for msob, "controlled release", shared only
#   optional fourth field is "stub product".  default is No
No No No No
# multilingual, has NLADD.sql
Yes Yes
# install oracle id, default ORACLE username, default ORACLE password
20024 XXFS XXFS

Note: In above line, XXFS is not necessarily the current db password. If it does not exist in database, adsplice will create it.

ADSplice does not complain it and created $XXFS_TOP. But when I later applied patches, "adop phase=prepare" failed with error:

Custom product XXFS entry is NOT there in FND_APPLICATIONS table
ERROR: The following custom products have not been correctly added: XXFS

We also tried to created two custom tops $XXFFS_TOP and $XXFS_TOP (with a new application id assigned, such as 30034), while they share same existing database account XXFS.  ADSplice also created two without complain. But "adop phase=prepare" failed again without giving a specific error:

message_status: ERROR
Adsplice action did not go through successfully.
*******FATAL ERROR*******
PROGRAM : ($AD_TOP/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Wed Xxx  5 11:52:14 2021
FUNCTION: main::execADSPLICE [ Level 1 ]
ERRORMSG: Adsplice action did not go through successfully.

I had to de-register $XXFFS_TOP first and then de-register $XXFS_TOP by using 
$ perl $AD_TOP/bin/adDeregisterCustomProd.pl

After de-register them, ADOP worked in the full patching cycle.

Before that, I tried the suggestion in Doc ID 1932207.1 (Adop Prepare ERRORMSG: Adsplice Action Did Not Go Through Successfully After Add new Custom Application in R12.2). But I was not sure (do not remember) it helped or not.

SQL> UPDATE FND_ORACLE_USERID SET ENABLED_FLAG = 'N'
WHERE ORACLE_USERNAME in ('XXFS') ;    -- from "(null)" 

2. "perl $AD_TOP/bin/adDeregisterCustomProd.pl" will delete data from tables FND_OAM_CONTEXT_CUSTOM, FNS_APPLICATION. But it will not delete the row in table FND_PRODUCT_INSTALLATION.

3. If the password of Oracle database account XXEF was changed not by FNDCPASS, ADSplice will give ORA-01017 error.  The fix is to use below line to change its password:
$ FNDCPASS apps/apps_PWD 0 Y system/'systemPWD' ORACLE xxef xxef_PWD

4.  "adop phase=abort,cleanup cleanup_mode=full" failed with error message:

AutoPatch - aidafoGetFileDbVersion: INFO: ORA-01403: no data found
AutoPatch error:
Unable to get APPS_DDL package version from db <XXFS> <APPS_DDL> <PACKAGE>
AutoPatch error:
adpmrp: Error while installing apps_ddl packages.

Run below query to see its status.  The fix is to run an UPDATE on the row. 

SQL> select oracle_username, read_only_flag from fnd_oracle_userid 
 where read_only_flag <> 'A';

ORACLE_USERNAME  READ_ONLY_FLAG
------------------------------ ---------------------------
APPLSYS                        E
APPS                               U
XXFS                               D
APPLSYSPUB                C
APPS_NE                        Z

SQL> update fnd_oracle_userid set read_only_flag='A' 
where ORACLE_USERNAME='XXFS';

Then "adop phase=abort" worked.  (Note: In my another instance, it has the same status, but ADOP does not fail).

5. After a custom top is created, if it is used by a Responsibility or concurrent jobs, run autoconfig and bounce apps services to avoid error/message:

APP-FND-00362: Routine afpbep cannot execute request &REQUEST for program &PROGRAM, because the environment variable &BASEPATH is not set for the application to which the concurrent program executable &EXECUTABLE belongs.

Shut down the concurrent managers. Set the basepath environment variable for the application. Restart the concurrent managers. (VARIABLE=)
Routine AFPEOT cannot construct the name of an executable file for your concurrent request 430346.

Check that the file name components are correct and valid on your system. Check that the environment for the person who started the concurrent manager 

6. R12.2 GUI forms will not create a custom top, because navigation Application => Register gives message:
Forms Registration for Custom Application or Schema is now blocked. Please consult ADSplice documentation for the current standards for deploying Customizations in EBS

Doc ID 2272551.1 says this is intended functionality. Application Forms cannot be used to add or remove applications. This is restricted in 12.2 and going forward.

7. If a custom top is not necessary but an env variable is needed as a pointer to a folder under $APPL_TOP, modify file $FND_TOP/fndenv.env by adding lines:
#Begin Customizations
AEUT_TOP=${APPL_TOP}/xxedi/12.0.0; export XXEDI_TOP
#End Customizations

With the use of adsplice, I am not sure if modifying Forms file is necessary in R12.2
$INST_TOP/ora/10.1.2/forms/server/default.env:
#Begin Customizations
XXEDI_TOP=/path/to/appl/xxedi/12.0.0
#End Customizations

Related Documents: 
How to remove a CUSTOM top  (Doc ID 1903037.1)
How To De-register Custom Applications (Doc ID 2085355.1)

No comments: