Enable Java Web Start with Oracle E-Business Suite R12.2

After R12.1.3 is upgraded to R12.2.10, I see  JRE 1.6.0_27 is the version in the new R12.2.10 instance. It is a low version, which makes R12.2 GUI Forms work only with IE browser. For R12.2 to work with MS Edge and Google Chrome browsers, we have to upgrade JRE and enable JWS.

1) First, per Doc ID 2188898.1 (Using Java Web Start with Oracle E-Business Suite), confirmed patch 26825525 (replaces 25441839 Merge Request On Top Of 10.1.2.3.2 PSU, Doc ID 2554599.1) was applied during R12.2.10 upgrade. (Section 4.2.2). 

$ opatch lsinventory | grep 26825525

     Patch  26825525 applied on xxxx, 2021

2) Confirmed ALL below 13 patches were applied (Section 4.2.3). Some patches may show up twice in below query.

SQL> SELECT bug_number, creation_date, ad_patch.is_patch_applied('R12',-1, bug_number) status
FROM ad_bugs
where bug_number in (
'24498616', -- AD: Add Java Web Start support to Oracle E-Business Suite
'25449925', -- TXK: Add Java Web Start support to Oracle E-Business Suite
'25380324', -- Oracle E-Business Suite Java Applets launching with Java Web Start
'29058008', -- Oracle E-Business Suite Java Applets Launching with JWS rel 2
'29024389', -- WS: New Preference To Indicate Launch of New Forms Session
'28713780', -- 12.2.6+:Oracle Workflow Java Applets launching with Java Web Start
'23645622', -- GL: Add Java Web Start Support to AHM Java applet
'23586286', -- MSC: Add Java Web Start Support to PS/SNO 12.2.6
'26100397', -- R12.TXK.C (see Doc ID 393931.1 for deploying JRE)
'22806350',  -- R12.OWF.C (see Doc ID 1367293.1 for enabling TLS 1.2)
'21473055', -- R12.ICX.D (iProcurement, Doc ID 1937220.1 for TLS1.2)
'22522877', -- R12.IBY.C (iPayment)
'22326911'  -- R12.ECX.C (Oracle XML Gateway)
) order by bug_number;

3) Backup CNOTEXT_FILE

$ cd (to a folder for backup folder)

$ cp -p $CONTEXT_FILE ${CONTEXT_NAME}.xml_BK_JRE

Replace "browser" by "jws" for <s_forms_launch_method>

$ vi $CONTEXT_FILE

Optionally, stop EBS services by adstpall.sh if they are running.

4) Follow ID 393931.1 (Deploying JRE for Windows Clients, go to a long table in Appendix E section) to download patch 30425890 - p30425890_180241_WINNT.zip (Oracle JRE 8 Update 241 for Windows 32-bit). I used JRE 1.8.0_241 because it is the version installed currently on all users' Window PC. If a higher version on the apps server is deployed, all users have to remove the lower version and re-install a matching version on their PC. (Note: later, I used p32140627_180281_WINNT.zip after users' PC got a newer JRE).

$ cd $COMMON_TOP/webapps/oacore/util/javaplugin

$ unzip p32140627_180281_WINNT.zip
Archive:  p32140627_180281_WINNT.zip
  inflating: jre-8u281-windows-i586.exe
  inflating: jre-8u281-windows-i586.tar.gz
  inflating: readme.txt

$ mv jre-8u281-windows-i586.exe j2se18281.exe

$ echo $FILE_EDITION

run

$ $FND_TOP/bin/txkSetPlugin.sh 18281

/u02/app/$TWO_TASK/fs1/EBSapps/appl/fnd/12.0.0/bin

  Starting interoperability upgrade script...

Sending logfile output to:
       $INST_TOP/logs/txkSetPlugin.log
etc … … 

AutoConfig completed successfully.
Done

$ sh $ADMIN_SCRIPTS_HOME/adautocfg.sh    (optional)

Go to the backup folder and confirm the configuration changes:

$ diff $CONTEXT_NAME.xml_BK_JRE $CONTEXT_FILE

<   <config_option type="techstack" oa_var="s_forms_launch_method">browser</config_option>
>   <config_option type="techstack" oa_var="s_forms_launch_method">jws</config_option>

<   <sun_plugin_ver oa_var="s_sun_plugin_ver">1.6.0_27</sun_plugin_ver>
>   <sun_plugin_ver oa_var="s_sun_plugin_ver">1.8.0_281</sun_plugin_ver>

<   <sun_clsid oa_var="s_sun_clsid">CAFEEFAC-0016-0000-0027-ABCDEFFEDCBA</sun_clsid>
>   <sun_clsid oa_var="s_sun_clsid">CAFEEFAC-0018-0000-0281-ABCDEFFEDCBA</sun_clsid>

... ...

$ cat $FORMS_WEB_CONFIG_FILE|grep sun_plugin_version| cut -c 1-35

sun_plugin_version=1.8.0_281

Next is to sign JAR file. Otherwise, launching EBS Forms will give a popup with Exception: JAR resources in JNLP file are not signed by same certificate.

5) Jar signing

Follow Doc ID 1591073.1 (Enhanced Jar Signing for Oracle E-Business Suite)

a) copy two files adkeystore.dat and adsign.txt (from old $APPL_TOP/admin of R12.1.3) to folder  $NE_BASE/EBSapps/appl/ad/admin, assuming JAR signing worked in R12.1.3 instance. Verify basic info:

SQL> set serveroutput on

SQL> declare

spass varchar2(30);

kpass varchar2(30);

begin

ad_jar.get_jripasswords(spass, kpass);

dbms_output.put_line(spass);

dbms_output.put_line(kpass);

end;  

/

$ keytool -list -keystore adkeystore.dat

Enter keystore password:

Keystore type: JKS

Keystore provider: SUN

Your keystore contains 3 entries

ebsdev_nodename, April 10, 2021, PrivateKeyEntry, ...

...  ...

If necessary, use command to create/re-new .dat file from the one for instance1 on the node1:

$ keytool -changealias -alias instance1Name_node1Name -destalias EBSDEV_nodeName -keystore adkeystore.dat

b) Make sure all apps services are stopped

c) $ adadmin

     => 1 (Generate Applications Files menu)
     => 4 (Generate product JAR files)

    Do you wish to force regeneration of all jar files? [No] ? Yes

Forcing generation of all product jar files.
Creating and signing every jar file can take about thirty minutes depending on the hardware being used.
You can watch the file $NE_BASE/EBSapps/log/adadmin/log/Gen_JAR.log to see the progress of jar file generation.

 Recording Adadmin action :ADADMIN_GEN_JARS
 Tokens:FORCE_JAR_GENERATION=Yes

  Signing product JAR files in  JAVA_TOP - $JAVA_TOP
   using entity $CONTEXT_NAME and certificate 1.

Successfully created javaVersionFile.

  Generating product JAR files in JAVA_TOP -
 $JAVA_TOP with command:

adjava -mx2048m -nojit oracle.apps.ad.jri.adjmx @$APPL_TOP/admin/$TWO_TASK/out/genjars.cmd

Successfully generated product JAR files in  JAVA_TOP -

 $JAVA_TOP.

  Copying Registry.dat from the Forms Java directory to $JAVA_TOP ...

Generating customall.jar ...

Not creating customall.jar as no custom java directories found under JAVA_TOP.

customall.jar generated successfully.

   ** Updating appsborg.zip ...

    Reading adjborg.txt...

adearea();
No files listed in $APPL_TOP/admin/adjborg.txt
Ignoring ...
adjborg.txt file seems to be empty.

    Done reading adjborg.txt.
As adjborg.txt file is empty, skipping the  generation of appsborg.zip

   ** Updating appsborg2.zip ...

    Reading adjborg2.txt...

adearea();
No files listed in $APPL_TOP/admin/adjborg2.txt
Ignoring ...
adjborg.txt file seems to be empty.

    Done reading adjborg2.txt.
As adjborg2.txt file is empty, skipping the  generation of appsborg2.zip

Review the messages above, then press [Return] to continue.

Notes from Doc ID 2065496.1:  In release 12.2 the files appsborg.zip and appsborg2.zip have been replaced with file ebsAppsborgManifest.jar. The new file ebsAppsborgManifest.jar should appear in the CLASSPATH. In summary for R12.2 the files appsborg.zip and appsborg2.zip are not used. The messages are expected and can be safely ignored.

$ echo $CLASSPATH | grep ebsAppsborgManifest

will give a long list

$ cd $FMW_HOME/Oracle_EBS-app1/shared-libs/ebs-appsborg/WEB-INF/lib

$ ls -al 

-rw-r--r-- 1 user group 806 Feb 23 16:32 ebsAppsborgManifest.jar

d) To verify the digital signature of the Jar file:

$ jarsigner -verify -verbose -certs $AD_TOP/java/jar/adxlib.jar

… … 

sm      6924 Fri Jan 14 03:00:00 EST 2011 oracle/apps/ad/util/zip/ZipOutputStream.class

      X.509, CN=Company_name Inc., O=Company_name Inc., L=City, ST=State, C=US
      [certificate is valid from 1/3/19 7:00 PM to 1/12/22 7:00 AM]
      X.509, CN=DigiCert SHA2 Assured ID Code Signing CA, OU=www.digicert.com, O=DigiCert Inc, C=US
      [certificate is valid from 10/22/13 8:00 AM to 10/22/28 8:00 AM]
      X.509, CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
      [certificate is valid from 11/9/06 7:00 PM to 11/9/31 7:00 PM]

      X.509, CN=nodename, O=nodename, OU=apps, L=, ST=, C=US
      [certificate is valid from 2/12/21 12:02 PM to 2/2/61 12:02 PM]
      [CertPath not validated: Path does not chain with any of the trust anchors]

  s = signature was verified
  m = entry is listed in manifest
  k = at least one certificate was found in keystore
  i = at least one certificate was found in identity scope

jar verified.

Warning:
This jar contains signatures that does not include a timestamp. Without a timestamp, users may not be able to validate this jar after the signer certificate's expiration date (2022-01-12) or after any future revocation date.

If you see message "This jar contains entries whose certificate chain is not validated.", it had a signing problem.

e) Start apps services by adstrtal.sh (or, start them later after TLS1.2 is enabled). 

After about steps, R12.2.10 webpage and Forms can be launched from MS Edge and Google Chrome. The file download may be slow, and so do not click too quick. If EBS forms does not open up in the 1st launch, you may go the downlead directory in  Window's Explore and double click on file frmservlet.jnlp to fire the Java form up.

6) Useful commands

- To view the keystore contents

$ keytool -list -v -keystore $NE_BASE/EBSapps/appl/ad/admin/adkeystore.dat
Enter keystore password:                     <== get it by a SQL if forgot. 

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: <ebsdev_nodename>      <== all are lower case and $TWO_TASK is EBSDEV
Creation date: Apr 2, 2019
Entry type: PrivateKeyEntry
Certificate chain length: 3
Certificate[1]:

etc ... … 

- To sign a single jar file fndgantt.jar:

$ jarsigner -keystore $NE_BASE/EBSapps/appl/ad/admin/adkeystore.dat $FND_TOP/java/jar/fndgantt.jar ebsdev_nodename
Enter Passphrase for keystore:
Enter key password for <ebsdev_nodename>:
jar signed.

Warning:
No -tsa or -tsacert is provided and this jar is not timestamped. Without a timestamp, users may not be able to validate this jar after the signer certificate's expiration date (2022-01-12) or after any future revocation date.

- Another way to to verify a digital signing by adding option "-keystore":

$ jarsigner -verify -verbose  -keystore $NE_BASE/EBSapps/appl/ad/admin/adkeystore.dat -certs $JAVA_TOP/oracle/apps/fnd/jar/fndaol.jar

- In custom development by Java, use below line to generate and sign mycustom.jar file, assuming mycustom.zip has all .class files and .xml files (and .xml file was imported to database by java oracle.jrad.tools.xml.importer.XMLImporter):

$ adjava oracle.apps.ad.jri.adjmx -areas $JAVA_TOP/mycustom.zip -outputFile $JAVA_TOP/mycustom.jar -jar $CONTEXT_NAME 1 CUST jarsigner -storePass <KeyStore Password> -keyPass <Key Password>

12.2.10 patch 30399999 failed on POS_SUPPLIER_SEARCH_INDEX.sql

After all pre-steps were completed and conditions met the requirement, I downloaded zip files for 12.2.10 patch 30399999 and unzipped to folder /path/to/R12210_Patches. Command line to apply it:

$ adop phase=apply apply_mode=downtime patches=30399999 patchtop=/path/to/R12210_Patches 

ADOP failed and exited with error after ran for hours:

You should check the file

$NE_BASE/EBSapps/log/adop/.../apply/node_name/30399999/log/u30399999.log

for errors.

 [UNEXPECTED]Error occurred executing "adpatch workers=16 options=hotpatch console=no interactive=no defaultsfile=/path/to/R12210_Patches//30399999 driver=u30399999.drv logfile=u30399999.log"

[UNEXPECTED]Refer to the log files for more information.

[UNEXPECTED]Apply phase has failed.

[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.

adop exiting with status = 1 (Fail)

Check adwork001.log file and see the real error:

sqlplus -s APPS/***** @$AD_TOP/patch/115/sql/adsqlwrapper.sql '$POS_TOP/patch/115/sql/POS_SUPPLIER_SEARCH_INDEX.sql '

Connected.

PL/SQL procedure successfully completed.

DECLARE

*

ERROR at line 1:

ORA-20000: possearchindex.sql(500): ORA-20000: Exception at

POS_SUPPLIER_SEARCH_INDEX_PKG.create_index(1800): ORA-20000: Exception at

POS_SUPPLIER_SEARCH_INDEX_PKG.create_index(1800): ORA-29855: error occurred in

the execution of ODCIINDEXCREATE routine

ORA-20000: Oracle Text error:

DRG-10754: Size (in bytes) must be between 1024 and 52428800

ORA-06512: at line 56

Time when worker failed: Fri Sep 24 2021 13:56:21

SQL> select index_name, table_name, index_type, status, domidx_opstatus 

from all_indexes where domidx_opstatus != 'VALID';

        Control

Worker  Code      Context            Filename                    Status

------  --------  -----------------  --------------------------  --------------

     1  Done      AutoPatch R120     POS_SUPPLIER_SEARCH_INDEX.  FAILED

At this stage, we do not want to make any changes to the instance. The only option is to skip this SQL code and then to re-start the adop session. 

Fortunately adctrl still shows the workers' status, while ADOP session already exited by the error. I choose option #8 to skip the failed worker.

After that, I ran below line to re-start from where it failed and it completed:

$ adop phase=apply apply_mode=downtime patches=30399999 patchtop=/path/to/R12210_Patches restart=yes

$ vi u30399999.log

... ...

Worker process 15 started.

Worker process 16 started.

Checking if all jobs have their actual and symbolic arguments in sync....

Done.

Reading jobs from FND_INSTALL_PROCESSES table ...

    Fixed: file POS_SUPPLIER_SEARCH_INDEX.sql on worker  1 for product pos username POS.

Time is: Fri Sep 24 2021 16:22:18

Done reading jobs from FND_INSTALL_PROCESSES table ...

Telling workers to read 'todo' restart file.

Done.

Completed: file POS_SUPPLIER_SEARCH_INDEX.sql on worker  1 for product pos username POS.

Time is: Fri Sep 24 2021 16:22:18

Connecting to APPS......Connected successfully.

Even u30399999.log says file POS_SUPPLIER_SEARCH_INDEX.sql was completed. But the worker log shows it was skipped. Index table still shows a failure as well.

$ vi adwork001.log

... ...

Ready to run jobs.

Setting parallel context: PARALLEL AUTOPATCH AT R120

Restarting job that failed - will skip a single step.

Time when worker restarted job: Fri Sep 24 2021 16:22:18

Skipping step POS_SUPPLIER_SEARCH_INDEX.sql.

Step skipped at time: Fri Sep 24 2021 16:22:18

Time when worker completed job: Fri Sep 24 2021 16:22:18

SQL> select index_name, table_name, index_type, status, domidx_opstatus 

from all_indexes where domidx_opstatus != 'VALID';

INDEX_NAME                                 TABLE_NAME                            INDEX_TYPE  STATUS   DOMIDX_OPSTATUS

---------------------------------------------  --------------------------------------------   ------------------  ------------   ----------------------------

POS_SUPPLIER_SEARCH_INDEX POS_SUPPLIER_ENTITY_DATA   DOMAIN       VALID      FAILED

I believe our instance does not use this index and just ignored it. I do not know exactly what causes the failure on creating the index. I believe it relates to the 0 value on two database init parameters:

SQL> show parameter PGA_AGGREGATE_TARGET;

NAME                             TYPE        VALUE

----------------------------- ----------- ------------------------------

pga_aggregate_target     big integer     0          (Note: 0 means unlimited ) 

SQL> show parameter SGA_TARGET;

NAME                                 TYPE        VALUE

------------------------------ ----------- ------------------------------

sga_target                        big integer     0

Run ADSplice in R12.2

ADSplice is the only way to create custom tops in EBS R12.2. Steps of running it is in Doc ID 1577707.1 (Creating a Custom Application in Oracle E-Business Suite Release 12.2). If it is a multi-node instance, you have to run it on each node.  

1. Before running ADSplice, make sure latest AD and TXK for R12.2 patches were applied already:

SQL> select abbreviation ,codelevel from ad_trackable_entities 

where abbreviation in( 'ad','txk') order by abbreviation;

ABBREVIATION CODELEVEL

--------------------- ---------

ad                         C.12

txk                        C.12

2. For a regular/normal custom top, it is not difficult to follow the Oracle document for creating or modifying it. Three files are required by ADSplice in folder $APPL_TOP/admin for one custom top $XXFS_TOP, for example

xxefprod.txt
xxefterr.txt
newprods.txt

3. Then run adsplice. It will ask for SYSTEM and APPS password. It also give option to run AutoConfig. So, it is better to shut down apps services before running adsplice.

 

$ echo $FILE_EDITION

run

$ cd $APPL_TOP/admin

$ adsplice

ADSplice will create folder structure for $XXFS_TOP under $APPL_TOP. 

For multiple nodes, run adsplice on all nodes. After 1st node, adsplice runs much faster because it knows all database part was completed.

4. Most custom top has custom codes/files. Migrate custom codes/files to the correct new folder, and then compile them or link them to right executable files.

5. After ADSplice worked, add entries for all your custom files to the custom synchronization driver file located at $APPL_TOP_NE/ad/custom/adop_sync.drv. I used below line which works well in copying folder xxfs in <s_current_base> (source) to <s_other_base> (target)

rsync -a --delete %s_current_base%/EBSapps/appl/xxfs %s_other_base%/EBSapps/appl

The option -a is very powerful, it will copy the directory and all its sub-directories recursively, and preserve symbolic links, modification times, group, ownership, and permissions. It is a combination of a bunch of other options:

-r  Recurse into directories  

-l  Copy symlinks as symlinks

-p  Preserve file permissions

-t  Preserve modification times

-g  Preserve group

-o  Preserve owner

-D  Preserve device and special files

-- delete option will delete files and sub-directories in target folder but not in source folder.

6. Run FS_CLONE

ADSplice creates folders only in RUN Edition file system. You must run "adop phase=fs_clone". This ensures that all custom products are synchronized/copied to PATCH file system. Doc ID 2167878.1 indicates FS_CLONE does use adop_sync.drv.

I ran into a error with "adop phase=prepare". ADOP does not tell what is the specific problem at all!

=============================

Inside evalADPATCHStatus()...

=============================

message_status: ERROR

Adsplice action did not go through successfully.

*******FATAL ERROR*******

PROGRAM : ($AD_TOP/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)

TIME    : Wed Jul 20 14:00:36 2022

FUNCTION: main::execADSPLICE [ Level 1 ]

ERRORMSG: Adsplice action did not go through successfully. 

Before the error, log file txkADOPPreparePhaseSynchronize.log posts a SQL statement similar to below lines to check table AD_ADOP_SESSION_PATCHES:

SELECT * FROM ad_adop_session_patches 

WHERE bug_number like 'ADSPLICE%' AND 

ORDER BY TO_CHAR(end_date,'YYYY.MM.DD:HH24:MI:SS') DESC

Without running fs_clone, column PATCH_FILE_SYSTEM_BASE is empty for one CUSTOM TOP. I think that causes ADOP PREPARE failed.

After "adop phase=prepare" failed, "adop phase=fs_clone" will not work. You have to below steps in order:

$ adop phase=abort

$ adop phase=cleanup cleanup_mode=full

$ adop phase=fs_clone

Notes: Similarly, anytime after a change was made in RUN file system, e.g. a patch applied in HOTPATCH mode, you have to run "adop phase=fs_clone" to sync the change to PATCH file system.

7. A useful query to verify if a new custom top in R12.2 was created by ADSplice 

SQL> set linesize 300

set pagesize 300

col APPL_ID format 99999;

col app_short_name format a15

col basepath format a 15

col basepath format a15

col application_name format a35

col inst_status format a10

col product_version a20

col product_version format a20

col patchset format a25

col oracle_id format 99999

SQL> SELECT fav.application_id APPL_ID, fav.application_short_name app_short_name, fav.basepath, fav.application_name,

decode(fpi.status,'I','Installed','S','Shared','N','No', fpi.status) inst_status,  fpi.product_version,

nvl(fpi.patch_level, 'Not Available') patchset, fpi.oracle_id

FROM apps.fnd_application_vl fav, apps.fnd_product_installations fpi

WHERE fav.application_id = fpi.application_id

and (fav.application_short_name like 'XX%' )

ORDER BY app_short_name asc;

- TROUBLESHOOTING and lessons learned on ADSplice:

1. We had a custom top that was created in 9i with mismatch between SHORT_NAME and BASEPATH (the name of custom top):

SQL> select ORACLE_USERNAME,ORACLE_ID, dba.USERNAME

   from fnd_oracle_userid fnd, dba_users dba

 where fnd.oracle_username = dba.username and lower(oracle_username) in ('xxfs') ;

ORACLE_USERNAME ORACLE_ID USERNAME 

------------------------------ ----------------  ---------------- 

XXFS                               20024             XXFS             

SQL> select application_id,application_short_name short_name, 

last_update_date, basepath, product_code

   from fnd_application

 where application_short_name like '%XXFS%' or application_id = 20024;  

APPLICATION_ID SHORT_NAME LAST_UPDATE BASEPATH PRODUCT_CODE

------------------------ -------------------- -------------------- --------------- ------------------------

                     20024                XXFFS    16-DEC-97       XXFS_TOP  XXFFS

We tried to keep it as the old way to minimize the re-write of custom codes, by putting below lines in file xxefprod.txt

# Product specific Product ID's

xxfs     20024

END_OF_PRODUCT_ABBREVIATIONS   -999

#

# ##################################################

# Oracle Support Diagnostic Tools

# ##################################################

#

# application id, abbreviation, shortname, prefix

20024 aefs XXFFS APP

# multiple product installations for msob, "controlled release", shared only

#   optional fourth field is "stub product".  default is No

No No No No

# multilingual, has NLADD.sql

Yes Yes

# install oracle id, default ORACLE username, default ORACLE password

20024 XXFS XXFS

Note: In above line, XXFS is not necessarily the current db password. If it does not exist in database, adsplice will create it.

ADSplice does not complain it and created $XXFS_TOP. But when I later applied patches, "adop phase=prepare" failed with error:

Custom product XXFS entry is NOT there in FND_APPLICATIONS table

ERROR: The following custom products have not been correctly added: XXFS

We also tried to created two custom tops $XXFFS_TOP and $XXFS_TOP (with a new application id assigned, such as 30034), while they share same existing database account XXFS.  ADSplice also created two without complain. But "adop phase=prepare" failed again without giving a specific error:

message_status: ERROR

Adsplice action did not go through successfully.

*******FATAL ERROR*******

PROGRAM : ($AD_TOP/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)

TIME    : Wed Xxx  5 11:52:14 2021

FUNCTION: main::execADSPLICE [ Level 1 ]

ERRORMSG: Adsplice action did not go through successfully.

I had to de-register $XXFFS_TOP first and then de-register $XXFS_TOP by using 

$ perl $AD_TOP/bin/adDeregisterCustomProd.pl

After de-register them, ADOP worked in the full patching cycle.

Before that, I tried the suggestion in Doc ID 1932207.1 (Adop Prepare ERRORMSG: Adsplice Action Did Not Go Through Successfully After Add new Custom Application in R12.2). But I was not sure (do not remember) it helped or not.

SQL> UPDATE FND_ORACLE_USERID SET ENABLED_FLAG = 'N'

WHERE ORACLE_USERNAME in ('XXFS') ;    -- from "(null)" 

2. "perl $AD_TOP/bin/adDeregisterCustomProd.pl" will delete data from tables FND_OAM_CONTEXT_CUSTOM, FNS_APPLICATION. But it will not delete the row in table FND_PRODUCT_INSTALLATION.

3. If the password of Oracle database account XXEF was changed not by FNDCPASS, ADSplice will give ORA-01017 error.  The fix is to use below line to change its password:

$ FNDCPASS apps/apps_PWD 0 Y system/'systemPWD' ORACLE xxef xxef_PWD

4.  "adop phase=abort,cleanup cleanup_mode=full" failed with error message:

AutoPatch - aidafoGetFileDbVersion: INFO: ORA-01403: no data found

AutoPatch error:

Unable to get APPS_DDL package version from db <XXFS> <APPS_DDL> <PACKAGE>

AutoPatch error:

adpmrp: Error while installing apps_ddl packages.

Run below query to see its status.  The fix is to run an UPDATE on the row. 

SQL> select oracle_username, read_only_flag from fnd_oracle_userid 

 where read_only_flag <> 'A';

ORACLE_USERNAME  READ_ONLY_FLAG

------------------------------ ---------------------------

APPLSYS                        E

APPS                               U

XXFS                               D

APPLSYSPUB                C

APPS_NE                        Z

SQL> update fnd_oracle_userid set read_only_flag='A' 

where ORACLE_USERNAME='XXFS';

Then "adop phase=abort" worked.  (Note: In my another instance, it has the same status, but ADOP does not fail).

5. After a custom top is created, if it is used by a Responsibility or concurrent jobs, run autoconfig and bounce apps services to avoid error/message:

APP-FND-00362: Routine afpbep cannot execute request &REQUEST for program &PROGRAM, because the environment variable &BASEPATH is not set for the application to which the concurrent program executable &EXECUTABLE belongs.

Shut down the concurrent managers. Set the basepath environment variable for the application. Restart the concurrent managers. (VARIABLE=)

Routine AFPEOT cannot construct the name of an executable file for your concurrent request 430346.

Check that the file name components are correct and valid on your system. Check that the environment for the person who started the concurrent manager 

6. R12.2 GUI forms will not create a custom top, because navigation Application => Register gives message:

Forms Registration for Custom Application or Schema is now blocked. Please consult ADSplice documentation for the current standards for deploying Customizations in EBS

Doc ID 2272551.1 says this is intended functionality. Application Forms cannot be used to add or remove applications. This is restricted in 12.2 and going forward.

7. If a custom top is not necessary but an env variable is needed as a pointer to a folder under $APPL_TOP, modify file $FND_TOP/fndenv.env by adding lines:

#Begin Customizations

AEUT_TOP=${APPL_TOP}/xxedi/12.0.0; export XXEDI_TOP

#End Customizations

With the use of adsplice, I am not sure if modifying Forms file is necessary in R12.2

$INST_TOP/ora/10.1.2/forms/server/default.env:

#Begin Customizations

XXEDI_TOP=/path/to/appl/xxedi/12.0.0

#End Customizations

Related Documents: 

How to remove a CUSTOM top  (Doc ID 1903037.1)

How To De-register Custom Applications (Doc ID 2085355.1)

Use adop to apply AD & TXK patches, and R12.2.10 patch

Steps in Doc ID 1617461.1 (Applying the Latest AD and TXK Release Update Packs to Oracle E-Business Suite Release 12.2) using ADOP and to complete R12.2.10 upgrade.

1). Checked database parameters for R12.2 (Doc ID: 396009.1) in 12c database:

SQL>  select SERVICE_ID,NAME from dba_services where name='ebs_patch';

SERVICE_ID NAME

---------- ------------------------

         8 ebs_patch

SQL> show parameter recyclebin

NAME                               TYPE        VALUE

-------------------------------- ----------- ------------

recyclebin                           string        OFF

SQL> show parameter _SYSTEM_TRIG_ENABLED

NAME                            TYPE       VALUE

----------------------------- ----------- ------------

_system_trig_enabled     boolean     TRUE

SQL> show parameter optimizer_adaptive_features

NAME                                      TYPE      VALUE

------------------------------------ ----------- --------------

optimizer_adaptive_features    boolean     FALSE

SQL> show parameter _disable_actualization_for_grant

NAME                                            TYPE        VALUE

------------------------------------------ ----------- ----------------

_disable_actualization_for_grant     boolean     TRUE

SQL> select fnd_profile.value('APPS_MAINTENANCE_MODE') from dual;

FND_PROFILE.VALUE('APPS_MAINTENANCE_MODE')

--------------------------------------------------------------------------

NORMAL

SQL> set lines 333 pages 123;

col ABBREVIATION for a12;

col NAME for a45;

col codelevel for a12;

SQL> select  ABBREVIATION, NAME, codelevel FROM AD_TRACKABLE_ENTITIES 

where abbreviation in ('txk','ad');

ABBREVIATION   NAME                                  CODELEVEL

------------ --------------------------------------------- ------------

ad           Applications DBA                                       C

txk         Oracle Applications Technology Stack        C

SQL> select node_name from apps.fnd_nodes;

NODE_NAME

------------------------------

db_nodename

AUTHENTICATION

apps_nodename

SQL> select node_name from apps.TXK_TCC_RESULTS;

NODE_NAME

------------------------------------------------------------------------

db_nodename.domain_name.com         

                         <== Should have db_nodename only

                         <== Not sure, but July 2020 db PSU 31113348 may fix the issue

$ grep s_dbhost $CONTEXT_FILE

         <dbhost oa_var="s_dbhost">db_nodename</dbhost>       <== mis-match

2). run checkDBpatch.sh

3). run checkMTpatch.sh 

$ . $APPL_TOP/APPS$CONTEXT_NAME.env       <= Source the env file

$ echo $FILE_EDITION

run

$ echo $RUN_BASE

/u02/app/$TWO_TASK/fs1

$ ./checkMTpatch.sh

Notes: without 2) and 3),  later ADOP may report error even two scripts were previously executed:

Validating system setup.

    Node registry is valid.

    [ERROR]     ETCC not run in the database node db_nodename

                The EBS Technology Codelevel Checker needs to be run on the database node.

                It is available as Patch 17537119.

    [WARNING]   MTCC has not been run. Apply Patch 17537119 and run checkMTpatch.sh.

    Encountered the above errors when performing database validations.

    Resolve the above errors and restart adop.

If the issue is persistent, check a solution in Doc ID 2191678.1

SQL> select node_name from apps.TXK_TCC_RESULTS;

4). Start WebLogic Admin Server

$ sh $ADMIN_SCRIPTS_HOME/adadminsrvctl.sh start

NMProcess: <Mn 7, 2021 4:47:14 PM> <INFO> <Loading domains file: $FMW_Home/wlserver_10.3/common/nodemanager/nmHome1/nodemanager.domains>

NMProcess: Mn 07, 2021 4:47:14 PM weblogic.nodemanager.server.NMServerConfig initDomainsMap

NMProcess: INFO: Loading domains file: $FMW_Home/wlserver_10.3/common/nodemanager/nmHome1/nodemanager.domains

NMProcess: <Mn 7, 2021 4:47:14 PM> <INFO> <Loaded node manager configuration properties from '$FMW_Home/wlserver_10.3/common/nodemanager/nmHome1/nodemanager.properties'>

NMProcess: Mn 07, 2021 4:47:14 PM weblogic.nodemanager.server.NMServer <init>

NMProcess: INFO: Loaded node manager configuration properties from '$FMW_Home/wlserver_10.3/common/nodemanager/nmHome1/nodemanager.properties'

NMProcess: Node manager v10.3

NMProcess:

NMProcess: Configuration settings:

... ...

adadminsrvctl.sh: exiting with status 0

adadminsrvctl.sh: check the logfile $LOG_HOME/appl/admin/log/adadminsrvctl.txt for more information ...

$ grep wls_admin $CONTEXT_FILE

5). Test WebLogic console works by login as "weblogic".

http://server_name.domain.com:s_wls_adminport/console

(After logged in, see "Warning: This server is not secure. Upgrade to Java 1.7.0_191 or higher.")

Below line shall also work:

$ wget http://server_name.domain.com:s_wls_adminport/console

… … 

100%[===============================>] 3,162       --.-K/s   in 0s

2021-Mn-12 10:25:54 (392 MB/s) - ‘console’ saved [3162/3162]

$ ps -ef | grep -i fs1 | wc -l

shall be 5 new sessions

Notes:  at this moment, environment variables PATCH_TOP and NE_BASE are not defined yet, and ADOP status gives error because it has no history yet:

$ adop -status

Enter the APPS username: apps

Enter the APPS password:

where adop_session_id=;

                      *

ERROR at line 6:

ORA-06550: line 6, column 23:

PL/SQL: ORA-00936: missing expression

ORA-06550: line 4, column 1:

PL/SQL: SQL Statement ignored

adop exiting with status = 1 (Fail)

6). Download/copy patches' .zip files to 

$ cd /path/to/$TWO_TASK/fs_ne/EBSapps/patch

$ ls

$ unzip p30628681_R12.AD.C_R12_LINUX.zip

7). Run 30628681/admin/adgrants.sql as SYSDBA. 

(confirmed the same version was executed before and so skipped this step)

8). Apply patch 30628681   (Do not merge it with any other patches)

$ echo $RUN_BASE

/u02/app/$TWO_TASK/fs1

$ echo $FILE_EDITION

run

$ pwd

/path/to/$TWO_TASK/fs_ne/EBSapps/patch

$ adop phase=apply patches=30628681 hotpatch=yes patchtop=/path/to/$TWO_TASK/fs_ne/EBSapps/patch

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

 Please wait. Validating credentials...

Enter the RUN file system context file name [$RUN_BASE/inst/apps/$CONTEXT_NAME/appl/admin/$CONTEXT_NAME.xml]:

[STATEMENT] [END   2021/0X/12 11:53:01] Performing verification of parameters

[STATEMENT] [START 2021/0X/12 11:53:01] Checking for the required ENV setup

[STATEMENT] [END   2021/0X/12 11:53:01] Checking for the required ENV setup

************* Start of  session *************

 version: 12.2.0

 started at: Fri XXX 12 2021 11:53:02

… … 

[STATEMENT] ADPATCH Log directory: $NE_BASE/EBSapps/log/adop/2/apply_20210X12_115025/$CONTEXT_NAME/30628681/log

[STATEMENT] Running: adpatch  workers=24    options=hotpatch    flags=autoskip   console=no interactive=no  defaultsfile=$APPL_TOP/admin/$TWO_TASK/adalldefaults.txt patchtop=$NE_BASE/EBSapps/patch/30628681 driver=u30628681.drv logfile=u30628681.log stdin=yes

Successfully created out directory        <== stay here for about 10 minutes

                                                                      If it takes hours, something was wrong.

    [STATEMENT] [START 2021/09/14 13:32:57] Running finalize since in hotpatch mode

    [STATEMENT] [END   2021/09/14 13:36:22] Running finalize since in hotpatch mode

    [STATEMENT] [START 2021/09/14 13:36:23] Performing Automatic DB cutover

… … 

[STATEMENT] adop phase=apply - Completed Successfully

[STATEMENT] Log file: $NE_BASE/EBSapps/log/adop/2/adop_20210X12_115025.log

adop exiting with status = 0 (Success)

Check logs to confirm it had no issue. It takes about 15 minutes.

Notes: Found ADOP does not like SYSTEM password has special char "!" at the end, with error:

ERROR: Invalid system user credentials.

Warning: Your system may have a limit on the number of failed login attempts before the account is locked out.

9). Use hotpatch mode to apply Patch 30735865 (R12.TXK.C.Delta.12) plus any additional critical patches (which is patch 31904550:R12.TXK.C for addressing fs_clone issue. See Section 4.2). 

$ unzip p30735865_R12.TXK.C_R12_GENERIC.zip

$ unzip p31904550_R12.TXK.C_R12_GENERIC.zip

$ echo $RUN_BASE

/u02/app/$TWO_TASK/fs1

$ adop phase=apply patches=30735865,31904550 hotpatch=yes merge=yes patchtop=/path/to/$TWO_TASK/fs_ne/EBSapps/patch

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

Validating credentials.

Initializing.

    Run Edition context  : $CONTEXT_FILE

    Patch edition context: /path/to/fs2/inst/apps/…/$CONTEXT_NAME.xml

    Patch file system free space: 89.36 GB

Validating system setup.

    Node registry is valid.

    [WARNING]   MTCC has not been run. Apply Patch 17537119 and run checkMTpatch.sh.

Checking for existing adop sessions.

    Application tier services are down.

    Continuing with the existing session [Session ID: 2].

==============================================

ADOP (C.Delta.12)

Session ID: 2

Node: node_name

Phase: apply

Log: $NE_BASE/EBSapps/log/adop/2/20210X15_094233/adop.log

==============================================

Starting patch merge.

    Merging patches where applicable.

    Log: $NE_BASE/EBSapps/log/adop/2/20210X15_094233/apply/node_name/admrgpch.log

Applying patch ADOP_MRG_20210X15_1613400209:u_1809065289.drv.

    Log: $NE_BASE/EBSapps/log/adop/2/20210X15_094233/apply/node_name/ADOP_MRG_20210X15_1613400209/log/u_1809065289.log

        New product CMI spliced. Use FNDCPASS to change the password.

        New product YMS spliced. Use FNDCPASS to change the password.      

                                                                       <= it takes about 15 minutes after this line totally

Running finalize actions for the patches being applied.

    Log: @ADZDSHOWLOG.sql "2021/0X/15 09:54:40"

Running cutover actions for the patches being applied.

    Creating workers to process cutover DDL in parallel

    Log: $NE_BASE/EBSapps/log/adop/2/20210X15_094233/apply/node_name/log/cutover.log

    Performing database cutover in Quick mode

Generating post apply reports.

Generating log report.

    Output: $NE_BASE/EBSapps/log/adop/2/20210X15_094233/apply/node_name/adzdshowlog.out

The apply phase completed successfully.

adop exiting with status = 0 (Success)

$ egrep -i 'fail|error|ora-' u_xxx.log

$ egrep -i 'fail|error|ora-' 

$NE_BASE/EBSapps/log/adop/2/20210X15_094233/apply/node_name/adzdshowlog.out

It reports ORA- errors on material view MRP_COMPANY_USERS_SN. This error can be ignored per Doc ID 1957323.1.

Notes: if ETCC version was old and some databases patches were not applied, ADOP will give warnings, e.g.

    [WARNING]   ETCC: The following required database fixes have not been applied to node db_nodename:

                  21322448

                  22331786

                  31404014

                  31550110

                Refer to My Oracle Support Knowledge Document 1594274.1 for instructions.

10). Confirm all worked

$ adop -status

Enter the APPS password:

Connected.

==================================================

ADOP (C.Delta.12)

Session Id: 2

Command: status

Output: $NE_BASE/EBSapps/log/adop/2/20210X15_100245/adzdshowstatus.out

=============================================================

Node Name       Node Type  Phase           Status          Started              Finished             Elapsed

--------------- ---------- --------------- --------------- -------------------- -------------------- ------------

$EBS_HOSTNAME          master    APPLY          ACTIVE          2021/0X/12 11:53:31  2021/0X/15 09:56:28  70:02:57

                                                          FINALIZE     NOT STARTED

                                                          CLEANUP     NOT STARTED

File System Synchronization Type: None

adop exiting with status = 0 (Success)

Use query to confirm 3 patches were applied:

SQL> select ad_patch.is_patch_applied('R12',-1,30735865) from dual;

SQL> select ad_patch.is_patch_applied('R12',-1,31904550) from dual;

SQL> select ad_patch.is_patch_applied('R12',-1,30628681) from dual;

SQL> select ADOP_SESSION_ID, BUG_NUMBER, STATUS, APPLIED_FILE_SYSTEM_BASE, 

ADPATCH_OPTIONS, round((end_date-start_date)*24*60, 1) EXEC_TIME, AUTOCONFIG_STATUS,  DRIVER_FILE_NAME, NODE_NAME, 

END_DATE, CLONE_STATUS

from ad_adop_session_patches

order by end_date desc;

30735865

31904550

30628681

SQL> set lines 333 pages 123;

col ABBREVIATION for a12;

col NAME for a45;

col codelevel for a12;

SQL> select  ABBREVIATION, NAME, codelevel FROM AD_TRACKABLE_ENTITIES where abbreviation in ('txk','ad','fnd','fwk','atg_pf','icx');

ABBREVIATION NAME                                          CODELEVEL

------------ --------------------------------------------- ------------

ad             Applications DBA                                   C.12

atg_pf      Applications Technology                         C

fnd           Oracle Application Object Library           C

fwk          Applications Framework                          C

icx           Oracle iProcurement                               D

txk           Oracle Applications Technology Stack   C.12

6 rows selected.

SQL> select release_name from apps.fnd_product_groups;

RELEASE_NAME

------------------------------

12.2.0

SQL> select node_name from apps.fnd_nodes;

NODE_NAME

------------------------------

db_nodename

AUTHENTICATION

apps_nodename

SQL> col EDITION_NAME for a15;

col PARENT_EDITION_NAME for a20;

col USABLE for a10;

SQL> select * from dba_editions;

EDITION_NAME  PARENT_EDITION_NAME USABLE

----------------------- ------------------------------------- ----------

ORA$BASE                                                              YES

11). Activate the new env file

Now, in a new session, PATCH_TOP and NE_BASE are setup, and new env file EBSapps.env is created under /path/to/$TWO_TASK. Add below line to .profile (of Linux applmgr account), and then start a new OS session:

. /u02/app/$TWO_TASK/EBSapps.env RUN

12). Run autoconfig to update Database Tier to latest code (Section 5 of Doc ID 1617461.1)

$ ./checkMTpatch.sh

No new patches are asked by the script.

Stop Weblogic Server Admin Server:

$ sh $ADMIN_SCRIPTS_HOME/adadminsrvctl.sh stop   

You are running adadminsrvctl.sh version 120.10.12020000.2

Enter the WebLogic Admin password:

Enter the APPS Schema password:

Stopping WLS Admin Server...

Refer $LOG_HOME/appl/admin/log/adadminsrvctl.txt for details

adadminsrvctl.sh: exiting with status 0

$ ps -ef | grep $LOGNAME

Stop WebLogic Server Node Manager service:

$ sh $ADMIN_SCRIPTS_HOME/adnodemgrctl.sh stop

You are running adnodemgrctl.sh version 120.11.12020000.4

Enter the WebLogic Admin password:

adnodemgrctl.sh: exiting with status 0

adnodemgrctl.sh: check the logfile $LOG_HOME/appl/admin/log/adnodemgrctl.txt for more information ...

$ ls -lrt $AD_TOP/bin/admkappsutil.pl

-rwxr-xr-x 1 user group 7232 Nov 24  2012 /u02/app/$TWO_TASK/fs1/EBSapps/appl/ad/12.0.0/bin/admkappsutil.pl

$ perl $AD_TOP/bin/admkappsutil.pl

Starting the generation of appsutil.zip

Log file located at $INST_TOP/admin/log/MakeAppsUtil_02151153.log

output located at $INST_TOP/admin/out/appsutil.zip

MakeAppsUtil completed successfully.

DBA to run adautocfg.sh on <RDBMS ORACLE_HOME> after unzip appsutil.zip.

a) Source the environment for RDBMS ORACLE_HOME

b) Uncompress appsutil.zip, under <RDBMS ORACLE_HOME>.

c) Run AutoConfig on <RDBMS ORACLE_HOME>

 $ sh <RDBMS_ORACLE_HOME>/appsutil/scripts/<CONTEXT_NAME>/adautocfg.sh

13) Run autoconfig on Apps node, After autoconfig completed on database node:

$ echo $FILE_EDITION

run

$ cd $ADMIN_SCRIPTS_HOME

$ ./adautocfg.sh

… … 

AutoConfig completed successfully.

SQL> select node_name from apps.fnd_nodes;

NODE_NAME

------------------------------

db_nodename

AUTHENTICATION

apps_nodename

14) Run ADSplice to create CUSTOM TOPs. Steps are in Doc ID 1577707.1 (Creating a Custom Application in Oracle E-Business Suite Release 12.2). 

SQL> select * from FND_OAM_CONTEXT_CUSTOM;

no row returned 

$ cd $APPL_TOP/admin

$ copy .txt files to here (from an instance where adsplice worked)

$ adsplice

... ...

If you added new products, use FNDCPASS to change their default passwords.

AD Splicer is complete.

Errors and warnings are listed in the log file $APPL_TOP/admin/$TWO_TASK/log/adsplice.log

and in other log files in the same directory.

New entries shall show up in below queries:

SQL> select * from FND_OAM_CONTEXT_CUSTOM;

SQL> select fav.application_id APPL_ID, fav.application_short_name app_short_name, fav.basepath, fav.application_name,

decode(fpi.status,'I','Installed','S','Shared','N','No', fpi.status) inst_status,  fpi.product_version,  

nvl(fpi.patch_level, 'Not Available') patchset, fpi.oracle_id

from apps.fnd_application_vl fav, apps.fnd_product_installations fpi  

where fav.application_id = fpi.application_id 

order by app_short_name asc;

Backup Apps file system & database (Recommended)

15). Applied RUP (Release Update Pack - R12.2.10) patch 30399999 (See Doc ID 2666934.1) 

a) download 11 .zip files from patch 30399999 to folder /path/to/R12_2_10

b) run "$ unzip unzip p30399999_R12_LINUX_Xof11.zip" on each of 11 zip files

c) copy software to $PATCH_TOP

    $ cd $PATCH_TOP

    $ cp -rp /path/to/R12_2_10/30399999 .

    $ du -sk 30399999

       12981020        30399999

Notes: patching process unzipped some files in the installation software folder. After it is applied:

    $ du -sk 30399999

      21489484        30399999

d) confirm nothing is running on Apps node.

     $ ps -ef | grep $LOGNAME

e) Apply patch 30399999

$ echo $FILE_EDITION

   run

$ adop phase=apply apply_mode=downtime patches=30399999 patchtop=/path/to/PATCH_TOP

                                                                 <== It takes almost 7 hours

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

Validating credentials.

Initializing.

    Run Edition context  : $CONTEXT_FILE

    Patch edition context: $CONTEXT_FILE on /fs2/

    Patch file system free space: 76.62 GB

Validating system setup.

    Node registry is valid.

Checking for existing adop sessions.

    Application tier services are down.

    Continuing with the existing session [Session ID: 2].

========================================================

ADOP (C.Delta.12)

Session ID: 2

Node: node_name

Phase: apply

Log: $NE_BASE/EBSapps/log/adop/2/20210X16_155144/adop.log

========================================================

Applying patch 30399999.

    Log: $NE_BASE/EBSapps/log/adop/2/20210X16_155144/apply/node_name/30399999/log/u30399999.log

                  <== It takes upto 7 hours. adctrl can be used to check workers.

                          After about two hour, it runs adutlrcmp.sql for about 30 minutes.

                          After about 4 hours, it re-runs adutlrcmp.sql for about 20 minutes

Running finalize actions for the patches being applied.

    Log: @ADZDSHOWLOG.sql "2021/0X/16 20:55:04"

Running cutover actions for the patches being applied.

    Creating workers to process cutover DDL in parallel

    Log: $NE_BASE/EBSapps/log/adop/2/20210X16_155144/apply/node_name/log/cutover.log

    Performing database cutover in Quick mode

Generating post apply reports.

Generating log report.

    Output: $NE_BASE/EBSapps/log/adop/2/20210X16_155144/apply/node_name/adzdshowlog.out

The apply phase completed successfully.

adop exiting with status = 0 (Success)

$ adopscanlog -latest=yes              <== its message is not easy to read!

SQL> select release_name from apps.fnd_product_groups;

RELEASE_NAME

------------------------------

12.2.10

SQL> set lines 333 pages 123;

col owner for a27

col object_name for a45;

col object_typr for a27;

SQL> select owner,object_type,count(*) from dba_objects where status <> 'VALID'

group by owner,object_type order by owner, object_type;

OWNER                       OBJECT_TYPE            COUNT(*)

--------------------------- ----------------------------- ----------

APPS                        PACKAGE                          1

APPS                        PACKAGE BODY              19

APPS                        PROCEDURE                     2

APPS                        VIEW                                  7

APPS_NE                 TYPE BODY                      2

PUBLIC                    SYNONYM                       4181

... ... ... 

SQL> select * from dba_editions;

return one line.

Verify and confirm below 4 important patches were applied:

SQL> SELECT adb.bug_number, adb.creation_date, 

ad_patch.is_patch_applied('R12',-1, adb.bug_number) status

FROM ad_bugs adb

WHERE adb.bug_number in ('30628681', '30735865', '30399994', '30399999');

BUG_NUMBER CREATION_DATE STATUS

--------------------- ----------------------- -----------

30399994   16-FEB-21  EXPLICIT

30399999   16-FEB-21  EXPLICIT

30628681   12-FEB-21  EXPLICIT

30735865   15-FEB-21  EXPLICIT

30399994 Patch R12.ATG_PF.C.delta.9 (Application Technology Family)

30399999 R12.2.10

30628681 Patch R12.AD.C.delta.12  (Application DBA)

30735865 Patch R12.TXK.C.delta.12  (Application Technology Stack)

16). Start R12.2 services

$ sh $ADMIN_SCRIPTS_HOME/adstrtal.sh apps/apps_PWD

It stayed long and may post errors in some instances from running "adadminsrvctl.sh start -nopromptmsg":

Executing service control script: $INST_TOP/admin/scripts/adadminsrvctl.sh start -nopromptmsg

Timeout specified in context file: -1 second(s)

script returned:

************************

You are running adadminsrvctl.sh version 120.10.12020000.11

Starting WLS Admin Server...

Refer $INST_TOP/logs/appl/admin/log/adadminsrvctl.txt for details

AdminServer logs are located at $FMW_HOME/user_projects/domains/EBS_domain_${TWO_TASK}/servers/AdminServer/logs

adadminsrvctl.sh: exiting with status 0

adadminsrvctl.sh: check the logfile $INST_TOP/logs/appl/admin/log/adadminsrvctl.txt for more information ...

.end std out.

java.rmi.NoSuchObjectException: The object identified by: '299' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.

        at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:237)

        at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:223)

        at javax.management.remote.rmi.RMIConnectionImpl_1036_WLStub.getMBeanCount(Unknown Source)

        at weblogic.management.remote.common.RMIConnectionWrapper$11.run(ClientProviderBase.java:782)

       … … … 

.end err out.

During services startup, CPU usage is high on the server. Seems adadminsrvctl.sh did some configuration in the first execution. In its 2nd run, the Java errors did not show up again. And the errors do not show always up in my other instances. The Java errors match with Doc ID 2335354.1. But there are many folders called nodemanager. I do not know yet which file causes the problem.

$ find . -type d -name nodemanager 

After adstrtal.sh finished, I was able to log onto EBS R12.2 site. The first page is Secure Configuration Console. Click on Unlock and choose Ignore .., then click on Proceed. 

At this stage, after site http URL is added to Java Security tab, EBS Forms and concurrent jobs work only in IE browser. MS Edge does not open the Forms because default R12.2 JRE version 1.6 is lower than the JRE version on my PC.

There are two methods to access the Secure Configuration Console home page later (See https://docs.oracle.com/cd/E26401_01/doc.122/e22952/T156458T663583.htm).

. Using the Functional Administrator responsibility:

On the Oracle E-Business Suite home page, select the Functional Administrator responsibility in the Navigator pane. Then, on the Functional Administrator page, select the Configuration Manager tab.

. Using the OAM Security Dashboard:

On the Oracle E-Business Suite home page, select System Administrator in the Navigator pane. Then select Oracle Applications Manager, then OAM Security Dashboard. On the dashboard, under the Configuration Management section is a link to the Secure Configuration Console.

WLS Admin console at http://nodename.domian.com:s_wls_adminport/console is not accessible by error:

The Server is not able to service this request: [Socket:000445]Connection rejected, filter blocked Socket, weblogic.security.net.FilterException: [Security:090220] rule 2

The error is also in AdminServer.log under $FMW_HOME/user_projects/domains/EBS_domain_$TWO_TASK/servers/AdminServer/logs. The fix is replace one line in file config.xml under $FMW_HOME/user_projects/domains/EBS_domain_$TWO_TASK/config

    <connection-filter-rule>0.0.0.0/0 * * deny</connection-filter-rule>

    <connection-filter-rule>0.0.0.0/0 * * allow</connection-filter-rule>  (<= good one)

After shutdown Admin Server services and re-start them, Admin Server console worked.

$ ps -ef | grep $LOGNAME | wc -l

81

17). Post actions when Apps services are running (See Doc ID 2666934.1 section 8.1)

Notes: ADOP cleanup and fs_clone steps here can be deferred and run after Post steps in which more patches are applied. 

$ adop -status

Enter the APPS password:

Connected.

==============================================================

ADOP (C.Delta.12)

Session Id: 2

Command: status

Output: $NE_BASE/EBSapps/log/adop/2/20210217_114609/adzdshowstatus.out

====================================================================

Node Name  Node Type  Phase           Status            Started                       Finished                      Elapsed

--------------- ---------- --------------- --------------- -------------------- -------------------- ------------

node_name  master        APPLY             ACTIVE    2021/02/12 11:53:31  2021/02/16 21:15:16  105:21:45

                                       FINALIZE       NOT STARTED

                                       CLEANUP       NOT STARTED

$ adop phase=cleanup           (See Doc ID 2666934.1 section 8.1)

                                        <== it takes 5 minutes

The cleanup phase completed successfully.

adop exiting with status = 0 (Success)

Run FS_CLONE. This action will copy the new run edition code and configuration to the other file system, to ensure that both file systems are in sync before applying patches using the regular adop cycle on the other file system.

$ echo $FILE_EDITION

run

$ adop phase=fs_clone

                                            <= it takes 40 minutes or so

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

Validating credentials.

Initializing.

    Run Edition context  : $CONTEXT_FILE

    Patch edition context: %CONTEXT_FILE on /fs2/

    Patch file system free space: 62.92 GB

Validating system setup.

    Node registry is valid.

Checking for existing adop sessions.

    No pending session exists.

    Starting new adop session.

===================================================

ADOP (C.Delta.12)

Session ID: 3

Node: $EBS_HOSTNAME

Phase: fs_clone

Log: $NE_BASE/EBSapps/log/adop/3/20210218_182108/adop.log

==================================================

Validating configuration on node: [$EBS_HOSTNAME].

    Log: $NE_BASE/EBSapps/log/adop/3/20210218_182108/fs_clone/validate/$EBS_HOSTNAME

        [WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.

Checking for pending cleanup actions.

    No pending cleanup actions found.

Blocking managed server ports.

    Log: $NE_BASE/EBSapps/log/adop/3/20210218_182108/fs_clone/$EBS_HOSTNAME/txkCloneAcquirePort.log

Performing CLONE steps.

    Log: $NE_BASE/EBSapps/log/adop/3/20210218_182108/fs_clone/$EBS_HOSTNAME

Beginning application tier FSCloneStage - Thu Feb 18 18:23:54 2021

$COMMON_TOP/util/jdk32/bin/java -Xmx600M -Doracle.jdbc.autoCommitSpecCompliant=false -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath $FMW_HOME/webtier/lib/xmlparserv2.jar: …  oracle.apps.ad.clone.FSCloneStageAppsTier -e $CONTEXT_FILE -targ -stage $COMMON_TOP/adopclone_$EBS_HOSTNAME -tmp /tmp -component appsTier -nopromptmsg

Log file located at $INST_TOP/admin/log/clone/FSCloneStageAppsTier_02181823.log

Completed FSCloneStage...

Thu Feb 18 18:30:49 2021

Beginning application tier FSCloneApply - Thu Feb 18 18:33:21 2021

$COMMON_TOP/adopclone_$EBS_HOSTNAME/FMW/t2pjdk/bin/java -Xmx600M -Doracle.jdbc.autoCommitSpecCompliant=false -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath $FMW_HOME/adopclone_$EBS_HOSTNAME/jlib/xmlparserv2.jar: …  oracle.apps.ad.clone.FSCloneApplyAppsTier -e $CONTEXT_FILE ……

Log file located at $INST_TOP/admin/log/clone/FSCloneApplyAppsTier_02181833.log

Log file located at $INST_TOP/admin/log/clone/FSCloneApplyAppsTier_02181833.log

Target System Fusion Middleware Home set to $PATCH_BASE/FMW_Home

Target System Web Oracle Home set to $PATCH_BASE/FMW_Home/webtier

Target System Appl TOP set to $PATCH_BASE/EBSapps/appl

Target System COMMON TOP set to $PATCH_BASE/EBSapps/comn

Target System Instance Top set to $PATCH_BASE/inst/apps/$CONTEXT_NAME

Report file located at $PATCH_BASE/inst/apps/$CONTEXT_NAME/temp/portpool.lst

The new APPL_TOP context file has been created : $CONTEXT_FILE on /fs2/

contextfile=$CONTEXT_FILE on /fs2/

Completed FSCloneApply...

Thu Feb 18 18:58:25 2021

Resetting FARM name...

runDomainName: EBS_domain_$TWO_TASK

patchDomainName:

targets_xml_loc: $FMW_HOME/user_projects/domains//sysman/state/targets.xml

$FMW_HOME/user_projects/domains//sysman/state/targets.xml file not found or it does not have read/write permissions.

Releasing managed server ports.

    Log: $NE_BASE/EBSapps/log/adop/3/20210218_182108/fs_clone/$EBS_HOSTNAME/txkCloneAcquirePort.log

Synchronizing snapshots.

Stopping services on patch file system.

    Stopping admin server.

You are running adadminsrvctl.sh version 120.10.12020000.11

Stopping WLS Admin Server...

Refer $PATCH_BASE/inst/apps/$CONTEXT_NAME/logs/appl/admin/log/adadminsrvctl.txt for details

AdminServer logs are located at $FMW_HOME/user_projects/domains/EBS_domain/servers/AdminServer/logs

adadminsrvctl.sh: exiting with status 0

adadminsrvctl.sh: check the logfile $PATCH_BASE/inst/apps/$CONTEXT_NAME/logs/appl/admin/log/adadminsrvctl.txt for more information ... 

    Stopping node manager.

You are running adnodemgrctl.sh version 120.11.12020000.12

The Node Manager is already shutdown

NodeManager log is located at $PATCH_BASE/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1

adnodemgrctl.sh: exiting with status 2

adnodemgrctl.sh: check the logfile $PATCH_BASE/inst/apps/$CONTEXT_NAME/logs/appl/admin/log/adnodemgrctl.txt for more information ... 

Generating log report.

    Output: $NE_BASE/EBSapps/log/adop/3/20210218_182108/fs_clone/$EBS_HOSTNAME/adzdshowlog.out

The fs_clone phase completed successfully.

adop exiting with status = 0 (Success)

$ adop -status

Enter the APPS password:

Connected.

=================================

ADOP (C.Delta.12)

Session Id: 3

Command: status

Output: $NE_BASE/EBSapps/log/adop/3/20210219_134510/adzdshowstatus.out

=========================================

Node Name                 Node Type  Phase                        Status      Started                       Finished                    Elapsed

--------------- ---------- --------------- --------------- -------------------- -------------------- ------------

$EBS_HOSTNAME    master     FS_CLONE   COMPLETED   2021/02/18 18:21:59  2021/02/18 18:58:36  0:36:37

18) Post-upgrade steps & Additional critical patches (Doc Sections 9 & 10)

Apply patches 30399996 (12.2.10 Online Help. And, may need more product specific patches)

$ echo $APPL_TOP

/u02/app/$TWO_TASK/fs1/EBSapps/appl

$ echo $FILE_EDITION

run

$ cd $PATCH_TOP

$ unzip p30399996_R12_GENERIC.zip

$ adop phase=apply patches=30399996 hotpatch=yes

… …                                                          <= it takes 2 hours or more

Patch file system free space: 48.54 GB

Validating system setup.

    Node registry is valid.

Checking for existing adop sessions.

        [INFO] ICM is not down

        [INFO] Connection to http://node_name.domain.com:8041 successful

    [WARNING]   You should only specify hotpatch mode when directed to by the patch readme.

    No pending session exists.

    Starting new adop session.

===========================================================

ADOP (C.Delta.12)

Session ID: 4

Node: $EBS_HOSTNAME

Phase: apply

Log: $NE_BASE/EBSapps/log/adop/4/20210219_141932/adop.log

===========================================================

Applying patch 30399996.

    Log: $NE_BASE/EBSapps/log/adop/4/20210219_141932/apply/$EBS_HOSTNAME/30399996/log/u30399996.log

Running finalize actions for the patches being applied.

    Log: @ADZDSHOWLOG.sql "2021/02/19 15:47:20"

Running cutover actions for the patches being applied.

    Creating workers to process cutover DDL in parallel

    Log: $NE_BASE/EBSapps/log/adop/4/20210219_141932/apply/$EBS_HOSTNAME/log/cutover.log

    Performing database cutover in Quick mode

Generating post apply reports.

Generating log report.

    Output: $NE_BASE/EBSapps/log/adop/4/20210219_141932/apply/$EBS_HOSTNAME/adzdshowlog.out

The apply phase completed successfully.

adop exiting with status = 0 (Success)

Apply Patch 31948857  (additional critical patch.  Section 10)      

$ cd $PATCH_TOP

$ unzip p31948857_R12.FND.C_R12_GENERIC.zip

$ echo $FILE_EDITION

run

$ adop phase=apply patches=31948857 hotpatch=yes

                                                                      <= it takes less than 10 minutes

$ adop -status

Enter the APPS password:

Connected.

==================================================

ADOP (C.Delta.12)

Session Id: 4

Command: status

Output: $NE_BASE/EBSapps/log/adop/4/20210222_105721/adzdshowstatus.out

==================================================

Node Name                     Node Type  Phase           Status         Started              Finished             Elapsed

-------------------------- ----------------- ----------------- -------------------- -------------------- ------------

$EBS_HOSTNAME      master    APPLY            ACTIVE        2021/02/19 14:20:27  2021/02/19 16:41:32  2:21:05

                                                      CLEANUP      NOT STARTED

Update the snapshot manually with the steps below. It is safe to run it (see Doc ID 1320300.1 Section 1.9):

1) Start adadmin after source the RUN FS env.

2) Select "2. Maintain Applications Files menu" in "AD Administration Main Menu".

3) In "Maintain Applications Files", select "4. Maintain snapshot information".

4) Select "2. Update current view snapshot" in the "Maintain Snapshot Information".

5) Select "1. Update Complete APPL_TOP" in the "Maintain Current View Snapshot Information".

$ adadmin       <== it takes 20 to 30 minutes

… … 

Note: Your APPL_TOP has 16 files missing. Some of these files are referenced in your file drivers but are missing in the APPL_TOP.

Cleaning up temporary files...

      ***** File-system snapshot utility completed successfully ******

 Recording Adadmin action :ADADMIN_UPDATE_CURRENT_VIEW

 Tokens:UPDATE_CURR_VW=1

Review the messages above, then press [Return] to continue.

Confirm all 6 patches were applied:

SQL> SELECT adb.bug_number, adb.creation_date, ad_patch.is_patch_applied('R12',-1, adb.bug_number) status

FROM ad_bugs adb

WHERE adb.bug_number in 

('30628681', '30735865', '30399994', '30399999', '30399996', '31948857');

Confirm EBS R12.2 services are still up and running. I can log onto the site to submit concurrent job by IE (Internet Explorer version 1909).

$ ps -ef | grep $LOGNAME | wc -l

81

It is good time to check the diskspace usage and do a full backup.

$ cd $TWO_TASK

$ du -h --max-depth=1

33G     ./apps

3.0G    ./inst

43G     ./fs1

36G     ./fs2

22G     ./fs_ne

138G    .

From R12.2 upgrade installation, JRE version is 1.6 and JWS is not enabled by default:

$  cat $FORMS_WEB_CONFIG_FILE|grep sun_plugin_version| cut -c 1-35

sun_plugin_version=1.6.0_27

$ grep s_forms_launch_method $CONTEXT_FILE

         <config_option type="techstack" oa_var="s_forms_launch_method">browser</config_option>

R12.2 comes with JDK 1.7.0_85.  If JDK upgrade is necessary, follow Doc ID 1530033.1

- Using the Latest JDK 7.0 Update with Oracle E-Business Suite Release 12.2.

$  $ADJVAPRG -version

java version "1.7.0_85"

Java(TM) SE Runtime Environment (build 1.7.0_85-b15)

Java HotSpot(TM) Client VM (build 24.85-b06, mixed mode)

$ $AFJVAPRG -version

java version "1.7.0_85"

Java(TM) SE Runtime Environment (build 1.7.0_85-b15)

Java HotSpot(TM) 64-Bit Server VM (build 24.85-b06, mixed mode)

By now, EBS is successfully upgraded to R12.2.10.