ERP on DB: April 2021

Sunday, April 25, 2021

Post actions after R12.2 upgrade worked

After EBS instance is upgraded to R12.2.10, patches and other custom steps are necessary for it to meet business needs. Before adding more nodes to the instance, I did followings in my site.

1. Apply required R12.2 apps patches for database 19c upgrade (Section 2.1, Doc ID 2552181.1 - Interoperability Notes: Oracle E-Business Suite Release 12.2 with Oracle Database 19c).

The following 9 patches must exist in the environment before database is upgraded to 19c.

SELECT adb.bug_number, aat.name appl_top_name, adb.language, adb.creation_date,
decode(ad_patch.is_patch_applied('R12',aat.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status
FROM ad_bugs adb,
(select aat.name, aat.appl_top_id
from applsys.ad_appl_tops aat,
(select distinct fat.name from applsys.fnd_appl_tops fat) fat
where aat.name=fat.name ) aat
where adb.bug_number in (
'25452805', -- Applied
'26052406', -- Applied
'26521736', -- Need (19c interoperability patch for Release 12.2)
'30433124', -- Applied
'31088182', -- Need to apply
'31349591', -- Need to apply
'31800803', -- Need to apply
'30611319', -- Applied (only if using Integrated SOA Gateway (ISG))
'28732161' -- Need (Release 12.2.6 or later versions)
) order by adb.bug_number,aat.name,adb.language;

In my instance, 4 patches were applied. Use downtime mode to apply 5 patches:

26521736
31088182
31349591

31800803
28732161

DOWNTIME mode will apply the patch to RUN file system. It is not necessary to use downtime mode to apply them, but it saves time to run adop cut-over.

$ sh $ADMIN_SCRIPTS_HOME/adstpall.sh apps/apps_PWD

$ ps -ef | grep $LOGNAME

$ echo $FILE_EDITION

run

$ echo $PATCH_TOP

$NE_BASE/EBSapps/patch

$ cd $PATCH_TOP

$ unzip p26521736_12.2.0_R12_LINUX.zip

$ unzip p31088182_R12.FND.C_R12_GENERIC.zip

$ unzip p31349591_R12.OTA.C_R12_GENERIC.zip

$ unzip p31800803_R12.TXK.C_R12_GENERIC.zip

$ unzip p28732161_R12.FRM.C_R12_GENERIC.zip

$ adop phase=apply apply_mode=downtime patches=26521736,31088182,31349591,31800803,28732161

Run the SQL statement again to confirm all 9 patches were applied by now.

Before completing the patching cycle, which is required by R12.2 Clone script, I applied 5 more patches to the instance:

2. Two more patches:

Patch 31405457 (See Doc ID 2554156.1 for 19c database Export/Import):

$ unzip p31405457_R12_GENERIC.zip

Patch 32357755 to fix issue on changing sysadmin password (Doc ID 2743640.1 - FRM-40735: WHEN-VALIDATE-ITEM Trigger Raised Unhandled Exception ORA-29261 )

$ unzip p32357755_R12.FND.C_R12_LINUX.zip

$ adop phase=apply apply_mode=downtime patches=31405457,32357755

2. Four patches for 1099 tax asked by Business:

$ unzip p32072819_R12.AP.C_R12_GENERIC.zip

$ unzip p32157516_R12.AP.C_R12_GENERIC.zip
$ unzip p32181474_R12.AP.C_R12_GENERIC.zip

$ unzip p32266274_R12.AP.C_R12_GENERIC.zip

$ adop phase=apply apply_mode=downtime patches=32072819,32157516,32181474,32266274

Confirm 6 patches were applied:

SQL> SELECT adb.bug_number, aat.name appl_top_name, adb.language, adb.creation_date,
decode(ad_patch.is_patch_applied('R12',aat.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status
FROM ad_bugs adb,
(select aat.name, aat.appl_top_id
from applsys.ad_appl_tops aat,
(select distinct fat.name from applsys.fnd_appl_tops fat) fat
where aat.name=fat.name ) aat
where adb.bug_number in (
'31405457',

'32357755',
'32072819',
'32157516',
'32181474',
'32266274'

) order by adb.bug_number,aat.name,adb.language;

3. Apply EBS CPU patches

Oracle E-Business Suite Release 12 Critical Patch Update Knowledge Document (January 2021) (Doc ID 2737201.1)

4. Install 3rd party application (such as Excel4app)

5. If a custom top is not necessary but an env variable is needed as a pointer to a folder under $APPL_TOP, modify file $FND_TOP/fndenv.env by adding lines:

#Begin Customizations

XXEDI_TOP=${APPL_TOP}/xxedi/12.0.0; export XXEDI_TOP

#End Customizations

Start a new OS session to see $XXEDI_TOP

$ echo $XXEDI_TOP

Since ADSplice adds lines to Forms config file $INST_TOP/ora/10.1.2/forms/server/default.env when creating custom top, manual modification on file default.env is not necessary in R12.2:

#Begin Customizations

XXEDI_TOP=/path/to/appl/xxedi/12.0.0

#End Customizations

6. Copy Custom Top folders to R12.2. Then re-link folders & executables, and compile custom forms.

7. Custom OAF page development

Modify $FND_TOP/admin/template/custom/ebsProductManifest_xml.tmp to add the entry below for customprod.jar (after cu

After EBS instance is upgraded to R12.2.10, Patches and other custom steps are necessary for it to meet the business needs. I did followings in my site.

1. . Apply required R12.2 apps patches for database 19c upgrade (Section 2.1, Doc ID 2552181.1 - Interoperability Notes: Oracle E-Business Suite Release 12.2 with Oracle Database 19c).

The following 9 patches must exist in the environment before database is upgraded to 19c.

In my instance, 4 patches were applied. Use downtime mode to apply 5 patches:

26521736
31088182
31349591

31800803
28732161

DOWNTIME mode will apply the patch to RUN file system.

* It is not necessary to use downtime mode to apply them, if FS_CLONE was executed after R12.2.10 patch.

$ sh $ADMIN_SCRIPTS_HOME/adstpall.sh apps/apps_PWD

$ ps -ef | grep $LOGNAME

$ echo $FILE_EDITION

run

$ echo $PATCH_TOP

$NE_BASE/EBSapps/patch

$ cd $PATCH_TOP

$ unzip p26521736_12.2.0_R12_LINUX.zip

$ unzip p31088182_R12.FND.C_R12_GENERIC.zip

$ unzip p31349591_R12.OTA.C_R12_GENERIC.zip

$ unzip p31800803_R12.TXK.C_R12_GENERIC.zip

$ unzip p28732161_R12.FRM.C_R12_GENERIC.zip

$ adop phase=apply apply_mode=downtime patches=26521736,31088182,31349591,31800803,28732161

Run the SQL statement again to confirm all 9 patches were applied by now.

Before completing the patching cycle, which is required by R12.2 Clone script, I applied 5 more patches to the instance:

2. Two more patches:

Patch 31405457 (See Doc ID 2554156.1 for 19c database Export/Import):

$ unzip p31405457_R12_GENERIC.zip

Patch 32357755 to fix issue on changing sysadmin password (Doc ID 2743640.1 - FRM-40735: WHEN-VALIDATE-ITEM Trigger Raised Unhandled Exception ORA-29261 )

$ unzip p32357755_R12.FND.C_R12_LINUX.zip

$ adop phase=apply apply_mode=downtime patches=31405457,32357755

2. Four patches for 1099 tax asked by Business:

$ unzip p32072819_R12.AP.C_R12_GENERIC.zip

$ unzip p32157516_R12.AP.C_R12_GENERIC.zip
$ unzip p32181474_R12.AP.C_R12_GENERIC.zip

$ unzip p32266274_R12.AP.C_R12_GENERIC.zip

$ adop phase=apply apply_mode=downtime patches=32072819,32157516,32181474,32266274

Confirm 6 patches were applied:

'32357755',
'32072819',
'32157516',
'32181474',
'32266274'

) order by adb.bug_number,aat.name,adb.language;

3. Apply 2021 January CPU patches

Oracle E-Business Suite Release 12 Critical Patch Update Knowledge Document (January 2021) (Doc ID 2737201.1)

(a) Apply EBS patch 32071646

SQL> select ad_patch.is_patch_applied('R12',-1,32071646) from dual;

AD_PATCH.IS_PATCH_APPLIED('R12',-1,32071646)

-------------------------------------------------------------------

NOT_APPLIED

SQL> SELECT adb.bug_number, aat.name appl_top_name, adb.language, adb.creation_date,

decode(ad_patch.is_patch_applied('R12',aat.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status

FROM ad_bugs adb,

(select aat.name, aat.appl_top_id

from applsys.ad_appl_tops aat,

(select distinct fat.name from applsys.fnd_appl_tops fat) fat

where aat.name=fat.name ) aat

where adb.bug_number in (

'32071646',

'32163187',

'32004048'

) order by adb.bug_number,aat.name,adb.language;

no rows selected

$ echo APPL_TOP

$ cd $PATCH_TOP

$ unzip p32071646_12.2.0_R12_LINUX.zip

$ adop phase=apply apply_mode=downtime patches=32071646

$ unzip p32163187_R12.FWK.C_R12_GENERIC.zip

$ unzip p32004048_R12.OKC.C_R12_GENERIC.zip

$ adop phase=apply apply_mode=downtime patches=32163187,32004048

... ...

Applying patch 32163187.

Log: $NE_BASE/EBSapps/log/adop/4/.../32163187/log/u32163187.log

Applying patch 32004048.

Log: $NE_BASE/EBSapps/log/adop/4/.../32004048/log/u32004048.log

Running finalize actions for the patches being applied.

Log: @ADZDSHOWLOG.sql "2021/07/13 12:05:43"

Running cutover actions for the patches being applied.

Creating workers to process cutover DDL in parallel

... ...

The apply phase completed successfully.

adop exiting with status = 0 (Success)

Run SQL statement again to confirm 3 patches are applied.

(b) WebLogic PSU patch 32052267 was applied as a ETCC requirement.

See Doc ID 1306505.1 for more on Oracle WebLogic Server PSUs (Patch Set Updates).

Patch 31304503 (OSS Security Patch Update CPUJul2020): applied

$ export ORACLE_HOME=$IAS_ORACLE_HOME

$ export PATH=$IAS_ORACLE_HOME/OPatch:$PATH

$ opatch lsinventory | grep 31304503

Patch 31304503 : applied on Sat Jun 26 12:07:37 EDT 2021

(d) Oracle Fusion Middleware 11.1.1.9 OHS - Web Tier Home

Patch 31047338 (OHS Security Patch Update CPUApr2020)

$ opatch lsinventory | grep -i 31047338

Patch 31047338 : applied on Sat Jun 26 12:20:11 EDT 2021

(e) Oracle Fusion Middleware 11.1.1.9 - Oracle Common Home

Patch 30368663 (Security Patch Update CPUOct2019)

Note: Patch 31985571 is a superset of patch 30368663 in Oct 2020

$ export ORACLE_HOME=$FMW_HOME/oracle_common

$ export PATH=$ORACLE_HOME/OPatch:$PATH

$ opatch lsinventory | grep 30368663

Patch 30368663 : applied on Sat Jun 26 12:31:37 EDT 2021

$ adop -status

Enter the APPS password:

Connected.

==================================================

ADOP (C.Delta.12)

Session Id: 4

Command: status

Output: $NE_BASE/EBSapps/log/adop/4/20210713_130234/adzdshowstatus.out

==================================================

Node Name Node Type Phase Status Started Finished Elapsed

--------------- ---------- --------------- --------------- -------------------- -------------------- ------------

node_name master APPLY ACTIVE 2021/0X/12 10:50:40 2021/0X/13 12:55:03 26:04:23

CLEANUP NOT STARTED

4. Install 3rd party application (such as Excel4app)

5. If a custom top is not necessary but an env variable is needed as a pointer to a folder under $APPL_TOP, modify file $FND_TOP/fndenv.env by adding lines:

#Begin Customizations

XXEDI_TOP=${APPL_TOP}/xxedi/12.0.0; export XXEDI_TOP

#End Customizations

Start a new OS session to see $XXEDI_TOP

$ echo $XXEDI_TOP

Since ADSplice adds lines to Forms config file $INST_TOP/ora/10.1.2/forms/server/default.env when creating custom top, manual modification on file default.env is not necessary in R12.2:

#Begin Customizations

XXEDI_TOP=/path/to/appl/xxedi/12.0.0

#End Customizations

6. Copy Custom Top folders to R12.2. Then re-link folders & executables, and compile custom forms.

7. Custom OAF page development

Notes: key entry is saved in file $FND_TOP/admin/template/custom/ebsProductManifest_xml.tmp (after customall.jar):

8. Apply 3 AP patch to fix AP issues

$ export FORMS_PATH=$FORMS_PATH:$AU_TOP/forms/US:AU_TOP/resource

$ frmcmp_batch userid=apps/apps_PWD module=$AU_TOP/resource/IBYFDOBJ.pll output_file=$AU_TOP/resource/IBYFDOBJ.plx module_type=library batch=no compile_all=special

Forms 10.1 (Form Compiler) Version 10.1.2.3.0 (Production)

Forms 10.1 (Form Compiler): Release - Production

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production

With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

PL/SQL Version 10.1.0.5.0 (Production)

Oracle Procedure Builder V10.1.2.3.0 - Production

Oracle Virtual Graphics System Version 10.1.2.0.0 (Production)

Oracle Multimedia Version 10.1.2.0.2 (Production)

Oracle Tools Integration Version 10.1.2.0.2 (Production)

Oracle Tools Common Area Version 10.1.2.0.2

Oracle CORE 10.1.0.5.0 Production

Compiling library IBYFDOBJ...

Invalidating Package Spec IBY_PMTATTRIB......

Invalidating Package Body IBY_PMTATTRIB......

Compiling Package Spec IBY_PMTATTRIB......

Compiling Package Body IBY_PMTATTRIB......

Done.

NOTES: after compiled by frmcmp_batch, IBYFDOBJ.plx got newer file:

$ ls -al $AU_TOP/resource/IBYFDOBJ.plx

-rw-r--r-- 1 user group 40960 Mmm 20 14:01 IBYFDOBJ.plx

$ adident Header $AU_TOP/resource/IBYFDOBJ.plx

$Header IBYFDOBJ.pld 120.7.12020000.2 2012/07/12 15:13:21 sgogula ship $

$ adident Header $AU_TOP/resource/IBYFDOBJ.pll

$Header IBYFDOBJ.pld 120.7.12020000.2 2012/07/12 15:13:21 sgogula ship $

$ sqlplus apps/apps_PWD

SQL> alter package IBY_DISBURSEMENT_COMP_PUB compile body;

Package body altered.

SQL> alter package IBY_DISBURSEMENT_COMP_PUB compile;

Package altered.

SQL> show parameter optimizer_capture_sql_plan_baselines

NAME TYPE VALUE

----------------------------------------------- ----------- ------------------

optimizer_capture_sql_plan_baselines boolean FALSE

$ unzip p32385168_R12.AP.C_R12_GENERIC.zip

$ adop phase=apply apply_mode=downtime patches=32385168

$ strings -a $AP_TOP/patch/115/sql/apaprvlb.pls |grep '$Header'

/* $Header: apaprvlb.pls 120.231.12020000.57 2021/02/24 09:40:39 anankitg ship $ */

$ unzip p32768426_R12.AP.C_R12_GENERIC.zip

$ unzip p31211521_R12.AP.C_R12_GENERIC.zip

$ adop phase=apply apply_mode=downtime patches=32768426,31211521

Confirm they are applied:

SQL> SELECT adb.bug_number, aat.name appl_top_name, adb.language, adb.creation_date,

decode(ad_patch.is_patch_applied('R12',aat.appl_top_id,adb.bug_number,adb.language),'EXPLICIT','APPLIED','NOT_APPLIED','NOT APPLIED') status

FROM ad_bugs adb,

(select aat.name, aat.appl_top_id

from applsys.ad_appl_tops aat,

(select distinct fat.name from applsys.fnd_appl_tops fat) fat

where aat.name=fat.name ) aat

where adb.bug_number in (

'32385168',

'32768426',

'31211521'

) order by adb.bug_number,aat.name,adb.language;

32768426:R12.AP.C

---------------------------------

$AU_TOP/forms/US/APXINWKB.fmb 120.983.12020000.181

$AP_TOP/forms/US/APXINWKB.fmx 120.983.12020000.181

31211521:R12.AP.C

---------------------------------

$AP_TOP/patch/115/sql/apetxsrb.pls 120.223.12020000.38

Follow-up steps asked by Oracle:

1) Validate APPS schema using adadmin.

a. start adadmin

b. Pick Option 4 Maintain Applications Database Entities menu

c. Pick Validate APPS schema using adadmin (1).

2) Re-create grants and synonyms for APPS schema using adadmin.

a. Pick Option 4 Maintain Applications Database Entities menu

b. Pick Re-create grants and synonyms for APPS schema (2)

3) Generate product JAR files using adadmin.

a. Pick Option 1 Generate Applications Files menu

b. Pick Generate product JAR files (4, and answer Yes)

4) Reload JAR files to database using adadmin.

a. Pick Option 3 Compile/Reload Applications Database Entities menu

b. Pick Reload JAR files to database (4)

5) Compile all the INVALIDS using adadmin.

a. Pick Option 3 Compile/Reload Applications Database Entities menu

b. Pick Option 1 Compile APPS schema

6) Compile all the menus.

a. Pick Option 3 Compile/Reload Applications Database Entities menu

b. Pick Option 2 Compile menu information

7) Compile all the ZX, IBY, XLA, PO, AP forms including libraries using adadmin.

a. Pick Option 1 Generate Applications Files menu

b. Pick Option 2 Generate form files

8)* Bounce the Database (Shutdown & Startup, optional)

9)* Start all the application services (adstrtal.sh apps/<apps password>).

10)* Submit the concurrent request "Gather Schema Statistics" with Schema Name: ALL and Estimate Percent: 20% Atleast and wait for the request to complete.

11)* Retest the issue.

* They can be deferred till configuration on all nodes is completed.

9. Verify database parameters for ADOP:

SQL> show parameter local_listener

NAME TYPE VALUE

--------------------- ---- ----------- ------------------------------

local_listener string EBSDEV_LOCAL

SQL> show parameter service_names

NAME TYPE VALUE

-------------------------- ----------- ------------------------------

service_names string EBSDEV, ebs_patch,EBSDEV_ebs_patch

SQL> select SERVICE_ID,NAME from dba_services where name='ebs_patch';

SERVICE_ID NAME

----------------- ----------------------------------------------------------------

6 ebs_patch

Notes: This line may got lost somehow. If so, add it (by "alter system ...").

SQL> select * from dba_editions;

EDITION_NAME PARENT_EDITION_NAME USABLE

---------------------- ------------------------------------ ------------

ORA$BASE YES

$ adop -status

Enter the APPS password:

Connected.

=======================================================

ADOP (C.Delta.12)

Session Id: 4

Command: status

Output: $NE_BASE/EBSapps/log/adop/4/20210X14_130108/adzdshowstatus.out

========================================================

Node Name Node Type Phase Status Started Finished Elapsed

--------------- ---------- --------------- --------------- -------------------- -------------------- ------------

$EBS_HOSTNAME master APPLY ACTIVE 2021/0X/12 10:50:40 2021/0X/13 12:55:03 26:04:23

CLEANUP NOT STARTED

10. Run FS_CLONE (when EBS Services are running) before adding/cloning 2 node

$ adop phase=cleanup (or, adop phase=cleanup cleanup_mode=full )

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

Validating credentials.

... ...

Patch file system free space: 49.76 GB

Validating system setup.

Node registry is valid.

Checking for existing adop sessions.

===============================================

ADOP (C.Delta.12)

Session ID: 4

Node: $EBS_HOSTNAME

Phase: cleanup

Log: $NE_BASE/EBSapps/log/adop/4/20210X14_130309/adop.log

===============================================

Processing cleanup actions in parallel.

Log: $NE_BASE/EBSapps/log/adop/4/20210X14_130309/cleanup/$EBS_HOSTNAME/log/cleanup.log

Generating report of queued DDL actions.

Output: $NE_BASE/EBSapps/log/adop/4/20210X14_130309/cleanup/$EBS_HOSTNAME/adzdallddls.out

Running cleanup in STANDARD mode.

Log: @ADZDSHOWLOG.sql "2021/0X/14 13:05:32"

Generating log report.

Output: $NE_BASE/EBSapps/log/adop/4/20210X14_130309/cleanup/$EBS_HOSTNAME/adzdshowlog.out

The cleanup phase completed successfully.

adop exiting with status = 0 (Success)

$ adop -status

Enter the APPS password:

Connected.

========================================================

ADOP (C.Delta.12)

Session Id: 4

Command: status

Output: $NE_BASE/EBSapps/log/adop/4/20210X14_131226/adzdshowstatus.out

========================================================

Node Name Node Type Phase Status Started Finished Elapsed

-------------------------- --------------- --------------- -------------------- -------------------- ------------

$EBS_HOSTNAME master APPLY COMPLETED 2021/0X/12 10:50:40 2021/0X/13 12:55:03 26:04:23

CLEANUP COMPLETED 2021/0X/14 13:04:04 2021/0X/14 13:05:54 0:01:50

$ echo $FILE_EDITION

run

NOTES: fs_clone will delete folders from fs2 (PATCH file system) and then copy folders from fs1 (RUN file system) to replace them. If it failed in the middle, some folders or files may be missed on fs2.

$ adop phase=fs_clone (or, adop phase=fs_clone force=yes)

<== it may take up to 2 hours

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

Validating credentials.

Initializing.

... ...

Patch file system free space: 49.75 GB

Validating system setup.

Node registry is valid.

Checking for existing adop sessions.

No pending session exists.

Starting new adop session.

==========================================================

ADOP (C.Delta.12)

Session ID: 5

Node: $EBS_HOSTNAME

Phase: fs_clone

Log: $NE_BASE/EBSapps/log/adop/5/20210X14_131409/adop.log

==========================================================

Validating configuration on node: [$EBS_HOSTNAME].

Log: $NE_BASE/EBSapps/log/adop/5/20210X14_131409/fs_clone/validate/$EBS_HOSTNAME

[WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.

Checking for pending cleanup actions.

No pending cleanup actions found.

Blocking managed server ports.

Log: $NE_BASE/EBSapps/log/adop/5/20210X14_131409/fs_clone/$EBS_HOSTNAME/txkCloneAcquirePort.log

Performing CLONE steps.

Log: $NE_BASE/EBSapps/log/adop/5/20210X14_131409/fs_clone/$EBS_HOSTNAME

Beginning application tier FSCloneStage - Wed Mmm 14 13:17:31 2021

$COMMON_TOP/util/jdk32/bin/java -Xmx600M -Doracle.jdbc.autoCommitSpecCompliant=false -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath ... ...

Log file located at $INST_TOP/admin/log/clone/FSCloneStageAppsTier_0X141317.log

Completed FSCloneStage...

Wed Jul 14 13:24:27 2021

Beginning application tier FSCloneApply - Wed Mmm 14 13:26:59 2021

... ...

Completed FSCloneApply...

Wed Mmm 14 13:52:59 2021

Resetting FARM name...

runDomainName: EBS_domain_$TWO_TASK

patchDomainName: EBS_domain

targets_xml_loc: $FMW_HOME/user_projects/domains/EBS_domain/sysman/state/targets.xml

FARM name reset is successful.

Releasing managed server ports.

Log: $NE_BASE/EBSapps/log/adop/5/20210X14_131409/fs_clone/$EBS_HOSTNAME/txkCloneAcquirePort.log

Synchronizing snapshots.

Stopping services on patch file system.

Stopping admin server.

You are running adadminsrvctl.sh version 120.10.12020000.11

Stopping WLS Admin Server...

Refer $PATCH_BASE/inst/apps/$CONTEXT_NAME/logs/appl/admin/log/adadminsrvctl.txt for details

AdminServer logs are located at $PATCH_BASE/FMW_Home/user_projects/domains/EBS_domain/servers/AdminServer/logs

adadminsrvctl.sh: exiting with status 0

adadminsrvctl.sh: check the logfile $PARCH_BASE/inst/apps/$CONTEXT_NAME/logs/appl/admin/log/adadminsrvctl.txt for more information ...

Stopping node manager.

You are running adnodemgrctl.sh version 120.11.12020000.12

The Node Manager is already shutdown

NodeManager log is located at $PATCH_BASE/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1

adnodemgrctl.sh: exiting with status 2

adnodemgrctl.sh: check the logfile $LOG_HOME/appl/admin/log/adnodemgrctl.txt for more information ...

Generating log report.

Output: $NE_BASE/EBSapps/log/adop/5/20210X14_131409/fs_clone/$EBS_HOSTNAME/adzdshowlog.out

The fs_clone phase completed successfully.

adop exiting with status = 0 (Success)

$ adop -status

Enter the APPS password:

Connected.

=========================================================

ADOP (C.Delta.12)

Session Id: 5

Command: status

Output: $NE_BASE/EBSapps/log/adop/5/20210X14_140211/adzdshowstatus.out

=========================================================

Node Name Node Type Phase Status Started Finished Elapsed

------------------------ -------------- ----------------- ------------------ --------------------- ------------

$EBS_HOSTNAME master FS_CLONE COMPLETED 2021/0X/14 13:15:26 2021/0X/14 14:53:11 1:37:45

File System Synchronization Type: Full

adop exiting with status = 0 (Success)

Now, it can be cloned to 2nd node.

Thursday, April 15, 2021

Enable TLS 1.2 in R12.2 right after it is upgraded from R12.1.3

This post assume all cert files are good and TLS 1.2 was enabled in R12.1.3. After it was upgraded to R12.2, SSL and TLS1.2 were not enabled automatically. It is optional, but is normally required by company's Security to enable it. Here are steps to enable them only for inbound connections as described in Doc ID 1367293.1 (Enabling TLS in Oracle E-Business Suite Release 12.2). The use of TLS 1.2 requires Java 7 at a minimum and Oracle Fusion Middleware (FMW) 1.1.1.9.

Since all wallet files and cert files exist in old $INST_TOP/certs/Apache folder, that saves a lot of time from generating them. I also verified all necessary patches exist in my new R12.2.10 instances. Here is what I did.

1. Copy Apache folder from R12.1.3 $INST_TOP/certs/Apache to R12.2.10 $NE_BASE/inst/$CONTEXT_NAME/certs/Apache (the same as <s_web_ssl_directory>/Apache).

$ cd $NE_BASE/inst/$CONTEXT_NAME/certs

$ ls

$ cp -rp $HOME/$TWO_TASK/inst_R12.1/apps/$CONTEXT_NAME/certs/Apache .

(assume R12.1 folder inst was renamed to inst_R12.1 right after R12.2 upgrade completed. Possible files in R12.1.3 includes:)

$ ls -al Apache

-rw-r--r-- 1 user group 1367 Mar 17 2020 ca.crt

-rw------- 1 user group 6377 Jul 17 2020 cwallet.sso

-rw-r--r-- 1 user group 6349 Jul 17 2020 ewallet.p12

-rw-r--r-- 1 user group 1684 Mar 17 2020 intermediate.crt

-rw-r--r-- 1 user group 5389 Jul 17 2020 opmn.crt

-rw-r--r-- 1 user group 2338 Jul 17 2020 server.crt

-rw-r--r-- 1 user group 1675 Jul 17 2020 server.key

Notes: Section 5.2 Step 2 says "The s_web_ssl_directory location is still used by some Oracle E-Business Suite Release 12.2 components (for example, XML Gateway Transportation Agent OXTA) and during the Oracle Fusion Middleware cloning process". Step 6 says Apache folder "is not used by the Oracle HTTP Server" (Apache).

In my instances, cert files worked in R12.1 are still worked in R12.2. So, I did not need to do any additional steps but use orapki in R12.2 to verify its password.

$ which orapki <= do not use the one in 10.1.2 ORACLE_HOME

$FMW_HOME/oracle_common/bin/orapki

$ orapki wallet display -wallet $NE_BASE/inst/$CONTEXT_NAME/certs/ewallet.p12

If it expired, it has to be renewed first because Apache and opmn will not start by error: [opmn] [ERROR:1] [] [internal] $FMW_HOME/webtier/opmn/bin/opmn: unexpected exit: status 4200

2. Copy cwallet.sso to 2 locations: HTTP Server & OPMN

$ grep s_ohs_instance_loc $CONTEXT_FILE

Define Linux variables for scripting:

$ iName=$(tr < $CONTEXT_FILE '<>' ' ' | awk '/"s_ohs_instance"/ {print $(NF-1)}' )

$ echo $iName <== this tells EBS_web_OHS1 or EBS_web_OHS2 is used, e.g.

$ SUBiName=${iName%?????}

$ echo $SUBiName

EBS_web

Modify the Oracle HTTP Server wallet:

$ cd $FMW_HOME/webtier/instances/$iName/config/OHS/$SUBiName/keystores/default

$ mv cwallet.sso cwallet.sso_ORIG

$ cp -p $NE_BASE/inst/$CONTEXT_NAME/certs/Apache/cwallet.sso .

$ ls

Modify the OPMN wallet:

$ cd $FMW_HOME/webtier/instances/$iName/config/OPMN/opmn

$ mv wallet wallet_ORIG

$ mkdir wallet

$ cd wallet

$ cp -p $NE_BASE/inst/$CONTEXT_NAME/certs/Apache/cwallet.sso .

$ ls

3. Modify 3 files: opmn.xml, and admin.conf & ssl.conf in folder <s_ohs_instance_loc>/config/OHS/<s_ohs_component> (see Section 6.1.2 Configure Inbound Connections)

NOTES:

1) If the environment is cloned from another instance, those configuration files will be copied from the Source instance and so their contents may not be the same as in below.

2) Unlike R12.1, WebLogic configuration files in R12.2 can be edited and AutoConfig may not update them. For more details on modifying OHS port values, see Doc ID 1905593.1 (Managing Configuration of Oracle HTTP Server and Web Application Services in Oracle E-Business Suite Release 12.2).

Configure the OPMN Cipher Suites:

$ cd ..

(or, $ cd $FMW_HOME/webtier/instances/$iName/config/OPMN/opmn )

$ cp -p opmn.xml opmn.xml_ORIG

$ vi opmn.xml

$ diff opmn.xml opmn.xml_ORIG

< <ssl enabled="true" wallet-file="....../config/OPMN/opmn/wallet" ssl-versions="TLSv1.2" ssl-ciphers="SSL_RSA_WITH_AES_256_CBC_SHA,SSL_RSA_WITH_AES_128_CBC_SHA"/>

---

> <ssl enabled="true" wallet-file="....../config/OPMN/opmn/wallet" ssl-versions="TLSv1.0" ssl-ciphers="SSL_RSA_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA"/>

110c110

< </opmn>

---

> </opmn>

\ No newline at end of file

$ cd ../../OHS/$SUBiName

(cd $FMW_HOME/webtier/instances/$iName/config/OHS/$SUBiName )

$ cp -p admin.conf admin.conf_ORIG

$ vi admin.conf

$ diff admin.conf admin.conf_ORIG

< SSLCipherSuite SSL_RSA_WITH_AES_256_CBC_SHA:SSL_RSA_WITH_AES_128_CBC_SHA

< SSLProtocol nzos_Version_1_2

---

> SSLCipherSuite SSL_RSA_WITH_AES_128_CBC_SHA:SSL_RSA_WITH_3DES_EDE_CBC_SHA

> SSLProtocol nzos_Version_1_0

$ cp -p ssl.conf ssl.conf_ORIG

$ vi ssl.conf

$ diff ssl.conf ssl.conf_ORIG

< SSLProtocol TLSv1.2

< SSLCipherSuite HIGH:MEDIUM:!aNULL:!RC4:!3DES:!SEED:!IDEA:!CAMELLIA:+HIGH:+MEDIUM

---

> SSLProtocol all +TLSv1 +SSLv3 <= the file may have slightly different entries

> SSLCipherSuite HIGH:MEDIUM:!aNULL:+SHA1:+MD5:+HIGH:+MEDIUM

Notes: Section 6.5.2 indicates that edit ssl.conf is acceptable.

Also, if entry in ssl.conf is wrong, adapcctl.sh will fail to start Oracle HTTP Server (OHS). Its logs are at $FMW_HOME/webtier/instances/$iName/diagnostics/logs/OHS/$SUBiName .

And opmn.log is at $FMW_HOME/webtier/instances/$iName/diagnostics/logs/OPMN/opmn

If want to make WLS console use SSL, more changes may be needed.

4. Confirm "Listen <port_number>" in ssl.conf and <s_webssl_port> in $CONTEXT_FILE are the same. In this post, it is 4480.

$ grep Listen ssl.conf

$ grep s_webssl_port $CONTEXT_FILE

5. Keep WLS Admin Server running on Primary node (to avoid java.net.ConnectException error), and run below line (on each application tier nodes) to propagate the changes made to the context file variables. (Optional?)

$ perl $AD_TOP/bin/adSyncContext.pl contextfile=$CONTEXT_FILE

Enter the APPS user password:

Enter the WebLogic AdminServer password:

Log file $INST_TOP/logs/appl/rgf/Wed_XXX_14_17_47_45_2021/adSyncContext.log says

... ...

Time taken for adSyncContext.pl to complete: 0 mins 24 secs

The program exited with status 0

6. Edit $CONTEXT_FILE to change value for variables as asked in Section 5.2 step 9. Here is the settings in my EBS site https://example.domain.com:

s_url_protocol => https

s_local_url_protocol => https

s_webentryurlprotocol => https

s_active_webport => 443 (4480 when no mapping)

s_https_listen_parameter => 4480

s_login_page, => https://example.domain.com/OA_HTML/AppsLogin

(copy it from R12.1.3 which worked)

s_webentryhost => example

(without it, login page will direct to a wrong URL)

s_external_url => https://example.domain.com

I do not enter port 443 to any files hold port info in OHS configuration folder when F5 is used for load balance. F5 may map port 443 to port 4480. (But in R12.1.3, line "Port 443" is in files ssl.conf and httpd.conf)

Some context variables will be changed to default by R12.2 upgrade. Update or confirm below variables in $CONTEXT_FILE before running adautocfg.sh.

sitename (a Profile reads it), s_smtphost (company-wide smtp server for mailer), s_forms_tmpdir (do not use /tmp for production instance), s_sesstimeout (a Profile reads it), s_appcpnam (to REQID.OUT - see Doc ID 134007.1 to set output file type), s_forms_catchterm to 0 (FORMS_CATCHTERM - see Doc ID 745711.1 to avoid forms LOVs consume 100% CPU. R12.2 default is 1).

Notes: in multi-node instance, enabling TLS 1.2 is not necessary (or is optional) on concurrent node. But if it is enabled, ssl cert file has to be renewed before it expires (to avoid error in starting Apache from running FS_CLONE on that node).

7. Additional notes.

cwallet.sso can be found in other 3 folders for Fusion Middleware Control Wallets. But my instance does not use them (TLS 1.0 may use them):

$ iDomain=$(tr < $CONTEXT_FILE '<>' ' ' | awk '/"s_wls_domain_name"/ {print $(NF-1)}' )

$ echo $iDomain

EBS_domain <== e.g.

$ ls -al $FMW_HOME/webtier/instances/$iName/config/OHS/$SUBiName/proxy-wallet

(Notes: during one troubleshooting, copying the right cwallet.sso file to above folder fixed the failure in starting Apache. Not sure why).

$ ls -al $FMW_HOME/user_projects/domains/$iDomain/opmn/$iName/wallet

$ ls -al $FMW_HOME/user_projects/domains/$iDomain/opmn/$iName/$SUBiName/wallet

8. Run adautocfg.sh.

Note: R12.2 autoconfig does not update files opmn.xml, admin.conf, ssl.conf in Step 3.

Verify some info:

$ cd $FMW_HOME/webtier/instances/$iName/config/OHS/$SUBiName

$ grep 4480 *.*

ssl.conf:Listen 4480
ssl.conf: <VirtualHost _default_:4480>

$ grep 443 *.*
trusted.conf:#RedirectMatch ^/OA_HTML/fwk/t/(.*) https://example.domain.com:443/OA_IMAGE/fwk/t/$1
url_fw.conf:RewriteRule ^/$ https://example.domain.com:443/OA_HTML/AppsLogin [R,L]

$ grep example *.*
trusted.conf:#RedirectMatch ^/OA_HTML/fwk/t/(.*) https://example.domain.com:443/OA_IMAGE/fwk/t/$1
url_fw.conf:RewriteRule ^/$ https://example.domain.com:443/OA_HTML/AppsLogin [R,L]

$ nslookup example.domain.com
Server: 1XX.121.153.1XX
Address: 1XX.121.153.1XX#53

example.domain.com canonical name = example.np.vip.domain.com.
Name: example.np.vip.domain.com
Address: 16X.63.75.6X

9. Run adstpall.sh/adstrtal.sh script in $ADMIN_SCRIPTS_HOME directory to stop and restart all services.

Now, my R12.2.10 site is SSL and TLS 1.2 enabled. All webpages can be accessed in https from browsers, and EBS Form can be launched (if JWS was enabled).

10. Doc ID 1367293.1 Section 5.2 Step 12 gives details on using $APPL_TOP_NE/ad/custom/adop_sync.drv to synchronize the TLS setup between the two file systems. I found that all .conf files are synchronized automatically, but .sso files are not. Adding below lines to file adop_sync.drv will synchronize them. But, if FS_CLONE is executed after this change, all settings are copied to PATCH file system and this step can be skipped. Note File b64InternetCertificate.txt is used if Oracle® Application Management Pack for Oracle E-Business Suite is used to monitor EBS instance.

#TLS SECTION - START

# Required for TLS setup migration from RUN to PATCH file-system.

# Please alter the commands in the event that rsync is not available or the platform does not support the example syntax.

#Oracle HTTP Server Wallet - cwallet.sso

rsync -zr %s_current_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/keystores/default/cwallet.sso %s_other_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/keystores/default/cwallet.sso

#OPMN Wallet - cwallet.sso

rsync -zr %s_current_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OPMN/opmn/wallet/cwallet.sso %s_other_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OPMN/opmn/wallet/cwallet.sso

#Fusion Middleware Control Wallets - cwallet.sso

rsync -zr %s_current_base%/FMW_Home/user_projects/domains/%s_wls_domain_name%/opmn/%s_ohs_instance%/%s_ohs_component%/wallet/cwallet.sso %s_other_base%/FMW_Home/user_projects/domains/%s_wls_domain_name%/opmn/%s_ohs_instance%/%s_ohs_component%/wallet/cwallet.sso

rsync -zr %s_current_base%/FMW_Home/user_projects/domains/%s_wls_domain_name%/opmn/%s_ohs_instance%/wallet/cwallet.sso %s_other_base%/FMW_Home/user_projects/domains/%s_wls_domain_name%/opmn/%s_ohs_instance%/wallet/cwallet.sso

rsync -zr %s_current_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/proxy-wallet/cwallet.sso %s_other_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/proxy-wallet/cwallet.sso

11. Above changes are on RUN file system only. Run FS_CLONE to sync up it to PATCH file system.

Wednesday, April 14, 2021

Enable Java Web Start with Oracle E-Business Suite R12.2

After R12.1.3 is upgraded to R12.2.10, I see JRE 1.6.0_27 is the version in the new R12.2.10 instance. It is a low version, which makes R12.2 GUI Forms work only with IE browser. For R12.2 to work with MS Edge and Google Chrome browsers, we have to upgrade JRE and enable JWS.

1) First, per Doc ID 2188898.1 (Using Java Web Start with Oracle E-Business Suite), confirmed patch 26825525 (replaces 25441839 Merge Request On Top Of 10.1.2.3.2 PSU, Doc ID 2554599.1) was applied during R12.2.10 upgrade. (Section 4.2.2).

$ opatch lsinventory | grep 26825525

Patch 26825525 applied on xxxx, 2021

2) Confirmed ALL below 13 patches were applied (Section 4.2.3). Some patches may show up twice in below query.

SQL> SELECT bug_number, creation_date, ad_patch.is_patch_applied('R12',-1, bug_number) status
FROM ad_bugs
where bug_number in (
'24498616', -- AD: Add Java Web Start support to Oracle E-Business Suite
'25449925', -- TXK: Add Java Web Start support to Oracle E-Business Suite
'25380324', -- Oracle E-Business Suite Java Applets launching with Java Web Start
'29058008', -- Oracle E-Business Suite Java Applets Launching with JWS rel 2
'29024389', -- WS: New Preference To Indicate Launch of New Forms Session
'28713780', -- 12.2.6+:Oracle Workflow Java Applets launching with Java Web Start
'23645622', -- GL: Add Java Web Start Support to AHM Java applet
'23586286', -- MSC: Add Java Web Start Support to PS/SNO 12.2.6
'26100397', -- R12.TXK.C (see Doc ID 393931.1 for deploying JRE)
'22806350', -- R12.OWF.C (see Doc ID 1367293.1 for enabling TLS 1.2)
'21473055', -- R12.ICX.D (iProcurement, Doc ID 1937220.1 for TLS1.2)
'22522877', -- R12.IBY.C (iPayment)
'22326911' -- R12.ECX.C (Oracle XML Gateway)
) order by bug_number;

3) Backup CNOTEXT_FILE

$ cd (to a folder for backup folder)

$ cp -p $CONTEXT_FILE ${CONTEXT_NAME}.xml_BK_JRE

Replace "browser" by "jws" for <s_forms_launch_method>

$ vi $CONTEXT_FILE

Optionally, stop EBS services by adstpall.sh if they are running.

4) Follow ID 393931.1 (Deploying JRE for Windows Clients, go to a long table in Appendix E section) to download patch 30425890 - p30425890_180241_WINNT.zip (Oracle JRE 8 Update 241 for Windows 32-bit). I used JRE 1.8.0_241 because it is the version installed currently on all users' Window PC. If a higher version on the apps server is deployed, all users have to remove the lower version and re-install a matching version on their PC. (Note: later, I used p32140627_180281_WINNT.zip after users' PC got a newer JRE).

$ cd $COMMON_TOP/webapps/oacore/util/javaplugin

$ unzip p32140627_180281_WINNT.zip
Archive: p32140627_180281_WINNT.zip
inflating: jre-8u281-windows-i586.exe
inflating: jre-8u281-windows-i586.tar.gz
inflating: readme.txt

$ mv jre-8u281-windows-i586.exe j2se18281.exe

$ echo $FILE_EDITION

run

$ $FND_TOP/bin/txkSetPlugin.sh 18281

/u02/app/$TWO_TASK/fs1/EBSapps/appl/fnd/12.0.0/bin

Starting interoperability upgrade script...

Sending logfile output to:
$INST_TOP/logs/txkSetPlugin.log
etc … …

AutoConfig completed successfully.
Done

$ sh $ADMIN_SCRIPTS_HOME/adautocfg.sh (optional)

Go to the backup folder and confirm the configuration changes:

$ diff $CONTEXT_NAME.xml_BK_JRE $CONTEXT_FILE

< <config_option type="techstack" oa_var="s_forms_launch_method">browser</config_option>
> <config_option type="techstack" oa_var="s_forms_launch_method">jws</config_option>

< <sun_plugin_ver oa_var="s_sun_plugin_ver">1.6.0_27</sun_plugin_ver>
> <sun_plugin_ver oa_var="s_sun_plugin_ver">1.8.0_281</sun_plugin_ver>

< <sun_clsid oa_var="s_sun_clsid">CAFEEFAC-0016-0000-0027-ABCDEFFEDCBA</sun_clsid>
> <sun_clsid oa_var="s_sun_clsid">CAFEEFAC-0018-0000-0281-ABCDEFFEDCBA</sun_clsid>

... ...

$ cat $FORMS_WEB_CONFIG_FILE|grep sun_plugin_version| cut -c 1-35

sun_plugin_version=1.8.0_281

Next is to sign JAR file. Otherwise, launching EBS Forms will give a popup with Exception: JAR resources in JNLP file are not signed by same certificate.

5) Jar signing

Follow Doc ID 1591073.1 (Enhanced Jar Signing for Oracle E-Business Suite)

a) copy two files adkeystore.dat and adsign.txt (from old $APPL_TOP/admin of R12.1.3) to folder $NE_BASE/EBSapps/appl/ad/admin, assuming JAR signing worked in R12.1.3 instance. Verify basic info:

SQL> set serveroutput on

SQL> declare

spass varchar2(30);

kpass varchar2(30);

begin

ad_jar.get_jripasswords(spass, kpass);

dbms_output.put_line(spass);

dbms_output.put_line(kpass);

end;

$ keytool -list -keystore adkeystore.dat

Enter keystore password:

Keystore type: JKS

Keystore provider: SUN

Your keystore contains 3 entries

ebsdev_nodename, April 10, 2021, PrivateKeyEntry, ...

... ...

If necessary, use command to create/re-new .dat file from the one for instance1 on the node1:

$ keytool -changealias -alias instance1Name_node1Name -destalias EBSDEV_nodeName -keystore adkeystore.dat

b) Make sure all apps services are stopped

c) $ adadmin

=> 1 (Generate Applications Files menu)
=> 4 (Generate product JAR files)

Do you wish to force regeneration of all jar files? [No] ? Yes

Forcing generation of all product jar files.
Creating and signing every jar file can take about thirty minutes depending on the hardware being used.
You can watch the file $NE_BASE/EBSapps/log/adadmin/log/Gen_JAR.log to see the progress of jar file generation.

Recording Adadmin action :ADADMIN_GEN_JARS
Tokens:FORCE_JAR_GENERATION=Yes

Signing product JAR files in JAVA_TOP - $JAVA_TOP
using entity $CONTEXT_NAME and certificate 1.

Successfully created javaVersionFile.

Generating product JAR files in JAVA_TOP -
$JAVA_TOP with command:

adjava -mx2048m -nojit oracle.apps.ad.jri.adjmx @$APPL_TOP/admin/$TWO_TASK/out/genjars.cmd

Successfully generated product JAR files in JAVA_TOP -

$JAVA_TOP.

Copying Registry.dat from the Forms Java directory to $JAVA_TOP ...

Generating customall.jar ...

Not creating customall.jar as no custom java directories found under JAVA_TOP.

customall.jar generated successfully.

** Updating appsborg.zip ...

Reading adjborg.txt...

adearea();
No files listed in $APPL_TOP/admin/adjborg.txt
Ignoring ...
adjborg.txt file seems to be empty.

Done reading adjborg.txt.
As adjborg.txt file is empty, skipping the generation of appsborg.zip

** Updating appsborg2.zip ...

Reading adjborg2.txt...

adearea();
No files listed in $APPL_TOP/admin/adjborg2.txt
Ignoring ...
adjborg.txt file seems to be empty.

Done reading adjborg2.txt.
As adjborg2.txt file is empty, skipping the generation of appsborg2.zip

Review the messages above, then press [Return] to continue.

Notes from Doc ID 2065496.1: In release 12.2 the files appsborg.zip and appsborg2.zip have been replaced with file ebsAppsborgManifest.jar. The new file ebsAppsborgManifest.jar should appear in the CLASSPATH. In summary for R12.2 the files appsborg.zip and appsborg2.zip are not used. The messages are expected and can be safely ignored.

$ echo $CLASSPATH | grep ebsAppsborgManifest

will give a long list

$ cd $FMW_HOME/Oracle_EBS-app1/shared-libs/ebs-appsborg/WEB-INF/lib

$ ls -al

-rw-r--r-- 1 user group 806 Feb 23 16:32 ebsAppsborgManifest.jar

d) To verify the digital signature of the Jar file:

$ jarsigner -verify -verbose -certs $AD_TOP/java/jar/adxlib.jar

… …

sm 6924 Fri Jan 14 03:00:00 EST 2011 oracle/apps/ad/util/zip/ZipOutputStream.class

      X.509, CN=Company_name Inc., O=Company_name Inc., L=City, ST=State, C=US
      [certificate is valid from 1/3/19 7:00 PM to 1/12/22 7:00 AM]
      X.509, CN=DigiCert SHA2 Assured ID Code Signing CA, OU=www.digicert.com, O=DigiCert Inc, C=US
      [certificate is valid from 10/22/13 8:00 AM to 10/22/28 8:00 AM]
      X.509, CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
      [certificate is valid from 11/9/06 7:00 PM to 11/9/31 7:00 PM]

      X.509, CN=nodename, O=nodename, OU=apps, L=, ST=, C=US
      [certificate is valid from 2/12/21 12:02 PM to 2/2/61 12:02 PM]
      [CertPath not validated: Path does not chain with any of the trust anchors]

s = signature was verified
m = entry is listed in manifest
k = at least one certificate was found in keystore
i = at least one certificate was found in identity scope

jar verified.

Warning:
This jar contains signatures that does not include a timestamp. Without a timestamp, users may not be able to validate this jar after the signer certificate's expiration date (2022-01-12) or after any future revocation date.

If you see message "This jar contains entries whose certificate chain is not validated.", it had a signing problem.

e) Start apps services by adstrtal.sh (or, start them later after TLS1.2 is enabled).

After about steps, R12.2.10 webpage and Forms can be launched from MS Edge and Google Chrome. The file download may be slow, and so do not click too quick. If EBS forms does not open up in the 1st launch, you may go the downlead directory in Window's Explore and double click on file frmservlet.jnlp to fire the Java form up.

6) Useful commands

- To view the keystore contents

$ keytool -list -v -keystore $NE_BASE/EBSapps/appl/ad/admin/adkeystore.dat
Enter keystore password: <== get it by a SQL if forgot.

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: <ebsdev_nodename> <== all are lower case and $TWO_TASK is EBSDEV
Creation date: Apr 2, 2019
Entry type: PrivateKeyEntry
Certificate chain length: 3
Certificate[1]:

etc ... …

- To sign a single jar file fndgantt.jar:

$ jarsigner -keystore $NE_BASE/EBSapps/appl/ad/admin/adkeystore.dat $FND_TOP/java/jar/fndgantt.jar ebsdev_nodename
Enter Passphrase for keystore:
Enter key password for <ebsdev_nodename>:
jar signed.

Warning:
No -tsa or -tsacert is provided and this jar is not timestamped. Without a timestamp, users may not be able to validate this jar after the signer certificate's expiration date (2022-01-12) or after any future revocation date.

- Another way to to verify a digital signing by adding option "-keystore":

$ jarsigner -verify -verbose -keystore $NE_BASE/EBSapps/appl/ad/admin/adkeystore.dat -certs $JAVA_TOP/oracle/apps/fnd/jar/fndaol.jar

- In custom development by Java, use below line to generate and sign mycustom.jar file, assuming mycustom.zip has all .class files and .xml files (and .xml file was imported to database by java oracle.jrad.tools.xml.importer.XMLImporter):

$ adjava oracle.apps.ad.jri.adjmx -areas $JAVA_TOP/mycustom.zip -outputFile $JAVA_TOP/mycustom.jar -jar $CONTEXT_NAME 1 CUST jarsigner -storePass <KeyStore Password> -keyPass <Key Password>