FND_OAM_CONTEXT_FILES misses a row of context file

I got below error from using ADOP to apply patches:

*******FATAL ERROR*******

PROGRAM : ($AD_TOP/patch/115/bin/txkADOPEvalSrvStatus.pl)

TIME    : Thu Feb 24 12:26:26 2022

FUNCTION: TXK::XML::load_doc [ Level 1 ]

MESSAGES:

error = Cannot open XML file for load

errorno = No such file or directory

file = $NE_BASE/EBSapps/log/adop/.../apply/node_name/TXK_EVAL_apply_.../ctx_files/$CONTEXT_NAME.xml

STACK TRACE

 at $AU_TOP/perl/TXK/Error.pm line 168

        TXK::Error::abort('TXK::Error', 'HASH(0x3d523b0)') called at $AU_TOP/perl/TXK/XML.pm line 280

        TXK::XML::load_doc('TXK::XML=HASH(0x3d49840)', 'HASH(0x3d1c698)') called at $AU_TOP/perl/TXK/XML.pm line 266

        TXK::XML::loadDocument('TXK::XML=HASH(0x3d49840)', 'HASH(0x3d1c698)') called at $AD_TOP/patch/115/bin/txkADOPEvalSrvStatus.pl line 1238

... ...        

The error and logs do not tell what is the problem. But when I ran ADOP validation, it gave me direction to find the root cause.

$ adop -validate

... ... 

    [UNEXPECTED] Could not find patch context file from database

    [UNEXPECTED]Remote action failed.

... ...

"adop phase=fs_clone" can hit the same error:

   Verifying existence of context files in database.

       [UNEXPECTED] Could not find patch context file from database

       UNEXPECTED]Patch edition context file not found in database for host node_NAME

1. What was the problem?

Most likely "EXEC FND_CONC_CLONE.SETUP_CLEAN;" was executed during a clone or somewhere, which deleted rows in tables:

SQL> select * from fnd_nodes;

SQL> select * from fnd_oam_context_files;

I used query to confirm the row for PATCH context file of the newly added node was not in table FND_OAM_CONTEXT_FILES:

SQL> select distinct(PATH) from FND_OAM_CONTEXT_FILES 

where NAME not in ('TEMPLATE','METADATA','config.txt') and CTX_TYPE='A' 

and (status is null or upper(status) in ('S','F')) and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = 'patch';

2. Fix: run AutoConfig on PATCH file system

a) Disable a trigger

SQL> conn system/systemPWD

Connected.

SQL> alter trigger ebs_logon disable;

Trigger altered.

b) Set PATCH env and run autoConfig

$ . /<EBS_HOME_BASE>/EBSapps.env patch

$ echo $TWO_TASK

EBSDEV_patch

$ sqlplus apps/appsPWD   -- modify tnsnames.ora if it failed

SQL> show user

USER is "APPS"

SQL> exit

$ cd $ADMIN_SCRIPTS_HOME

$ echo $FILE_EDITION

patch

$ ./adautocfg.sh

Enter the APPS user password:

The log file for this session is located at: $INST_TOP/admin/log/MMDDHHMI/adconfig.log

... ...

AutoConfig completed successfully.

c). Enable the trigger

SQL> conn system/systemPWD

Connected.

SQL> alter trigger ebs_logon enable;

Trigger altered.

d). Confirm two rows in FND_OAM_CONTEXT_FILES for patch filesystem now

SQL> connect apps/appsPWD

SQL> select distinct(PATH) from FND_OAM_CONTEXT_FILES 

where NAME not in ('TEMPLATE','METADATA','config.txt') and CTX_TYPE='A' 

and (status is null or upper(status) in ('S','F')) 

and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = 'patch';

AutoConfig also inserted one row to table applsys.FND_APPL_TOPS.

3. Run ADOP again from the beginning. All shall work.

Start a new OS session 

$ echo $FILE_EDITION

run

$  adop -validate

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

.. ...

adop exiting with status = 0 (Success)

Notes: 

1. After AutoConfig on patch filesystem, it may bring a wrong login settings to the site. You will have to run AutoConfig in RUN filesystem to overwrite them.

2. If this problem was from "adop phase=fs_clone", run it again by "adop phase=fs_clone force=yes" to start it from the beginning to avoid the Apache failure (because some wrong info was saved in adop from previous run).

Unable to initialize SSL environment, nzos call nzosSetCredential returned 28791

Another possible way to fix the problem:

$ . ./EBSapps.env patch

$ echo $FILE_EDITION

patch

$ echo $PATCH_BASE

<EBS_HOME_BASE>/fs2

$ $ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer \

        action=upload \ contextfile=/path on fs2/to/$CONTEXT_FILE

        logfile=/path/to/patchupload.log

REFERENCE:  Doc ID 2090393.1 (Error "Use of uninitialized value $result in split" In File txkADOPValidations.error When Running fs_clone)

Clear BNE / WebADI cache in R12.2

In R12.2, URL link https://[hostname.domain]:portnumber/OA_HTML/BneAdminServlet does not work by default.  It works for user with Desktop Integration Manager responsibility.

After logged in under Desktop Integration Manager responsibility, click on a link to open any Oracle Application Framework (FWK) page. Then, try:

https://[hostname.domain]:portnumber/OA_HTML/BneAdminServlet 

It provides a link to clear BNE cache.

The URL may give error in the first try:

Desktop Integration Manager is not a valid responsibility for the current user. Please contact your System Administrator.

Clear the middle tier cache to resolve this issue, because middle tier has not picked the change in assigned responsibility yet:

a) Navigate to Functional administrator > Core Service > Caching Framework > Global Configuration

b) Click on Clear All Cache 

c) Try the link again.

Upgrade JDK 7 in EBS R12.2

EBS installation brings in low version of JDK. It is necessary to upgrade JDK to address security vulnerabilities and to meet the need of product support. The easiest way to find the latest JDK version that works with EBS is Oracle release document for quarterly CPU patches. Currently. EBS R12.2 is certified only with JDK 1.7.

EBS R12.2 uses both 64-bit and 32-bit JDK. Use below lines to find current JDK version in an EBS environment:

$ echo $CLASSPATH

$COMMON_TOP/util/jdk32/lib/dt.jar:/... ... etc

$ $ADJVAPRG -version

java version "1.7.0_85"

Java(TM) SE Runtime Environment (build 1.7.0_85-b15)

Java HotSpot(TM) Server VM (build 24.85-b06, mixed mode)

$ echo $AF_CLASSPATH

$COMMON_TOP/util/jdk64/lib/dt.jar:/... ... etc

$ $AFJVAPRG -version

java version "1.7.0_85"

Java(TM) SE Runtime Environment (build 1.7.0_85-b15)

Java HotSpot(TM) 64-Bit Server VM (build 24.85-b06, mixed mode)

The CLASSPATH and AF_CLASSPATH environment variable must contain the necessary Java 7.0 libraries, which should include:

[COMMON_TOP]/util/<Java7>/lib/dt.jar

[COMMON_TOP]/util/<Java7>/lib/tools.jar

[COMMON_TOP]/util/<Java7>/jre/lib/rt.jar

1.7.0_85 is the default version from R12.2 upgrade, and Weblogic console does give warning:

Warning: This server is not secure. Upgrade to Java 1.7.0_191 or higher.

Follow steps in Scenario 3.5.2 of Doc ID 1530033.1 (Using the Latest JDK 7.0 Update with Oracle E-Business Suite Release 12.2) to upgrade it. Currently, JDK 7 Update 331 is recommended by the most recent R12.2 CPU patch (January 2022, Doc ID 2815550.1)

1. First of all, use link in Doc ID 1439822.1 to download patch 33518566 (Oracle JDK 7 Update 331) for both 64 & 32 bit, and then save two files p33518566_170331_Linux-x86-64.zip & p33518566_170331_LINUX.zip to a shared location /u46/path/to/JDK331. After unzip, two files jdk-7u331-linux-x64.tar.gz & jdk-7u331-linux-i586.tar.gz provide two folders under /u46/path/to/JDK331 for staging:

$ tar vzxfp jdk-7u331-linux-x64.tar.gz

$ mv jdk1.7.0_331 jdk1.7.0_331_64

$ tar vzxfp jdk-7u331-linux-i586.tar.gz

$ mv jdk1.7.0_331 jdk1.7.0_331_32

UPDATES in February 2023:

Oracle ended support for JDK 7 after July 2022 (See NOTE:2809708.1 - Oracle Java SE 7 End of Extended Support). So, It is important to read CPU release documents to find where to download the right JDK file for upgrading JDK embedded in E-Business Suites. For example, January 2023 CPU document 2897309.1 ( Oracle E-Business Suite Release 12.2 Critical Patch Update Availability Document (January 2023)) points to Doc ID 2897309.1 (Oracle Critical Patch Update (CPU) Oct 2022 for Oracle Java SE), which has a link to download JDK 1.7.0_361 (patch 34569003) files p34569003_170361_LINUX.zip and p34569003_170361_Linux-x86-64.zip.

2. Confirmed below patches were applied to $ORACLE_HOME:

OracleAS 10.1.2 Patch 16271876

OracleAS 10.1.2 Patch 17907988

OracleAS 10.1.2 Forms Patch 17653437

OracleAS 10.1.2 Patch 16241466

OracleAS 10.1.2 Patch 17645157

Then, shutdown Apps services

$ { echo apps ; echo appsPWD; echo wlsPWD ; } | adstpall.sh @ -mode=allnodes -nopromptmsg

$ ps -ef | grep $LOGNAME

$ echo $FILE_EDITION

run

Make sure there is no open online patching cycle (which is started by "adop phase=prepare"):

$ adop -status

... ...

Node Name       Node Type  Phase           Status              Started            Finished      Elapsed

--------------- ------------------ --------------- ----------------- ---------------- -------------- ------------

node1       master            APPLY           ACTIVE             xxxxx            xxxxx            xxxxx

                                       CLEANUP     NOT STARTED

node2       slave              APPLY           ACTIVE              xxxxx            xxxxx           xxxxx

                                       CLEANUP     NOT STARTED

I upgraded JDK folders in RUN file system directly. I did not use ADOP patching cycle for that, and only ran FS_CLONE after JDK folders are replaced in RUN file system on all nodes.

$ JDKfolder=/u46/path/to/JDK331

$ cd $JDKfolder

$ ls

jdk1.7.0_331_32  jdk-7u331-linux-i586.tar.gz  p33518566_170331_Linux-x86-64.zip  readme.txt

jdk1.7.0_331_64  jdk-7u331-linux-x64.tar.gz   p33518566_170331_LINUX.zip

3. Install New JDK 7.0 and Replace the Existing JDK Home Contents

$ echo $COMMON_TOP/util

$RUN_BASE/EBSapps/comn/util

$ cd $COMMON_TOP/util

$ mv jdk64 jdk64_OLD_2022        <== Or, tar it up by "tar -cvzf jdk64_OLD.tar.gz jdk64"

                                                         <== then delete old folder jdk64

$ mv jdk32 jdk32_OLD_2022

$ cp -rp $JDKfolder/jdk1.7.0_331_64 jdk64

$ cp -rp $JDKfolder/jdk1.7.0_331_32 jdk32

4. Upgrade Java (64 bit) in Oracle Fusion Middleware 11g Web Tier

$ echo $FMW_HOME/webtier

$RUN_BASE/FMW_Home/webtier

$ cd $FMW_HOME/webtier

$ mv jdk jdk_OLD_2022

$ cp -rp $JDKfolder/jdk1.7.0_331_64 jdk  

5. Upgrade Java (32 bit) in OracleAS 10.1.2 Oracle_Home

$ echo $ORACLE_HOME

$RUN_BASE/EBSapps/10.1.2

$ cd $ORACLE_HOME

$ mv jdk jdk_OLD_2022

$ cp -rp $JDKfolder/jdk1.7.0_331_32 jdk

6. Rebuild Forms and Reports Executables (For UNIX/Linux Only)

Make sure ORACLE_HOME points to 10.1.2 Oracle Home before relinking forms and reports executables.

$ cd $ORACLE_HOME/forms/lib

$ make -f ins_forms.mk sharedlib install

... ...

/usr/bin/ld: warning: -z lazyload ignored.

$ cd $ORACLE_HOME/reports/lib

$ make -f ins_reports.mk install

7. Repeat the steps in other nodes

UPDATE in 2025: I wrote a a script to run above steps in one execution.

8. Run autoconfig on all apps-tier nodes

$ cd $ADMIN_SCRIPTS_HOME

$ ./adautocfg.sh

9. Verify and start Apps services

$ $ADJVAPRG -version

java version "1.7.0_331"

Java(TM) SE Runtime Environment (build 1.7.0_331-b06)

Java HotSpot(TM) Server VM (build 24.331-b06, mixed mode)

$ $AFJVAPRG -version

java version "1.7.0_331"

Java(TM) SE Runtime Environment (build 1.7.0_331-b06)

Java HotSpot(TM) 64-Bit Server VM (build 24.331-b06, mixed mode)

$ { echo apps ; echo appsPWD ; echo wlsPWD ; } | adstrtal.sh @ -mode=allnodes -nopromptmsg

A JSP file can be used as a test.

$ cd $OA_HTML

$ vi JDKtest.jsp

$ more JDKtest.jsp

The JDK version is: <%= System.getProperty("java.version") %>

$ perl -x $FND_TOP/patch/115/bin/ojspCompile.pl --compile -s 'JDKtest.jsp' 

Access the JSP file from a browser, using the URL:

http://[web server]:[port]/OA_HTML/JDKtest.jsp

10. If no other changes, sync patch file system by fs_clone:

$ adop phase=fs_clone

Additional NOTES:

A)  If downtime is not allowed, the upgrade has to be in PATCH file system first. The upgrade steps are in the Oracle document. 

a) Check that there is no open online patching cycle

$ adop -status

$ adop phase=prepare

b) Source the EBS environment with the patch file system

$ cd <EBS_BASE_HOME> 

         /* <EBS_BASE_HOME>  is where fs1, fs2, and others are installed. */

$ . EBSapps.env PATCH

$ echo $FILE_EDITION

c) Complete sames steps 3 - 7 above.

d) Run AutoConfig on PATCH file system on all nodes

e) Cut over to the newly installed JDK 7.0 patch file system

$ adop phase=cutover

B)  I used the same way to upgrade old JDK in an Oracle 11g Client by replacing the jdk folder under $ORACLE_HOME to address vulnerability found by Qualys scan report.

C)  JDK is also installed by rpm on Linux server level. I am not sure EBS needs it or not. Here is from one of my EBS RHEL7 servers:

$ rpm -qa | grep jdk

copy-jdk-configs-3.3-11.el7_9.noarch

java-1.8.0-openjdk-1.8.0.382.b05-1.el7_9.x86_64

java-1.8.0-openjdk-headless-1.8.0.382.b05-1.el7_9.x86_64

jdk1.7-1.7.0_391-fcs.x86_64

$ which java

/bin/java

$ java -version

openjdk version "1.8.0_382"      <== It is the OS Java

OpenJDK Runtime Environment (build 1.8.0_382-b05)

OpenJDK 64-Bit Server VM (build 25.382-b05, mixed mode)

$ cd /usr/java                              <== It is for apps. I guess it may not be used

$ ls -altr

lrwxrwxrwx   1 root root          16 Feb 19  2020 default -> /usr/java/latest

drwxr-xr-x      8 root root      4096 Aug  3 09:27 jdk1.7.0_391-amd64

lrwxrwxrwx   1 root root           28 Aug  3 09:27 latest -> /usr/java/jdk1.7.0_391-amd64

Renew Java Signing

When the current Java Code Signing file will expire in days, a big question is: if we do not renew it, will the EBS Forms still work as usual?  From my experience, even the Java Signing cert expired, EBS Forms will continue functioning. 

But, if a expired adkeystore.dat file is used to re-sign JAR files by adadmin (as a patching step, etc),  the newly signed forms may be blocked by pop-up message in a "Java Application Blocked" box: Your security settings have blocked an application signed with an expired or not-yet-valid certificate from running.

In this case, you can go to Java console on local machine, Java >> Security >> Exception Site List >> add the EBS site URL. Then, you can open EBS GUI forms as a workaround.

When a wrong adkeystore.dat is used by adadmin to generate JAR files, EBS Forms will not open up by popup message:

JAR resources in JNLP file are not signed by same certificate

Also see Oracle Doc ID 2605947.1 (Forms Jar File Certificate Expiration and Timestamp Functionality). 

Warning on JAR file when it closes to expiration:

$ jarsigner -verify -verbose -certs $AD_TOP/java/jar/adxlib.jar

... ...

Warning:

This jar contains entries whose signer certificate will expire within six months.

This jar contains signatures that does not include a timestamp. Without a timestamp, users may not be able to validate this jar after the signer certificate's expiration date (2022-01-12) or after any future revocation date.

Warning after expired:

$ jarsigner -verify -verbose -certs $AD_TOP/java/jar/adxlib.jar

... ...

Warning:

This jar contains entries whose signer certificate has expired.

Steps to renew Java Signing cert, if you have to:

1. I emailed the old adkeystore.dat file to a colleague in Security team. He emailed back the renewed file. He said he downloaded it from Venafi (Venafi Trust Protection Platform) site as JKS (.jks) file with the same old password, and then he renamed it to adkeystore.dat. 

There is no need to create separate cert for each server. Code signing is the same for all servers but has a different alias. Key command to change the alias for another instance (e.g. from EBSDEV instance to EBSQA):

$ keytool -changealias -alias ebsdev_devNode1d -destalias EBSQA_qaNode1q -keystore adkeystore.dat

SQL> set serveroutput on    -- pull out passwords

SQL> declare

spass varchar2(30);

kpass varchar2(30);

begin

ad_jar.get_jripasswords(spass, kpass);

dbms_output.put_line(spass);

dbms_output.put_line(kpass);

end;  

/

puneet

myxuan

2. Copy the new cert file to $NE_BASE/EBSapps/appl/ad/admin (in R12.2)

$ cd $NE_BASE/EBSapps/appl/ad/admin

$ mv adkeystore.dat adkeystore.dat_OLD

$ keytool -list -keystore adkeystore.dat_OLD

Enter keystore password:

Keystore type: JKS

Keystore provider: SUN

Your keystore contains 1 entry

ebsdev_devNode1d, Apr 5, 2019, PrivateKeyEntry,      <== that is the file creation/modification date

Certificate fingerprint (SHA1): A7:01:6E:76:D7:2D:F5:69:8B:93:2B:C0:CA:E5:41:7C:DB:51:45:AA

$ keytool -list -keystore adkeystore.dat    <== new file from renewal

Enter keystore password:

Keystore type: JKS

Keystore provider: SUN

Your keystore contains 3 entries      <== new file could still say "1 entry". Not sure why

cn=digicert assured id root ca, ou=www.digicert.com, o=digicert inc, c=us, Jan 3, 2022, trustedCertEntry,

Certificate fingerprint (SHA1): 05:63:B8:63:0D:62:D7:5A:BB:C8:AB:1E:4B:DF:B5:A8:99:B2:4D:43

cn=digicert sha2 assured id code signing ca, ou=www.digicert.com, o=digicert inc, c=us, Jan 3, 2022, trustedCertEntry,

Certificate fingerprint (SHA1): 92:C1:58:8E:85:AF:22:01:CE:79:15:E8:53:8B:49:2F:60:5B:80:C6

ebsdev_devNode1d, Jan 3, 2022, PrivateKeyEntry,

Certificate fingerprint (SHA1): AF:78:49:69:75:3D:D8:6E:70:F9:74:69:42:BB:A9:19:6B:03:9D:E

Use "keytool -list -v -keystore adkeystore.dat" to see the keystore's validation period and other details.

3. Run command to change the alias for QA server:

$ keytool -changealias -alias ebsdev_devNode1d -destalias EBSQA_qaNode1q -keystore adkeystore.dat

Enter keystore password:

Enter key password for <ebsdev_devNode1d>

Warning:

The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore adkeystore.dat  -destkeystore adkeystore.dat -deststoretype pkcs12".

$ keytool -list -keystore adkeystore.dat

Enter keystore password:

Keystore type: JKS

Keystore provider: SUN

Your keystore contains 3 entries

cn=digicert assured id root ca, ou=www.digicert.com, o=digicert inc, c=us, Jan 3, 2022, trustedCertEntry,

Certificate fingerprint (SHA1): 05:63:B8:63:0D:62:D7:5A:BB:C8:AB:1E:4B:DF:B5:A8:99:B2:4D:43

cn=digicert sha2 assured id code signing ca, ou=www.digicert.com, o=digicert inc, c=us, Jan 3, 2022, trustedCertEntry,

Certificate fingerprint (SHA1): 92:C1:58:8E:85:AF:22:01:CE:79:15:E8:53:8B:49:2F:60:5B:80:C6

ebsqa_qaNode1q, Jan 3, 2022, PrivateKeyEntry,

Certificate fingerprint (SHA1): AF:78:49:69:75:3D:D8:6E:70:F9:74:69:42:BB:A9:19:6B:03:9D:E

Extra steps (that may not be necessary):

$ cp -p adkeystore.dat adkeystore.dat_BK

$ keytool -importkeystore -srckeystore adkeystore.dat -destkeystore adkeystore.dat -deststoretype pkcs12

Enter source keystore password:

Entry for alias cn=digicert assured id root ca, ou=www.digicert.com, o=digicert inc, c=us successfully imported.

Enter key password for <ebsqa_qaNode1q>

Entry for alias ebsqa_qaNode1q successfully imported.

Entry for alias cn=digicert sha2 assured id code signing ca, ou=www.digicert.com, o=digicert inc, c=us successfully imported.

Import command completed:  3 entries successfully imported, 0 entries failed or cancelled

Warning:

<cn=digicert assured id root ca, ou=www.digicert.com, o=digicert inc, c=us> uses the SHA1withRSA signature algorithm which is considered a security risk. This algorithm will be disabled in a future update.

Migrated "adkeystore.dat" to Non JKS/JCEKS. The JKS keystore is backed up as "adkeystore.dat.old".

$ ls -altr adkeystore.dat*

-rw-r----- 1 user users 4992 Apr  2  2019 adkeystore.dat_OLD

-rw-r--r-- 1 user users 7498 Mar  1 11:39 adkeystore.dat_BK

-rw-r--r-- 1 user users 7498 Mar  1 11:41 adkeystore.dat.old

-rw-r--r-- 1 user users 8721 Mar  1 11:41 adkeystore.dat

4. Make sure file adsign.txt is correct

$ more adsign.txt

EBSQA_qanode1q 1 CUST

5. Run adadmin to generate JAR files

$ adamin

 ==> 1 ==> 4

Do you wish to force regeneration of all jar files? [No] ? Yes

Forcing generation of all product jar files.

Creating and signing every jar file can take about thirty minutes depending on the hardware being used.

... ...

Generating customall.jar ...

Not creating customall.jar as no custom java directories found under JAVA_TOP.

customall.jar generated successfully.

   ** Updating appsborg.zip ...

    Reading adjborg.txt...

adearea();

No files listed in $RUN_BASE/EBSapps/appl/admin/adjborg.txt

Ignoring ...

adjborg.txt file seems to be empty.

    Done reading adjborg.txt.

As adjborg.txt file is empty, skipping the  generation of appsborg.zip

   ** Updating appsborg2.zip ...

    Reading adjborg2.txt...

adearea();

No files listed in $RUN_BASE/EBSapps/appl/admin/adjborg2.txt

Ignoring ...

adjborg.txt file seems to be empty.

    Done reading adjborg2.txt.

As adjborg2.txt file is empty, skipping the  generation of appsborg2.zip

6. Confirm the new expiration date

jarsigner -verify -verbose -certs $AD_TOP/java/jar/adxlib.jar

... ...

- Signed by "CN=Company_name, O=Company_name, L=City, ST=State_, C=US"

    Digest algorithm: SHA-256

    Signature algorithm: SHA256withRSA, 4096-bit key

jar verified.

Warning:

This jar contains signatures that do not include a timestamp. Without a timestamp, users may not be able to validate this jar after any of the signer certificates expire (as early as 2024-04-05).

The signer certificate will expire on 2024-04-05.

7. Test the result by opening a few GUI forms. 

It is not necessary to test the whole site.

Also see Enable Java Web Start with Oracle E-Business Suite R12.2

Re-new R12.2 ssl certificate

Certificate has to be renewed before it expires, because after it expires, "adapcctl.sh start" will fail to start Apache service in R12.2. Steps to renew it in R12.2.10:

1. Renew the cert file.

This the key step and may be outside of your control. The process of creating a certificate is always:

- Create public\private key pair.

- Create CSR.

- CA issues certificate based on CSR.

In most cases, the first two steps are done in a cert tool, such as Venafi. Then Venafi talks to DigiCert to get the signed certificate which can be download as .pfx file. After it is renamed to ewallet.p12, OWM (Oracle Wallet Manager) 10.1.0.5.0 in 10.1.3 ORACLE_HOME of R12.1 can read it and turn "Auto Login" on.

But OWM 11.1.1.9.0 at $FMW_HOME/webtier/bin/ in R12.2.10 can not read .pfx file correctly, simply showing "Certificate[Empty]". We have to use orapki under $FMW_HOME/oracle_common/bin to turn "Auto Login" on as a workaround.

2. Copy the new cert file to $NE_BASE/inst/$CONTEXT_NAME/certs/Apache as ewallet.p12

This Apache folder can serve as a staging location. Depending on instance setup, it may not be used in R12.2 at all.

3. Verify the cert can be open and its password is good (optional):

$ alias orapki=$FMW_HOME/oracle_common/bin/orapki

$ which orapki                     <= do not use the one in 10.1.2 ORACLE_HOME

$ orapki wallet display -wallet $NE_BASE/inst/$CONTEXT_NAME/certs/Apache/ewallet.p12

Enter wallet password:   xxxxx

Requested Certificates:

User Certificates:

Subject:        CN=site_name.domian.com,O=company_name,L=city,ST=AZ,C=US

Trusted Certificates:

Subject:        CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US

Subject:        CN=company_name Secure CA2,O=company_name,C=US

Notes:  Without specifying the path to file ewallet.p12, orapki in $FMW_HOME (vs. in 10.1.2 ORACLE_HOME) will read a default cert file in somewhere and may give incorrect password error.

4. Run below line to enable Auto Login

$ orapki wallet create -wallet $NE_BASE/inst/$CONTEXT_NAME/certs/Apache/ewallet.p12 -pwd 'sslPWD' -auto_login

Oracle PKI Tool : Version 11.1.1.9.0

Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.

"orapki wallet create ..." will not destroy or modify the original wallet! It creates cwallet.sso in the current directory. You can now view its contents without a password:

$ orapki wallet display -wallet $NE_BASE/inst/$CONTEXT_NAME/certs/Apache/cwallet.sso

Oracle PKI Tool : Version 11.1.1.9.0

Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:

User Certificates:

Subject:        CN= ... ...

... ...

5. Stop apps services, and copy cwallet.sso to other locations:

$ adstpall.sh apps/appsPWD

$ iName=$(tr < $CONTEXT_FILE '<>' '  ' | awk '/"s_ohs_instance"/ {print $(NF-1)}' )

$ echo $iName

EBS_web_EBSDEV_OHS1

$ cd $FMW_HOME/webtier/instances/$iName

$ find . -name cwallet.sso | fgrep -v /webgate/

./config/OPMN/opmn/wallet_ORIG/cwallet.sso

./config/OPMN/opmn/wallet/cwallet.sso

./config/OHS/EBS_web_EBSDEV/keystores/default/cwallet.sso

./config/OHS/EBS_web_EBSDEV/proxy-wallet/cwallet.sso

Use one line to see their contents:

$ find . -name cwallet.sso | fgrep -v /webgate/ | while read w ; do echo -e "\n$w"; orapki wallet display -nologo -complete -wallet $w ; done

$ ls -al config/OPMN/opmn/wallet/cwallet.sso

-rw------- 1 user group 6369 Feb 13  2020 config/OPMN/opmn/wallet/cwallet.sso

$ ls -al config/OHS/EBS_web_EBSDEV/keystores/default/cwallet.sso

-rw------- 1 user group 6369 Feb 13  2020 config/OHS/EBS_web_EBSDEV/keystores/default/cwallet.sso

$ ls -al config/OHS/EBS_web_EBSDEV/proxy-wallet/cwallet.sso

-rw------- 1 user group 4373 Jun 29  2021 config/OHS/EBS_web_EBSDEV/proxy-wallet/cwallet.sso

                         <== a default file from upgrade installation? Seems it is not used.

Now, back up old .sso files and use new file $NE_BASE/inst/$CONTEXT_NAME/certs/Apache/cwallet.sso to replace them.

6. Copy ewallet.p12 and cwallet.sso to all other nodes.

Note: New cwallet.sso has to be copied to all folders on concurrent node as well, only IF ssl was enabled on that node. Otherwise, "adop phase=fs_clone" may fail with message "Unable to initialize SSL environment, nzos call nzosSetCredential returned 28791" and strange error on the node:

LOG FILE: $INST_TOP/logs/appl/rgf/TXK/txkSetAppsConf_10141635.log ***

ERROR: The value <$FMW_HOME/webtier/instances/EBS_web_OHS3> for s_ohs_instance_loc in $CONTEXT_FILE isn't a directory.

7. Start/stop Apache as a quick test

$ adapcctl.sh start             <== Do NOT use "adopmnctl.sh startall"

$ adopmnctl.sh status

$ adopmnctl.sh stopall

8. Run autoconfig 

9. Star apps services

UPDATE in 2024: I put all above steps to a shell script for renewing cert.

- TROUBLESHOOTING

$ SUBiName=${iName%?????}

$ echo $SUBiName

Its OHS logs from adapcctl.sh are at $FMW_HOME/webtier/instances/$iName/diagnostics/logs/OHS/$SUBiName

And opmn.log is at $FMW_HOME/webtier/instances/$iName/diagnostics/logs/OPMN/opmn

a cwallet.sso file also exists in two folders under $EBS_DOMAIN_HOME/opmn/EBS*/wallet. I am not sure what it is for. But you can use orapki to view them, such as 

$ $ $FMW_HOME/oracle_common/bin/orapki wallet display -wallet $EBS_DOMAIN_HOME/opmn/EBS_web_OHS1/wallet/cwallet.sso

$ $FMW_HOME/oracle_common/bin/orapki wallet display -wallet $EBS_DOMAIN_HOME/opmn/EBS_web_OHS2/wallet/cwallet.sso

- A way to get server.cer file, read the contents of cert file, and find the expiration date of a cert file (Doc ID 2674842.1):

a) Verify the password works

$ $FMW_HOME/oracle_common/bin/orapki wallet display -wallet /path/to/ewallet.p12

Oracle PKI Tool : Version 11.1.1.9.0

Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.

Enter wallet password:

Requested Certificates:

User Certificates:

Subject:        CN=site_name.domian.com,O=company_name,L=city,ST=AZ,C=US

Trusted Certificates:

Subject:        CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US

Subject:        CN=company_name. Secure CA2,O=company_name,C=US

b) If troubleshooting is needed, export the certificate present in the wallet to a file

Copy the "Subject" information under the "User Certificates" section above and run command to generate server.cer file:

$ orapki wallet export -wallet /path/to/ewallet.p12 -dn 'CN=site_name.domian.com,O=company_name,L=city,ST=AZ,C=US' -cert server.crt

                                  <== it generates .crt file or .cer file, or both are the same>

$ more server.crt

c) Use the keytool to check the signature algorithm and expiry details

$ which keytool

$COMMON_TOP/util/jdk32/jre/bin/keytool      (It is a Java tool)

$ keytool -printcert -file server.crt

Owner: CN=site_name.domian.com,O=company_name,L=city,ST=AZ,C=US

Issuer: CN=company_name Secure CA2,O=company_name,C=US

Serial number: bbef1120a01b29a5c3c213623xxxxxxxx

Valid from: Thu Jan 12 20:00:00 EDT 2021 until: Tue Jan 16 19:59:59 EDT 2022

... ...

d) To read the full contents of server.cer, run openssl:

$ which openssl

/usr/bin/openssl

$ openssl version

OpenSSL 1.0.2k-fips  26 Jan 2017

$ openssl x509 -in server.crt -noout -text >> server.txt

$ more server.txt

Certificate:

    Data:

        Version: 3 (0x2)

        Serial Number:

            05:36:63:2d:8f:8d:4b:1f:64:45:84:5a:ce:49:1a:8e

    Signature Algorithm: sha256WithRSAEncryption

        Issuer: ... ...

... ...

- Workarounds

Since OWM 11.1.1.9 in R12.2 can not be used to read cert file, one option is to create a key pair and CSR in OWM using old cert file. After the CSR is used within Venafi to enroll for the certificate, manually install the certificate back in OWM. So, we have to create the request within OWM in multiple steps. In this case, the password for old certificate must be good. 

The another option is to use OWM in 10.1.3 ORACLE_HOME on R12.1 file system to open and save the new cert file (if old R12.1 file system is still available).

References (for creating/renewing cert file. I did not use all of them):

- Oracle Wallet Manager Shows "Certificate:Empty" When Generating Wallets Using An Existing Certificate & Key (Doc ID 1582186.1)

- How to Check Validity of Server Certificate from ORAPKI Wallet in Command Line (Doc ID 2674842.1)

- How to Replace an Expired or Expiring Certificate in FMW 11g Wallets? (Doc ID 1371209.1)

- 12.2 E-Business Suite Technology Stack Upgrade Steps To Reuse/ Migrate 11i Certificates To 12.2 Wallet Format To Replace Cwallet.sso (Doc ID 2315134.1)