Friday, February 11, 2022

Renew Java Signing

When the current Java Code Signing file will expire in days, a big question is: if we do not renew it, will the EBS Forms still work as usual?  From my experience, even the Java Signing cert expired, EBS Forms will continue functioning. 

But, if a expired adkeystore.dat file is used to re-sign JAR files by adadmin (as a patching step, etc),  the newly signed forms may be blocked by pop-up message in a "Java Application Blocked" box: Your security settings have blocked an application signed with an expired or not-yet-valid certificate from running.

In this case, you can go to Java console on local machine, Java >> Security >> Exception Site List >> add the EBS site URL. Then, you can open EBS GUI forms as a workaround.

When a wrong adkeystore.dat is used by adadmin to generate JAR files, EBS Forms will not open up by popup message:
JAR resources in JNLP file are not signed by same certificate

Also see Oracle Doc ID 2605947.1 (Forms Jar File Certificate Expiration and Timestamp Functionality). 

Warning on JAR file when it closes to expiration:
$ jarsigner -verify -verbose -certs $AD_TOP/java/jar/adxlib.jar
... ...
Warning:
This jar contains entries whose signer certificate will expire within six months.
This jar contains signatures that does not include a timestamp. Without a timestamp, users may not be able to validate this jar after the signer certificate's expiration date (2022-01-12) or after any future revocation date.

Warning after expired:
$ jarsigner -verify -verbose -certs $AD_TOP/java/jar/adxlib.jar
... ...
Warning:
This jar contains entries whose signer certificate has expired.

Steps to renew Java Signing cert, if you have to:

1. I emailed the old adkeystore.dat file to a colleague in Security team. He emailed back the renewed file. He said he downloaded it from Venafi (Venafi Trust Protection Platform) site as JKS (.jks) file with the same old password, and then he renamed it to adkeystore.dat. 

There is no need to create separate cert for each server. Code signing is the same for all servers but has a different alias. Key command to change the alias for another instance (e.g. from EBSDEV instance to EBSQA):

$ keytool -changealias -alias ebsdev_devNode1d -destalias EBSQA_qaNode1q -keystore adkeystore.dat

SQL> set serveroutput on    -- pull out passwords
SQL> declare
spass varchar2(30);
kpass varchar2(30);
begin
ad_jar.get_jripasswords(spass, kpass);
dbms_output.put_line(spass);
dbms_output.put_line(kpass);
end;  
/
puneet
myxuan

2. Copy the new cert file to $NE_BASE/EBSapps/appl/ad/admin (in R12.2)

$ cd $NE_BASE/EBSapps/appl/ad/admin

$ mv adkeystore.dat adkeystore.dat_OLD
$ keytool -list -keystore adkeystore.dat_OLD
Enter keystore password:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

ebsdev_devNode1d, Apr 5, 2019, PrivateKeyEntry,      <== that is the file creation/modification date
Certificate fingerprint (SHA1): A7:01:6E:76:D7:2D:F5:69:8B:93:2B:C0:CA:E5:41:7C:DB:51:45:AA

$ keytool -list -keystore adkeystore.dat    <== new file from renewal
Enter keystore password:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 3 entries      <== new file could still say "1 entry". Not sure why

cn=digicert assured id root ca, ou=www.digicert.com, o=digicert inc, c=us, Jan 3, 2022, trustedCertEntry,
Certificate fingerprint (SHA1): 05:63:B8:63:0D:62:D7:5A:BB:C8:AB:1E:4B:DF:B5:A8:99:B2:4D:43
cn=digicert sha2 assured id code signing ca, ou=www.digicert.com, o=digicert inc, c=us, Jan 3, 2022, trustedCertEntry,
Certificate fingerprint (SHA1): 92:C1:58:8E:85:AF:22:01:CE:79:15:E8:53:8B:49:2F:60:5B:80:C6
ebsdev_devNode1d, Jan 3, 2022, PrivateKeyEntry,
Certificate fingerprint (SHA1): AF:78:49:69:75:3D:D8:6E:70:F9:74:69:42:BB:A9:19:6B:03:9D:E

Use "keytool -list -v -keystore adkeystore.dat" to see the keystore's validation period and other details.

3. Run command to change the alias for QA server:

$ keytool -changealias -alias ebsdev_devNode1d -destalias EBSQA_qaNode1q -keystore adkeystore.dat
Enter keystore password:
Enter key password for <ebsdev_devNode1d>

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore adkeystore.dat  -destkeystore adkeystore.dat -deststoretype pkcs12".

$ keytool -list -keystore adkeystore.dat
Enter keystore password:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 3 entries

cn=digicert assured id root ca, ou=www.digicert.com, o=digicert inc, c=us, Jan 3, 2022, trustedCertEntry,
Certificate fingerprint (SHA1): 05:63:B8:63:0D:62:D7:5A:BB:C8:AB:1E:4B:DF:B5:A8:99:B2:4D:43
cn=digicert sha2 assured id code signing ca, ou=www.digicert.com, o=digicert inc, c=us, Jan 3, 2022, trustedCertEntry,
Certificate fingerprint (SHA1): 92:C1:58:8E:85:AF:22:01:CE:79:15:E8:53:8B:49:2F:60:5B:80:C6
ebsqa_qaNode1q, Jan 3, 2022, PrivateKeyEntry,
Certificate fingerprint (SHA1): AF:78:49:69:75:3D:D8:6E:70:F9:74:69:42:BB:A9:19:6B:03:9D:E

Extra steps (that may not be necessary):
$ cp -p adkeystore.dat adkeystore.dat_BK

$ keytool -importkeystore -srckeystore adkeystore.dat -destkeystore adkeystore.dat -deststoretype pkcs12
Enter source keystore password:
Entry for alias cn=digicert assured id root ca, ou=www.digicert.com, o=digicert inc, c=us successfully imported.
Enter key password for <ebsqa_qaNode1q>
Entry for alias ebsqa_qaNode1q successfully imported.
Entry for alias cn=digicert sha2 assured id code signing ca, ou=www.digicert.com, o=digicert inc, c=us successfully imported.
Import command completed:  3 entries successfully imported, 0 entries failed or cancelled

Warning:
<cn=digicert assured id root ca, ou=www.digicert.com, o=digicert inc, c=us> uses the SHA1withRSA signature algorithm which is considered a security risk. This algorithm will be disabled in a future update.
Migrated "adkeystore.dat" to Non JKS/JCEKS. The JKS keystore is backed up as "adkeystore.dat.old".

$ ls -altr adkeystore.dat*
-rw-r----- 1 user users 4992 Apr  2  2019 adkeystore.dat_OLD
-rw-r--r-- 1 user users 7498 Mar  1 11:39 adkeystore.dat_BK
-rw-r--r-- 1 user users 7498 Mar  1 11:41 adkeystore.dat.old
-rw-r--r-- 1 user users 8721 Mar  1 11:41 adkeystore.dat

4. Make sure file adsign.txt is correct
$ more adsign.txt
EBSQA_qanode1q 1 CUST

5. Run adadmin to generate JAR files
$ adamin
 ==> 1 ==> 4
Do you wish to force regeneration of all jar files? [No] ? Yes

Forcing generation of all product jar files.
Creating and signing every jar file can take about thirty minutes depending on the hardware being used.
... ...
Generating customall.jar ...
Not creating customall.jar as no custom java directories found under JAVA_TOP.
customall.jar generated successfully.
   ** Updating appsborg.zip ...
    Reading adjborg.txt...
adearea();
No files listed in $RUN_BASE/EBSapps/appl/admin/adjborg.txt
Ignoring ...
adjborg.txt file seems to be empty.
    Done reading adjborg.txt.
As adjborg.txt file is empty, skipping the  generation of appsborg.zip
   ** Updating appsborg2.zip ...
    Reading adjborg2.txt...
adearea();
No files listed in $RUN_BASE/EBSapps/appl/admin/adjborg2.txt
Ignoring ...
adjborg.txt file seems to be empty.
    Done reading adjborg2.txt.
As adjborg2.txt file is empty, skipping the  generation of appsborg2.zip

6. Confirm the new expiration date
jarsigner -verify -verbose -certs $AD_TOP/java/jar/adxlib.jar
... ...
- Signed by "CN=Company_name, O=Company_name, L=City, ST=State_, C=US"
    Digest algorithm: SHA-256
    Signature algorithm: SHA256withRSA, 4096-bit key

jar verified.

Warning:
This jar contains signatures that do not include a timestamp. Without a timestamp, users may not be able to validate this jar after any of the signer certificates expire (as early as 2024-04-05).

The signer certificate will expire on 2024-04-05.

7. Test the result by opening a few GUI forms. 
It is not necessary to test the whole site.

Tuesday, February 1, 2022

Re-new R12.2 ssl certificate

Certificate has to be renewed before it expires, because after it expires, "adapcctl.sh start" will fail to start Apache service in R12.2. Steps to renew it in R12.2.10:

1. Renew the cert file.
This the key step and may be outside of your control. The process of creating a certificate is always:
- Create public\private key pair.
- Create CSR.
- CA issues certificate based on CSR.

In most cases, the first two steps are done in a cert tool, such as Venafi. Then Venafi talks to DigiCert to get the signed certificate which can be download as .pfx file. After it is renamed to ewallet.p12, OWM (Oracle Wallet Manager) 10.1.0.5.0 in 10.1.3 ORACLE_HOME of R12.1 can read it and turn "Auto Login" on.

But OWM 11.1.1.9.0 at $FMW_HOME/webtier/bin/ in R12.2.10 can not read .pfx file correctly, simply showing "Certificate[Empty]". We have to use orapki under $FMW_HOME/oracle_common/bin to turn "Auto Login" on as a workaround.

2. Copy the new cert file to $NE_BASE/inst/$CONTEXT_NAME/certs/Apache as ewallet.p12
This Apache folder can serve as a staging location. Depending on instance setup, it may not be used in R12.2 at all.

3. Verify the cert can be open and its password is good (optional):
$ alias orapki=$FMW_HOME/oracle_common/bin/orapki
$ which orapki                     <= do not use the one in 10.1.2 ORACLE_HOME

$ orapki wallet display -wallet $NE_BASE/inst/$CONTEXT_NAME/certs/Apache/ewallet.p12
Enter wallet password:   xxxxx
Requested Certificates:
User Certificates:
Subject:        CN=site_name.domian.com,O=company_name,L=city,ST=AZ,C=US
Trusted Certificates:
Subject:        CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US
Subject:        CN=company_name Secure CA2,O=company_name,C=US

Notes:  Without specifying the path to file ewallet.p12, orapki in $FMW_HOME (vs. in 10.1.2 ORACLE_HOME) will read a default cert file in somewhere and may give incorrect password error.

4. Run below line to enable Auto Login

$ orapki wallet create -wallet $NE_BASE/inst/$CONTEXT_NAME/certs/Apache/ewallet.p12 -pwd 'sslPWD' -auto_login
Oracle PKI Tool : Version 11.1.1.9.0
Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.

"orapki wallet create ..." will not destroy or modify the original wallet! It creates cwallet.sso in the current directory. You can now view its contents without a password:
$ orapki wallet display -wallet $NE_BASE/inst/$CONTEXT_NAME/certs/Apache/cwallet.sso
Oracle PKI Tool : Version 11.1.1.9.0
Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
User Certificates:
Subject:        CN= ... ...
... ...

5. Stop apps services, and copy cwallet.sso to other locations:

$ adstpall.sh apps/appsPWD

$ iName=$(tr < $CONTEXT_FILE '<>' '  ' | awk '/"s_ohs_instance"/ {print $(NF-1)}' )
$ echo $iName
EBS_web_EBSDEV_OHS1

$ cd $FMW_HOME/webtier/instances/$iName

$ find . -name cwallet.sso | fgrep -v /webgate/
./config/OPMN/opmn/wallet_ORIG/cwallet.sso
./config/OPMN/opmn/wallet/cwallet.sso
./config/OHS/EBS_web_EBSDEV/keystores/default/cwallet.sso
./config/OHS/EBS_web_EBSDEV/proxy-wallet/cwallet.sso

Use one line to see their contents:
$ find . -name cwallet.sso | fgrep -v /webgate/ | while read w ; do echo -e "\n$w"; orapki wallet display -nologo -complete -wallet $w ; done

$ ls -al config/OPMN/opmn/wallet/cwallet.sso
-rw------- 1 user group 6369 Feb 13  2020 config/OPMN/opmn/wallet/cwallet.sso
$ ls -al config/OHS/EBS_web_EBSDEV/keystores/default/cwallet.sso
-rw------- 1 user group 6369 Feb 13  2020 config/OHS/EBS_web_EBSDEV/keystores/default/cwallet.sso
$ ls -al config/OHS/EBS_web_EBSDEV/proxy-wallet/cwallet.sso
-rw------- 1 user group 4373 Jun 29  2021 config/OHS/EBS_web_EBSDEV/proxy-wallet/cwallet.sso
                         <== a default file from upgrade installation? Seems it is not used.

Now, back up old .sso files and use new file $NE_BASE/inst/$CONTEXT_NAME/certs/Apache/cwallet.sso to replace them.

6. Copy ewallet.p12 and cwallet.sso to all other nodes.
Note: New cwallet.sso has to be copied to all folders on concurrent node as well, only IF ssl was enabled on that node. Otherwise, "adop phase=fs_clone" may fail with message "Unable to initialize SSL environment, nzos call nzosSetCredential returned 28791" and strange error on the node:
LOG FILE: $INST_TOP/logs/appl/rgf/TXK/txkSetAppsConf_10141635.log ***
ERROR: The value <$FMW_HOME/webtier/instances/EBS_web_OHS3> for s_ohs_instance_loc in $CONTEXT_FILE isn't a directory.

7. Start/stop Apache as a quick test
$ adapcctl.sh start             <== Do NOT use "adopmnctl.sh startall"
$ adopmnctl.sh status
$ adopmnctl.sh stopall

8. Run autoconfig 
9. Star apps services

- TROUBLESHOOTING
$ SUBiName=${iName%?????}
$ echo $SUBiName

Its OHS logs from adapcctl.sh are at $FMW_HOME/webtier/instances/$iName/diagnostics/logs/OHS/$SUBiName
And opmn.log is at $FMW_HOME/webtier/instances/$iName/diagnostics/logs/OPMN/opmn

a cwallet.sso file also exists in two folders under $EBS_DOMAIN_HOME/opmn/EBS*/wallet. I am not sure what it is for. But you can use orapki to view them, such as 
$ $ $FMW_HOME/oracle_common/bin/orapki wallet display -wallet $EBS_DOMAIN_HOME/opmn/EBS_web_OHS1/wallet/cwallet.sso

$ $FMW_HOME/oracle_common/bin/orapki wallet display -wallet $EBS_DOMAIN_HOME/opmn/EBS_web_OHS2/wallet/cwallet.sso

- A way to get server.cer file, read the contents of cert file, and find the expiration date of a cert file (Doc ID 2674842.1):
a) Verify the password works
$ $FMW_HOME/oracle_common/bin/orapki wallet display -wallet /path/to/ewallet.p12
Oracle PKI Tool : Version 11.1.1.9.0
Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.

Enter wallet password:
Requested Certificates:
User Certificates:
Subject:        CN=site_name.domian.com,O=company_name,L=city,ST=AZ,C=US
Trusted Certificates:
Subject:        CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US
Subject:        CN=company_name. Secure CA2,O=company_name,C=US

b) If troubleshooting is needed, export the certificate present in the wallet to a file
Copy the "Subject" information under the "User Certificates" section above and run command to generate server.cer file:

$ orapki wallet export -wallet /path/to/ewallet.p12 -dn 'CN=site_name.domian.com,O=company_name,L=city,ST=AZ,C=US' -cert server.crt
                                  <== it generates .crt file or .cer file, or both are the same>
$ more server.crt

c) Use the keytool to check the signature algorithm and expiry details
$ which keytool
$COMMON_TOP/util/jdk32/jre/bin/keytool      (It is a Java tool)
$ keytool -printcert -file server.crt
Owner: CN=site_name.domian.com,O=company_name,L=city,ST=AZ,C=US
Issuer: CN=company_name Secure CA2,O=company_name,C=US
Serial number: bbef1120a01b29a5c3c213623xxxxxxxx
Valid from: Thu Jan 12 20:00:00 EDT 2021 until: Tue Jan 16 19:59:59 EDT 2022
... ...
d) To read the full contents of server.cer, run openssl:
$ which openssl
/usr/bin/openssl
$ openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017
$ openssl x509 -in server.crt -noout -text >> server.txt
$ more server.txt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            05:36:63:2d:8f:8d:4b:1f:64:45:84:5a:ce:49:1a:8e
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: ... ...
... ...
- Workarounds
Since OWM 11.1.1.9 in R12.2 can not be used to read cert file, one option is to create a key pair and CSR in OWM using old cert file. After the CSR is used within Venafi to enroll for the certificate, manually install the certificate back in OWM. So, we have to create the request within OWM in multiple steps. In this case, the password for old certificate must be good. 

The another option is to use OWM in 10.1.3 ORACLE_HOME on R12.1 file system to open and save the new cert file (if old R12.1 file system is still available).

References (for creating/renewing cert file. I did not use all of them):

- Oracle Wallet Manager Shows "Certificate:Empty" When Generating Wallets Using An Existing Certificate & Key (Doc ID 1582186.1)
- How to Check Validity of Server Certificate from ORAPKI Wallet in Command Line (Doc ID 2674842.1)
- How to Replace an Expired or Expiring Certificate in FMW 11g Wallets? (Doc ID 1371209.1)
- 12.2 E-Business Suite Technology Stack Upgrade Steps To Reuse/ Migrate 11i Certificates To 12.2 Wallet Format To Replace Cwallet.sso (Doc ID 2315134.1)