Tuesday, February 1, 2022

Re-new R12.2 ssl certificate

Certificate has to be renewed before it expires, because after it expires, "adapcctl.sh start" will fail to start Apache service in R12.2. Steps to renew it in R12.2.10:

1. Renew the cert file.
This the key step and may be outside of your control. The process of creating a certificate is always:
- Create public\private key pair.
- Create CSR.
- CA issues certificate based on CSR.

In most cases, the first two steps are done in a cert tool, such as Venafi. Then Venafi talks to DigiCert to get the signed certificate which can be download as .pfx file. After it is renamed to ewallet.p12, OWM (Oracle Wallet Manager) 10.1.0.5.0 in 10.1.3 ORACLE_HOME of R12.1 can read it and turn "Auto Login" on.

But OWM 11.1.1.9.0 at $FMW_HOME/webtier/bin/ in R12.2.10 can not read .pfx file correctly, simply showing "Certificate[Empty]". We have to use orapki under $FMW_HOME/oracle_common/bin to turn "Auto Login" on as a workaround.

2. Copy the new cert file to $NE_BASE/inst/$CONTEXT_NAME/certs/Apache as ewallet.p12
This Apache folder can serve as a staging location. Depending on instance setup, it may not be used in R12.2 at all.

3. Verify the cert can be open and its password is good (optional):
$ alias orapki=$FMW_HOME/oracle_common/bin/orapki
$ which orapki                     <= do not use the one in 10.1.2 ORACLE_HOME

$ orapki wallet display -wallet $NE_BASE/inst/$CONTEXT_NAME/certs/Apache/ewallet.p12
Enter wallet password:   xxxxx
Requested Certificates:
User Certificates:
Subject:        CN=site_name.domian.com,O=company_name,L=city,ST=AZ,C=US
Trusted Certificates:
Subject:        CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US
Subject:        CN=company_name Secure CA2,O=company_name,C=US

Notes:  Without specifying the path to file ewallet.p12, orapki in $FMW_HOME (vs. in 10.1.2 ORACLE_HOME) will read a default cert file in somewhere and may give incorrect password error.

4. Run below line to enable Auto Login

$ orapki wallet create -wallet $NE_BASE/inst/$CONTEXT_NAME/certs/Apache/ewallet.p12 -pwd 'sslPWD' -auto_login
Oracle PKI Tool : Version 11.1.1.9.0
Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.

"orapki wallet create ..." will not destroy or modify the original wallet! It creates cwallet.sso in the current directory. You can now view its contents without a password:
$ orapki wallet display -wallet $NE_BASE/inst/$CONTEXT_NAME/certs/Apache/cwallet.sso
Oracle PKI Tool : Version 11.1.1.9.0
Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
User Certificates:
Subject:        CN= ... ...
... ...

5. Stop apps services, and copy cwallet.sso to other locations:

$ adstpall.sh apps/appsPWD

$ iName=$(tr < $CONTEXT_FILE '<>' '  ' | awk '/"s_ohs_instance"/ {print $(NF-1)}' )
$ echo $iName
EBS_web_EBSDEV_OHS1

$ cd $FMW_HOME/webtier/instances/$iName

$ find . -name cwallet.sso | fgrep -v /webgate/
./config/OPMN/opmn/wallet_ORIG/cwallet.sso
./config/OPMN/opmn/wallet/cwallet.sso
./config/OHS/EBS_web_EBSDEV/keystores/default/cwallet.sso
./config/OHS/EBS_web_EBSDEV/proxy-wallet/cwallet.sso

Use one line to see their contents:
$ find . -name cwallet.sso | fgrep -v /webgate/ | while read w ; do echo -e "\n$w"; orapki wallet display -nologo -complete -wallet $w ; done

$ ls -al config/OPMN/opmn/wallet/cwallet.sso
-rw------- 1 user group 6369 Feb 13  2020 config/OPMN/opmn/wallet/cwallet.sso
$ ls -al config/OHS/EBS_web_EBSDEV/keystores/default/cwallet.sso
-rw------- 1 user group 6369 Feb 13  2020 config/OHS/EBS_web_EBSDEV/keystores/default/cwallet.sso
$ ls -al config/OHS/EBS_web_EBSDEV/proxy-wallet/cwallet.sso
-rw------- 1 user group 4373 Jun 29  2021 config/OHS/EBS_web_EBSDEV/proxy-wallet/cwallet.sso
                         <== a default file from upgrade installation? Seems it is not used.

Now, back up old .sso files and use new file $NE_BASE/inst/$CONTEXT_NAME/certs/Apache/cwallet.sso to replace them.

6. Copy ewallet.p12 and cwallet.sso to all other nodes.
Note: New cwallet.sso has to be copied to all folders on concurrent node as well, only IF ssl was enabled on that node. Otherwise, "adop phase=fs_clone" may fail with message "Unable to initialize SSL environment, nzos call nzosSetCredential returned 28791" and strange error on the node:
LOG FILE: $INST_TOP/logs/appl/rgf/TXK/txkSetAppsConf_10141635.log ***
ERROR: The value <$FMW_HOME/webtier/instances/EBS_web_OHS3> for s_ohs_instance_loc in $CONTEXT_FILE isn't a directory.

7. Start/stop Apache as a quick test
$ adapcctl.sh start             <== Do NOT use "adopmnctl.sh startall"
$ adopmnctl.sh status
$ adopmnctl.sh stopall

8. Run autoconfig 
9. Star apps services

- TROUBLESHOOTING
$ SUBiName=${iName%?????}
$ echo $SUBiName

Its OHS logs from adapcctl.sh are at $FMW_HOME/webtier/instances/$iName/diagnostics/logs/OHS/$SUBiName
And opmn.log is at $FMW_HOME/webtier/instances/$iName/diagnostics/logs/OPMN/opmn

a cwallet.sso file also exists in two folders under $EBS_DOMAIN_HOME/opmn/EBS*/wallet. I am not sure what it is for. But you can use orapki to view them, such as 
$ $ $FMW_HOME/oracle_common/bin/orapki wallet display -wallet $EBS_DOMAIN_HOME/opmn/EBS_web_OHS1/wallet/cwallet.sso

$ $FMW_HOME/oracle_common/bin/orapki wallet display -wallet $EBS_DOMAIN_HOME/opmn/EBS_web_OHS2/wallet/cwallet.sso

- A way to get server.cer file, read the contents of cert file, and find the expiration date of a cert file (Doc ID 2674842.1):
a) Verify the password works
$ $FMW_HOME/oracle_common/bin/orapki wallet display -wallet /path/to/ewallet.p12
Oracle PKI Tool : Version 11.1.1.9.0
Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.

Enter wallet password:
Requested Certificates:
User Certificates:
Subject:        CN=site_name.domian.com,O=company_name,L=city,ST=AZ,C=US
Trusted Certificates:
Subject:        CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US
Subject:        CN=company_name. Secure CA2,O=company_name,C=US

b) If troubleshooting is needed, export the certificate present in the wallet to a file
Copy the "Subject" information under the "User Certificates" section above and run command to generate server.cer file:

$ orapki wallet export -wallet /path/to/ewallet.p12 -dn 'CN=site_name.domian.com,O=company_name,L=city,ST=AZ,C=US' -cert server.crt
                                  <== it generates .crt file or .cer file, or both are the same>
$ more server.crt

c) Use the keytool to check the signature algorithm and expiry details
$ which keytool
$COMMON_TOP/util/jdk32/jre/bin/keytool      (It is a Java tool)
$ keytool -printcert -file server.crt
Owner: CN=site_name.domian.com,O=company_name,L=city,ST=AZ,C=US
Issuer: CN=company_name Secure CA2,O=company_name,C=US
Serial number: bbef1120a01b29a5c3c213623xxxxxxxx
Valid from: Thu Jan 12 20:00:00 EDT 2021 until: Tue Jan 16 19:59:59 EDT 2022
... ...
d) To read the full contents of server.cer, run openssl:
$ which openssl
/usr/bin/openssl
$ openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017
$ openssl x509 -in server.crt -noout -text >> server.txt
$ more server.txt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            05:36:63:2d:8f:8d:4b:1f:64:45:84:5a:ce:49:1a:8e
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: ... ...
... ...
- Workarounds
Since OWM 11.1.1.9 in R12.2 can not be used to read cert file, one option is to create a key pair and CSR in OWM using old cert file. After the CSR is used within Venafi to enroll for the certificate, manually install the certificate back in OWM. So, we have to create the request within OWM in multiple steps. In this case, the password for old certificate must be good. 

The another option is to use OWM in 10.1.3 ORACLE_HOME on R12.1 file system to open and save the new cert file (if old R12.1 file system is still available).

References (for creating/renewing cert file. I did not use all of them):

- Oracle Wallet Manager Shows "Certificate:Empty" When Generating Wallets Using An Existing Certificate & Key (Doc ID 1582186.1)
- How to Check Validity of Server Certificate from ORAPKI Wallet in Command Line (Doc ID 2674842.1)
- How to Replace an Expired or Expiring Certificate in FMW 11g Wallets? (Doc ID 1371209.1)
- 12.2 E-Business Suite Technology Stack Upgrade Steps To Reuse/ Migrate 11i Certificates To 12.2 Wallet Format To Replace Cwallet.sso (Doc ID 2315134.1)

No comments: