Tuesday, March 24, 2020

Enable TLS1.2 in EBS R12.1

I followed Doc ID 376700.1 (Enabling TLS in Oracle E-Business Suite Release 12.1) to encrypt all connections for Oracle E-Business Suite Release 12.1.3 in RHEL7 using Transport Layer Security (TLS) protocol 1.2.

To check which TLS 1.X is used, on Chrome Ctrl+Shift+I to bring up a DevTool. Click on Security tab to see security info.

======================
- PRE REQUISITES
======================
1) HTTP Server (OHS) 10.1.3.5.0

If needed, use Doc ID 454811.1 (Upgrading to the Latest OracleAS 10g 10.1.3.x Patch Set in Oracle E-Business Suite Release 12) to upgrade it.
$ cd $INST_TOP/ora/10.1.3
$ . XXXX_xxxx.env

2) JDK 7 is required

I followed Doc ID 1467892.1 to upgrade JDK 6 to JDK 7 before working on this. Verify:
$ echo $IAS_ORACLE_HOME
/path/to/apps/tech_st/10.1.3
$ $IAS_ORACLE_HOME/appsutil/jdk/jre/bin/java -fullversion
java full version "1.7.0_231-b08"

$ echo $ORACLE_HOME
/path/to/apps/tech_st/10.1.2
$ $ORACLE_HOME/jdk/bin/java -fullversion
java full version "1.7.0_231-b08"

3) EBS site uses HTTPS

In my EBS sites, digital certificate uses PKCS#12 (Public Key Cryptography Standard #12) encryption. We download and re-new it from a website managed by company Security team. When we download it, the website ask to enter a password, say 'sslPWD01'.
cd $INST_TOP/certs/Apache
$ ls -al
-rw-r----- 1 user ogroup 6413  Jan 15 16:05 ewallet.p12
-rw------- 1 user ogroup 6441  Jan 15 16:05 cwallet.sso


======================
- Apply patches for enabling TLS1.2
======================

1) 3 patches to 10.1.3.5 HOME

$ cd $ADMIN_SCRIPTS_HOME
$ adstpall.sh

$ cd $INST_TOP/ora/10.1.3
$ . XXXXX_xxxxxx.env
$ echo $ORACLE_HOME
$ vi /etc/oraInst.loc
$ opatch lsinventory | grep 21845942   <== yes, 2015 CPU was applied before

$ cd /ebsu01/app/patchTLS12   (where all files are saved in)

-- 29292327 (Product Patched: Oracle HTTP Server. It includes newer "openssl")
-- unzip p29292327_101350_LINUX.zip
$ cd 29292327
$ opatch apply

-- 27208670 (Product Patched: OPMN. It include newer "opmn")
Note it is necessary to install a newer SDK to 10.1.3.5 OH to avoid error (from applying patch 27208670, Doc ID 2555323.1):
OPATCH_JAVA_ERROR=CheckConflict: OPatch cannot process overlay patches because of no OUI support. Please take latest OUI 10.1 patchset from "My Oracle Support" and try again.

$ cd $ORACLE_HOME
$ mv OPatch OPatch.pre_6640838
$ cd /ebsu01/app/patchForms      (where p6640838_10106_Linux-x86-64.zip is unzipped)
$ cd cd/Disk1/install
$ export DISPLAY=66.666.666.666:0.0
$ ./runInstaller -ignoreSysPrereqs &

Steps for install (very slowly):
a) Choose next on the welcome page.
b) Choose the install type as Custom
c) Enter the 10.1.3.5 ORACLE_HOME where OUI components have to be installed.
d) On the Available product components page, select the checkbox to show all components.
e) Then, click on Expend All
f) You can de-select
       Oracle Installation Libraries 10.1.0.6.0
    Ensure below are selected.
     x Oracle Universal Installer
             x Installer SDK Component 10.1.0.6.0
     x Oracle One-Off Patch Installer
g) Then, click on Next to Install.

$ cd $ORACLE_HOME
$ mv OPatch OPatch_delete
$ mv OPatch.pre_6640838 OPatch

-- Now, apply patch 27208670
$ cd /ebsu01/app/patchTLS12
$ unzip p27208670_101350_LINUX.zip
$ cd 27208670 
$ opatch apply    <== "27208670" overlays " 21845942 "

$ cd ..
$ rm -rf 27208670

-- 22322938  Product Patched: OC4J (update Java Mail API to 1.5.4)
-- unzip p22322938_101350_Generic.zip
$ cd 22322938
$ opatch apply   => Answer N to continue (Rolling back patch 8999551)

2) 4 patches by adpatch

SQL> select * from ad_bugs where bug_number in (
'23645824',  -- 23645824:R12.TXK.B (R12.TXK.B.delta.3. for FORMSAPP.EAR DEPLOYMENT FAILS)
'22974534',  -- 22974534:R12.OWF.B (for OAF EMAIL)
'27881758',  -- 27881758:R12.OWF.B (connect to OUTLOOK.OFFICE365.COM. superseded by 28779647)
'24677849'   -- 24677849:R12.TXK.B (fix MAILER FAILS)
);

Merge those 4 patches to tls12_merge.zip

NOTE:  Start a new OS session to take 10.1.2 ORACLE_HOME to avoid adpatch error:
adogjf() Unable to copy Registry.Dat.

$ adadmin   => to enable maintenance mode
$ cd /ebsu01/app/patchTLS12
-- unzip tls12_merge.zip

$ cd tls12_merge
$ echo $ORACLE_HOME  =>  10.1.2
$ ls
$ adpatch

3) Using EBS openssl (that comes from patch 29292327)

$ openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017
$ which openssl
/usr/bin/openssl     <== from OS

$ ls -al $IAS_ORACLE_HOME/Apache/open_ssl/bin
-rw-r--r-- 1 user ogroup 616303 Mar 23 12:48 openssl
-rw-r--r-- 1 user ogroup   10835 Mar 23 12:48 openssl.cnf

$ cd $IAS_ORACLE_HOME/Apache/open_ssl/bin
$ chmod 755 openssl

$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$IAS_ORACLE_HOME/lib
$ export OPENSSL_CONF=$IAS_ORACLE_HOME/Apache/open_ssl/bin/openssl.cnf

NOTES: without them, openssl may give error/warning
openssl: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory
WARNING: can't open config file: /home/nse/workspace/openssl-linux_x86_OL5/install/openssl.cnf


$ export PATH=$IAS_ORACLE_HOME/Apache/open_ssl/bin:$PATH
$ which openssl
~/$TWO_TASK/apps/tech_st/10.1.3/Apache/open_ssl/bin/openssl
$ openssl version
OpenSSL 1.0.2q  20 Nov 2018

=========================
- Configure files to enable TLS1.2
=========================

1)  Get server key and certificates

$ cd $INST_TOP/certs/Apache

$ openssl pkcs12 -in ewallet.p12 -out jyy.key -nocerts -passout pass:'EBS1t' -password pass:'sslPWD01'
MAC verified OK
Warning unsupported bag type: secretBag

$ openssl rsa -in jyy.key -out server.key -passin pass:'EBS1t'   <= Use same PASS PHRASE
writing RSA key

$ openssl pkcs12 -in ewallet.p12 -out allcerts.crt -nokeys -password pass:sslPWD01
MAC verified OK
Warning unsupported bag type: secretBag

NOTES: It creates allcerts.crt file in which has 3 certs. Need to manually extract certs from this file:
server.crt (keyword: server name) -
subject=/C=US/ST=state/L=city/O=company name/CN=servername.domain.com
intermediate.crt (with company name) -
subject=/C=US/O=company name/CN=company name. Secure CA2
root cert (ca.crt) -
subject=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA

But, intermediate.crt and root cert (ca.crt) are universal within the company network. They can be copied from other instances. So, only server.crt is truly new.

$ cat server.crt intermediate.crt ca.crt > opmn.crt

By now, $INST_TOP/certs/Apache has below files:

ewallet.p12
cwallet.sso
ca.crt
intermediate.crt
jyy.key
server.key
server.crt
opmn.crt

Additional commands

$ openssl pkcs12 -in ewallet.p12 -info
will list 3 certs and the key

Do not know what below will do:
openssl pkcs12 -in ewallet.p12 -out A_certs.crt -nokeys -nodes -password pass:sslPWD01
openssl pkcs12 -in ewallet.p12 -out CA_cert.crt -nokeys -cacerts -chain -password pass:sslPWD01

2)  Modify or create template files in $FND_TOP/admin/template/custom. They can be shared out to other EBS instances. Below are for enabling TLS 1.2 only (and disabling others):

$ cd $FND_TOP/admin/template/custom

$ mv opmn_xml_1013.tmp opmn_xml_1013.tmp_BK   (<= in case there was one)
$ cp -p ../opmn_xml_1013.tmp .
$ vi opmn_xml_1013.tmp

Replace this line in the template:
<ssl enabled="true" wallet-file="%s_web_ssl_directory%/opmn"/>
With the following:
<ssl enabled="true" openssl-certfile="%s_web_ssl_directory%/Apache/opmn.crt" openssl-keyfile="%s_web_ssl_directory%/Apache/server.key" openssl-password="change1t" openssl-lib="%s_weboh_oh%/lib" ssl-versions="TLSv1.2" ssl-ciphers="ECDHE-RSA-AES128-GCM-SHA256,ECDHE-RSA-AES256-GCM-SHA384"/>

$ diff opmn_xml_1013.tmp opmn_xml_1013.tmp_BK
14c14
<       <ssl enabled="true" openssl-certfile="%s_web_ssl_directory%/Apache/opmn.crt" openssl-keyfile="%s_web_ssl_directory%/Apache/server.key" openssl-password="change1t" openssl-lib="%s_weboh_oh%/lib" ssl-versions="TLSv1.2" ssl-ciphers="ECDHE-RSA-AES128-GCM-SHA256,ECDHE-RSA-AES256-GCM-SHA384"/>
---
>       <ssl enabled="true" wallet-file="%s_web_ssl_directory%/opmn"/>

$ cp -p ../ssl_conf_1013.tmp .
Perform the following

Step 1 - Comment out the following line in the template:
#SSLWallet file:%s_web_ssl_directory%/Apache
Step 2 - Add the following 3 lines into the template:
SSLCertificateFile %s_web_ssl_directory%/Apache/server.crt
SSLCertificateKeyFile %s_web_ssl_directory%/Apache/server.key
SSLCertificateChainFile %s_web_ssl_directory%/Apache/intermediate.crt
Step 3 - Comment out  two lines
# SSLProtocol    -all +TLSv1 +SSLv3
# SSLCipherSuite HIGH:MEDIUM:!aNULL:+SHA1:+MD5:+HIGH:+MEDIUM
Step 4 - Add two lines to it:
SSLProtocol TLSv1.2
SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384


Follow Oracle document to make similar changes in other 4 template files:

$ diff httpd_conf_1013.tmp ../httpd_conf_1013.tmp
261,262c261
< # LoadModule ossl_module       libexec/mod_ossl.so
< LoadModule ssl_module            libexec/mod_ssl.so
---
> LoadModule ossl_module          libexec/mod_ossl.so

$ diff oc4j_properties_1013.tmp ../oc4j_properties_1013.tmp
72,73d71
< https.protocols=TLSv1.2
<

$ diff oafm_oc4j_properties_1013.tmp ../oafm_oc4j_properties_1013.tmp
68,69d67
< https.protocols=TLSv1.2
<

$ diff forms_oc4j_properties_1013.tmp ../forms_oc4j_properties_1013.tmp
72,73d71
< https.protocols=TLSv1.2
<

3)   Run autoconfig and start services

$ adadmin   => disable maintenance mode
$ cd $ADMIN_SCRIPTS_HOME
$ adautocfg.sh
$ adstrtal.sh

$ openssl s_client -connect site_name.domian.com:port -tls1_2

TROUBLESHOOTING:

1. After all above steps completed, adstrtal.sh failed to start oacore service in the first time by Timed-out error (code 204). It is fixed by cleaning session lock files in persistence directories .
2. If the SSL root certificate (ca.crt) is incorrect, EBS GUI will give warning in popup:  Java warning: The certificate is not valid and cannot be used to verify the identity of this website . You have to find the problem in ca.crt or manually import it into the Java 'Secure Site CA' certificate store (which I did not try):
Java Control Panel -> Security (tab) -> Manage Certificates (button) -> Certificate Type: Secure Site CA -> Import (button)
3. While we had EBS site still using TLS 1.0, its Forms worked with Java 1.8.0_241 but did not work with Java 1.8.0_261. The pop-up box shows Java error ExitException javax.net.ssl.SSLProtocol Exception: Received close_notify during handshake . It seems some built-in security check in JRE 1.8.0_261 blocks TLS1.0 connection or other reason. So, enable TLS1.2 in EBS is necessary to avoid issue from newer Java pushed to PC desktop.

After first instance worked with enabling TLS1.2, setup steps can be done by a shell script:

#!/bin/bash
# This script get crts and keys from ssl certificate ewallet.p12 in $INST_TOP/certs/Apache. And copy
6 template files from current location to $FND_TOP/admin/template/custom. This will reduce manual steps and human error.
#
p12PWD='N0Pe'
echo -n "p12 cert password > "
read p12PWD

if [ $p12PWD != 'N0Pe' ]; then
 echo "$p12PWD"
 CURRPWD=$PWD
 echo $CURRPWD

echo "working on template files ..."
echo "$FND_TOP/admin/template/custom"
cp -p *.tmp $FND_TOP/admin/template/custom/.
ls -altr $FND_TOP/admin/template/custom

echo "working on cert files ..."
# Assume below two certs are universal within the company. Just copy over to  use them.
cp -p ca.crt $INST_TOP/certs/Apache
cp -p intermediate.crt $INST_TOP/certs/Apache

# certs files
export PATH=$IAS_ORACLE_HOME/Apache/open_ssl/bin:$PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$IAS_ORACLE_HOME/lib
export OPENSSL_CONF=$IAS_ORACLE_HOME/Apache/open_ssl/bin/openssl.cnf

cd $IAS_ORACLE_HOME/Apache/open_ssl/bin
chmod 755 openssl
which openssl

cd $INST_TOP/certs/Apache
openssl pkcs12 -in ewallet.p12 -out jyy.key -nocerts -passout pass:'change1t' -password pass:$p12PWD
openssl rsa -in jyy.key -out server.key -passin pass:'change1t'

# below line assumes the SERVER cert is the 1st position in ewallet.p12. If it is not, need to manual
ly copy it from file allcerts.crt generated by next command line.
openssl pkcs12 -in ewallet.p12 -clcerts -nokeys -password pass:$p12PWD | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p;/-END CERTIFICATE-/q' > server.crt
#
# Use this to get more details if needed
## openssl pkcs12 -in ewallet.p12 -out allcerts.crt -nokeys -password pass:$p12PWD

#
cat server.crt intermediate.crt ca.crt > opmn.crt
#
# Not sure if the order in below output file is always correct
openssl pkcs12 -in ewallet.p12 -clcerts -nokeys -password pass:$p12PWD | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > opmn.crt_2nd
# opmn.crt and opmn.crt_2nd shall be identical except the key order in the files
# echo "cehck difference. should have nothing"
# diff opmn.crt opmn.cer_2nd
ls -altr
echo $INST_TOP/certs/Apache
cd $CURRPWD

else
  echo "password seems wrong."
  exit 1
fi


Execute the script in the directory where file ca.crt, intermediate.crt and all .tmp files are saved:
$ ./openssl_cert.sh
p12 cert password > sslpwd01

/u01/app/patchTLS12
working on template files ...
$FND_TOP/admin/template/custom
-rw-r--r-- 1 user ogroup 40516 Mar 17 18:07  httpd_conf_1013.tmp
-rw-r--r-- 1 user ogroup  8447  May 19 15:44  ssl_conf_1013.tmp
-rw-r--r-- 1 user ogroup  2707  May 20 14:27  oc4j_properties_1013.tmp
-rw-r--r-- 1 user ogroup  2820  May 20 14:28  oafm_oc4j_properties_1013.tmp
-rw-r--r-- 1 user ogroup  2528  May 20 14:28  forms_oc4j_properties_1013.tmp
-rw-r--r-- 1 user ogroup 11207 May 22 16:13  opmn_xml_1013.tmp
working on cert files ...
$IAS_ORACLE_HOME/Apache/open_ssl/bin/openssl
MAC verified OK
Warning unsupported bag type: secretBag
writing RSA key
MAC verified OK
Warning unsupported bag type: secretBag
MAC verified OK
Warning unsupported bag type: secretBag
-rw-r--r-- 1 user ogroup  1367 Mar 17 17:16 ca.crt
-rw-r--r-- 1 user ogroup  1684 Mar 17 17:20 intermediate.crt
-rw-r--r-- 1 user ogroup  6229 Sep  1  13:50 ewallet.p12
-rw------- 1 user ogroup  6257 Sep  1  13:50 cwallet.sso
-rw-r--r-- 1 user ogroup  1970 Sep  2  00:01 jyy.key
-rw-r--r-- 1 user ogroup  1675 Sep  2  00:01 server.key
-rw-r--r-- 1 user ogroup  2175 Sep  2  00:01 server.crt
-rw-r--r-- 1 user ogroup  5226 Sep  2  00:01 opmn.crt
-rw-r--r-- 1 user ogroup  5226 Sep  2  00:01 opmn.crt_2nd
$INST_TOP/certs/Apache

No comments: