ERP on DB: Enable TLS 1.2 in R12.2 right after it is upgraded from R12.1.3

This post assume all cert files are good and TLS 1.2 was enabled in R12.1.3. After it was upgraded to R12.2, SSL and TLS1.2 were not enabled automatically. It is optional, but is normally required by company's Security to enable it. Here are steps to enable them only for inbound connections as described in Doc ID 1367293.1 (Enabling TLS in Oracle E-Business Suite Release 12.2). The use of TLS 1.2 requires Java 7 at a minimum and Oracle Fusion Middleware (FMW) 1.1.1.9.

Since all wallet files and cert files exist in old $INST_TOP/certs/Apache folder, that saves a lot of time from generating them. I also verified all necessary patches exist in my new R12.2.10 instances. Here is what I did.

1. Copy Apache folder from R12.1.3 $INST_TOP/certs/Apache to R12.2.10 $NE_BASE/inst/$CONTEXT_NAME/certs/Apache (the same as <s_web_ssl_directory>/Apache).

$ cd $NE_BASE/inst/$CONTEXT_NAME/certs

$ ls

$ cp -rp $HOME/$TWO_TASK/inst_R12.1/apps/$CONTEXT_NAME/certs/Apache .

(assume R12.1 folder inst was renamed to inst_R12.1 right after R12.2 upgrade completed. Possible files in R12.1.3 includes:)

$ ls -al Apache

-rw-r--r-- 1 user group 1367 Mar 17 2020 ca.crt

-rw------- 1 user group 6377 Jul 17 2020 cwallet.sso

-rw-r--r-- 1 user group 6349 Jul 17 2020 ewallet.p12

-rw-r--r-- 1 user group 1684 Mar 17 2020 intermediate.crt

-rw-r--r-- 1 user group 5389 Jul 17 2020 opmn.crt

-rw-r--r-- 1 user group 2338 Jul 17 2020 server.crt

-rw-r--r-- 1 user group 1675 Jul 17 2020 server.key

Notes: Section 5.2 Step 2 says "The s_web_ssl_directory location is still used by some Oracle E-Business Suite Release 12.2 components (for example, XML Gateway Transportation Agent OXTA) and during the Oracle Fusion Middleware cloning process". Step 6 says Apache folder "is not used by the Oracle HTTP Server" (Apache).

In my instances, cert files worked in R12.1 are still worked in R12.2. So, I did not need to do any additional steps but use orapki in R12.2 to verify its password.

$ which orapki <= do not use the one in 10.1.2 ORACLE_HOME

$FMW_HOME/oracle_common/bin/orapki

$ orapki wallet display -wallet $NE_BASE/inst/$CONTEXT_NAME/certs/ewallet.p12

If it expired, it has to be renewed first because Apache and opmn will not start by error: [opmn] [ERROR:1] [] [internal] $FMW_HOME/webtier/opmn/bin/opmn: unexpected exit: status 4200

2. Copy cwallet.sso to 2 locations: HTTP Server & OPMN

$ grep s_ohs_instance_loc $CONTEXT_FILE

Define Linux variables for scripting:

$ iName=$(tr < $CONTEXT_FILE '<>' ' ' | awk '/"s_ohs_instance"/ {print $(NF-1)}' )

$ echo $iName <== this tells EBS_web_OHS1 or EBS_web_OHS2 is used, e.g.

$ SUBiName=${iName%?????}

$ echo $SUBiName

EBS_web

Modify the Oracle HTTP Server wallet:

$ cd $FMW_HOME/webtier/instances/$iName/config/OHS/$SUBiName/keystores/default

$ mv cwallet.sso cwallet.sso_ORIG

$ cp -p $NE_BASE/inst/$CONTEXT_NAME/certs/Apache/cwallet.sso .

$ ls

Modify the OPMN wallet:

$ cd $FMW_HOME/webtier/instances/$iName/config/OPMN/opmn

$ mv wallet wallet_ORIG

$ mkdir wallet

$ cd wallet

$ cp -p $NE_BASE/inst/$CONTEXT_NAME/certs/Apache/cwallet.sso .

$ ls

3. Modify 3 files: opmn.xml, and admin.conf & ssl.conf in folder <s_ohs_instance_loc>/config/OHS/<s_ohs_component> (see Section 6.1.2 Configure Inbound Connections)

NOTES:

1) If the environment is cloned from another instance, those configuration files will be copied from the Source instance and so their contents may not be the same as in below.

2) Unlike R12.1, WebLogic configuration files in R12.2 can be edited and AutoConfig may not update them. For more details on modifying OHS port values, see Doc ID 1905593.1 (Managing Configuration of Oracle HTTP Server and Web Application Services in Oracle E-Business Suite Release 12.2).

Configure the OPMN Cipher Suites:

$ cd ..

(or, $ cd $FMW_HOME/webtier/instances/$iName/config/OPMN/opmn )

$ cp -p opmn.xml opmn.xml_ORIG

$ vi opmn.xml

$ diff opmn.xml opmn.xml_ORIG

< <ssl enabled="true" wallet-file="....../config/OPMN/opmn/wallet" ssl-versions="TLSv1.2" ssl-ciphers="SSL_RSA_WITH_AES_256_CBC_SHA,SSL_RSA_WITH_AES_128_CBC_SHA"/>

---

> <ssl enabled="true" wallet-file="....../config/OPMN/opmn/wallet" ssl-versions="TLSv1.0" ssl-ciphers="SSL_RSA_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA"/>

110c110

< </opmn>

---

> </opmn>

\ No newline at end of file

$ cd ../../OHS/$SUBiName

(cd $FMW_HOME/webtier/instances/$iName/config/OHS/$SUBiName )

$ cp -p admin.conf admin.conf_ORIG

$ vi admin.conf

$ diff admin.conf admin.conf_ORIG

< SSLCipherSuite SSL_RSA_WITH_AES_256_CBC_SHA:SSL_RSA_WITH_AES_128_CBC_SHA

< SSLProtocol nzos_Version_1_2

---

> SSLCipherSuite SSL_RSA_WITH_AES_128_CBC_SHA:SSL_RSA_WITH_3DES_EDE_CBC_SHA

> SSLProtocol nzos_Version_1_0

$ cp -p ssl.conf ssl.conf_ORIG

$ vi ssl.conf

$ diff ssl.conf ssl.conf_ORIG

< SSLProtocol TLSv1.2

< SSLCipherSuite HIGH:MEDIUM:!aNULL:!RC4:!3DES:!SEED:!IDEA:!CAMELLIA:+HIGH:+MEDIUM

---

> SSLProtocol all +TLSv1 +SSLv3 <= the file may have slightly different entries

> SSLCipherSuite HIGH:MEDIUM:!aNULL:+SHA1:+MD5:+HIGH:+MEDIUM

Notes: Section 6.5.2 indicates that edit ssl.conf is acceptable.

Also, if entry in ssl.conf is wrong, adapcctl.sh will fail to start Oracle HTTP Server (OHS). Its logs are at $FMW_HOME/webtier/instances/$iName/diagnostics/logs/OHS/$SUBiName .

And opmn.log is at $FMW_HOME/webtier/instances/$iName/diagnostics/logs/OPMN/opmn

If want to make WLS console use SSL, more changes may be needed.

4. Confirm "Listen <port_number>" in ssl.conf and <s_webssl_port> in $CONTEXT_FILE are the same. In this post, it is 4480.

$ grep Listen ssl.conf

$ grep s_webssl_port $CONTEXT_FILE

5. Keep WLS Admin Server running on Primary node (to avoid java.net.ConnectException error), and run below line (on each application tier nodes) to propagate the changes made to the context file variables. (Optional?)

$ perl $AD_TOP/bin/adSyncContext.pl contextfile=$CONTEXT_FILE

Enter the APPS user password:

Enter the WebLogic AdminServer password:

Log file $INST_TOP/logs/appl/rgf/Wed_XXX_14_17_47_45_2021/adSyncContext.log says

... ...

Time taken for adSyncContext.pl to complete: 0 mins 24 secs

The program exited with status 0

6. Edit $CONTEXT_FILE to change value for variables as asked in Section 5.2 step 9. Here is the settings in my EBS site https://example.domain.com:

s_url_protocol => https

s_local_url_protocol => https

s_webentryurlprotocol => https

s_active_webport => 443 (4480 when no mapping)

s_https_listen_parameter => 4480

s_login_page, => https://example.domain.com/OA_HTML/AppsLogin

(copy it from R12.1.3 which worked)

s_webentryhost => example

(without it, login page will direct to a wrong URL)

s_external_url => https://example.domain.com

I do not enter port 443 to any files hold port info in OHS configuration folder when F5 is used for load balance. F5 may map port 443 to port 4480. (But in R12.1.3, line "Port 443" is in files ssl.conf and httpd.conf)

Some context variables will be changed to default by R12.2 upgrade. Update or confirm below variables in $CONTEXT_FILE before running adautocfg.sh.

sitename (a Profile reads it), s_smtphost (company-wide smtp server for mailer), s_forms_tmpdir (do not use /tmp for production instance), s_sesstimeout (a Profile reads it), s_appcpnam (to REQID.OUT - see Doc ID 134007.1 to set output file type), s_forms_catchterm to 0 (FORMS_CATCHTERM - see Doc ID 745711.1 to avoid forms LOVs consume 100% CPU. R12.2 default is 1).

Notes: in multi-node instance, enabling TLS 1.2 is not necessary (or is optional) on concurrent node. But if it is enabled, ssl cert file has to be renewed before it expires (to avoid error in starting Apache from running FS_CLONE on that node).

7. Additional notes.

cwallet.sso can be found in other 3 folders for Fusion Middleware Control Wallets. But my instance does not use them (TLS 1.0 may use them):

$ iDomain=$(tr < $CONTEXT_FILE '<>' ' ' | awk '/"s_wls_domain_name"/ {print $(NF-1)}' )

$ echo $iDomain

EBS_domain <== e.g.

$ ls -al $FMW_HOME/webtier/instances/$iName/config/OHS/$SUBiName/proxy-wallet

(Notes: during one troubleshooting, copying the right cwallet.sso file to above folder fixed the failure in starting Apache. Not sure why).

$ ls -al $FMW_HOME/user_projects/domains/$iDomain/opmn/$iName/wallet

$ ls -al $FMW_HOME/user_projects/domains/$iDomain/opmn/$iName/$SUBiName/wallet

8. Run adautocfg.sh.

Note: R12.2 autoconfig does not update files opmn.xml, admin.conf, ssl.conf in Step 3.

Verify some info:

$ cd $FMW_HOME/webtier/instances/$iName/config/OHS/$SUBiName

$ grep 4480 *.*

ssl.conf:Listen 4480
ssl.conf: <VirtualHost _default_:4480>

$ grep 443 *.*
trusted.conf:#RedirectMatch ^/OA_HTML/fwk/t/(.*) https://example.domain.com:443/OA_IMAGE/fwk/t/$1
url_fw.conf:RewriteRule ^/$ https://example.domain.com:443/OA_HTML/AppsLogin [R,L]

$ grep example *.*
trusted.conf:#RedirectMatch ^/OA_HTML/fwk/t/(.*) https://example.domain.com:443/OA_IMAGE/fwk/t/$1
url_fw.conf:RewriteRule ^/$ https://example.domain.com:443/OA_HTML/AppsLogin [R,L]

$ nslookup example.domain.com
Server: 1XX.121.153.1XX
Address: 1XX.121.153.1XX#53

example.domain.com canonical name = example.np.vip.domain.com.
Name: example.np.vip.domain.com
Address: 16X.63.75.6X

9. Run adstpall.sh/adstrtal.sh script in $ADMIN_SCRIPTS_HOME directory to stop and restart all services.

Now, my R12.2.10 site is SSL and TLS 1.2 enabled. All webpages can be accessed in https from browsers, and EBS Form can be launched (if JWS was enabled).

10. Doc ID 1367293.1 Section 5.2 Step 12 gives details on using $APPL_TOP_NE/ad/custom/adop_sync.drv to synchronize the TLS setup between the two file systems. I found that all .conf files are synchronized automatically, but .sso files are not. Adding below lines to file adop_sync.drv will synchronize them. But, if FS_CLONE is executed after this change, all settings are copied to PATCH file system and this step can be skipped. Note File b64InternetCertificate.txt is used if Oracle® Application Management Pack for Oracle E-Business Suite is used to monitor EBS instance.

#TLS SECTION - START

# Required for TLS setup migration from RUN to PATCH file-system.

# Please alter the commands in the event that rsync is not available or the platform does not support the example syntax.

#Oracle HTTP Server Wallet - cwallet.sso

rsync -zr %s_current_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/keystores/default/cwallet.sso %s_other_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/keystores/default/cwallet.sso

#OPMN Wallet - cwallet.sso

rsync -zr %s_current_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OPMN/opmn/wallet/cwallet.sso %s_other_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OPMN/opmn/wallet/cwallet.sso

#Fusion Middleware Control Wallets - cwallet.sso

rsync -zr %s_current_base%/FMW_Home/user_projects/domains/%s_wls_domain_name%/opmn/%s_ohs_instance%/%s_ohs_component%/wallet/cwallet.sso %s_other_base%/FMW_Home/user_projects/domains/%s_wls_domain_name%/opmn/%s_ohs_instance%/%s_ohs_component%/wallet/cwallet.sso

rsync -zr %s_current_base%/FMW_Home/user_projects/domains/%s_wls_domain_name%/opmn/%s_ohs_instance%/wallet/cwallet.sso %s_other_base%/FMW_Home/user_projects/domains/%s_wls_domain_name%/opmn/%s_ohs_instance%/wallet/cwallet.sso

rsync -zr %s_current_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/proxy-wallet/cwallet.sso %s_other_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/proxy-wallet/cwallet.sso

11. Above changes are on RUN file system only. Run FS_CLONE to sync up it to PATCH file system.

ERP on DB

Thursday, April 15, 2021

Enable TLS 1.2 in R12.2 right after it is upgraded from R12.1.3

No comments: