When ssl certificate expired (or something is wrong in the certificate .sso file), Apache will not start by adapcctl.sh in R12.2. Possible error message in adapcctl.txt (or adopmnctl.txt) under $LOG_HOME/appl/admin/log:
[opmn] [ERROR:1] [] [internal] $FMW_HOME/webtier/opmn/bin/opmn: unexpected exit: status 4200
opmnctl start: opmn failed to start.
ias-component/process-type/process-set: EBS_web/OHS/OHS/
Error
--> Process (index=1,uid=1246640827,pid=29336)
failed to start a managed process after the maximum retry limit
Log: $FMW_HOME/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/console~OHS~1.log
... ... :: adapcctl.sh: exiting with status 204
$FMW_HOME/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/EBS_web.log may give errors from starting Apache:
[OHS] [ERROR:32] [] [core.c] [host_id: node_name.domain.com] [host_addr: 167.69.xx.xx] [pid: 1851] [tid: 139696124196736] [user: applmgr] [VirtualHost: site_name.domain.com:0] Init: (site_name.domain.com:443) Unable to initialize SSL environment, nzos call nzosSetCredential returned 28791
[OHS] [ERROR:32] [] [core.c] [host_id: node_name.domain.com] [host_addr: 167.69.xx.xx] [pid: 1851] [tid: 139696124196736] [user: applmgr] [VirtualHost: site_name.domain.com:0] NZ Library Error: Unknown error
"adopmnctl.sh start" can be used to get more error message in $FMW_HOME/webtier/instances/EBS_web_OHS1/diagnostics/logs/OPMN/opmn/opmn.log, such as errors (that matches Doc ID 2676628.1):
[opmn] [ERROR:1] [] [ons-secure] Connection server SSL set credentials failed (28791)
[opmn] [ERROR:1] [222] [ons-secure] SSL initialization failed
Note: If cert file .../config/OPMN/opmn/wallet/cwallet.sso is a wrong file, "adapcctl.sh start" will fail quickly and may give misleading error in adapcctl.txt:
[opmn] [ERROR:1] [] [internal] $FMW_HOME/webtier/opmn/bin/opmn: unexpected exit: status 4200
opmnctl start: opmn failed to start.
So its first webpage and so the entire EBS site is not accessible. The error messages in the log do not point out the real problem. Most likely (but not 100% true), the cert expired.
I tried to renew the cert file. But if the new one did not make Apache start, it is impossible to tell whether CA software webpage (such as Venafi) gave me a valid cert file. At that situation, there is no good way to test the cert renewal. It became an urgent problem.
The solution is to create a temporary cert to bring the site up. Doc ID 2555355.1 (Prerequisite Steps to Configure Oracle Fusion Middleware 11.1.1.9 Components for Oracle E-Business Suite Release 12.2 Before Applying the July 2019 and Later FMW OSS Security Patch) gives steps for creating a temporary cert file. I had to keep Admin Server ("adadminsrvctl.sh start") up during this process in R12.2.10.
First of all, make sure to use the right orapki:
$ alias orapki=$FMW_HOME/oracle_common/bin/orapki
$ cd /u01/app/temp
$ mkdir ss
$ cd ss
Create a new wallet with an acceptable self-signed certificate in /u01/app/temp/ss:
$ orapki wallet create -wallet ./ -auto_login_only
Oracle PKI Tool : Version 11.1.1.9.0
Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.
$ orapki wallet add -wallet . -dn "CN=FMWSmallCircleOfTrust" -asym_alg RSA -keysize 2048 -sign_alg sha256 -self_signed -validity 3652 -auto_login_only
Oracle PKI Tool : Version 11.1.1.9.0
Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.
$ orapki wallet display -wallet . <== to verify/see the new wallet
Oracle PKI Tool : Version 11.1.1.9.0
Copyright (c) 2004, 2015, Oracle and/or its affiliates. All rights reserved.
Requested Certificates:
User Certificates:
Subject: CN=FMWSmallCircleOfTrust
Trusted Certificates:
Subject: CN=FMWSmallCircleOfTrust
Define useful OS variable $iName for next steps:
$ tr < $CONTEXT_FILE '<>' ' ' | awk '/"s_ohs_instance"/ {print $(NF-1)}'
EBS_web_OHS1
$ iName=$(tr < $CONTEXT_FILE '<>' ' ' | awk '/"s_ohs_instance"/ {print $(NF-1)}' )
$ pwd
/u01/app/temp/ss
$ cd $FMW_HOME/webtier/instances/$iName
$ pwd
$FMW_HOME/webtier/instances/EBS_web_OHS1
$ find . -name cwallet.sso <== to find cwallet.sso is used in 3 locations
./config/OPMN/opmn/wallet_ORIG/cwallet.sso
./config/OPMN/opmn/wallet/cwallet.sso
./config/OHS/EBS_web/keystores/default/cwallet.sso
./config/OHS/EBS_web/proxy-wallet/cwallet.sso
Back up the existing cwallet.sso and replace it by the temporary cert file in 3 locations.
$ find . -name cwallet.sso | fgrep -v /webgate/ | while read w ; do echo $w; cp -p /u01/app/temp/ss/cwallet.sso $w ; done
./config/OPMN/opmn/wallet/cwallet.sso
./config/OHS/EBS_web/keystores/default/cwallet.sso
./config/OHS/EBS_web/proxy-wallet/cwallet.sso
$ ls -al ./config/OPMN/opmn/wallet
total 12
-rw------- 1 user group 3853 Oct 1 16:44 cwallet.sso
-rw------- 1 user group 4365 May 28 15:53 cwallet.sso_BK_1001
-rw------- 1 user group 0 May 28 15:53 cwallet.sso.lck
$ ls -al
drwx------ 3 user group 17 May 28 15:54 auditlogs
drwx------ 2 user group 21 May 28 15:53 bin
drwx------ 4 user group 29 May 28 15:53 config
drwx------ 3 user group 18 May 28 15:53 diagnostics
drwx------ 3 user group 21 May 28 15:53 OHS
drwx------ 3 user group 23 Jun 14 03:24 tmp
$ find . -name cwallet.sso | fgrep -v /webgate/ | while read w ; do echo -e "\n$w"; orapki wallet display -nologo -wallet $w ; done
./config/OPMN/opmn/wallet_ORIG/cwallet.sso
Requested Certificates:
User Certificates:
Subject: CN=Self-Signed Certificate for EBS_web_OHS1\20,OU=OAS,O=ORACLE,L=REDWOODSHORES,ST=CA,C=US
Trusted Certificates:
Subject: CN=Self-Signed Certificate for EBS_web_OHS1\20,OU=OAS,O=ORACLE,L=REDWOODSHORES,ST=CA,C=US
./config/OPMN/opmn/wallet/cwallet.sso
Requested Certificates:
User Certificates:
Subject: CN=FMWSmallCircleOfTrust
Trusted Certificates:
Subject: CN=FMWSmallCircleOfTrust
./config/OHS/EBS_web/keystores/default/cwallet.sso
Requested Certificates:
User Certificates:
Subject: CN=FMWSmallCircleOfTrust
Trusted Certificates:
Subject: CN=FMWSmallCircleOfTrust
./config/OHS/EBS_web/proxy-wallet/cwallet.sso
Requested Certificates:
User Certificates:
Subject: CN=FMWSmallCircleOfTrust
Trusted Certificates:
Subject: CN=FMWSmallCircleOfTrust
Re-register OHS and its new certificate with Fusion Middleware Control. Seems to me this step is necessary (while I do not know what it really does).
$ aHost=$( tr < $CONTEXT_FILE '<>' ' ' | awk '/"s_wls_admin_host"/ {print $(NF-1)}' )
$ aPort=$( tr < $CONTEXT_FILE '<>' ' ' | awk '/"s_wls_adminport"/ {print $(NF-1)}' )
$ aUser=$( tr < $CONTEXT_FILE '<>' ' ' | awk '/"s_wls_admin_user"/ {print $(NF-1)}' )
$ echo $aHost
node_name
$ echo $aPort
7032
$ echo $aUser
weblogic
$ cd $FMW_HOME/webtier/instances/$iName/bin
$ ./opmnctl unregisterinstance -adminHost $aHost -adminPort $aPort -adminUsername $aUser -instanceName $iName
Command requires login to weblogic admin server (node_name):
Username: weblogic
Password:
Unregistering instance
Command succeeded.
$ ./opmnctl registerinstance -adminHost $aHost -adminPort $aPort -adminUsername $aUser
Command requires login to weblogic admin server (node_name ):
Username: weblogic
Password:
Registering instance
Command succeeded.
I logged onto EM site at http://node_name.domain.com:7032/em (vs. Console) and saw OHS was still down (Somehow, my EM always shows Web Tier sites are down), and adopmnctl.sh reported OHS in Down status.
Now, when I ran adapcctl.sh, it started Apache successfully and the webpage worked in "unsafe" mode! Then, I shutdown everything and ran an autoConfig before started all EBS services.
$ cd $ADMIN_SCRIPTS_HOME
$ ./adapcctl.sh start
$ ./adopmnctl.sh status
You are running adopmnctl.sh version 120.0.12020000.2
Checking status of OPMN managed processes...
Processes in Instance: EBS_web_OHS1
----------------------+--------------------+---------+---------
ias-component | process-type | pid | status
----------------------+--------------------+---------+---------
EBS_web | OHS | 1166 | Alive
With that, I had the R12.2.10 site available to test the certificate renewal and got the expired cert renewed after replacing the temporary cert.
TROUBLESHOOTING
If Apache does still not start, check console~OHS~1.log under $FMW_HOME/webtier/instances/$iName/diagnostics/logs/OHS/EBS_xxx
If Apache started in Alive status, but the login webpage is still not available, below line shall return "connected"
$ wget http://node_name.domain.com:s_webport
If the login page shows ERR_SSL_PROTOCOL_ERROR, most likely some parameter in .xml file(s) for enabling TLS 1.2 is wrong.
If the login page shows ERR_CONNECTION_RESET, one of the possibilities is F5 listens to a wrong port if F5 is used in company network. It shall listen to s_webssl_port. Also check Oracle Doc ID 2771703.1 for other possible causes.
Notes: ADOP will not automatically copy cert file cwallet.sso from RUN file system to PATCH file system. You have to modify adop_sync.drv located under $APPL_TOP_NE/ad/custom to include the followings:
#Oracle HTTP Server Wallet - cwallet.sso
rsync -zr %s_current_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/keystores/default/cwallet.sso %s_other_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/keystores/default/cwallet.sso
#OPMN Wallet - cwallet.sso
rsync -zr %s_current_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OPMN/opmn/wallet/cwallet.sso %s_other_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OPMN/opmn/wallet/cwallet.sso
rsync -zr %s_current_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/proxy-wallet/cwallet.sso %s_other_base%/FMW_Home/webtier/instances/%s_ohs_instance%/config/OHS/%s_ohs_component%/proxy-wallet/cwallet.sso
