Apache (OHS) in R12.2 failed to stop and refused to start

After Linux server patching and reboot, alerts were sent out that httpd process for OHS (Oracle HTTP Server) in a production instance was not running on the server. I tried to stop/start it without luck. Somehow it tied to a pid owned by root very strangely (or, it ties to a pid that does not exist). 

$ ps -ef | grep httpd         <== No httpd running

$ adapcctl.sh start

$ adopmnctl.sh status

Processes in Instance: EBS_web_EBSPROD_OHS1

------------------------ --------------+------------------+-------+-------

ias-component                     | process-type    |     pid | status

------------------------- -------------+------------------+-------+-------

EBS_web_EBSPROD         | OHS                |    919 | Stop

$ ps -ef | grep 919

root       919     2  0 09:32 ?        00:00:00 [xxxxxx]

$ iName=$(tr < $CONTEXT_FILE '<>' '  ' | awk '/"s_ohs_instance"/ {print $(NF-1)}' )

$ SUBiName=${iName%?????}

$ cd $FMW_HOME/webtier/instances/$iName/diagnostics/logs/OHS/$SUBiName

The log file shows many lines of message:

--------

22/0X/05 02:47:03 Stop process

--------

$FMW_HOME/webtier/ohs/bin/apachectl stop: httpd (no pid file) not running

--------

22/0X/05 02:48:03 Stop process

--------

$FMW_HOME/webtier/ohs/bin/apachectl hardstop: httpd (no pid file) not running

File httpd.pid shall reside in this log folder, which is defined in httpd.conf in $FMW_HOME/webtier/instances/$iName/config/OHS/$SUBiName (or, $IAS_ORACLE_HOME/instances/$iName/config/OHS/$SUBiName) in R12.2. I believe the problem is httpd.pid was removed BEFORE "adapcctl.sh stop" fully completed, maybe due to Linux server crash or power off.  Normally, "adapcctl.sh stop" checks it and then removes it. Because of that, adapcctl.sh failed on checking a status and refused to start Apache.

Additionally, opmn logs can be found in $FMW_HOME/webtier/instances/$iName/diagnostics/logs/OPMN/opmn

The workaround:

1. Stop/kill all opmn processes  (keeping WLS related processes will be fine) 

$ sh $ADMIN_SCRIPTS_HOME/adopmnctl.sh stop

$ ps -ef | grep opmn

2. Create a empty file 

$ cd $FMW_HOME/webtier/instances/$iName/diagnostics/logs/OHS/$SUBiName

$ touch httpd.pid

3. Clear a folder

$ cd $FMW_HOME/webtier/instances/$iName/config/OPMN/opmn

$ ls -al states

-rw-r----- 1 user group 19 Jun 21 18:57 .opmndat

-rw-r----- 1 user group 579 Jun 21 18:54 p1878855085

$ mv states states_BK

$ mkdir states

$ ls -al states

4. Now, starting Apache shall work

$ ./adapcctl.sh start

$ ./adopmnctl.sh status

$ ps -ef | grep httpd | wc -l

4                  <== 3 httpd.worker processes running

5. Make sure all work

./adstpall.sh apps/appsPWD

./adstrtal.sh apps/appsPWD

./adopmnctl.sh status

When Apache (OHS) starts up, it writes the process ID (PID) of the parent httpd process to the httpd.pid file. When Apache is running, file httpd.pid shall exist and not be empty.