EBS login page get frozen and then timed out

A R12 EBS web site stayed frozen and did not re-direct to the login page. It seems it was waiting something or was trying to gain an access, until timed-out with Internal error. The message in Apache log:

[Thu Sep  8 07:17:03 2016] [warn] [client 172.xxx.xx.xxx] oc4j_socket_recvfull timed out
[Thu Sep  8 07:17:03 2016] [error] [client 172.xxx.xx.xxx] [ecid: xxx] mod_oc4j: request to OC4J hostname.domain.com:21700 failed: Connect failed

I checked around and saw errors in below log file:

$LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_1/application.log

16/09/07 16:01:54.978 10.1.3.5.0 Started
16/09/07 16:02:03.327 html: 10.1.3.5.0 Started
16/09/07 16:15:40.920 html: Error initializing servlet
java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.profiles.Profiles    at oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect.java:239)
        at oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect.java:287)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.loadServlet(HttpApplication.java:2529)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.findServlet(HttpApplication.java:5008)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.findServlet(HttpApplication.java:4932)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.getRequestDispatcher(HttpApplication.java:3140)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:775)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:458)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:313)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:199)
        at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
        at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234)
        at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29)
        at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
        at java.lang.Thread.run(Thread.java:619)

This error could be port for s_java_object_cache_port is not available.

$ grep s_java_object_cache_port $CONTEXT_FILE
      <jcache_port oa_var="s_java_object_cache_port" oa_type="PORT" base="12345" step="1" range="-1" label="Java Object Cache Port">12385</jcache_port>

I verified port 12385 was in use:     
$ netstat -an | grep 12385
tcp     1480      0 ::ffff:157.121.53.0:12385   ::ffff:167.69.38.184:1571   ESTABLISHED

Then, I asked System Admin to run below lines as root to get the detail:
#  lsof -i :12385
COMMAND  PID     USER     FD   TYPE DEVICE SIZE/OFF NODE NAME
java             7135   ebsdev2  194u  IPv6 156352      0t0  TCP hostname.domain.com:12385->dbHost.domian.com:rdb-dbs-disp (ESTABLISHED)

# ls -l /proc/7135/exe
lrwxrwxrwx 1 applmgr2 users 0 Sep  7 13:31 /proc/7135/exe -> /path/to/apps/tech_st/10.1.3/appsutil/jdk/bin/java

It appears port 12385 was used by another EBS instance EBSDEV2 on the same server.

The fix is easy. I stopped all EBS services for instance EBSDEV2, and then started the ones that was frozen first. All worked fine.

Additional Notes:
- In my R12.1.3 instances,  I experienced EBS site is inaccessible with below errors in EBS first webpage. The cause could be a change in network switch or some interruption on host/database. The fix in my cases is to make sure apps account in database is not locked and recycle EBS services.

Unable to generate forwarding URL. Exception: oracle.apps.fnd.common.AppsException: oracle.apps.jtf.base.resources.FrameworkException

Unable to generate forwarding URL. Exception: java.lang.NullPointerException

- Same fix for intermittent and sporadic login issue with error
You have encountered an unexpected error. Please contact the System Administrator for assistance.

- Below error in EBS first webpage could be caused by database error (e.g. database is down or a tablespace is full):
Unable to generate forwarding URL. Exception: oracle.apps.fnd.common.AppsException: oracle.apps.fnd.common.PoolException: Exception creating new Poolable object. 

- Got below error on 2nd page (login page) from re-directing 1st webpage on IE browser. The problem went away after deleted IE cache files.
Unable to authenticate session

- After Apps nodes rebooted unexpectedly, EBS site webpage does not work even Apache started fine without any error. The problem was related to some kind of cache issue. The final fix is to bounce the database (surprisingly).

- After DBA applied patches and made changes to Java on database server, EBS site re-directs to login page that is not coming up, instead a blank page is being displayed. I check around and get below result:

SQL> select fnd_web_sec.validate_password('GUEST','oracle') from dual;

FND_WEB_SEC.VALIDATE_PASSWORD('GUEST','ORACLE')
--------------------------------------------------------------------------------
N

SQL> select fnd_message.get from dual;

GET
--------------------------------------------------------------------------------
Oracle error -29548: ORA-29548: Java system class reported: release of Java system classes in the database (12.1.0.2.200714 1.6) does not match that of the oracle executable (12.1.0.2.171017 1.6) has been detected in FND_WEB_SEC.VALIDATE_PASSWORD.

It matches exact same error as in Doc ID 1673030.1, which says "If it shows ORA-29548 errors, then issue is with the database OJVM. Rebuild it using Note 2149019.1 How to Install, Remove, Reload, Validate and Repair the JVM Component in an Oracle Database". DBA had to fix the problem.

Run concurrent jobs on EBS host in a different account

Users need to run shell scripts to submit concurrent jobs and then do other things. If you do not want to share the Applmgr password to other users, you can set up a separate OS account for users to run their scripts. Assume Oracle EBS is installed under /u01/app/ by OS user Applmgr.

$ hostname
hostname1q
$ echo $APPL_TOP
/u01/app/EBSQA/apps/apps_st/appl
$ echo $ORACLE_HOME
/u01/app/EBSQA/apps/tech_st/10.1.2
$ echo $TWO_TASK
EBSQA

1. create user batchuser on same host hostname1q

2. install Oracle client on the same host but under a different location 

$ whoami
batchuser
$ echo $ORACLE_HOME
/batchu01/app/product/11.2.0/EBIZ
$ echo ORACLE_SID
EBSQA

3. create a file to setup the environment variables
$ whoami
batchuser
$ more setenv.ebsqa
export ORACLE_HOME=/batchu01/app/product/11.2.0/EBIZ
export ORACLE_SID=EBSQA

export TNS_ADMIN=/batchu01/app/product/11.2.0/EBIZ/network/admin
export LD_LIBRARY_PATH=/batchu01/app/product/11.2.0/EBIZ/lib
export PATH=/batchu01/app/product/11.2.0/EBIZ/bin:/usr/bin:/usr/local/bin:/etc

. /u01/app/EBSQA/apps/apps_st/appl/EBSQA_hostname1q.env

export PATH=$PATH:/batchu01/app/product/11.2.0/EBIZ/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH

echo `which CONCSUB`
echo `which sqlldr`

$ . ./setenv.ebsqa
/u01/app/EBSQA/apps/apps_st/appl/fnd/12.0.0/bin/CONCSUB
/batchu01/app/product/11.2.0/EBIZ/bin/sqlldr

4. now, submit a concurrent job by $FND_TOP/bin/CONCSUB

$ CONCSUB APPS/appsPWD SYSADMIN "System Administrator" EBSadmin WAIT=Y CONCURRENT FND FNDSCURS PROGRAM_NAME='"Active Users"'
Submitted request 4654627 for CONCURRENT FND FNDSCURS PROGRAM_NAME="Active Users"
Normal completion
 
$ more $APPLCSF/log/l4654627.req
+---------------------------------------------------------------------------+
Application Object Library: Version : 12.0.0
Copyright (c) 1979, 1999, Oracle Corporation. All rights reserved.
FNDSCURS module: Active Users
+---------------------------------------------------------------------------+
Current system time is 28-JUL-2016 14:08:59
.. ... ...

5. Troubleshooting: if you get below error, the library file may be wrong or bad. Copy the file from EBS installation folder /u01/app/EBSQA/apps/tech_st/10.1.2/lib to current library folder /batchu01/app/product/11.2.0/EBIZ/lib.

CONCSUB: error while loading shared libraries: libclntsh.so.10.1: wrong ELF class: ELFCLASS64
CONCSUB: error while loading shared libraries: libnnz10.so: cannot open shared object file: No such file or directory

~~~~~~~~~~~~~~~~~~~~ A sample script ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#!/bin/ksh

# Some variables
CONC_APPUSR=EBSBATCH    # an EBS user who runs this cm job
CONC_LOGUSR=apps
MYFA_RESP_NAME=MY_FA_Resp
MYFA_SHORTNAME=MYFA

CONTROL_LOG=$APPLPTMP/log/${EBS_SYSCODE}.log 
SECU_HOME=$HOME/ebsbatch   # <- folder holds file orauser.pwd in which has apps password.
                                  # That means who has permission to run this script will know apps password
                                  # So, in production, put it a scheduler server with high security
date >> $CONTROL_LOG
print "CONCSUB $CONC_LOGUSR $MYFA_SHORTNAME $MYFA_RESP_NAME $CONC_APPUSR WAIT=Y CONCURRENT $MYFA_SHORTNAME FAB2RPT1" >> $CONTROL_LOG
## CONCSUB apps MYFA MY_FA_Resp EBSBATCH WAIT=Y CONCURRENT MYFA FAB2RPT1
submission=$(CONCSUB $CONC_LOGUSR $MYFA_SHORTNAME \
     ${MYFA_RESP_NAME} $CONC_APPUSR WAIT=Y CONCURRENT \
     $MYFA_SHORTNAME FABLD2RPT1 "BEFORE" < $SECU_HOME/orauser.pwd)
if (( $? != 0 ))
then
   print "Customization : CONCSUB invoke failed" >> $CONTROL_LOG
   exit 99
fi

print "Customization: Invoking CM for invoice lines from AP/PO report" >> $CONTROL_LOG

#-------------------------------------------------------------------------
#  Parse out request number from concsub output. Password prompt
#  may be included depending on invoke style, so trim up to key words.
#-------------------------------------------------------------------------

reqno=$( echo ${submission#*Submitted request}|awk {'print $1'} )

#-------------------------------------------------------------------------
#  Query Concurrent Mgr table selecting status code for this reqno
#-------------------------------------------------------------------------
MY_MASTER_PWD=apps/appsPWD@EBSQA   #Or, use Oracle Wallet to hide the password

completion_code=$(sqlplus -s $MY_MASTER_PWD <<-EOF
set heading off
set verify off
select fcr.status_code
from   applsys.fnd_concurrent_requests fcr
where  fcr.request_id = $reqno
/
exit
EOF)

# Trim off white space that sometimes occurs
completion_code=$( echo $completion_code|awk {'print $1'} )

#-------------------------------------------------------------------------
#  Exit based on completion code
#-------------------------------------------------------------------------
if [[ $completion_code = C ]]
then
  print "Concurrent Manager job "$reqno" completed succesfully" >> $CONTROL_LOG
  exit 0
else
  print "Concurrent Manager job #"$reqno" did not complete succesfully" >> $CONTROL_LOG
  cat $APPLCSF/$APPLLOG/l${reqno}.req $APPLCSF/$APPLOUT/o${reqno}.out  >> $CONTROL_LOG
  exit 1
fi

exit 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~