EBS login page get frozen and then timed out

A R12 EBS web site stayed frozen and did not re-direct to the login page. It seems it was waiting something or was trying to gain an access, until timed-out with Internal error. The message in Apache log:

[Thu Sep  8 07:17:03 2016] [warn] [client 172.xxx.xx.xxx] oc4j_socket_recvfull timed out
[Thu Sep  8 07:17:03 2016] [error] [client 172.xxx.xx.xxx] [ecid: xxx] mod_oc4j: request to OC4J hostname.domain.com:21700 failed: Connect failed

I checked around and saw errors in below log file:

$LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_1/application.log

16/09/07 16:01:54.978 10.1.3.5.0 Started
16/09/07 16:02:03.327 html: 10.1.3.5.0 Started
16/09/07 16:15:40.920 html: Error initializing servlet
java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.profiles.Profiles    at oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect.java:239)
        at oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect.java:287)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.loadServlet(HttpApplication.java:2529)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.findServlet(HttpApplication.java:5008)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.findServlet(HttpApplication.java:4932)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.getRequestDispatcher(HttpApplication.java:3140)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:775)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:458)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:313)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:199)
        at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
        at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234)
        at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29)
        at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
        at java.lang.Thread.run(Thread.java:619)

This error could be port for s_java_object_cache_port is not available.

$ grep s_java_object_cache_port $CONTEXT_FILE
      <jcache_port oa_var="s_java_object_cache_port" oa_type="PORT" base="12345" step="1" range="-1" label="Java Object Cache Port">12385</jcache_port>

I verified port 12385 was in use:     
$ netstat -an | grep 12385
tcp     1480      0 ::ffff:157.121.53.0:12385   ::ffff:167.69.38.184:1571   ESTABLISHED

Then, I asked System Admin to run below lines as root to get the detail:
#  lsof -i :12385
COMMAND  PID     USER     FD   TYPE DEVICE SIZE/OFF NODE NAME
java             7135   ebsdev2  194u  IPv6 156352      0t0  TCP hostname.domain.com:12385->dbHost.domian.com:rdb-dbs-disp (ESTABLISHED)

# ls -l /proc/7135/exe
lrwxrwxrwx 1 applmgr2 users 0 Sep  7 13:31 /proc/7135/exe -> /path/to/apps/tech_st/10.1.3/appsutil/jdk/bin/java

It appears port 12385 was used by another EBS instance EBSDEV2 on the same server.

The fix is easy. I stopped all EBS services for instance EBSDEV2, and then started the ones that was frozen first. All worked fine.

Additional Notes:
- In my R12.1.3 instances,  I experienced EBS site is inaccessible with below errors in EBS first webpage. The cause could be a change in network switch or some interruption on host/database. The fix in my cases is to make sure apps account in database is not locked and recycle EBS services.

Unable to generate forwarding URL. Exception: oracle.apps.fnd.common.AppsException: oracle.apps.jtf.base.resources.FrameworkException

Unable to generate forwarding URL. Exception: java.lang.NullPointerException

- Same fix for intermittent and sporadic login issue with error
You have encountered an unexpected error. Please contact the System Administrator for assistance.

- Below error in EBS first webpage could be caused by database error (e.g. database is down or a tablespace is full):
Unable to generate forwarding URL. Exception: oracle.apps.fnd.common.AppsException: oracle.apps.fnd.common.PoolException: Exception creating new Poolable object. 

- Got below error on 2nd page (login page) from re-directing 1st webpage on IE browser. The problem went away after deleted IE cache files.
Unable to authenticate session

- After Apps nodes rebooted unexpectedly, EBS site webpage does not work even Apache started fine without any error. The problem was related to some kind of cache issue. The final fix is to bounce the database (surprisingly).

- After DBA applied patches and made changes to Java on database server, EBS site re-directs to login page that is not coming up, instead a blank page is being displayed. I check around and get below result:

SQL> select fnd_web_sec.validate_password('GUEST','oracle') from dual;

FND_WEB_SEC.VALIDATE_PASSWORD('GUEST','ORACLE')
--------------------------------------------------------------------------------
N

SQL> select fnd_message.get from dual;

GET
--------------------------------------------------------------------------------
Oracle error -29548: ORA-29548: Java system class reported: release of Java system classes in the database (12.1.0.2.200714 1.6) does not match that of the oracle executable (12.1.0.2.171017 1.6) has been detected in FND_WEB_SEC.VALIDATE_PASSWORD.

It matches exact same error as in Doc ID 1673030.1, which says "If it shows ORA-29548 errors, then issue is with the database OJVM. Rebuild it using Note 2149019.1 How to Install, Remove, Reload, Validate and Repair the JVM Component in an Oracle Database". DBA had to fix the problem.