EBS login page get frozen and then timed out

A R12 EBS web site stayed frozen and did not re-direct to the login page. It seems it was waiting something or was trying to gain an access, until timed-out with Internal error. The message in Apache log:

[Thu Sep  8 07:17:03 2016] [warn] [client 172.xxx.xx.xxx] oc4j_socket_recvfull timed out
[Thu Sep  8 07:17:03 2016] [error] [client 172.xxx.xx.xxx] [ecid: xxx] mod_oc4j: request to OC4J hostname.domain.com:21700 failed: Connect failed

I checked around and saw errors in below log file:

$LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_1/application.log

16/09/07 16:01:54.978 10.1.3.5.0 Started
16/09/07 16:02:03.327 html: 10.1.3.5.0 Started
16/09/07 16:15:40.920 html: Error initializing servlet
java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.profiles.Profiles    at oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect.java:239)
        at oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect.java:287)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.loadServlet(HttpApplication.java:2529)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.findServlet(HttpApplication.java:5008)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.findServlet(HttpApplication.java:4932)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.getRequestDispatcher(HttpApplication.java:3140)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:775)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:458)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:313)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:199)
        at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
        at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234)
        at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29)
        at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
        at java.lang.Thread.run(Thread.java:619)

This error could be port for s_java_object_cache_port is not available.

$ grep s_java_object_cache_port $CONTEXT_FILE
      <jcache_port oa_var="s_java_object_cache_port" oa_type="PORT" base="12345" step="1" range="-1" label="Java Object Cache Port">12385</jcache_port>

I verified port 12385 was in use:     
$ netstat -an | grep 12385
tcp     1480      0 ::ffff:157.121.53.0:12385   ::ffff:167.69.38.184:1571   ESTABLISHED

Then, I asked System Admin to run below lines as root to get the detail:
#  lsof -i :12385
COMMAND  PID     USER     FD   TYPE DEVICE SIZE/OFF NODE NAME
java             7135   ebsdev2  194u  IPv6 156352      0t0  TCP hostname.domain.com:12385->dbHost.domian.com:rdb-dbs-disp (ESTABLISHED)

# ls -l /proc/7135/exe
lrwxrwxrwx 1 applmgr2 users 0 Sep  7 13:31 /proc/7135/exe -> /path/to/apps/tech_st/10.1.3/appsutil/jdk/bin/java

It appears port 12385 was used by another EBS instance EBSDEV2 on the same server.

The fix is easy. I stopped all EBS services for instance EBSDEV2, and then started the ones that was frozen first. All worked fine.

Additional Notes:
- In my R12.1.3 instances,  I experienced EBS site is inaccessible with below errors in EBS first webpage. The cause could be a change in network switch or some interruption on host/database. The fix in my cases is to make sure apps account in database is not locked and recycle EBS services.

Unable to generate forwarding URL. Exception: oracle.apps.fnd.common.AppsException: oracle.apps.jtf.base.resources.FrameworkException

Unable to generate forwarding URL. Exception: java.lang.NullPointerException

- Same fix for intermittent and sporadic login issue with error
You have encountered an unexpected error. Please contact the System Administrator for assistance.

- Below error in EBS first webpage could be caused by database error (e.g. database is down or a tablespace is full):
Unable to generate forwarding URL. Exception: oracle.apps.fnd.common.AppsException: oracle.apps.fnd.common.PoolException: Exception creating new Poolable object. 

- Got below error on 2nd page (login page) from re-directing 1st webpage on IE browser. The problem went away after deleted IE cache files.
Unable to authenticate session

- After Apps nodes rebooted unexpectedly, EBS site webpage does not work even Apache started fine without any error. The problem was related to some kind of cache issue. The final fix is to bounce the database (surprisingly).

- After DBA applied patches and made changes to Java on database server, EBS site re-directs to login page that is not coming up, instead a blank page is being displayed. I check around and get below result:

SQL> select fnd_web_sec.validate_password('GUEST','oracle') from dual;

FND_WEB_SEC.VALIDATE_PASSWORD('GUEST','ORACLE')
--------------------------------------------------------------------------------
N

SQL> select fnd_message.get from dual;

GET
--------------------------------------------------------------------------------
Oracle error -29548: ORA-29548: Java system class reported: release of Java system classes in the database (12.1.0.2.200714 1.6) does not match that of the oracle executable (12.1.0.2.171017 1.6) has been detected in FND_WEB_SEC.VALIDATE_PASSWORD.

It matches exact same error as in Doc ID 1673030.1, which says "If it shows ORA-29548 errors, then issue is with the database OJVM. Rebuild it using Note 2149019.1 How to Install, Remove, Reload, Validate and Repair the JVM Component in an Oracle Database". DBA had to fix the problem.

Run concurrent jobs on EBS host in a different account

Users need to run shell scripts to submit concurrent jobs and then do other things. If you do not want to share the Applmgr password to other users, you can set up a separate OS account for users to run their scripts. Assume Oracle EBS is installed under /u01/app/ by OS user Applmgr.

$ hostname
hostname1q
$ echo $APPL_TOP
/u01/app/EBSQA/apps/apps_st/appl
$ echo $ORACLE_HOME
/u01/app/EBSQA/apps/tech_st/10.1.2
$ echo $TWO_TASK
EBSQA

1. create user batchuser on same host hostname1q

2. install Oracle client on the same host but under a different location 

$ whoami
batchuser
$ echo $ORACLE_HOME
/batchu01/app/product/11.2.0/EBIZ
$ echo ORACLE_SID
EBSQA

3. create a file to setup the environment variables
$ whoami
batchuser
$ more setenv.ebsqa
export ORACLE_HOME=/batchu01/app/product/11.2.0/EBIZ
export ORACLE_SID=EBSQA

export TNS_ADMIN=/batchu01/app/product/11.2.0/EBIZ/network/admin
export LD_LIBRARY_PATH=/batchu01/app/product/11.2.0/EBIZ/lib
export PATH=/batchu01/app/product/11.2.0/EBIZ/bin:/usr/bin:/usr/local/bin:/etc

. /u01/app/EBSQA/apps/apps_st/appl/EBSQA_hostname1q.env

export PATH=$PATH:/batchu01/app/product/11.2.0/EBIZ/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH

echo `which CONCSUB`
echo `which sqlldr`

$ . ./setenv.ebsqa
/u01/app/EBSQA/apps/apps_st/appl/fnd/12.0.0/bin/CONCSUB
/batchu01/app/product/11.2.0/EBIZ/bin/sqlldr

4. now, submit a concurrent job by $FND_TOP/bin/CONCSUB

$ CONCSUB APPS/appsPWD SYSADMIN "System Administrator" EBSadmin WAIT=Y CONCURRENT FND FNDSCURS PROGRAM_NAME='"Active Users"'
Submitted request 4654627 for CONCURRENT FND FNDSCURS PROGRAM_NAME="Active Users"
Normal completion
 
$ more $APPLCSF/log/l4654627.req
+---------------------------------------------------------------------------+
Application Object Library: Version : 12.0.0
Copyright (c) 1979, 1999, Oracle Corporation. All rights reserved.
FNDSCURS module: Active Users
+---------------------------------------------------------------------------+
Current system time is 28-JUL-2016 14:08:59
.. ... ...

5. Troubleshooting: if you get below error, the library file may be wrong or bad. Copy the file from EBS installation folder /u01/app/EBSQA/apps/tech_st/10.1.2/lib to current library folder /batchu01/app/product/11.2.0/EBIZ/lib.

CONCSUB: error while loading shared libraries: libclntsh.so.10.1: wrong ELF class: ELFCLASS64
CONCSUB: error while loading shared libraries: libnnz10.so: cannot open shared object file: No such file or directory

~~~~~~~~~~~~~~~~~~~~ A sample script ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#!/bin/ksh

# Some variables
CONC_APPUSR=EBSBATCH    # an EBS user who runs this cm job
CONC_LOGUSR=apps
MYFA_RESP_NAME=MY_FA_Resp
MYFA_SHORTNAME=MYFA

CONTROL_LOG=$APPLPTMP/log/${EBS_SYSCODE}.log 
SECU_HOME=$HOME/ebsbatch   # <- folder holds file orauser.pwd in which has apps password.
                                  # That means who has permission to run this script will know apps password
                                  # So, in production, put it a scheduler server with high security
date >> $CONTROL_LOG
print "CONCSUB $CONC_LOGUSR $MYFA_SHORTNAME $MYFA_RESP_NAME $CONC_APPUSR WAIT=Y CONCURRENT $MYFA_SHORTNAME FAB2RPT1" >> $CONTROL_LOG
## CONCSUB apps MYFA MY_FA_Resp EBSBATCH WAIT=Y CONCURRENT MYFA FAB2RPT1
submission=$(CONCSUB $CONC_LOGUSR $MYFA_SHORTNAME \
     ${MYFA_RESP_NAME} $CONC_APPUSR WAIT=Y CONCURRENT \
     $MYFA_SHORTNAME FABLD2RPT1 "BEFORE" < $SECU_HOME/orauser.pwd)
if (( $? != 0 ))
then
   print "Customization : CONCSUB invoke failed" >> $CONTROL_LOG
   exit 99
fi

print "Customization: Invoking CM for invoice lines from AP/PO report" >> $CONTROL_LOG

#-------------------------------------------------------------------------
#  Parse out request number from concsub output. Password prompt
#  may be included depending on invoke style, so trim up to key words.
#-------------------------------------------------------------------------

reqno=$( echo ${submission#*Submitted request}|awk {'print $1'} )

#-------------------------------------------------------------------------
#  Query Concurrent Mgr table selecting status code for this reqno
#-------------------------------------------------------------------------
MY_MASTER_PWD=apps/appsPWD@EBSQA   #Or, use Oracle Wallet to hide the password

completion_code=$(sqlplus -s $MY_MASTER_PWD <<-EOF
set heading off
set verify off
select fcr.status_code
from   applsys.fnd_concurrent_requests fcr
where  fcr.request_id = $reqno
/
exit
EOF)

# Trim off white space that sometimes occurs
completion_code=$( echo $completion_code|awk {'print $1'} )

#-------------------------------------------------------------------------
#  Exit based on completion code
#-------------------------------------------------------------------------
if [[ $completion_code = C ]]
then
  print "Concurrent Manager job "$reqno" completed succesfully" >> $CONTROL_LOG
  exit 0
else
  print "Concurrent Manager job #"$reqno" did not complete succesfully" >> $CONTROL_LOG
  cat $APPLCSF/$APPLLOG/l${reqno}.req $APPLCSF/$APPLOUT/o${reqno}.out  >> $CONTROL_LOG
  exit 1
fi

exit 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Profile option FND_INIT_SQL may cause error

When switching Responsibility, forms gave error:

Oracle error -6550: ORA-06550 : line 1, column 7:
 PLS-00306: wrong number or types of arguments in call to 'INIT'
 ORA-06550 : line 1, column 7:
 PL/SQL: Statement ignored
 has been detected in fnd_global.initialize[fnd_init_sql].

I checked profile 'Initialization SQL Statement - Custom' as suggested by Doc ID 577753.1 and did see anything on Site level. But when I ran below code, I sew one at Application level (by ID 10002).

SQL> select PROFILE_OPTION_ID, LEVEL_ID, LEVEL_VALUE, PROFILE_OPTION_VALUE
    from  apps.fnd_profile_option_values
    where profile_option_id = (
     select   profile_option_id   from  apps.fnd_profile_options
    where  profile_option_name = 'FND_INIT_SQL' );
PROFILE_OPTION_ID   LEVEL_ID LEVEL_VALUE   PROFILE_OPTION_VALUE
---------------------------  --------------- --------------------  --------------------------------
             3157                  10002               20004       begin my_global.init; end;

I ran a script in check R12 profile options to got the custom Application name. After I changed profile 'Initialization SQL Statement - Custom' on that Application level to null, which deletes that row from table apps.fnd_profile_option_values, the Oracle errors on forms were fixed.

Run FTP and submit concurrent job by SQL

This is a SQL script to FTP get a file from a remote Windows server to database server, and then to submit an EBS concurrent job which will read that file. The database version is Oracle 11.2.0.4.

/** pre setups:
SQL> select fnd_profile.value('WIN_UPS_FILE_INFO') from dual;
FND_PROFILE.VALUE('WIN_UPS_FILE_INFO')
--------------------------------------------------------------------------------
PUL/NT_USERID passWD WIN_HOSTNAME.domain.com

SQL> select * from all_directories where directory_name = 'CUSTOM_FTPIN';
OWNER   DIRECTORY_NAME  DIRECTORY_PATH
---------   -----------------------   -----------------------------
SYS         CUSTOM_FTPIN        /path/to/ftp/in
**/

SQL> set serveroutput on
SQL> DECLARE
      --  l_utl_file_handler          UTL_FILE.FILE_TYPE;
      l_conn  UTL_TCP.connection;
      l_chr_file_line varchar2(200);
      l_chr_user_id varchar2(20);
      l_chr_password varchar2(20);
      l_chr_directory_path varchar2(200);
      l_chr_server_name varchar2(40);
      l_chr_file_name varchar2(20) := 'WIN07-24-2016.csv';
      -- l_chr_file_name varchar2(20) := '&1';
      l_request_id Number;
 BEGIN
    l_chr_file_line := fnd_profile.value('WIN_UPS_FILE_INFO');
    select substr(l_chr_file_line,1,12) into l_chr_user_id from dual;
    select substr(l_chr_file_line,14,7) into l_chr_password from dual;
    select substr(l_chr_file_line,22,30) into l_chr_server_name from dual;

    l_conn := ftp.login(l_chr_server_name, '21',l_chr_user_id,l_chr_password);
    ftp.ascii(p_conn => l_conn);
    ftp.get(p_conn      => l_conn,
               p_from_file => l_chr_file_name,
               p_to_dir    => 'CUSTOM_FTPIN',
               p_to_file   => l_chr_file_name);
     ftp.logout(l_conn);
    utl_tcp.close_all_connections;

/**  commend this out for testing FTP
 l_request_id := FND_REQUEST.SUBMIT_REQUEST('MYAP','MYAPUFVW','','',FALSE,''||l_chr_file_name||'',
chr(0), '', '','','',
                  '', '', '', '', '', '', '', '', '', '',
                  '', '', '', '', '', '', '', '', '', '',
                  '', '', '', '', '', '', '', '', '', '',
                  '', '', '', '', '', '', '', '', '', '',
                  '', '', '', '', '', '', '', '', '', '',
                  '', '', '', '', '', '', '', '', '', '',
                  '', '', '', '', '', '', '', '', '', '',
                  '', '', '', '', '', '', '', '', '', '',
                  '', '', '', '', '', '', '', '', '');

**/
commit;
END;
 /

SQL>
220 Microsoft FTP Service
331 Password required for PUL/NT_USERID.
230 User logged in.
200 Type set to A.
227 Entering Passive Mode (177,79,176,170,278,170).
125 Data connection already open; Transfer starting.
226 Transfer complete.
221 Goodbye.

PL/SQL procedure successfully completed.

SQL> exit

Network error FRM-92102 and F5

A R12.1.3 EBS site has one concurrent server and two web/forms servers which are monitored by F5 for load balance. If I started Apache services on two web hosts, forms got network error very quick. But if I started services just on one host, this problem would not happen. The error is

FRM-92102: A network error has occurred. The Forms Client has attempted to reestablish its
connection to the server 5 times(s) without success. Please check the network connection and try again later.

The Java Console log showed " oracle.forms.net.ConnectionException: 500 " error.

I believed this was something in the network, not in the EBS apps config. I worked with F5 admin on this and they found Persistence profile in F5 setups was null for this site somehow. After they set the value to two hours, this problem does not happen again in this site.

I had similar problem in another R12 site before that, when forms got FRM-92102 errors after user opened forms for a few minutes. That problem was caused by a wrong timeout value for Persistence profile in F5.