FTP and SFTP

1. Run ftp using a specific account interactively

$ ftp -inv remote_host.domain.com
Connected to remote_host.domain.com (10.133.67.38).
220 Microsoft FTP Service
Remote system type is Windows_NT.
ftp> user ftpuser
331 Password required for ftpuser.
Password:
230 User logged in.
ftp>

Note below line does not work:
$ ftp ftpuser@remote_host.domain.com
ftp: ftpuser@remote_host.domain.com: Name or service not known
ftp> exit

2. Run FTP in a shell script

Assume FTP login info is saved in file $HOME/.netpw in format:

machine WintEdi.domain.COM login ANET/userID password userPWD

REMOTE_PATH=......
LOCAL_PATH=......
REMOTE_MACHINE=WintEdi.domain.COM

function ftp_to_remote

{

ftpuserid=`grep $REMOTE_MACHINE $HOME/.netpw|cut -f4 -d' '`

ftpuserpwd=`grep $REMOTE_MACHINE $HOME/.netpw|cut -f6 -d' '`

ftp -inv <<EndFTP

open $REMOTE_MACHINE

user $ftpuserid $ftpuserpwd

cd $REMOTE_PATH
put $LOCAL_PATH/output.txt ouput1.txt
bye
EndFTP
}

3. Try "man ftp" for help.

4. SFTP options:

$ sftp -oport=2222 ftpuser@remote_host.domain.com
Connecting to remote_host.domain.com...
WARNING: Logon attempts are audited. Access and use allowed for authorized purposes only. Violators will be prosecuted

$ sftp -oIdentityFile=/path/to/sftp_key/id_dsa ftpuser@remote_host.domain.com
Also see https://erpondb.blogspot.com/2015/11/run-scp-or-sftp-without-password.html

Troubleshhot UTL_FILE error

Ran a PL/SQL code and got error message: 

ORA-06512: at
"SYS.UTL_FILE", line 536
ORA-29283: invalid file ope
declare
*
ERROR at line 1:
ORA-29283: invalid file operation
ORA-06512: at "SYS.UTL_FILE", line 536
ORA-29283: invalid file operation
ORA-06512: at line 90

Two steps to troubleshoot the error:

A. Find which directory it tries to write the file to, and verify it is defined in the database:

SQL> set pages 100 lines 100

SQL> select * from dba_directories where directory_name='FTPOUT';

The directory could be owned by SYS. But APPS should be granted READ and WRITE.
SQL> SELECT * FROM dba_tab_privs WHERE table_name = 'FTPOUT';

B. Run a short code as APPS to test it. If it works, file test_UTL.txt shall create in the directory.
SQL> set serveroutput on

DECLARE
  fileHandler UTL_FILE.FILE_TYPE;
  v_error_msg  varchar2(1000);
BEGIN
  fileHandler := UTL_FILE.FOPEN('FTPOUT', 'test_UTL.txt', 'W');
  UTL_FILE.PUT_LINE(fileHandler, 'Writing TO a test file\n');
  UTL_FILE.PUTF(fileHandler, 'Writing 2nd line to test file\n');
  UTL_FILE.FCLOSE(fileHandler);
EXCEPTION
  WHEN utl_file.invalid_path THEN
     raise_application_error(-20001, 'ERROR: Invalid PATH for the file.');
  when others then
     v_error_msg := substr(sqlerrm,1,100);
     utl_file.fclose(fileHandler);
     dbms_output.put_line('Error message is - ' || v_error_msg);
     raise;
END;
/

A MAX linesize written into file test_UTL.txt can be specified in the call:
UTL_FILE.FOPEN('FTPOUT', 'test_UTL.txt', 'W', 32767)

In EBS environment, the path for $APPLPTMP is not necessary to be in DBA_DIRECTORIES.

UPDATES in 2018:
During a database 12c upgrade, somehow the OS env variable ORA_NLS10 was unset. That made " utl_file.fcopy ('FTPOUT','test.txt','FTPOUT','test_copy.txt'); ", which is used by Export of AAD rules using Oracle Forms Lock/Unlock option, generate file test_copy.txt 0 in size and UTL_FILE.PUT_LINE fail if output file test_UTL.txt is larger than 1 MB with below errors:

declare
*
ERROR at line 1:
ORA-29282: invalid file ID
ORA-06512: at "SYS.UTL_FILE", line 77
ORA-06512: at "SYS.UTL_FILE", line 690
ORA-06512: at line 256
ORA-29285: file write error

or

Also check the init parameter file and value for
utl_file_dir in v$parameter table.
declare
*
ERROR at line 1:
ORA-29285: file write error
ORA-06512: at line 258

The fix is to set env variable ORA_NLS10 on database server to $ORACLE_HOME/nls/data/9idata

By the way, the quick way to test $APPLPTMP and UTL_FILE_DIR in EBS is to run one line:

SQL> exec FND_FILE.PUT_LINE(fnd_file.log, 'Hello World!');

A new file with a name like l0009966.tmp shall be created in the first directory specified in the db parameter utl_file_dir, containing 'Hello World!'.

EBS login page get frozen and then timed out

A R12 EBS web site stayed frozen and did not re-direct to the login page. It seems it was waiting something or was trying to gain an access, until timed-out with Internal error. The message in Apache log:

[Thu Sep  8 07:17:03 2016] [warn] [client 172.xxx.xx.xxx] oc4j_socket_recvfull timed out
[Thu Sep  8 07:17:03 2016] [error] [client 172.xxx.xx.xxx] [ecid: xxx] mod_oc4j: request to OC4J hostname.domain.com:21700 failed: Connect failed

I checked around and saw errors in below log file:

$LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_1/application.log

16/09/07 16:01:54.978 10.1.3.5.0 Started
16/09/07 16:02:03.327 html: 10.1.3.5.0 Started
16/09/07 16:15:40.920 html: Error initializing servlet
java.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.profiles.Profiles    at oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect.java:239)
        at oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect.java:287)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.loadServlet(HttpApplication.java:2529)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.findServlet(HttpApplication.java:5008)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.findServlet(HttpApplication.java:4932)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpApplication.getRequestDispatcher(HttpApplication.java:3140)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:775)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:458)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:313)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].server.http.AJPRequestHandler.run(AJPRequestHandler.java:199)
        at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
        at oracle.oc4j.network.ServerSocketAcceptHandler.procClientSocket(ServerSocketAcceptHandler.java:234)
        at oracle.oc4j.network.ServerSocketAcceptHandler.access$700(ServerSocketAcceptHandler.java:29)
        at oracle.oc4j.network.ServerSocketAcceptHandler$AcceptHandlerHorse.run(ServerSocketAcceptHandler.java:879)
        at com.evermind[Oracle Containers for J2EE 10g (10.1.3.5.0) ].util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
        at java.lang.Thread.run(Thread.java:619)

This error could be port for s_java_object_cache_port is not available.

$ grep s_java_object_cache_port $CONTEXT_FILE
      <jcache_port oa_var="s_java_object_cache_port" oa_type="PORT" base="12345" step="1" range="-1" label="Java Object Cache Port">12385</jcache_port>

I verified port 12385 was in use:     
$ netstat -an | grep 12385
tcp     1480      0 ::ffff:157.121.53.0:12385   ::ffff:167.69.38.184:1571   ESTABLISHED

Then, I asked System Admin to run below lines as root to get the detail:
#  lsof -i :12385
COMMAND  PID     USER     FD   TYPE DEVICE SIZE/OFF NODE NAME
java             7135   ebsdev2  194u  IPv6 156352      0t0  TCP hostname.domain.com:12385->dbHost.domian.com:rdb-dbs-disp (ESTABLISHED)

# ls -l /proc/7135/exe
lrwxrwxrwx 1 applmgr2 users 0 Sep  7 13:31 /proc/7135/exe -> /path/to/apps/tech_st/10.1.3/appsutil/jdk/bin/java

It appears port 12385 was used by another EBS instance EBSDEV2 on the same server.

The fix is easy. I stopped all EBS services for instance EBSDEV2, and then started the ones that was frozen first. All worked fine.

Additional Notes:
- In my R12.1.3 instances,  I experienced EBS site is inaccessible with below errors in EBS first webpage. The cause could be a change in network switch or some interruption on host/database. The fix in my cases is to make sure apps account in database is not locked and recycle EBS services.

Unable to generate forwarding URL. Exception: oracle.apps.fnd.common.AppsException: oracle.apps.jtf.base.resources.FrameworkException

Unable to generate forwarding URL. Exception: java.lang.NullPointerException

- Same fix for intermittent and sporadic login issue with error
You have encountered an unexpected error. Please contact the System Administrator for assistance.

- Below error in EBS first webpage could be caused by database error (e.g. database is down or a tablespace is full):
Unable to generate forwarding URL. Exception: oracle.apps.fnd.common.AppsException: oracle.apps.fnd.common.PoolException: Exception creating new Poolable object. 

- Got below error on 2nd page (login page) from re-directing 1st webpage on IE browser. The problem went away after deleted IE cache files.
Unable to authenticate session

- After Apps nodes rebooted unexpectedly, EBS site webpage does not work even Apache started fine without any error. The problem was related to some kind of cache issue. The final fix is to bounce the database (surprisingly).

- After DBA applied patches and made changes to Java on database server, EBS site re-directs to login page that is not coming up, instead a blank page is being displayed. I check around and get below result:

SQL> select fnd_web_sec.validate_password('GUEST','oracle') from dual;

FND_WEB_SEC.VALIDATE_PASSWORD('GUEST','ORACLE')
--------------------------------------------------------------------------------
N

SQL> select fnd_message.get from dual;

GET
--------------------------------------------------------------------------------
Oracle error -29548: ORA-29548: Java system class reported: release of Java system classes in the database (12.1.0.2.200714 1.6) does not match that of the oracle executable (12.1.0.2.171017 1.6) has been detected in FND_WEB_SEC.VALIDATE_PASSWORD.

It matches exact same error as in Doc ID 1673030.1, which says "If it shows ORA-29548 errors, then issue is with the database OJVM. Rebuild it using Note 2149019.1 How to Install, Remove, Reload, Validate and Repair the JVM Component in an Oracle Database". DBA had to fix the problem.

Run concurrent jobs on EBS host in a different account

Users need to run shell scripts to submit concurrent jobs and then do other things. If you do not want to share the Applmgr password to other users, you can set up a separate OS account for users to run their scripts. Assume Oracle EBS is installed under /u01/app/ by OS user Applmgr.

$ hostname
hostname1q
$ echo $APPL_TOP
/u01/app/EBSQA/apps/apps_st/appl
$ echo $ORACLE_HOME
/u01/app/EBSQA/apps/tech_st/10.1.2
$ echo $TWO_TASK
EBSQA

1. create user batchuser on same host hostname1q

2. install Oracle client on the same host but under a different location 

$ whoami
batchuser
$ echo $ORACLE_HOME
/batchu01/app/product/11.2.0/EBIZ
$ echo ORACLE_SID
EBSQA

3. create a file to setup the environment variables
$ whoami
batchuser
$ more setenv.ebsqa
export ORACLE_HOME=/batchu01/app/product/11.2.0/EBIZ
export ORACLE_SID=EBSQA

export TNS_ADMIN=/batchu01/app/product/11.2.0/EBIZ/network/admin
export LD_LIBRARY_PATH=/batchu01/app/product/11.2.0/EBIZ/lib
export PATH=/batchu01/app/product/11.2.0/EBIZ/bin:/usr/bin:/usr/local/bin:/etc

. /u01/app/EBSQA/apps/apps_st/appl/EBSQA_hostname1q.env

export PATH=$PATH:/batchu01/app/product/11.2.0/EBIZ/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH

echo `which CONCSUB`
echo `which sqlldr`

$ . ./setenv.ebsqa
/u01/app/EBSQA/apps/apps_st/appl/fnd/12.0.0/bin/CONCSUB
/batchu01/app/product/11.2.0/EBIZ/bin/sqlldr

4. now, submit a concurrent job by $FND_TOP/bin/CONCSUB

$ CONCSUB APPS/appsPWD SYSADMIN "System Administrator" EBSadmin WAIT=Y CONCURRENT FND FNDSCURS PROGRAM_NAME='"Active Users"'
Submitted request 4654627 for CONCURRENT FND FNDSCURS PROGRAM_NAME="Active Users"
Normal completion
 
$ more $APPLCSF/log/l4654627.req
+---------------------------------------------------------------------------+
Application Object Library: Version : 12.0.0
Copyright (c) 1979, 1999, Oracle Corporation. All rights reserved.
FNDSCURS module: Active Users
+---------------------------------------------------------------------------+
Current system time is 28-JUL-2016 14:08:59
.. ... ...

5. Troubleshooting: if you get below error, the library file may be wrong or bad. Copy the file from EBS installation folder /u01/app/EBSQA/apps/tech_st/10.1.2/lib to current library folder /batchu01/app/product/11.2.0/EBIZ/lib.

CONCSUB: error while loading shared libraries: libclntsh.so.10.1: wrong ELF class: ELFCLASS64
CONCSUB: error while loading shared libraries: libnnz10.so: cannot open shared object file: No such file or directory

~~~~~~~~~~~~~~~~~~~~ A sample script ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#!/bin/ksh

# Some variables
CONC_APPUSR=EBSBATCH    # an EBS user who runs this cm job
CONC_LOGUSR=apps
MYFA_RESP_NAME=MY_FA_Resp
MYFA_SHORTNAME=MYFA

CONTROL_LOG=$APPLPTMP/log/${EBS_SYSCODE}.log 
SECU_HOME=$HOME/ebsbatch   # <- folder holds file orauser.pwd in which has apps password.
                                  # That means who has permission to run this script will know apps password
                                  # So, in production, put it a scheduler server with high security
date >> $CONTROL_LOG
print "CONCSUB $CONC_LOGUSR $MYFA_SHORTNAME $MYFA_RESP_NAME $CONC_APPUSR WAIT=Y CONCURRENT $MYFA_SHORTNAME FAB2RPT1" >> $CONTROL_LOG
## CONCSUB apps MYFA MY_FA_Resp EBSBATCH WAIT=Y CONCURRENT MYFA FAB2RPT1
submission=$(CONCSUB $CONC_LOGUSR $MYFA_SHORTNAME \
     ${MYFA_RESP_NAME} $CONC_APPUSR WAIT=Y CONCURRENT \
     $MYFA_SHORTNAME FABLD2RPT1 "BEFORE" < $SECU_HOME/orauser.pwd)
if (( $? != 0 ))
then
   print "Customization : CONCSUB invoke failed" >> $CONTROL_LOG
   exit 99
fi

print "Customization: Invoking CM for invoice lines from AP/PO report" >> $CONTROL_LOG

#-------------------------------------------------------------------------
#  Parse out request number from concsub output. Password prompt
#  may be included depending on invoke style, so trim up to key words.
#-------------------------------------------------------------------------

reqno=$( echo ${submission#*Submitted request}|awk {'print $1'} )

#-------------------------------------------------------------------------
#  Query Concurrent Mgr table selecting status code for this reqno
#-------------------------------------------------------------------------
MY_MASTER_PWD=apps/appsPWD@EBSQA   #Or, use Oracle Wallet to hide the password

completion_code=$(sqlplus -s $MY_MASTER_PWD <<-EOF
set heading off
set verify off
select fcr.status_code
from   applsys.fnd_concurrent_requests fcr
where  fcr.request_id = $reqno
/
exit
EOF)

# Trim off white space that sometimes occurs
completion_code=$( echo $completion_code|awk {'print $1'} )

#-------------------------------------------------------------------------
#  Exit based on completion code
#-------------------------------------------------------------------------
if [[ $completion_code = C ]]
then
  print "Concurrent Manager job "$reqno" completed succesfully" >> $CONTROL_LOG
  exit 0
else
  print "Concurrent Manager job #"$reqno" did not complete succesfully" >> $CONTROL_LOG
  cat $APPLCSF/$APPLLOG/l${reqno}.req $APPLCSF/$APPLOUT/o${reqno}.out  >> $CONTROL_LOG
  exit 1
fi

exit 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Profile option FND_INIT_SQL may cause error

When switching Responsibility, forms gave error:

Oracle error -6550: ORA-06550 : line 1, column 7:
 PLS-00306: wrong number or types of arguments in call to 'INIT'
 ORA-06550 : line 1, column 7:
 PL/SQL: Statement ignored
 has been detected in fnd_global.initialize[fnd_init_sql].

I checked profile 'Initialization SQL Statement - Custom' as suggested by Doc ID 577753.1 and did see anything on Site level. But when I ran below code, I sew one at Application level (by ID 10002).

SQL> select PROFILE_OPTION_ID, LEVEL_ID, LEVEL_VALUE, PROFILE_OPTION_VALUE
    from  apps.fnd_profile_option_values
    where profile_option_id = (
     select   profile_option_id   from  apps.fnd_profile_options
    where  profile_option_name = 'FND_INIT_SQL' );
PROFILE_OPTION_ID   LEVEL_ID LEVEL_VALUE   PROFILE_OPTION_VALUE
---------------------------  --------------- --------------------  --------------------------------
             3157                  10002               20004       begin my_global.init; end;

I ran a script in check R12 profile options to got the custom Application name. After I changed profile 'Initialization SQL Statement - Custom' on that Application level to null, which deletes that row from table apps.fnd_profile_option_values, the Oracle errors on forms were fixed.