Monday, August 18, 2014

Compile Oracle EBS forms and/or libraries

1. Use below steps as Applmgr to compile one R12 form in $CUSTOM_TOP/forms/US directory. Note frmcmp_batch.sh shall exist in $ORACLE_HOME/bin.

$ export FORMS_PATH=$FORMS_PATH:$AU_TOP/forms/US:$AU_TOP/resource:$CUSTOM_TOP/forms/US
$ cd $CUSTOM_TOP/forms/US
$ mv FORM_NAME.fmx FORM_NAME.fmx_BK (or delete it)
$ chmod 664 FORM_NAME.fmb (to make it -rw-rw-r--. This step seems optional).

$ frmcmp_batch.sh module=FORM_NAME.fmb userid=apps/XXXpwd output_file=FORM_NAME.fmx module_type=form compile_all=yes

If it works, it shall say "Created form file FORM_NAME.fmx". And new time stamp for two files FORM_NAME.fmb and FORM_NAME.fmx.

2. To compile only library (.pll) files, env variable $FORMS_PATH is not needed. Go to $AU_TOP/resources and run the command:

$ cd $AU_TOP/resource
$ frmcmp_batch.sh module=FORM_NAME.pll userid=apps/XXXpwd module_type=library compile_all=special batch=yes

After it completes, new timestamp will be on FORM_NAME.pll and FORM_NAME.plx without doing anything on .fmb and .fmx files (in $CUSTOM_TOP/forms/US if it is a custom form). The file size on FORM_NAME.pll may become bigger. Use command to check and compare contents:

$ adident Header $AU_TOP/resource/FORM_NAME.pll 

Notes:
(1). If option "batch=yes" is used, progress and error messages will not echo out on the screen when the commend runs. But they will be written to .err file in current directory.
(2). If it gets ORA-12154 error in R12, the workaround is to modify $TNS_ADMIN variable in frmcmp_batch.sh (Doc ID 431324.1).
export TNS_ADMIN=${TNS_ADMIN:-$ORACLE_HOME/network/admin}
(3). For a non-Applmgr user to compile a R12 form, it is necessary to change permission on a set of files (Doc ID 561334.1). Make sure ORACLE_HOME points to 10.1.2 location.
chmod +rx $ORACLE_HOME
chmod +rx $ORACLE_HOME/bin
chmod +rx $ORACLE_HOME/bin/frm*
chmod -R +rx $ORACLE_HOME/lib
chmod +rx $ORACLE_HOME/forms
chmod -R +rx $ORACLE_HOME/forms/mesg
chmod -R +rx $ORACLE_HOME/forms/admin
chmod -R +rx $ORACLE_HOME/oracore
chmod +rx $ORACLE_HOME/nls
chmod -R +rx $ORACLE_HOME/nls/data
chmod +rx $ORACLE_HOME/rdbms
chmod -R +rx $ORACLE_HOME/rdbms/mesg
chmod +rx $ORACLE_HOME/ldap
chmod -R +rx $ORACLE_HOME/ldap/mesg
chmod -R +rx $ORACLE_HOME/ldap/admin
chmod +rx $ORACLE_HOME/network
chmod -R +rx $ORACLE_HOME/network/admin
chmod -R +rx $ORACLE_HOME/jre
chmod -R +rx $ORACLE_HOME/procbuilder

(also go to $CUSTOM_TOP/forms/US, and give 666 to .fmx and .fmb files so that user can re-create the form files.) 
Note: after above grants, sqlplus is still not available to other users but form compile shall work.

3. Use ADADMIN to compile (or re-generate) multiple forms. The steps are documented in very good document: How To Regenerate Forms Library Files Using "adadmin" (11i & R12) (Doc ID 141118.1)
A) Select the "1. Generate Applications Files menu" option.
B) Then select the "2. Generate forms files" option.
C) Choose the number of workers (default is the # of CPUs in the server).
D) Enter "yes" to regenerate PL/SQL library files (.pll extension)
E) Enter "yes" to regenerate Forms Menu files (.mmb -> .mmx extension)
F) Enter "yes" to regenerate Forms Executable files (.fmb->.fmx extension)
G) Enter the list of products, by short name as they are listed in "$APPL_TOP/" (all OR ont, po, fnd, ar, gl)
H) Enter "No" to generate specific files for each product. If you like to generate specific files for each product, you may answer "Yes" here and a list of files will be displayed
I) Enter "all" to generate all the previously selected items for all languages, or only some of the installed languages.
J) Shows your default and/or selected languages
K) Enter "yes" if the languages selected are the desired languages

yes on F) will give new timestamp on .fmx files. After answered yes on D), some .plx files get new timestamp, but no .pll files get new timestamp.  I used adadmin to compile all forms in an instance and got one error:

The following Oracle Forms objects did not generate successfully:
ap      forms/US        APXWCARD.fmx

An error occurred while generating Oracle Forms objects.
Continue as if it were successful [No] :


adadmin log does not give much details on the error. So, I ran the command to compile this form individually:

$ cd $AP_TOP/forms/US
$ mv APXWCARD.fmx APXWCARD.fmx_BK_04292015
$ export FORMS_PATH=$FORMS_PATH:$AU_TOP/forms/US:$AU_TOP/resource:$AP_TOP/forms/US

$ frmcmp_batch.sh module=$AU_TOP/forms/US/APXWCARD.fmb userid=apps/APPS_PWD output_file=APXWCARD.fmx module_type=form compile_all=yes

FRM-30064: Unable to parse statement select distinct full_name, employee_number, date_of_birth, pa.person_id
from per_all_people_f pap,
financials_system_parameters fsp,
per_assignments_f pa
where (trunc(sysdate) between pap.effective_start_date and pap.effective_end_date)
and (trunc(sysdate) between pa.effective_start_date and pa.effective_end_date)
and pap.business_group_id = fsp.business_group_id
and pa.business_group_id = fsp.business_group_id
and pap.person_id = pa.person_id
and pa.assignment_type = 'E'
order by upper(full_name).
ORA-01791: not a SELECTed expression
Record Group FULL_NAME
Form: APXWCARD

FRM-30085: Unable to adjust form for output.
Form not created

To fix the problem, check the file version:
$ strings $AU_TOP/forms/US/APXWCARD.fmb | grep '$Header'
FDRCSID('$Header: APPSTAND.fmb 120.6.12010000.6 2010/03/10 12:58  dbowles ship     $');
FDRCSID('$Header: APXWCARD.fmb 120.16.12010000.14 2010/02/22 22:27  meesubra ship   $');

Doc ID 1314734.1 says it hits a database bug. So, there is always a risk of failures from compiling all forms.

4. Write a script to compile forms file

~~~~~~~~~~~~~~ Script to compile one custom form ~~~~~~~~~~~~~~
# Call:  $ ./script_name.sh forms_name
#
export CUSTOM_TOP=$APPL_TOP/custom     # where the custom form .fmb file is saved under
export FORMS_PATH=$FORMS_PATH:$AU_TOP/forms/US:$AU_TOP/resource:$CUSTOM_TOP/forms/US

Passwd='appsPWD'
custom_top=$CUSTOM_TOP/forms/US

cd $custom_top
custom_form=$1

frmcmp_batch module=$custom_form.fmb userid=apps/${Passwd} output_file=$custom_form.fmx module_type=form compile_all=yes batch=yes

exit 0

~~~~~~~~~~~~~~~~ Compile multiple forms ~~~~~~~~~~~~~~~~~
/path/to/script_name.sh forms_name1
/path/to/script_name.sh forms_name2
/path/to/script_name.sh forms_name3


UPDATES: If frmcmp_batch.sh in R12.1.3 gets ORA-12154 error (or error relates to listener), modify it with workaround described in Dec ID 431324.1 on $TNS_ADMIN:

TNS_ADMIN=${TNS_ADMIN:-$ORACLE_HOME/network/admin}
export TNS_ADMIN

Relinek apps executable file

1. Relink the AutoInvoice Master program (report) as follows: (refer to Doc ID 1209514.1)

a) Log into the Operating System as APPLMGR (on CM node)
b) $ cd $AR_TOP/bin
c) Make a backup copy on file RAXMTR
d) Make sure that Autoinvoice is not running (by checking concurrent jobs)
e) Run the relink script
    $ adrelink.sh force=y ranlib=y "ar RAXMTR"
 
=====log on the screen ==============================
  .....
  Done with link of ar executable 'RAXMTR' on Fri Jul 18 12:51:41 EDT 2014
  Done relinking module RAXMTR in product ar
  Done with link of product 'ar' on Fri Jul 18 12:51:41 EDT 2014
  adrelink is exiting with status 0
  End of adrelink session
  Date/time is Fri Jul 18 12:51:41 EDT 2014

2. To relink 10.1.2 reports executables:
a) $ cd $ORACLE_HOME/reports/lib (or $ORACLE_HOME/reports/lib32)
b) $ make -f ins_reports.mk install

3.To relink 10.1.2 forms executables:
a) $ cd $ORACLE_HOME/forms/lib (or $ORACLE_HOME/forms/lib32)
b) $ make -f ins_forms.mk install

4. Use adadmin to relink all apps programs
   choices:  2 => 1
    If all worked, you will see " adrelink is exiting with status 0 " at the end.
    Log file is $APPL_TOP/admin/${TWO_TASK}/log/adrelink.log

5. Regenerate the jar files with force:
 $ adadmin
   1 => 4. Make sure that you choose force equals Yes.

NOTES/UPDATES
1: Oracle usually suggests "regenerate the jar files with force" after "relink" all form excutables.

2: If below errors exist in the make log file:
    /usr/lib/libXtst.so.6: undefined reference to `__stack_chk_fail@GLIBC_2.4'
    /usr/lib/libXtst.so.6: undefined reference to `__fprintf_chk@GLIBC_2.3.4'
    /usr/lib/libXtst.so.6: undefined reference to `__sprintf_chk@GLIBC_2.3.4'

The fix is to update a required link to a Motif library (as root)
  # unlink /usr/lib/libXtst.so.6
  # ln -s /usr/X11R6/lib/libXtst.so.6.1 /usr/lib/libXtst.so.6

Before the change
$ ls -al /usr/lib/libXtst.so.6
lrwxrwxrwx 1 root root 16 Oct 11  2012 /usr/lib/libXtst.so.6 -> libXtst.so.6.1.0

After the fix, the result shall be
$ ls -al /usr/lib/libXtst.so.6
lrwxrwxrwx 1 root root 29 Jun 18 2014 /usr/lib/libXtst.so.6 -> /usr/X11R6/lib/libXtst.so.6.1
$ ls -al /usr/X11R6/lib/libXtst*
lrwxrwxrwx 1 root root    14 Oct 18  2011 /usr/X11R6/lib/libXtst.so -> libXtst.so.6.1
lrwxrwxrwx 1 root root    14 Oct 18  2011 /usr/X11R6/lib/libXtst.so.6 -> libXtst.so.6.1
-rwxr-xr-x 1 root root 17204 Feb 26  2008 /usr/X11R6/lib/libXtst.so.6.1


3: On Linux RHEL 6 and 7, R12.1 adadmin relink and/or adpatch may hit error
    libgcc_s.so: undefined reference to `__stack_chk_fail@GLIBC_2.4'

The fix is to apply patch 12415211 to 10.1.2 AND 10.1.3 ORACLE HOMES manually (see Doc ID 1525823.1). Below script shall do the job.

#!/bin/ksh
# Steps from Doc ID 761566.1 - search keyword 12415211 (patch number)
# copy file libgcc_s-2.3.2-stub.so from patch folder (
$HOME/RHEL6 in this script) to 
# $ORACLE_HOME/lib/stubs and $IAS_ORACLE_HOME/lib/stubs

echo $ORACLE_HOME
cd $ORACLE_HOME/lib
cp -p -R stubs stubsORIG_`date +%H%M`
pwd
cd stubs
ls -al libgcc_s-2.3.2-stub.so*
cp -p $HOME/RHEL6/libgcc_s-2.3.2-stub.so .
ln -s libgcc_s-2.3.2-stub.so libgcc_s.so.1
ln -s libgcc_s.so.1 libgcc_s.so
ls -al libgcc_s*.*

# -rw-r--r-- 1 id xxx 12246 Jul 27  2011
$ORACLE_HOME/lib/stubs/libgcc_s-2.3.2-stub.so
# lrwxrwxrwx 1 id xxx  13 Jun  2 16:32
$ORACLE_HOME/lib/stubs/libgcc_s.so -> libgcc_s.so.1
# lrwxrwxrwx 1 if xxx  22 Jun  2 16:32
$ORACLE_HOME/lib/stubs/libgcc_s.so.1 -> libgcc_s-2.3.2-stub.so
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
echo $IAS_ORACLE_HOME
cd $IAS_ORACLE_HOME/lib

cp -p -R stubs stubsORIG_`date +%H%M`
pwd
cd stubs
ls -al libgcc_s-2.3.2-stub.so*
cp -p $HOME/RHEL6/libgcc_s-2.3.2-stub.so .
ln -s libgcc_s-2.3.2-stub.so libgcc_s.so.1
ln -s libgcc_s.so.1 libgcc_s.so
ls -al libgcc_s*.*


4: One day, after I applied patches to ORACLE HOMEs and applied an AR patch, some AR forms hang and get frozen forever without any error or warning message. The issue was fixed by running steps 3 and 5 above (on each Web node).


Friday, August 8, 2014

R12 Patchset Level, R12 Apps code Level

How to determine R12 Patchset Level (or Module code level) , and module is installed/implemented or not (Doc ID 443699.1)

1. As applmgr or OS environment owner
(a) source environment
(b) login as apps
(c) SQL> @$AD_TOP/sql/adutconf.sql

The adutconf.sql script will create a text file adutconf.lst containing version information and product top information.  There is a section->”Product Installation Status, Version Info and Patch Level”, e.g.
     Product    Appl Status    Version    Patchset Level
     AD           Shared           12.0.0      R12.AD.B.3         

     FND         Installed         12.0.0      R12.FND.B.3       <-- call this "FND code level" also
  • Implemented product has status Installed.
  • Not installed Product has the Status Inactive.
  • Products with status Shared are not fully implemented - only partially installed for dependent Product(s).
  Note: there are exceptions. e.g. WebEDI module (BNE) may be used even its status is Inactive.
           AD = Applications DBA,  FND = Application Object Library (AOL, i.e. Foundation tables). 

2.  Doc ID 550654.1 (for R12.1)
Login to OAM. Choose menu “OAM Support Cart” in upper right hand corner
Navigate to :
(a) Support Cart –> Applications Signature –> Collect –> Check “Product Information” box
(b) Click on “View” (eyeglasses) to get the patchset level report

3.  Script (similar to the one for 11i in Doc ID 443699.1)
SELECT a.application_name, a.application_short_name, a.product_code,  b.patch_level, DECODE (b.status, 'I', 'Installed', 'S', 'Shared', 'Not Installed - N/A') status, b.last_update_date, b.tablespace
FROM  apps.fnd_product_installations b, apps.fnd_application_vl a,
WHERE a.application_id = b.application_id
-- and b.status=’I’
order by a.application_short_name asc;

--  Not sure what is in this table
SELECT app_short_name, MAX(patch_level)
FROM apps.ad_patch_driver_minipks
GROUP BY app_short_name order by app_short_name; 

Applications code level?

SQL> select release_name from apps.fnd_product_groups;
12.1.3

Tuesday, July 22, 2014

WFLOAD, FNDLOAD

1. Oracle EBS provides a UNIX command to get workflow definition file.  

WFLOAD <apps_user_name>/<password>@db 0 Y DOWNLOAD apinvapr.wft APINVAPR

apinvapr.wft will be created in the same directory that this command was ran from. APINVAPR is the workflow name.

2. FNDLOAD utility:

Here is an example on how to use FNDLOAD:

a) Use FNDLOAD to download a custom current program from Dev environment: 
FNDLOAD apps/<PWD> 0 Y DOWNLOAD $FND_TOP/patch/115/import/afcpprog.lct XXX_AWARD_FEEDBACK.ldt PROGRAM APPLICATION_SHORT_NAME="SQLAP" CONCURRENT_PROGRAM_NAME="XXX_AWARD_FEEDBACK"

NOTES: Somehow, FNDLOAD does not get full execution, if above line is saved to a shell script file .sh to run it.

b) Copy file XXX_AWARD_FEEDBACK.ldt to QA environment

c) Use FNSLOAD to upload it to QA environment
FNDLOAD apps/<QA_PWD> 0 Y UPLOAD $FND_TOP/patch/115/import/afcpprog.lct XXX_AWARD_FEEDBACK.ldt WARNING=YES UPLOAD_MODE=REPLACE CUSTOM_MODE=FORCE

Additional examples:

Download a profile option:
FNDLOAD apps/<PWD> O Y DOWNLOAD $FND_TOP/patch/115/import/afscprof.lct XXX_LOOKUP_CODE.ldt PROFILE PROFILE_NAME="XXX_LOOKUP_CODE" APPLICATION_SHORT_NAME="SQLAP"

Download a lookup type:
FNDLOAD apps/<PWD> O Y DOWNLOAD $FND_TOP/patch/115/import/aflvmlu.lct XXX_UNIT_LT.ldt FND_LOOKUP_TYPE APPLICATION_SHORT_NAME="SQLAP" LOOKUP_TYPE="XXX_UNIT_LT"

Download a value set:
FNDLOAD apps/Ebs#8myPwd 0 Y DOWNLOAD $FND_TOP/patch/115/import/afffload.lct XXX_OPTIONS_VS.ldt VALUE_SET FLEX_VALUE_SET_NAME="XXX_OPTIONS_VS"

Download a request group:
FNDLOAD apps/<PWD> O Y DOWNLOAD $FND_TOP/patch/115/import/afcpreqg.lct filename.ldt REQUEST_GROUP REQUEST_GROUP_NAME='GL Concurrent Program Group' APPLICATION_SHORT_NAME='SQLGL'

Doc ID 2058269.1 is about Transaction Analyzer. But it can serve as a good example on how to use FNDLOAD to register a concurrent program (Doc ID 1587455.1 EBS Payments Funds Disbursement Analyzer does also):
$FND_TOP/bin/FNDLOAD apps/<PWD> 0 Y UPLOAD $FND_TOP/patch/115/import/afcpprog.lct artrxanalzr.ldt CUSTOM_MODE=FORCE UPLOAD_MODE=REPLACE

NOTES additional steps to assign / grant "Transaction Analyzer" to custom responsibility AR_SupperUser:
Security --> Responsibilites --> Define --> "AR_SuperUser" to find it belongs to which "Request Group": AR SuperUser
Security --> Responsibilitie --> Request --> "AR SuperUser" --> add "Transaction Analyzer" to the group.
Now, responsibility AR_SupperUser can submit the new concurrent job "Transaction Analyzer".

Thursday, June 12, 2014

du / find to check space & Linux commands

More OS commands are used as an EBS apps admin than a database DBA. More notes are added to this post after I feel they are useful.

~~~~~~~~~~~~~~~~~~~~~~~ du ~~~~~~~~~~~~~~~~~~~~~~~
Below command will show the disk space used by first level of each folder in RH Linux. It is a good way to find which folder fills up the disk space.

$ du -h --max-depth=1

To get size of current directory:

du -sh *              <== List space used by each file & folder under it
$ du -sh (or, $ du -sk )     <== Get the total space used in current location
du -k .               <== List all details under current location. Not useful.

But, file size reported by "du -sh filename.txt" is much less than the size from "ls -al filename.txt". The difference is big for a large file. A good thing is both "du --apparent-size -m filename.txt" and "ls -lh filename.txt" agree on file size on my RHEL servers.   

~~~~~~~~~~~~~~~~~~~~~ find big files ~~~~~~~~~~~~~~~~~
Find the big files under a directory in Linux, which is useful when disk space is filling up:

$ find /home -size +1000000 -exec ls -alh {} \;

For example
$ find /u01/app -type f -size +1000000 -exec ls -alh {} \;
-rw-rw-r-- 1 ebs oracle 953M Aug  9 2011 /path/to/ftp/out/gl_journal_to_adam.txt
-rw-r----- 1 ebs oracle 1.8G Sep 27 2013  /path/to/ora/10.1.3/Apache/ssl_engine_log
-rw-r--r-- 1 ebs oracle 8.3G Mar 30 13:24   /path/to/ora/10.1.3/j2ee/oacore/oacore_default_group_1/application.log

You can search for exact file size, or just for bigger (+) or smaller (–) files. For example all bigger than 512k files would be found with command:

$ find /home -type f -size +512k -exec ls -lh {} \;

units:
    b – for 512-byte blocks (this is the default if no suffix is used)
    c – for bytes
    w – for two-byte words
    k – for Kilobytes (units of 1024 bytes)
    M – for Megabytes (units of 1048576 bytes)
    G – for Gigabytes (units of 1073741824 bytes)
~~~~~~~~~~~~~~~~~~~~ find the total files ~~~~~~~~~~~~~~~~~~~
Find the total number of files under a directory:

$ find folder_name -type f | wc -l
$ find folder_name -maxdepth 1 -type f | wc -l    <== to exclude subdirectories

~~~~~~~~~~~~~~~~~~~~ find then sort ~~~~~~~~~~~~~~~~~~~~~~
Find and sort by timestamp:

$ find /path/.  -name '*.*' -printf "%T+\t%s\t%p\n" | sort | more

%T+ - modification time (and date)
 %s - size
 %p - path of file

~~~~~~~~~~~~~~~~~~~~ find then replace ~~~~~~~~~~~~~~~~~~~
Use below line to replace wrong file owner under a directory:
# find /u01 -user wrongUserID -print | wc -l
find /u01 -user wrongUserID -exec chown rightUserID:userGroup {} \;

To find/list all files under current directory tree that is not owned by you (applMgr):
$ find . ! -user applMgr
find . ! -user applMgr -exec ls -altr {} \;

~~~~~~~~~~~~~~~~~~~ find then delete ~~~~~~~~~~~~~~~~~~~~
Use find to delete files:
One day, CM log file and Output file can not be opened by browser with error "Authentication failed."  I found the disk space was full with 11GB of Apache folder. Use "find" to delete old files (-r will also delete sub-folders):

$ cd $LOG_HOME/ora/10.1.3/Apache
$ find *.* -type f -mtime +7 -exec rm -rf {} \; 

NOTES: If there is a huge pile of files, above command may get error. Try a different way by placing * inside the single quotes so that it is used as a matching wildcard and not for shell filename expansion:

$ find . -name '*.tmp' | wc -l
$ find . -name '*.tmp' -mtime +7 -exec rm -f {} \;

Below first line worked for me (in cron also) to delete files from a folder with more than 15K files. But the 2nd line failed.
$ /usr/bin/find /u01/path/utl_dir/. -name '*.tmp' -mtime +60 -exec rm -f {} \;

$ /usr/bin/find /u01/path/utl_dir/*.tmp -mtime +60 -exec rm -f {} \;
-ksh: /usr/bin/find: /usr/bin/find: cannot execute [Argument list too long]

~~~~~~~~~~~~~~~~~~~~~ zip ~~~~~~~~~~~~~~~~~~~~~~~~~~
To Zip up a folder, including all subdirectories:
$ cd /path/to/parent_folderName     <= go to the upper folder first
$ zip -r file_name.zip folderName

this is useful when use "admrgpch -s source_path -d target_path -manifest manifest.txt" to merge patches and then copy the merged folder to different node.

~~~~~~~~~~~~~~~~~~ grep / egrep ~~~~~~~~~~~~~~~~~~~~~~~
To find multiple words in a text file:

$ /bin/grep -i -e 'drop ' -e 'alter ' -e 'grant ' -e 'commit ' ascii_file.sql

$ egrep -i 'fail|error' long_file_name.log

~~~~~~~~~~~~~~~~~~~ lsof ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
lsof  to find open files.  it is useful when the system acts strangely

$ lsof /home | wc -l
$ lsof | grep deleted | wc -l   (files in being deleted queue)

~~~~~~~~~~~~~~~~~~ port ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To find what is using a port in Linux:

Below line will tell if a port is used or not on the server:

$ netstat -tuanp | grep 6230 
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp    0 0 167.69.109.82:6230 0.0.0.0:*   LISTEN                 31876/opmn

Then the process owner (or root) can use below line to see more details on using the port: 

$ lsof -i :6230 
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
opmn 31876 ebsdev 9u IPv4 42886593 0t0 TCP server3d.domain.com:6230 (LISTEN)

Below line may also works (Note: "lsof" may show result only to root or process owner):

$ ls -l /proc/31876/exe        <= use the PID from above line
lrwxrwxrwx 1 ebiz user 0 Mar 5 11:19 /proc/31876/exe -> /u05/app/EBSDEV/apps/tech_st/10.1.3/opmn/bin/opmn

NOTES: even a port is free and available on a server, it can be blocked by firewall and such to prevent access from other servers.
~~~~~~~~~~~~~~~~~ screen ~~~~~~~~~~~~~~~~~~~~~~~
First of all, make sure "screen" is installed (in /usr/bin). To check its version, type:
$ screen -version

To start a "screen" session for a long-run job:
$ screen
$ adop phase=fs_clone
... ...

To list running screen/background session:
$ screen -list   (Or, $ screen -l)
There are screens on:
1653.pts-5.server2d   (Attached)
23515.pts-3.server2d  (Attached)
2 Sockets in /var/run/screen/S-userID.

To re-attach a session:
$ screen -r -d 23515
Session ID: 16
Node: server2d
Phase: fs_clone
Phase: fs_clone
Log: $ADOP_LOG_HOME/16/xxxxxx_201956/adop.log
... ...
~~~~~~~~~~~~~~~~~  rsync ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To copy a folder to another folder on the SAME server and preserve exactly all attributes. For example, in EBS R12.2, to synchronize custom folder xxap from PATCH file system to RUN file system:

$ rsync -a --delete $PATCH_BASE/EBSapps/appl/xxap $RUN_BASE/EBSapps/appl

Notes:  add   rsync -a --delete %s_current_base%/EBSapps/appl/xxap %s_other_base%/EBSapps/appl   to file $APPL_TOP_NE/ad/custom/adop_sync.drv
If contents of a folder are static, add   rsync -zr %s_current_base%/EBSapps/appl/xxe4a %s_other_base%/EBSapps/appl   to file $APPL_TOP_NE/ad/custom/adop_sync.drv

The option -a is very powerful, it will copy the directory and all its sub-directories recursively, and preserve symbolic links, modification times, group, ownership, and permissions.
-a option is a combination of a bunch of other options:
    -r  Recurse into directories  
    -l  Copy symlinks as symlinks
    -p  Preserve file permissions
    -t  Preserve modification times
    -g  Preserve group
    -o  Preserve owner
    -D  Preserve device and special files
"-- delete" will DELETE files in Target folder but not in Source folder.

Below line will copy the folder structure with same permissions to current location, without copying files:
$ cd /path/to/target
$ rsync -av -f"+ */" -f"- *" /path1/to/source .

Copy a folder to a different server (and preserve exactly all attributes):

$ cd /to/path/source_FOLDER
$ rsync --progress -avze ssh applmgr@nodeName:/to/path/target_FOLDER

~~~~~~~~~~~~~~~~~~ Stickybit ~~~~~~~~~~~~~~~~~~~~~
If your ID owns /path/to/LOG directory and you want to allow users in other Groups to rename the files in that directory, you have to remove Stickybit on LOG directory (by chmod -t) if it is enabled, because enabled Stickybit makes other users unable to rename/delete the files under LOG directory.

$ ls -al /path/to/LOG
drwxrwxrwt 10 user  group    134 Aug 14 09:57

$ chmod -t /path/to/LOG    # remove Stickybit
$ ls -al /path/to/LOG
drwxrwxrwx 10 user  group    134 Aug 14 09:57

Right permission is needed for others to write/modify your file. To get 664 permissions (-rw-rw-r--) on all files created by your ID, add one line to .bashrc or .kshrc or .profile of your OS ID, or add the line to the shell script for generating the file: 

set umask to 0002  (or 002 ?)
or 
umask u=rwx,g=rwx,o=rx

~~~~~~~~~~~~~~~~~ chmod -R ~~~~~~~~~~~~~~~~~~~~
Run "chmod -R" to change permission on a folder and all subfolders & files under it. For example, to make a folder and all subfolders & all files under it "readable": 
$ chmod -R +r /path/to/LOG

~~~~~~~~~~~~~~~~~ list folders ~~~~~~~~~~~~~~~~~
- To list all folders (sub-directories called "fs_clone")  under a location:
$ find . -name fs_clone -type d -print

- Get the list of directories. Below were tested in K Shell
$ echo $SHELL
/bin/ksh
1.
$ find /u01/app/patches/* -prune -type d   (Or go to the directory first, $ find * -prune -type d )
/u01/app/patches/11902981
/u01/app/patches/13006289
/u01/app/patches/16367827

Note: if just want the list of files, use "find * -prune -type f"

2. -- List directories older than two days
$ find * -prune -type d -mtime +2

3.
$ ls -l |egrep "^drw" |awk '{print $9}'
-- Loop through the list to do whatever you want
$ for x in `ls -l |egrep "^drw" |awk '{print $9}'`
do
echo $x;
done

4.
-- Note: FS is used to define the delimiter in AWK
The default is defined by OS env variable $FS
$ ls -1d */ | awk 'BEGIN {FS="/"};{print $1}'