How to setup a custom env variable in R12.2

In R12.2, custom env variables can be defined in file $INST_TOP/appl/admin/custom$CONTEXT_NAME.env (after it is created), which is called by $APPL_TOP/APPS$CONTEXT_NAME.env (and so, by the main env file EBSapps.env).  For example, use two lines in custom$CONTEXT_NAME.env to define a custom variable:

$ cd $INST_TOP/appl/admin

$ more custom$CONTEXT_NAME.env

FORMS_RECORD_GROUP_MAX=30000

export FORMS_RECORD_GROUP_MAX

$ echo $FORMS_RECORD_GROUP_MAX

30000

Another way is to use the "custom" folder under $AD_TOP/admin/template. Oracle Doc ID 745711.1 gives steps on defining a Forms env variable:

a) Go to the autoconfig Template folder:

$ cd $AD_TOP/admin/template

b) Create new directory named (custom)

$ mkdir custom

c) Make sure that new directory has same file permissions as ($AD_TOP/admin/template)

d) Copy the following autoconfig template to the new custom directory:

$ cp $AD_TOP/admin/template/APPLSYS_ux.env $AD_TOP/admin/template/custom/APPLSYS_ux.env

e) Edit the file copied file under custom directory and add the following 2 lines at the end of section:

####################################

# Oracle Forms environment variables

####################################

FORMS_RECORD_GROUP_MAX=30000

export FORMS_RECORD_GROUP_MAX

f) Save and exit from the file.

g) Next time autoconfig run, it will read the custom directory and check for any customizations there.

After AutoConfig, above two lines will be copied to file $APPL_TOP/$CONTEXT_NAME.env (and other AD files?).

Notes: "FORMS_RECORD_GROUP_MAX=30000" can be equivalently added to Forms file $INST_TOP/ora/10.1.2/forms/server/default.env. But that file will be overwritten by AutoConfig. See Oracle Doc ID 745711.1.

EBS session status in the database & on OS

When an OS process uses high CPU or other OS resources, use below line to find its start time:

$  ps -eo pid,lstart,cmd | grep 21846

74063 Tue Dec 19 12:29:31 2022 grep --color=auto 21486

21486 Tue Dec 19 07:39:01 2022 frmweb server webfile=HTTP-0,0,1,default

If a forms process (frmweb) exists on OS level but does not have a responding session in the database, it is a run-away and it can be terminated.  

Below SQL statement will tell and monitor what an EBS session is doing in the database after its process ID is identified (e.g. 21846) on OS level of EBS apps server. 

SQL> SELECT to_char(S.logon_time, 'DD-MON-RRRR HH24:MI:SS') "logon_time", to_char(S.PREV_EXEC_START, 'DD-MON-RRRR HH24:MI:SS') "last op started", 

    client_identifier, module, action, status, machine, round(last_call_et/60,1) "Minutes in current status",

    sql_id, blocking_session, process, sid, serial#,

    STATE,

    ROUND((S.WAIT_TIME_MICRO/1000000)/60/60,1) "total wait hrs",

    DECODE(S.TIME_REMAINING_MICRO,'-1', 'indefinite', '0','Wait timed out',NULL,'session not waiting') "remaining wait time", 

    DECODE(S.TIME_SINCE_LAST_WAIT_MICRO, '0','Still waiting') "current wait status",

    S.time_since_last_wait_micro

   FROM v$session s

WHERE process = '21846' 

-- order by client_identifier

;

  

  BLOCKING_SESSION - Tell if it is blocked by another session.

  STATE - The state of the wait event.

  WAIT_TIME_MICRO - The wait time of the wait event in microseconds.

  TIME_REMAINING_MICRO - The time remaining in microseconds before the wait event times out.

  TIME_SINCE_LAST_WAIT_MICRO - The amount of time in microseconds elapsed from the last wait to the current wait.

Normally, a session for a running concurrent request will show the SQL_ID.

An oacore process may have many sessions in the database. To find which process ID is for oacore, try "$ ps -ef | grep oacore " on OS level of apps server, or run below line in the database (if XX is part of a CUSTOM top name).

SQL> select distinct process, machine from v$session 

where action like '%XX%' order by machine;

When an OACORE process taking high CPU, find the process ID and run following commands :

1) ps -eLo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm | grep PID

==> which thread is taking CPU.  Most of them should show 0.0 (which is the CPU column).

I did see an OACORE process (e.g. 13030) supports 125 JVM threads in one of my instances. So, it is not good to simply kill an oacore process.

$ ps -eLo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm | grep 13030 | wc -l

125

2) jmap -histo <jvm_process_id> > /tmp/ <jvm_process_id>.histo

==> will dump heap usage details.  

3) jmap -dump:format=b,file= <jvm_process_id>.jmap <jvm_process_id>

==> will generate an actual heap dump, binary format

4) kill -3 <jvm_process_id>

==> output goes to 12.2 oacore_server1.out. It will generate stack trace, run twice a minute apart.

kill -3 <jvm_process_id>

5) lsof -p <jvm_process_id> > <jvm_process_id>.log

==> List of open ports/files

6) SELECT audsid, module, last_call_et, action from gv$session where process = '&jvm_process_id';

==> corresponding DB sessions

7) log / incident files from that oacore :

$EBS_DOMAIN_HOME/servers/oa*/logs/*

$EBS_DOMAIN_HOME/servers/oa*/adr/diag/ofm/EBS_domain*/oacore*/incident/incdir*/*

SQL statement to find blocking session

Use SQL to identify and monitor the database session that blocks other session:

SQL> SELECT DECODE(request,0,'Holder: ','Waiter: ') status, sid sess_ID, id1, id2, lmode, request, type

FROM    V$LOCK

WHERE (id1, id2, type) IN

(SELECT id1, id2, type FROM V$LOCK WHERE request > 0)

ORDER BY id1, request;

Use below to confirm the blocking status in the database:

SQL> SELECT blocking_session block_holder,

  sid, serial#, process, machine, client_identifier, wait_class, seconds_in_wait

FROM v$session

WHERE blocking_session IS NOT NULL

ORDER BY blocking_session; 

Purge R12.2 log files in Linux OS

Folder structure in R12.2 file system is more complicated than in R12.1, because RUN and PATCH may switch their location. After an adop cutover, the path to log files will be changed to a different one. Below are folders that have a growing number of log files. "ls" can be used to list the log files but "cd" do not work to reach the folder, while the path uses "*" in it.

$IAS_ORACLE_HOME/instances/*/diagnostics/logs/OHS/*/*.log*

$EBS_DOMAIN_HOME/servers/oacore*/logs/*.log*

$EBS_DOMAIN_HOME/servers/form*/logs/*.log*

$EBS_DOMAIN_HOME/servers/AdminServer/logs/*.log*

Fortunately, "find" works with path having "*" in it. Below lines work in cron to purge/delete old log files (on web/forms node).

$ crontab -l

05 05 * * * . /u07/app/.profile; /usr/bin/find $IAS_ORACLE_HOME/instances/*/diagnostics/logs/OHS/*/*log* -type f -mtime +30 -exec rm -f {} \;

05 15 * * * . /u07/app/.profile; /usr/bin/find $EBS_DOMAIN_HOME/servers/oacore*/logs/*log* -type f -mtime +30 -exec rm -f {} \;

Other logs or their locations:

$ADOP_LOG_HOME/*                   <== each ADOP session ID has a sub-folder

$INST_TOP/admin/log/MMDDHHMM/adconfig.log

$LOG_HOME/appl/admin/log/*      <== adstrtal.sh & adstpall.sh 

$LOG_HOME/appl/rgf/javacache.log 

$IAS_ORACLE_HOME/instances/*/diagnostics/logs/OPMN/opmn/* 

$EBS_DOMAIN_HOME/sysman/log/* 

$EBS_DOMAIN_HOME/servers/oac*/adr/diag/ofm/EBS_domain*/oac*/incident/* 

$EBS_DOMAIN_HOME/servers/forms_s*/adr/diag/ofm/EBS_domain*/forms_s*/incident/*

$INST_TOP/logs/ora/10.1.2/reports/cache/*.*

Locations or files for configuration and setups:

$IAS_ORACLE_HOME/instances/EBS_web*/config/OHS/EBS_web* 

$FMW_HOME/webtier/instances/EBS_web*/config/OPMN/opmn/* 

$CONTEXT_FILE

$INST_TOP/appl/admin/*.properties

$INST_TOP/ora/10.1.2/forms/server/default.env    (AutoConfig overwrites it)

$FND_TOP/fndenv.env

$EBS_DOMAIN_HOME/config/config.xml

$EBS_APPS_DEPLOYMENT_DIR/*/APP-INF/node_info.txt

.profile file (for account runs the crontab jobs on RHEL)

$ more .profile

PATH=/bin:/usr/bin:/usr/local/bin

export PATH

MANPATH=/usr/share/man:/usr/local/share/man:/usr/X11R6/man

export MANPATH

EDITOR=/bin/vi

export EDITOR

. /u07/app/EBSDEV/EBSapps.env RUN

isMaster="no"

if [ ! -z $APPS_VERSION ] && [ ${APPS_VERSION:0:4} == "12.2" ]

then

s_status=`cat $CONTEXT_FILE | grep -i s_adminserverstatus`

isMaster="${s_status:60:7}"

fi

if [ $isMaster == "enabled" ]

then

if [ `echo -n ${HOSTNAME%%.*} | tail -c -1` != "p" ]

then

PS1=$'

\e[0;31m$USER@${HOSTNAME%%.*}[$TWO_TASK]\e[m$PWD

-->$ '

else

PS1=$'

\e[0;31m$USER@${HOSTNAME%%.*}[$TWO_TASK]\e[m\E[32m$PWD \E[0m

-->$ '

fi

else

if [ `echo -n ${HOSTNAME%%.*} | tail -c -1` != "p" ]

then

PS1='

$USER@${HOSTNAME%%.*}[$TWO_TASK]$PWD

-->$ '

else

PS1=$'

$USER@${HOSTNAME%%.*}[$TWO_TASK]\E[32m$PWD \E[0m

-->$ '

fi

fi

alias rm='rm -i'

stty erase ^?