adoacorectl.sh: exiting with status 150

when the services are not brought down gracefully, or shutdown scripts got errors:

$ADMIN_SCRIPTS_HOME/adoafmctl.sh stop
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adoafmctl.sh version 120.8
Stopping OPMN managed OAFM OC4J instance ...
****************************************************

Executing service control script:
$ADMIN_SCRIPTS_HOME/adformsctl.sh stop
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh  version 120.16.12010000.3
Stopping OPMN managed FORMS OC4J instance  ...
****************************************************

Executing service control script:
$ADMIN_SCRIPTS_HOME/adoacorectl.sh stop
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adoacorectl.sh version 120.13
Stopping OPMN managed OACORE OC4J instance ...

then in that case the lock file or other files will be still present and they will create problem when we try to bring up the environment. For example

$ADMIN_SCRIPTS_HOME/adoacorectl.sh start
script returned:
adoacorectl.sh: exiting with status 150
or
adoacorectl.sh: exiting with status 204

  - check startup log at $LOG_HOME/appl/admin/log/ adoacorectl.txt, adoafmctl.txt, adoaformsctl.txt
  - check log at $LOG_HOME/ora/10.1.3/opmn/opmn.log
  - check Apache logs. Location is "$LOG_HOME/ora/10.1.3/Apache". The latest ones error_log and access_log.

Fix:

stop all R12 processes

rm -rf $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/persistence/*
rm -rf $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/persistence/*
rm -rf $ORA_CONFIG_HOME/10.1.3/j2ee/forms/persistence/*

start all service again

Use TAR to backup a folder

Use right options of TAR to backup entire folder EBSDEV under /u01/app to a different location in Linux. It also shows all errors/warnings during the file copy.

$ which tar
/bin/tar
$ cd /u01/app       NOTE: do not have /u01/app in front of folder name EBSDEV below
$ tar -zcvf /u04/app/backup/m3d_DEV_022015_mt.tar.gz EBSDEV >> /u04/app/ebsde v_backup/backup022015.log

To avoid interruption and keep tar running on the background, use nohup in Linux:

$ nohup tar -zcvf /u04/app/backup/m3d_DEV_022015_mt.tar.gz EBSDEV &

It will create log file nohup.out in current directory and keep grabbing everything thru folder EBSDEV. Use ps to check the status.

$ ps -ef | grep backup

Potential errors:
tar: EBSDEV/apps/apps_st/appl/aagl/bin/.sasa.swp: Cannot open: Permission denied
tar: EBSDEV/apps/apps_st/appl/docs/install/.vi_v_pkb.sql.swp: Cannot open: Permission denied
tar: EBSDEV/apps/apps_st/appl/aafs/bin/fssd01_old: Cannot stat: No such file or directory
tar: EBSDEV/apps/apps_st/appl/aafs/bin/text.txt: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors

To get detail on above errors, use "ls -al" to find that /u01/app/EBSDEV/apps/apps_st/appl/aagl/bin/.sasa.swp is owned by a different OS user, and find file link is missing /u01/app/EBSDEV/apps/apps_st/appl/aafs/bin/fssd01_old -> /bin/fndcpesr, etc.

NOTES:

1. If "-h" is used (e.g. tar -zhcvf  ...), it ditches the symlinks and copies/brings in the actual files they pointed to. Without '-h", it retains the soft links.
2. It can back up multiple folders to one file. It is useful in EBS R12.2.

$ cd /u04/app/EBSDEV

$ tar -zcvf /u04/app/backup/m3d_DEV_beforeSSL.tar.gz  fs1 fs2 fs_ne EBSapps.env >> /u04/app/backup/backup1005.log

To decompress/untar/restore backed-up folder EBSDEV in file /path/to/m3d_DEV_022015_mt.tar.gz (or copy it to a new location first):

$ mkdir temp
$ cd temp
$ tar vzxfp /path/to/m3d_DEV_022015_mt.tar.gz
It will create folder EBSDEV under current folder /temp/

The options stand for verbose, extract, pass thru gzip first, specify file, preserve permissions. "p" is important when trying to restore a directory.

Now, "find EBSDEV -type f | wc -l" shall match the counts on source and target folders. With "-type f", find does not count symlinks (symbolic link).

BTW, zip also can be used to backup a (smaller) folder:

$ zip -r flie_name.zip folder_name
"-r" is to get all subfolders.

gzip can be used to compress a big file (but it is slow):

$ ls -al java*.*

-rw-------  1 user group 5842435627 Jan 18 12:07 java_pid1583.hpro

$ gzip java_pid1583.hprof

 $ ls -al java*.*

-rw-------  1 user group 697002093 Jan 18 12:07 java_pid1583.hprof.gz

$ which gzip

/usr/bin/gzip

Environment file settings in Oracle EBS R12

If APPL_TOP = /u01/apps/apps_st/appl, the main env file that shall be included in .profile (on Linux server) is
. /u01/apps/apps_st/appl/APPS< CONTEXT_NAME >.env

It calls three files:

. $APPL_TOP/custom< CONTEXT_NAME >.env (if there is one)
. $ORACLE_CONFIG_HOME/< CONTEXT_NAME >.env (or $INST_TOP/ora/10.1.2/< CONTEXT_NAME >.env)
. $APPL_TOP/< CONTEXT_NAME >.env

In database server, there is a < CONTEXT_NAME >.env under $ORACLE_HOME.

Area env files:

adovars_< CONTEXT_NAME >.env file
The adovars.env file, located in $APPL_TOP/admin, specifies the location of various files such as
Java files, HTML files, and JRE (Java Runtime Environment) files. The adovars.env file includes commentson the purpose and recommended setting of each variable. In a R12 environment, this file is maintained by AutoConfig, and should not be edited manually. It is called from the main
applications environment file, $APPL_TOP/< CONTEXT_NAME >.env

fndenv.env file
Located in the FND_TOP directory, this file sets additional environment variables used by the Application Object Library. For example, it sets APPLBIN as the name of the subdirectory where product executabl eprograms and shell scripts are stored (bin). This file is called by $APPL_TOP/< CONTEXT_NAME >.env and is maintained by AutoConfig. But, custom env variables can be put in this file by using below format to keep custom variables untouched by AutoConfig:

#Begin Customizations
CUST_TOP=/u01/apps/apps_st/appl/cust; export CUST_TOP
#End Customizations

devenv.env file
This file sets variables for linking third-party software and custom-developed applications with Oracle E-Business Suite, and for allowing you to compile and link custom OracleForms user exits and concurrent programs with Oracle EBS. This script is located in $FND_TOP/usrxit, and is automatically called by fndenv.env.

Other files:

adconfig.txt file
This file stores configuration information when Oracle E-Business Suite is installed. It is for AD utilities to run successfully, and is in the $APPL_TOP/admin directory. AD utility programs perform a variety of database and file management tasks.

dbc file
There is a .dbc file under $FND_SECURE containing various parameters which are responsible for the connection to the database upon receiving a request from Apache Jserv. AutoConfig generates this file after reading variables from $CONTEXT_FILE. See Doc ID 362851.1

default.env for Form
There is a default.env file under $INST_TOP/ora/10.1.2/forms/server for Form applications. If you have custom-developed forms, you can add lines to this file to tell the location for custom form files:

#Begin Customizations
CUST_TOP=/u01/apps/apps_st/appl/cust
#End Customizations

Note: This file does not define env variables. If the path is wrong in this file, Oracle EBS will give error:
FRM-40010 Cannot read from /....../xxx.fmx

10GiAS home:

To apply patch to or work on 10gAS (10.1.3.X) in EBS R12.1, go to directory $INST_TOP/ora/10.1.3 and source the .env file there. Now, the ORACLE_HOME points to a different location (with 10.1.3 in the path), which holds10gAS components, such as Apache/modplsql, oc4j, etc.

Apache httpd services fail to start

R12 Apache server does not start. But the log file reports exiting status 0, which means normal.

Error
   --> Process (index=1,uid=41488097,pid=14132)
   failed to start a managed process after the maximum retry limit
   Log:
   /u02/app/EBSPROD/inst/apps/<CONTEXT name>/logs/ora/10.1.3/opmn/HTTP_Server~1.log
   10/26/2014-10:04:39 : : adapcctl.sh: existing with status 0
 
HTTP_Server~1.log file does not give any useful information as well:
   14/10/26 10:04:39 Start process
   --------
   /u02/app/EBSPROD/inst/apps/<CONTEXT name>/ora/10.1.3/Apache/Apache/bin/apachectl startssl: execing httpd

After deleted httpd.pid file in $INST_TOP/pids/10.1.3/Apache and added below debug options to httpd.conf file in $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf, "adapcctl.sh start" can start the httpd services. See Doc ID 422419.1 on how to enable debug logs.

LogLevel debug
OraLogMode oracle
OraLogSeverity TRACE:32
OraLogDir /u02/app/EBSPROD/inst/apps/<CONTEXT name>/logs/ora/10.1.3/Apache/oracle
 
But, The real fix seems to delete Apache folder under $LOG_HOME/ora/10.1.3 with all httpd log files, and then re-create it (and sub-folder oracle). I believe the cause was some log file exceeded 2GB limit in size.

Troubleshooting:

Logs for troubleshooting on startup:
$LOG_HOME/ora/10.1.3/Apache
$LOG_HOME/ora/10.1.3/opmn/     <= Check file HTTP_Server~1.log here if Apache does not start & no logs
$LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_1/application.log      <= Notes: For example, this file got 8GB in size due to all 10.1.3.5.0 Container error entries (bug 10126440 ? )
$LOG_HOME/ora/10.1.3/j2ee/forms/forms_default_group_1/application.log

Other files can be checked with egrep -i 'fail|error' to see if any issues, when getting "Page not found":
$LOG_HOME/ora/10.1.3/opmn/oacore_default_group_1/oacorestd.err
$LOG_HOME/appl/rgf/javacache.log   (Java Object Cache (JOC) log, $APPLRGF)

- start APACHE by adapcctl.sh start
Make sure below lines do not give back any error:
./adapcctl.sh status
./adalnctl.sh status
If Apache is up and running with enabled ssl, below URL shall work
https://siteName.domian.com:ssl_port
siteName.domain.com can be replaced by its DNS ip address for troubleshooting (with possible "Certificate error" because the certs file is issued for the original site URL).
By this time, the re-direction to login page will not work because it needs oacore process running.

- start OACORE process (in 10.1.3 Oracle_Home) by adoacorectl.sh start
If the login page is unavailable immediately, check the opmn log first
$LOG_HOME/ora/10.1.3/opmn/default_group~oacore~default_group~1.log
If the login page gets internal errors or hangs, but below URLs work (in R12):
https://ip_address:ssl_port/OA_HTML/ServletPing
https://ip_address:ssl_port/OA_HTML/jsp/fnd/aoljtest.jsp   (it may not be configured in some sites)
https://siteName.domian.com:ssl_port/OA_HTML/jsp/fnd/aoljtest.jsp    -- bring up a test HTML form
check log files in $LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_1 to get the real error.

-start FORMS process by adformsctl.sh start
After forms services are running, you shall be able to open forms by JRE popup. If forms does not open, check logs in $LOG_HOME/ora/10.1.3/j2ee/forms/forms_default_group_1 for any error.

- start OAFM OC4J by adoafmctl.sh start

If Apache is running on a server with IP 188.xx.67.77, TELNET on port s_webport (8020) returns below message in my R12.1 environment:
$ telnet 188.xx.67.77 8020
Trying...
Connected to 188.xx.67.77.
Escape character is '^]'.
GET
  <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <HTML><HEAD>
  <TITLE>400 Bad Request</TITLE>
  </HEAD><BODY>
  <H1>Bad Request</H1>
  Your browser sent a request that this server could not understand.<P>
  invalid request-URI <P>
  </BODY></HTML>
  Connection closed.
But "$ telnet 188.xx.67.77 4463" and GET return only "Connection closed." (for unknown reason). 4463 is the ssl_prot in the ssl-enabled configuration.

Most of times, Apache works fine. The problem may come from company Security policy change, fire wall or F5 network settings. All you need is to provide evidence as proof.

NOTE1: after using "adapcctl.sh start" to start Apache services, do not use "adapcctl.sh stop" to stop httpd services, which may leave file httpd.pid on the file system and leave opmn processes still running. Instead, Use "adopmnctl.sh stopall" to stop opmn processes and httpd processes.

NOTE2: All Apaches logs and opmn logs  (or log directories) can be deleted safely, including
    $LOG_HOME/appl/admin/log/adapcctl.text
    $LOG_HOME/ora/10.1.3/opmn/HTTP_Server~1.log
See Doc ID 1964851.1 to clean the logs and enable debug.

NOTE3: If a port defined for Apache is not available, Apache will not start. One day, "adstrtal.sh" gave a similar message above. After added debug options to Apache conf file, I found the problem "make_sock: could not bind to port 4482". Usually profile option APPS_SERVLET_AGENT has the site URL (name and port).

Trun on trace in EBS forms and Forms Node

EBS R12.1 has different ways to trace a user's session. Usually, apps tracing will help address issues on performance.

1. Trace a concurrent program with database .trc file

1) As System Administrator, go to Concurrent-> programs -> Define, and query program PROGRAM_NAME ie ARRGTA, or RAXTRX

2) Enable (or make sure) the check box called 'Enable Trace' and save

3) Before starting trace, make sure the following DB parameters are set:

MAX_DUMP_FILE_SIZE = UNLIMITED
TIMED_STATISTICS = TRUE
STATISTICS_LEVEL = ALL

If they are not set as above, run the following as SYS:

SQL> alter system set MAX_DUMP_FILE_SIZE=UNLIMITED;
SQL> alter system set TIMED_STATISTICS=TRUE;
SQL> alter system set STATISTICS_LEVEL=ALL;

4) Ask the user to rerun the report reproducing the issue

5) After the process finishes, run tkprof for the trace file with sort=exeela,prsela,fchela and
explain=<apps owner>/<apps password>. Example:
tkprof <sourcefile> <outputfile> sys=no explain=apps/<pw> sort=exeela,prsela,fchela

Please make sure the process already finished. If the process has not finished the rows column in tkprof will be all zeros and useless. Be sure to use the TKPROF under the RDBMS Oracle Home /bin directory. The TKPROF under the Applications Oracle Home will not yield accurate results.

2.  Oracle Support steps to trace just one session (when trace can not be turned on on a program) using SQL code.

1) Make sure trace is NOT enabled for any of the concurrent programs we will be running:
System Administrator > Concurrent > Programs > Define
Query the program ARRGTA
Make sure trace is DISABLED (It is important trace is disabled for the program so it does not interfere with the rest of the steps)

2) Create the following Procedure as database user APPS:

CREATE OR REPLACE PROCEDURE cp_level12_trace_util AS
req_id number:=fnd_global.conc_request_id;
prog_id number:=fnd_global.conc_program_id;
appl_id number:=fnd_global.RESP_APPL_ID;
prog_name varchar2(128);
trc_label varchar2(128);

BEGIN

if req_id > 0 then
select max(concurrent_program_name) into prog_name
from fnd_concurrent_programs
where concurrent_program_id=prog_id
and application_id=appl_id;

trc_label:='CP_'||prog_name||'_REQ_'||req_id;

FND_CTL.FND_SESS_CTL(''
,''
,''
,'TRUE'
,''
,'ALTER SESSION SET MAX_DUMP_FILE_SIZE=UNLIMITED');

FND_CTL.FND_SESS_CTL(''
,''
,''
,'TRUE'
,''
,'ALTER SESSION SET TIMED_STATISTICS=TRUE');

FND_CTL.FND_SESS_CTL(''
,''
,''
,'TRUE'
,''
,'ALTER SESSION SET STATISTICS_LEVEL=ALL');


FND_CTL.FND_SESS_CTL(''
, ''
, ''
, 'TRUE'
, ''
, 'ALTER SESSION SET TRACEFILE_IDENTIFIER='||''''
|| trc_label
|| ''''
|| ' EVENTS ='
|| ''''
|| ' 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12 '
|| '''');

end if;
END;
/

3) Set the following profile option at user level, only for the user who reproduces the issue:

Profile option: Initialization SQL Statement - Custom
Value: begin cp_level12_trace_util; end;

NOTES: Be careful, as sometimes it may block user from login, and also see FND_INIT_SQL may cause error.

4) Run the concurrent request by the user, and upload all trace files, which have the request_id in their name (It will have something like CP_PROGNAME_REQ_999999, where PROGNAME is the short name of the program, and 999999 is the request id). If Trace files are too big, do not upload them for now

5) Run tkprof for the above trace files, and upload the output:
tkprof <trace file> <output file> sort=exeela,prsela,fchela explain=apps/<password>

Make sure the process already finished (if it hangs, please leave it running for a few hours before canceling). If the process has not finished the rows column in tkprof will be all zeros.

6) Upload also the log files for the concurrent requests

7) Do not forget to set the profile option “Initialization SQL Statement – Custom” back to BLANK afterwards.

3.  Trace EBS Forms session

1) Change Profile option Utilities:Diagnostics to "Yes" at user level
2) Help --> Diagnostics --> Trace --> to choose a tracing level. Then, a popup box will tell where the trace file .trc is saved.

4.  Find which node an EBS R12 session is connecting to

1) Change Profile option FND: Diagnostics to "Yes" at user level
2) Help -> About Oracle Applications (below section will tell which form node you have connected):
----------------------------------------
Forms Server
----------------------------------------
Oracle Forms Version : 10.1.2.3.0
Application Object Library : 12.0.0
Machine : node_name.domain.COM

NOTES:

- Turn "FND: Diagnostics" profile to Yes at user level will also add "About this Page" to EBS Homepage's left bottom after login and add "Diagnostics" on the right top of OA Framework pages. Clean the cache if they do not show up immediately after re-login.
- This profile option is different from turning "Utilities:Diagnostics" to Yes, which enables tracing on sub-menus of Help on GUI forms.
- I found that even "FND: Diagnostics" profile is set to No, "About this Page" may still shows up. That could be controlled by other things, such as other profile options for Personalization:

Personalize Self-Service Defn = YES

FND: Personalization Region Link Enabled = YES

Disable Self-Service Personal = NO (Only at Site)