How to turn on Debug log on user's session

During troubleshooting on data issue, use below steps in Doc ID 1320736.1 to turn on FND debug log on user's forms session or concurrent requests.

Steps for On-Line processes
  1) From a System Administrator responsibility
       Navigation path: Profile -> System
           a) Set the following profile option at SITE level:
                  FND: Diagnostics -> Yes
           b) Set the following profile options at USER level:
                  FND: Debug Log Module -> %
                  FND: Debug Log Enabled -> Yes
                  FND: Debug Log Level -> Statement

  2) Run the following query to get the starting log sequence(seq1)
           --- ---
           SELECT MAX(log_sequence)
           FROM FND_LOG_MESSAGES;
           --- ---
  3) Go to the Payables (or any) responsibility
       Open the involved Forms or run the concurrent job and reproduce the error.

  4) Run the following query again to get the finishing log sequence(seq2)
           --- ---
           SELECT MAX(log_sequence)
           FROM FND_LOG_MESSAGES;
           --- ---
  5) Run the following query to generate the log file:
           --- ---
           SELECT module, message_text
           FROM fnd_log_messages
           WHERE log_sequence between &seq1 and &seq2
           ORDER BY log_sequence;
           --- ---
  6) Provide the results in an EXCEL spreadsheet with column headings for readability.

  Notes: If table fnd_log_messages becomes huge, it's safe to just truncate it.
              SQL> truncate table APPLSYS.FND_LOG_MESSAGES;

adoacorectl.sh: exiting with status 150

when the services are not brought down gracefully, or shutdown scripts got errors:

$ADMIN_SCRIPTS_HOME/adoafmctl.sh stop
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adoafmctl.sh version 120.8
Stopping OPMN managed OAFM OC4J instance ...
****************************************************

Executing service control script:
$ADMIN_SCRIPTS_HOME/adformsctl.sh stop
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh  version 120.16.12010000.3
Stopping OPMN managed FORMS OC4J instance  ...
****************************************************

Executing service control script:
$ADMIN_SCRIPTS_HOME/adoacorectl.sh stop
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adoacorectl.sh version 120.13
Stopping OPMN managed OACORE OC4J instance ...

then in that case the lock file or other files will be still present and they will create problem when we try to bring up the environment. For example

$ADMIN_SCRIPTS_HOME/adoacorectl.sh start
script returned:
adoacorectl.sh: exiting with status 150
or
adoacorectl.sh: exiting with status 204

  - check startup log at $LOG_HOME/appl/admin/log/ adoacorectl.txt, adoafmctl.txt, adoaformsctl.txt
  - check log at $LOG_HOME/ora/10.1.3/opmn/opmn.log
  - check Apache logs. Location is "$LOG_HOME/ora/10.1.3/Apache". The latest ones error_log and access_log.

Fix:

stop all R12 processes

rm -rf $ORA_CONFIG_HOME/10.1.3/j2ee/oacore/persistence/*
rm -rf $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/persistence/*
rm -rf $ORA_CONFIG_HOME/10.1.3/j2ee/forms/persistence/*

start all service again

Use TAR to backup a folder

Use right options of TAR to backup entire folder EBSDEV under /u01/app to a different location in Linux. It also shows all errors/warnings during the file copy.

$ which tar
/bin/tar
$ cd /u01/app       NOTE: do not have /u01/app in front of folder name EBSDEV below
$ tar -zcvf /u04/app/backup/m3d_DEV_022015_mt.tar.gz EBSDEV >> /u04/app/ebsde v_backup/backup022015.log

To avoid interruption and keep tar running on the background, use nohup in Linux:

$ nohup tar -zcvf /u04/app/backup/m3d_DEV_022015_mt.tar.gz EBSDEV &

It will create log file nohup.out in current directory and keep grabbing everything thru folder EBSDEV. Use ps to check the status.

$ ps -ef | grep backup

Potential errors:
tar: EBSDEV/apps/apps_st/appl/aagl/bin/.sasa.swp: Cannot open: Permission denied
tar: EBSDEV/apps/apps_st/appl/docs/install/.vi_v_pkb.sql.swp: Cannot open: Permission denied
tar: EBSDEV/apps/apps_st/appl/aafs/bin/fssd01_old: Cannot stat: No such file or directory
tar: EBSDEV/apps/apps_st/appl/aafs/bin/text.txt: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors

To get detail on above errors, use "ls -al" to find that /u01/app/EBSDEV/apps/apps_st/appl/aagl/bin/.sasa.swp is owned by a different OS user, and find file link is missing /u01/app/EBSDEV/apps/apps_st/appl/aafs/bin/fssd01_old -> /bin/fndcpesr, etc.

NOTES:

1. If "-h" is used (e.g. tar -zhcvf  ...), it ditches the symlinks and copies/brings in the actual files they pointed to. Without '-h", it retains the soft links.
2. It can back up multiple folders to one file. It is useful in EBS R12.2.

$ cd /u04/app/EBSDEV

$ tar -zcvf /u04/app/backup/m3d_DEV_beforeSSL.tar.gz  fs1 fs2 fs_ne EBSapps.env >> /u04/app/backup/backup1005.log

To decompress/untar/restore backed-up folder EBSDEV in file /path/to/m3d_DEV_022015_mt.tar.gz (or copy it to a new location first):

$ mkdir temp
$ cd temp
$ tar vzxfp /path/to/m3d_DEV_022015_mt.tar.gz
It will create folder EBSDEV under current folder /temp/

The options stand for verbose, extract, pass thru gzip first, specify file, preserve permissions. "p" is important when trying to restore a directory.

Now, "find EBSDEV -type f | wc -l" shall match the counts on source and target folders. With "-type f", find does not count symlinks (symbolic link).

BTW, zip also can be used to backup a (smaller) folder:

$ zip -r flie_name.zip folder_name
"-r" is to get all subfolders.

gzip can be used to compress a big file (but it is slow):

$ ls -al java*.*

-rw-------  1 user group 5842435627 Jan 18 12:07 java_pid1583.hpro

$ gzip java_pid1583.hprof

 $ ls -al java*.*

-rw-------  1 user group 697002093 Jan 18 12:07 java_pid1583.hprof.gz

$ which gzip

/usr/bin/gzip

Environment file settings in Oracle EBS R12

If APPL_TOP = /u01/apps/apps_st/appl, the main env file that shall be included in .profile (on Linux server) is
. /u01/apps/apps_st/appl/APPS< CONTEXT_NAME >.env

It calls three files:

. $APPL_TOP/custom< CONTEXT_NAME >.env (if there is one)
. $ORACLE_CONFIG_HOME/< CONTEXT_NAME >.env (or $INST_TOP/ora/10.1.2/< CONTEXT_NAME >.env)
. $APPL_TOP/< CONTEXT_NAME >.env

In database server, there is a < CONTEXT_NAME >.env under $ORACLE_HOME.

Area env files:

adovars_< CONTEXT_NAME >.env file
The adovars.env file, located in $APPL_TOP/admin, specifies the location of various files such as
Java files, HTML files, and JRE (Java Runtime Environment) files. The adovars.env file includes commentson the purpose and recommended setting of each variable. In a R12 environment, this file is maintained by AutoConfig, and should not be edited manually. It is called from the main
applications environment file, $APPL_TOP/< CONTEXT_NAME >.env

fndenv.env file
Located in the FND_TOP directory, this file sets additional environment variables used by the Application Object Library. For example, it sets APPLBIN as the name of the subdirectory where product executabl eprograms and shell scripts are stored (bin). This file is called by $APPL_TOP/< CONTEXT_NAME >.env and is maintained by AutoConfig. But, custom env variables can be put in this file by using below format to keep custom variables untouched by AutoConfig:

#Begin Customizations
CUST_TOP=/u01/apps/apps_st/appl/cust; export CUST_TOP
#End Customizations

devenv.env file
This file sets variables for linking third-party software and custom-developed applications with Oracle E-Business Suite, and for allowing you to compile and link custom OracleForms user exits and concurrent programs with Oracle EBS. This script is located in $FND_TOP/usrxit, and is automatically called by fndenv.env.

Other files:

adconfig.txt file
This file stores configuration information when Oracle E-Business Suite is installed. It is for AD utilities to run successfully, and is in the $APPL_TOP/admin directory. AD utility programs perform a variety of database and file management tasks.

dbc file
There is a .dbc file under $FND_SECURE containing various parameters which are responsible for the connection to the database upon receiving a request from Apache Jserv. AutoConfig generates this file after reading variables from $CONTEXT_FILE. See Doc ID 362851.1

default.env for Form
There is a default.env file under $INST_TOP/ora/10.1.2/forms/server for Form applications. If you have custom-developed forms, you can add lines to this file to tell the location for custom form files:

#Begin Customizations
CUST_TOP=/u01/apps/apps_st/appl/cust
#End Customizations

Note: This file does not define env variables. If the path is wrong in this file, Oracle EBS will give error:
FRM-40010 Cannot read from /....../xxx.fmx

10GiAS home:

To apply patch to or work on 10gAS (10.1.3.X) in EBS R12.1, go to directory $INST_TOP/ora/10.1.3 and source the .env file there. Now, the ORACLE_HOME points to a different location (with 10.1.3 in the path), which holds10gAS components, such as Apache/modplsql, oc4j, etc.

Apache httpd services fail to start

R12 Apache server does not start. But the log file reports exiting status 0, which means normal.

Error
   --> Process (index=1,uid=41488097,pid=14132)
   failed to start a managed process after the maximum retry limit
   Log:
   /u02/app/EBSPROD/inst/apps/<CONTEXT name>/logs/ora/10.1.3/opmn/HTTP_Server~1.log
   10/26/2014-10:04:39 : : adapcctl.sh: existing with status 0
 
HTTP_Server~1.log file does not give any useful information as well:
   14/10/26 10:04:39 Start process
   --------
   /u02/app/EBSPROD/inst/apps/<CONTEXT name>/ora/10.1.3/Apache/Apache/bin/apachectl startssl: execing httpd

After deleted httpd.pid file in $INST_TOP/pids/10.1.3/Apache and added below debug options to httpd.conf file in $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf, "adapcctl.sh start" can start the httpd services. See Doc ID 422419.1 on how to enable debug logs.

LogLevel debug
OraLogMode oracle
OraLogSeverity TRACE:32
OraLogDir /u02/app/EBSPROD/inst/apps/<CONTEXT name>/logs/ora/10.1.3/Apache/oracle
 
But, The real fix seems to delete Apache folder under $LOG_HOME/ora/10.1.3 with all httpd log files, and then re-create it (and sub-folder oracle). I believe the cause was some log file exceeded 2GB limit in size.

Troubleshooting:

Logs for troubleshooting on startup:
$LOG_HOME/ora/10.1.3/Apache
$LOG_HOME/ora/10.1.3/opmn/     <= Check file HTTP_Server~1.log here if Apache does not start & no logs
$LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_1/application.log      <= Notes: For example, this file got 8GB in size due to all 10.1.3.5.0 Container error entries (bug 10126440 ? )
$LOG_HOME/ora/10.1.3/j2ee/forms/forms_default_group_1/application.log

Other files can be checked with egrep -i 'fail|error' to see if any issues, when getting "Page not found":
$LOG_HOME/ora/10.1.3/opmn/oacore_default_group_1/oacorestd.err
$LOG_HOME/appl/rgf/javacache.log   (Java Object Cache (JOC) log, $APPLRGF)

- start APACHE by adapcctl.sh start
Make sure below lines do not give back any error:
./adapcctl.sh status
./adalnctl.sh status
If Apache is up and running with enabled ssl, below URL shall work
https://siteName.domian.com:ssl_port
siteName.domain.com can be replaced by its DNS ip address for troubleshooting (with possible "Certificate error" because the certs file is issued for the original site URL).
By this time, the re-direction to login page will not work because it needs oacore process running.

- start OACORE process (in 10.1.3 Oracle_Home) by adoacorectl.sh start
If the login page is unavailable immediately, check the opmn log first
$LOG_HOME/ora/10.1.3/opmn/default_group~oacore~default_group~1.log
If the login page gets internal errors or hangs, but below URLs work (in R12):
https://ip_address:ssl_port/OA_HTML/ServletPing
https://ip_address:ssl_port/OA_HTML/jsp/fnd/aoljtest.jsp   (it may not be configured in some sites)
https://siteName.domian.com:ssl_port/OA_HTML/jsp/fnd/aoljtest.jsp    -- bring up a test HTML form
check log files in $LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_1 to get the real error.

-start FORMS process by adformsctl.sh start
After forms services are running, you shall be able to open forms by JRE popup. If forms does not open, check logs in $LOG_HOME/ora/10.1.3/j2ee/forms/forms_default_group_1 for any error.

- start OAFM OC4J by adoafmctl.sh start

If Apache is running on a server with IP 188.xx.67.77, TELNET on port s_webport (8020) returns below message in my R12.1 environment:
$ telnet 188.xx.67.77 8020
Trying...
Connected to 188.xx.67.77.
Escape character is '^]'.
GET
  <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <HTML><HEAD>
  <TITLE>400 Bad Request</TITLE>
  </HEAD><BODY>
  <H1>Bad Request</H1>
  Your browser sent a request that this server could not understand.<P>
  invalid request-URI <P>
  </BODY></HTML>
  Connection closed.
But "$ telnet 188.xx.67.77 4463" and GET return only "Connection closed." (for unknown reason). 4463 is the ssl_prot in the ssl-enabled configuration.

Most of times, Apache works fine. The problem may come from company Security policy change, fire wall or F5 network settings. All you need is to provide evidence as proof.

NOTE1: after using "adapcctl.sh start" to start Apache services, do not use "adapcctl.sh stop" to stop httpd services, which may leave file httpd.pid on the file system and leave opmn processes still running. Instead, Use "adopmnctl.sh stopall" to stop opmn processes and httpd processes.

NOTE2: All Apaches logs and opmn logs  (or log directories) can be deleted safely, including
    $LOG_HOME/appl/admin/log/adapcctl.text
    $LOG_HOME/ora/10.1.3/opmn/HTTP_Server~1.log
See Doc ID 1964851.1 to clean the logs and enable debug.

NOTE3: If a port defined for Apache is not available, Apache will not start. One day, "adstrtal.sh" gave a similar message above. After added debug options to Apache conf file, I found the problem "make_sock: could not bind to port 4482". Usually profile option APPS_SERVLET_AGENT has the site URL (name and port).