JSP files for OAF pages

Our developers coded a custom OAF (Oracle Application Framework) apps. After code files were saved to folders under $JAVA_TOP/oracle/apps/custom/Rebate/webui and .xml files were loaded into the database, webpage https://sitename.domain.com/OA_HTML/OA.jsp?page=/oracle/apps/custom/Rebate/webui/RebatePG&language_code=US&.... got error:

Error Page  You have encountered an unexpected error. Please contact the System Administrator for assistance

 I enabled Profile option "FND: Diagnostics" to see more detailed exception.

 Exception Details. 
 oracle.apps.fnd.framework.OAException: oracle.adf.mds.exception.MDSRuntimeException:
 Unable to find component with absolute reference = /oracle/apps/aear/Rebate/webui/SearchPage, XML Path = null.
 Please verify that the reference is valid and the definition of the component exists either on the File System or in the MDS Repository.
    at oracle.apps.fnd.framework.OAException.wrapperException(OAException.java:912)
    ... ... ...

We verified file $JAVA_TOP/oracle/apps/custom/Rebate/webui/SearchPage.xml does exist on in the OS folder, and then we ran a short script to upload it to the database

APPS_USER_PARAM=$1
APPS_PASS_PARAM=$2
HOST_NAME_PARAM=$3
DB_SID_PARAM=$4
DB_PORT_PARAM=$5
LOG_CONTROL=$LOG_TOP/log/SearchPage`date '+%Y%m%d%H%M'`.log  
... ... ....
 java oracle.jrad.tools.xml.importer.XMLImporter $JAVA_TOP/oracle/apps/custom/Rebate/webui/SearchPage.xml \
-username $APPS_USER_PARAM -password $APPS_PASS_PARAM -rootdir . \
-dbconnection "(DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=$HOST_NAME_PARAM)(PORT=$DB_PORT_PARAM)) (CONNECT_DATA= (SID=$DB_SID_PARAM)))"

After the xml is imported successfully, below code should display its definition stored in MDS repository.

SQL> set serveroutput on
SQL> Begin
SQL> apps.jdr_utils.printdocument('/oracle/apps/custom/Rebate/webui/SearchPage');
SQL> End;

Now, after we click on the OAF page again, it works! The problem was fixed.

Additional notes:

1. all .java files in a folder can be compiled to .class file by javac. For example,

$ cd $JAVA_TOP/oracle/apps/custom/Rebate/webui
$ javac *.java

2. For JSP apps, Java .class file is not loaded into the database. All compiled JSP files are stored in $COMMON/_pages. In R12, the jsp files does not get compiled automatically. If OC4J doesn’t find the .class file in the _pages directory, it will just display a blank webpage and will not even attempt to compile the JSP. This is different from 11i. So, do not modify or delete files in folder $COMMON/_pages in R12 if you do not know how to compile JSP files.

3. The Perl script to compile JSP file is $FND_TOP/patch/115/bin/ojspCompile.pl. Run below line will clear cache and force compile all jsp pages

 $ perl -x $FND_TOP/patch/115/bin/ojspCompile.pl --compile --flush

Good documents on compiling JSP files:
JSP Pages Hanging in R12 After Removing Cached Class Files in _pages (Doc ID 433386.1)
How to Enable Automatic Compilation of JSP pages in R12 Environment (Doc ID 458338.1)

UPDATES:  How jsp file works in R12.2.10?

For R12.2, JSP file is saved in $OA_HTML (same as $EBS_APPS_DEPLOYMENT_DIR/oacore/html). They are not uploaded to database. You can use same command to compile a single custom .jsp file, such as 

$ cd $OA_HTML

$ more JDKtest.jsp

The JDK version is: <%= System.getProperty("java.version") %>

$ perl -x $FND_TOP/patch/115/bin/ojspCompile.pl --compile -s 'JDKtest.jsp' --flush 

logfile set: $LOG_HOME/appl/rgf/ojsp/ojspc_error.log

starting...(compiling delta)

using 10i internal ojsp ver: 10.3.6.0

synchronizing dependency file:

  loading deplist...7874

  enumerating jsps...7875

  updating dependency...7875

  parsing jsp...7875

  writing deplist...7875

initializing compilation:

  files to compile...1

  eliminating children...1 (-0)

  searching uncompiled...1

translating and compiling:

  searching untranslated...1

  translating jsps...1/1 in 23s

  compiling jsps...1/1 in 3s

Finished!

After compilation, file jdktest.class is saved in $OA_HTML/WEB-INF/classes/_pages. But it may give error on page https://[web node]:[port]/OA_HTML/JDKtest.jsp :

Requested resource or page is not allowed in this site

This feature is controlled by new profile options in R12.2. To resolve the issue, change "Security: Allowed Resources" to ALL (and may also change "Allow Unrestricted JSP Access [FND_SEC_ALLOW_JSP_UNRESTRICTED_ACCESS] to Yes).  

Then, bounce all Apps services. Now page http://[web node]:[port]/OA_HTML/JDKtest.jsp shall say "The JDK version is: 1.7.0_xxx".

s_oacore_nprocs and s_forms_nprocs

An EBS R12.1.1 instance had a lot of database sessions kept in INACTIVE for a long time (15+ days) and never got closed. So, the number of db sessions increased daily and easily exceeded database parameter PROCESSES (which is set to 3000). I had to recycle Apps services each two weeks to kill the idle DB sessions. Most idle sessions were from "e::bes:oracle.apps.icx.security.session.created" by JDBC Thin Client. Modules AR and GL are used most in this instance.

After the instance was upgraded to R12.1.3 and its database was upgraded to 12c, the idle session problem went away surprisingly. Before that, I tried below setting parameters and did NOT get much luck.

Two documents were used to tune CONTEXT variables:
- How To Prevent Inactive JDBC Connections In Oracle Applications 11i and R12 ( Doc ID 427759.1 )
- JVM: Guidelines to setup the Java Virtual Machine in Apps Ebusiness Suite 11i and R12 ( Doc ID 362851.1 )

The number of jvms (oc4j in R12) is configured by the autoconfig variables s_oacore_nprocs, s_forms_nprocs, s_disco_nprocs, and s_xmlsvcs_nprocs. They can be updated in the $CONTEXT_FILE (using the autoconfig editor from OAM).

The document suggests use 1 JVM per CPU for 100 active connected users to OACoreGroup. Use the script to determine "active users" for OACoreGroup :

REM
REM SQL to count number of Apps 11i users
REM Run as APPS user
REM
 select 'Number of user sessions : ' || count( distinct session_id) How_many_user_sessions
  from icx_sessions icx
where disabled_flag != 'Y' and PSEUDO_FLAG = 'N'
and (last_connect + decode(FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT'), NULL,limit_time, 0,limit_time, FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT')/60)/24) > sysdate and counter < limit_connects;
REM
REM END OF SQL
REM

I got "Number of user sessions : 372" in my instance. So, I set s_oacore_nprocs to 4 to handle the volume ( assuming that 372 is peak level), and set s_forms_nprocs to 4 as well.

s_oacore_nprocs will decide how the number of sub-folders under $LOG_HOME/ora/10.1.3/j2ee/oacore

I set Heap configuration (4 parameters forms_jvm_start_options, oacore_jvm_start_options, forms_jvm_stop_options, oacore_jvm_stop_options in $CONTEXT_FILE) to
-Xmx1024M -Xms512M -XX:MaxPermSize=256M -XX:NewRatio=2 -XX:+PrintGCTimeStamps

I also added the following parameter to the DBC file:
JDBC\:oracle.jdbc.maxCachedBufferSize=262144

Changed the DBC file settings for dbc file under $FND_SECURE directory:
FND_JDBC_BUFFER_DECAY_INTERVAL=120
FND_JDBC_BUFFER_MIN=0
FND_JDBC_BUFFER_MAX=0
FND_MAX_JDBC_CONNECTIONS=256
FND_JDBC_USABLE_CHECK=true
FND_JDBC_BUFFER_DECAY_SIZE=5

But all above changes did not help much :(

UPDATE in November 2016:
After we moved EBS R12.1.3 instance to a new Linux 6 server (from Linux 5), adstrtal.sh got error when starting services.

Executing service control script:
$ADMIN_SCRIPTS_HOME/adoacorectl.sh start
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adoacorectl.sh version 120.13
Starting OPMN managed OACORE OC4J instance  ...
****************************************************

Executing service control script:$ADMIN_SCRIPTS_HOME/adformsctl.sh start
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh  version 120.16.12010000.3
Starting OPMN managed FORMS OC4J instance  ...

But I had no problem in log onto the EBS site and launching Forms.

The error (or warning) could be fixed by increasing CONTEXT FILE parameters for "Timeout specified in context file: 100 second(s)".   Since parameters s_oacore_nprocs=4 and s_forms_nprocs=4 did not help, I just re-set them back to 2 (and fnd_jdbc_usable_check to false) in $CONTEXT_FILE and ran adautocfg.sh. After that, the adstrtal.sh Timed-out error did not show up.

Output page cannot be displayed

Concurrent job completed successfully. But the Output can not be opened due to its size.
Error:  Page Cannot Be Displayed, Cannot View Output of Large concurrent reports

I tried below actions in R12.1, which may help or may not help. Seems to me it also depends on the resources of client PC. VMware mostly has this trouble when the Output file size is huge.

1. Follow Doc ID 845841.1 (without applying patches)

a) Set "ICX: Session Timeout" profile option value (120 minutes)
Note: s_oc4j_sesstimeout shall match this value

b) Set "OC4J Session Timeout" in orion-web.xml
Note: two orion-web.xml files. One for OC4J, and one for OAFM (which may not be important to the time-out)
$ORA_CONFIG_HOME/10.1.3/j2ee/oacore/application-deployments/oacore/html/orion-web.xml
$ORA_CONFIG_HOME/10.1.3/j2ee/oafm/application-deployments/oafm/webservices/orion-web.xml
increase <session-timeout> from 30 to 120

c) Set Timeout in $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf
Increase Timeout from 300 to 900 (seconds)

2. Doc ID 780081.1 Java Heap Errors when Running An FSG
Section: To Increase Heap Size of oacore process:

In file $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml

<process-type id="oacore" module-id="OC4J" status="enabled" working-dir="$ORACLE_HOME/j2ee/home">
<module-data>
<category id="start-parameters">
   <data id="java-options" value="-server -verbose:gc -Xmx1024M -Xms256M -XX:MaxPermSize=160M

oacore java-options meaning:

-Xmx1024M -- Specifies maximum heap memory.(1024 MB)
-Xms256M -- Initial heap memory.(256 MB)

3.  Bounce the Concurrent Manager, Apache Server (and maybe database listener).

opatch and Oracle Home inventory

There are two inventory locations. One is the central inventory defined by file /etc/oraInst.loc (in Linux host). For each ORACLE HOME on the server, there is a local inventory at $ORACLE_HOME/inventory.

opatch lists the information first when applying a patch:

Oracle Home                : /path/to/apps/tech_st/10.1.2
Oracle Home Inventory  : $ORACLE_HOME/inventory
Central Inventory           : /path/to/oraEbsInventory
   from                          : /etc/oraInst.loc

Because the central inventory /path/to/oraEbsInventory may be shared by multiple ORACLE HOMEs on the server, it is more vulnerable to error or corruption. If opatch throws out some error, troubleshoot central inventory first.

If there is a problem with local inventory, it may turn out you have to re-install it. So, do not try to change anything without backing it up.

When I apply a small patch to 10.1.2 ORACLE HOME, it failed with below messages in log file $ORACLE_HOME/.patch_storage/8551790/xxx.log. "opatch lsinventory" also did not show the patch was applied.

Patch 8551790 has been applied successfully
... ... ...

verifying patch
  verifying that patch ID is in the Oracle Home inventory
INVENTORY PROBLEM: Patch 8551790 is not present in Oracle Home inventory.
  Verifying copy files.
Comparing "/path/to/8551790/files/procbuilder/lib/cus_procbuilder.mk" and "$ORACLE_HOME/procbuilder/lib/cus_procbuilder.mk"
... ... ...
OPATCH_JAVA_ERROR: Patch was not successfully applied.

Verification of the patch failed.
ERROR: OPatch failed as verification of the patch failed.

The real message is "INVENTORY PROBLEM ...".  Doc ID 438067.1 (Updating the Inventory Is Failing During Patching) provides a solution to it.

Fix:
I renamed the existing Contents folder and created an empty folder Contents under oraEbsInventory. After I ran "opatch apply" again, the issue was fixed.

The same fix may address error "Inventory check failed: Patch ID is NOT registered in Oracle Home inventory." or "Verification of patch failed: Patch is not found in the Inventory." as well.

That Oracle document also says "opatch lsinventory -detail" accesses the inventory stored in text format in central inventory while "opatch lsinventory" gets inventory info from binary inventory (under oraEbsInventory/Contents ?).

R12.1 log locations and log collection

To troubleshoot a technology stack (Web issue or Forms error),  Oracle Support asks a lot of logs, before they take a look on the service request (SR) or even they may not know what they are doing.  Recently I followed Doc ID 1942889.1 (SRDC - Data Collection Request for EBS: IAS) to collect diagnostic data/logs.

1. follow Doc ID 2026081.1 to run 5 sql scripts to generate 5 .txt files and save them to /tmp.

2. run below line to zip all needed files
$ more zip1942889_1.sh
zip -r /tmp/$TWO_TASK'_'`uname -n`_`date +%m%d%y`_Techstack_info.zip \
/tmp/apps_version_info.txt \
/tmp/fndnodes_oss.txt \
/tmp/web_urls.txt \
/tmp/atg_ver_patches.txt \
/tmp/login_profiles.txt \
$OA_HTML/bin/appsweb* \
$CONTEXT_FILE \
$FND_SECURE/* \
$ORA_CONFIG_HOME/10.1.3/Apache/* \
$ORA_CONFIG_HOME/10.1.3/j2ee/* \
$ORA_CONFIG_HOME/10.1.3/javacache/* \
$ORA_CONFIG_HOME/10.1.3/opmn/*

3. run below line from Doc  (Doc ID 601736.1 to get techstack Component Versions (Forms, Http Server, JDK, Framework)

$ cd $FND_TOP/patch/115/bin   
$ $ADPERLPRG $FND_TOP/patch/115/bin/TXKScript.pl \
-script=$FND_TOP/patch/115/bin/txkInventory.pl -txktop=$APPLTMP \
-contextfile=$CONTEXT_FILE \
-appspass=appsPWD \
-outfile=/tmp/Report_App_Inventory.html
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** STDOUT   = /$LOG_HOME/appl/rgf/TXK/txkInventory_Thu_Jan_14_10_21_05_2016_stdout.log
Reportfile /tmp/Report_App_Inventory.html generated successfully.

Summary on R12.1 log locations. Note env variable $LOG_HOME and $INST_TOP/logs are the same.

-----------------------------------------------------------------------------
-- Startup/Shutdown services (by $ADMIN_SCRIPTS_HOME scripts) log files:

$LOG_HOME/appl/admin/log

----------------------------------------------------------------------------
-- Apache, OC4J and OPMN (see Doc ID 454178.1): runtime

$LOG_HOME/ora/10.1.3/Apache

$LOG_HOME/ora/10.1.3/opmn
$LOG_HOME/ora/10.1.3/j2ee (each sub-folder /forms and /oacore has files application.log and server.log)

 $LOG_HOME/ora/10.1.3/opmn/opmn.log
 $LOG_HOME/ora/10.1.3/opmn/oacore_default_group_X/oacorestd.err   (X depends on s_oacore_nprocs)
 $LOG_HOME/ora/10.1.3/opmn/default_group~oacore~default_group~X.log
 $LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_X/application.log
 $LOG_HOME/ora/10.1.3/j2ee/forms/forms_default_group_X/application.log    X depends on s_forms_nprocs

$APPLRGF/javacache.log

- user's DB connection log (and trace if enabled)
 $LOG_HOME/ora/10.1.2/network

--------------------------------------------------------------------------
-- Concurrent program log & out

$APPLCSF/log
$APPLCSF/out
($INST_TOP/logs/appl/conc/log  <== default)

SQL> SELECT logfile_name, outfile_name, outfile_node_name, last_update_date
FROM apps.FND_CONCURRENT_REQUESTS
WHERE REQUEST_ID = &requestID;

- temporary directories
 $APPLPTMP  
 $APPLTMP     <== temporary files for EBS forms.
 $INST_TOP/logs/ora/10.1.2/forms  ($FORMS_TRACE_DR)  <== holds forms runtime files
- Forms Runtime Diagnostics (FRD) tracing AND logging files (Doc ID 438652.1)

----------------------------------------------------------------------------
-- Autoconfig log file:
--
Apps:
$INST_TOP/admin/log/MMDDHHMM/adconfig.log
($INST_TOP/admin/out/MMDDHHMM has run-time files)

DB tier:
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/<MMDDHHMM>/

----------------------------------------------------------------------------
-- adadmin, adpatch and adctrl

$APPL_TOP/admin/${TWO_TASK}/log

----------------------------------------------------------------------------
-- Patching related log files

i)   $APPL_TOP/admin/${TWO_TASK}/log/ <== Application Tier -- adpatch log
ii)  $ORACLE_HOME/.patch_storage            <== Forms (Forms & Reports 10.1.2/Developer) patch
iii) $IAS_ORACLE_HOME/.patch_storage    <== Apache (Web Server) patch

----------------------------------------------------------------------------
-- Clone logs:

Pre-clone log files in source instance

Apps: => $INST_TOP/admin/log/StageAppsTier_MMDDHHMM.log
DB:    => $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/StageDBTier_MMDDHHMM.log

Clone log files in target instance

Apps : => $INST_TOP/admin/log/ApplyAppsTier_<time>.log
DB:     => $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTier_<time>.log

---------------------------------------------------------------------------
-- Output Post Processor (OPP) log from the Application

Getting OPP Log from the application itself
a. System Administrator > Concurrent > Manager > Administer
b. Search for 'Output Post Processor'
c. Click the 'Processes' button
d. Click the Manager Log button. This will open the 'OPP'