An EBS R12.1.1 instance had a lot of database sessions kept in INACTIVE for a long time (15+ days)
and never got closed. So, the number of db sessions increased daily and easily
exceeded database parameter PROCESSES (which is set to 3000). I had to recycle
Apps services each two weeks to kill the idle DB sessions. Most idle sessions were from
"e::bes:oracle.apps.icx.security.session.created" by JDBC Thin Client. Modules AR and GL are used most in this
instance.
After the instance was upgraded to R12.1.3 and its database was
upgraded to 12c, the idle session problem went away surprisingly. Before that, I tried below setting parameters and did NOT get much luck.
Two documents were used to tune CONTEXT variables:
- How To Prevent Inactive JDBC Connections In Oracle Applications 11i and R12 ( Doc ID 427759.1 )
- JVM: Guidelines to setup the Java Virtual Machine in Apps Ebusiness Suite 11i and R12 ( Doc ID 362851.1 )
The number of jvms (oc4j in R12) is configured by the autoconfig
variables s_oacore_nprocs, s_forms_nprocs, s_disco_nprocs, and
s_xmlsvcs_nprocs. They can be updated in the $CONTEXT_FILE (using the autoconfig editor from OAM).
The document suggests use 1 JVM per CPU for 100 active connected users to OACoreGroup. Use the script to determine "active users" for OACoreGroup :
REM
REM SQL to count number of Apps 11i users
REM Run as APPS user
REM
select 'Number of user sessions : ' || count( distinct session_id)
How_many_user_sessions
from icx_sessions icx
where disabled_flag != 'Y' and PSEUDO_FLAG = 'N'
and (last_connect + decode(FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT'),
NULL,limit_time,
0,limit_time, FND_PROFILE.VALUE('ICX_SESSION_TIMEOUT')/60)/24) >
sysdate and counter < limit_connects;
REM
REM END OF SQL
REM
I got "Number of user sessions : 372" in my instance. So, I set s_oacore_nprocs to 4 to handle the volume ( assuming that 372 is peak level), and set s_forms_nprocs to 4 as well.
s_oacore_nprocs will decide how the number of sub-folders under $LOG_HOME/ora/10.1.3/j2ee/oacore
I set Heap configuration (4 parameters forms_jvm_start_options, oacore_jvm_start_options, forms_jvm_stop_options, oacore_jvm_stop_options in $CONTEXT_FILE) to
-Xmx1024M -Xms512M -XX:MaxPermSize=256M -XX:NewRatio=2 -XX:+PrintGCTimeStamps
I also added the following parameter to the DBC file:
JDBC\:oracle.jdbc.maxCachedBufferSize=262144
Changed the DBC file settings for dbc file under $FND_SECURE directory:
FND_JDBC_BUFFER_DECAY_INTERVAL=120
FND_JDBC_BUFFER_MIN=0
FND_JDBC_BUFFER_MAX=0
FND_MAX_JDBC_CONNECTIONS=256
FND_JDBC_USABLE_CHECK=true
FND_JDBC_BUFFER_DECAY_SIZE=5
But all above changes did not help much :(
UPDATE in November 2016:
After we moved EBS R12.1.3 instance to a new Linux 6 server (from Linux 5), adstrtal.sh got error when starting services.
Executing service control script:
$ADMIN_SCRIPTS_HOME/adoacorectl.sh start
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adoacorectl.sh version 120.13
Starting OPMN managed OACORE OC4J instance ...
****************************************************
Executing service control script:$ADMIN_SCRIPTS_HOME/adformsctl.sh start
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh version 120.16.12010000.3
Starting OPMN managed FORMS OC4J instance ...
But I had no problem in log onto the EBS site and launching Forms.
The error (or warning) could be fixed by increasing CONTEXT FILE parameters for "Timeout specified in context file: 100 second(s)". Since parameters s_oacore_nprocs=4 and s_forms_nprocs=4 did not help, I just re-set them back to 2 (and fnd_jdbc_usable_check to false) in $CONTEXT_FILE and ran adautocfg.sh. After that, the adstrtal.sh Timed-out error did not show up.
Sunday, June 26, 2016
Wednesday, June 15, 2016
Output page cannot be displayed
Concurrent job completed successfully. But the Output can not be opened due to its size.
Error: Page Cannot Be Displayed, Cannot View Output of Large concurrent reports
I tried below actions in R12.1, which may help or may not help. Seems to me it also depends on the resources of client PC. VMware mostly has this trouble when the Output file size is huge.
1. Follow Doc ID 845841.1 (without applying patches)
a) Set "ICX: Session Timeout" profile option value (120 minutes)
Note: s_oc4j_sesstimeout shall match this value
b) Set "OC4J Session Timeout" in orion-web.xml
Note: two orion-web.xml files. One for OC4J, and one for OAFM (which may not be important to the time-out)
$ORA_CONFIG_HOME/10.1.3/j2ee/oacore/application-deployments/oacore/html/orion-web.xml
$ORA_CONFIG_HOME/10.1.3/j2ee/oafm/application-deployments/oafm/webservices/orion-web.xml
increase <session-timeout> from 30 to 120
c) Set Timeout in $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf
Increase Timeout from 300 to 900 (seconds)
2. Doc ID 780081.1 Java Heap Errors when Running An FSG
Section: To Increase Heap Size of oacore process:
In file $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml
<process-type id="oacore" module-id="OC4J" status="enabled" working-dir="$ORACLE_HOME/j2ee/home">
<module-data>
<category id="start-parameters">
<data id="java-options" value="-server -verbose:gc -Xmx1024M -Xms256M -XX:MaxPermSize=160M
oacore java-options meaning:
-Xmx1024M -- Specifies maximum heap memory.(1024 MB)
-Xms256M -- Initial heap memory.(256 MB)
3. Bounce the Concurrent Manager, Apache Server (and maybe database listener).
Error: Page Cannot Be Displayed, Cannot View Output of Large concurrent reports
I tried below actions in R12.1, which may help or may not help. Seems to me it also depends on the resources of client PC. VMware mostly has this trouble when the Output file size is huge.
1. Follow Doc ID 845841.1 (without applying patches)
a) Set "ICX: Session Timeout" profile option value (120 minutes)
Note: s_oc4j_sesstimeout shall match this value
b) Set "OC4J Session Timeout" in orion-web.xml
Note: two orion-web.xml files. One for OC4J, and one for OAFM (which may not be important to the time-out)
$ORA_CONFIG_HOME/10.1.3/j2ee/oacore/application-deployments/oacore/html/orion-web.xml
$ORA_CONFIG_HOME/10.1.3/j2ee/oafm/application-deployments/oafm/webservices/orion-web.xml
increase <session-timeout> from 30 to 120
c) Set Timeout in $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf
Increase Timeout from 300 to 900 (seconds)
2. Doc ID 780081.1 Java Heap Errors when Running An FSG
Section: To Increase Heap Size of oacore process:
In file $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml
<process-type id="oacore" module-id="OC4J" status="enabled" working-dir="$ORACLE_HOME/j2ee/home">
<module-data>
<category id="start-parameters">
<data id="java-options" value="-server -verbose:gc -Xmx1024M -Xms256M -XX:MaxPermSize=160M
oacore java-options meaning:
-Xmx1024M -- Specifies maximum heap memory.(1024 MB)
-Xms256M -- Initial heap memory.(256 MB)
3. Bounce the Concurrent Manager, Apache Server (and maybe database listener).
Wednesday, May 25, 2016
opatch and Oracle Home inventory
There are two inventory locations. One is the central inventory defined by file /etc/oraInst.loc (in Linux host). For each ORACLE HOME on the server, there is a local inventory at $ORACLE_HOME/inventory.
opatch lists the information first when applying a patch:
Oracle Home : /path/to/apps/tech_st/10.1.2
Oracle Home Inventory : $ORACLE_HOME/inventory
Central Inventory : /path/to/oraEbsInventory
from : /etc/oraInst.loc
Because the central inventory /path/to/oraEbsInventory may be shared by multiple ORACLE HOMEs on the server, it is more vulnerable to error or corruption. If opatch throws out some error, troubleshoot central inventory first.
If there is a problem with local inventory, it may turn out you have to re-install it. So, do not try to change anything without backing it up.
When I apply a small patch to 10.1.2 ORACLE HOME, it failed with below messages in log file $ORACLE_HOME/.patch_storage/8551790/xxx.log. "opatch lsinventory" also did not show the patch was applied.
Patch 8551790 has been applied successfully
... ... ...
verifying patch
verifying that patch ID is in the Oracle Home inventory
INVENTORY PROBLEM: Patch 8551790 is not present in Oracle Home inventory.
Verifying copy files.
Comparing "/path/to/8551790/files/procbuilder/lib/cus_procbuilder.mk" and "$ORACLE_HOME/procbuilder/lib/cus_procbuilder.mk"
... ... ...
OPATCH_JAVA_ERROR: Patch was not successfully applied.
Verification of the patch failed.
ERROR: OPatch failed as verification of the patch failed.
The real message is "INVENTORY PROBLEM ...". Doc ID 438067.1 (Updating the Inventory Is Failing During Patching) provides a solution to it.
Fix:
I renamed the existing Contents folder and created an empty folder Contents under oraEbsInventory. After I ran "opatch apply" again, the issue was fixed.
The same fix may address error "Inventory check failed: Patch ID is NOT registered in Oracle Home inventory." or "Verification of patch failed: Patch is not found in the Inventory." as well.
That Oracle document also says "opatch lsinventory -detail" accesses the inventory stored in text format in central inventory while "opatch lsinventory" gets inventory info from binary inventory (under oraEbsInventory/Contents ?).
opatch lists the information first when applying a patch:
Oracle Home : /path/to/apps/tech_st/10.1.2
Oracle Home Inventory : $ORACLE_HOME/inventory
Central Inventory : /path/to/oraEbsInventory
from : /etc/oraInst.loc
Because the central inventory /path/to/oraEbsInventory may be shared by multiple ORACLE HOMEs on the server, it is more vulnerable to error or corruption. If opatch throws out some error, troubleshoot central inventory first.
If there is a problem with local inventory, it may turn out you have to re-install it. So, do not try to change anything without backing it up.
When I apply a small patch to 10.1.2 ORACLE HOME, it failed with below messages in log file $ORACLE_HOME/.patch_storage/8551790/xxx.log. "opatch lsinventory" also did not show the patch was applied.
Patch 8551790 has been applied successfully
... ... ...
verifying patch
verifying that patch ID is in the Oracle Home inventory
INVENTORY PROBLEM: Patch 8551790 is not present in Oracle Home inventory.
Verifying copy files.
Comparing "/path/to/8551790/files/procbuilder/lib/cus_procbuilder.mk" and "$ORACLE_HOME/procbuilder/lib/cus_procbuilder.mk"
... ... ...
OPATCH_JAVA_ERROR: Patch was not successfully applied.
Verification of the patch failed.
ERROR: OPatch failed as verification of the patch failed.
The real message is "INVENTORY PROBLEM ...". Doc ID 438067.1 (Updating the Inventory Is Failing During Patching) provides a solution to it.
Fix:
I renamed the existing Contents folder and created an empty folder Contents under oraEbsInventory. After I ran "opatch apply" again, the issue was fixed.
The same fix may address error "Inventory check failed: Patch ID is NOT registered in Oracle Home inventory." or "Verification of patch failed: Patch is not found in the Inventory." as well.
That Oracle document also says "opatch lsinventory -detail" accesses the inventory stored in text format in central inventory while "opatch lsinventory" gets inventory info from binary inventory (under oraEbsInventory/Contents ?).
Sunday, May 22, 2016
R12.1 log locations and log collection
To troubleshoot a technology stack (Web issue or Forms error), Oracle Support asks a lot of logs, before they take a look on the service request (SR) or even they may not know what they are doing. Recently I followed Doc ID 1942889.1 (SRDC - Data Collection Request for EBS: IAS) to collect diagnostic data/logs.
1. follow Doc ID 2026081.1 to run 5 sql scripts to generate 5 .txt files and save them to /tmp.
2. run below line to zip all needed files
$ more zip1942889_1.sh
zip -r /tmp/$TWO_TASK'_'`uname -n`_`date +%m%d%y`_Techstack_info.zip \
/tmp/apps_version_info.txt \
/tmp/fndnodes_oss.txt \
/tmp/web_urls.txt \
/tmp/atg_ver_patches.txt \
/tmp/login_profiles.txt \
$OA_HTML/bin/appsweb* \
$CONTEXT_FILE \
$FND_SECURE/* \
$ORA_CONFIG_HOME/10.1.3/Apache/* \
$ORA_CONFIG_HOME/10.1.3/j2ee/* \
$ORA_CONFIG_HOME/10.1.3/javacache/* \
$ORA_CONFIG_HOME/10.1.3/opmn/*
3. run below line from Doc (Doc ID 601736.1 to get techstack Component Versions (Forms, Http Server, JDK, Framework)
$ cd $FND_TOP/patch/115/bin
$ $ADPERLPRG $FND_TOP/patch/115/bin/TXKScript.pl \
-script=$FND_TOP/patch/115/bin/txkInventory.pl -txktop=$APPLTMP \
-contextfile=$CONTEXT_FILE \
-appspass=appsPWD \
-outfile=/tmp/Report_App_Inventory.html
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** STDOUT = /$LOG_HOME/appl/rgf/TXK/txkInventory_Thu_Jan_14_10_21_05_2016_stdout.log
Reportfile /tmp/Report_App_Inventory.html generated successfully.
Summary on R12.1 log locations. Note env variable $LOG_HOME and $INST_TOP/logs are the same.
-----------------------------------------------------------------------------
-- Startup/Shutdown services (by $ADMIN_SCRIPTS_HOME scripts) log files:
$LOG_HOME/appl/admin/log
----------------------------------------------------------------------------
-- Apache, OC4J and OPMN (see Doc ID 454178.1): runtime
$LOG_HOME/ora/10.1.3/Apache
$LOG_HOME/ora/10.1.3/opmn
$LOG_HOME/ora/10.1.3/j2ee (each sub-folder /forms and /oacore has files application.log and server.log)
$LOG_HOME/ora/10.1.3/opmn/opmn.log
$LOG_HOME/ora/10.1.3/opmn/oacore_default_group_X/oacorestd.err (X depends on s_oacore_nprocs)
$LOG_HOME/ora/10.1.3/opmn/default_group~oacore~default_group~X.log
$LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_X/application.log
$LOG_HOME/ora/10.1.3/j2ee/forms/forms_default_group_X/application.log X depends on s_forms_nprocs
$APPLRGF/javacache.log
- user's DB connection log (and trace if enabled)
$LOG_HOME/ora/10.1.2/network
--------------------------------------------------------------------------
-- Concurrent program log & out
$APPLCSF/log
$APPLCSF/out
($INST_TOP/logs/appl/conc/log <== default)
SQL> SELECT logfile_name, outfile_name, outfile_node_name, last_update_date
FROM apps.FND_CONCURRENT_REQUESTS
WHERE REQUEST_ID = &requestID;
- temporary directories
$APPLPTMP
$APPLTMP <== temporary files for EBS forms.
$INST_TOP/logs/ora/10.1.2/forms ($FORMS_TRACE_DR) <== holds forms runtime files
- Forms Runtime Diagnostics (FRD) tracing AND logging files (Doc ID 438652.1)
----------------------------------------------------------------------------
-- Autoconfig log file:
--
Apps:
$INST_TOP/admin/log/MMDDHHMM/adconfig.log
($INST_TOP/admin/out/MMDDHHMM has run-time files)
DB tier:
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/<MMDDHHMM>/
----------------------------------------------------------------------------
-- adadmin, adpatch and adctrl
$APPL_TOP/admin/${TWO_TASK}/log
----------------------------------------------------------------------------
-- Patching related log files
i) $APPL_TOP/admin/${TWO_TASK}/log/ <== Application Tier -- adpatch log
ii) $ORACLE_HOME/.patch_storage <== Forms (Forms & Reports 10.1.2/Developer) patch
iii) $IAS_ORACLE_HOME/.patch_storage <== Apache (Web Server) patch
----------------------------------------------------------------------------
-- Clone logs:
Pre-clone log files in source instance
Apps: => $INST_TOP/admin/log/StageAppsTier_MMDDHHMM.log
DB: => $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/StageDBTier_MMDDHHMM.log
Clone log files in target instance
Apps : => $INST_TOP/admin/log/ApplyAppsTier_<time>.log
DB: => $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTier_<time>.log
---------------------------------------------------------------------------
-- Output Post Processor (OPP) log from the Application
Getting OPP Log from the application itself
a. System Administrator > Concurrent > Manager > Administer
b. Search for 'Output Post Processor'
c. Click the 'Processes' button
d. Click the Manager Log button. This will open the 'OPP'
1. follow Doc ID 2026081.1 to run 5 sql scripts to generate 5 .txt files and save them to /tmp.
2. run below line to zip all needed files
$ more zip1942889_1.sh
zip -r /tmp/$TWO_TASK'_'`uname -n`_`date +%m%d%y`_Techstack_info.zip \
/tmp/apps_version_info.txt \
/tmp/fndnodes_oss.txt \
/tmp/web_urls.txt \
/tmp/atg_ver_patches.txt \
/tmp/login_profiles.txt \
$OA_HTML/bin/appsweb* \
$CONTEXT_FILE \
$FND_SECURE/* \
$ORA_CONFIG_HOME/10.1.3/Apache/* \
$ORA_CONFIG_HOME/10.1.3/j2ee/* \
$ORA_CONFIG_HOME/10.1.3/javacache/* \
$ORA_CONFIG_HOME/10.1.3/opmn/*
3. run below line from Doc (Doc ID 601736.1 to get techstack Component Versions (Forms, Http Server, JDK, Framework)
$ cd $FND_TOP/patch/115/bin
$ $ADPERLPRG $FND_TOP/patch/115/bin/TXKScript.pl \
-script=$FND_TOP/patch/115/bin/txkInventory.pl -txktop=$APPLTMP \
-contextfile=$CONTEXT_FILE \
-appspass=appsPWD \
-outfile=/tmp/Report_App_Inventory.html
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** STDOUT = /$LOG_HOME/appl/rgf/TXK/txkInventory_Thu_Jan_14_10_21_05_2016_stdout.log
Reportfile /tmp/Report_App_Inventory.html generated successfully.
Summary on R12.1 log locations. Note env variable $LOG_HOME and $INST_TOP/logs are the same.
-----------------------------------------------------------------------------
-- Startup/Shutdown services (by $ADMIN_SCRIPTS_HOME scripts) log files:
$LOG_HOME/appl/admin/log
----------------------------------------------------------------------------
-- Apache, OC4J and OPMN (see Doc ID 454178.1): runtime
$LOG_HOME/ora/10.1.3/Apache
$LOG_HOME/ora/10.1.3/opmn
$LOG_HOME/ora/10.1.3/j2ee (each sub-folder /forms and /oacore has files application.log and server.log)
$LOG_HOME/ora/10.1.3/opmn/opmn.log
$LOG_HOME/ora/10.1.3/opmn/oacore_default_group_X/oacorestd.err (X depends on s_oacore_nprocs)
$LOG_HOME/ora/10.1.3/opmn/default_group~oacore~default_group~X.log
$LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_X/application.log
$LOG_HOME/ora/10.1.3/j2ee/forms/forms_default_group_X/application.log X depends on s_forms_nprocs
$APPLRGF/javacache.log
- user's DB connection log (and trace if enabled)
$LOG_HOME/ora/10.1.2/network
--------------------------------------------------------------------------
-- Concurrent program log & out
$APPLCSF/log
$APPLCSF/out
($INST_TOP/logs/appl/conc/log <== default)
SQL> SELECT logfile_name, outfile_name, outfile_node_name, last_update_date
FROM apps.FND_CONCURRENT_REQUESTS
WHERE REQUEST_ID = &requestID;
- temporary directories
$APPLPTMP
$APPLTMP <== temporary files for EBS forms.
$INST_TOP/logs/ora/10.1.2/forms ($FORMS_TRACE_DR) <== holds forms runtime files
- Forms Runtime Diagnostics (FRD) tracing AND logging files (Doc ID 438652.1)
----------------------------------------------------------------------------
-- Autoconfig log file:
--
Apps:
$INST_TOP/admin/log/MMDDHHMM/adconfig.log
($INST_TOP/admin/out/MMDDHHMM has run-time files)
DB tier:
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/<MMDDHHMM>/
----------------------------------------------------------------------------
-- adadmin, adpatch and adctrl
$APPL_TOP/admin/${TWO_TASK}/log
----------------------------------------------------------------------------
-- Patching related log files
i) $APPL_TOP/admin/${TWO_TASK}/log/ <== Application Tier -- adpatch log
ii) $ORACLE_HOME/.patch_storage <== Forms (Forms & Reports 10.1.2/Developer) patch
iii) $IAS_ORACLE_HOME/.patch_storage <== Apache (Web Server) patch
----------------------------------------------------------------------------
-- Clone logs:
Pre-clone log files in source instance
Apps: => $INST_TOP/admin/log/StageAppsTier_MMDDHHMM.log
DB: => $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/StageDBTier_MMDDHHMM.log
Clone log files in target instance
Apps : => $INST_TOP/admin/log/ApplyAppsTier_<time>.log
DB: => $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTier_<time>.log
---------------------------------------------------------------------------
-- Output Post Processor (OPP) log from the Application
Getting OPP Log from the application itself
a. System Administrator > Concurrent > Manager > Administer
b. Search for 'Output Post Processor'
c. Click the 'Processes' button
d. Click the Manager Log button. This will open the 'OPP'
Saturday, April 23, 2016
Manually deploy EAR file
After a patch (such as JE patch 21950151) is applied or adadmin generates new JAR (Java Archive) file, the apps startup script adformsctl.sh may deploy forms EAR file. You may see a Warning (with status 0) in log $LOG_HOME/appl/admin/log/adstrtal.log.
Starting OPMN managed FORMS OC4J instance ...
Calling txkChkFormsDeployment.pl to check whether latest FORMSAPP.EAR is deployed...
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl started @ Mon Apr 18 1
6:58:03 2016
*** Log File = $LOG_HOME/appl/rgf/TXK/txkChkFormsDeployment_Mon_Apr_18_16_58_02_2016/txkChkFormsDeployment_Mon_Apr_18_16_58_02_2016.log
File "$IAS_ORACLE_HOME/j2ee/forms/applications/forms/formsweb/WEB-INF/lib/frmsrv.jar" exists. Proceeding to check the size...
=================================================
*** Latest formsapp.ear has NOT been deployed ***
Deploying the latest EAR file...
=================================================
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl completed @ Mon Apr 18 16:59:04 2016
*****************************************
* W A R N I N G *
*****************************************
Error while executing the perl script txkChkFormsDeployment.pl
We have determined that you need to redeploy Forms (using txkChkFormsDeployment.pl).
But could not do it automatically for you, due to some issues.
For details refer the log files.
Follow Note: 397174.1 to redeploy Forms manually.
*****************************************
adformsctl.sh: exiting with status 0
I followed steps in Doc ID 397174.1 to get over this. Without this steps, some forms may not open up.
1) back up file $INST_TOP/ora/10.1.3/j2ee/forms/config/system-jazn-data.xml (for deploying the formsapp.ear file). Note this file's timestamp always gets updated.
2) when the oc4jadmin Password is unknown, search keyword oc4jadmin for pattern in file system-jazn-data.xml , and replace line
<description>OC4J Administrator</description>
<credentials>{903}9VrhYTuhd7DyBJf7J/4KwbSEIlETQJOO</credentials>
with
<description>OC4J Administrator</description>
<credentials>!welcome123</credentials>
Optional: after above change and bounce apps services, below line can verify the password is correct :
$ $AF_JRE_TOP/bin/java -jar $IAS_ORACLE_HOME/j2ee/home/admin_client.jar deployer:oc4j:opmn://hostname.domain.com:6530/forms oc4jadmin welcome123 -validateURI
URI deployer:oc4j:opmn://hostname.domain.com:6530/forms is valid and connected
-- where to get the port number
$ grep 6530 $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml
<port local="6130" remote="6230" request="6530"/>
-- one more validation
$ cd $ORA_CONFIG_HOME/10.1.3/opmn/bin
$ ./opmnctl validate
opmnctl: opmn validation succeeded.
3) stop apps services, and run the Perl script on each Forms tier to deploy the EAR file
$ $FND_TOP/bin/txkrun.pl -script=CfgOC4JApp
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = $LOG_HOME/appl/rgf/TXK/txkCfgOC4JApp_Mon_Apr_18_17_17_52_2016.log
Program : $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl started @ Mon Apr 18 17:17:52 2016
*** Log File = $LOG_HOME/appl/rgf/TXK/txkCfgOC4JApp_Mon_Apr_18_17_17_52_2016.log
Enter Application name for re-deployment ? forms
Enter Oc4j Instance password for re-deployment ? welcome123
Run Autoconfig <Yes/No> ? No
*****************************************************
Required values for starting OC4J instance "forms":
====================================================
s_formsstatus = enabled
s_forms_nprocs = 1 (value should be greater than 0)
Existing values from the context file:
======================================
s_formsstatus = enabled
s_forms_nprocs = 2
----------------------------------------------
*** Values for context variables are VALID ***
----------------------------------------------
*****************************************************
Stopping all OPMN processes.
OPMN stopped.
OPMN started.
Deplolying Application : "forms" onto OC4J instance: "forms"
Application deployed successfully.
Stopping and starting OC4J instances.
Started OC4J instances.
Binding webApp : "forms" with webmodule : "formsweb" for OC4J instance: "forms"
Web application bound successfully.
Stopping OPMN.
OPMN stopped.
Program : $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl completed @ Mon Apr 18 17:42:21 2016
End of $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl : No Errors encountered
4) restore the original system-jazn-data.xml file.
5) run AutoConfig on each Forms server. (Without this step, the forms may not pop up after click due to failure in back-end authentication.)
$ADMIN_SCRIPTS_HOME/adautocfg.sh
Now, you shall see new timestamp on $IAS_ORACLE_HOME/j2ee/forms/applications/forms.ear file. And, adformsctl.sh will start forms successfully.
___________________________________________________________________
~~~~~~~ Additional notes (Real paths are replaced with env variables) ~~~~~~~
Manually deploying EAR file may not always work. Here is what I experienced when execution "$FND_TOP/bin/txkrun.pl -script=CfgOC4JApp" gets failure in instances where adstpall.sh (to stop apps) get Time out error from "adoafmctl.sh stop" and "adformsctl.sh stop".
1. I first saw error in startup log adstrtal.log in a situation when adstrtal.sh called the Perl script to deploy EAR file.
Executing service control script:
$ADMIN_SCRIPTS_HOME/adformsctl.sh start
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh version 120.16.12010000.3
Starting OPMN managed FORMS OC4J instance ...
Calling txkChkFormsDeployment.pl to check whether latest FORMSAPP.EAR is deployed...
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl started @ Wed Jul 14 15:47:39 2016
*** Log File = $LOG_HOME/appl/rgf/TXK/
txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log
File "$IAS_ORACLE_HOME/j2ee/forms/applications/forms/formsweb/WEB-INF/lib/frmsrv.jar" exists. Proceeding to check the size...
=================================================
*** Latest formsapp.ear has NOT been deployed ***
Deploying the latest EAR file...
=================================================
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = $LOG_HOME/appl/rgf/TXK/
txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log
****************************************************
2. Because of the error, adopmnctl.sh may show OC4J: forms is DOWN. When I looked into further, I saw errors in the EAR deployment log:
$ more $LOG_HOME/appl/rgf/TXK/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log
- - - - - - - - -
Application deployed successfully. <== this line (and above) show deployment went good
===========================
Inside manageFormsOc4j()...
===========================
*** OPERATION is stopproc ***
Execute SYSTEM command : $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
opmnctl: stopping opmn managed processes...
opmn id=hostname.domin.com:6230
0 of 1 processes stopped.
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
default_group/forms/default_group/
Error
--> Process (index=1,uid=2087060759,pid=14495)
time out while waiting for a managed process to stop
Log:
$LOG_HOME/ora/10.1.3/opmn/default_group~forms~default_group~1.log
===========================
Inside manageFormsOc4j()...
===========================
*** OPERATION is startproc ***
Execute SYSTEM command : $IAS_ORACLE_HOME/opmn/bin/opmnctl startproc instancename=forms
ERROR DESCRIPTION:
(*******FATAL ERROR*******
PROGRAM : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl($FND_TOP/bin/txkrun.pl)
TIME : Thu Jul 14 15:50:07 2016
FUNCTION: TXK::Process::run [ Level 3 ]
MESSAGES:
Command error: <rc> = 38400, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl startproc instancename=forms
Command error: <rc> = 52224, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
Command error: <rc> = 38400, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
STACK TRACE
TXK::Error::abort('TXK::Error','HASH(0x8942208)') called at $AU_TOP/perl/TXK/Common.pm line 299
TXK::Common::doError('TXK::Process=HASH(0x89eec34)','Command error: <rc> = 38400, <command> = /path/to/EBSQA/...','undef')
called at $AU_TOP/perl/TXK/Common.pm line 314
... ... ... ...
3. Seems the errors are at the last stage of EAR deployment. When I shutdown apps service and started it again by adstrtal.sh, it did not try to deploy the EAR file again and just showed all in status 0. But, when I checked out the login and forms, I could not launch forms.
4. When I tried to deploy the EAR file manually by following the steps above, I got similar errors. Seems the error is from the last stage.
. . . . . . . . .
Stopping all OPMN processes.
OPMN stopped.
OPMN started.
Deplolying Application : "forms" onto OC4J instance: "forms"
Application deployed successfully.
Stopping and starting OC4J instances.
Errors encountered running $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl
*******FATAL ERROR*******
PROGRAM : $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl($FND_TOP/bin/txkrun.pl)
TIME : Wed Jul 6 17:11:06 2016
FUNCTION: TXK::Process::run [ Level 3 ]
MESSAGES:
Command error: <rc> = 52224, <command> =$IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
STACK TRACE
TXK::Error::abort('TXK::Error','HASH(0x91118f4)') called at $AU_TOP/perl/TXK/Common.pm line 299
TXK::Common::doError('TXK::Process=HASH(0x9a86c10)','Command error: <rc> = 52224, <command> = /path/to/EBSQA/a...','undef') called at $AU_TOP/perl/TXK/Common.pm line 314
. . . . . . . . .
5. I did try things around and made forms work before I ran Step 6 below.
6. The deployment errors match the issue in Doc ID 1399491.1, which recommends patch 12965674 as the fix. After I applied it to 10.1.3.5 ORACLE_HOME (for Apache), it not only fixed the errors in EAR deployment but also fixed below Time out error in adstpall.log during apps services shutdown !
You are running adstpall.sh version 120.10.12010000.4
Executing service control script:
$ADMIN_SCRIPTS_HOME/adoafmctl.sh stop
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adoafmctl.sh version 120.8
Stopping OPMN managed OAFM OC4J instance ...
****************************************************
Executing service control script:
$ADMIN_SCRIPTS_HOME/adformsctl.sh stop
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh version 120.16.12010000.3
Stopping OPMN managed FORMS OC4J instance ...
NOTES: Oracle gives a document with a easier way to manually deploy R12 Ear file $ORACLE_HOME/forms/j2ee/formsapp.ear. See Doc ID 1961805.1
Starting OPMN managed FORMS OC4J instance ...
Calling txkChkFormsDeployment.pl to check whether latest FORMSAPP.EAR is deployed...
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl started @ Mon Apr 18 1
6:58:03 2016
*** Log File = $LOG_HOME/appl/rgf/TXK/txkChkFormsDeployment_Mon_Apr_18_16_58_02_2016/txkChkFormsDeployment_Mon_Apr_18_16_58_02_2016.log
File "$IAS_ORACLE_HOME/j2ee/forms/applications/forms/formsweb/WEB-INF/lib/frmsrv.jar" exists. Proceeding to check the size...
=================================================
*** Latest formsapp.ear has NOT been deployed ***
Deploying the latest EAR file...
=================================================
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl completed @ Mon Apr 18 16:59:04 2016
*****************************************
* W A R N I N G *
*****************************************
Error while executing the perl script txkChkFormsDeployment.pl
We have determined that you need to redeploy Forms (using txkChkFormsDeployment.pl).
But could not do it automatically for you, due to some issues.
For details refer the log files.
Follow Note: 397174.1 to redeploy Forms manually.
*****************************************
adformsctl.sh: exiting with status 0
I followed steps in Doc ID 397174.1 to get over this. Without this steps, some forms may not open up.
1) back up file $INST_TOP/ora/10.1.3/j2ee/forms/config/system-jazn-data.xml (for deploying the formsapp.ear file). Note this file's timestamp always gets updated.
2) when the oc4jadmin Password is unknown, search keyword oc4jadmin for pattern in file system-jazn-data.xml , and replace line
<description>OC4J Administrator</description>
<credentials>{903}9VrhYTuhd7DyBJf7J/4KwbSEIlETQJOO</credentials>
with
<description>OC4J Administrator</description>
<credentials>!welcome123</credentials>
Optional: after above change and bounce apps services, below line can verify the password is correct :
$ $AF_JRE_TOP/bin/java -jar $IAS_ORACLE_HOME/j2ee/home/admin_client.jar deployer:oc4j:opmn://hostname.domain.com:6530/forms oc4jadmin welcome123 -validateURI
URI deployer:oc4j:opmn://hostname.domain.com:6530/forms is valid and connected
-- where to get the port number
$ grep 6530 $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml
<port local="6130" remote="6230" request="6530"/>
-- one more validation
$ cd $ORA_CONFIG_HOME/10.1.3/opmn/bin
$ ./opmnctl validate
opmnctl: opmn validation succeeded.
3) stop apps services, and run the Perl script on each Forms tier to deploy the EAR file
$ $FND_TOP/bin/txkrun.pl -script=CfgOC4JApp
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = $LOG_HOME/appl/rgf/TXK/txkCfgOC4JApp_Mon_Apr_18_17_17_52_2016.log
Program : $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl started @ Mon Apr 18 17:17:52 2016
*** Log File = $LOG_HOME/appl/rgf/TXK/txkCfgOC4JApp_Mon_Apr_18_17_17_52_2016.log
Enter Application name for re-deployment ? forms
Enter Oc4j Instance password for re-deployment ? welcome123
Run Autoconfig <Yes/No> ? No
*****************************************************
Required values for starting OC4J instance "forms":
====================================================
s_formsstatus = enabled
s_forms_nprocs = 1 (value should be greater than 0)
Existing values from the context file:
======================================
s_formsstatus = enabled
s_forms_nprocs = 2
----------------------------------------------
*** Values for context variables are VALID ***
----------------------------------------------
*****************************************************
Stopping all OPMN processes.
OPMN stopped.
OPMN started.
Deplolying Application : "forms" onto OC4J instance: "forms"
Application deployed successfully.
Stopping and starting OC4J instances.
Started OC4J instances.
Binding webApp : "forms" with webmodule : "formsweb" for OC4J instance: "forms"
Web application bound successfully.
Stopping OPMN.
OPMN stopped.
Program : $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl completed @ Mon Apr 18 17:42:21 2016
End of $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl : No Errors encountered
4) restore the original system-jazn-data.xml file.
5) run AutoConfig on each Forms server. (Without this step, the forms may not pop up after click due to failure in back-end authentication.)
$ADMIN_SCRIPTS_HOME/adautocfg.sh
Now, you shall see new timestamp on $IAS_ORACLE_HOME/j2ee/forms/applications/forms.ear file. And, adformsctl.sh will start forms successfully.
___________________________________________________________________
~~~~~~~ Additional notes (Real paths are replaced with env variables) ~~~~~~~
Manually deploying EAR file may not always work. Here is what I experienced when execution "$FND_TOP/bin/txkrun.pl -script=CfgOC4JApp" gets failure in instances where adstpall.sh (to stop apps) get Time out error from "adoafmctl.sh stop" and "adformsctl.sh stop".
1. I first saw error in startup log adstrtal.log in a situation when adstrtal.sh called the Perl script to deploy EAR file.
Executing service control script:
$ADMIN_SCRIPTS_HOME/adformsctl.sh start
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh version 120.16.12010000.3
Starting OPMN managed FORMS OC4J instance ...
Calling txkChkFormsDeployment.pl to check whether latest FORMSAPP.EAR is deployed...
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl started @ Wed Jul 14 15:47:39 2016
*** Log File = $LOG_HOME/appl/rgf/TXK/
txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log
File "$IAS_ORACLE_HOME/j2ee/forms/applications/forms/formsweb/WEB-INF/lib/frmsrv.jar" exists. Proceeding to check the size...
=================================================
*** Latest formsapp.ear has NOT been deployed ***
Deploying the latest EAR file...
=================================================
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = $LOG_HOME/appl/rgf/TXK/
txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log
****************************************************
2. Because of the error, adopmnctl.sh may show OC4J: forms is DOWN. When I looked into further, I saw errors in the EAR deployment log:
$ more $LOG_HOME/appl/rgf/TXK/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log
- - - - - - - - -
Application deployed successfully. <== this line (and above) show deployment went good
===========================
Inside manageFormsOc4j()...
===========================
*** OPERATION is stopproc ***
Execute SYSTEM command : $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
opmnctl: stopping opmn managed processes...
opmn id=hostname.domin.com:6230
0 of 1 processes stopped.
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
default_group/forms/default_group/
Error
--> Process (index=1,uid=2087060759,pid=14495)
time out while waiting for a managed process to stop
Log:
$LOG_HOME/ora/10.1.3/opmn/default_group~forms~default_group~1.log
===========================
Inside manageFormsOc4j()...
===========================
*** OPERATION is startproc ***
Execute SYSTEM command : $IAS_ORACLE_HOME/opmn/bin/opmnctl startproc instancename=forms
ERROR DESCRIPTION:
(*******FATAL ERROR*******
PROGRAM : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl($FND_TOP/bin/txkrun.pl)
TIME : Thu Jul 14 15:50:07 2016
FUNCTION: TXK::Process::run [ Level 3 ]
MESSAGES:
Command error: <rc> = 38400, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl startproc instancename=forms
Command error: <rc> = 52224, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
Command error: <rc> = 38400, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
STACK TRACE
TXK::Error::abort('TXK::Error','HASH(0x8942208)') called at $AU_TOP/perl/TXK/Common.pm line 299
TXK::Common::doError('TXK::Process=HASH(0x89eec34)','Command error: <rc> = 38400, <command> = /path/to/EBSQA/...','undef')
called at $AU_TOP/perl/TXK/Common.pm line 314
... ... ... ...
3. Seems the errors are at the last stage of EAR deployment. When I shutdown apps service and started it again by adstrtal.sh, it did not try to deploy the EAR file again and just showed all in status 0. But, when I checked out the login and forms, I could not launch forms.
4. When I tried to deploy the EAR file manually by following the steps above, I got similar errors. Seems the error is from the last stage.
. . . . . . . . .
Stopping all OPMN processes.
OPMN stopped.
OPMN started.
Deplolying Application : "forms" onto OC4J instance: "forms"
Application deployed successfully.
Stopping and starting OC4J instances.
Errors encountered running $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl
*******FATAL ERROR*******
PROGRAM : $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl($FND_TOP/bin/txkrun.pl)
TIME : Wed Jul 6 17:11:06 2016
FUNCTION: TXK::Process::run [ Level 3 ]
MESSAGES:
Command error: <rc> = 52224, <command> =$IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
STACK TRACE
TXK::Error::abort('TXK::Error','HASH(0x91118f4)') called at $AU_TOP/perl/TXK/Common.pm line 299
TXK::Common::doError('TXK::Process=HASH(0x9a86c10)','Command error: <rc> = 52224, <command> = /path/to/EBSQA/a...','undef') called at $AU_TOP/perl/TXK/Common.pm line 314
. . . . . . . . .
5. I did try things around and made forms work before I ran Step 6 below.
6. The deployment errors match the issue in Doc ID 1399491.1, which recommends patch 12965674 as the fix. After I applied it to 10.1.3.5 ORACLE_HOME (for Apache), it not only fixed the errors in EAR deployment but also fixed below Time out error in adstpall.log during apps services shutdown !
You are running adstpall.sh version 120.10.12010000.4
Executing service control script:
$ADMIN_SCRIPTS_HOME/adoafmctl.sh stop
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adoafmctl.sh version 120.8
Stopping OPMN managed OAFM OC4J instance ...
****************************************************
Executing service control script:
$ADMIN_SCRIPTS_HOME/adformsctl.sh stop
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh version 120.16.12010000.3
Stopping OPMN managed FORMS OC4J instance ...
NOTES: Oracle gives a document with a easier way to manually deploy R12 Ear file $ORACLE_HOME/forms/j2ee/formsapp.ear. See Doc ID 1961805.1
Subscribe to:
Comments (Atom)