Concurrent job completed successfully. But the Output can not be opened due to its size.
Error: Page Cannot Be Displayed, Cannot View Output of Large concurrent reports
I tried below actions in R12.1, which may help or may not help. Seems to me it also depends on the resources of client PC. VMware mostly has this trouble when the Output file size is huge.
1. Follow Doc ID 845841.1 (without applying patches)
a) Set "ICX: Session Timeout" profile option value (120 minutes)
Note: s_oc4j_sesstimeout shall match this value
b) Set "OC4J Session Timeout" in orion-web.xml
Note: two orion-web.xml files. One for OC4J, and one for OAFM (which may not be important to the time-out)
$ORA_CONFIG_HOME/10.1.3/j2ee/oacore/application-deployments/oacore/html/orion-web.xml
$ORA_CONFIG_HOME/10.1.3/j2ee/oafm/application-deployments/oafm/webservices/orion-web.xml
increase <session-timeout> from 30 to 120
c) Set Timeout in $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf
Increase Timeout from 300 to 900 (seconds)
2. Doc ID 780081.1 Java Heap Errors when Running An FSG
Section: To Increase Heap Size of oacore process:
In file $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml
<process-type id="oacore" module-id="OC4J" status="enabled" working-dir="$ORACLE_HOME/j2ee/home">
<module-data>
<category id="start-parameters">
<data id="java-options" value="-server -verbose:gc -Xmx1024M -Xms256M -XX:MaxPermSize=160M
oacore java-options meaning:
-Xmx1024M -- Specifies maximum heap memory.(1024 MB)
-Xms256M -- Initial heap memory.(256 MB)
3. Bounce the Concurrent Manager, Apache Server (and maybe database listener).
Wednesday, June 15, 2016
Wednesday, May 25, 2016
opatch and Oracle Home inventory
There are two inventory locations. One is the central inventory defined by file /etc/oraInst.loc (in Linux host). For each ORACLE HOME on the server, there is a local inventory at $ORACLE_HOME/inventory.
opatch lists the information first when applying a patch:
Oracle Home : /path/to/apps/tech_st/10.1.2
Oracle Home Inventory : $ORACLE_HOME/inventory
Central Inventory : /path/to/oraEbsInventory
from : /etc/oraInst.loc
Because the central inventory /path/to/oraEbsInventory may be shared by multiple ORACLE HOMEs on the server, it is more vulnerable to error or corruption. If opatch throws out some error, troubleshoot central inventory first.
If there is a problem with local inventory, it may turn out you have to re-install it. So, do not try to change anything without backing it up.
When I apply a small patch to 10.1.2 ORACLE HOME, it failed with below messages in log file $ORACLE_HOME/.patch_storage/8551790/xxx.log. "opatch lsinventory" also did not show the patch was applied.
Patch 8551790 has been applied successfully
... ... ...
verifying patch
verifying that patch ID is in the Oracle Home inventory
INVENTORY PROBLEM: Patch 8551790 is not present in Oracle Home inventory.
Verifying copy files.
Comparing "/path/to/8551790/files/procbuilder/lib/cus_procbuilder.mk" and "$ORACLE_HOME/procbuilder/lib/cus_procbuilder.mk"
... ... ...
OPATCH_JAVA_ERROR: Patch was not successfully applied.
Verification of the patch failed.
ERROR: OPatch failed as verification of the patch failed.
The real message is "INVENTORY PROBLEM ...". Doc ID 438067.1 (Updating the Inventory Is Failing During Patching) provides a solution to it.
Fix:
I renamed the existing Contents folder and created an empty folder Contents under oraEbsInventory. After I ran "opatch apply" again, the issue was fixed.
The same fix may address error "Inventory check failed: Patch ID is NOT registered in Oracle Home inventory." or "Verification of patch failed: Patch is not found in the Inventory." as well.
That Oracle document also says "opatch lsinventory -detail" accesses the inventory stored in text format in central inventory while "opatch lsinventory" gets inventory info from binary inventory (under oraEbsInventory/Contents ?).
opatch lists the information first when applying a patch:
Oracle Home : /path/to/apps/tech_st/10.1.2
Oracle Home Inventory : $ORACLE_HOME/inventory
Central Inventory : /path/to/oraEbsInventory
from : /etc/oraInst.loc
Because the central inventory /path/to/oraEbsInventory may be shared by multiple ORACLE HOMEs on the server, it is more vulnerable to error or corruption. If opatch throws out some error, troubleshoot central inventory first.
If there is a problem with local inventory, it may turn out you have to re-install it. So, do not try to change anything without backing it up.
When I apply a small patch to 10.1.2 ORACLE HOME, it failed with below messages in log file $ORACLE_HOME/.patch_storage/8551790/xxx.log. "opatch lsinventory" also did not show the patch was applied.
Patch 8551790 has been applied successfully
... ... ...
verifying patch
verifying that patch ID is in the Oracle Home inventory
INVENTORY PROBLEM: Patch 8551790 is not present in Oracle Home inventory.
Verifying copy files.
Comparing "/path/to/8551790/files/procbuilder/lib/cus_procbuilder.mk" and "$ORACLE_HOME/procbuilder/lib/cus_procbuilder.mk"
... ... ...
OPATCH_JAVA_ERROR: Patch was not successfully applied.
Verification of the patch failed.
ERROR: OPatch failed as verification of the patch failed.
The real message is "INVENTORY PROBLEM ...". Doc ID 438067.1 (Updating the Inventory Is Failing During Patching) provides a solution to it.
Fix:
I renamed the existing Contents folder and created an empty folder Contents under oraEbsInventory. After I ran "opatch apply" again, the issue was fixed.
The same fix may address error "Inventory check failed: Patch ID is NOT registered in Oracle Home inventory." or "Verification of patch failed: Patch is not found in the Inventory." as well.
That Oracle document also says "opatch lsinventory -detail" accesses the inventory stored in text format in central inventory while "opatch lsinventory" gets inventory info from binary inventory (under oraEbsInventory/Contents ?).
Sunday, May 22, 2016
R12.1 log locations and log collection
To troubleshoot a technology stack (Web issue or Forms error), Oracle Support asks a lot of logs, before they take a look on the service request (SR) or even they may not know what they are doing. Recently I followed Doc ID 1942889.1 (SRDC - Data Collection Request for EBS: IAS) to collect diagnostic data/logs.
1. follow Doc ID 2026081.1 to run 5 sql scripts to generate 5 .txt files and save them to /tmp.
2. run below line to zip all needed files
$ more zip1942889_1.sh
zip -r /tmp/$TWO_TASK'_'`uname -n`_`date +%m%d%y`_Techstack_info.zip \
/tmp/apps_version_info.txt \
/tmp/fndnodes_oss.txt \
/tmp/web_urls.txt \
/tmp/atg_ver_patches.txt \
/tmp/login_profiles.txt \
$OA_HTML/bin/appsweb* \
$CONTEXT_FILE \
$FND_SECURE/* \
$ORA_CONFIG_HOME/10.1.3/Apache/* \
$ORA_CONFIG_HOME/10.1.3/j2ee/* \
$ORA_CONFIG_HOME/10.1.3/javacache/* \
$ORA_CONFIG_HOME/10.1.3/opmn/*
3. run below line from Doc (Doc ID 601736.1 to get techstack Component Versions (Forms, Http Server, JDK, Framework)
$ cd $FND_TOP/patch/115/bin
$ $ADPERLPRG $FND_TOP/patch/115/bin/TXKScript.pl \
-script=$FND_TOP/patch/115/bin/txkInventory.pl -txktop=$APPLTMP \
-contextfile=$CONTEXT_FILE \
-appspass=appsPWD \
-outfile=/tmp/Report_App_Inventory.html
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** STDOUT = /$LOG_HOME/appl/rgf/TXK/txkInventory_Thu_Jan_14_10_21_05_2016_stdout.log
Reportfile /tmp/Report_App_Inventory.html generated successfully.
Summary on R12.1 log locations. Note env variable $LOG_HOME and $INST_TOP/logs are the same.
-----------------------------------------------------------------------------
-- Startup/Shutdown services (by $ADMIN_SCRIPTS_HOME scripts) log files:
$LOG_HOME/appl/admin/log
----------------------------------------------------------------------------
-- Apache, OC4J and OPMN (see Doc ID 454178.1): runtime
$LOG_HOME/ora/10.1.3/Apache
$LOG_HOME/ora/10.1.3/opmn
$LOG_HOME/ora/10.1.3/j2ee (each sub-folder /forms and /oacore has files application.log and server.log)
$LOG_HOME/ora/10.1.3/opmn/opmn.log
$LOG_HOME/ora/10.1.3/opmn/oacore_default_group_X/oacorestd.err (X depends on s_oacore_nprocs)
$LOG_HOME/ora/10.1.3/opmn/default_group~oacore~default_group~X.log
$LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_X/application.log
$LOG_HOME/ora/10.1.3/j2ee/forms/forms_default_group_X/application.log X depends on s_forms_nprocs
$APPLRGF/javacache.log
- user's DB connection log (and trace if enabled)
$LOG_HOME/ora/10.1.2/network
--------------------------------------------------------------------------
-- Concurrent program log & out
$APPLCSF/log
$APPLCSF/out
($INST_TOP/logs/appl/conc/log <== default)
SQL> SELECT logfile_name, outfile_name, outfile_node_name, last_update_date
FROM apps.FND_CONCURRENT_REQUESTS
WHERE REQUEST_ID = &requestID;
- temporary directories
$APPLPTMP
$APPLTMP <== temporary files for EBS forms.
$INST_TOP/logs/ora/10.1.2/forms ($FORMS_TRACE_DR) <== holds forms runtime files
- Forms Runtime Diagnostics (FRD) tracing AND logging files (Doc ID 438652.1)
----------------------------------------------------------------------------
-- Autoconfig log file:
--
Apps:
$INST_TOP/admin/log/MMDDHHMM/adconfig.log
($INST_TOP/admin/out/MMDDHHMM has run-time files)
DB tier:
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/<MMDDHHMM>/
----------------------------------------------------------------------------
-- adadmin, adpatch and adctrl
$APPL_TOP/admin/${TWO_TASK}/log
----------------------------------------------------------------------------
-- Patching related log files
i) $APPL_TOP/admin/${TWO_TASK}/log/ <== Application Tier -- adpatch log
ii) $ORACLE_HOME/.patch_storage <== Forms (Forms & Reports 10.1.2/Developer) patch
iii) $IAS_ORACLE_HOME/.patch_storage <== Apache (Web Server) patch
----------------------------------------------------------------------------
-- Clone logs:
Pre-clone log files in source instance
Apps: => $INST_TOP/admin/log/StageAppsTier_MMDDHHMM.log
DB: => $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/StageDBTier_MMDDHHMM.log
Clone log files in target instance
Apps : => $INST_TOP/admin/log/ApplyAppsTier_<time>.log
DB: => $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTier_<time>.log
---------------------------------------------------------------------------
-- Output Post Processor (OPP) log from the Application
Getting OPP Log from the application itself
a. System Administrator > Concurrent > Manager > Administer
b. Search for 'Output Post Processor'
c. Click the 'Processes' button
d. Click the Manager Log button. This will open the 'OPP'
1. follow Doc ID 2026081.1 to run 5 sql scripts to generate 5 .txt files and save them to /tmp.
2. run below line to zip all needed files
$ more zip1942889_1.sh
zip -r /tmp/$TWO_TASK'_'`uname -n`_`date +%m%d%y`_Techstack_info.zip \
/tmp/apps_version_info.txt \
/tmp/fndnodes_oss.txt \
/tmp/web_urls.txt \
/tmp/atg_ver_patches.txt \
/tmp/login_profiles.txt \
$OA_HTML/bin/appsweb* \
$CONTEXT_FILE \
$FND_SECURE/* \
$ORA_CONFIG_HOME/10.1.3/Apache/* \
$ORA_CONFIG_HOME/10.1.3/j2ee/* \
$ORA_CONFIG_HOME/10.1.3/javacache/* \
$ORA_CONFIG_HOME/10.1.3/opmn/*
3. run below line from Doc (Doc ID 601736.1 to get techstack Component Versions (Forms, Http Server, JDK, Framework)
$ cd $FND_TOP/patch/115/bin
$ $ADPERLPRG $FND_TOP/patch/115/bin/TXKScript.pl \
-script=$FND_TOP/patch/115/bin/txkInventory.pl -txktop=$APPLTMP \
-contextfile=$CONTEXT_FILE \
-appspass=appsPWD \
-outfile=/tmp/Report_App_Inventory.html
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** STDOUT = /$LOG_HOME/appl/rgf/TXK/txkInventory_Thu_Jan_14_10_21_05_2016_stdout.log
Reportfile /tmp/Report_App_Inventory.html generated successfully.
Summary on R12.1 log locations. Note env variable $LOG_HOME and $INST_TOP/logs are the same.
-----------------------------------------------------------------------------
-- Startup/Shutdown services (by $ADMIN_SCRIPTS_HOME scripts) log files:
$LOG_HOME/appl/admin/log
----------------------------------------------------------------------------
-- Apache, OC4J and OPMN (see Doc ID 454178.1): runtime
$LOG_HOME/ora/10.1.3/Apache
$LOG_HOME/ora/10.1.3/opmn
$LOG_HOME/ora/10.1.3/j2ee (each sub-folder /forms and /oacore has files application.log and server.log)
$LOG_HOME/ora/10.1.3/opmn/opmn.log
$LOG_HOME/ora/10.1.3/opmn/oacore_default_group_X/oacorestd.err (X depends on s_oacore_nprocs)
$LOG_HOME/ora/10.1.3/opmn/default_group~oacore~default_group~X.log
$LOG_HOME/ora/10.1.3/j2ee/oacore/oacore_default_group_X/application.log
$LOG_HOME/ora/10.1.3/j2ee/forms/forms_default_group_X/application.log X depends on s_forms_nprocs
$APPLRGF/javacache.log
- user's DB connection log (and trace if enabled)
$LOG_HOME/ora/10.1.2/network
--------------------------------------------------------------------------
-- Concurrent program log & out
$APPLCSF/log
$APPLCSF/out
($INST_TOP/logs/appl/conc/log <== default)
SQL> SELECT logfile_name, outfile_name, outfile_node_name, last_update_date
FROM apps.FND_CONCURRENT_REQUESTS
WHERE REQUEST_ID = &requestID;
- temporary directories
$APPLPTMP
$APPLTMP <== temporary files for EBS forms.
$INST_TOP/logs/ora/10.1.2/forms ($FORMS_TRACE_DR) <== holds forms runtime files
- Forms Runtime Diagnostics (FRD) tracing AND logging files (Doc ID 438652.1)
----------------------------------------------------------------------------
-- Autoconfig log file:
--
Apps:
$INST_TOP/admin/log/MMDDHHMM/adconfig.log
($INST_TOP/admin/out/MMDDHHMM has run-time files)
DB tier:
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/<MMDDHHMM>/
----------------------------------------------------------------------------
-- adadmin, adpatch and adctrl
$APPL_TOP/admin/${TWO_TASK}/log
----------------------------------------------------------------------------
-- Patching related log files
i) $APPL_TOP/admin/${TWO_TASK}/log/ <== Application Tier -- adpatch log
ii) $ORACLE_HOME/.patch_storage <== Forms (Forms & Reports 10.1.2/Developer) patch
iii) $IAS_ORACLE_HOME/.patch_storage <== Apache (Web Server) patch
----------------------------------------------------------------------------
-- Clone logs:
Pre-clone log files in source instance
Apps: => $INST_TOP/admin/log/StageAppsTier_MMDDHHMM.log
DB: => $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/StageDBTier_MMDDHHMM.log
Clone log files in target instance
Apps : => $INST_TOP/admin/log/ApplyAppsTier_<time>.log
DB: => $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTier_<time>.log
---------------------------------------------------------------------------
-- Output Post Processor (OPP) log from the Application
Getting OPP Log from the application itself
a. System Administrator > Concurrent > Manager > Administer
b. Search for 'Output Post Processor'
c. Click the 'Processes' button
d. Click the Manager Log button. This will open the 'OPP'
Saturday, April 23, 2016
Manually deploy EAR file
After a patch (such as JE patch 21950151) is applied or adadmin generates new JAR (Java Archive) file, the apps startup script adformsctl.sh may deploy forms EAR file. You may see a Warning (with status 0) in log $LOG_HOME/appl/admin/log/adstrtal.log.
Starting OPMN managed FORMS OC4J instance ...
Calling txkChkFormsDeployment.pl to check whether latest FORMSAPP.EAR is deployed...
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl started @ Mon Apr 18 1
6:58:03 2016
*** Log File = $LOG_HOME/appl/rgf/TXK/txkChkFormsDeployment_Mon_Apr_18_16_58_02_2016/txkChkFormsDeployment_Mon_Apr_18_16_58_02_2016.log
File "$IAS_ORACLE_HOME/j2ee/forms/applications/forms/formsweb/WEB-INF/lib/frmsrv.jar" exists. Proceeding to check the size...
=================================================
*** Latest formsapp.ear has NOT been deployed ***
Deploying the latest EAR file...
=================================================
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl completed @ Mon Apr 18 16:59:04 2016
*****************************************
* W A R N I N G *
*****************************************
Error while executing the perl script txkChkFormsDeployment.pl
We have determined that you need to redeploy Forms (using txkChkFormsDeployment.pl).
But could not do it automatically for you, due to some issues.
For details refer the log files.
Follow Note: 397174.1 to redeploy Forms manually.
*****************************************
adformsctl.sh: exiting with status 0
I followed steps in Doc ID 397174.1 to get over this. Without this steps, some forms may not open up.
1) back up file $INST_TOP/ora/10.1.3/j2ee/forms/config/system-jazn-data.xml (for deploying the formsapp.ear file). Note this file's timestamp always gets updated.
2) when the oc4jadmin Password is unknown, search keyword oc4jadmin for pattern in file system-jazn-data.xml , and replace line
<description>OC4J Administrator</description>
<credentials>{903}9VrhYTuhd7DyBJf7J/4KwbSEIlETQJOO</credentials>
with
<description>OC4J Administrator</description>
<credentials>!welcome123</credentials>
Optional: after above change and bounce apps services, below line can verify the password is correct :
$ $AF_JRE_TOP/bin/java -jar $IAS_ORACLE_HOME/j2ee/home/admin_client.jar deployer:oc4j:opmn://hostname.domain.com:6530/forms oc4jadmin welcome123 -validateURI
URI deployer:oc4j:opmn://hostname.domain.com:6530/forms is valid and connected
-- where to get the port number
$ grep 6530 $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml
<port local="6130" remote="6230" request="6530"/>
-- one more validation
$ cd $ORA_CONFIG_HOME/10.1.3/opmn/bin
$ ./opmnctl validate
opmnctl: opmn validation succeeded.
3) stop apps services, and run the Perl script on each Forms tier to deploy the EAR file
$ $FND_TOP/bin/txkrun.pl -script=CfgOC4JApp
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = $LOG_HOME/appl/rgf/TXK/txkCfgOC4JApp_Mon_Apr_18_17_17_52_2016.log
Program : $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl started @ Mon Apr 18 17:17:52 2016
*** Log File = $LOG_HOME/appl/rgf/TXK/txkCfgOC4JApp_Mon_Apr_18_17_17_52_2016.log
Enter Application name for re-deployment ? forms
Enter Oc4j Instance password for re-deployment ? welcome123
Run Autoconfig <Yes/No> ? No
*****************************************************
Required values for starting OC4J instance "forms":
====================================================
s_formsstatus = enabled
s_forms_nprocs = 1 (value should be greater than 0)
Existing values from the context file:
======================================
s_formsstatus = enabled
s_forms_nprocs = 2
----------------------------------------------
*** Values for context variables are VALID ***
----------------------------------------------
*****************************************************
Stopping all OPMN processes.
OPMN stopped.
OPMN started.
Deplolying Application : "forms" onto OC4J instance: "forms"
Application deployed successfully.
Stopping and starting OC4J instances.
Started OC4J instances.
Binding webApp : "forms" with webmodule : "formsweb" for OC4J instance: "forms"
Web application bound successfully.
Stopping OPMN.
OPMN stopped.
Program : $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl completed @ Mon Apr 18 17:42:21 2016
End of $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl : No Errors encountered
4) restore the original system-jazn-data.xml file.
5) run AutoConfig on each Forms server. (Without this step, the forms may not pop up after click due to failure in back-end authentication.)
$ADMIN_SCRIPTS_HOME/adautocfg.sh
Now, you shall see new timestamp on $IAS_ORACLE_HOME/j2ee/forms/applications/forms.ear file. And, adformsctl.sh will start forms successfully.
___________________________________________________________________
~~~~~~~ Additional notes (Real paths are replaced with env variables) ~~~~~~~
Manually deploying EAR file may not always work. Here is what I experienced when execution "$FND_TOP/bin/txkrun.pl -script=CfgOC4JApp" gets failure in instances where adstpall.sh (to stop apps) get Time out error from "adoafmctl.sh stop" and "adformsctl.sh stop".
1. I first saw error in startup log adstrtal.log in a situation when adstrtal.sh called the Perl script to deploy EAR file.
Executing service control script:
$ADMIN_SCRIPTS_HOME/adformsctl.sh start
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh version 120.16.12010000.3
Starting OPMN managed FORMS OC4J instance ...
Calling txkChkFormsDeployment.pl to check whether latest FORMSAPP.EAR is deployed...
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl started @ Wed Jul 14 15:47:39 2016
*** Log File = $LOG_HOME/appl/rgf/TXK/
txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log
File "$IAS_ORACLE_HOME/j2ee/forms/applications/forms/formsweb/WEB-INF/lib/frmsrv.jar" exists. Proceeding to check the size...
=================================================
*** Latest formsapp.ear has NOT been deployed ***
Deploying the latest EAR file...
=================================================
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = $LOG_HOME/appl/rgf/TXK/
txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log
****************************************************
2. Because of the error, adopmnctl.sh may show OC4J: forms is DOWN. When I looked into further, I saw errors in the EAR deployment log:
$ more $LOG_HOME/appl/rgf/TXK/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log
- - - - - - - - -
Application deployed successfully. <== this line (and above) show deployment went good
===========================
Inside manageFormsOc4j()...
===========================
*** OPERATION is stopproc ***
Execute SYSTEM command : $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
opmnctl: stopping opmn managed processes...
opmn id=hostname.domin.com:6230
0 of 1 processes stopped.
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
default_group/forms/default_group/
Error
--> Process (index=1,uid=2087060759,pid=14495)
time out while waiting for a managed process to stop
Log:
$LOG_HOME/ora/10.1.3/opmn/default_group~forms~default_group~1.log
===========================
Inside manageFormsOc4j()...
===========================
*** OPERATION is startproc ***
Execute SYSTEM command : $IAS_ORACLE_HOME/opmn/bin/opmnctl startproc instancename=forms
ERROR DESCRIPTION:
(*******FATAL ERROR*******
PROGRAM : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl($FND_TOP/bin/txkrun.pl)
TIME : Thu Jul 14 15:50:07 2016
FUNCTION: TXK::Process::run [ Level 3 ]
MESSAGES:
Command error: <rc> = 38400, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl startproc instancename=forms
Command error: <rc> = 52224, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
Command error: <rc> = 38400, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
STACK TRACE
TXK::Error::abort('TXK::Error','HASH(0x8942208)') called at $AU_TOP/perl/TXK/Common.pm line 299
TXK::Common::doError('TXK::Process=HASH(0x89eec34)','Command error: <rc> = 38400, <command> = /path/to/EBSQA/...','undef')
called at $AU_TOP/perl/TXK/Common.pm line 314
... ... ... ...
3. Seems the errors are at the last stage of EAR deployment. When I shutdown apps service and started it again by adstrtal.sh, it did not try to deploy the EAR file again and just showed all in status 0. But, when I checked out the login and forms, I could not launch forms.
4. When I tried to deploy the EAR file manually by following the steps above, I got similar errors. Seems the error is from the last stage.
. . . . . . . . .
Stopping all OPMN processes.
OPMN stopped.
OPMN started.
Deplolying Application : "forms" onto OC4J instance: "forms"
Application deployed successfully.
Stopping and starting OC4J instances.
Errors encountered running $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl
*******FATAL ERROR*******
PROGRAM : $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl($FND_TOP/bin/txkrun.pl)
TIME : Wed Jul 6 17:11:06 2016
FUNCTION: TXK::Process::run [ Level 3 ]
MESSAGES:
Command error: <rc> = 52224, <command> =$IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
STACK TRACE
TXK::Error::abort('TXK::Error','HASH(0x91118f4)') called at $AU_TOP/perl/TXK/Common.pm line 299
TXK::Common::doError('TXK::Process=HASH(0x9a86c10)','Command error: <rc> = 52224, <command> = /path/to/EBSQA/a...','undef') called at $AU_TOP/perl/TXK/Common.pm line 314
. . . . . . . . .
5. I did try things around and made forms work before I ran Step 6 below.
6. The deployment errors match the issue in Doc ID 1399491.1, which recommends patch 12965674 as the fix. After I applied it to 10.1.3.5 ORACLE_HOME (for Apache), it not only fixed the errors in EAR deployment but also fixed below Time out error in adstpall.log during apps services shutdown !
You are running adstpall.sh version 120.10.12010000.4
Executing service control script:
$ADMIN_SCRIPTS_HOME/adoafmctl.sh stop
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adoafmctl.sh version 120.8
Stopping OPMN managed OAFM OC4J instance ...
****************************************************
Executing service control script:
$ADMIN_SCRIPTS_HOME/adformsctl.sh stop
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh version 120.16.12010000.3
Stopping OPMN managed FORMS OC4J instance ...
NOTES: Oracle gives a document with a easier way to manually deploy R12 Ear file $ORACLE_HOME/forms/j2ee/formsapp.ear. See Doc ID 1961805.1
Starting OPMN managed FORMS OC4J instance ...
Calling txkChkFormsDeployment.pl to check whether latest FORMSAPP.EAR is deployed...
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl started @ Mon Apr 18 1
6:58:03 2016
*** Log File = $LOG_HOME/appl/rgf/TXK/txkChkFormsDeployment_Mon_Apr_18_16_58_02_2016/txkChkFormsDeployment_Mon_Apr_18_16_58_02_2016.log
File "$IAS_ORACLE_HOME/j2ee/forms/applications/forms/formsweb/WEB-INF/lib/frmsrv.jar" exists. Proceeding to check the size...
=================================================
*** Latest formsapp.ear has NOT been deployed ***
Deploying the latest EAR file...
=================================================
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl completed @ Mon Apr 18 16:59:04 2016
*****************************************
* W A R N I N G *
*****************************************
Error while executing the perl script txkChkFormsDeployment.pl
We have determined that you need to redeploy Forms (using txkChkFormsDeployment.pl).
But could not do it automatically for you, due to some issues.
For details refer the log files.
Follow Note: 397174.1 to redeploy Forms manually.
*****************************************
adformsctl.sh: exiting with status 0
I followed steps in Doc ID 397174.1 to get over this. Without this steps, some forms may not open up.
1) back up file $INST_TOP/ora/10.1.3/j2ee/forms/config/system-jazn-data.xml (for deploying the formsapp.ear file). Note this file's timestamp always gets updated.
2) when the oc4jadmin Password is unknown, search keyword oc4jadmin for pattern in file system-jazn-data.xml , and replace line
<description>OC4J Administrator</description>
<credentials>{903}9VrhYTuhd7DyBJf7J/4KwbSEIlETQJOO</credentials>
with
<description>OC4J Administrator</description>
<credentials>!welcome123</credentials>
Optional: after above change and bounce apps services, below line can verify the password is correct :
$ $AF_JRE_TOP/bin/java -jar $IAS_ORACLE_HOME/j2ee/home/admin_client.jar deployer:oc4j:opmn://hostname.domain.com:6530/forms oc4jadmin welcome123 -validateURI
URI deployer:oc4j:opmn://hostname.domain.com:6530/forms is valid and connected
-- where to get the port number
$ grep 6530 $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml
<port local="6130" remote="6230" request="6530"/>
-- one more validation
$ cd $ORA_CONFIG_HOME/10.1.3/opmn/bin
$ ./opmnctl validate
opmnctl: opmn validation succeeded.
3) stop apps services, and run the Perl script on each Forms tier to deploy the EAR file
$ $FND_TOP/bin/txkrun.pl -script=CfgOC4JApp
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = $LOG_HOME/appl/rgf/TXK/txkCfgOC4JApp_Mon_Apr_18_17_17_52_2016.log
Program : $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl started @ Mon Apr 18 17:17:52 2016
*** Log File = $LOG_HOME/appl/rgf/TXK/txkCfgOC4JApp_Mon_Apr_18_17_17_52_2016.log
Enter Application name for re-deployment ? forms
Enter Oc4j Instance password for re-deployment ? welcome123
Run Autoconfig <Yes/No> ? No
*****************************************************
Required values for starting OC4J instance "forms":
====================================================
s_formsstatus = enabled
s_forms_nprocs = 1 (value should be greater than 0)
Existing values from the context file:
======================================
s_formsstatus = enabled
s_forms_nprocs = 2
----------------------------------------------
*** Values for context variables are VALID ***
----------------------------------------------
*****************************************************
Stopping all OPMN processes.
OPMN stopped.
OPMN started.
Deplolying Application : "forms" onto OC4J instance: "forms"
Application deployed successfully.
Stopping and starting OC4J instances.
Started OC4J instances.
Binding webApp : "forms" with webmodule : "formsweb" for OC4J instance: "forms"
Web application bound successfully.
Stopping OPMN.
OPMN stopped.
Program : $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl completed @ Mon Apr 18 17:42:21 2016
End of $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl : No Errors encountered
4) restore the original system-jazn-data.xml file.
5) run AutoConfig on each Forms server. (Without this step, the forms may not pop up after click due to failure in back-end authentication.)
$ADMIN_SCRIPTS_HOME/adautocfg.sh
Now, you shall see new timestamp on $IAS_ORACLE_HOME/j2ee/forms/applications/forms.ear file. And, adformsctl.sh will start forms successfully.
___________________________________________________________________
~~~~~~~ Additional notes (Real paths are replaced with env variables) ~~~~~~~
Manually deploying EAR file may not always work. Here is what I experienced when execution "$FND_TOP/bin/txkrun.pl -script=CfgOC4JApp" gets failure in instances where adstpall.sh (to stop apps) get Time out error from "adoafmctl.sh stop" and "adformsctl.sh stop".
1. I first saw error in startup log adstrtal.log in a situation when adstrtal.sh called the Perl script to deploy EAR file.
Executing service control script:
$ADMIN_SCRIPTS_HOME/adformsctl.sh start
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh version 120.16.12010000.3
Starting OPMN managed FORMS OC4J instance ...
Calling txkChkFormsDeployment.pl to check whether latest FORMSAPP.EAR is deployed...
Program : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl started @ Wed Jul 14 15:47:39 2016
*** Log File = $LOG_HOME/appl/rgf/TXK/
txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log
File "$IAS_ORACLE_HOME/j2ee/forms/applications/forms/formsweb/WEB-INF/lib/frmsrv.jar" exists. Proceeding to check the size...
=================================================
*** Latest formsapp.ear has NOT been deployed ***
Deploying the latest EAR file...
=================================================
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = $LOG_HOME/appl/rgf/TXK/
txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log
****************************************************
2. Because of the error, adopmnctl.sh may show OC4J: forms is DOWN. When I looked into further, I saw errors in the EAR deployment log:
$ more $LOG_HOME/appl/rgf/TXK/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016/txkChkFormsDeployment_Thu_Jul_14_15_47_39_2016.log
- - - - - - - - -
Application deployed successfully. <== this line (and above) show deployment went good
===========================
Inside manageFormsOc4j()...
===========================
*** OPERATION is stopproc ***
Execute SYSTEM command : $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
opmnctl: stopping opmn managed processes...
opmn id=hostname.domin.com:6230
0 of 1 processes stopped.
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
default_group/forms/default_group/
Error
--> Process (index=1,uid=2087060759,pid=14495)
time out while waiting for a managed process to stop
Log:
$LOG_HOME/ora/10.1.3/opmn/default_group~forms~default_group~1.log
===========================
Inside manageFormsOc4j()...
===========================
*** OPERATION is startproc ***
Execute SYSTEM command : $IAS_ORACLE_HOME/opmn/bin/opmnctl startproc instancename=forms
ERROR DESCRIPTION:
(*******FATAL ERROR*******
PROGRAM : $FND_TOP/patch/115/bin/txkChkFormsDeployment.pl($FND_TOP/bin/txkrun.pl)
TIME : Thu Jul 14 15:50:07 2016
FUNCTION: TXK::Process::run [ Level 3 ]
MESSAGES:
Command error: <rc> = 38400, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl startproc instancename=forms
Command error: <rc> = 52224, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
Command error: <rc> = 38400, <command> = $IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
STACK TRACE
TXK::Error::abort('TXK::Error','HASH(0x8942208)') called at $AU_TOP/perl/TXK/Common.pm line 299
TXK::Common::doError('TXK::Process=HASH(0x89eec34)','Command error: <rc> = 38400, <command> = /path/to/EBSQA/...','undef')
called at $AU_TOP/perl/TXK/Common.pm line 314
... ... ... ...
3. Seems the errors are at the last stage of EAR deployment. When I shutdown apps service and started it again by adstrtal.sh, it did not try to deploy the EAR file again and just showed all in status 0. But, when I checked out the login and forms, I could not launch forms.
4. When I tried to deploy the EAR file manually by following the steps above, I got similar errors. Seems the error is from the last stage.
. . . . . . . . .
Stopping all OPMN processes.
OPMN stopped.
OPMN started.
Deplolying Application : "forms" onto OC4J instance: "forms"
Application deployed successfully.
Stopping and starting OC4J instances.
Errors encountered running $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl
*******FATAL ERROR*******
PROGRAM : $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl($FND_TOP/bin/txkrun.pl)
TIME : Wed Jul 6 17:11:06 2016
FUNCTION: TXK::Process::run [ Level 3 ]
MESSAGES:
Command error: <rc> = 52224, <command> =$IAS_ORACLE_HOME/opmn/bin/opmnctl stopproc instancename=forms
STACK TRACE
TXK::Error::abort('TXK::Error','HASH(0x91118f4)') called at $AU_TOP/perl/TXK/Common.pm line 299
TXK::Common::doError('TXK::Process=HASH(0x9a86c10)','Command error: <rc> = 52224, <command> = /path/to/EBSQA/a...','undef') called at $AU_TOP/perl/TXK/Common.pm line 314
. . . . . . . . .
5. I did try things around and made forms work before I ran Step 6 below.
6. The deployment errors match the issue in Doc ID 1399491.1, which recommends patch 12965674 as the fix. After I applied it to 10.1.3.5 ORACLE_HOME (for Apache), it not only fixed the errors in EAR deployment but also fixed below Time out error in adstpall.log during apps services shutdown !
You are running adstpall.sh version 120.10.12010000.4
Executing service control script:
$ADMIN_SCRIPTS_HOME/adoafmctl.sh stop
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adoafmctl.sh version 120.8
Stopping OPMN managed OAFM OC4J instance ...
****************************************************
Executing service control script:
$ADMIN_SCRIPTS_HOME/adformsctl.sh stop
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
ERROR : Timed out( 100000 ): Interrupted Exception
You are running adformsctl.sh version 120.16.12010000.3
Stopping OPMN managed FORMS OC4J instance ...
NOTES: Oracle gives a document with a easier way to manually deploy R12 Ear file $ORACLE_HOME/forms/j2ee/formsapp.ear. See Doc ID 1961805.1
Monday, April 4, 2016
Custom EBS password complexity
When I tried to create a new user in a non-prod R12.1 instance, I got below error:
Unable to load java class oracle.apps.custom.security.XXXXXPasswordValidation specified profile option SIGNON_PASSWORD_CUSTOM. Please verify that the class exists and that it implements the java interface oracle.apps.custom.security.XXXXXPasswordValidation
I checked profile option Signon Password Custom which had "oracle.apps.custom.security.XXXXXPasswordValidation" on site level. I ran below query and did not see the custom Java class in the database. So that error makes sense because it was not uploaded to the database.
SQL> SELECT dbms_java.longname(object_name), status
FROM user_objects
WHERE object_type = 'JAVA CLASS'
AND dbms_java.longname(object_name) like '%PasswordValidation';
DBMS_JAVA.LONGNAME(OBJECT_NAME) STATUS
oracle/apps/fnd/security/PasswordValidation VALID
oracle/apps/fnd/security/AppsPasswordValidation VALID
Fortunately I found the Java files are still saved in the file system. So I uploaded it to the database:
$ cd $JAVA_TOP/oracle/apps/custom/security
$ ls -al
-rwxrwxrwx 1 trainer users 1927 Nov 8 2013 XXXXXPasswordValidation.class
-rwxrwxrwx 1 trainer users 2629 Feb 6 2012 XXXXXPasswordValidation.java
$ loadjava -user apps/appsPWD -verbose -resolve -force XXXXXPasswordValidation.class
arguments: '-user' 'apps/appsPWD' '-verbose' '-resolve' '-force' 'XXXXXPasswordValidation.class'
creating : class oracle/apps/custom/security/XXXXXPasswordValidation
loading : class oracle/apps/custom/security/XXXXXPasswordValidation
resolving: class oracle/apps/custom/security/XXXXXPasswordValidation
SQL> SELECT dbms_java.longname(object_name), status
FROM user_objects
WHERE object_type = 'JAVA CLASS'
AND dbms_java.longname(object_name) like '%PasswordValidation';
DBMS_JAVA.LONGNAME(OBJECT_NAME) STATUS
oracle/apps/custom/security/XXXXXPasswordValidation VALID
oracle/apps/fnd/security/PasswordValidation VALID
oracle/apps/fnd/security/AppsPasswordValidation VALID
After those steps, I was able to create EBS user if the password meets the complexity defined in the custom Java file.
Since oracle/apps/fnd/security/PasswordValidation exists in the database, I assumed it is an Oracle standard class and so I changed profile option Signon Password Custom from "oracle.apps.custom.security.XXXXXPasswordValidation" to "oracle.apps.fnd.security.PasswordValidation". But when I tried to create a user now , I got error:
Unable to load Java class oracle.apps.fnd.security.PasswordValidation specified in profile option SIGNON_PASSWORD_CUSTOM. Please verify that the class exists and that it implements the Java interface oracle.apps.fnd.security.PasswordValidation.
As an exercise, I re-uploaded the Java class to the database:
$ cd $JAVA_TOP/oracle/apps/fnd/security
$ ls -al PasswordValidation.class
-rwxrwxrwx 1 trainer users 673 Jan 3 2007 PasswordValidation.class
$ loadjava -user apps/appsPWD -verbose -resolve -force PasswordValidation.class
arguments: '-user' 'apps/appsPWD' '-verbose' '-resolve' '-force' 'PasswordValidation.class'
creating : class oracle/apps/fnd/security/PasswordValidation
loading : class oracle/apps/fnd/security/PasswordValidation
resolving: class oracle/apps/fnd/security/PasswordValidation
Interestingly, that made it and other Java classes INVALID in the database.I had to run adadmin to compile APPS schema to make them VALID again. Seems that oracle.apps.fnd.security.PasswordValidation can not be used or it is a wrong value to put in profile option Signon Password Custom.
The custom Java code looks like:
$ view XXXXXPasswordValidation.java
package oracle.apps.custom.security;
import oracle.apps.fnd.common.VersionInfo;
import oracle.apps.fnd.security.*;
import java.*;
// Referenced classes of package oracle.apps.fnd.security:
// PasswordValidation
public class XXXXXPasswordValidation implements PasswordValidation
{
due to copy right, no detail here...
}
Updates:
Profile option "Signon Password Failure Limit" defines maximum number of tries allowed before user account becomes locked.
Profile option "Signon Password No Reuse" defines minimum number of days that a user must wait before being allowed to reuse a password.
Unable to load java class oracle.apps.custom.security.XXXXXPasswordValidation specified profile option SIGNON_PASSWORD_CUSTOM. Please verify that the class exists and that it implements the java interface oracle.apps.custom.security.XXXXXPasswordValidation
I checked profile option Signon Password Custom which had "oracle.apps.custom.security.XXXXXPasswordValidation" on site level. I ran below query and did not see the custom Java class in the database. So that error makes sense because it was not uploaded to the database.
SQL> SELECT dbms_java.longname(object_name), status
FROM user_objects
WHERE object_type = 'JAVA CLASS'
AND dbms_java.longname(object_name) like '%PasswordValidation';
DBMS_JAVA.LONGNAME(OBJECT_NAME) STATUS
oracle/apps/fnd/security/PasswordValidation VALID
oracle/apps/fnd/security/AppsPasswordValidation VALID
Fortunately I found the Java files are still saved in the file system. So I uploaded it to the database:
$ cd $JAVA_TOP/oracle/apps/custom/security
$ ls -al
-rwxrwxrwx 1 trainer users 1927 Nov 8 2013 XXXXXPasswordValidation.class
-rwxrwxrwx 1 trainer users 2629 Feb 6 2012 XXXXXPasswordValidation.java
$ loadjava -user apps/appsPWD -verbose -resolve -force XXXXXPasswordValidation.class
arguments: '-user' 'apps/appsPWD' '-verbose' '-resolve' '-force' 'XXXXXPasswordValidation.class'
creating : class oracle/apps/custom/security/XXXXXPasswordValidation
loading : class oracle/apps/custom/security/XXXXXPasswordValidation
resolving: class oracle/apps/custom/security/XXXXXPasswordValidation
SQL> SELECT dbms_java.longname(object_name), status
FROM user_objects
WHERE object_type = 'JAVA CLASS'
AND dbms_java.longname(object_name) like '%PasswordValidation';
DBMS_JAVA.LONGNAME(OBJECT_NAME) STATUS
oracle/apps/custom/security/XXXXXPasswordValidation VALID
oracle/apps/fnd/security/PasswordValidation VALID
oracle/apps/fnd/security/AppsPasswordValidation VALID
After those steps, I was able to create EBS user if the password meets the complexity defined in the custom Java file.
Since oracle/apps/fnd/security/PasswordValidation exists in the database, I assumed it is an Oracle standard class and so I changed profile option Signon Password Custom from "oracle.apps.custom.security.XXXXXPasswordValidation" to "oracle.apps.fnd.security.PasswordValidation". But when I tried to create a user now , I got error:
Unable to load Java class oracle.apps.fnd.security.PasswordValidation specified in profile option SIGNON_PASSWORD_CUSTOM. Please verify that the class exists and that it implements the Java interface oracle.apps.fnd.security.PasswordValidation.
As an exercise, I re-uploaded the Java class to the database:
$ cd $JAVA_TOP/oracle/apps/fnd/security
$ ls -al PasswordValidation.class
-rwxrwxrwx 1 trainer users 673 Jan 3 2007 PasswordValidation.class
$ loadjava -user apps/appsPWD -verbose -resolve -force PasswordValidation.class
arguments: '-user' 'apps/appsPWD' '-verbose' '-resolve' '-force' 'PasswordValidation.class'
creating : class oracle/apps/fnd/security/PasswordValidation
loading : class oracle/apps/fnd/security/PasswordValidation
resolving: class oracle/apps/fnd/security/PasswordValidation
Interestingly, that made it and other Java classes INVALID in the database.I had to run adadmin to compile APPS schema to make them VALID again. Seems that oracle.apps.fnd.security.PasswordValidation can not be used or it is a wrong value to put in profile option Signon Password Custom.
The custom Java code looks like:
$ view XXXXXPasswordValidation.java
package oracle.apps.custom.security;
import oracle.apps.fnd.common.VersionInfo;
import oracle.apps.fnd.security.*;
import java.*;
// Referenced classes of package oracle.apps.fnd.security:
// PasswordValidation
public class XXXXXPasswordValidation implements PasswordValidation
{
due to copy right, no detail here...
}
Updates:
Profile option "Signon Password Failure Limit" defines maximum number of tries allowed before user account becomes locked.
Profile option "Signon Password No Reuse" defines minimum number of days that a user must wait before being allowed to reuse a password.
Subscribe to:
Comments (Atom)